system

Abstract

We provide the system. [Solution] A sensor means that acquires state information in response to user operations, An analysis means for detecting an anomaly based on state information acquired from the aforementioned sensor means, Information generation means that generates information related to repairs based on the abnormality detected by the analysis means, Information display means for presenting the generated information to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, when diagnosing a bicycle failure, the user himself / herself had to identify the failure location. However, due to lack of knowledge, accurate diagnosis was difficult, and as a result, unnecessary repairs and costs often occurred. Also, the time and effort required to take the bicycle to a repair shop were a significant burden. The present invention aims to solve such problems, and an object thereof is to enable a user to easily identify a bicycle failure location and immediately grasp whether repair is possible and what measures are necessary.

Means for Solving the Problems

[0005] This invention incorporates a sensor means for acquiring bicycle status information in response to user operation, and uses an analysis means for detecting abnormalities from the collected information. Furthermore, by combining this with an information generation means for generating repair information based on the detected abnormalities, a system is realized that presents the user with appropriate repair methods and urgency levels. With this configuration, the user can quickly and accurately identify the location of the bicycle malfunction and take appropriate action.

[0006] A "sensor means" is a device or method for detecting the state of a bicycle through user operation and acquiring necessary information.

[0007] "Analysis means" refers to a device or method for processing state information acquired by sensor means and detecting abnormalities occurring in a bicycle based on that information.

[0008] "Information generation means" refers to a device or method for generating information necessary for repair based on anomalies identified by analysis means and providing it to the user.

[0009] "Information display means" refers to a device or method for presenting generated repair information to the user in a visual, auditory, or other format.

[0010] An "abnormality" refers to a condition or sign that impairs the normal operation of a bicycle, and indicates a situation that requires diagnosis or repair.

[0011] "Inference means" refers to a device or method for identifying the location of an anomaly and the priority of repairs based on the detected anomaly. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

MODE FOR CARRYING OUT THE INVENTION

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] The system of the present invention is designed to enable users to efficiently diagnose bicycle malfunctions and understand the need for and methods of repair. Specific embodiments of this system are described below.

[0034] First, the user installs an application on their smartphone or tablet to diagnose their bicycle. When the user launches the app, the device prompts them to activate the sensor device installed on the bicycle. This sensor device captures the physical condition of the bicycle, for example, by including a camera and microphone.

[0035] The terminal collects data from sensors in real time and constantly monitors for any abnormalities. Once data is collected, it is transmitted to a server via the network. The server uses a generative AI model to analyze the received data and identify which part of the bicycle is experiencing problems. This process includes pattern recognition of image and audio data supplied by the sensors.

[0036] When an anomaly is detected, the server processes the information to generate repair instructions and estimated repair costs corresponding to the location of the anomaly. The generated results are returned to the terminal in a structured format, and the terminal presents these results to the user within the application. The user can review the diagnostic results in detail on the screen and understand the recommended repair items and their urgency. If necessary, they can also search for information on the nearest repair shop and make a reservation through the application.

[0037] For example, if a bicycle brake makes an unusual noise, the user can use the app to record the sound. The device sends the audio data to a server, which analyzes it and diagnoses brake pad wear. The system then displays on the user's device that replacement is necessary and provides an estimated cost. In this way, users can effectively understand the current state of their bicycle and take prompt action.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The user launches the smartphone application and presses the button to start the bicycle diagnostic. This prompts the application to activate the sensors attached to the bicycle.

[0041] Step 2:

[0042] The device activates various sensors mounted on the bicycle (e.g., camera, microphone, vibration sensor) and begins collecting data. The camera captures the visible state of the bicycle, the microphone records sound, and the vibration sensor detects abnormal vibrations.

[0043] Step 3:

[0044] The terminal processes the collected data in real time to detect anomalies. If the collected data indicates an anomaly, it is converted to an appropriate format and prepared for the next processing step.

[0045] Step 4:

[0046] The terminal collects and processes data, which is then sent to the server. This transmission utilizes a network connection to ensure that the data reaches the server safely and quickly.

[0047] Step 5:

[0048] The server inputs the received data into a generating AI model, which then performs analysis to detect anomalies. The AI ​​model uses pattern recognition technology to analyze, for example, audio data or image data, and identify the location of the anomaly.

[0049] Step 6:

[0050] The server generates necessary repair information based on the anomalies detected by the AI. Repair procedures and estimated repair costs are calculated and compiled into information provided to the user.

[0051] Step 7:

[0052] The server sends the generated results to the terminal. This data includes the location of the problem, the urgency of the repair, the estimated cost, and recommended actions.

[0053] Step 8:

[0054] The application screen displays the diagnostic results received by the terminal from the server. The user can refer to these results and, if necessary, arrange for repairs or search for the nearest repair shop.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] For many users, quickly and accurately diagnosing the location of a malfunction in objects such as bicycles, and determining the appropriate repair method and urgency, is a technically challenging task. Furthermore, expert diagnosis and repair are time-consuming and costly, so there is a need for a system that allows users to easily understand the cause of a malfunction and quickly decide on a course of action.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes sensing means for acquiring state information of an object in response to user operation, analysis means for detecting anomalies based on the state information acquired from the sensing means, and information generation means for generating information related to repairs based on the anomalies detected by the analysis means. This enables the user to understand the specific circumstances of the failure and to take appropriate repair action quickly.

[0060] A "user" is someone who operates the system and obtains diagnostic and repair information for an object.

[0061] "Sensing means" refers to functions or devices for acquiring state information of an object in response to user operations.

[0062] "Analysis means" refers to functions or devices for detecting abnormalities based on acquired state information.

[0063] "Information generation means" refers to functions or devices for generating information related to repairs based on detected anomalies.

[0064] "Information display means" refers to functions or devices for presenting generated information to the user.

[0065] "Communication means" refers to functions or devices for transmitting state information to analysis means via a network.

[0066] "Identification means" refers to functions or devices that perform pattern recognition using generative AI models.

[0067] "Recording means" refers to functions or devices for acquiring audio data and image data.

[0068] "Inference means" refers to functions or devices that indicate the location of an anomaly and the priority of its repair.

[0069] This system allows users to efficiently diagnose the condition of objects such as bicycles and determine the need for repairs and how to do so. First, the user installs a diagnostic application on their smartphone or tablet. The application uses sensing devices attached to the object to collect condition information, including audio and image data.

[0070] The terminal acquires data from the sensing device in real time and transmits that data to the server via a network connection. The server analyzes this data using a generative AI model to identify anomalies. Specifically, it uses image recognition technology and voice analysis technology to identify the location and nature of the malfunction.

[0071] The server generates repair information based on the analysis results. This information includes identifying the problem area, repair procedures, estimated repair costs, and urgency. The generated information is presented to the user via a terminal. Based on the information obtained from the terminal, the user can understand the current state of their bicycle and take appropriate action.

[0072] For example, if a bicycle brake makes an unusual noise, the user uses the application to record the sound. The device sends this audio data to a server, which diagnoses brake pad wear. As a result, it indicates that the brake pads need replacing and provides an estimated cost.

[0073] In this process, using a generative AI model allows users to understand the condition of an object without the help of an expert and take prompt repair action. By using prompts such as, "Please perform a diagnostic analysis of the unusual noise coming from my bicycle brakes and tell me what repairs are needed and the estimated cost," the AI ​​model can perform a highly accurate diagnosis.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The user launches the application.

[0077] The user prepares to begin the diagnosis by launching the app on their smartphone or tablet. The app displays an interface to activate the sensing device and prompts the user to take action.

[0078] Step 2:

[0079] The terminal acquires status information from the sensing device.

[0080] The terminal activates a sensor attached to the bicycle and collects audio and image data. The input is data about the physical state of the bicycle, and the output is formatted information containing this data.

[0081] Step 3:

[0082] The device sends the collected data to the server.

[0083] The terminal compresses the collected audio and image data using a network connection and sends it to the server. The input is raw data from the sensing device, and the output is data converted into a format that the server can analyze.

[0084] Step 4:

[0085] The server analyzes the data and identifies anomalies.

[0086] The server analyzes the transmitted data using a generative AI model. This process involves pattern recognition to identify the location and extent of anomalies. The input is the data sent from the terminal, and the output is detailed information about the anomalies as a result of the analysis.

[0087] Step 5:

[0088] The server generates information about the repair.

[0089] The server generates information, including repair procedures, estimated costs, and the urgency of the repair, based on the analysis results. The input is the analyzed anomaly information, and the output is information structured in a way that is easy for the user to understand.

[0090] Step 6:

[0091] The terminal presents the generated information to the user.

[0092] The terminal displays information received from the server within the application, informing the user about the problem and recommended repair methods. The input is structured information from the server, and the output is specific repair actions displayed in the user interface.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] In modern factories, machine failures significantly reduce production efficiency. However, conventional maintenance methods often make it difficult to detect failures early and respond appropriately. This invention aims to provide a technology that enables efficient diagnosis of machine failures and prompt maintenance.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes a measurement means for acquiring machine status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the measurement means, and an information generation means for generating maintenance information based on the abnormalities detected by the analysis means. This enables real-time monitoring of the machine's status and prompt response when abnormalities are detected.

[0098] "Measurement means" refers to devices or methods that acquire machine status information in response to user operations.

[0099] "Analysis means" refers to devices or methods for detecting abnormalities based on state information obtained from measurement means.

[0100] "Information generation means" refers to a device or method that generates maintenance-related information based on anomalies detected by analysis means.

[0101] "Information display means" refers to a device or method for presenting generated information to a user.

[0102] "Reading means" refers to mechanisms and technologies for acquiring audio data and image data.

[0103] "Predictive means" refers to devices or methods for identifying the location of an anomaly and the priority of its maintenance.

[0104] This system efficiently monitors the status of machinery within the factory, quickly detects abnormalities, and provides maintenance information. The system is configured as follows:

[0105] The user first installs sensors on machinery within the factory, and these sensors act as measurement tools. Specific hardware includes audio sensors and cameras. These sensors acquire status information, such as machine operating sounds and visual data, in real time.

[0106] Information acquired from sensors is transmitted to a server via devices such as smartphones and tablets. The server uses a generative AI model as an analysis tool to detect anomalies from this data. The analysis algorithm performs image recognition and audio pattern analysis to determine whether or not an anomaly is present.

[0107] When an anomaly is detected, the server, as an information generation mechanism, generates data identifying the location of the anomaly and determining the maintenance priority. Furthermore, detailed information regarding repair methods and estimated costs is also generated. This generated information is transmitted to the user's device (terminal) and displayed to the user through an information display mechanism.

[0108] This system allows users to constantly monitor the status of their machinery and take quick and appropriate action when an anomaly occurs. For example, if an unusual noise is detected on a production line's conveyor belt, the user can record the sound through the application and have it analyzed by the server. The server then diagnoses that the cause is bearing wear and presents a list of replacement parts and instructions to the user's device.

[0109] For example, a prompt message such as, "Please record sensor data and begin analysis to diagnose the machine's condition immediately. If abnormal noises are detected from the bearings, the AI ​​will identify the cause and provide instructions if repairs are necessary," can be used. This prompt encourages the user to take action and makes the system easier to use.

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The user performs operations to acquire measurement data through sensors installed on the machine. The device acquires machine status information, such as audio and image data, in real time. The input is audio and images from the machine, and the output is digital data transferred to the user's device.

[0113] Step 2:

[0114] The terminal sends the acquired measurement data to the server. The transmission process securely transfers digital data over the internet. The input is the digital data from the terminal, and the output is the transmitted data that reaches the server.

[0115] Step 3:

[0116] The server uses a generative AI model to analyze the received data. Data analysis is performed using image recognition algorithms and speech pattern recognition models to detect anomalies. The input is measurement data, and the output is whether or not an anomaly was detected and detailed information about it.

[0117] Step 4:

[0118] The server generates maintenance information based on the analysis results. It identifies abnormal areas, prioritizes them, and creates information including necessary repair procedures and estimated costs. The input to this process is the analysis results, and the output is the maintenance information provided to the user.

[0119] Step 5:

[0120] The terminal receives maintenance information sent from the server and presents it to the user. The displayed information includes the location of the problem, repair procedures, and a list of necessary parts. The input is maintenance information from the server, and the output is data displayed for the user to view.

[0121] This series of processes allows for efficient assessment of the machine's condition and enables prompt maintenance responses.

[0122] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0123] The present invention provides a system that can take into account the user's emotions when diagnosing bicycle malfunctions. This system comprises sensor means, analysis means, information generation means, and information display means, and further incorporates an emotion engine, making it possible to adjust the diagnostic process according to the user's emotions. The embodiments thereof are described in detail below.

[0124] The user launches the application on a smartphone or other device and begins diagnosing the bicycle. The device acquires the bicycle's status through sensor means. This status information includes image data from a camera, audio data from a microphone, and data from other sensors. The collected data is transmitted to a server in real time, and analysis means installed on the server analyze the data to detect abnormalities.

[0125] Once the anomaly has been identified, the server generates repair information using an information generation system. Meanwhile, the emotion engine analyzes data such as the user's voice tone and facial expressions to understand their emotional state. This emotional state is identified as, for example, excitement, worry, or calmness. Based on the user's emotional state, the emotion engine adjusts the level of detail and presentation of the diagnostic results that the information display system presents to the user.

[0126] For example, if the emotion engine determines that the user is experiencing high levels of stress due to a problem with their bicycle, the device will adjust to gently and carefully explain the diagnostic results to reassure the user. It will also calmly and concisely communicate the need for repairs and their urgency to prevent user confusion.

[0127] Thus, the present invention can improve the user experience and support the optimal repair approach by providing information tailored to the emotional state of each individual user.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The user launches a smartphone application and enters a command to start the diagnosis. In response, the device activates various sensors on the bicycle and prepares to monitor the bicycle's condition.

[0131] Step 2:

[0132] The device uses sensors to collect status information, such as taking pictures of the bicycle's appearance with a camera, collecting abnormal sounds with a microphone, and acquiring vibration data from other sensors. The collected data serves as basic information for determining abnormalities.

[0133] Step 3:

[0134] The device transmits data acquired by its sensors to the server in real time. This data includes image information, audio data, and vibration patterns, and the server analyzes this information in detail after receiving it.

[0135] Step 4:

[0136] The server uses analytical tools to analyze the received data and identify which part of the bicycle is abnormal. This analysis process uses generative AI models for pattern recognition and anomaly detection.

[0137] Step 5:

[0138] The server generates repair recommendations and estimated costs based on detected anomalies using a repair information generation mechanism. This clearly indicates specific repair methods and urgency.

[0139] Step 6:

[0140] The server uses an emotion engine to analyze the user's voice tone and facial expression data to assess the user's emotional state. If it determines that the user is stressed, it adjusts the tone and level of detail of the information provided.

[0141] Step 7:

[0142] Based on the results from the emotion engine, the server sends optimized diagnostic information and repair recommendations to the device.

[0143] Step 8:

[0144] The device displays information received from the server to the user. The results are presented carefully and accurately, taking the user's feelings into consideration, allowing the user to review the diagnostic results and recommended actions.

[0145] (Example 2)

[0146] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0147] When bicycles or other equipment malfunction, it is necessary to quickly and accurately identify the cause of the malfunction, provide appropriate repair instructions to the user, and present information while taking the user's emotional state into consideration. In particular, a challenge is to provide information that allows users to work on repairs with peace of mind without feeling excessively stressed about the malfunction.

[0148] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0149] In this invention, the server includes a sensor means for acquiring device status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the sensor means, an information generation means for generating repair information based on the abnormalities detected by the analysis means, an emotion analysis means for analyzing the user's emotional information, and a display control means for presenting information to the user based on the generated information and the user's emotional information. This enables the rapid and accurate identification of the cause of device failure and the presentation of appropriate and reassuring information according to the user's emotional state.

[0150] A "user" refers to a person who operates this system and receives information regarding the diagnosis and repair of bicycles and equipment.

[0151] "Sensor means" refers to a set of devices or methods for acquiring status information of equipment through user operation.

[0152] "Analysis means" refers to a technology or process for analyzing state information acquired by sensor means and detecting anomalies.

[0153] "Information generation means" refers to a technology or system for creating information related to repairs based on anomalies identified by analysis means.

[0154] "Emotional analysis methods" refer to means of analyzing a user's emotional information to understand their stress levels and emotional state.

[0155] "Display control means" refers to a technology or method for presenting information to the user in the most optimal form based on generated information and emotional information.

[0156] The system of the present invention diagnoses equipment failures via a user-owned terminal and provides appropriate information regarding repairs. Specific embodiments are described below.

[0157] The user first launches the application on their device. The device is equipped with sensory means for collecting data. These sensory means include a camera, microphone, and various other sensors, which acquire image and audio data regarding the device's status. This collected status data is transmitted from the device to the server in real time.

[0158] The server processes the large volume of data it receives using analytical tools. Generative AI models are used for analysis, identifying the cause of failures and abnormal locations based on machine learning algorithms. For example, it can detect abnormal damage from images captured by cameras and identify unusual sounds from audio recorded by microphones.

[0159] Furthermore, the server is equipped with emotion analysis capabilities. This allows it to extract emotional information from the user's voice tone and camera footage, thereby understanding their emotional state. Emotion recognition technology is used for emotion analysis, visually and audibly evaluating the user's stress and anxiety.

[0160] The server integrates analysis results and emotional information to create repair information to be provided to the user through an information generation means. The information is adjusted considering the user's emotional state and presented to the user via a display control means.

[0161] For example, if the server determines that the user is experiencing stress, the terminal will display a gentle and reassuring guide during the repair process. An example of a prompt message is input into the generating AI model: "Based on the user's voice and facial expression data, identify their current emotional state and suggest the most appropriate method for presenting the diagnostic results." This input then performs emotion analysis.

[0162] Thus, the system of the present invention enables users to repair equipment with peace of mind through highly accurate fault diagnosis that takes into account the user's emotional state and the presentation of appropriate information.

[0163] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0164] Step 1:

[0165] The user launches an application on their device to diagnose a problem with their bicycle. The device prepares the sensors and begins collecting data from the equipment. Inputs include image data from the camera, audio data from the microphone, and acceleration and vibration data from other sensors. This data is acquired in real time from each sensor. The output is the collected equipment status data.

[0166] Step 2:

[0167] The terminal sends the collected data to the server. For security reasons, the data is encrypted before transmission. The server processes the received data using analytical methods. Inputs include image data, audio data, and sensor data transmitted from the terminal. Data processing uses a generative AI model and applies anomaly detection algorithms to identify faulty parts and abnormal conditions in the equipment. The output is the analyzed anomaly information.

[0168] Step 3:

[0169] The server then uses emotion analysis to determine the user's emotional state. The inputs are the user's voice tone and facial expression data captured by the camera. Using emotion analysis technology, the server identifies the user's current emotional state by analyzing their stress level and emotional condition. The output is an evaluation of the user's emotional state.

[0170] Step 4:

[0171] The server combines analyzed anomaly information and emotional information to create appropriate repair information for the user using an information generation mechanism. Here, the input consists of two parts: anomaly information and emotional information. Logic and prompt statements are used for information generation, and a generation AI model produces information optimized for the user's state. The output is the adjusted repair information.

[0172] Step 5:

[0173] The terminal presents information to the user through display control means, based on information transmitted from the server. The terminal considers the user's emotional state and presents information with appropriate tone and visual effects. Specifically, it displays voice guidance in a gentle tone to calm the user and concise and clear text messages. Ultimately, the user is more likely to understand the specific actions that need to be taken for repair.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0176] In bicycle fault diagnosis, conventional systems provided diagnostic information uniformly without considering the user's emotions, resulting in an unoptimized user experience. This invention aims to support efficient bicycle maintenance while reducing stress by adjusting the diagnostic process according to the user's emotional state.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes an information gathering means for acquiring state information in response to user operations, an anomaly analysis means for detecting anomalies based on the state information acquired from the information gathering means, an information generation means for generating repair-related information based on the anomalies detected by the anomaly analysis means, and an emotion analysis means for analyzing the user's voice tone and facial expressions to determine their emotional state. This enables the presentation of optimal information according to the user's emotional state.

[0179] "Information gathering means" refers to a device or method that has the function of acquiring bicycle status information in response to user operations.

[0180] An "anomaly analysis means" is a device or process for detecting abnormalities in a bicycle based on state information obtained from an information gathering means.

[0181] "Information generation means" refers to devices or algorithms that generate information related to repairs based on detected anomalies.

[0182] An "information display means" is a device or software that has the function of presenting information to the user and adjusts the method of information presentation through sentiment analysis.

[0183] "Emotional analysis methods" refer to technologies and systems that analyze a user's voice tone and facial expressions to determine their emotional state.

[0184] "Means of recording" refers to devices such as cameras and microphones used to acquire audio and image information.

[0185] "Predictive means" refers to devices or methods that have the function of identifying the location of an anomaly and the urgency of its repair.

[0186] In order to implement this invention, it is necessary to build a system in which servers, terminals, and users cooperate with each other. In this system, processing is carried out in the following flow.

[0187] First, the user launches a dedicated application on a device (such as smart glasses or a smartphone). The device is equipped with information gathering capabilities, and uses a camera and microphone to collect audio and image information to obtain the status of the bicycle. This includes hands-free operation using smart glasses.

[0188] Next, the data collected by the terminal is transmitted to the server via the network. The server uses an anomaly analysis means to analyze the data in real time to identify any abnormalities in the bicycle. Based on the analysis results, the information generation means constructs information related to repairs.

[0189] Furthermore, the server integrates emotion analysis capabilities, determining the user's emotional state by analyzing their voice tone and facial expressions. This analysis is then used to adjust the content and presentation of information displayed to the user. This ensures that the user receives reassuring information appropriate to their emotional state. For example, if the user is feeling anxious, the system adjusts to guide them through the repair process in a gentle tone.

[0190] For example, when a customer visits a bicycle shop, the staff wearing smart glasses can quickly identify the problem and provide reassuring customer service based on the customer's facial expressions.

[0191] An example of a prompt from a generated AI model is, "Understand the customer's emotions and suggest how to adjust your response." This prompt is used by the system to consider appropriate responses based on the user's emotions.

[0192] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0193] Step 1:

[0194] The user launches the application on the device and starts the bicycle diagnostic. The device uses its camera and microphone to collect image data and audio tones of the bicycle. The input at this time is the start signal from the user's operation, and the output is the collected visual and audio data. This data is collected by the device's information collection means.

[0195] Step 2:

[0196] The terminal sends the collected data to the server. The server uses an anomaly analysis tool to detect anomalies in the bicycle from the input video and audio information. This analysis tool performs data processing such as image comparison and audio pattern recognition to identify the location and type of anomaly. The output is detailed information about which part of the bicycle is abnormal.

[0197] Step 3:

[0198] Based on the results of the anomaly analysis, the server uses information generation tools to create repair procedures and recommendations. The input is detailed information about the anomaly, and the output is specific repair procedures and precautions that the user should take. This information serves as a guide to help the user perform the repair work smoothly.

[0199] Step 4:

[0200] The server determines the user's emotional state from voice tone and facial expression data collected using emotion analysis tools. The input is the aforementioned voice and facial expression data, and the output is the analysis result regarding the user's emotions (e.g., tension, anxiety, relaxation). For example, if the voice tone is high and the tempo is fast, the server will determine that the user is in an anxious state.

[0201] Step 5:

[0202] The server adjusts how the generated repair information is presented based on the results of sentiment analysis. Through the information display means, it presents the information with a tone and level of detail that matches the user's emotions. The input is the user's emotional state and repair information, and the output is displayed to the user by the terminal as information adjusted to be easily understood by the user.

[0203] Step 6:

[0204] Finally, the user can confidently repair the bicycle based on the information provided by the device. By following the instructions displayed on the device, the problem can be resolved efficiently. The input is the adjusted information from the device, and the output is the result of the user's completed repair work.

[0205] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0206] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0207] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0208] [Second Embodiment]

[0209] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0210] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0211] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0212] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0213] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0215] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0216] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0217] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0218] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0219] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0220] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0221] The system of the present invention is designed to enable users to efficiently diagnose bicycle malfunctions and understand the need for and methods of repair. Specific embodiments of this system are described below.

[0222] First, the user installs an application on their smartphone or tablet to diagnose their bicycle. When the user launches the app, the device prompts them to activate the sensor device installed on the bicycle. This sensor device captures the physical condition of the bicycle, for example, by including a camera and microphone.

[0223] The terminal collects data from sensors in real time and constantly monitors for any abnormalities. Once data is collected, it is transmitted to a server via the network. The server uses a generative AI model to analyze the received data and identify which part of the bicycle is experiencing problems. This process includes pattern recognition of image and audio data supplied by the sensors.

[0224] When an anomaly is detected, the server processes the information to generate repair instructions and estimated repair costs corresponding to the location of the anomaly. The generated results are returned to the terminal in a structured format, and the terminal presents these results to the user within the application. The user can review the diagnostic results in detail on the screen and understand the recommended repair items and their urgency. If necessary, they can also search for information on the nearest repair shop and make a reservation through the application.

[0225] For example, if a bicycle brake makes an unusual noise, the user can use the app to record the sound. The device sends the audio data to a server, which analyzes it and diagnoses brake pad wear. The system then displays on the user's device that replacement is necessary and provides an estimated cost. In this way, users can effectively understand the current state of their bicycle and take prompt action.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The user launches the smartphone application and presses the button to start the bicycle diagnostic. This prompts the application to activate the sensors attached to the bicycle.

[0229] Step 2:

[0230] The device activates various sensors mounted on the bicycle (e.g., camera, microphone, vibration sensor) and begins collecting data. The camera captures the visible state of the bicycle, the microphone records sound, and the vibration sensor detects abnormal vibrations.

[0231] Step 3:

[0232] The terminal processes the collected data in real time to detect anomalies. If the collected data indicates an anomaly, it is converted to an appropriate format and prepared for the next processing step.

[0233] Step 4:

[0234] The terminal collects and processes data, which is then sent to the server. This transmission utilizes a network connection to ensure that the data reaches the server safely and quickly.

[0235] Step 5:

[0236] The server inputs the received data into a generating AI model, which then performs analysis to detect anomalies. The AI ​​model uses pattern recognition technology to analyze, for example, audio data or image data, and identify the location of the anomaly.

[0237] Step 6:

[0238] The server generates necessary repair information based on the anomalies detected by the AI. Repair procedures and estimated repair costs are calculated and compiled into information provided to the user.

[0239] Step 7:

[0240] The server sends the generated results to the terminal. This data includes the location of the problem, the urgency of the repair, the estimated cost, and recommended actions.

[0241] Step 8:

[0242] The application screen displays the diagnostic results received by the terminal from the server. The user can refer to these results and, if necessary, arrange for repairs or search for the nearest repair shop.

[0243] (Example 1)

[0244] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0245] For many users, quickly and accurately diagnosing the location of a malfunction in objects such as bicycles, and determining the appropriate repair method and urgency, is a technically challenging task. Furthermore, expert diagnosis and repair are time-consuming and costly, so there is a need for a system that allows users to easily understand the cause of a malfunction and quickly decide on a course of action.

[0246] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0247] In this invention, the server includes sensing means for acquiring state information of an object in response to user operation, analysis means for detecting anomalies based on the state information acquired from the sensing means, and information generation means for generating information related to repairs based on the anomalies detected by the analysis means. This enables the user to understand the specific circumstances of the failure and to take appropriate repair action quickly.

[0248] A "user" is someone who operates the system and obtains diagnostic and repair information for an object.

[0249] "Sensing means" refers to functions or devices for acquiring state information of an object in response to user operations.

[0250] "Analysis means" refers to functions or devices for detecting abnormalities based on acquired state information.

[0251] "Information generation means" refers to functions or devices for generating information related to repairs based on detected anomalies.

[0252] "Information display means" refers to functions or devices for presenting generated information to the user.

[0253] "Communication means" refers to functions or devices for transmitting state information to analysis means via a network.

[0254] "Identification means" refers to functions or devices that perform pattern recognition using generative AI models.

[0255] "Recording means" refers to functions or devices for acquiring audio data and image data.

[0256] "Inference means" refers to functions or devices that indicate the location of an anomaly and the priority of its repair.

[0257] This system allows users to efficiently diagnose the condition of objects such as bicycles and determine the need for repairs and how to do so. First, the user installs a diagnostic application on their smartphone or tablet. The application uses sensing devices attached to the object to collect condition information, including audio and image data.

[0258] The terminal acquires data from the sensing device in real time and transmits that data to the server via a network connection. The server analyzes this data using a generative AI model to identify anomalies. Specifically, it uses image recognition technology and voice analysis technology to identify the location and nature of the malfunction.

[0259] The server generates repair information based on the analysis results. This information includes identifying the problem area, repair procedures, estimated repair costs, and urgency. The generated information is presented to the user via a terminal. Based on the information obtained from the terminal, the user can understand the current state of their bicycle and take appropriate action.

[0260] For example, if a bicycle brake makes an unusual noise, the user uses the application to record the sound. The device sends this audio data to a server, which diagnoses brake pad wear. As a result, it indicates that the brake pads need replacing and provides an estimated cost.

[0261] In this process, using a generative AI model allows users to understand the condition of an object without the help of an expert and take prompt repair action. By using prompts such as, "Please perform a diagnostic analysis of the unusual noise coming from my bicycle brakes and tell me what repairs are needed and the estimated cost," the AI ​​model can perform a highly accurate diagnosis.

[0262] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0263] Step 1:

[0264] The user launches the application.

[0265] The user prepares to begin the diagnosis by launching the app on their smartphone or tablet. The app displays an interface to activate the sensing device and prompts the user to take action.

[0266] Step 2:

[0267] The terminal acquires status information from the sensing device.

[0268] The terminal activates a sensor attached to the bicycle and collects audio and image data. The input is data about the physical state of the bicycle, and the output is formatted information containing this data.

[0269] Step 3:

[0270] The device sends the collected data to the server.

[0271] The terminal compresses the collected audio and image data using a network connection and sends it to the server. The input is raw data from the sensing device, and the output is data converted into a format that the server can analyze.

[0272] Step 4:

[0273] The server analyzes the data and identifies anomalies.

[0274] The server analyzes the transmitted data using a generative AI model. This process involves pattern recognition to identify the location and extent of anomalies. The input is the data sent from the terminal, and the output is detailed information about the anomalies as a result of the analysis.

[0275] Step 5:

[0276] The server generates information about the repair.

[0277] The server generates information, including repair procedures, estimated costs, and the urgency of the repair, based on the analysis results. The input is the analyzed anomaly information, and the output is information structured in a way that is easy for the user to understand.

[0278] Step 6:

[0279] The terminal presents the generated information to the user.

[0280] The terminal displays information received from the server within the application, informing the user about the problem and recommended repair methods. The input is structured information from the server, and the output is specific repair actions displayed in the user interface.

[0281] (Application Example 1)

[0282] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0283] In modern factories, machine failures are a factor that significantly reduces production efficiency. However, in conventional maintenance methods, it is often difficult to detect failures at an early stage and take appropriate actions. The purpose of the present invention is to provide a technology that can efficiently diagnose machine failures and enable prompt maintenance.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0285] In this invention, the server includes a measurement means for acquiring machine state information according to a user's operation, an analysis means for detecting an abnormality based on the state information acquired from the measurement means, and an information generation means for generating information regarding maintenance based on the abnormality detected by the analysis means. Thereby, it becomes possible to monitor the state of the machine in real time and respond promptly when an abnormality is detected.

[0286] The "measurement means" is a device or method for acquiring machine state information according to a user's operation.

[0287] The "analysis means" is a device or method for detecting an abnormality based on the state information acquired from the measurement means

[0288] The "information generation means" is a device or method for generating information regarding maintenance based on the abnormality detected by the analysis means.

[0289] The "information display means" is a device or method for presenting the generated information to the user.

[0290] The "reading means" is a mechanism or technology for acquiring audio data and image data.

[0291] "Predictive means" refers to devices or methods for identifying the location of an anomaly and the priority of its maintenance.

[0292] This system efficiently monitors the status of machinery within the factory, quickly detects abnormalities, and provides maintenance information. The system is configured as follows:

[0293] The user first installs sensors on machinery within the factory, and these sensors act as measurement tools. Specific hardware includes audio sensors and cameras. These sensors acquire status information, such as machine operating sounds and visual data, in real time.

[0294] Information acquired from sensors is transmitted to a server via devices such as smartphones and tablets. The server uses a generative AI model as an analysis tool to detect anomalies from this data. The analysis algorithm performs image recognition and audio pattern analysis to determine whether or not an anomaly is present.

[0295] When an anomaly is detected, the server, as an information generation mechanism, generates data identifying the location of the anomaly and determining the maintenance priority. Furthermore, detailed information regarding repair methods and estimated costs is also generated. This generated information is transmitted to the user's device (terminal) and displayed to the user through an information display mechanism.

[0296] This system allows users to constantly monitor the status of their machinery and take quick and appropriate action when an anomaly occurs. For example, if an unusual noise is detected on a production line's conveyor belt, the user can record the sound through the application and have it analyzed by the server. The server then diagnoses that the cause is bearing wear and presents a list of replacement parts and instructions to the user's device.

[0297] For example, a prompt message such as, "Please record sensor data and begin analysis to diagnose the machine's condition immediately. If abnormal noises are detected from the bearings, the AI ​​will identify the cause and provide instructions if repairs are necessary," can be used. This prompt encourages the user to take action and makes the system easier to use.

[0298] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0299] Step 1:

[0300] The user performs operations to acquire measurement data through sensors installed on the machine. The device acquires machine status information, such as audio and image data, in real time. The input is audio and images from the machine, and the output is digital data transferred to the user's device.

[0301] Step 2:

[0302] The terminal sends the acquired measurement data to the server. The transmission process securely transfers digital data over the internet. The input is the digital data from the terminal, and the output is the transmitted data that reaches the server.

[0303] Step 3:

[0304] The server uses a generative AI model to analyze the received data. Data analysis is performed using image recognition algorithms and speech pattern recognition models to detect anomalies. The input is measurement data, and the output is whether or not an anomaly was detected and detailed information about it.

[0305] Step 4:

[0306] The server generates maintenance information based on the analysis results. It identifies abnormal areas, prioritizes them, and creates information including necessary repair procedures and estimated costs. The input to this process is the analysis results, and the output is the maintenance information provided to the user.

[0307] Step 5:

[0308] The terminal receives the maintenance information sent from the server and presents the information to the user. The displayed information includes the abnormal location, repair procedures, list of necessary parts, etc. The input is the maintenance information from the server, and the output is the data display viewable by the user.

[0309] Through this series of processes, the state of the machine can be efficiently grasped, enabling prompt maintenance response.

[0310] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0311] The present invention is a system that can take into account the user's emotion when diagnosing a bicycle failure. This system includes sensor means, analysis means, information generation means, and information display means, and by further incorporating an emotion engine, it is possible to adjust the diagnostic process according to the user's emotion. The embodiments will be described in detail below.

[0312] The user starts the application on a terminal such as a smartphone and initiates the diagnosis of the bicycle. The terminal acquires the state of the bicycle through the sensor means. This state information includes image data using a camera, audio data using a microphone, and data from other sensors. The collected data is transmitted to the server in real time, and the analysis means installed on the server analyzes the data to detect abnormalities.

[0313] Once the anomaly has been identified, the server generates repair information using an information generation system. Meanwhile, the emotion engine analyzes data such as the user's voice tone and facial expressions to understand their emotional state. This emotional state is identified as, for example, excitement, worry, or calmness. Based on the user's emotional state, the emotion engine adjusts the level of detail and presentation of the diagnostic results that the information display system presents to the user.

[0314] For example, if the emotion engine determines that the user is experiencing high levels of stress due to a problem with their bicycle, the device will adjust to gently and carefully explain the diagnostic results to reassure the user. It will also calmly and concisely communicate the need for repairs and their urgency to prevent user confusion.

[0315] Thus, the present invention can improve the user experience and support the optimal repair approach by providing information tailored to the emotional state of each individual user.

[0316] The following describes the processing flow.

[0317] Step 1:

[0318] The user launches a smartphone application and enters a command to start the diagnosis. In response, the device activates various sensors on the bicycle and prepares to monitor the bicycle's condition.

[0319] Step 2:

[0320] The device uses sensors to collect status information, such as taking pictures of the bicycle's appearance with a camera, collecting abnormal sounds with a microphone, and acquiring vibration data from other sensors. The collected data serves as basic information for determining abnormalities.

[0321] Step 3:

[0322] The device transmits data acquired by its sensors to the server in real time. This data includes image information, audio data, and vibration patterns, and the server analyzes this information in detail after receiving it.

[0323] Step 4:

[0324] The server uses analytical tools to analyze the received data and identify which part of the bicycle is abnormal. This analysis process uses generative AI models for pattern recognition and anomaly detection.

[0325] Step 5:

[0326] The server generates repair recommendations and estimated costs based on detected anomalies using a repair information generation mechanism. This clearly indicates specific repair methods and urgency.

[0327] Step 6:

[0328] The server uses an emotion engine to analyze the user's voice tone and facial expression data to assess the user's emotional state. If it determines that the user is stressed, it adjusts the tone and level of detail of the information provided.

[0329] Step 7:

[0330] Based on the results from the emotion engine, the server sends optimized diagnostic information and repair recommendations to the device.

[0331] Step 8:

[0332] The device displays information received from the server to the user. The results are presented carefully and accurately, taking the user's feelings into consideration, allowing the user to review the diagnostic results and recommended actions.

[0333] (Example 2)

[0334] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0335] When bicycles or other equipment malfunction, it is necessary to quickly and accurately identify the cause of the malfunction, provide appropriate repair instructions to the user, and present information while taking the user's emotional state into consideration. In particular, a challenge is to provide information that allows users to work on repairs with peace of mind without feeling excessively stressed about the malfunction.

[0336] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0337] In this invention, the server includes a sensor means for acquiring device status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the sensor means, an information generation means for generating repair information based on the abnormalities detected by the analysis means, an emotion analysis means for analyzing the user's emotional information, and a display control means for presenting information to the user based on the generated information and the user's emotional information. This enables the rapid and accurate identification of the cause of device failure and the presentation of appropriate and reassuring information according to the user's emotional state.

[0338] A "user" refers to a person who operates this system and receives information regarding the diagnosis and repair of bicycles and equipment.

[0339] "Sensor means" refers to a set of devices or methods for acquiring status information of equipment through user operation.

[0340] "Analysis means" refers to a technology or process for analyzing state information acquired by sensor means and detecting anomalies.

[0341] "Information generation means" refers to a technology or system for creating information related to repairs based on anomalies identified by analysis means.

[0342] "Emotional analysis methods" refer to means of analyzing a user's emotional information to understand their stress levels and emotional state.

[0343] "Display control means" refers to a technology or method for presenting information to the user in the most optimal form based on generated information and emotional information.

[0344] The system of the present invention diagnoses equipment failures via a user-owned terminal and provides appropriate information regarding repairs. Specific embodiments are described below.

[0345] The user first launches the application on their device. The device is equipped with sensory means for collecting data. These sensory means include a camera, microphone, and various other sensors, which acquire image and audio data regarding the device's status. This collected status data is transmitted from the device to the server in real time.

[0346] The server processes the large volume of data it receives using analytical tools. Generative AI models are used for analysis, identifying the cause of failures and abnormal locations based on machine learning algorithms. For example, it can detect abnormal damage from images captured by cameras and identify unusual sounds from audio recorded by microphones.

[0347] Furthermore, the server is equipped with emotion analysis capabilities. This allows it to extract emotional information from the user's voice tone and camera footage, thereby understanding their emotional state. Emotion recognition technology is used for emotion analysis, visually and audibly evaluating the user's stress and anxiety.

[0348] The server integrates analysis results and emotional information to create repair information to be provided to the user through an information generation means. The information is adjusted considering the user's emotional state and presented to the user via a display control means.

[0349] For example, if the server determines that the user is experiencing stress, the terminal will display a gentle and reassuring guide during the repair process. An example of a prompt message is input into the generating AI model: "Based on the user's voice and facial expression data, identify their current emotional state and suggest the most appropriate method for presenting the diagnostic results." This input then performs emotion analysis.

[0350] Thus, the system of the present invention enables users to repair equipment with peace of mind through highly accurate fault diagnosis that takes into account the user's emotional state and the presentation of appropriate information.

[0351] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0352] Step 1:

[0353] The user launches an application on their device to diagnose a problem with their bicycle. The device prepares the sensors and begins collecting data from the equipment. Inputs include image data from the camera, audio data from the microphone, and acceleration and vibration data from other sensors. This data is acquired in real time from each sensor. The output is the collected equipment status data.

[0354] Step 2:

[0355] The terminal sends the collected data to the server. For security reasons, the data is encrypted before transmission. The server processes the received data using analytical methods. Inputs include image data, audio data, and sensor data transmitted from the terminal. Data processing uses a generative AI model and applies anomaly detection algorithms to identify faulty parts and abnormal conditions in the equipment. The output is the analyzed anomaly information.

[0356] Step 3:

[0357] The server then uses emotion analysis to determine the user's emotional state. The inputs are the user's voice tone and facial expression data captured by the camera. Using emotion analysis technology, the server identifies the user's current emotional state by analyzing their stress level and emotional condition. The output is an evaluation of the user's emotional state.

[0358] Step 4:

[0359] The server combines analyzed anomaly information and emotional information to create appropriate repair information for the user using an information generation mechanism. Here, the input consists of two parts: anomaly information and emotional information. Logic and prompt statements are used for information generation, and a generation AI model produces information optimized for the user's state. The output is the adjusted repair information.

[0360] Step 5:

[0361] The terminal presents information to the user through display control means, based on information transmitted from the server. The terminal considers the user's emotional state and presents information with appropriate tone and visual effects. Specifically, it displays voice guidance in a gentle tone to calm the user and concise and clear text messages. Ultimately, the user is more likely to understand the specific actions that need to be taken for repair.

[0362] (Application Example 2)

[0363] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0364] In bicycle fault diagnosis, conventional systems provided diagnostic information uniformly without considering the user's emotions, resulting in an unoptimized user experience. This invention aims to support efficient bicycle maintenance while reducing stress by adjusting the diagnostic process according to the user's emotional state.

[0365] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0366] In this invention, the server includes an information gathering means for acquiring state information in response to user operations, an anomaly analysis means for detecting anomalies based on the state information acquired from the information gathering means, an information generation means for generating repair-related information based on the anomalies detected by the anomaly analysis means, and an emotion analysis means for analyzing the user's voice tone and facial expressions to determine their emotional state. This enables the presentation of optimal information according to the user's emotional state.

[0367] "Information gathering means" refers to a device or method that has the function of acquiring bicycle status information in response to user operations.

[0368] An "anomaly analysis means" is a device or process for detecting abnormalities in a bicycle based on state information obtained from an information gathering means.

[0369] "Information generation means" refers to devices or algorithms that generate information related to repairs based on detected anomalies.

[0370] An "information display means" is a device or software that has the function of presenting information to the user and adjusts the method of information presentation through sentiment analysis.

[0371] "Emotional analysis methods" refer to technologies and systems that analyze a user's voice tone and facial expressions to determine their emotional state.

[0372] "Means of recording" refers to devices such as cameras and microphones used to acquire audio and image information.

[0373] "Predictive means" refers to devices or methods that have the function of identifying the location of an anomaly and the urgency of its repair.

[0374] In order to implement this invention, it is necessary to build a system in which servers, terminals, and users cooperate with each other. In this system, processing is carried out in the following flow.

[0375] First, the user launches a dedicated application on a device (such as smart glasses or a smartphone). The device is equipped with information gathering capabilities, and uses a camera and microphone to collect audio and image information to obtain the status of the bicycle. This includes hands-free operation using smart glasses.

[0376] Next, the data collected by the terminal is transmitted to the server via the network. The server uses an anomaly analysis means to analyze the data in real time to identify any abnormalities in the bicycle. Based on the analysis results, the information generation means constructs information related to repairs.

[0377] Furthermore, the server integrates emotion analysis capabilities, determining the user's emotional state by analyzing their voice tone and facial expressions. This analysis is then used to adjust the content and presentation of information displayed to the user. This ensures that the user receives reassuring information appropriate to their emotional state. For example, if the user is feeling anxious, the system adjusts to guide them through the repair process in a gentle tone.

[0378] For example, when a customer visits a bicycle shop, the staff wearing smart glasses can quickly identify the problem and provide reassuring customer service based on the customer's facial expressions.

[0379] An example of a prompt from a generated AI model is, "Understand the customer's emotions and suggest how to adjust your response." This prompt is used by the system to consider appropriate responses based on the user's emotions.

[0380] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0381] Step 1:

[0382] The user launches the application on the device and starts the bicycle diagnostic. The device uses its camera and microphone to collect image data and audio tones of the bicycle. The input at this time is the start signal from the user's operation, and the output is the collected visual and audio data. This data is collected by the device's information collection means.

[0383] Step 2:

[0384] The terminal sends the collected data to the server. The server uses an anomaly analysis tool to detect anomalies in the bicycle from the input video and audio information. This analysis tool performs data processing such as image comparison and audio pattern recognition to identify the location and type of anomaly. The output is detailed information about which part of the bicycle is abnormal.

[0385] Step 3:

[0386] Based on the results of the anomaly analysis, the server uses information generation tools to create repair procedures and recommendations. The input is detailed information about the anomaly, and the output is specific repair procedures and precautions that the user should take. This information serves as a guide to help the user perform the repair work smoothly.

[0387] Step 4:

[0388] The server determines the user's emotional state from voice tone and facial expression data collected using emotion analysis tools. The input is the aforementioned voice and facial expression data, and the output is the analysis result regarding the user's emotions (e.g., tension, anxiety, relaxation). For example, if the voice tone is high and the tempo is fast, the server will determine that the user is in an anxious state.

[0389] Step 5:

[0390] The server adjusts how the generated repair information is presented based on the results of sentiment analysis. Through the information display means, it presents the information with a tone and level of detail that matches the user's emotions. The input is the user's emotional state and repair information, and the output is displayed to the user by the terminal as information adjusted to be easily understood by the user.

[0391] Step 6:

[0392] Finally, the user can confidently repair the bicycle based on the information provided by the device. By following the instructions displayed on the device, the problem can be resolved efficiently. The input is the adjusted information from the device, and the output is the result of the user's completed repair work.

[0393] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0396] [Third Embodiment]

[0397] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0398] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0400] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0404] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0405] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0406] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0407] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0408] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0409] The system of the present invention is designed to enable users to efficiently diagnose bicycle malfunctions and understand the need for and methods of repair. Specific embodiments of this system are described below.

[0410] First, the user installs an application on their smartphone or tablet to diagnose their bicycle. When the user launches the app, the device prompts them to activate the sensor device installed on the bicycle. This sensor device captures the physical condition of the bicycle, for example, by including a camera and microphone.

[0411] The terminal collects data from sensors in real time and constantly monitors for any abnormalities. Once data is collected, it is transmitted to a server via the network. The server uses a generative AI model to analyze the received data and identify which part of the bicycle is experiencing problems. This process includes pattern recognition of image and audio data supplied by the sensors.

[0412] When an anomaly is detected, the server processes the information to generate repair instructions and estimated repair costs corresponding to the location of the anomaly. The generated results are returned to the terminal in a structured format, and the terminal presents these results to the user within the application. The user can review the diagnostic results in detail on the screen and understand the recommended repair items and their urgency. If necessary, they can also search for information on the nearest repair shop and make a reservation through the application.

[0413] For example, if a bicycle brake makes an unusual noise, the user can use the app to record the sound. The device sends the audio data to a server, which analyzes it and diagnoses brake pad wear. The system then displays on the user's device that replacement is necessary and provides an estimated cost. In this way, users can effectively understand the current state of their bicycle and take prompt action.

[0414] The following describes the processing flow.

[0415] Step 1:

[0416] The user launches the smartphone application and presses the button to start the bicycle diagnostic. This prompts the application to activate the sensors attached to the bicycle.

[0417] Step 2:

[0418] The device activates various sensors mounted on the bicycle (e.g., camera, microphone, vibration sensor) and begins collecting data. The camera captures the visible state of the bicycle, the microphone records sound, and the vibration sensor detects abnormal vibrations.

[0419] Step 3:

[0420] The terminal processes the collected data in real time to detect anomalies. If the collected data indicates an anomaly, it is converted to an appropriate format and prepared for the next processing step.

[0421] Step 4:

[0422] The terminal collects and processes data, which is then sent to the server. This transmission utilizes a network connection to ensure that the data reaches the server safely and quickly.

[0423] Step 5:

[0424] The server inputs the received data into a generating AI model, which then performs analysis to detect anomalies. The AI ​​model uses pattern recognition technology to analyze, for example, audio data or image data, and identify the location of the anomaly.

[0425] Step 6:

[0426] The server generates necessary repair information based on the anomalies detected by the AI. Repair procedures and estimated repair costs are calculated and compiled into information provided to the user.

[0427] Step 7:

[0428] The server sends the generated results to the terminal. This data includes the location of the problem, the urgency of the repair, the estimated cost, and recommended actions.

[0429] Step 8:

[0430] The application screen displays the diagnostic results received by the terminal from the server. The user can refer to these results and, if necessary, arrange for repairs or search for the nearest repair shop.

[0431] (Example 1)

[0432] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0433] For many users, quickly and accurately diagnosing the location of a malfunction in objects such as bicycles, and determining the appropriate repair method and urgency, is a technically challenging task. Furthermore, expert diagnosis and repair are time-consuming and costly, so there is a need for a system that allows users to easily understand the cause of a malfunction and quickly decide on a course of action.

[0434] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0435] In this invention, the server includes sensing means for acquiring state information of an object in response to user operation, analysis means for detecting anomalies based on the state information acquired from the sensing means, and information generation means for generating information related to repairs based on the anomalies detected by the analysis means. This enables the user to understand the specific circumstances of the failure and to take appropriate repair action quickly.

[0436] A "user" is someone who operates the system and obtains diagnostic and repair information for an object.

[0437] "Sensing means" refers to functions or devices for acquiring state information of an object in response to user operations.

[0438] "Analysis means" refers to functions or devices for detecting abnormalities based on acquired state information.

[0439] "Information generation means" refers to functions or devices for generating information related to repairs based on detected anomalies.

[0440] "Information display means" refers to functions or devices for presenting generated information to the user.

[0441] "Communication means" refers to functions or devices for transmitting state information to analysis means via a network.

[0442] "Identification means" refers to functions or devices that perform pattern recognition using generative AI models.

[0443] "Recording means" refers to functions or devices for acquiring audio data and image data.

[0444] "Inference means" refers to functions or devices that indicate the location of an anomaly and the priority of its repair.

[0445] This system allows users to efficiently diagnose the condition of objects such as bicycles and determine the need for repairs and how to do so. First, the user installs a diagnostic application on their smartphone or tablet. The application uses sensing devices attached to the object to collect condition information, including audio and image data.

[0446] The terminal acquires data from the sensing device in real time and transmits that data to the server via a network connection. The server analyzes this data using a generative AI model to identify anomalies. Specifically, it uses image recognition technology and voice analysis technology to identify the location and nature of the malfunction.

[0447] The server generates repair information based on the analysis results. This information includes identifying the problem area, repair procedures, estimated repair costs, and urgency. The generated information is presented to the user via a terminal. Based on the information obtained from the terminal, the user can understand the current state of their bicycle and take appropriate action.

[0448] For example, if a bicycle brake makes an unusual noise, the user uses the application to record the sound. The device sends this audio data to a server, which diagnoses brake pad wear. As a result, it indicates that the brake pads need replacing and provides an estimated cost.

[0449] In this process, using a generative AI model allows users to understand the condition of an object without the help of an expert and take prompt repair action. By using prompts such as, "Please perform a diagnostic analysis of the unusual noise coming from my bicycle brakes and tell me what repairs are needed and the estimated cost," the AI ​​model can perform a highly accurate diagnosis.

[0450] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0451] Step 1:

[0452] The user launches the application.

[0453] The user prepares to begin the diagnosis by launching the app on their smartphone or tablet. The app displays an interface to activate the sensing device and prompts the user to take action.

[0454] Step 2:

[0455] The terminal acquires status information from the sensing device.

[0456] The terminal activates a sensor attached to the bicycle and collects audio and image data. The input is data about the physical state of the bicycle, and the output is formatted information containing this data.

[0457] Step 3:

[0458] The device sends the collected data to the server.

[0459] The terminal compresses the collected audio and image data using a network connection and sends it to the server. The input is raw data from the sensing device, and the output is data converted into a format that the server can analyze.

[0460] Step 4:

[0461] The server analyzes the data and identifies anomalies.

[0462] The server analyzes the transmitted data using a generative AI model. This process involves pattern recognition to identify the location and extent of anomalies. The input is the data sent from the terminal, and the output is detailed information about the anomalies as a result of the analysis.

[0463] Step 5:

[0464] The server generates information about the repair.

[0465] The server generates information, including repair procedures, estimated costs, and the urgency of the repair, based on the analysis results. The input is the analyzed anomaly information, and the output is information structured in a way that is easy for the user to understand.

[0466] Step 6:

[0467] The terminal presents the generated information to the user.

[0468] The terminal displays information received from the server within the application, informing the user about the problem and recommended repair methods. The input is structured information from the server, and the output is specific repair actions displayed in the user interface.

[0469] (Application Example 1)

[0470] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0471] In modern factories, machine failures significantly reduce production efficiency. However, conventional maintenance methods often make it difficult to detect failures early and respond appropriately. This invention aims to provide a technology that enables efficient diagnosis of machine failures and prompt maintenance.

[0472] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0473] In this invention, the server includes a measurement means for acquiring machine status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the measurement means, and an information generation means for generating maintenance information based on the abnormalities detected by the analysis means. This enables real-time monitoring of the machine's status and prompt response when abnormalities are detected.

[0474] "Measurement means" refers to devices or methods that acquire machine status information in response to user operations.

[0475] "Analysis means" refers to devices or methods for detecting abnormalities based on state information obtained from measurement means.

[0476] "Information generation means" refers to a device or method that generates maintenance-related information based on anomalies detected by analysis means.

[0477] "Information display means" refers to a device or method for presenting generated information to a user.

[0478] "Reading means" refers to mechanisms and technologies for acquiring audio data and image data.

[0479] "Predictive means" refers to devices or methods for identifying the location of an anomaly and the priority of its maintenance.

[0480] This system efficiently monitors the status of machinery within the factory, quickly detects abnormalities, and provides maintenance information. The system is configured as follows:

[0481] The user first installs sensors on machinery within the factory, and these sensors act as measurement tools. Specific hardware includes audio sensors and cameras. These sensors acquire status information, such as machine operating sounds and visual data, in real time.

[0482] Information acquired from sensors is transmitted to a server via devices such as smartphones and tablets. The server uses a generative AI model as an analysis tool to detect anomalies from this data. The analysis algorithm performs image recognition and audio pattern analysis to determine whether or not an anomaly is present.

[0483] When an anomaly is detected, the server, as an information generation mechanism, generates data identifying the location of the anomaly and determining the maintenance priority. Furthermore, detailed information regarding repair methods and estimated costs is also generated. This generated information is transmitted to the user's device (terminal) and displayed to the user through an information display mechanism.

[0484] This system allows users to constantly monitor the status of their machinery and take quick and appropriate action when an anomaly occurs. For example, if an unusual noise is detected on a production line's conveyor belt, the user can record the sound through the application and have it analyzed by the server. The server then diagnoses that the cause is bearing wear and presents a list of replacement parts and instructions to the user's device.

[0485] For example, a prompt message such as, "Please record sensor data and begin analysis to diagnose the machine's condition immediately. If abnormal noises are detected from the bearings, the AI ​​will identify the cause and provide instructions if repairs are necessary," can be used. This prompt encourages the user to take action and makes the system easier to use.

[0486] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0487] Step 1:

[0488] The user performs operations to acquire measurement data through sensors installed on the machine. The device acquires machine status information, such as audio and image data, in real time. The input is audio and images from the machine, and the output is digital data transferred to the user's device.

[0489] Step 2:

[0490] The terminal sends the acquired measurement data to the server. The transmission process securely transfers digital data over the internet. The input is the digital data from the terminal, and the output is the transmitted data that reaches the server.

[0491] Step 3:

[0492] The server uses a generative AI model to analyze the received data. Data analysis is performed using image recognition algorithms and speech pattern recognition models to detect anomalies. The input is measurement data, and the output is whether or not an anomaly was detected and detailed information about it.

[0493] Step 4:

[0494] The server generates maintenance information based on the analysis results. It identifies abnormal areas, prioritizes them, and creates information including necessary repair procedures and estimated costs. The input to this process is the analysis results, and the output is the maintenance information provided to the user.

[0495] Step 5:

[0496] The terminal receives maintenance information sent from the server and presents it to the user. The displayed information includes the location of the problem, repair procedures, and a list of necessary parts. The input is maintenance information from the server, and the output is data displayed for the user to view.

[0497] This series of processes allows for efficient assessment of the machine's condition and enables prompt maintenance responses.

[0498] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0499] The present invention provides a system that can take into account the user's emotions when diagnosing bicycle malfunctions. This system comprises sensor means, analysis means, information generation means, and information display means, and further incorporates an emotion engine, making it possible to adjust the diagnostic process according to the user's emotions. The embodiments thereof are described in detail below.

[0500] The user launches the application on a smartphone or other device and begins diagnosing the bicycle. The device acquires the bicycle's status through sensor means. This status information includes image data from a camera, audio data from a microphone, and data from other sensors. The collected data is transmitted to a server in real time, and analysis means installed on the server analyze the data to detect abnormalities.

[0501] Once the anomaly has been identified, the server generates repair information using an information generation system. Meanwhile, the emotion engine analyzes data such as the user's voice tone and facial expressions to understand their emotional state. This emotional state is identified as, for example, excitement, worry, or calmness. Based on the user's emotional state, the emotion engine adjusts the level of detail and presentation of the diagnostic results that the information display system presents to the user.

[0502] For example, if the emotion engine determines that the user is experiencing high levels of stress due to a problem with their bicycle, the device will adjust to gently and carefully explain the diagnostic results to reassure the user. It will also calmly and concisely communicate the need for repairs and their urgency to prevent user confusion.

[0503] Thus, the present invention can improve the user experience and support the optimal repair approach by providing information tailored to the emotional state of each individual user.

[0504] The following describes the processing flow.

[0505] Step 1:

[0506] The user launches a smartphone application and enters a command to start the diagnosis. In response, the device activates various sensors on the bicycle and prepares to monitor the bicycle's condition.

[0507] Step 2:

[0508] The device uses sensors to collect status information, such as taking pictures of the bicycle's appearance with a camera, collecting abnormal sounds with a microphone, and acquiring vibration data from other sensors. The collected data serves as basic information for determining abnormalities.

[0509] Step 3:

[0510] The device transmits data acquired by its sensors to the server in real time. This data includes image information, audio data, and vibration patterns, and the server analyzes this information in detail after receiving it.

[0511] Step 4:

[0512] The server uses analytical tools to analyze the received data and identify which part of the bicycle is abnormal. This analysis process uses generative AI models for pattern recognition and anomaly detection.

[0513] Step 5:

[0514] The server generates repair recommendations and estimated costs based on detected anomalies using a repair information generation mechanism. This clearly indicates specific repair methods and urgency.

[0515] Step 6:

[0516] The server uses an emotion engine to analyze the user's voice tone and facial expression data to assess the user's emotional state. If it determines that the user is stressed, it adjusts the tone and level of detail of the information provided.

[0517] Step 7:

[0518] Based on the results from the emotion engine, the server sends optimized diagnostic information and repair recommendations to the device.

[0519] Step 8:

[0520] The device displays information received from the server to the user. The results are presented carefully and accurately, taking the user's feelings into consideration, allowing the user to review the diagnostic results and recommended actions.

[0521] (Example 2)

[0522] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0523] When bicycles or other equipment malfunction, it is necessary to quickly and accurately identify the cause of the malfunction, provide appropriate repair instructions to the user, and present information while taking the user's emotional state into consideration. In particular, a challenge is to provide information that allows users to work on repairs with peace of mind without feeling excessively stressed about the malfunction.

[0524] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0525] In this invention, the server includes a sensor means for acquiring device status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the sensor means, an information generation means for generating repair information based on the abnormalities detected by the analysis means, an emotion analysis means for analyzing the user's emotional information, and a display control means for presenting information to the user based on the generated information and the user's emotional information. This enables the rapid and accurate identification of the cause of device failure and the presentation of appropriate and reassuring information according to the user's emotional state.

[0526] A "user" refers to a person who operates this system and receives information regarding the diagnosis and repair of bicycles and equipment.

[0527] "Sensor means" refers to a set of devices or methods for acquiring status information of equipment through user operation.

[0528] "Analysis means" refers to a technology or process for analyzing state information acquired by sensor means and detecting anomalies.

[0529] "Information generation means" refers to a technology or system for creating information related to repairs based on anomalies identified by analysis means.

[0530] "Emotional analysis methods" refer to means of analyzing a user's emotional information to understand their stress levels and emotional state.

[0531] "Display control means" refers to a technology or method for presenting information to the user in the most optimal form based on generated information and emotional information.

[0532] The system of the present invention diagnoses equipment failures via a user-owned terminal and provides appropriate information regarding repairs. Specific embodiments are described below.

[0533] The user first launches the application on their device. The device is equipped with sensory means for collecting data. These sensory means include a camera, microphone, and various other sensors, which acquire image and audio data regarding the device's status. This collected status data is transmitted from the device to the server in real time.

[0534] The server processes the large volume of data it receives using analytical tools. Generative AI models are used for analysis, identifying the cause of failures and abnormal locations based on machine learning algorithms. For example, it can detect abnormal damage from images captured by cameras and identify unusual sounds from audio recorded by microphones.

[0535] Furthermore, the server is equipped with emotion analysis capabilities. This allows it to extract emotional information from the user's voice tone and camera footage, thereby understanding their emotional state. Emotion recognition technology is used for emotion analysis, visually and audibly evaluating the user's stress and anxiety.

[0536] The server integrates analysis results and emotional information to create repair information to be provided to the user through an information generation means. The information is adjusted considering the user's emotional state and presented to the user via a display control means.

[0537] For example, if the server determines that the user is experiencing stress, the terminal will display a gentle and reassuring guide during the repair process. An example of a prompt message is input into the generating AI model: "Based on the user's voice and facial expression data, identify their current emotional state and suggest the most appropriate method for presenting the diagnostic results." This input then performs emotion analysis.

[0538] Thus, the system of the present invention enables users to repair equipment with peace of mind through highly accurate fault diagnosis that takes into account the user's emotional state and the presentation of appropriate information.

[0539] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0540] Step 1:

[0541] The user launches an application on their device to diagnose a problem with their bicycle. The device prepares the sensors and begins collecting data from the equipment. Inputs include image data from the camera, audio data from the microphone, and acceleration and vibration data from other sensors. This data is acquired in real time from each sensor. The output is the collected equipment status data.

[0542] Step 2:

[0543] The terminal sends the collected data to the server. For security reasons, the data is encrypted before transmission. The server processes the received data using analytical methods. Inputs include image data, audio data, and sensor data transmitted from the terminal. Data processing uses a generative AI model and applies anomaly detection algorithms to identify faulty parts and abnormal conditions in the equipment. The output is the analyzed anomaly information.

[0544] Step 3:

[0545] The server then uses emotion analysis to determine the user's emotional state. The inputs are the user's voice tone and facial expression data captured by the camera. Using emotion analysis technology, the server identifies the user's current emotional state by analyzing their stress level and emotional condition. The output is an evaluation of the user's emotional state.

[0546] Step 4:

[0547] The server combines analyzed anomaly information and emotional information to create appropriate repair information for the user using an information generation mechanism. Here, the input consists of two parts: anomaly information and emotional information. Logic and prompt statements are used for information generation, and a generation AI model produces information optimized for the user's state. The output is the adjusted repair information.

[0548] Step 5:

[0549] The terminal presents information to the user through display control means, based on information transmitted from the server. The terminal considers the user's emotional state and presents information with appropriate tone and visual effects. Specifically, it displays voice guidance in a gentle tone to calm the user and concise and clear text messages. Ultimately, the user is more likely to understand the specific actions that need to be taken for repair.

[0550] (Application Example 2)

[0551] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0552] In bicycle fault diagnosis, conventional systems provided diagnostic information uniformly without considering the user's emotions, resulting in an unoptimized user experience. This invention aims to support efficient bicycle maintenance while reducing stress by adjusting the diagnostic process according to the user's emotional state.

[0553] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0554] In this invention, the server includes an information gathering means for acquiring state information in response to user operations, an anomaly analysis means for detecting anomalies based on the state information acquired from the information gathering means, an information generation means for generating repair-related information based on the anomalies detected by the anomaly analysis means, and an emotion analysis means for analyzing the user's voice tone and facial expressions to determine their emotional state. This enables the presentation of optimal information according to the user's emotional state.

[0555] "Information gathering means" refers to a device or method that has the function of acquiring bicycle status information in response to user operations.

[0556] An "anomaly analysis means" is a device or process for detecting abnormalities in a bicycle based on state information obtained from an information gathering means.

[0557] "Information generation means" refers to devices or algorithms that generate information related to repairs based on detected anomalies.

[0558] An "information display means" is a device or software that has the function of presenting information to the user and adjusts the method of information presentation through sentiment analysis.

[0559] "Emotional analysis methods" refer to technologies and systems that analyze a user's voice tone and facial expressions to determine their emotional state.

[0560] "Means of recording" refers to devices such as cameras and microphones used to acquire audio and image information.

[0561] "Predictive means" refers to devices or methods that have the function of identifying the location of an anomaly and the urgency of its repair.

[0562] In order to implement this invention, it is necessary to build a system in which servers, terminals, and users cooperate with each other. In this system, processing is carried out in the following flow.

[0563] First, the user launches a dedicated application on a device (such as smart glasses or a smartphone). The device is equipped with information gathering capabilities, and uses a camera and microphone to collect audio and image information to obtain the status of the bicycle. This includes hands-free operation using smart glasses.

[0564] Next, the data collected by the terminal is transmitted to the server via the network. The server uses an anomaly analysis means to analyze the data in real time to identify any abnormalities in the bicycle. Based on the analysis results, the information generation means constructs information related to repairs.

[0565] Furthermore, the server integrates emotion analysis capabilities, determining the user's emotional state by analyzing their voice tone and facial expressions. This analysis is then used to adjust the content and presentation of information displayed to the user. This ensures that the user receives reassuring information appropriate to their emotional state. For example, if the user is feeling anxious, the system adjusts to guide them through the repair process in a gentle tone.

[0566] For example, when a customer visits a bicycle shop, the staff wearing smart glasses can quickly identify the problem and provide reassuring customer service based on the customer's facial expressions.

[0567] An example of a prompt from a generated AI model is, "Understand the customer's emotions and suggest how to adjust your response." This prompt is used by the system to consider appropriate responses based on the user's emotions.

[0568] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0569] Step 1:

[0570] The user launches the application on the device and starts the bicycle diagnostic. The device uses its camera and microphone to collect image data and audio tones of the bicycle. The input at this time is the start signal from the user's operation, and the output is the collected visual and audio data. This data is collected by the device's information collection means.

[0571] Step 2:

[0572] The terminal sends the collected data to the server. The server uses an anomaly analysis tool to detect anomalies in the bicycle from the input video and audio information. This analysis tool performs data processing such as image comparison and audio pattern recognition to identify the location and type of anomaly. The output is detailed information about which part of the bicycle is abnormal.

[0573] Step 3:

[0574] Based on the results of the anomaly analysis, the server uses information generation tools to create repair procedures and recommendations. The input is detailed information about the anomaly, and the output is specific repair procedures and precautions that the user should take. This information serves as a guide to help the user perform the repair work smoothly.

[0575] Step 4:

[0576] The server determines the user's emotional state from voice tone and facial expression data collected using emotion analysis tools. The input is the aforementioned voice and facial expression data, and the output is the analysis result regarding the user's emotions (e.g., tension, anxiety, relaxation). For example, if the voice tone is high and the tempo is fast, the server will determine that the user is in an anxious state.

[0577] Step 5:

[0578] The server adjusts how the generated repair information is presented based on the results of sentiment analysis. Through the information display means, it presents the information with a tone and level of detail that matches the user's emotions. The input is the user's emotional state and repair information, and the output is displayed to the user by the terminal as information adjusted to be easily understood by the user.

[0579] Step 6:

[0580] Finally, the user can confidently repair the bicycle based on the information provided by the device. By following the instructions displayed on the device, the problem can be resolved efficiently. The input is the adjusted information from the device, and the output is the result of the user's completed repair work.

[0581] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0582] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0584] [Fourth Embodiment]

[0585] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0586] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0587] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0589] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0591] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0592] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0593] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0594] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0595] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0596] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0597] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0598] The system of the present invention is designed to enable users to efficiently diagnose bicycle malfunctions and understand the need for and methods of repair. Specific embodiments of this system are described below.

[0599] First, the user installs an application on their smartphone or tablet to diagnose their bicycle. When the user launches the app, the device prompts them to activate the sensor device installed on the bicycle. This sensor device captures the physical condition of the bicycle, for example, by including a camera and microphone.

[0600] The terminal collects data from sensors in real time and constantly monitors for any abnormalities. Once data is collected, it is transmitted to a server via the network. The server uses a generative AI model to analyze the received data and identify which part of the bicycle is experiencing problems. This process includes pattern recognition of image and audio data supplied by the sensors.

[0601] When an anomaly is detected, the server processes the information to generate repair instructions and estimated repair costs corresponding to the location of the anomaly. The generated results are returned to the terminal in a structured format, and the terminal presents these results to the user within the application. The user can review the diagnostic results in detail on the screen and understand the recommended repair items and their urgency. If necessary, they can also search for information on the nearest repair shop and make a reservation through the application.

[0602] For example, if a bicycle brake makes an unusual noise, the user can use the app to record the sound. The device sends the audio data to a server, which analyzes it and diagnoses brake pad wear. The system then displays on the user's device that replacement is necessary and provides an estimated cost. In this way, users can effectively understand the current state of their bicycle and take prompt action.

[0603] The following describes the processing flow.

[0604] Step 1:

[0605] The user launches the smartphone application and presses the button to start the bicycle diagnostic. This prompts the application to activate the sensors attached to the bicycle.

[0606] Step 2:

[0607] The device activates various sensors mounted on the bicycle (e.g., camera, microphone, vibration sensor) and begins collecting data. The camera captures the visible state of the bicycle, the microphone records sound, and the vibration sensor detects abnormal vibrations.

[0608] Step 3:

[0609] The terminal processes the collected data in real time to detect anomalies. If the collected data indicates an anomaly, it is converted to an appropriate format and prepared for the next processing step.

[0610] Step 4:

[0611] The terminal collects and processes data, which is then sent to the server. This transmission utilizes a network connection to ensure that the data reaches the server safely and quickly.

[0612] Step 5:

[0613] The server inputs the received data into a generating AI model, which then performs analysis to detect anomalies. The AI ​​model uses pattern recognition technology to analyze, for example, audio data or image data, and identify the location of the anomaly.

[0614] Step 6:

[0615] The server generates necessary repair information based on the anomalies detected by the AI. Repair procedures and estimated repair costs are calculated and compiled into information provided to the user.

[0616] Step 7:

[0617] The server sends the generated results to the terminal. This data includes the location of the problem, the urgency of the repair, the estimated cost, and recommended actions.

[0618] Step 8:

[0619] The application screen displays the diagnostic results received by the terminal from the server. The user can refer to these results and, if necessary, arrange for repairs or search for the nearest repair shop.

[0620] (Example 1)

[0621] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0622] For many users, quickly and accurately diagnosing the location of a malfunction in objects such as bicycles, and determining the appropriate repair method and urgency, is a technically challenging task. Furthermore, expert diagnosis and repair are time-consuming and costly, so there is a need for a system that allows users to easily understand the cause of a malfunction and quickly decide on a course of action.

[0623] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0624] In this invention, the server includes sensing means for acquiring state information of an object in response to user operation, analysis means for detecting anomalies based on the state information acquired from the sensing means, and information generation means for generating information related to repairs based on the anomalies detected by the analysis means. This enables the user to understand the specific circumstances of the failure and to take appropriate repair action quickly.

[0625] A "user" is someone who operates the system and obtains diagnostic and repair information for an object.

[0626] "Sensing means" refers to functions or devices for acquiring state information of an object in response to user operations.

[0627] "Analysis means" refers to functions or devices for detecting abnormalities based on acquired state information.

[0628] "Information generation means" refers to functions or devices for generating information related to repairs based on detected anomalies.

[0629] "Information display means" refers to functions or devices for presenting generated information to the user.

[0630] "Communication means" refers to functions or devices for transmitting state information to analysis means via a network.

[0631] "Identification means" refers to functions or devices that perform pattern recognition using generative AI models.

[0632] "Recording means" refers to functions or devices for acquiring audio data and image data.

[0633] "Inference means" refers to functions or devices that indicate the location of an anomaly and the priority of its repair.

[0634] This system allows users to efficiently diagnose the condition of objects such as bicycles and determine the need for repairs and how to do so. First, the user installs a diagnostic application on their smartphone or tablet. The application uses sensing devices attached to the object to collect condition information, including audio and image data.

[0635] The terminal acquires data from the sensing device in real time and transmits that data to the server via a network connection. The server analyzes this data using a generative AI model to identify anomalies. Specifically, it uses image recognition technology and voice analysis technology to identify the location and nature of the malfunction.

[0636] The server generates repair information based on the analysis results. This information includes identifying the problem area, repair procedures, estimated repair costs, and urgency. The generated information is presented to the user via a terminal. Based on the information obtained from the terminal, the user can understand the current state of their bicycle and take appropriate action.

[0637] For example, if a bicycle brake makes an unusual noise, the user uses the application to record the sound. The device sends this audio data to a server, which diagnoses brake pad wear. As a result, it indicates that the brake pads need replacing and provides an estimated cost.

[0638] In this process, using a generative AI model allows users to understand the condition of an object without the help of an expert and take prompt repair action. By using prompts such as, "Please perform a diagnostic analysis of the unusual noise coming from my bicycle brakes and tell me what repairs are needed and the estimated cost," the AI ​​model can perform a highly accurate diagnosis.

[0639] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0640] Step 1:

[0641] The user launches the application.

[0642] The user prepares to begin the diagnosis by launching the app on their smartphone or tablet. The app displays an interface to activate the sensing device and prompts the user to take action.

[0643] Step 2:

[0644] The terminal acquires status information from the sensing device.

[0645] The terminal activates a sensor attached to the bicycle and collects audio and image data. The input is data about the physical state of the bicycle, and the output is formatted information containing this data.

[0646] Step 3:

[0647] The device sends the collected data to the server.

[0648] The terminal compresses the collected audio and image data using a network connection and sends it to the server. The input is raw data from the sensing device, and the output is data converted into a format that the server can analyze.

[0649] Step 4:

[0650] The server analyzes the data and identifies anomalies.

[0651] The server analyzes the transmitted data using a generative AI model. This process involves pattern recognition to identify the location and extent of anomalies. The input is the data sent from the terminal, and the output is detailed information about the anomalies as a result of the analysis.

[0652] Step 5:

[0653] The server generates information about the repair.

[0654] The server generates information, including repair procedures, estimated costs, and the urgency of the repair, based on the analysis results. The input is the analyzed anomaly information, and the output is information structured in a way that is easy for the user to understand.

[0655] Step 6:

[0656] The terminal presents the generated information to the user.

[0657] The terminal displays information received from the server within the application, informing the user about the problem and recommended repair methods. The input is structured information from the server, and the output is specific repair actions displayed in the user interface.

[0658] (Application Example 1)

[0659] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0660] In modern factories, machine failures significantly reduce production efficiency. However, conventional maintenance methods often make it difficult to detect failures early and respond appropriately. This invention aims to provide a technology that enables efficient diagnosis of machine failures and prompt maintenance.

[0661] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0662] In this invention, the server includes a measurement means for acquiring machine status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the measurement means, and an information generation means for generating maintenance information based on the abnormalities detected by the analysis means. This enables real-time monitoring of the machine's status and prompt response when abnormalities are detected.

[0663] "Measurement means" refers to devices or methods that acquire machine status information in response to user operations.

[0664] "Analysis means" refers to devices or methods for detecting abnormalities based on state information obtained from measurement means.

[0665] "Information generation means" refers to a device or method that generates maintenance-related information based on anomalies detected by analysis means.

[0666] "Information display means" refers to a device or method for presenting generated information to a user.

[0667] "Reading means" refers to mechanisms and technologies for acquiring audio data and image data.

[0668] "Predictive means" refers to devices or methods for identifying the location of an anomaly and the priority of its maintenance.

[0669] This system efficiently monitors the status of machinery within the factory, quickly detects abnormalities, and provides maintenance information. The system is configured as follows:

[0670] The user first installs sensors on machinery within the factory, and these sensors act as measurement tools. Specific hardware includes audio sensors and cameras. These sensors acquire status information, such as machine operating sounds and visual data, in real time.

[0671] Information acquired from sensors is transmitted to a server via devices such as smartphones and tablets. The server uses a generative AI model as an analysis tool to detect anomalies from this data. The analysis algorithm performs image recognition and audio pattern analysis to determine whether or not an anomaly is present.

[0672] When an anomaly is detected, the server, as an information generation mechanism, generates data identifying the location of the anomaly and determining the maintenance priority. Furthermore, detailed information regarding repair methods and estimated costs is also generated. This generated information is transmitted to the user's device (terminal) and displayed to the user through an information display mechanism.

[0673] This system allows users to constantly monitor the status of their machinery and take quick and appropriate action when an anomaly occurs. For example, if an unusual noise is detected on a production line's conveyor belt, the user can record the sound through the application and have it analyzed by the server. The server then diagnoses that the cause is bearing wear and presents a list of replacement parts and instructions to the user's device.

[0674] For example, a prompt message such as, "Please record sensor data and begin analysis to diagnose the machine's condition immediately. If abnormal noises are detected from the bearings, the AI ​​will identify the cause and provide instructions if repairs are necessary," can be used. This prompt encourages the user to take action and makes the system easier to use.

[0675] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0676] Step 1:

[0677] The user performs operations to acquire measurement data through sensors installed on the machine. The device acquires machine status information, such as audio and image data, in real time. The input is audio and images from the machine, and the output is digital data transferred to the user's device.

[0678] Step 2:

[0679] The terminal sends the acquired measurement data to the server. The transmission process securely transfers digital data over the internet. The input is the digital data from the terminal, and the output is the transmitted data that reaches the server.

[0680] Step 3:

[0681] The server uses a generative AI model to analyze the received data. Data analysis is performed using image recognition algorithms and speech pattern recognition models to detect anomalies. The input is measurement data, and the output is whether or not an anomaly was detected and detailed information about it.

[0682] Step 4:

[0683] The server generates maintenance information based on the analysis results. It identifies abnormal areas, prioritizes them, and creates information including necessary repair procedures and estimated costs. The input to this process is the analysis results, and the output is the maintenance information provided to the user.

[0684] Step 5:

[0685] The terminal receives maintenance information sent from the server and presents it to the user. The displayed information includes the location of the problem, repair procedures, and a list of necessary parts. The input is maintenance information from the server, and the output is data displayed for the user to view.

[0686] This series of processes allows for efficient assessment of the machine's condition and enables prompt maintenance responses.

[0687] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0688] The present invention provides a system that can take into account the user's emotions when diagnosing bicycle malfunctions. This system comprises sensor means, analysis means, information generation means, and information display means, and further incorporates an emotion engine, making it possible to adjust the diagnostic process according to the user's emotions. The embodiments thereof are described in detail below.

[0689] The user launches the application on a smartphone or other device and begins diagnosing the bicycle. The device acquires the bicycle's status through sensor means. This status information includes image data from a camera, audio data from a microphone, and data from other sensors. The collected data is transmitted to a server in real time, and analysis means installed on the server analyze the data to detect abnormalities.

[0690] Once the anomaly has been identified, the server generates repair information using an information generation system. Meanwhile, the emotion engine analyzes data such as the user's voice tone and facial expressions to understand their emotional state. This emotional state is identified as, for example, excitement, worry, or calmness. Based on the user's emotional state, the emotion engine adjusts the level of detail and presentation of the diagnostic results that the information display system presents to the user.

[0691] For example, if the emotion engine determines that the user is experiencing high levels of stress due to a problem with their bicycle, the device will adjust to gently and carefully explain the diagnostic results to reassure the user. It will also calmly and concisely communicate the need for repairs and their urgency to prevent user confusion.

[0692] Thus, the present invention can improve the user experience and support the optimal repair approach by providing information tailored to the emotional state of each individual user.

[0693] The following describes the processing flow.

[0694] Step 1:

[0695] The user launches a smartphone application and enters a command to start the diagnosis. In response, the device activates various sensors on the bicycle and prepares to monitor the bicycle's condition.

[0696] Step 2:

[0697] The device uses sensors to collect status information, such as taking pictures of the bicycle's appearance with a camera, collecting abnormal sounds with a microphone, and acquiring vibration data from other sensors. The collected data serves as basic information for determining abnormalities.

[0698] Step 3:

[0699] The device transmits data acquired by its sensors to the server in real time. This data includes image information, audio data, and vibration patterns, and the server analyzes this information in detail after receiving it.

[0700] Step 4:

[0701] The server uses analytical tools to analyze the received data and identify which part of the bicycle is abnormal. This analysis process uses generative AI models for pattern recognition and anomaly detection.

[0702] Step 5:

[0703] The server generates repair recommendations and estimated costs based on detected anomalies using a repair information generation mechanism. This clearly indicates specific repair methods and urgency.

[0704] Step 6:

[0705] The server uses an emotion engine to analyze the user's voice tone and facial expression data to assess the user's emotional state. If it determines that the user is stressed, it adjusts the tone and level of detail of the information provided.

[0706] Step 7:

[0707] Based on the results from the emotion engine, the server sends optimized diagnostic information and repair recommendations to the device.

[0708] Step 8:

[0709] The device displays information received from the server to the user. The results are presented carefully and accurately, taking the user's feelings into consideration, allowing the user to review the diagnostic results and recommended actions.

[0710] (Example 2)

[0711] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0712] When bicycles or other equipment malfunction, it is necessary to quickly and accurately identify the cause of the malfunction, provide appropriate repair instructions to the user, and present information while taking the user's emotional state into consideration. In particular, a challenge is to provide information that allows users to work on repairs with peace of mind without feeling excessively stressed about the malfunction.

[0713] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0714] In this invention, the server includes a sensor means for acquiring device status information in response to user operations, an analysis means for detecting abnormalities based on the status information acquired from the sensor means, an information generation means for generating repair information based on the abnormalities detected by the analysis means, an emotion analysis means for analyzing the user's emotional information, and a display control means for presenting information to the user based on the generated information and the user's emotional information. This enables the rapid and accurate identification of the cause of device failure and the presentation of appropriate and reassuring information according to the user's emotional state.

[0715] A "user" refers to a person who operates this system and receives information regarding the diagnosis and repair of bicycles and equipment.

[0716] "Sensor means" refers to a set of devices or methods for acquiring status information of equipment through user operation.

[0717] "Analysis means" refers to a technology or process for analyzing state information acquired by sensor means and detecting anomalies.

[0718] "Information generation means" refers to a technology or system for creating information related to repairs based on anomalies identified by analysis means.

[0719] "Emotional analysis methods" refer to means of analyzing a user's emotional information to understand their stress levels and emotional state.

[0720] "Display control means" refers to a technology or method for presenting information to the user in the most optimal form based on generated information and emotional information.

[0721] The system of the present invention diagnoses equipment failures via a user-owned terminal and provides appropriate information regarding repairs. Specific embodiments are described below.

[0722] The user first launches the application on their device. The device is equipped with sensory means for collecting data. These sensory means include a camera, microphone, and various other sensors, which acquire image and audio data regarding the device's status. This collected status data is transmitted from the device to the server in real time.

[0723] The server processes the large volume of data it receives using analytical tools. Generative AI models are used for analysis, identifying the cause of failures and abnormal locations based on machine learning algorithms. For example, it can detect abnormal damage from images captured by cameras and identify unusual sounds from audio recorded by microphones.

[0724] Furthermore, the server is equipped with emotion analysis capabilities. This allows it to extract emotional information from the user's voice tone and camera footage, thereby understanding their emotional state. Emotion recognition technology is used for emotion analysis, visually and audibly evaluating the user's stress and anxiety.

[0725] The server integrates analysis results and emotional information to create repair information to be provided to the user through an information generation means. The information is adjusted considering the user's emotional state and presented to the user via a display control means.

[0726] For example, if the server determines that the user is experiencing stress, the terminal will display a gentle and reassuring guide during the repair process. An example of a prompt message is input into the generating AI model: "Based on the user's voice and facial expression data, identify their current emotional state and suggest the most appropriate method for presenting the diagnostic results." This input then performs emotion analysis.

[0727] Thus, the system of the present invention enables users to repair equipment with peace of mind through highly accurate fault diagnosis that takes into account the user's emotional state and the presentation of appropriate information.

[0728] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0729] Step 1:

[0730] The user launches an application on their device to diagnose a problem with their bicycle. The device prepares the sensors and begins collecting data from the equipment. Inputs include image data from the camera, audio data from the microphone, and acceleration and vibration data from other sensors. This data is acquired in real time from each sensor. The output is the collected equipment status data.

[0731] Step 2:

[0732] The terminal sends the collected data to the server. For security reasons, the data is encrypted before transmission. The server processes the received data using analytical methods. Inputs include image data, audio data, and sensor data transmitted from the terminal. Data processing uses a generative AI model and applies anomaly detection algorithms to identify faulty parts and abnormal conditions in the equipment. The output is the analyzed anomaly information.

[0733] Step 3:

[0734] The server then uses emotion analysis to determine the user's emotional state. The inputs are the user's voice tone and facial expression data captured by the camera. Using emotion analysis technology, the server identifies the user's current emotional state by analyzing their stress level and emotional condition. The output is an evaluation of the user's emotional state.

[0735] Step 4:

[0736] The server combines analyzed anomaly information and emotional information to create appropriate repair information for the user using an information generation mechanism. Here, the input consists of two parts: anomaly information and emotional information. Logic and prompt statements are used for information generation, and a generation AI model produces information optimized for the user's state. The output is the adjusted repair information.

[0737] Step 5:

[0738] The terminal presents information to the user through display control means, based on information transmitted from the server. The terminal considers the user's emotional state and presents information with appropriate tone and visual effects. Specifically, it displays voice guidance in a gentle tone to calm the user and concise and clear text messages. Ultimately, the user is more likely to understand the specific actions that need to be taken for repair.

[0739] (Application Example 2)

[0740] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0741] In bicycle fault diagnosis, conventional systems provided diagnostic information uniformly without considering the user's emotions, resulting in an unoptimized user experience. This invention aims to support efficient bicycle maintenance while reducing stress by adjusting the diagnostic process according to the user's emotional state.

[0742] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0743] In this invention, the server includes an information gathering means for acquiring state information in response to user operations, an anomaly analysis means for detecting anomalies based on the state information acquired from the information gathering means, an information generation means for generating repair-related information based on the anomalies detected by the anomaly analysis means, and an emotion analysis means for analyzing the user's voice tone and facial expressions to determine their emotional state. This enables the presentation of optimal information according to the user's emotional state.

[0744] "Information gathering means" refers to a device or method that has the function of acquiring bicycle status information in response to user operations.

[0745] An "anomaly analysis means" is a device or process for detecting abnormalities in a bicycle based on state information obtained from an information gathering means.

[0746] "Information generation means" refers to devices or algorithms that generate information related to repairs based on detected anomalies.

[0747] An "information display means" is a device or software that has the function of presenting information to the user and adjusts the method of information presentation through sentiment analysis.

[0748] "Emotional analysis methods" refer to technologies and systems that analyze a user's voice tone and facial expressions to determine their emotional state.

[0749] "Means of recording" refers to devices such as cameras and microphones used to acquire audio and image information.

[0750] "Predictive means" refers to devices or methods that have the function of identifying the location of an anomaly and the urgency of its repair.

[0751] In order to implement this invention, it is necessary to build a system in which servers, terminals, and users cooperate with each other. In this system, processing is carried out in the following flow.

[0752] First, the user launches a dedicated application on a device (such as smart glasses or a smartphone). The device is equipped with information gathering capabilities, and uses a camera and microphone to collect audio and image information to obtain the status of the bicycle. This includes hands-free operation using smart glasses.

[0753] Next, the data collected by the terminal is transmitted to the server via the network. The server uses an anomaly analysis means to analyze the data in real time to identify any abnormalities in the bicycle. Based on the analysis results, the information generation means constructs information related to repairs.

[0754] Furthermore, the server integrates emotion analysis capabilities, determining the user's emotional state by analyzing their voice tone and facial expressions. This analysis is then used to adjust the content and presentation of information displayed to the user. This ensures that the user receives reassuring information appropriate to their emotional state. For example, if the user is feeling anxious, the system adjusts to guide them through the repair process in a gentle tone.

[0755] For example, when a customer visits a bicycle shop, the staff wearing smart glasses can quickly identify the problem and provide reassuring customer service based on the customer's facial expressions.

[0756] An example of a prompt from a generated AI model is, "Understand the customer's emotions and suggest how to adjust your response." This prompt is used by the system to consider appropriate responses based on the user's emotions.

[0757] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0758] Step 1:

[0759] The user launches the application on the device and starts the bicycle diagnostic. The device uses its camera and microphone to collect image data and audio tones of the bicycle. The input at this time is the start signal from the user's operation, and the output is the collected visual and audio data. This data is collected by the device's information collection means.

[0760] Step 2:

[0761] The terminal sends the collected data to the server. The server uses an anomaly analysis tool to detect anomalies in the bicycle from the input video and audio information. This analysis tool performs data processing such as image comparison and audio pattern recognition to identify the location and type of anomaly. The output is detailed information about which part of the bicycle is abnormal.

[0762] Step 3:

[0763] Based on the results of the anomaly analysis, the server uses information generation tools to create repair procedures and recommendations. The input is detailed information about the anomaly, and the output is specific repair procedures and precautions that the user should take. This information serves as a guide to help the user perform the repair work smoothly.

[0764] Step 4:

[0765] The server determines the user's emotional state from voice tone and facial expression data collected using emotion analysis tools. The input is the aforementioned voice and facial expression data, and the output is the analysis result regarding the user's emotions (e.g., tension, anxiety, relaxation). For example, if the voice tone is high and the tempo is fast, the server will determine that the user is in an anxious state.

[0766] Step 5:

[0767] The server adjusts how the generated repair information is presented based on the results of sentiment analysis. Through the information display means, it presents the information with a tone and level of detail that matches the user's emotions. The input is the user's emotional state and repair information, and the output is displayed to the user by the terminal as information adjusted to be easily understood by the user.

[0768] Step 6:

[0769] Finally, the user can confidently repair the bicycle based on the information provided by the device. By following the instructions displayed on the device, the problem can be resolved efficiently. The input is the adjusted information from the device, and the output is the result of the user's completed repair work.

[0770] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0771] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0773] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0774] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0775] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0776] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0777] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0778] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0779] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0780] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0781] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0782] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0783] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0784] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0785] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0786] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0787] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0788] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0789] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0790] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0791] The following is further disclosed regarding the embodiments described above.

[0792] (Claim 1)

[0793] A sensor means that acquires state information in response to user operations,

[0794] An analysis means for detecting an anomaly based on state information acquired from the aforementioned sensor means,

[0795] Information generation means that generates information related to repairs based on the abnormality detected by the analysis means,

[0796] Information display means for presenting the generated information to the user,

[0797] A system that includes this.

[0798] (Claim 2)

[0799] The system according to claim 1, wherein the sensor means includes a scanner means for acquiring audio data and image data.

[0800] (Claim 3)

[0801] The system according to claim 1, wherein the information generation means includes an estimation means for identifying the location of an abnormality and the priority of its repair.

[0802] "Example 1"

[0803] (Claim 1)

[0804] A sensing means that acquires state information of an object in response to user operation,

[0805] An analysis means for detecting an anomaly based on state information acquired from the sensing means,

[0806] Information generation means that generates information related to repairs based on the abnormality detected by the analysis means,

[0807] Information display means for presenting the generated information to the user,

[0808] A communication means for transmitting the aforementioned state information to the analysis means via a network,

[0809] The aforementioned analysis means includes an identification means that performs pattern recognition using a generated AI model,

[0810] A system that includes this.

[0811] (Claim 2)

[0812] The system according to claim 1, wherein the sensing means includes recording means for acquiring audio data and image data.

[0813] (Claim 3)

[0814] The system according to claim 1, wherein the information generation means includes an inference means for indicating the location of an abnormality and the priority of its repair.

[0815] "Application Example 1"

[0816] (Claim 1)

[0817] A measurement means for acquiring machine status information in response to user operations,

[0818] An analysis means for detecting an anomaly based on state information obtained from the measurement means,

[0819] Information generation means that generates maintenance information based on the abnormality detected by the analysis means,

[0820] Information display means for presenting the generated information to the user,

[0821] A system that includes this.

[0822] (Claim 2)

[0823] The system according to claim 1, wherein the measurement means includes a reading means for acquiring audio data and image data.

[0824] (Claim 3)

[0825] The system according to claim 1, wherein the information generation means includes an estimation means for identifying the location of an anomaly and the priority of its maintenance.

[0826] "Example 2 of combining an emotion engine"

[0827] (Claim 1)

[0828] A sensor means that acquires device status information in response to user operation,

[0829] An analysis means for detecting an anomaly based on state information acquired from the aforementioned sensor means,

[0830] Information generation means that generates information related to repairs based on the abnormality detected by the analysis means,

[0831] A means of analyzing user emotional information,

[0832] A display control means that presents information to the user based on the generated information and the user's emotional information,

[0833] A system that includes this.

[0834] (Claim 2)

[0835] The system according to claim 1, wherein the sensor means includes a collection function for acquiring audio data and image data.

[0836] (Claim 3)

[0837] The system according to claim 1, wherein the information generation means includes prediction technology for identifying the location of an anomaly and the priority of its repair, and the display control means generates a message corresponding to emotional information.

[0838] "Application example 2 when combining with an emotional engine"

[0839] (Claim 1)

[0840] Information gathering means for acquiring state information in response to user operations,

[0841] An anomaly analysis means for detecting anomalies based on state information obtained from the aforementioned information collection means,

[0842] An information generation means that generates information related to repairs based on the abnormality detected by the abnormality analysis means,

[0843] Information display means that adjusts and presents the generated information according to the user's emotional state,

[0844] An emotion analysis method that analyzes the user's voice tone and facial expressions to determine their emotional state,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, wherein the information gathering means includes a shooting means for acquiring audio information and image information.

[0848] (Claim 3)

[0849] The system according to claim 1, wherein the information generation means includes an estimation means for identifying the location of an abnormality and the urgency of its repair. [Explanation of symbols]

[0850] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A sensor means that acquires state information in response to user operations, An analysis means for detecting an anomaly based on state information acquired from the aforementioned sensor means, Information generation means that generates information related to repairs based on the abnormality detected by the analysis means, Information display means for presenting the generated information to the user, A system that includes this.

2. The system according to claim 1, wherein the sensor means includes a scanner means for acquiring audio data and image data.

3. The system according to claim 1, wherein the information generation means includes an estimation means for identifying the location of an abnormality and the priority of its repair.

Images