System

Abstract

A system is provided.SOLUTION: A system comprising: means for performing voice pattern analysis using a AI model trained in advance based on a feature amount extracted by the server; means for determining presence or absence of an abnormal sound or a specific pattern based on a result of the analysis and generating a determination result; and means for transmitting the determination result to the device and notifying a user of the determination result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional acoustic testing relies primarily on human subjective judgment, resulting in inconsistent and inaccurate results due to individual differences and differences in experience. Furthermore, accurate acoustic testing is difficult outdoors or in noisy environments, requiring highly skilled experts, resulting in labor shortages and training costs. The purpose of this invention is to solve these problems and improve the efficiency and reliability of acoustic testing. [Means for solving the problem]

[0005] The present invention provides an acoustic judgment support system that includes a terminal equipped with a high-sensitivity microphone that collects audio data in real time, a server that receives the audio data transmitted from the terminal and performs noise reduction and feature extraction, means for analyzing audio patterns using a pre-trained AI model based on the features extracted by the server, means for determining the presence or absence of abnormal sounds or specific patterns based on the analysis results and generating a judgment result, and means for transmitting the judgment result to the terminal and notifying the user. This system improves the objectivity and accuracy of acoustic inspections, and can solve problems such as labor shortages and the cost of training experts.

[0006] "Audio data" refers to data that converts sound waves into digital signals for recording and processing.

[0007] A "high-sensitivity microphone" is a microphone that can collect minute fluctuations in sound with high precision.

[0008] A "terminal" is a device for collecting, processing, and transmitting voice data.

[0009] "Server" means a computer system that processes and analyzes received voice data and manages and provides the results.

[0010] "Noise reduction" is the process of removing unwanted background sounds and noise from collected audio data.

[0011] "Feature extraction" is the process of extracting important parts or features of sound from audio data.

[0012] "AI Model" means a computational model that is pre-trained to perform voice analysis using artificial intelligence algorithms.

[0013] "Voice pattern analysis" is the process of analyzing the characteristics of voice data and identifying abnormal sounds or specific patterns.

[0014] "Determination results" are information regarding the presence or absence of abnormal sounds or specific patterns generated as a result of analysis by the AI ​​model.

[0015] "Filtering" is a process that emphasizes specific frequency bands within audio data or suppresses unnecessary frequency bands. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] To implement this invention, the terminals, servers, and programs installed on them that constitute the Acoustic Discrimination Support System (ADSS) are required. Specific processes performed by the server, terminals, and users are described below.

[0038] Specific processing of the terminal

[0039] Audio data collection

[0040] The device uses a highly sensitive microphone to collect audio data in real time, allowing it to accurately capture data from specific sound sources, such as a vehicle engine or machinery on a production line.

[0041] Digital Conversion and Filtering

[0042] The collected audio data is converted from analog to digital format and then denoised and filtered, which suppresses background noise and emphasises the desired frequency bands.

[0043] Sending data

[0044] The pre-processed audio data is sent over the network to a server, where it is compressed and encrypted for secure and efficient transmission.

[0045] Server specific processing

[0046] Data Receipt and Storage

[0047] The server receives the voice data sent from the device and stores it in temporary storage, verifies the integrity of the data, and prepares it for analysis.

[0048] Feature extraction and analysis preparation

[0049] The server loads the trained AI model and extracts features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for voice pattern analysis.

[0050] Voice pattern analysis and judgment

[0051] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns. Based on the results, it is determined whether an abnormality exists or whether a specific pattern exists.

[0052] Generation and notification of judgment results

[0053] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The generated judgment result is sent to the terminal and notified to the user.

[0054] User specific processing

[0055] Check the results

[0056] The user can check the results of the assessment through their device. For example, if an abnormality is detected, the user is notified by an alert sound or a warning icon.

[0057] View detailed data and take action

[0058] If necessary, the user can check the detailed audio data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, machine maintenance, part replacement, or further inspection can be carried out.

[0059] Example: Vehicle engine sound inspection

[0060] Device behavior

[0061] A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time. The collected audio data is converted into a digital format, and then noise reduction and filtering are performed. The pre-processed data is then sent to a server.

[0062] Server Operation

[0063] The server temporarily stores the received voice data and extracts features from the voice data. The extracted features are input into an AI model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the device.

[0064] User Actions

[0065] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, the user takes measures such as replacing the bearings.

[0066] In this way, acoustic inspection work can be carried out quickly and accurately. This system significantly improves the efficiency and reliability of acoustic inspections, and also solves the problems of labor shortages and training costs.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or factory machinery.

[0070] Step 2:

[0071] The device converts the collected voice data from analog to digital signals, making it in a format that can be analyzed.

[0072] Step 3:

[0073] The device then filters the digital audio data, reducing background noise and emphasizing important frequency bands.

[0074] Step 4:

[0075] The terminal then packetizes the pre-processed voice data, encrypts it, and sends it to the server, ensuring secure data transmission.

[0076] Step 5:

[0077] The server receives the voice data sent from the device and temporarily stores it in storage. It also performs error checking to ensure data integrity.

[0078] Step 6:

[0079] The server loads the AI ​​model and prepares it for analysis, using a pre-trained model.

[0080] Step 7:

[0081] The server extracts features from the audio data, specifically, frequency spectrum, amplitude, phase, etc.

[0082] Step 8:

[0083] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[0084] Step 9:

[0085] The server generates a judgment result based on the analysis results, such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[0086] Step 10:

[0087] The server generates a judgment result and sends it to the terminal, compressing and encrypting the data as necessary.

[0088] Step 11:

[0089] The terminal displays the result of the judgment received on the user interface, for example, by notifying the user using an alert sound or a warning icon.

[0090] Step 12:

[0091] The user checks the judgment results displayed on the device and, if necessary, can also view detailed voice data analysis results.

[0092] Step 13:

[0093] The user takes appropriate measures based on the judgment results. For example, if an abnormal sound is detected, the user performs machine maintenance or replaces parts.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Conventional acoustic detection systems lack the ability to collect acoustic data in real time and analyze it quickly and accurately, making it difficult to distinguish between normal and abnormal movement patterns. They also lacked the functionality to efficiently determine the presence or absence of abnormal sounds or specific patterns, resulting in delays in providing appropriate notifications and feedback to users. Furthermore, the entire process, from collecting acoustic data to analyzing it and notifying users, was not integrated, resulting in low efficiency for the entire system.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes a terminal equipped with a highly sensitive sensor that collects acoustic data in real time, a processing device that receives the acoustic data transmitted from the terminal and performs noise removal and feature extraction, means for performing acoustic pattern analysis using a pre-trained machine learning model based on the features extracted by the processing device, means for determining the presence or absence of abnormal acoustics or specific patterns based on the analysis results and generating a determination result, and means for transmitting the determination result to the terminal and notifying a user. This makes it possible to collect acoustic data in real time, analyze it quickly and accurately, efficiently determine the presence or absence of abnormal acoustics or specific patterns, and provide appropriate notification and feedback to the user.

[0099] "Acoustic data" is information that is generated by converting sound into electrical signals and storing or processing them in digital form.

[0100] A "high-sensitivity sensor" is an acoustic detection device capable of detecting minute acoustic changes with high precision.

[0101] A "terminal" is a device that collects acoustic data and sends it to a server for processing.

[0102] A "processor" is a computer system that has the computational power to analyze acoustic data.

[0103] A "machine learning model" is an algorithm that is trained on large amounts of data to perform pattern recognition and prediction.

[0104] "Acoustic pattern analysis" is a process that identifies abnormalities and specific patterns based on the features of acoustic data.

[0105] "Abnormal sound" is a sound that is different from normal operating sounds and indicates an abnormality.

[0106] The "determination result" is diagnostic information generated based on the analysis of the acoustic data.

[0107] A "user" is a person who operates the system and checks the judgment results.

[0108] "Noise reduction" is a process for removing unnecessary background sounds from audio data.

[0109] "Feature extraction" is the process of extracting important patterns and properties from acoustic data.

[0110] "Real-time" refers to the simultaneous collection and processing of acoustic data.

[0111] MODE FOR CARRYING OUT THE INVENTION

[0112] To implement this invention, an Acoustic Decision Support System (ADSS) is required, which includes the following components: ADSS is realized by the roles of terminal, server, and user.

[0113] Terminal configuration and operation

[0114] Acoustic data collection

[0115] The device is equipped with highly sensitive sensors that collect acoustic data in real time from specific sound sources, such as a vehicle engine or a machine on a production line, with high accuracy. The device also includes location and time information to provide context for the collected acoustic data.

[0116] Digital Conversion and Filtering

[0117] The collected acoustic data is in analog format and is converted to digital format using an ADC (analog-to-digital converter) within the device. It is then denoised and filtered using audio processing software such as Audacity, which removes unwanted background noise and emphasises the desired frequency bands.

[0118] Sending data

[0119] The pre-processed digital audio data is then sent to the server via the network. During data transmission, the data is compressed and encrypted using gzip, TLS / SSL, etc., for safe and efficient transmission.

[0120] Server configuration and operation

[0121] Data Receipt and Storage

[0122] The server receives the acoustic data sent from the device and stores it in temporary storage (e.g., Amazon S3). It checks the integrity of the data and verifies that it is not corrupted. This temporary storage makes the data easily available for subsequent processing.

[0123] Feature extraction and analysis preparation

[0124] The server loads a trained machine learning model (e.g., TENSORFLOW® or PyTorch) and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data, preparing it for analysis.

[0125] Acoustic pattern analysis and judgment

[0126] The server uses the extracted features to perform acoustic pattern analysis. The features are input into a machine learning model, and the degree of match is calculated by comparing normal and abnormal sound patterns. Based on the results of this analysis, it determines whether an abnormality exists or whether a specific pattern exists.

[0127] Generation and notification of judgment results

[0128] The server generates a judgment result based on the analysis results. The result is displayed in the form of "No abnormality," "Abnormality detected," "Specific abnormal sound pattern detected," etc. These judgment results are sent to the terminal and notified to the user.

[0129] User Configuration and Behavior

[0130] Check the results

[0131] The user can check the results of the assessment through their device. For example, if an abnormality is detected, an alert sound or a warning icon will be displayed to alert the user.

[0132] View detailed data and take action

[0133] If necessary, the user can check the detailed acoustic data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, the user can perform machine maintenance, replace parts, or perform further inspections.

[0134] Example: Inspecting vehicle engine noise

[0135] Device behavior

[0136] Highly sensitive sensors are installed in the vehicle's engine to collect engine sounds in real time. The collected acoustic data is then digitally converted and filtered before being sent to a server via a network.

[0137] Server Operation

[0138] The server temporarily stores the received acoustic data and extracts features from the acoustic data. The extracted features are input into a machine learning model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the terminal.

[0139] User Actions

[0140] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, countermeasures such as replacing the bearings are implemented.

[0141] Examples of prompt statements

[0142] "Generate Python code to collect vehicle engine sounds in real time and detect abnormal sounds."

[0143] In this way, the Acoustic Decision Support System (ADSS) can quickly and efficiently carry out the entire process from collection to analysis and notification, improving the efficiency and reliability of acoustic inspections and resolving the problems of labor shortages and training costs.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] Acoustic data collection

[0147] The device uses a highly sensitive sensor to collect acoustic data in real time from specific sound sources. For example, it can capture the sound of a vehicle engine or the machine noise of a production line. The input is an analog acoustic signal captured by the acoustic sensor, and the output is analog acoustic data. Specifically, the device's highly sensitive sensor is placed near the target sound source and continuously collects acoustic data.

[0148] Step 2:

[0149] Digital Conversion and Filtering

[0150] The device converts the collected analog acoustic data into digital format using a built-in analog-to-digital converter (ADC). Next, the converted digital acoustic data is processed using audio processing software such as Audacity to remove noise and perform filtering. The input is analog acoustic data, and the output is filtered digital acoustic data. Specifically, the audio data is converted into digital format using the built-in ADC, and then white noise and unwanted frequencies are removed using Audacity.

[0151] Step 3:

[0152] Sending data

[0153] The device sends the preprocessed digital audio data to a server over the Internet or a local network. The data is sent in a compressed format such as gzip and encrypted format such as TLS / SSL. The input is the filtered digital audio data, and the output is compressed and encrypted audio data packets. Specifically, the digital audio data is compressed and securely sent to the server using an encryption protocol.

[0154] Step 4:

[0155] Data reception and storage

[0156] The server receives the digital audio data packets sent from the device, decompresses and decrypts them, and stores the received data in temporary storage (e.g., cloud storage). The input is the compressed and encrypted audio data packets, and the output is the digital audio data stored in the storage. Specifically, the server decompresses and decrypts the received data packets before storing them in cloud storage.

[0157] Step 5:

[0158] Feature extraction and analysis preparation

[0159] The server loads a pre-trained machine learning model (e.g., TensorFlow or PyTorch) based on the stored acoustic data, and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data. The input is the stored digital acoustic data, and the output is the extracted feature data. Specifically, the acoustic data is converted into a frequency spectrum, and amplitude and other important features are extracted.

[0160] Step 6:

[0161] Acoustic pattern analysis and judgment

[0162] The server inputs the feature data into the AI ​​model and analyzes whether there is an abnormality by comparing normal and abnormal acoustic patterns. The input is the feature data, and the output is the judgment result of whether there is an abnormality or a specific acoustic pattern. Specifically, the server inputs the feature data into the AI ​​model and analyzes whether the model detects an abnormal sound or a specific pattern.

[0163] Step 7:

[0164] Generation and notification of judgment results

[0165] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality", "abnormality detected", "specific abnormal sound pattern detected", etc. The generated judgment result is sent to the terminal and notified to the user. The input is the analysis result, and the output is a notification of the judgment result. Specifically, the server forms the judgment result and sends it to the terminal as a data packet.

[0166] Step 8:

[0167] Confirm the results and implement countermeasures

[0168] The user checks the judgment results through the terminal. If an abnormality is detected, an alert sound or warning icon is displayed, and the user can refer to the detailed analysis results and take appropriate measures. The input is the notification of the judgment results, and the output is the countermeasures to be implemented by the user. Specifically, the terminal displays an alert, and the user checks the detailed data before carrying out maintenance or repairs.

[0169] (Application example 1)

[0170] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0171] For autonomous vehicles, it is important to detect abnormalities in the vehicle early and respond quickly and appropriately. However, conventional methods require advanced expertise and expensive equipment to detect abnormal sounds, making it difficult for ordinary vehicle owners. Furthermore, even if abnormal sounds were detected, there was a lack of systems that could present specific countermeasures to users, resulting in many cases of delays in repairs and maintenance. For this reason, there is a need for a system that can automatically detect abnormal sounds, present countermeasures, and notify users in an easy-to-understand manner.

[0172] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0173] In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving the voice data transmitted from the terminal and performing noise reduction and feature extraction, and means for performing voice pattern analysis using a pre-trained generative AI model based on the features extracted by the server. This makes it possible to determine the presence or absence of abnormal sounds or specific patterns and generate a determination result. Furthermore, if an abnormal sound is detected based on the analysis result, it is possible to propose a detailed analysis result and countermeasures to the user. This allows the user to quickly identify vehicle abnormalities and take appropriate measures, significantly improving vehicle safety and operational efficiency.

[0174] A "high-sensitivity microphone" is a microphone that can accurately pick up very quiet sounds and subtle changes in sound.

[0175] "Terminal" refers to an electronic device that has the functionality to collect, convert, and transmit data.

[0176] "Audio data" refers to data in which sound waveforms are recorded and stored in digital format.

[0177] "Digital format" is a format in which analog signals are expressed as numbers.

[0178] "Filtering" is a process that removes unnecessary noise and interference from collected audio data and emphasizes only the necessary frequency bands.

[0179] A "server" is a computer system that receives data from other terminals or devices and provides functions such as analysis, storage, and transmission.

[0180] "Noise reduction" is a process that removes background noise and unnecessary sounds from audio data.

[0181] "Feature extraction" is the process of extracting useful features (such as frequency components and amplitude) from audio data.

[0182] A "generative AI model" is an artificial intelligence model that has been pre-trained using machine learning algorithms to perform a specific task (in this case, speech pattern analysis).

[0183] "Voice pattern analysis" is an analytical method that detects the presence of specific voice patterns (for example, abnormal sounds) based on the features of voice data.

[0184] The "determination result" is the result obtained by the analysis, and is information indicating whether or not there is an abnormal sound or a specific audio pattern.

[0185] A "cloud server" is a remote computer system that provides data and services over the Internet.

[0186] The "criteria set in advance by experts" are criteria or rules set in advance by experts for determining abnormal sounds or specific audio patterns.

[0187] To implement this invention, a system is required that collects audio data in real time, detects abnormal sounds, and notifies the user. This system is composed of a terminal, a server, and programs installed on these. Specific processes performed by the server, terminal, and user are described below.

[0188] Specific processing of the terminal

[0189] The device is equipped with a highly sensitive microphone and a mechanism for collecting voice data in real time. This voice data is captured in analog form and then converted to digital form. During this process, noise reduction and filtering are performed to suppress background noise and emphasize desired frequency bands. The pre-processed voice data is then securely and efficiently transmitted over the network to a cloud server.

[0190] Server specific processing

[0191] The server receives the voice data sent from the device and stores it in temporary storage. It checks the integrity of this data and prepares it for analysis. Next, it uses a pre-trained generative AI model to extract features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for analysis. These features are then input into the AI ​​model, which compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. Based on the results, it determines whether there is an abnormality or the existence of a specific pattern, and generates a judgment result. If an abnormal sound is detected, it also generates detailed analysis results and information to suggest countermeasures to the user.

[0192] User specific processing

[0193] The user checks the results of the assessment through their device. If an abnormality is detected, an alert sound or a warning icon will be displayed to notify the user. The user can then review the detailed analysis results and take appropriate action according to the presented countermeasures. For example, if an abnormal sound is detected in the vehicle's engine, it may be recommended to replace the bearings or conduct further inspections.

[0194] Specific examples

[0195] For example, monitoring the engine noise of a self-driving vehicle might proceed as follows:

[0196] 1. A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time.

[0197] 2. Collected audio data is converted into digital form and then denoised and filtered.

[0198] 3. The preprocessed audio data is sent to the cloud server.

[0199] 4. The server receives the audio data, extracts features using the generative AI model, and analyzes whether there are any abnormal sounds.

[0200] 5. If an abnormal sound is detected, a judgment result such as "Abnormal bearing noise detected" and specific countermeasures are generated and notified to the terminal.

[0201] 6. The user should check the notified judgment result and take appropriate measures such as replacing the bearing.

[0202] Prompt Sentence Examples

[0203] "input:

[0204] Audio data: [0.1, 0.2, -0.1, ...]

[0205] output:

[0206] Verification result: Abnormal noise detected. Details: Abnormal engine bearing noise detected. Confidence level: 95%

[0207] In this way, an anomaly detection system using acoustic data will significantly improve vehicle safety and operational efficiency.

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] The device uses a highly sensitive microphone to collect audio data in real time. The input is ambient sounds, such as the sound of a vehicle engine, which are captured through the microphone. These sounds are collected in analog format, and the output is analog audio data.

[0211] Step 2:

[0212] The terminal converts the collected analog voice data into a digital format and performs noise reduction and filtering. Analog voice data is used as input. Noise reduction removes unwanted background noise, and filtering emphasizes the desired frequency band. The output is pre-processed digital voice data.

[0213] Step 3:

[0214] The device sends preprocessed digital audio data to the cloud server. The input is the source data, which is encrypted and compressed before being sent over the network. The output is the audio data sent to the cloud server.

[0215] Step 4:

[0216] The server receives the voice data sent from the device and stores it in temporary storage. The received data is used as input. The server checks the integrity of the data and performs the storage process. The output is the temporarily stored digital voice data.

[0217] Step 5:

[0218] The server extracts features from the stored voice data. In this process, features such as frequency spectrum and amplitude required for the generative AI model are extracted. Temporarily stored digital voice data is used as input. The voice data is analyzed and features are extracted, resulting in feature data as output.

[0219] Step 6:

[0220] The server inputs the extracted features into the generative AI model and performs voice pattern analysis. The extracted feature data is used as input. The generative AI model has learned normal and abnormal voice patterns and performs analysis by comparing them with the feature data. The output is the analysis results.

[0221] Step 7:

[0222] Based on the analysis results, the server determines whether there are any abnormal sounds or specific patterns and generates a judgment result. The results of the voice pattern analysis are used as input. The judgment results include "no abnormality," "abnormality present," "specific abnormal sound pattern present," etc. The output is the judgment result data.

[0223] Step 8:

[0224] The server sends the generated judgment result to the terminal and notifies the user. The judgment result data is used as input. The judgment result is compressed and encrypted and sent to the terminal. The output is the judgment result received by the user through the terminal.

[0225] Step 9:

[0226] The user checks the judgment results through a terminal. The judgment results displayed on the terminal are used as input. If an abnormality is detected, the user is notified of the detailed analysis results and countermeasures, along with an alert sound and a warning icon. The output is the confirmed judgment results and proposed countermeasures.

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[0229] Specific processing of the terminal

[0230] Audio data collection

[0231] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[0232] Digital Conversion and Filtering

[0233] The collected audio data is converted from analog to digital, then filtered to reduce background noise and emphasize the desired frequency band.

[0234] Sending data

[0235] The pre-processed audio data is packetized and sent to the server using a secure protocol, where data is compressed and encrypted to ensure secure communication.

[0236] Server specific processing

[0237] Data Receipt and Storage

[0238] The server receives the voice data sent from the device and stores it in temporary storage. It checks for errors while verifying the integrity of the data.

[0239] Feature extraction and analysis preparation

[0240] The server loads the trained AI model and extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[0241] Voice pattern analysis and judgment

[0242] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns, which determines whether an abnormality exists or the existence of a specific pattern.

[0243] Generating judgment results

[0244] The server generates a judgment result based on the analysis results, which is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc.

[0245] Sending the judgment result

[0246] The judgment result is sent to the terminal, and the data is compressed and encrypted as necessary.

[0247] Specific processing of the emotion engine

[0248] emotion recognition

[0249] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. The emotion engine detects multiple emotional states, such as the user's stress level, excitement, or calmness.

[0250] Emotion-based notification adjustment

[0251] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, a notification that emphasizes a more serious warning will be sent.

[0252] User specific processing

[0253] Check the results

[0254] The user checks the results on their device, and notifications are dynamically adjusted based on the emotion engine's results, using audio and visual alerts.

[0255] View detailed data and take action

[0256] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the user's stress level is high, the user will be provided with support information to take prompt and appropriate action.

[0257] Examples: Vehicle engine sound inspection and user emotion recognition

[0258] Device behavior

[0259] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[0260] Server Operation

[0261] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[0262] User Actions

[0263] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[0264] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[0265] The processing flow will be explained below.

[0266] Step 1:

[0267] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[0268] Step 2:

[0269] The device converts the collected voice data from analog to digital signals, putting the voice data into an analyzable format.

[0270] Step 3:

[0271] The device then performs a filtering process on the digital audio signal, reducing background noise and emphasizing important frequency bands.

[0272] Step 4:

[0273] The terminal packetizes the pre-processed voice data, compresses and encrypts the data, and then transmits it to the server.

[0274] Step 5:

[0275] The server receives the voice data sent from the device and temporarily stores it in storage. It checks the data for consistency and performs error checks.

[0276] Step 6:

[0277] The server loads the trained AI model and prepares it for analysis.

[0278] Step 7:

[0279] The server extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[0280] Step 8:

[0281] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[0282] Step 9:

[0283] Based on the analysis results, the server generates a judgment result such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[0284] Step 10:

[0285] The server sends the result of the judgment to the terminal and compresses and encrypts the data as necessary.

[0286] Step 11:

[0287] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, it evaluates their stress level, excitement, calmness, etc. based on their tone of voice and facial expressions.

[0288] Step 12:

[0289] The server combines the output information of the emotion engine with the judgment results and selects the most appropriate notification method for the user. For example, if the user is in a high-stress state, a high-level warning will be issued.

[0290] Step 13:

[0291] The device displays the judgment results and the emotion engine's evaluation results on the user interface, and the user is notified using audio and visual alerts.

[0292] Step 14:

[0293] The user can check the judgment results and the emotion engine's evaluation results displayed on the device. They can also refer to the detailed voice data analysis results.

[0294] Step 15:

[0295] The user can take appropriate measures based on the judgment results and the emotion engine's evaluation results. For example, if an abnormal sound is detected and the user's stress level is high, machine maintenance or part replacement will be carried out promptly.

[0296] Example 2

[0297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] Conventional sound detection support systems did not take into account the user's emotional state and fixed the way and timing of notifications were received, which increased stress for the user. Furthermore, the security of audio data was not adequately ensured, which posed a risk of important data being leaked. Furthermore, because the criteria for determining abnormal sounds were fixed, flexible responses were difficult and the system relied on dedicated experts.

[0299] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for performing voice pattern analysis using a machine learning model, means for generating a determination result, means for recognizing the emotional state of the user by analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the emotion recognition result. This allows the user to receive notifications at an appropriate time and in an appropriate manner without feeling stressed. In addition, data confidentiality is maintained by compressing and encrypting the data using a secure protocol before transmission. Furthermore, flexible and highly accurate voice analysis is possible by comparing the criteria for determining abnormal sounds with criteria set in advance by experts.

[0300] A "high-sensitivity microphone" is a sound collection device that has the ability to pick up even the faintest sounds.

[0301] A "terminal" is an electronic device that collects and digitally processes voice data.

[0302] "Digital format" refers to a format in which an analog signal is converted into digital code.

[0303] "Filtering" is a process for removing unnecessary noise from the original audio data.

[0304] "Noise reduction" is a technique for reducing or removing background noise from collected audio data.

[0305] "Feature extraction" is the process of extracting useful information or patterns from speech data.

[0306] A "machine learning model" is an algorithm that learns from large amounts of data and enables pattern recognition and prediction.

[0307] "Voice pattern analysis" is a technology that analyzes voice data to detect specific voice patterns and anomalies.

[0308] The "determination result" is information that indicates the presence or absence of an abnormality or a specific state generated as a result of the analysis.

[0309] "User" means an individual or organization that uses the system.

[0310] "Emotional state" is a state that indicates a user's stress level and emotional response.

[0311] The "notification method" is the method for notifying the user of the judgment results and other information.

[0312] A "secure protocol" is a communication protocol that ensures confidentiality and integrity when sending and receiving data.

[0313] "Data compression" is a technique for reducing the volume of data.

[0314] "Encryption" is a technology that uses a specific algorithm to convert data into a format that cannot be deciphered by third parties.

[0315] The "criteria for determining abnormal sounds" are criteria for detecting abnormalities used in the sound analysis.

[0316] An "expert" is someone who has advanced knowledge and skills in a particular field.

[0317] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[0318] Specific processing of the terminal

[0319] Audio data collection

[0320] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line. The collected audio data is temporarily stored in the device as an analog signal.

[0321] Digital Conversion and Filtering

[0322] The terminal converts the collected analog voice data into a digital signal using an Analog-to-Digital Converter (ADC). The signal is then filtered using an FIR filter to remove high-frequency noise and emphasize key frequency bands.

[0323] Sending data

[0324] The pre-processed audio data is packetized and sent to the server using a secure protocol (e.g., TLS or SSL), with algorithms such as GZIP used to compress the data and AES encryption used for encryption.

[0325] Server specific processing

[0326] Data Receipt and Storage

[0327] The server receives the voice data sent from the device and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks.

[0328] Feature extraction and analysis preparation

[0329] The server then loads a pre-trained machine learning model to extract features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data, often using MFCC (Mel-Frequency Cepstral Coefficients) and STFT (Short-Time Fourier Transform).

[0330] Voice pattern analysis and judgment

[0331] Based on the extracted features, the server inputs them into a machine learning model and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. This method uses a convolutional neural network (CNN), which is good at analyzing time series data and spatial data of voice.

[0332] Generating judgment results

[0333] Based on the analysis results, the server generates a judgment result, which can be expressed as "no abnormality," "abnormality," "specific abnormal sound pattern," etc. For example, it can identify a specific abnormal sound, such as "abnormal sound in the engine bearing."

[0334] Sending the judgment result

[0335] The server sends the result to the device, compressing and encrypting the data as needed, again using TLS / SSL and AES encryption in the process.

[0336] Specific processing of the emotion engine

[0337] emotion recognition

[0338] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. For example, it uses the device's camera and microphone to analyze the user's facial expressions and tone of voice. For analysis, OpenCV is used to capture the user's facial expressions in real time and input them into an emotion analysis model.

[0339] Emotion-based notification adjustment

[0340] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, the server adjusts the notification method to be more specific and urgent. In addition to voice notifications, it also uses text notifications and graphical alerts.

[0341] User specific processing

[0342] Check the results

[0343] The user checks the results on their device. Notifications are dynamically adjusted based on the emotion engine's results, providing appropriate audible and visual alerts. For example, the device might display a message saying, "There is an abnormal noise in the engine bearings."

[0344] View detailed data and take action

[0345] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the stress level is high, specific instructions for considering replacing bearings will be displayed. This allows for quick and appropriate action.

[0346] Examples: Vehicle engine sound inspection and user emotion recognition

[0347] Device behavior

[0348] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[0349] Server Operation

[0350] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[0351] User Actions

[0352] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[0353] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[0354] Example prompts for generative AI models

[0355] If you need an explanation of specific system operations or each piece of hardware and software, enter prompts like the following into the generative AI model:

[0356] Example prompt sentence:

[0357] "Please explain in detail the benefits of combining the Acoustic Judgment Support System (ADSS) with an emotion engine and the specific processing steps. Please also mention the specific processing performed by the terminal, server, and user, as well as the hardware and software used. Also, as a concrete example, please explain how it works in the case of a vehicle engine sound inspection."

[0358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0359] Step 1: Collecting audio data

[0360] The device uses a high-sensitivity microphone to collect audio data in real time from a specific sound source (e.g., a vehicle engine). The input is an analog audio signal from the high-sensitivity microphone attached to the device. This analog audio data is temporarily stored within the device. The output is analog audio data.

[0361] Step 2: Digital Conversion and Filtering

[0362] The terminal converts the collected analog voice data (input) into a digital signal. This conversion is performed using an Analog-to-Digital Converter (ADC). The digital voice data is then filtered to reduce background noise and emphasize key frequency bands. Specifically, an FIR filter is used. The filtered digital voice data is output.

[0363] Step 3: Sending data

[0364] The device packets preprocessed digital audio data (input) and sends it to the server using a secure protocol (e.g., TLS or SSL). The data is compressed using an algorithm such as GZIP and encrypted using the AES cipher. The encrypted compressed data is sent to the server as output.

[0365] Step 4: Receiving and storing data

[0366] The server receives the encrypted compressed data (input) sent from the terminal and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks. The output is the original stored digital audio data.

[0367] Step 5: Feature extraction and analysis preparation

[0368] The server loads a pre-trained machine learning model based on the stored digital audio data (input) and extracts features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data. This process uses MFCC and STFT. The output is the extracted audio data features.

[0369] Step 6: Voice pattern analysis and decision

[0370] The server inputs the feature-extracted voice data (input) into a machine learning model and compares it with normal voice patterns and abnormal sound patterns to measure the degree of match. Specifically, the analysis is performed using a convolutional neural network (CNN). The output is the analysis result (e.g., "normal," "abnormal," or "specific abnormal sound pattern present").

[0371] Step 7: Generate a decision result

[0372] The server generates a judgment result based on the voice pattern analysis result (input). The judgment result is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The judgment result to be notified to the user is generated as output.

[0373] Step 8: Send the result

[0374] The server sends the generated judgment result (input) to the terminal, compressing and encrypting the data if necessary, again using a secure protocol (e.g., TLS or SSL) and AES encryption. The encrypted judgment result data is sent to the terminal as output.

[0375] Step 9: Emotion Recognition

[0376] The device or server analyzes the user's voice and facial expressions (input) to recognize their emotional state. Specifically, it uses the device's camera and microphone to capture facial expressions in real time using OpenCV and an emotion analysis model based on voice analysis. The output is the user's emotional state (e.g., stress level, excitement, calm).

[0377] Step 10: Adjust notifications based on emotion

[0378] The server combines the judgment results with the emotion recognition results (input) to select the optimal notification method for the user. Specifically, if the stress level is high, a notification method that emphasizes urgency and specificity is selected. The output is the adjusted notification content and format.

[0379] Step 11: Check the results

[0380] The user confirms the adjusted notification content (input) through the terminal. Notifications are provided through voice, text, graphical alerts, etc. The output allows the user to acknowledge and respond to the judgment results.

[0381] Step 12: View detailed data and take action

[0382] The user can view the detailed voice data analysis results and emotion recognition results (input) and take appropriate countermeasures. For example, if an abnormal sound is detected and the stress level is high, the user can quickly consider replacing the bearings. As an output, appropriate countermeasures are taken and the equipment continues to operate normally.

[0383] (Application example 2)

[0384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] In modern factories, it is extremely important to quickly detect abnormal machine sounds and perform maintenance at the appropriate time. However, conventional acoustic detection systems do not take the user's emotional state into consideration when notifying them of the detection results, which can lead to problems such as increased stress. In addition, there is a need for a comprehensive system that not only detects abnormal machine sounds but also efficiently monitors the entire acoustic environment.

[0386] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving voice data transmitted from the terminal and performing noise reduction and feature extraction, means for performing voice pattern analysis using a pre-trained AI model based on the features extracted by the server, means for determining the presence or absence of abnormal sounds or specific patterns based on the analysis results and generating a determination result, means for transmitting the determination result to the terminal and notifying the user, means for recognizing the user's emotional state using a camera and image recognition, and means for dynamically adjusting the notification method to the user depending on the emotional state. This makes it possible not only to accurately detect abnormal sounds from machines but also to provide appropriate notifications taking the user's emotional state into consideration.

[0387] A "terminal" is a device equipped with a highly sensitive microphone that collects audio data in real time.

[0388] The "server" is a device that receives voice data sent from the terminal, removes noise and extracts features, and analyzes the data to determine whether or not there are any abnormal sounds or specific patterns.

[0389] A "high-sensitivity microphone" is a microphone that can collect sound with high accuracy.

[0390] "Noise reduction" is the process of removing unnecessary background noise from collected audio data.

[0391] "Feature extraction" is the process of extracting important patterns and features from collected voice data.

[0392] An "AI model" is a model that uses artificial intelligence to analyze data and is trained to identify specific patterns.

[0393] "Voice pattern analysis" is a process that uses an AI model to analyze patterns in voice data and determine whether there are any abnormal sounds or specific patterns.

[0394] The "determination result" is a result regarding the presence of an abnormal sound or a specific pattern, which is generated based on the sound pattern analysis.

[0395] A "camera" is a device for taking still images and videos.

[0396] "Image recognition" is a technology that analyzes images taken with a camera and recognizes specific objects or situations.

[0397] "Emotional state" refers to the user's emotional state, such as stress level, excitement, or calmness.

[0398] "Notification method" refers to the means or format for conveying information or warnings to the user.

[0399] A specific system for implementing the present invention will now be described.

[0400] System Overview

[0401] This system is configured so that a device equipped with a highly sensitive microphone collects voice data in real time and sends it to a server. The server then removes noise from the voice data, extracts features, and analyzes the voice patterns using an AI model. Based on the analysis results, it determines whether there are any abnormal sounds or specific patterns, and dynamically adjusts the notification method taking into account the results of the analysis and the user's emotional state.

[0402] Specific processing method for the terminal

[0403] The device is equipped with a highly sensitive microphone that collects audio data in real time from specific sound sources, such as machinery sounds in a factory. The collected audio data is converted from analog to digital format and then filtered to reduce background noise and emphasize frequency bands that are useful for detecting abnormal sounds. After preprocessing, the audio data is packetized and sent to a server using a secure protocol.

[0404] Specific server processing method

[0405] The server receives the voice data sent from the device and temporarily stores it in storage. Next, it checks for errors while verifying the data's consistency. After that, it loads the trained AI model and extracts features (frequency spectrum, amplitude, phase, etc.) from the voice data. It performs voice pattern analysis based on the features and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match, thereby determining whether there is an abnormality or the presence of a specific pattern. The judgment result is generated as "no abnormality," "abnormality present," or "specific abnormal sound pattern present." The server then uses a camera and image recognition technology to recognize the user's emotional state from facial expressions, etc. The recognized emotional state is combined with the judgment result to select the most appropriate notification method and send it to the device.

[0406] Specific user operations

[0407] The user can check the results of the assessment through their device. Notifications are dynamically adjusted according to the user's emotional state; for example, if the user is in a high stress state, a more pronounced warning sound or visual alert is displayed. Based on the analysis results and the evaluation of the emotional state, support information is provided to take prompt and appropriate action if the user is in a high stress state. The user can then perform maintenance or repairs on the machine as needed.

[0408] Hardware and software used

[0409] Hardware used:

[0410] High-sensitivity microphone

[0411] Terminal (for collecting voice data)

[0412] Server (for data analysis and judgment)

[0413] Camera (for emotion recognition)

[0414] Software used:

[0415] pyaudio (audio data collection)

[0416] socket (data transmission)

[0417] Keras (AI model for analyzing voice patterns)

[0418] dlib (image recognition and emotional state recognition)

[0419] Specific examples

[0420] For example, if this system is used to detect abnormal sounds from machinery operating in a factory, a highly sensitive microphone collects the sounds and sends the data to a server. The server analyzes the data, and if an abnormal sound is detected, the results are sent to the terminal. At the same time, the camera analyzes the operator's facial expression, and if the user is in a state of high stress, an exaggerated warning sound or visual alert is displayed. This notification method takes the user's emotional state into account, enabling a quick and appropriate response.

[0421] Prompt Sentence Examples

[0422] "Write a Python program to help a factory robot collect mechanical sounds and detect abnormal sounds. Use a highly sensitive microphone to collect the audio in real time, and then convert and filter the data digitally before sending it to a server."

[0423] "Write a Python program that uses an AI model to analyze voice data to detect anomalous patterns, and then notifies the user based on the results. The notification content should dynamically adjust based on the user's emotional state (stress level, calmness)."

[0424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0425] Step 1:

[0426] The terminal collects machine sounds within the factory in real time using a highly sensitive microphone.

[0427] Specifically, a high-sensitivity microphone installed in the terminal collects analog audio data from specific sound sources within the factory, and this collected audio data becomes the input for the next processing step.

[0428] Step 2:

[0429] The analog voice data collected by the terminal is converted into digital format and filtered.

[0430] The input is analog audio data, which is converted to a digital signal, then filtered to reduce background noise and highlight abnormal sounds, and this digital audio data is the input for the next processing step.

[0431] Step 3:

[0432] The terminal converts the preprocessed voice data into packets and transmits them to the server.

[0433] The input is pre-processed digital audio data, which is then packetized and sent to the server using a secure protocol (e.g., TLS), where the packetized data is the input for processing.

[0434] Step 4:

[0435] The server receives the audio data, temporarily stores it in storage, and performs error checking.

[0436] The input is packetized voice data sent from the terminal, which the server receives and stores, and performs error checks to ensure data integrity. This temporarily stored data becomes the input for the next analysis step.

[0437] Step 5:

[0438] The server extracts features (frequency spectrum, amplitude, phase, etc.) from the stored audio data.

[0439] The input is stored digital audio data, and an AI model is used to extract features from the audio data, which then become the input for the next step of audio pattern analysis.

[0440] Step 6:

[0441] The server analyzes the audio pattern using an AI model based on the features and determines whether there are any abnormal sounds or specific patterns.

[0442] The input is the extracted features, which are then input into the AI ​​model for analysis. The analysis results determine whether there are any abnormal sounds or specific patterns, and a judgment result is generated. This judgment result becomes the input for the next notification adjustment step.

[0443] Step 7:

[0444] The server uses a camera to recognize the user's emotional state.

[0445] The input is image data of the user collected by a camera, and image recognition technology is used to analyze the user's emotional state (stress level, calmness, etc.). This analyzed emotional state becomes additional input for the next notification adjustment step.

[0446] Step 8:

[0447] The server dynamically adjusts the notification method based on the judgment result and the user's emotional state.

[0448] The inputs are the analyzed emotional state and the judgment result, and these are combined to select the optimal notification method. For example, if the user is in a high stress state, an enhanced warning sound or visual alert is selected. This adjusted notification method becomes the input for the next notification sending step.

[0449] Step 9:

[0450] The server transmits the adjusted notification method and the determination result to the terminal.

[0451] The input is the adjusted notification method and the judgment result, which is sent to the terminal. The sent notification becomes the input for the next user confirmation step.

[0452] Step 10:

[0453] The terminal notifies the user, and the user confirms the judgment result.

[0454] The input is the notification method and judgment result sent from the server, which the device notifies to the user. The user can take appropriate measures or actions based on the displayed judgment result and notification.

[0455] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0456] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0457] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0458] [Second embodiment]

[0459] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0460] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0461] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0462] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0463] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0464] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0465] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0466] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0467] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0468] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0469] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0470] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0471] To implement this invention, the terminals, servers, and programs installed on them that constitute the Acoustic Discrimination Support System (ADSS) are required. Specific processes performed by the server, terminals, and users are described below.

[0472] Specific processing of the terminal

[0473] Audio data collection

[0474] The device uses a highly sensitive microphone to collect audio data in real time, allowing it to accurately capture data from specific sound sources, such as a vehicle engine or machinery on a production line.

[0475] Digital Conversion and Filtering

[0476] The collected audio data is converted from analog to digital format and then denoised and filtered, which suppresses background noise and emphasises the desired frequency bands.

[0477] Sending data

[0478] The pre-processed audio data is sent over the network to a server, where it is compressed and encrypted for secure and efficient transmission.

[0479] Server specific processing

[0480] Data Receipt and Storage

[0481] The server receives the voice data sent from the device and stores it in temporary storage, verifies the integrity of the data, and prepares it for analysis.

[0482] Feature extraction and analysis preparation

[0483] The server loads the trained AI model and extracts features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for voice pattern analysis.

[0484] Voice pattern analysis and judgment

[0485] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns. Based on the results, it is determined whether an abnormality exists or whether a specific pattern exists.

[0486] Generation and notification of judgment results

[0487] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The generated judgment result is sent to the terminal and notified to the user.

[0488] User specific processing

[0489] Check the results

[0490] The user can check the results of the assessment through their device. For example, if an abnormality is detected, the user is notified by an alert sound or a warning icon.

[0491] View detailed data and take action

[0492] If necessary, the user can check the detailed audio data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, machine maintenance, part replacement, or further inspection can be carried out.

[0493] Example: Vehicle engine sound inspection

[0494] Device behavior

[0495] A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time. The collected audio data is converted into a digital format, and then noise reduction and filtering are performed. The pre-processed data is then sent to a server.

[0496] Server Operation

[0497] The server temporarily stores the received voice data and extracts features from the voice data. The extracted features are input into an AI model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the device.

[0498] User Actions

[0499] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, the user takes measures such as replacing the bearings.

[0500] In this way, acoustic inspection work can be carried out quickly and accurately. This system significantly improves the efficiency and reliability of acoustic inspections, and also solves the problems of labor shortages and training costs.

[0501] The processing flow will be explained below.

[0502] Step 1:

[0503] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or factory machinery.

[0504] Step 2:

[0505] The device converts the collected voice data from analog to digital signals, making it in a format that can be analyzed.

[0506] Step 3:

[0507] The device then filters the digital audio data, reducing background noise and emphasizing important frequency bands.

[0508] Step 4:

[0509] The terminal then packetizes the pre-processed voice data, encrypts it, and sends it to the server, ensuring secure data transmission.

[0510] Step 5:

[0511] The server receives the voice data sent from the device and temporarily stores it in storage. It also performs error checking to ensure data integrity.

[0512] Step 6:

[0513] The server loads the AI ​​model and prepares it for analysis, using a pre-trained model.

[0514] Step 7:

[0515] The server extracts features from the audio data, specifically, frequency spectrum, amplitude, phase, etc.

[0516] Step 8:

[0517] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[0518] Step 9:

[0519] The server generates a judgment result based on the analysis results, such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[0520] Step 10:

[0521] The server generates a judgment result and sends it to the terminal, compressing and encrypting the data as necessary.

[0522] Step 11:

[0523] The terminal displays the result of the judgment received on the user interface, for example, by notifying the user using an alert sound or a warning icon.

[0524] Step 12:

[0525] The user checks the judgment results displayed on the device and, if necessary, can also view detailed voice data analysis results.

[0526] Step 13:

[0527] The user takes appropriate measures based on the judgment results. For example, if an abnormal sound is detected, the user performs machine maintenance or replaces parts.

[0528] Example 1

[0529] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] Conventional acoustic detection systems lack the ability to collect acoustic data in real time and analyze it quickly and accurately, making it difficult to distinguish between normal and abnormal movement patterns. They also lacked the functionality to efficiently determine the presence or absence of abnormal sounds or specific patterns, resulting in delays in providing appropriate notifications and feedback to users. Furthermore, the entire process, from collecting acoustic data to analyzing it and notifying users, was not integrated, resulting in low efficiency for the entire system.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0532] In this invention, the server includes a terminal equipped with a highly sensitive sensor that collects acoustic data in real time, a processing device that receives the acoustic data transmitted from the terminal and performs noise removal and feature extraction, means for performing acoustic pattern analysis using a pre-trained machine learning model based on the features extracted by the processing device, means for determining the presence or absence of abnormal acoustics or specific patterns based on the analysis results and generating a determination result, and means for transmitting the determination result to the terminal and notifying a user. This makes it possible to collect acoustic data in real time, analyze it quickly and accurately, efficiently determine the presence or absence of abnormal acoustics or specific patterns, and provide appropriate notification and feedback to the user.

[0533] "Acoustic data" is information that is generated by converting sound into electrical signals and storing or processing them in digital form.

[0534] A "high-sensitivity sensor" is an acoustic detection device capable of detecting minute acoustic changes with high precision.

[0535] A "terminal" is a device that collects acoustic data and sends it to a server for processing.

[0536] A "processor" is a computer system that has the computational power to analyze acoustic data.

[0537] A "machine learning model" is an algorithm that is trained on large amounts of data to perform pattern recognition and prediction.

[0538] "Acoustic pattern analysis" is a process that identifies abnormalities and specific patterns based on the features of acoustic data.

[0539] "Abnormal sound" is a sound that is different from normal operating sounds and indicates an abnormality.

[0540] The "determination result" is diagnostic information generated based on the analysis of the acoustic data.

[0541] A "user" is a person who operates the system and checks the judgment results.

[0542] "Noise reduction" is a process for removing unnecessary background sounds from audio data.

[0543] "Feature extraction" is the process of extracting important patterns and properties from acoustic data.

[0544] "Real-time" refers to the simultaneous collection and processing of acoustic data.

[0545] MODE FOR CARRYING OUT THE INVENTION

[0546] To implement this invention, an Acoustic Decision Support System (ADSS) is required, which includes the following components: ADSS is realized by the roles of terminal, server, and user.

[0547] Terminal configuration and operation

[0548] Acoustic data collection

[0549] The device is equipped with highly sensitive sensors that collect acoustic data in real time from specific sound sources, such as a vehicle engine or a machine on a production line, with high accuracy. The device also includes location and time information to provide context for the collected acoustic data.

[0550] Digital Conversion and Filtering

[0551] The collected acoustic data is in analog format and is converted to digital format using an ADC (analog-to-digital converter) within the device. It is then denoised and filtered using audio processing software such as Audacity, which removes unwanted background noise and emphasises the desired frequency bands.

[0552] Sending data

[0553] The pre-processed digital audio data is then sent to the server via the network. During data transmission, the data is compressed and encrypted using gzip, TLS / SSL, etc., for safe and efficient transmission.

[0554] Server configuration and operation

[0555] Data Receipt and Storage

[0556] The server receives the acoustic data sent from the device and stores it in temporary storage (e.g., Amazon S3). It checks the integrity of the data and verifies that it is not corrupted. This temporary storage makes the data easily available for subsequent processing.

[0557] Feature extraction and analysis preparation

[0558] The server loads a trained machine learning model (e.g., TensorFlow or PyTorch) and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data, which is then ready for analysis.

[0559] Acoustic pattern analysis and judgment

[0560] The server uses the extracted features to perform acoustic pattern analysis. The features are input into a machine learning model, and the degree of match is calculated by comparing normal and abnormal sound patterns. Based on the results of this analysis, it determines whether an abnormality exists or whether a specific pattern exists.

[0561] Generation and notification of judgment results

[0562] The server generates a judgment result based on the analysis results. The result is displayed in the form of "No abnormality," "Abnormality detected," "Specific abnormal sound pattern detected," etc. These judgment results are sent to the terminal and notified to the user.

[0563] User Configuration and Behavior

[0564] Check the results

[0565] The user can check the results of the assessment through their device. For example, if an abnormality is detected, an alert sound or a warning icon will be displayed to alert the user.

[0566] View detailed data and take action

[0567] If necessary, the user can check the detailed acoustic data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, the user can perform machine maintenance, replace parts, or perform further inspections.

[0568] Example: Inspecting vehicle engine noise

[0569] Device behavior

[0570] Highly sensitive sensors are installed in the vehicle's engine to collect engine sounds in real time. The collected acoustic data is then digitally converted and filtered before being sent to a server via a network.

[0571] Server Operation

[0572] The server temporarily stores the received acoustic data and extracts features from the acoustic data. The extracted features are input into a machine learning model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the terminal.

[0573] User Actions

[0574] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, countermeasures such as replacing the bearings are implemented.

[0575] Examples of prompt statements

[0576] "Generate Python code to collect vehicle engine sounds in real time and detect abnormal sounds."

[0577] In this way, the Acoustic Decision Support System (ADSS) can quickly and efficiently carry out the entire process from collection to analysis and notification, improving the efficiency and reliability of acoustic inspections and resolving the problems of labor shortages and training costs.

[0578] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0579] Step 1:

[0580] Acoustic data collection

[0581] The device uses a highly sensitive sensor to collect acoustic data in real time from specific sound sources. For example, it can capture the sound of a vehicle engine or the machine noise of a production line. The input is an analog acoustic signal captured by the acoustic sensor, and the output is analog acoustic data. Specifically, the device's highly sensitive sensor is placed near the target sound source and continuously collects acoustic data.

[0582] Step 2:

[0583] Digital Conversion and Filtering

[0584] The device converts the collected analog acoustic data into digital format using a built-in analog-to-digital converter (ADC). Next, the converted digital acoustic data is processed using audio processing software such as Audacity to remove noise and perform filtering. The input is analog acoustic data, and the output is filtered digital acoustic data. Specifically, the audio data is converted into digital format using the built-in ADC, and then white noise and unwanted frequencies are removed using Audacity.

[0585] Step 3:

[0586] Sending data

[0587] The device sends the preprocessed digital audio data to a server over the Internet or a local network. The data is sent in a compressed format such as gzip and encrypted format such as TLS / SSL. The input is the filtered digital audio data, and the output is compressed and encrypted audio data packets. Specifically, the digital audio data is compressed and securely sent to the server using an encryption protocol.

[0588] Step 4:

[0589] Data reception and storage

[0590] The server receives the digital audio data packets sent from the device, decompresses and decrypts them, and stores the received data in temporary storage (e.g., cloud storage). The input is the compressed and encrypted audio data packets, and the output is the digital audio data stored in the storage. Specifically, the server decompresses and decrypts the received data packets before storing them in cloud storage.

[0591] Step 5:

[0592] Feature extraction and analysis preparation

[0593] The server loads a pre-trained machine learning model (e.g., TensorFlow or PyTorch) based on the stored acoustic data, and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data. The input is the stored digital acoustic data, and the output is the extracted feature data. Specifically, the acoustic data is converted into a frequency spectrum, and amplitude and other important features are extracted.

[0594] Step 6:

[0595] Acoustic pattern analysis and judgment

[0596] The server inputs the feature data into the AI ​​model and analyzes whether there is an abnormality by comparing normal and abnormal acoustic patterns. The input is the feature data, and the output is the judgment result of whether there is an abnormality or a specific acoustic pattern. Specifically, the server inputs the feature data into the AI ​​model and analyzes whether the model detects an abnormal sound or a specific pattern.

[0597] Step 7:

[0598] Generation and notification of judgment results

[0599] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality", "abnormality detected", "specific abnormal sound pattern detected", etc. The generated judgment result is sent to the terminal and notified to the user. The input is the analysis result, and the output is a notification of the judgment result. Specifically, the server forms the judgment result and sends it to the terminal as a data packet.

[0600] Step 8:

[0601] Confirm the results and implement countermeasures

[0602] The user checks the judgment results through the terminal. If an abnormality is detected, an alert sound or warning icon is displayed, and the user can refer to the detailed analysis results and take appropriate measures. The input is the notification of the judgment results, and the output is the countermeasures to be implemented by the user. Specifically, the terminal displays an alert, and the user checks the detailed data before carrying out maintenance or repairs.

[0603] (Application example 1)

[0604] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] For autonomous vehicles, it is important to detect abnormalities in the vehicle early and respond quickly and appropriately. However, conventional methods require advanced expertise and expensive equipment to detect abnormal sounds, making it difficult for ordinary vehicle owners. Furthermore, even if abnormal sounds were detected, there was a lack of systems that could present specific countermeasures to users, resulting in many cases of delays in repairs and maintenance. For this reason, there is a need for a system that can automatically detect abnormal sounds, present countermeasures, and notify users in an easy-to-understand manner.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0607] In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving the voice data transmitted from the terminal and performing noise reduction and feature extraction, and means for performing voice pattern analysis using a pre-trained generative AI model based on the features extracted by the server. This makes it possible to determine the presence or absence of abnormal sounds or specific patterns and generate a determination result. Furthermore, if an abnormal sound is detected based on the analysis result, it is possible to propose a detailed analysis result and countermeasures to the user. This allows the user to quickly identify vehicle abnormalities and take appropriate measures, significantly improving vehicle safety and operational efficiency.

[0608] A "high-sensitivity microphone" is a microphone that can accurately pick up very quiet sounds and subtle changes in sound.

[0609] "Terminal" refers to an electronic device that has the functionality to collect, convert, and transmit data.

[0610] "Audio data" refers to data in which sound waveforms are recorded and stored in digital format.

[0611] "Digital format" is a format in which analog signals are expressed as numbers.

[0612] "Filtering" is a process that removes unnecessary noise and interference from collected audio data and emphasizes only the necessary frequency bands.

[0613] A "server" is a computer system that receives data from other terminals or devices and provides functions such as analysis, storage, and transmission.

[0614] "Noise reduction" is a process that removes background noise and unnecessary sounds from audio data.

[0615] "Feature extraction" is the process of extracting useful features (such as frequency components and amplitude) from audio data.

[0616] A "generative AI model" is an artificial intelligence model that has been pre-trained using machine learning algorithms to perform a specific task (in this case, speech pattern analysis).

[0617] "Voice pattern analysis" is an analytical method that detects the presence of specific voice patterns (for example, abnormal sounds) based on the features of voice data.

[0618] The "determination result" is the result obtained by the analysis, and is information indicating whether or not there is an abnormal sound or a specific audio pattern.

[0619] A "cloud server" is a remote computer system that provides data and services over the Internet.

[0620] The "criteria set in advance by experts" are criteria or rules set in advance by experts for determining abnormal sounds or specific audio patterns.

[0621] To implement this invention, a system is required that collects audio data in real time, detects abnormal sounds, and notifies the user. This system is composed of a terminal, a server, and programs installed on these. Specific processes performed by the server, terminal, and user are described below.

[0622] Specific processing of the terminal

[0623] The device is equipped with a highly sensitive microphone and a mechanism for collecting voice data in real time. This voice data is captured in analog form and then converted to digital form. During this process, noise reduction and filtering are performed to suppress background noise and emphasize desired frequency bands. The pre-processed voice data is then securely and efficiently transmitted over the network to a cloud server.

[0624] Server specific processing

[0625] The server receives the voice data sent from the device and stores it in temporary storage. It checks the integrity of this data and prepares it for analysis. Next, it uses a pre-trained generative AI model to extract features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for analysis. These features are then input into the AI ​​model, which compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. Based on the results, it determines whether there is an abnormality or the existence of a specific pattern, and generates a judgment result. If an abnormal sound is detected, it also generates detailed analysis results and information to suggest countermeasures to the user.

[0626] User specific processing

[0627] The user checks the results of the assessment through their device. If an abnormality is detected, an alert sound or a warning icon will be displayed to notify the user. The user can then review the detailed analysis results and take appropriate action according to the presented countermeasures. For example, if an abnormal sound is detected in the vehicle's engine, it may be recommended to replace the bearings or conduct further inspections.

[0628] Specific examples

[0629] For example, monitoring the engine noise of a self-driving vehicle might proceed as follows:

[0630] 1. A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time.

[0631] 2. Collected audio data is converted into digital form and then denoised and filtered.

[0632] 3. The preprocessed audio data is sent to the cloud server.

[0633] 4. The server receives the audio data, extracts features using the generative AI model, and analyzes whether there are any abnormal sounds.

[0634] 5. If an abnormal sound is detected, a judgment result such as "Abnormal bearing noise detected" and specific countermeasures are generated and notified to the terminal.

[0635] 6. The user should check the notified judgment result and take appropriate measures such as replacing the bearing.

[0636] Prompt Sentence Examples

[0637] "input:

[0638] Audio data: [0.1, 0.2, -0.1, ...]

[0639] output:

[0640] Verification result: Abnormal noise detected. Details: Abnormal engine bearing noise detected. Confidence level: 95%

[0641] In this way, an anomaly detection system using acoustic data will significantly improve vehicle safety and operational efficiency.

[0642] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0643] Step 1:

[0644] The device uses a highly sensitive microphone to collect audio data in real time. The input is ambient sounds, such as the sound of a vehicle engine, which are captured through the microphone. These sounds are collected in analog format, and the output is analog audio data.

[0645] Step 2:

[0646] The terminal converts the collected analog voice data into a digital format and performs noise reduction and filtering. Analog voice data is used as input. Noise reduction removes unwanted background noise, and filtering emphasizes the desired frequency band. The output is pre-processed digital voice data.

[0647] Step 3:

[0648] The device sends preprocessed digital audio data to the cloud server. The input is the source data, which is encrypted and compressed before being sent over the network. The output is the audio data sent to the cloud server.

[0649] Step 4:

[0650] The server receives the voice data sent from the device and stores it in temporary storage. The received data is used as input. The server checks the integrity of the data and performs the storage process. The output is the temporarily stored digital voice data.

[0651] Step 5:

[0652] The server extracts features from the stored voice data. In this process, features such as frequency spectrum and amplitude required for the generative AI model are extracted. Temporarily stored digital voice data is used as input. The voice data is analyzed and features are extracted, resulting in feature data as output.

[0653] Step 6:

[0654] The server inputs the extracted features into the generative AI model and performs voice pattern analysis. The extracted feature data is used as input. The generative AI model has learned normal and abnormal voice patterns and performs analysis by comparing them with the feature data. The output is the analysis results.

[0655] Step 7:

[0656] Based on the analysis results, the server determines whether there are any abnormal sounds or specific patterns and generates a judgment result. The results of the voice pattern analysis are used as input. The judgment results include "no abnormality," "abnormality present," "specific abnormal sound pattern present," etc. The output is the judgment result data.

[0657] Step 8:

[0658] The server sends the generated judgment result to the terminal and notifies the user. The judgment result data is used as input. The judgment result is compressed and encrypted and sent to the terminal. The output is the judgment result received by the user through the terminal.

[0659] Step 9:

[0660] The user checks the judgment results through a terminal. The judgment results displayed on the terminal are used as input. If an abnormality is detected, the user is notified of the detailed analysis results and countermeasures, along with an alert sound and a warning icon. The output is the confirmed judgment results and proposed countermeasures.

[0661] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0662] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[0663] Specific processing of the terminal

[0664] Audio data collection

[0665] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[0666] Digital Conversion and Filtering

[0667] The collected audio data is converted from analog to digital, then filtered to reduce background noise and emphasize the desired frequency band.

[0668] Sending data

[0669] The pre-processed audio data is packetized and sent to the server using a secure protocol, where data is compressed and encrypted to ensure secure communication.

[0670] Server specific processing

[0671] Data Receipt and Storage

[0672] The server receives the voice data sent from the device and stores it in temporary storage. It checks for errors while verifying the integrity of the data.

[0673] Feature extraction and analysis preparation

[0674] The server loads the trained AI model and extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[0675] Voice pattern analysis and judgment

[0676] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns, which determines whether an abnormality exists or the existence of a specific pattern.

[0677] Generating judgment results

[0678] The server generates a judgment result based on the analysis results, which is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc.

[0679] Sending the judgment result

[0680] The judgment result is sent to the terminal, and the data is compressed and encrypted as necessary.

[0681] Specific processing of the emotion engine

[0682] emotion recognition

[0683] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. The emotion engine detects multiple emotional states, such as the user's stress level, excitement, or calmness.

[0684] Emotion-based notification adjustment

[0685] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, a notification that emphasizes a more serious warning will be sent.

[0686] User specific processing

[0687] Check the results

[0688] The user checks the results on their device, and notifications are dynamically adjusted based on the emotion engine's results, using audio and visual alerts.

[0689] View detailed data and take action

[0690] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the user's stress level is high, the user will be provided with support information to take prompt and appropriate action.

[0691] Examples: Vehicle engine sound inspection and user emotion recognition

[0692] Device behavior

[0693] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[0694] Server Operation

[0695] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[0696] User Actions

[0697] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[0698] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[0699] The processing flow will be explained below.

[0700] Step 1:

[0701] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[0702] Step 2:

[0703] The device converts the collected voice data from analog to digital signals, putting the voice data into an analyzable format.

[0704] Step 3:

[0705] The device then performs a filtering process on the digital audio signal, reducing background noise and emphasizing important frequency bands.

[0706] Step 4:

[0707] The terminal packetizes the pre-processed voice data, compresses and encrypts the data, and then transmits it to the server.

[0708] Step 5:

[0709] The server receives the voice data sent from the device and temporarily stores it in storage. It checks the data for consistency and performs error checks.

[0710] Step 6:

[0711] The server loads the trained AI model and prepares it for analysis.

[0712] Step 7:

[0713] The server extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[0714] Step 8:

[0715] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[0716] Step 9:

[0717] Based on the analysis results, the server generates a judgment result such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[0718] Step 10:

[0719] The server sends the result of the judgment to the terminal and compresses and encrypts the data as necessary.

[0720] Step 11:

[0721] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, it evaluates their stress level, excitement, calmness, etc. based on their tone of voice and facial expressions.

[0722] Step 12:

[0723] The server combines the output information of the emotion engine with the judgment results and selects the most appropriate notification method for the user. For example, if the user is in a high-stress state, a high-level warning will be issued.

[0724] Step 13:

[0725] The device displays the judgment results and the emotion engine's evaluation results on the user interface, and the user is notified using audio and visual alerts.

[0726] Step 14:

[0727] The user can check the judgment results and the emotion engine's evaluation results displayed on the device. They can also refer to the detailed voice data analysis results.

[0728] Step 15:

[0729] The user can take appropriate measures based on the judgment results and the emotion engine's evaluation results. For example, if an abnormal sound is detected and the user's stress level is high, machine maintenance or part replacement will be carried out promptly.

[0730] Example 2

[0731] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] Conventional sound detection support systems did not take into account the user's emotional state and fixed the way and timing of notifications were received, which increased stress for the user. Furthermore, the security of audio data was not adequately ensured, which posed a risk of important data being leaked. Furthermore, because the criteria for determining abnormal sounds were fixed, flexible responses were difficult and the system relied on dedicated experts.

[0733] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for performing voice pattern analysis using a machine learning model, means for generating a determination result, means for recognizing the emotional state of the user by analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the emotion recognition result. This allows the user to receive notifications at an appropriate time and in an appropriate manner without feeling stressed. In addition, data confidentiality is maintained by compressing and encrypting the data using a secure protocol before transmission. Furthermore, flexible and highly accurate voice analysis is possible by comparing the criteria for determining abnormal sounds with criteria set in advance by experts.

[0734] A "high-sensitivity microphone" is a sound collection device that has the ability to pick up even the faintest sounds.

[0735] A "terminal" is an electronic device that collects and digitally processes voice data.

[0736] "Digital format" refers to a format in which an analog signal is converted into digital code.

[0737] "Filtering" is a process for removing unnecessary noise from the original audio data.

[0738] "Noise reduction" is a technique for reducing or removing background noise from collected audio data.

[0739] "Feature extraction" is the process of extracting useful information or patterns from speech data.

[0740] A "machine learning model" is an algorithm that learns from large amounts of data and enables pattern recognition and prediction.

[0741] "Voice pattern analysis" is a technology that analyzes voice data to detect specific voice patterns and anomalies.

[0742] The "determination result" is information that indicates the presence or absence of an abnormality or a specific state generated as a result of the analysis.

[0743] "User" means an individual or organization that uses the system.

[0744] "Emotional state" is a state that indicates a user's stress level and emotional response.

[0745] The "notification method" is the method for notifying the user of the judgment results and other information.

[0746] A "secure protocol" is a communication protocol that ensures confidentiality and integrity when sending and receiving data.

[0747] "Data compression" is a technique for reducing the volume of data.

[0748] "Encryption" is a technology that uses a specific algorithm to convert data into a format that cannot be deciphered by third parties.

[0749] The "criteria for determining abnormal sounds" are criteria for detecting abnormalities used in the sound analysis.

[0750] An "expert" is someone who has advanced knowledge and skills in a particular field.

[0751] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[0752] Specific processing of the terminal

[0753] Audio data collection

[0754] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line. The collected audio data is temporarily stored in the device as an analog signal.

[0755] Digital Conversion and Filtering

[0756] The terminal converts the collected analog voice data into a digital signal using an Analog-to-Digital Converter (ADC). The signal is then filtered using an FIR filter to remove high-frequency noise and emphasize key frequency bands.

[0757] Sending data

[0758] The pre-processed audio data is packetized and sent to the server using a secure protocol (e.g., TLS or SSL), with algorithms such as GZIP used to compress the data and AES encryption used for encryption.

[0759] Server specific processing

[0760] Data Receipt and Storage

[0761] The server receives the voice data sent from the device and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks.

[0762] Feature extraction and analysis preparation

[0763] The server then loads a pre-trained machine learning model to extract features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data, often using MFCC (Mel-Frequency Cepstral Coefficients) and STFT (Short-Time Fourier Transform).

[0764] Voice pattern analysis and judgment

[0765] Based on the extracted features, the server inputs them into a machine learning model and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. This method uses a convolutional neural network (CNN), which is good at analyzing time series data and spatial data of voice.

[0766] Generating judgment results

[0767] Based on the analysis results, the server generates a judgment result, which can be expressed as "no abnormality," "abnormality," "specific abnormal sound pattern," etc. For example, it can identify a specific abnormal sound, such as "abnormal sound in the engine bearing."

[0768] Sending the judgment result

[0769] The server sends the result to the device, compressing and encrypting the data as needed, again using TLS / SSL and AES encryption in the process.

[0770] Specific processing of the emotion engine

[0771] emotion recognition

[0772] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. For example, it uses the device's camera and microphone to analyze the user's facial expressions and tone of voice. For analysis, OpenCV is used to capture the user's facial expressions in real time and input them into an emotion analysis model.

[0773] Emotion-based notification adjustment

[0774] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, the server adjusts the notification method to be more specific and urgent. In addition to voice notifications, it also uses text notifications and graphical alerts.

[0775] User specific processing

[0776] Check the results

[0777] The user checks the results on their device. Notifications are dynamically adjusted based on the emotion engine's results, providing appropriate audible and visual alerts. For example, the device might display a message saying, "There is an abnormal noise in the engine bearings."

[0778] View detailed data and take action

[0779] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the stress level is high, specific instructions for considering replacing bearings will be displayed. This allows for quick and appropriate action.

[0780] Examples: Vehicle engine sound inspection and user emotion recognition

[0781] Device behavior

[0782] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[0783] Server Operation

[0784] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[0785] User Actions

[0786] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[0787] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[0788] Example prompts for generative AI models

[0789] If you need an explanation of specific system operations or each piece of hardware and software, enter prompts like the following into the generative AI model:

[0790] Example prompt sentence:

[0791] "Please explain in detail the benefits of combining the Acoustic Judgment Support System (ADSS) with an emotion engine and the specific processing steps. Please also mention the specific processing performed by the terminal, server, and user, as well as the hardware and software used. Also, as a concrete example, please explain how it works in the case of a vehicle engine sound inspection."

[0792] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0793] Step 1: Collecting audio data

[0794] The device uses a high-sensitivity microphone to collect audio data in real time from a specific sound source (e.g., a vehicle engine). The input is an analog audio signal from the high-sensitivity microphone attached to the device. This analog audio data is temporarily stored within the device. The output is analog audio data.

[0795] Step 2: Digital Conversion and Filtering

[0796] The terminal converts the collected analog voice data (input) into a digital signal. This conversion is performed using an Analog-to-Digital Converter (ADC). The digital voice data is then filtered to reduce background noise and emphasize key frequency bands. Specifically, an FIR filter is used. The filtered digital voice data is output.

[0797] Step 3: Sending data

[0798] The device packets preprocessed digital audio data (input) and sends it to the server using a secure protocol (e.g., TLS or SSL). The data is compressed using an algorithm such as GZIP and encrypted using the AES cipher. The encrypted compressed data is sent to the server as output.

[0799] Step 4: Receiving and storing data

[0800] The server receives the encrypted compressed data (input) sent from the terminal and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks. The output is the original stored digital audio data.

[0801] Step 5: Feature extraction and analysis preparation

[0802] The server loads a pre-trained machine learning model based on the stored digital audio data (input) and extracts features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data. This process uses MFCC and STFT. The output is the extracted audio data features.

[0803] Step 6: Voice pattern analysis and decision

[0804] The server inputs the feature-extracted voice data (input) into a machine learning model and compares it with normal voice patterns and abnormal sound patterns to measure the degree of match. Specifically, the analysis is performed using a convolutional neural network (CNN). The output is the analysis result (e.g., "normal," "abnormal," or "specific abnormal sound pattern present").

[0805] Step 7: Generate a decision result

[0806] The server generates a judgment result based on the voice pattern analysis result (input). The judgment result is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The judgment result to be notified to the user is generated as output.

[0807] Step 8: Send the result

[0808] The server sends the generated judgment result (input) to the terminal, compressing and encrypting the data if necessary, again using a secure protocol (e.g., TLS or SSL) and AES encryption. The encrypted judgment result data is sent to the terminal as output.

[0809] Step 9: Emotion Recognition

[0810] The device or server analyzes the user's voice and facial expressions (input) to recognize their emotional state. Specifically, it uses the device's camera and microphone to capture facial expressions in real time using OpenCV and an emotion analysis model based on voice analysis. The output is the user's emotional state (e.g., stress level, excitement, calm).

[0811] Step 10: Adjust notifications based on emotion

[0812] The server combines the judgment results with the emotion recognition results (input) to select the optimal notification method for the user. Specifically, if the stress level is high, a notification method that emphasizes urgency and specificity is selected. The output is the adjusted notification content and format.

[0813] Step 11: Check the results

[0814] The user confirms the adjusted notification content (input) through the terminal. Notifications are provided through voice, text, graphical alerts, etc. The output allows the user to acknowledge and respond to the judgment results.

[0815] Step 12: View detailed data and take action

[0816] The user can view the detailed voice data analysis results and emotion recognition results (input) and take appropriate countermeasures. For example, if an abnormal sound is detected and the stress level is high, the user can quickly consider replacing the bearings. As an output, appropriate countermeasures are taken and the equipment continues to operate normally.

[0817] (Application example 2)

[0818] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] In modern factories, it is extremely important to quickly detect abnormal machine sounds and perform maintenance at the appropriate time. However, conventional acoustic detection systems do not take the user's emotional state into consideration when notifying them of the detection results, which can lead to problems such as increased stress. In addition, there is a need for a comprehensive system that not only detects abnormal machine sounds but also efficiently monitors the entire acoustic environment.

[0820] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving voice data transmitted from the terminal and performing noise reduction and feature extraction, means for performing voice pattern analysis using a pre-trained AI model based on the features extracted by the server, means for determining the presence or absence of abnormal sounds or specific patterns based on the analysis results and generating a determination result, means for transmitting the determination result to the terminal and notifying the user, means for recognizing the user's emotional state using a camera and image recognition, and means for dynamically adjusting the notification method to the user depending on the emotional state. This makes it possible not only to accurately detect abnormal sounds from machines but also to provide appropriate notifications taking the user's emotional state into consideration.

[0821] A "terminal" is a device equipped with a highly sensitive microphone that collects audio data in real time.

[0822] The "server" is a device that receives voice data sent from the terminal, removes noise and extracts features, and analyzes the data to determine whether or not there are any abnormal sounds or specific patterns.

[0823] A "high-sensitivity microphone" is a microphone that can collect sound with high accuracy.

[0824] "Noise reduction" is the process of removing unnecessary background noise from collected audio data.

[0825] "Feature extraction" is the process of extracting important patterns and features from collected voice data.

[0826] An "AI model" is a model that uses artificial intelligence to analyze data and is trained to identify specific patterns.

[0827] "Voice pattern analysis" is a process that uses an AI model to analyze patterns in voice data and determine whether there are any abnormal sounds or specific patterns.

[0828] The "determination result" is a result regarding the presence of an abnormal sound or a specific pattern, which is generated based on the sound pattern analysis.

[0829] A "camera" is a device for taking still images and videos.

[0830] "Image recognition" is a technology that analyzes images taken with a camera and recognizes specific objects or situations.

[0831] "Emotional state" refers to the user's emotional state, such as stress level, excitement, or calmness.

[0832] "Notification method" refers to the means or format for conveying information or warnings to the user.

[0833] A specific system for implementing the present invention will now be described.

[0834] System Overview

[0835] This system is configured so that a device equipped with a highly sensitive microphone collects voice data in real time and sends it to a server. The server then removes noise from the voice data, extracts features, and analyzes the voice patterns using an AI model. Based on the analysis results, it determines whether there are any abnormal sounds or specific patterns, and dynamically adjusts the notification method taking into account the results of the analysis and the user's emotional state.

[0836] Specific processing method for the terminal

[0837] The device is equipped with a highly sensitive microphone that collects audio data in real time from specific sound sources, such as machinery sounds in a factory. The collected audio data is converted from analog to digital format and then filtered to reduce background noise and emphasize frequency bands that are useful for detecting abnormal sounds. After preprocessing, the audio data is packetized and sent to a server using a secure protocol.

[0838] Specific server processing method

[0839] The server receives the voice data sent from the device and temporarily stores it in storage. Next, it checks for errors while verifying the data's consistency. After that, it loads the trained AI model and extracts features (frequency spectrum, amplitude, phase, etc.) from the voice data. It performs voice pattern analysis based on the features and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match, thereby determining whether there is an abnormality or the presence of a specific pattern. The judgment result is generated as "no abnormality," "abnormality present," or "specific abnormal sound pattern present." The server then uses a camera and image recognition technology to recognize the user's emotional state from facial expressions, etc. The recognized emotional state is combined with the judgment result to select the most appropriate notification method and send it to the device.

[0840] Specific user operations

[0841] The user can check the results of the assessment through their device. Notifications are dynamically adjusted according to the user's emotional state; for example, if the user is in a high stress state, a more pronounced warning sound or visual alert is displayed. Based on the analysis results and the evaluation of the emotional state, support information is provided to take prompt and appropriate action if the user is in a high stress state. The user can then perform maintenance or repairs on the machine as needed.

[0842] Hardware and software used

[0843] Hardware used:

[0844] High-sensitivity microphone

[0845] Terminal (for collecting voice data)

[0846] Server (for data analysis and judgment)

[0847] Camera (for emotion recognition)

[0848] Software used:

[0849] pyaudio (audio data collection)

[0850] socket (data transmission)

[0851] Keras (AI model for analyzing voice patterns)

[0852] dlib (image recognition and emotional state recognition)

[0853] Specific examples

[0854] For example, if this system is used to detect abnormal sounds from machinery operating in a factory, a highly sensitive microphone collects the sounds and sends the data to a server. The server analyzes the data, and if an abnormal sound is detected, the results are sent to the terminal. At the same time, the camera analyzes the operator's facial expression, and if the user is in a state of high stress, an exaggerated warning sound or visual alert is displayed. This notification method takes the user's emotional state into account, enabling a quick and appropriate response.

[0855] Prompt Sentence Examples

[0856] "Write a Python program to help a factory robot collect mechanical sounds and detect abnormal sounds. Use a highly sensitive microphone to collect the audio in real time, and then convert and filter the data digitally before sending it to a server."

[0857] "Write a Python program that uses an AI model to analyze voice data to detect anomalous patterns, and then notifies the user based on the results. The notification content should dynamically adjust based on the user's emotional state (stress level, calmness)."

[0858] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0859] Step 1:

[0860] The terminal collects machine sounds within the factory in real time using a highly sensitive microphone.

[0861] Specifically, a high-sensitivity microphone installed in the terminal collects analog audio data from specific sound sources within the factory, and this collected audio data becomes the input for the next processing step.

[0862] Step 2:

[0863] The analog voice data collected by the terminal is converted into digital format and filtered.

[0864] The input is analog audio data, which is converted to a digital signal, then filtered to reduce background noise and highlight abnormal sounds, and this digital audio data is the input for the next processing step.

[0865] Step 3:

[0866] The terminal converts the preprocessed voice data into packets and transmits them to the server.

[0867] The input is pre-processed digital audio data, which is then packetized and sent to the server using a secure protocol (e.g., TLS), where the packetized data is the input for processing.

[0868] Step 4:

[0869] The server receives the audio data, temporarily stores it in storage, and performs error checking.

[0870] The input is packetized voice data sent from the terminal, which the server receives and stores, and performs error checks to ensure data integrity. This temporarily stored data becomes the input for the next analysis step.

[0871] Step 5:

[0872] The server extracts features (frequency spectrum, amplitude, phase, etc.) from the stored audio data.

[0873] The input is stored digital audio data, and an AI model is used to extract features from the audio data, which then become the input for the next step of audio pattern analysis.

[0874] Step 6:

[0875] The server analyzes the audio pattern using an AI model based on the features and determines whether there are any abnormal sounds or specific patterns.

[0876] The input is the extracted features, which are then input into the AI ​​model for analysis. The analysis results determine whether there are any abnormal sounds or specific patterns, and a judgment result is generated. This judgment result becomes the input for the next notification adjustment step.

[0877] Step 7:

[0878] The server uses a camera to recognize the user's emotional state.

[0879] The input is image data of the user collected by a camera, and image recognition technology is used to analyze the user's emotional state (stress level, calmness, etc.). This analyzed emotional state becomes additional input for the next notification adjustment step.

[0880] Step 8:

[0881] The server dynamically adjusts the notification method based on the judgment result and the user's emotional state.

[0882] The inputs are the analyzed emotional state and the judgment result, and these are combined to select the optimal notification method. For example, if the user is in a high stress state, an enhanced warning sound or visual alert is selected. This adjusted notification method becomes the input for the next notification sending step.

[0883] Step 9:

[0884] The server transmits the adjusted notification method and the determination result to the terminal.

[0885] The input is the adjusted notification method and the judgment result, which is sent to the terminal. The sent notification becomes the input for the next user confirmation step.

[0886] Step 10:

[0887] The terminal notifies the user, and the user confirms the judgment result.

[0888] The input is the notification method and judgment result sent from the server, which the device notifies to the user. The user can take appropriate measures or actions based on the displayed judgment result and notification.

[0889] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0890] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0891] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0892] [Third embodiment]

[0893] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0894] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0895] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0896] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0897] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0898] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0899] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0900] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0901] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0902] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0903] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0904] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0905] To implement this invention, the terminals, servers, and programs installed on them that constitute the Acoustic Discrimination Support System (ADSS) are required. Specific processes performed by the server, terminals, and users are described below.

[0906] Specific processing of the terminal

[0907] Audio data collection

[0908] The device uses a highly sensitive microphone to collect audio data in real time, allowing it to accurately capture data from specific sound sources, such as a vehicle engine or machinery on a production line.

[0909] Digital Conversion and Filtering

[0910] The collected audio data is converted from analog to digital format and then denoised and filtered, which suppresses background noise and emphasises the desired frequency bands.

[0911] Sending data

[0912] The pre-processed audio data is sent over the network to a server, where it is compressed and encrypted for secure and efficient transmission.

[0913] Server specific processing

[0914] Data Receipt and Storage

[0915] The server receives the voice data sent from the device and stores it in temporary storage, verifies the integrity of the data, and prepares it for analysis.

[0916] Feature extraction and analysis preparation

[0917] The server loads the trained AI model and extracts features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for voice pattern analysis.

[0918] Voice pattern analysis and judgment

[0919] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns. Based on the results, it is determined whether an abnormality exists or whether a specific pattern exists.

[0920] Generation and notification of judgment results

[0921] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The generated judgment result is sent to the terminal and notified to the user.

[0922] User specific processing

[0923] Check the results

[0924] The user can check the results of the assessment through their device. For example, if an abnormality is detected, the user is notified by an alert sound or a warning icon.

[0925] View detailed data and take action

[0926] If necessary, the user can check the detailed audio data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, machine maintenance, part replacement, or further inspection can be carried out.

[0927] Example: Vehicle engine sound inspection

[0928] Device behavior

[0929] A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time. The collected audio data is converted into a digital format, and then noise reduction and filtering are performed. The pre-processed data is then sent to a server.

[0930] Server Operation

[0931] The server temporarily stores the received voice data and extracts features from the voice data. The extracted features are input into an AI model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the device.

[0932] User Actions

[0933] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, the user takes measures such as replacing the bearings.

[0934] In this way, acoustic inspection work can be carried out quickly and accurately. This system significantly improves the efficiency and reliability of acoustic inspections, and also solves the problems of labor shortages and training costs.

[0935] The processing flow will be explained below.

[0936] Step 1:

[0937] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or factory machinery.

[0938] Step 2:

[0939] The device converts the collected voice data from analog to digital signals, making it in a format that can be analyzed.

[0940] Step 3:

[0941] The device then filters the digital audio data, reducing background noise and emphasizing important frequency bands.

[0942] Step 4:

[0943] The terminal then packetizes the pre-processed voice data, encrypts it, and sends it to the server, ensuring secure data transmission.

[0944] Step 5:

[0945] The server receives the voice data sent from the device and temporarily stores it in storage. It also performs error checking to ensure data integrity.

[0946] Step 6:

[0947] The server loads the AI ​​model and prepares it for analysis, using a pre-trained model.

[0948] Step 7:

[0949] The server extracts features from the audio data, specifically, frequency spectrum, amplitude, phase, etc.

[0950] Step 8:

[0951] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[0952] Step 9:

[0953] The server generates a judgment result based on the analysis results, such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[0954] Step 10:

[0955] The server generates a judgment result and sends it to the terminal, compressing and encrypting the data as necessary.

[0956] Step 11:

[0957] The terminal displays the result of the judgment received on the user interface, for example, by notifying the user using an alert sound or a warning icon.

[0958] Step 12:

[0959] The user checks the judgment results displayed on the device and, if necessary, can also view detailed voice data analysis results.

[0960] Step 13:

[0961] The user takes appropriate measures based on the judgment results. For example, if an abnormal sound is detected, the user performs machine maintenance or replaces parts.

[0962] Example 1

[0963] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Conventional acoustic detection systems lack the ability to collect acoustic data in real time and analyze it quickly and accurately, making it difficult to distinguish between normal and abnormal movement patterns. They also lacked the functionality to efficiently determine the presence or absence of abnormal sounds or specific patterns, resulting in delays in providing appropriate notifications and feedback to users. Furthermore, the entire process, from collecting acoustic data to analyzing it and notifying users, was not integrated, resulting in low efficiency for the entire system.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0966] In this invention, the server includes a terminal equipped with a highly sensitive sensor that collects acoustic data in real time, a processing device that receives the acoustic data transmitted from the terminal and performs noise removal and feature extraction, means for performing acoustic pattern analysis using a pre-trained machine learning model based on the features extracted by the processing device, means for determining the presence or absence of abnormal acoustics or specific patterns based on the analysis results and generating a determination result, and means for transmitting the determination result to the terminal and notifying a user. This makes it possible to collect acoustic data in real time, analyze it quickly and accurately, efficiently determine the presence or absence of abnormal acoustics or specific patterns, and provide appropriate notification and feedback to the user.

[0967] "Acoustic data" is information that is generated by converting sound into electrical signals and storing or processing them in digital form.

[0968] A "high-sensitivity sensor" is an acoustic detection device capable of detecting minute acoustic changes with high precision.

[0969] A "terminal" is a device that collects acoustic data and sends it to a server for processing.

[0970] A "processor" is a computer system that has the computational power to analyze acoustic data.

[0971] A "machine learning model" is an algorithm that is trained on large amounts of data to perform pattern recognition and prediction.

[0972] "Acoustic pattern analysis" is a process that identifies abnormalities and specific patterns based on the features of acoustic data.

[0973] "Abnormal sound" is a sound that is different from normal operating sounds and indicates an abnormality.

[0974] The "determination result" is diagnostic information generated based on the analysis of the acoustic data.

[0975] A "user" is a person who operates the system and checks the judgment results.

[0976] "Noise reduction" is a process for removing unnecessary background sounds from audio data.

[0977] "Feature extraction" is the process of extracting important patterns and properties from acoustic data.

[0978] "Real-time" refers to the simultaneous collection and processing of acoustic data.

[0979] MODE FOR CARRYING OUT THE INVENTION

[0980] To implement this invention, an Acoustic Decision Support System (ADSS) is required, which includes the following components: ADSS is realized by the roles of terminal, server, and user.

[0981] Terminal configuration and operation

[0982] Acoustic data collection

[0983] The device is equipped with highly sensitive sensors that collect acoustic data in real time from specific sound sources, such as a vehicle engine or a machine on a production line, with high accuracy. The device also includes location and time information to provide context for the collected acoustic data.

[0984] Digital Conversion and Filtering

[0985] The collected acoustic data is in analog format and is converted to digital format using an ADC (analog-to-digital converter) within the device. It is then denoised and filtered using audio processing software such as Audacity, which removes unwanted background noise and emphasises the desired frequency bands.

[0986] Sending data

[0987] The pre-processed digital audio data is then sent to the server via the network. During data transmission, the data is compressed and encrypted using gzip, TLS / SSL, etc., for safe and efficient transmission.

[0988] Server configuration and operation

[0989] Data Receipt and Storage

[0990] The server receives the acoustic data sent from the device and stores it in temporary storage (e.g., Amazon S3). It checks the integrity of the data and verifies that it is not corrupted. This temporary storage makes the data easily available for subsequent processing.

[0991] Feature extraction and analysis preparation

[0992] The server loads a trained machine learning model (e.g., TensorFlow or PyTorch) and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data, which is then ready for analysis.

[0993] Acoustic pattern analysis and judgment

[0994] The server uses the extracted features to perform acoustic pattern analysis. The features are input into a machine learning model, and the degree of match is calculated by comparing normal and abnormal sound patterns. Based on the results of this analysis, it determines whether an abnormality exists or whether a specific pattern exists.

[0995] Generation and notification of judgment results

[0996] The server generates a judgment result based on the analysis results. The result is displayed in the form of "No abnormality," "Abnormality detected," "Specific abnormal sound pattern detected," etc. These judgment results are sent to the terminal and notified to the user.

[0997] User Configuration and Behavior

[0998] Check the results

[0999] The user can check the results of the assessment through their device. For example, if an abnormality is detected, an alert sound or a warning icon will be displayed to alert the user.

[1000] View detailed data and take action

[1001] If necessary, the user can check the detailed acoustic data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, the user can perform machine maintenance, replace parts, or perform further inspections.

[1002] Example: Inspecting vehicle engine noise

[1003] Device behavior

[1004] Highly sensitive sensors are installed in the vehicle's engine to collect engine sounds in real time. The collected acoustic data is then digitally converted and filtered before being sent to a server via a network.

[1005] Server Operation

[1006] The server temporarily stores the received acoustic data and extracts features from the acoustic data. The extracted features are input into a machine learning model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the terminal.

[1007] User Actions

[1008] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, countermeasures such as replacing the bearings are implemented.

[1009] Examples of prompt statements

[1010] "Generate Python code to collect vehicle engine sounds in real time and detect abnormal sounds."

[1011] In this way, the Acoustic Decision Support System (ADSS) can quickly and efficiently carry out the entire process from collection to analysis and notification, improving the efficiency and reliability of acoustic inspections and resolving the problems of labor shortages and training costs.

[1012] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1013] Step 1:

[1014] Acoustic data collection

[1015] The device uses a highly sensitive sensor to collect acoustic data in real time from specific sound sources. For example, it can capture the sound of a vehicle engine or the machine noise of a production line. The input is an analog acoustic signal captured by the acoustic sensor, and the output is analog acoustic data. Specifically, the device's highly sensitive sensor is placed near the target sound source and continuously collects acoustic data.

[1016] Step 2:

[1017] Digital Conversion and Filtering

[1018] The device converts the collected analog acoustic data into digital format using a built-in analog-to-digital converter (ADC). Next, the converted digital acoustic data is processed using audio processing software such as Audacity to remove noise and perform filtering. The input is analog acoustic data, and the output is filtered digital acoustic data. Specifically, the audio data is converted into digital format using the built-in ADC, and then white noise and unwanted frequencies are removed using Audacity.

[1019] Step 3:

[1020] Sending data

[1021] The device sends the preprocessed digital audio data to a server over the Internet or a local network. The data is sent in a compressed format such as gzip and encrypted format such as TLS / SSL. The input is the filtered digital audio data, and the output is compressed and encrypted audio data packets. Specifically, the digital audio data is compressed and securely sent to the server using an encryption protocol.

[1022] Step 4:

[1023] Data reception and storage

[1024] The server receives the digital audio data packets sent from the device, decompresses and decrypts them, and stores the received data in temporary storage (e.g., cloud storage). The input is the compressed and encrypted audio data packets, and the output is the digital audio data stored in the storage. Specifically, the server decompresses and decrypts the received data packets before storing them in cloud storage.

[1025] Step 5:

[1026] Feature extraction and analysis preparation

[1027] The server loads a pre-trained machine learning model (e.g., TensorFlow or PyTorch) based on the stored acoustic data, and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data. The input is the stored digital acoustic data, and the output is the extracted feature data. Specifically, the acoustic data is converted into a frequency spectrum, and amplitude and other important features are extracted.

[1028] Step 6:

[1029] Acoustic pattern analysis and judgment

[1030] The server inputs the feature data into the AI ​​model and analyzes whether there is an abnormality by comparing normal and abnormal acoustic patterns. The input is the feature data, and the output is the judgment result of whether there is an abnormality or a specific acoustic pattern. Specifically, the server inputs the feature data into the AI ​​model and analyzes whether the model detects an abnormal sound or a specific pattern.

[1031] Step 7:

[1032] Generation and notification of judgment results

[1033] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality", "abnormality detected", "specific abnormal sound pattern detected", etc. The generated judgment result is sent to the terminal and notified to the user. The input is the analysis result, and the output is a notification of the judgment result. Specifically, the server forms the judgment result and sends it to the terminal as a data packet.

[1034] Step 8:

[1035] Confirm the results and implement countermeasures

[1036] The user checks the judgment results through the terminal. If an abnormality is detected, an alert sound or warning icon is displayed, and the user can refer to the detailed analysis results and take appropriate measures. The input is the notification of the judgment results, and the output is the countermeasures to be implemented by the user. Specifically, the terminal displays an alert, and the user checks the detailed data before carrying out maintenance or repairs.

[1037] (Application example 1)

[1038] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1039] For autonomous vehicles, it is important to detect abnormalities in the vehicle early and respond quickly and appropriately. However, conventional methods require advanced expertise and expensive equipment to detect abnormal sounds, making it difficult for ordinary vehicle owners. Furthermore, even if abnormal sounds were detected, there was a lack of systems that could present specific countermeasures to users, resulting in many cases of delays in repairs and maintenance. For this reason, there is a need for a system that can automatically detect abnormal sounds, present countermeasures, and notify users in an easy-to-understand manner.

[1040] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1041] In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving the voice data transmitted from the terminal and performing noise reduction and feature extraction, and means for performing voice pattern analysis using a pre-trained generative AI model based on the features extracted by the server. This makes it possible to determine the presence or absence of abnormal sounds or specific patterns and generate a determination result. Furthermore, if an abnormal sound is detected based on the analysis result, it is possible to propose a detailed analysis result and countermeasures to the user. This allows the user to quickly identify vehicle abnormalities and take appropriate measures, significantly improving vehicle safety and operational efficiency.

[1042] A "high-sensitivity microphone" is a microphone that can accurately pick up very quiet sounds and subtle changes in sound.

[1043] "Terminal" refers to an electronic device that has the functionality to collect, convert, and transmit data.

[1044] "Audio data" refers to data in which sound waveforms are recorded and stored in digital format.

[1045] "Digital format" is a format in which analog signals are expressed as numbers.

[1046] "Filtering" is a process that removes unnecessary noise and interference from collected audio data and emphasizes only the necessary frequency bands.

[1047] A "server" is a computer system that receives data from other terminals or devices and provides functions such as analysis, storage, and transmission.

[1048] "Noise reduction" is a process that removes background noise and unnecessary sounds from audio data.

[1049] "Feature extraction" is the process of extracting useful features (such as frequency components and amplitude) from audio data.

[1050] A "generative AI model" is an artificial intelligence model that has been pre-trained using machine learning algorithms to perform a specific task (in this case, speech pattern analysis).

[1051] "Voice pattern analysis" is an analytical method that detects the presence of specific voice patterns (for example, abnormal sounds) based on the features of voice data.

[1052] The "determination result" is the result obtained by the analysis, and is information indicating whether or not there is an abnormal sound or a specific audio pattern.

[1053] A "cloud server" is a remote computer system that provides data and services over the Internet.

[1054] The "criteria set in advance by experts" are criteria or rules set in advance by experts for determining abnormal sounds or specific audio patterns.

[1055] To implement this invention, a system is required that collects audio data in real time, detects abnormal sounds, and notifies the user. This system is composed of a terminal, a server, and programs installed on these. Specific processes performed by the server, terminal, and user are described below.

[1056] Specific processing of the terminal

[1057] The device is equipped with a highly sensitive microphone and a mechanism for collecting voice data in real time. This voice data is captured in analog form and then converted to digital form. During this process, noise reduction and filtering are performed to suppress background noise and emphasize desired frequency bands. The pre-processed voice data is then securely and efficiently transmitted over the network to a cloud server.

[1058] Server specific processing

[1059] The server receives the voice data sent from the device and stores it in temporary storage. It checks the integrity of this data and prepares it for analysis. Next, it uses a pre-trained generative AI model to extract features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for analysis. These features are then input into the AI ​​model, which compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. Based on the results, it determines whether there is an abnormality or the existence of a specific pattern, and generates a judgment result. If an abnormal sound is detected, it also generates detailed analysis results and information to suggest countermeasures to the user.

[1060] User specific processing

[1061] The user checks the results of the assessment through their device. If an abnormality is detected, an alert sound or a warning icon will be displayed to notify the user. The user can then review the detailed analysis results and take appropriate action according to the presented countermeasures. For example, if an abnormal sound is detected in the vehicle's engine, it may be recommended to replace the bearings or conduct further inspections.

[1062] Specific examples

[1063] For example, monitoring the engine noise of a self-driving vehicle might proceed as follows:

[1064] 1. A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time.

[1065] 2. Collected audio data is converted into digital form and then denoised and filtered.

[1066] 3. The preprocessed audio data is sent to the cloud server.

[1067] 4. The server receives the audio data, extracts features using the generative AI model, and analyzes whether there are any abnormal sounds.

[1068] 5. If an abnormal sound is detected, a judgment result such as "Abnormal bearing noise detected" and specific countermeasures are generated and notified to the terminal.

[1069] 6. The user should check the notified judgment result and take appropriate measures such as replacing the bearing.

[1070] Prompt Sentence Examples

[1071] "input:

[1072] Audio data: [0.1, 0.2, -0.1, ...]

[1073] output:

[1074] Verification result: Abnormal noise detected. Details: Abnormal engine bearing noise detected. Confidence level: 95%

[1075] In this way, an anomaly detection system using acoustic data will significantly improve vehicle safety and operational efficiency.

[1076] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1077] Step 1:

[1078] The device uses a highly sensitive microphone to collect audio data in real time. The input is ambient sounds, such as the sound of a vehicle engine, which are captured through the microphone. These sounds are collected in analog format, and the output is analog audio data.

[1079] Step 2:

[1080] The terminal converts the collected analog voice data into a digital format and performs noise reduction and filtering. Analog voice data is used as input. Noise reduction removes unwanted background noise, and filtering emphasizes the desired frequency band. The output is pre-processed digital voice data.

[1081] Step 3:

[1082] The device sends preprocessed digital audio data to the cloud server. The input is the source data, which is encrypted and compressed before being sent over the network. The output is the audio data sent to the cloud server.

[1083] Step 4:

[1084] The server receives the voice data sent from the device and stores it in temporary storage. The received data is used as input. The server checks the integrity of the data and performs the storage process. The output is the temporarily stored digital voice data.

[1085] Step 5:

[1086] The server extracts features from the stored voice data. In this process, features such as frequency spectrum and amplitude required for the generative AI model are extracted. Temporarily stored digital voice data is used as input. The voice data is analyzed and features are extracted, resulting in feature data as output.

[1087] Step 6:

[1088] The server inputs the extracted features into the generative AI model and performs voice pattern analysis. The extracted feature data is used as input. The generative AI model has learned normal and abnormal voice patterns and performs analysis by comparing them with the feature data. The output is the analysis results.

[1089] Step 7:

[1090] Based on the analysis results, the server determines whether there are any abnormal sounds or specific patterns and generates a judgment result. The results of the voice pattern analysis are used as input. The judgment results include "no abnormality," "abnormality present," "specific abnormal sound pattern present," etc. The output is the judgment result data.

[1091] Step 8:

[1092] The server sends the generated judgment result to the terminal and notifies the user. The judgment result data is used as input. The judgment result is compressed and encrypted and sent to the terminal. The output is the judgment result received by the user through the terminal.

[1093] Step 9:

[1094] The user checks the judgment results through a terminal. The judgment results displayed on the terminal are used as input. If an abnormality is detected, the user is notified of the detailed analysis results and countermeasures, along with an alert sound and a warning icon. The output is the confirmed judgment results and proposed countermeasures.

[1095] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1096] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[1097] Specific processing of the terminal

[1098] Audio data collection

[1099] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[1100] Digital Conversion and Filtering

[1101] The collected audio data is converted from analog to digital, then filtered to reduce background noise and emphasize the desired frequency band.

[1102] Sending data

[1103] The pre-processed audio data is packetized and sent to the server using a secure protocol, where data is compressed and encrypted to ensure secure communication.

[1104] Server specific processing

[1105] Data Receipt and Storage

[1106] The server receives the voice data sent from the device and stores it in temporary storage. It checks for errors while verifying the integrity of the data.

[1107] Feature extraction and analysis preparation

[1108] The server loads the trained AI model and extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[1109] Voice pattern analysis and judgment

[1110] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns, which determines whether an abnormality exists or the existence of a specific pattern.

[1111] Generating judgment results

[1112] The server generates a judgment result based on the analysis results, which is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc.

[1113] Sending the judgment result

[1114] The judgment result is sent to the terminal, and the data is compressed and encrypted as necessary.

[1115] Specific processing of the emotion engine

[1116] emotion recognition

[1117] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. The emotion engine detects multiple emotional states, such as the user's stress level, excitement, or calmness.

[1118] Emotion-based notification adjustment

[1119] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, a notification that emphasizes a more serious warning will be sent.

[1120] User specific processing

[1121] Check the results

[1122] The user checks the results on their device, and notifications are dynamically adjusted based on the emotion engine's results, using audio and visual alerts.

[1123] View detailed data and take action

[1124] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the user's stress level is high, the user will be provided with support information to take prompt and appropriate action.

[1125] Examples: Vehicle engine sound inspection and user emotion recognition

[1126] Device behavior

[1127] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[1128] Server Operation

[1129] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[1130] User Actions

[1131] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[1132] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[1133] The processing flow will be explained below.

[1134] Step 1:

[1135] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[1136] Step 2:

[1137] The device converts the collected voice data from analog to digital signals, putting the voice data into an analyzable format.

[1138] Step 3:

[1139] The device then performs a filtering process on the digital audio signal, reducing background noise and emphasizing important frequency bands.

[1140] Step 4:

[1141] The terminal packetizes the pre-processed voice data, compresses and encrypts the data, and then transmits it to the server.

[1142] Step 5:

[1143] The server receives the voice data sent from the device and temporarily stores it in storage. It checks the data for consistency and performs error checks.

[1144] Step 6:

[1145] The server loads the trained AI model and prepares it for analysis.

[1146] Step 7:

[1147] The server extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[1148] Step 8:

[1149] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[1150] Step 9:

[1151] Based on the analysis results, the server generates a judgment result such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[1152] Step 10:

[1153] The server sends the result of the judgment to the terminal and compresses and encrypts the data as necessary.

[1154] Step 11:

[1155] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, it evaluates their stress level, excitement, calmness, etc. based on their tone of voice and facial expressions.

[1156] Step 12:

[1157] The server combines the output information of the emotion engine with the judgment results and selects the most appropriate notification method for the user. For example, if the user is in a high-stress state, a high-level warning will be issued.

[1158] Step 13:

[1159] The device displays the judgment results and the emotion engine's evaluation results on the user interface, and the user is notified using audio and visual alerts.

[1160] Step 14:

[1161] The user can check the judgment results and the emotion engine's evaluation results displayed on the device. They can also refer to the detailed voice data analysis results.

[1162] Step 15:

[1163] The user can take appropriate measures based on the judgment results and the emotion engine's evaluation results. For example, if an abnormal sound is detected and the user's stress level is high, machine maintenance or part replacement will be carried out promptly.

[1164] Example 2

[1165] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1166] Conventional sound detection support systems did not take into account the user's emotional state and fixed the way and timing of notifications were received, which increased stress for the user. Furthermore, the security of audio data was not adequately ensured, which posed a risk of important data being leaked. Furthermore, because the criteria for determining abnormal sounds were fixed, flexible responses were difficult and the system relied on dedicated experts.

[1167] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for performing voice pattern analysis using a machine learning model, means for generating a determination result, means for recognizing the emotional state of the user by analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the emotion recognition result. This allows the user to receive notifications at an appropriate time and in an appropriate manner without feeling stressed. In addition, data confidentiality is maintained by compressing and encrypting the data using a secure protocol before transmission. Furthermore, flexible and highly accurate voice analysis is possible by comparing the criteria for determining abnormal sounds with criteria set in advance by experts.

[1168] A "high-sensitivity microphone" is a sound collection device that has the ability to pick up even the faintest sounds.

[1169] A "terminal" is an electronic device that collects and digitally processes voice data.

[1170] "Digital format" refers to a format in which an analog signal is converted into digital code.

[1171] "Filtering" is a process for removing unnecessary noise from the original audio data.

[1172] "Noise reduction" is a technique for reducing or removing background noise from collected audio data.

[1173] "Feature extraction" is the process of extracting useful information or patterns from speech data.

[1174] A "machine learning model" is an algorithm that learns from large amounts of data and enables pattern recognition and prediction.

[1175] "Voice pattern analysis" is a technology that analyzes voice data to detect specific voice patterns and anomalies.

[1176] The "determination result" is information that indicates the presence or absence of an abnormality or a specific state generated as a result of the analysis.

[1177] "User" means an individual or organization that uses the system.

[1178] "Emotional state" is a state that indicates a user's stress level and emotional response.

[1179] The "notification method" is the method for notifying the user of the judgment results and other information.

[1180] A "secure protocol" is a communication protocol that ensures confidentiality and integrity when sending and receiving data.

[1181] "Data compression" is a technique for reducing the volume of data.

[1182] "Encryption" is a technology that uses a specific algorithm to convert data into a format that cannot be deciphered by third parties.

[1183] The "criteria for determining abnormal sounds" are criteria for detecting abnormalities used in the sound analysis.

[1184] An "expert" is someone who has advanced knowledge and skills in a particular field.

[1185] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[1186] Specific processing of the terminal

[1187] Audio data collection

[1188] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line. The collected audio data is temporarily stored in the device as an analog signal.

[1189] Digital Conversion and Filtering

[1190] The terminal converts the collected analog voice data into a digital signal using an Analog-to-Digital Converter (ADC). The signal is then filtered using an FIR filter to remove high-frequency noise and emphasize key frequency bands.

[1191] Sending data

[1192] The pre-processed audio data is packetized and sent to the server using a secure protocol (e.g., TLS or SSL), with algorithms such as GZIP used to compress the data and AES encryption used for encryption.

[1193] Server specific processing

[1194] Data Receipt and Storage

[1195] The server receives the voice data sent from the device and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks.

[1196] Feature extraction and analysis preparation

[1197] The server then loads a pre-trained machine learning model to extract features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data, often using MFCC (Mel-Frequency Cepstral Coefficients) and STFT (Short-Time Fourier Transform).

[1198] Voice pattern analysis and judgment

[1199] Based on the extracted features, the server inputs them into a machine learning model and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. This method uses a convolutional neural network (CNN), which is good at analyzing time series data and spatial data of voice.

[1200] Generating judgment results

[1201] Based on the analysis results, the server generates a judgment result, which can be expressed as "no abnormality," "abnormality," "specific abnormal sound pattern," etc. For example, it can identify a specific abnormal sound, such as "abnormal sound in the engine bearing."

[1202] Sending the judgment result

[1203] The server sends the result to the device, compressing and encrypting the data as needed, again using TLS / SSL and AES encryption in the process.

[1204] Specific processing of the emotion engine

[1205] emotion recognition

[1206] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. For example, it uses the device's camera and microphone to analyze the user's facial expressions and tone of voice. For analysis, OpenCV is used to capture the user's facial expressions in real time and input them into an emotion analysis model.

[1207] Emotion-based notification adjustment

[1208] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, the server adjusts the notification method to be more specific and urgent. In addition to voice notifications, it also uses text notifications and graphical alerts.

[1209] User specific processing

[1210] Check the results

[1211] The user checks the results on their device. Notifications are dynamically adjusted based on the emotion engine's results, providing appropriate audible and visual alerts. For example, the device might display a message saying, "There is an abnormal noise in the engine bearings."

[1212] View detailed data and take action

[1213] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the stress level is high, specific instructions for considering replacing bearings will be displayed. This allows for quick and appropriate action.

[1214] Examples: Vehicle engine sound inspection and user emotion recognition

[1215] Device behavior

[1216] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[1217] Server Operation

[1218] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[1219] User Actions

[1220] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[1221] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[1222] Example prompts for generative AI models

[1223] If you need an explanation of specific system operations or each piece of hardware and software, enter prompts like the following into the generative AI model:

[1224] Example prompt sentence:

[1225] "Please explain in detail the benefits of combining the Acoustic Judgment Support System (ADSS) with an emotion engine and the specific processing steps. Please also mention the specific processing performed by the terminal, server, and user, as well as the hardware and software used. Also, as a concrete example, please explain how it works in the case of a vehicle engine sound inspection."

[1226] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1227] Step 1: Collecting audio data

[1228] The device uses a high-sensitivity microphone to collect audio data in real time from a specific sound source (e.g., a vehicle engine). The input is an analog audio signal from the high-sensitivity microphone attached to the device. This analog audio data is temporarily stored within the device. The output is analog audio data.

[1229] Step 2: Digital Conversion and Filtering

[1230] The terminal converts the collected analog voice data (input) into a digital signal. This conversion is performed using an Analog-to-Digital Converter (ADC). The digital voice data is then filtered to reduce background noise and emphasize key frequency bands. Specifically, an FIR filter is used. The filtered digital voice data is output.

[1231] Step 3: Sending data

[1232] The device packets preprocessed digital audio data (input) and sends it to the server using a secure protocol (e.g., TLS or SSL). The data is compressed using an algorithm such as GZIP and encrypted using the AES cipher. The encrypted compressed data is sent to the server as output.

[1233] Step 4: Receiving and storing data

[1234] The server receives the encrypted compressed data (input) sent from the terminal and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks. The output is the original stored digital audio data.

[1235] Step 5: Feature extraction and analysis preparation

[1236] The server loads a pre-trained machine learning model based on the stored digital audio data (input) and extracts features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data. This process uses MFCC and STFT. The output is the extracted audio data features.

[1237] Step 6: Voice pattern analysis and decision

[1238] The server inputs the feature-extracted voice data (input) into a machine learning model and compares it with normal voice patterns and abnormal sound patterns to measure the degree of match. Specifically, the analysis is performed using a convolutional neural network (CNN). The output is the analysis result (e.g., "normal," "abnormal," or "specific abnormal sound pattern present").

[1239] Step 7: Generate a decision result

[1240] The server generates a judgment result based on the voice pattern analysis result (input). The judgment result is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The judgment result to be notified to the user is generated as output.

[1241] Step 8: Send the result

[1242] The server sends the generated judgment result (input) to the terminal, compressing and encrypting the data if necessary, again using a secure protocol (e.g., TLS or SSL) and AES encryption. The encrypted judgment result data is sent to the terminal as output.

[1243] Step 9: Emotion Recognition

[1244] The device or server analyzes the user's voice and facial expressions (input) to recognize their emotional state. Specifically, it uses the device's camera and microphone to capture facial expressions in real time using OpenCV and an emotion analysis model based on voice analysis. The output is the user's emotional state (e.g., stress level, excitement, calm).

[1245] Step 10: Adjust notifications based on emotion

[1246] The server combines the judgment results with the emotion recognition results (input) to select the optimal notification method for the user. Specifically, if the stress level is high, a notification method that emphasizes urgency and specificity is selected. The output is the adjusted notification content and format.

[1247] Step 11: Check the results

[1248] The user confirms the adjusted notification content (input) through the terminal. Notifications are provided through voice, text, graphical alerts, etc. The output allows the user to acknowledge and respond to the judgment results.

[1249] Step 12: View detailed data and take action

[1250] The user can view the detailed voice data analysis results and emotion recognition results (input) and take appropriate countermeasures. For example, if an abnormal sound is detected and the stress level is high, the user can quickly consider replacing the bearings. As an output, appropriate countermeasures are taken and the equipment continues to operate normally.

[1251] (Application example 2)

[1252] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1253] In modern factories, it is extremely important to quickly detect abnormal machine sounds and perform maintenance at the appropriate time. However, conventional acoustic detection systems do not take the user's emotional state into consideration when notifying them of the detection results, which can lead to problems such as increased stress. In addition, there is a need for a comprehensive system that not only detects abnormal machine sounds but also efficiently monitors the entire acoustic environment.

[1254] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving voice data transmitted from the terminal and performing noise reduction and feature extraction, means for performing voice pattern analysis using a pre-trained AI model based on the features extracted by the server, means for determining the presence or absence of abnormal sounds or specific patterns based on the analysis results and generating a determination result, means for transmitting the determination result to the terminal and notifying the user, means for recognizing the user's emotional state using a camera and image recognition, and means for dynamically adjusting the notification method to the user depending on the emotional state. This makes it possible not only to accurately detect abnormal sounds from machines but also to provide appropriate notifications taking the user's emotional state into consideration.

[1255] A "terminal" is a device equipped with a highly sensitive microphone that collects audio data in real time.

[1256] The "server" is a device that receives voice data sent from the terminal, removes noise and extracts features, and analyzes the data to determine whether or not there are any abnormal sounds or specific patterns.

[1257] A "high-sensitivity microphone" is a microphone that can collect sound with high accuracy.

[1258] "Noise reduction" is the process of removing unnecessary background noise from collected audio data.

[1259] "Feature extraction" is the process of extracting important patterns and features from collected voice data.

[1260] An "AI model" is a model that uses artificial intelligence to analyze data and is trained to identify specific patterns.

[1261] "Voice pattern analysis" is a process that uses an AI model to analyze patterns in voice data and determine whether there are any abnormal sounds or specific patterns.

[1262] The "determination result" is a result regarding the presence of an abnormal sound or a specific pattern, which is generated based on the sound pattern analysis.

[1263] A "camera" is a device for taking still images and videos.

[1264] "Image recognition" is a technology that analyzes images taken with a camera and recognizes specific objects or situations.

[1265] "Emotional state" refers to the user's emotional state, such as stress level, excitement, or calmness.

[1266] "Notification method" refers to the means or format for conveying information or warnings to the user.

[1267] A specific system for implementing the present invention will now be described.

[1268] System Overview

[1269] This system is configured so that a device equipped with a highly sensitive microphone collects voice data in real time and sends it to a server. The server then removes noise from the voice data, extracts features, and analyzes the voice patterns using an AI model. Based on the analysis results, it determines whether there are any abnormal sounds or specific patterns, and dynamically adjusts the notification method taking into account the results of the analysis and the user's emotional state.

[1270] Specific processing method for the terminal

[1271] The device is equipped with a highly sensitive microphone that collects audio data in real time from specific sound sources, such as machinery sounds in a factory. The collected audio data is converted from analog to digital format and then filtered to reduce background noise and emphasize frequency bands that are useful for detecting abnormal sounds. After preprocessing, the audio data is packetized and sent to a server using a secure protocol.

[1272] Specific server processing method

[1273] The server receives the voice data sent from the device and temporarily stores it in storage. Next, it checks for errors while verifying the data's consistency. After that, it loads the trained AI model and extracts features (frequency spectrum, amplitude, phase, etc.) from the voice data. It performs voice pattern analysis based on the features and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match, thereby determining whether there is an abnormality or the presence of a specific pattern. The judgment result is generated as "no abnormality," "abnormality present," or "specific abnormal sound pattern present." The server then uses a camera and image recognition technology to recognize the user's emotional state from facial expressions, etc. The recognized emotional state is combined with the judgment result to select the most appropriate notification method and send it to the device.

[1274] Specific user operations

[1275] The user can check the results of the assessment through their device. Notifications are dynamically adjusted according to the user's emotional state; for example, if the user is in a high stress state, a more pronounced warning sound or visual alert is displayed. Based on the analysis results and the evaluation of the emotional state, support information is provided to take prompt and appropriate action if the user is in a high stress state. The user can then perform maintenance or repairs on the machine as needed.

[1276] Hardware and software used

[1277] Hardware used:

[1278] High-sensitivity microphone

[1279] Terminal (for collecting voice data)

[1280] Server (for data analysis and judgment)

[1281] Camera (for emotion recognition)

[1282] Software used:

[1283] pyaudio (audio data collection)

[1284] socket (data transmission)

[1285] Keras (AI model for analyzing voice patterns)

[1286] dlib (image recognition and emotional state recognition)

[1287] Specific examples

[1288] For example, if this system is used to detect abnormal sounds from machinery operating in a factory, a highly sensitive microphone collects the sounds and sends the data to a server. The server analyzes the data, and if an abnormal sound is detected, the results are sent to the terminal. At the same time, the camera analyzes the operator's facial expression, and if the user is in a state of high stress, an exaggerated warning sound or visual alert is displayed. This notification method takes the user's emotional state into account, enabling a quick and appropriate response.

[1289] Prompt Sentence Examples

[1290] "Write a Python program to help a factory robot collect mechanical sounds and detect abnormal sounds. Use a highly sensitive microphone to collect the audio in real time, and then convert and filter the data digitally before sending it to a server."

[1291] "Write a Python program that uses an AI model to analyze voice data to detect anomalous patterns, and then notifies the user based on the results. The notification content should dynamically adjust based on the user's emotional state (stress level, calmness)."

[1292] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1293] Step 1:

[1294] The terminal collects machine sounds within the factory in real time using a highly sensitive microphone.

[1295] Specifically, a high-sensitivity microphone installed in the terminal collects analog audio data from specific sound sources within the factory, and this collected audio data becomes the input for the next processing step.

[1296] Step 2:

[1297] The analog voice data collected by the terminal is converted into digital format and filtered.

[1298] The input is analog audio data, which is converted to a digital signal, then filtered to reduce background noise and highlight abnormal sounds, and this digital audio data is the input for the next processing step.

[1299] Step 3:

[1300] The terminal converts the preprocessed voice data into packets and transmits them to the server.

[1301] The input is pre-processed digital audio data, which is then packetized and sent to the server using a secure protocol (e.g., TLS), where the packetized data is the input for processing.

[1302] Step 4:

[1303] The server receives the audio data, temporarily stores it in storage, and performs error checking.

[1304] The input is packetized voice data sent from the terminal, which the server receives and stores, and performs error checks to ensure data integrity. This temporarily stored data becomes the input for the next analysis step.

[1305] Step 5:

[1306] The server extracts features (frequency spectrum, amplitude, phase, etc.) from the stored audio data.

[1307] The input is stored digital audio data, and an AI model is used to extract features from the audio data, which then become the input for the next step of audio pattern analysis.

[1308] Step 6:

[1309] The server analyzes the audio pattern using an AI model based on the features and determines whether there are any abnormal sounds or specific patterns.

[1310] The input is the extracted features, which are then input into the AI ​​model for analysis. The analysis results determine whether there are any abnormal sounds or specific patterns, and a judgment result is generated. This judgment result becomes the input for the next notification adjustment step.

[1311] Step 7:

[1312] The server uses a camera to recognize the user's emotional state.

[1313] The input is image data of the user collected by a camera, and image recognition technology is used to analyze the user's emotional state (stress level, calmness, etc.). This analyzed emotional state becomes additional input for the next notification adjustment step.

[1314] Step 8:

[1315] The server dynamically adjusts the notification method based on the judgment result and the user's emotional state.

[1316] The inputs are the analyzed emotional state and the judgment result, and these are combined to select the optimal notification method. For example, if the user is in a high stress state, an enhanced warning sound or visual alert is selected. This adjusted notification method becomes the input for the next notification sending step.

[1317] Step 9:

[1318] The server transmits the adjusted notification method and the determination result to the terminal.

[1319] The input is the adjusted notification method and the judgment result, which is sent to the terminal. The sent notification becomes the input for the next user confirmation step.

[1320] Step 10:

[1321] The terminal notifies the user, and the user confirms the judgment result.

[1322] The input is the notification method and judgment result sent from the server, which the device notifies to the user. The user can take appropriate measures or actions based on the displayed judgment result and notification.

[1323] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1324] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1325] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1326] [Fourth embodiment]

[1327] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1328] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1329] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1330] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1331] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1332] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1333] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1334] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1335] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1336] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1337] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1338] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1339] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1340] To implement this invention, the terminals, servers, and programs installed on them that constitute the Acoustic Discrimination Support System (ADSS) are required. Specific processes performed by the server, terminals, and users are described below.

[1341] Specific processing of the terminal

[1342] Audio data collection

[1343] The device uses a highly sensitive microphone to collect audio data in real time, allowing it to accurately capture data from specific sound sources, such as a vehicle engine or machinery on a production line.

[1344] Digital Conversion and Filtering

[1345] The collected audio data is converted from analog to digital format and then denoised and filtered, which suppresses background noise and emphasises the desired frequency bands.

[1346] Sending data

[1347] The pre-processed audio data is sent over the network to a server, where it is compressed and encrypted for secure and efficient transmission.

[1348] Server specific processing

[1349] Data Receipt and Storage

[1350] The server receives the voice data sent from the device and stores it in temporary storage, verifies the integrity of the data, and prepares it for analysis.

[1351] Feature extraction and analysis preparation

[1352] The server loads the trained AI model and extracts features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for voice pattern analysis.

[1353] Voice pattern analysis and judgment

[1354] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns. Based on the results, it is determined whether an abnormality exists or whether a specific pattern exists.

[1355] Generation and notification of judgment results

[1356] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The generated judgment result is sent to the terminal and notified to the user.

[1357] User specific processing

[1358] Check the results

[1359] The user can check the results of the assessment through their device. For example, if an abnormality is detected, the user is notified by an alert sound or a warning icon.

[1360] View detailed data and take action

[1361] If necessary, the user can check the detailed audio data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, machine maintenance, part replacement, or further inspection can be carried out.

[1362] Example: Vehicle engine sound inspection

[1363] Device behavior

[1364] A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time. The collected audio data is converted into a digital format, and then noise reduction and filtering are performed. The pre-processed data is then sent to a server.

[1365] Server Operation

[1366] The server temporarily stores the received voice data and extracts features from the voice data. The extracted features are input into an AI model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the device.

[1367] User Actions

[1368] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, the user takes measures such as replacing the bearings.

[1369] In this way, acoustic inspection work can be carried out quickly and accurately. This system significantly improves the efficiency and reliability of acoustic inspections, and also solves the problems of labor shortages and training costs.

[1370] The processing flow will be explained below.

[1371] Step 1:

[1372] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or factory machinery.

[1373] Step 2:

[1374] The device converts the collected voice data from analog to digital signals, making it in a format that can be analyzed.

[1375] Step 3:

[1376] The device then filters the digital audio data, reducing background noise and emphasizing important frequency bands.

[1377] Step 4:

[1378] The terminal then packetizes the pre-processed voice data, encrypts it, and sends it to the server, ensuring secure data transmission.

[1379] Step 5:

[1380] The server receives the voice data sent from the device and temporarily stores it in storage. It also performs error checking to ensure data integrity.

[1381] Step 6:

[1382] The server loads the AI ​​model and prepares it for analysis, using a pre-trained model.

[1383] Step 7:

[1384] The server extracts features from the audio data, specifically, frequency spectrum, amplitude, phase, etc.

[1385] Step 8:

[1386] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[1387] Step 9:

[1388] The server generates a judgment result based on the analysis results, such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[1389] Step 10:

[1390] The server generates a judgment result and sends it to the terminal, compressing and encrypting the data as necessary.

[1391] Step 11:

[1392] The terminal displays the result of the judgment received on the user interface, for example, by notifying the user using an alert sound or a warning icon.

[1393] Step 12:

[1394] The user checks the judgment results displayed on the device and, if necessary, can also view detailed voice data analysis results.

[1395] Step 13:

[1396] The user takes appropriate measures based on the judgment results. For example, if an abnormal sound is detected, the user performs machine maintenance or replaces parts.

[1397] Example 1

[1398] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1399] Conventional acoustic detection systems lack the ability to collect acoustic data in real time and analyze it quickly and accurately, making it difficult to distinguish between normal and abnormal movement patterns. They also lacked the functionality to efficiently determine the presence or absence of abnormal sounds or specific patterns, resulting in delays in providing appropriate notifications and feedback to users. Furthermore, the entire process, from collecting acoustic data to analyzing it and notifying users, was not integrated, resulting in low efficiency for the entire system.

[1400] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1401] In this invention, the server includes a terminal equipped with a highly sensitive sensor that collects acoustic data in real time, a processing device that receives the acoustic data transmitted from the terminal and performs noise removal and feature extraction, means for performing acoustic pattern analysis using a pre-trained machine learning model based on the features extracted by the processing device, means for determining the presence or absence of abnormal acoustics or specific patterns based on the analysis results and generating a determination result, and means for transmitting the determination result to the terminal and notifying a user. This makes it possible to collect acoustic data in real time, analyze it quickly and accurately, efficiently determine the presence or absence of abnormal acoustics or specific patterns, and provide appropriate notification and feedback to the user.

[1402] "Acoustic data" is information that is generated by converting sound into electrical signals and storing or processing them in digital form.

[1403] A "high-sensitivity sensor" is an acoustic detection device capable of detecting minute acoustic changes with high precision.

[1404] A "terminal" is a device that collects acoustic data and sends it to a server for processing.

[1405] A "processor" is a computer system that has the computational power to analyze acoustic data.

[1406] A "machine learning model" is an algorithm that is trained on large amounts of data to perform pattern recognition and prediction.

[1407] "Acoustic pattern analysis" is a process that identifies abnormalities and specific patterns based on the features of acoustic data.

[1408] "Abnormal sound" is a sound that is different from normal operating sounds and indicates an abnormality.

[1409] The "determination result" is diagnostic information generated based on the analysis of the acoustic data.

[1410] A "user" is a person who operates the system and checks the judgment results.

[1411] "Noise reduction" is a process for removing unnecessary background sounds from audio data.

[1412] "Feature extraction" is the process of extracting important patterns and properties from acoustic data.

[1413] "Real-time" refers to the simultaneous collection and processing of acoustic data.

[1414] MODE FOR CARRYING OUT THE INVENTION

[1415] To implement this invention, an Acoustic Decision Support System (ADSS) is required, which includes the following components: ADSS is realized by the roles of terminal, server, and user.

[1416] Terminal configuration and operation

[1417] Acoustic data collection

[1418] The device is equipped with highly sensitive sensors that collect acoustic data in real time from specific sound sources, such as a vehicle engine or a machine on a production line, with high accuracy. The device also includes location and time information to provide context for the collected acoustic data.

[1419] Digital Conversion and Filtering

[1420] The collected acoustic data is in analog format and is converted to digital format using an ADC (analog-to-digital converter) within the device. It is then denoised and filtered using audio processing software such as Audacity, which removes unwanted background noise and emphasises the desired frequency bands.

[1421] Sending data

[1422] The pre-processed digital audio data is then sent to the server via the network. During data transmission, the data is compressed and encrypted using gzip, TLS / SSL, etc., for safe and efficient transmission.

[1423] Server configuration and operation

[1424] Data Receipt and Storage

[1425] The server receives the acoustic data sent from the device and stores it in temporary storage (e.g., Amazon S3). It checks the integrity of the data and verifies that it is not corrupted. This temporary storage makes the data easily available for subsequent processing.

[1426] Feature extraction and analysis preparation

[1427] The server loads a trained machine learning model (e.g., TensorFlow or PyTorch) and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data, which is then ready for analysis.

[1428] Acoustic pattern analysis and judgment

[1429] The server uses the extracted features to perform acoustic pattern analysis. The features are input into a machine learning model, and the degree of match is calculated by comparing normal and abnormal sound patterns. Based on the results of this analysis, it determines whether an abnormality exists or whether a specific pattern exists.

[1430] Generation and notification of judgment results

[1431] The server generates a judgment result based on the analysis results. The result is displayed in the form of "No abnormality," "Abnormality detected," "Specific abnormal sound pattern detected," etc. These judgment results are sent to the terminal and notified to the user.

[1432] User Configuration and Behavior

[1433] Check the results

[1434] The user can check the results of the assessment through their device. For example, if an abnormality is detected, an alert sound or a warning icon will be displayed to alert the user.

[1435] View detailed data and take action

[1436] If necessary, the user can check the detailed acoustic data analysis results and take appropriate measures based on the results. For example, if an abnormal sound is detected, the user can perform machine maintenance, replace parts, or perform further inspections.

[1437] Example: Inspecting vehicle engine noise

[1438] Device behavior

[1439] Highly sensitive sensors are installed in the vehicle's engine to collect engine sounds in real time. The collected acoustic data is then digitally converted and filtered before being sent to a server via a network.

[1440] Server Operation

[1441] The server temporarily stores the received acoustic data and extracts features from the acoustic data. The extracted features are input into a machine learning model and compared with normal engine sounds to determine whether there are any abnormalities. For example, a judgment result such as "abnormal bearing noise detected" is generated and sent to the terminal.

[1442] User Actions

[1443] The user (inspector) checks the notified judgment results and, if necessary, refers to the detailed analysis results. If an abnormality is detected, countermeasures such as replacing the bearings are implemented.

[1444] Examples of prompt statements

[1445] "Generate Python code to collect vehicle engine sounds in real time and detect abnormal sounds."

[1446] In this way, the Acoustic Decision Support System (ADSS) can quickly and efficiently carry out the entire process from collection to analysis and notification, improving the efficiency and reliability of acoustic inspections and resolving the problems of labor shortages and training costs.

[1447] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1448] Step 1:

[1449] Acoustic data collection

[1450] The device uses a highly sensitive sensor to collect acoustic data in real time from specific sound sources. For example, it can capture the sound of a vehicle engine or the machine noise of a production line. The input is an analog acoustic signal captured by the acoustic sensor, and the output is analog acoustic data. Specifically, the device's highly sensitive sensor is placed near the target sound source and continuously collects acoustic data.

[1451] Step 2:

[1452] Digital Conversion and Filtering

[1453] The device converts the collected analog acoustic data into digital format using a built-in analog-to-digital converter (ADC). Next, the converted digital acoustic data is processed using audio processing software such as Audacity to remove noise and perform filtering. The input is analog acoustic data, and the output is filtered digital acoustic data. Specifically, the audio data is converted into digital format using the built-in ADC, and then white noise and unwanted frequencies are removed using Audacity.

[1454] Step 3:

[1455] Sending data

[1456] The device sends the preprocessed digital audio data to a server over the Internet or a local network. The data is sent in a compressed format such as gzip and encrypted format such as TLS / SSL. The input is the filtered digital audio data, and the output is compressed and encrypted audio data packets. Specifically, the digital audio data is compressed and securely sent to the server using an encryption protocol.

[1457] Step 4:

[1458] Data reception and storage

[1459] The server receives the digital audio data packets sent from the device, decompresses and decrypts them, and stores the received data in temporary storage (e.g., cloud storage). The input is the compressed and encrypted audio data packets, and the output is the digital audio data stored in the storage. Specifically, the server decompresses and decrypts the received data packets before storing them in cloud storage.

[1460] Step 5:

[1461] Feature extraction and analysis preparation

[1462] The server loads a pre-trained machine learning model (e.g., TensorFlow or PyTorch) based on the stored acoustic data, and extracts features (frequency spectrum, amplitude, etc.) from the acoustic data. The input is the stored digital acoustic data, and the output is the extracted feature data. Specifically, the acoustic data is converted into a frequency spectrum, and amplitude and other important features are extracted.

[1463] Step 6:

[1464] Acoustic pattern analysis and judgment

[1465] The server inputs the feature data into the AI ​​model and analyzes whether there is an abnormality by comparing normal and abnormal acoustic patterns. The input is the feature data, and the output is the judgment result of whether there is an abnormality or a specific acoustic pattern. Specifically, the server inputs the feature data into the AI ​​model and analyzes whether the model detects an abnormal sound or a specific pattern.

[1466] Step 7:

[1467] Generation and notification of judgment results

[1468] The server generates a judgment result based on the analysis results. The judgment result is expressed in the form of "no abnormality", "abnormality detected", "specific abnormal sound pattern detected", etc. The generated judgment result is sent to the terminal and notified to the user. The input is the analysis result, and the output is a notification of the judgment result. Specifically, the server forms the judgment result and sends it to the terminal as a data packet.

[1469] Step 8:

[1470] Confirm the results and implement countermeasures

[1471] The user checks the judgment results through the terminal. If an abnormality is detected, an alert sound or warning icon is displayed, and the user can refer to the detailed analysis results and take appropriate measures. The input is the notification of the judgment results, and the output is the countermeasures to be implemented by the user. Specifically, the terminal displays an alert, and the user checks the detailed data before carrying out maintenance or repairs.

[1472] (Application example 1)

[1473] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1474] For autonomous vehicles, it is important to detect abnormalities in the vehicle early and respond quickly and appropriately. However, conventional methods require advanced expertise and expensive equipment to detect abnormal sounds, making it difficult for ordinary vehicle owners. Furthermore, even if abnormal sounds were detected, there was a lack of systems that could present specific countermeasures to users, resulting in many cases of delays in repairs and maintenance. For this reason, there is a need for a system that can automatically detect abnormal sounds, present countermeasures, and notify users in an easy-to-understand manner.

[1475] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1476] In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving the voice data transmitted from the terminal and performing noise reduction and feature extraction, and means for performing voice pattern analysis using a pre-trained generative AI model based on the features extracted by the server. This makes it possible to determine the presence or absence of abnormal sounds or specific patterns and generate a determination result. Furthermore, if an abnormal sound is detected based on the analysis result, it is possible to propose a detailed analysis result and countermeasures to the user. This allows the user to quickly identify vehicle abnormalities and take appropriate measures, significantly improving vehicle safety and operational efficiency.

[1477] A "high-sensitivity microphone" is a microphone that can accurately pick up very quiet sounds and subtle changes in sound.

[1478] "Terminal" refers to an electronic device that has the functionality to collect, convert, and transmit data.

[1479] "Audio data" refers to data in which sound waveforms are recorded and stored in digital format.

[1480] "Digital format" is a format in which analog signals are expressed as numbers.

[1481] "Filtering" is a process that removes unnecessary noise and interference from collected audio data and emphasizes only the necessary frequency bands.

[1482] A "server" is a computer system that receives data from other terminals or devices and provides functions such as analysis, storage, and transmission.

[1483] "Noise reduction" is a process that removes background noise and unnecessary sounds from audio data.

[1484] "Feature extraction" is the process of extracting useful features (such as frequency components and amplitude) from audio data.

[1485] A "generative AI model" is an artificial intelligence model that has been pre-trained using machine learning algorithms to perform a specific task (in this case, speech pattern analysis).

[1486] "Voice pattern analysis" is an analytical method that detects the presence of specific voice patterns (for example, abnormal sounds) based on the features of voice data.

[1487] The "determination result" is the result obtained by the analysis, and is information indicating whether or not there is an abnormal sound or a specific audio pattern.

[1488] A "cloud server" is a remote computer system that provides data and services over the Internet.

[1489] The "criteria set in advance by experts" are criteria or rules set in advance by experts for determining abnormal sounds or specific audio patterns.

[1490] To implement this invention, a system is required that collects audio data in real time, detects abnormal sounds, and notifies the user. This system is composed of a terminal, a server, and programs installed on these. Specific processes performed by the server, terminal, and user are described below.

[1491] Specific processing of the terminal

[1492] The device is equipped with a highly sensitive microphone and a mechanism for collecting voice data in real time. This voice data is captured in analog form and then converted to digital form. During this process, noise reduction and filtering are performed to suppress background noise and emphasize desired frequency bands. The pre-processed voice data is then securely and efficiently transmitted over the network to a cloud server.

[1493] Server specific processing

[1494] The server receives the voice data sent from the device and stores it in temporary storage. It checks the integrity of this data and prepares it for analysis. Next, it uses a pre-trained generative AI model to extract features (frequency spectrum, amplitude, etc.) from the voice data. These features are used for analysis. These features are then input into the AI ​​model, which compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. Based on the results, it determines whether there is an abnormality or the existence of a specific pattern, and generates a judgment result. If an abnormal sound is detected, it also generates detailed analysis results and information to suggest countermeasures to the user.

[1495] User specific processing

[1496] The user checks the results of the assessment through their device. If an abnormality is detected, an alert sound or a warning icon will be displayed to notify the user. The user can then review the detailed analysis results and take appropriate action according to the presented countermeasures. For example, if an abnormal sound is detected in the vehicle's engine, it may be recommended to replace the bearings or conduct further inspections.

[1497] Specific examples

[1498] For example, monitoring the engine noise of a self-driving vehicle might proceed as follows:

[1499] 1. A highly sensitive microphone is attached to the vehicle's engine to collect engine sounds in real time.

[1500] 2. Collected audio data is converted into digital form and then denoised and filtered.

[1501] 3. The preprocessed audio data is sent to the cloud server.

[1502] 4. The server receives the audio data, extracts features using the generative AI model, and analyzes whether there are any abnormal sounds.

[1503] 5. If an abnormal sound is detected, a judgment result such as "Abnormal bearing noise detected" and specific countermeasures are generated and notified to the terminal.

[1504] 6. The user should check the notified judgment result and take appropriate measures such as replacing the bearing.

[1505] Prompt Sentence Examples

[1506] "input:

[1507] Audio data: [0.1, 0.2, -0.1, ...]

[1508] output:

[1509] Verification result: Abnormal noise detected. Details: Abnormal engine bearing noise detected. Confidence level: 95%

[1510] In this way, an anomaly detection system using acoustic data will significantly improve vehicle safety and operational efficiency.

[1511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1512] Step 1:

[1513] The device uses a highly sensitive microphone to collect audio data in real time. The input is ambient sounds, such as the sound of a vehicle engine, which are captured through the microphone. These sounds are collected in analog format, and the output is analog audio data.

[1514] Step 2:

[1515] The terminal converts the collected analog voice data into a digital format and performs noise reduction and filtering. Analog voice data is used as input. Noise reduction removes unwanted background noise, and filtering emphasizes the desired frequency band. The output is pre-processed digital voice data.

[1516] Step 3:

[1517] The device sends preprocessed digital audio data to the cloud server. The input is the source data, which is encrypted and compressed before being sent over the network. The output is the audio data sent to the cloud server.

[1518] Step 4:

[1519] The server receives the voice data sent from the device and stores it in temporary storage. The received data is used as input. The server checks the integrity of the data and performs the storage process. The output is the temporarily stored digital voice data.

[1520] Step 5:

[1521] The server extracts features from the stored voice data. In this process, features such as frequency spectrum and amplitude required for the generative AI model are extracted. Temporarily stored digital voice data is used as input. The voice data is analyzed and features are extracted, resulting in feature data as output.

[1522] Step 6:

[1523] The server inputs the extracted features into the generative AI model and performs voice pattern analysis. The extracted feature data is used as input. The generative AI model has learned normal and abnormal voice patterns and performs analysis by comparing them with the feature data. The output is the analysis results.

[1524] Step 7:

[1525] Based on the analysis results, the server determines whether there are any abnormal sounds or specific patterns and generates a judgment result. The results of the voice pattern analysis are used as input. The judgment results include "no abnormality," "abnormality present," "specific abnormal sound pattern present," etc. The output is the judgment result data.

[1526] Step 8:

[1527] The server sends the generated judgment result to the terminal and notifies the user. The judgment result data is used as input. The judgment result is compressed and encrypted and sent to the terminal. The output is the judgment result received by the user through the terminal.

[1528] Step 9:

[1529] The user checks the judgment results through a terminal. The judgment results displayed on the terminal are used as input. If an abnormality is detected, the user is notified of the detailed analysis results and countermeasures, along with an alert sound and a warning icon. The output is the confirmed judgment results and proposed countermeasures.

[1530] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1531] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[1532] Specific processing of the terminal

[1533] Audio data collection

[1534] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[1535] Digital Conversion and Filtering

[1536] The collected audio data is converted from analog to digital, then filtered to reduce background noise and emphasize the desired frequency band.

[1537] Sending data

[1538] The pre-processed audio data is packetized and sent to the server using a secure protocol, where data is compressed and encrypted to ensure secure communication.

[1539] Server specific processing

[1540] Data Receipt and Storage

[1541] The server receives the voice data sent from the device and stores it in temporary storage. It checks for errors while verifying the integrity of the data.

[1542] Feature extraction and analysis preparation

[1543] The server loads the trained AI model and extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[1544] Voice pattern analysis and judgment

[1545] The extracted features are input into the AI ​​model, and the degree of match is measured by comparing them with normal voice patterns and abnormal sound patterns, which determines whether an abnormality exists or the existence of a specific pattern.

[1546] Generating judgment results

[1547] The server generates a judgment result based on the analysis results, which is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc.

[1548] Sending the judgment result

[1549] The judgment result is sent to the terminal, and the data is compressed and encrypted as necessary.

[1550] Specific processing of the emotion engine

[1551] emotion recognition

[1552] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. The emotion engine detects multiple emotional states, such as the user's stress level, excitement, or calmness.

[1553] Emotion-based notification adjustment

[1554] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, a notification that emphasizes a more serious warning will be sent.

[1555] User specific processing

[1556] Check the results

[1557] The user checks the results on their device, and notifications are dynamically adjusted based on the emotion engine's results, using audio and visual alerts.

[1558] View detailed data and take action

[1559] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the user's stress level is high, the user will be provided with support information to take prompt and appropriate action.

[1560] Examples: Vehicle engine sound inspection and user emotion recognition

[1561] Device behavior

[1562] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[1563] Server Operation

[1564] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[1565] User Actions

[1566] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[1567] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[1568] The processing flow will be explained below.

[1569] Step 1:

[1570] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line.

[1571] Step 2:

[1572] The device converts the collected voice data from analog to digital signals, putting the voice data into an analyzable format.

[1573] Step 3:

[1574] The device then performs a filtering process on the digital audio signal, reducing background noise and emphasizing important frequency bands.

[1575] Step 4:

[1576] The terminal packetizes the pre-processed voice data, compresses and encrypts the data, and then transmits it to the server.

[1577] Step 5:

[1578] The server receives the voice data sent from the device and temporarily stores it in storage. It checks the data for consistency and performs error checks.

[1579] Step 6:

[1580] The server loads the trained AI model and prepares it for analysis.

[1581] Step 7:

[1582] The server extracts features of the audio data (frequency spectrum, amplitude, phase, etc.).

[1583] Step 8:

[1584] The server inputs the extracted features into the AI ​​model and analyzes the voice pattern, comparing it with normal voice patterns and abnormal sound patterns to measure the degree of match.

[1585] Step 9:

[1586] Based on the analysis results, the server generates a judgment result such as "no abnormality," "abnormality present," or "specific abnormal sound pattern present."

[1587] Step 10:

[1588] The server sends the result of the judgment to the terminal and compresses and encrypts the data as necessary.

[1589] Step 11:

[1590] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, it evaluates their stress level, excitement, calmness, etc. based on their tone of voice and facial expressions.

[1591] Step 12:

[1592] The server combines the output information of the emotion engine with the judgment results and selects the most appropriate notification method for the user. For example, if the user is in a high-stress state, a high-level warning will be issued.

[1593] Step 13:

[1594] The device displays the judgment results and the emotion engine's evaluation results on the user interface, and the user is notified using audio and visual alerts.

[1595] Step 14:

[1596] The user can check the judgment results and the emotion engine's evaluation results displayed on the device. They can also refer to the detailed voice data analysis results.

[1597] Step 15:

[1598] The user can take appropriate measures based on the judgment results and the emotion engine's evaluation results. For example, if an abnormal sound is detected and the user's stress level is high, machine maintenance or part replacement will be carried out promptly.

[1599] Example 2

[1600] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1601] Conventional sound detection support systems did not take into account the user's emotional state and fixed the way and timing of notifications were received, which increased stress for the user. Furthermore, the security of audio data was not adequately ensured, which posed a risk of important data being leaked. Furthermore, because the criteria for determining abnormal sounds were fixed, flexible responses were difficult and the system relied on dedicated experts.

[1602] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for performing voice pattern analysis using a machine learning model, means for generating a determination result, means for recognizing the emotional state of the user by analyzing the user's voice and facial expressions, and means for dynamically adjusting the notification method based on the emotion recognition result. This allows the user to receive notifications at an appropriate time and in an appropriate manner without feeling stressed. In addition, data confidentiality is maintained by compressing and encrypting the data using a secure protocol before transmission. Furthermore, flexible and highly accurate voice analysis is possible by comparing the criteria for determining abnormal sounds with criteria set in advance by experts.

[1603] A "high-sensitivity microphone" is a sound collection device that has the ability to pick up even the faintest sounds.

[1604] A "terminal" is an electronic device that collects and digitally processes voice data.

[1605] "Digital format" refers to a format in which an analog signal is converted into digital code.

[1606] "Filtering" is a process for removing unnecessary noise from the original audio data.

[1607] "Noise reduction" is a technique for reducing or removing background noise from collected audio data.

[1608] "Feature extraction" is the process of extracting useful information or patterns from speech data.

[1609] A "machine learning model" is an algorithm that learns from large amounts of data and enables pattern recognition and prediction.

[1610] "Voice pattern analysis" is a technology that analyzes voice data to detect specific voice patterns and anomalies.

[1611] The "determination result" is information that indicates the presence or absence of an abnormality or a specific state generated as a result of the analysis.

[1612] "User" means an individual or organization that uses the system.

[1613] "Emotional state" is a state that indicates a user's stress level and emotional response.

[1614] The "notification method" is the method for notifying the user of the judgment results and other information.

[1615] A "secure protocol" is a communication protocol that ensures confidentiality and integrity when sending and receiving data.

[1616] "Data compression" is a technique for reducing the volume of data.

[1617] "Encryption" is a technology that uses a specific algorithm to convert data into a format that cannot be deciphered by third parties.

[1618] The "criteria for determining abnormal sounds" are criteria for detecting abnormalities used in the sound analysis.

[1619] An "expert" is someone who has advanced knowledge and skills in a particular field.

[1620] This invention configures a system that dynamically adjusts the content and format of notifications to users by combining an emotion engine with an Acoustic Discrimination Support System (ADSS), and provides optimal support according to the user's emotional state. Specific processes performed by the server, terminal, and user are explained below.

[1621] Specific processing of the terminal

[1622] Audio data collection

[1623] The device uses a highly sensitive microphone to collect audio data in real time from specific sound sources, such as the sound of a vehicle engine or machinery on a production line. The collected audio data is temporarily stored in the device as an analog signal.

[1624] Digital Conversion and Filtering

[1625] The terminal converts the collected analog voice data into a digital signal using an Analog-to-Digital Converter (ADC). The signal is then filtered using an FIR filter to remove high-frequency noise and emphasize key frequency bands.

[1626] Sending data

[1627] The pre-processed audio data is packetized and sent to the server using a secure protocol (e.g., TLS or SSL), with algorithms such as GZIP used to compress the data and AES encryption used for encryption.

[1628] Server specific processing

[1629] Data Receipt and Storage

[1630] The server receives the voice data sent from the device and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks.

[1631] Feature extraction and analysis preparation

[1632] The server then loads a pre-trained machine learning model to extract features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data, often using MFCC (Mel-Frequency Cepstral Coefficients) and STFT (Short-Time Fourier Transform).

[1633] Voice pattern analysis and judgment

[1634] Based on the extracted features, the server inputs them into a machine learning model and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match. This method uses a convolutional neural network (CNN), which is good at analyzing time series data and spatial data of voice.

[1635] Generating judgment results

[1636] Based on the analysis results, the server generates a judgment result, which can be expressed as "no abnormality," "abnormality," "specific abnormal sound pattern," etc. For example, it can identify a specific abnormal sound, such as "abnormal sound in the engine bearing."

[1637] Sending the judgment result

[1638] The server sends the result to the device, compressing and encrypting the data as needed, again using TLS / SSL and AES encryption in the process.

[1639] Specific processing of the emotion engine

[1640] emotion recognition

[1641] The server or device analyzes the user's voice and facial expressions to recognize their emotional state. For example, it uses the device's camera and microphone to analyze the user's facial expressions and tone of voice. For analysis, OpenCV is used to capture the user's facial expressions in real time and input them into an emotion analysis model.

[1642] Emotion-based notification adjustment

[1643] The server combines the judgment results with the output information of the emotion engine to select the optimal notification method for the user. For example, if the user is in a high stress state, the server adjusts the notification method to be more specific and urgent. In addition to voice notifications, it also uses text notifications and graphical alerts.

[1644] User specific processing

[1645] Check the results

[1646] The user checks the results on their device. Notifications are dynamically adjusted based on the emotion engine's results, providing appropriate audible and visual alerts. For example, the device might display a message saying, "There is an abnormal noise in the engine bearings."

[1647] View detailed data and take action

[1648] Users can view the detailed voice data analysis results and the emotion engine's evaluation results, and take appropriate measures based on the results. For example, if an abnormal sound is detected and the stress level is high, specific instructions for considering replacing bearings will be displayed. This allows for quick and appropriate action.

[1649] Examples: Vehicle engine sound inspection and user emotion recognition

[1650] Device behavior

[1651] The device is equipped with a highly sensitive microphone attached to the vehicle's engine, which collects engine sounds in real time. The audio data is then converted into digital data, filtered, and sent to a server.

[1652] Server Operation

[1653] The server stores the received voice data in storage and extracts features. It uses an AI model to analyze the voice pattern and generate a judgment result such as "abnormal bearing noise detected." It then uses an emotion engine to recognize the user's emotional state. For example, it determines the user's stress level from their tone of voice and facial expression. The judgment result and emotional information are combined to select the optimal notification method and send it to the device.

[1654] User Actions

[1655] The user (inspector) checks the judgment results on the device and also refers to the evaluation results of the emotion engine. If the stress level is high, the device will alert the user with an emphasized warning sound or visual alert. The user can then take measures such as replacing the bearings.

[1656] In this way, by combining ADSS with an emotion engine, the efficiency and reliability of acoustic testing can be further improved, and more personalized assistance can be provided to users.

[1657] Example prompts for generative AI models

[1658] If you need an explanation of specific system operations or each piece of hardware and software, enter prompts like the following into the generative AI model:

[1659] Example prompt sentence:

[1660] "Please explain in detail the benefits of combining the Acoustic Judgment Support System (ADSS) with an emotion engine and the specific processing steps. Please also mention the specific processing performed by the terminal, server, and user, as well as the hardware and software used. Also, as a concrete example, please explain how it works in the case of a vehicle engine sound inspection."

[1661] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1662] Step 1: Collecting audio data

[1663] The device uses a high-sensitivity microphone to collect audio data in real time from a specific sound source (e.g., a vehicle engine). The input is an analog audio signal from the high-sensitivity microphone attached to the device. This analog audio data is temporarily stored within the device. The output is analog audio data.

[1664] Step 2: Digital Conversion and Filtering

[1665] The terminal converts the collected analog voice data (input) into a digital signal. This conversion is performed using an Analog-to-Digital Converter (ADC). The digital voice data is then filtered to reduce background noise and emphasize key frequency bands. Specifically, an FIR filter is used. The filtered digital voice data is output.

[1666] Step 3: Sending data

[1667] The device packets preprocessed digital audio data (input) and sends it to the server using a secure protocol (e.g., TLS or SSL). The data is compressed using an algorithm such as GZIP and encrypted using the AES cipher. The encrypted compressed data is sent to the server as output.

[1668] Step 4: Receiving and storing data

[1669] The server receives the encrypted compressed data (input) sent from the terminal and stores it in temporary storage. At this time, it checks the integrity of the data and performs error checks such as CRC checks. The output is the original stored digital audio data.

[1670] Step 5: Feature extraction and analysis preparation

[1671] The server loads a pre-trained machine learning model based on the stored digital audio data (input) and extracts features (e.g., frequency spectrum, amplitude, phase, etc.) from the audio data. This process uses MFCC and STFT. The output is the extracted audio data features.

[1672] Step 6: Voice pattern analysis and decision

[1673] The server inputs the feature-extracted voice data (input) into a machine learning model and compares it with normal voice patterns and abnormal sound patterns to measure the degree of match. Specifically, the analysis is performed using a convolutional neural network (CNN). The output is the analysis result (e.g., "normal," "abnormal," or "specific abnormal sound pattern present").

[1674] Step 7: Generate a decision result

[1675] The server generates a judgment result based on the voice pattern analysis result (input). The judgment result is expressed as "no abnormality," "abnormality detected," "specific abnormal sound pattern detected," etc. The judgment result to be notified to the user is generated as output.

[1676] Step 8: Send the result

[1677] The server sends the generated judgment result (input) to the terminal, compressing and encrypting the data if necessary, again using a secure protocol (e.g., TLS or SSL) and AES encryption. The encrypted judgment result data is sent to the terminal as output.

[1678] Step 9: Emotion Recognition

[1679] The device or server analyzes the user's voice and facial expressions (input) to recognize their emotional state. Specifically, it uses the device's camera and microphone to capture facial expressions in real time using OpenCV and an emotion analysis model based on voice analysis. The output is the user's emotional state (e.g., stress level, excitement, calm).

[1680] Step 10: Adjust notifications based on emotion

[1681] The server combines the judgment results with the emotion recognition results (input) to select the optimal notification method for the user. Specifically, if the stress level is high, a notification method that emphasizes urgency and specificity is selected. The output is the adjusted notification content and format.

[1682] Step 11: Check the results

[1683] The user confirms the adjusted notification content (input) through the terminal. Notifications are provided through voice, text, graphical alerts, etc. The output allows the user to acknowledge and respond to the judgment results.

[1684] Step 12: View detailed data and take action

[1685] The user can view the detailed voice data analysis results and emotion recognition results (input) and take appropriate countermeasures. For example, if an abnormal sound is detected and the stress level is high, the user can quickly consider replacing the bearings. As an output, appropriate countermeasures are taken and the equipment continues to operate normally.

[1686] (Application example 2)

[1687] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1688] In modern factories, it is extremely important to quickly detect abnormal machine sounds and perform maintenance at the appropriate time. However, conventional acoustic detection systems do not take the user's emotional state into consideration when notifying them of the detection results, which can lead to problems such as increased stress. In addition, there is a need for a comprehensive system that not only detects abnormal machine sounds but also efficiently monitors the entire acoustic environment.

[1689] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a terminal equipped with a high-sensitivity microphone that collects voice data in real time, means for receiving voice data transmitted from the terminal and performing noise reduction and feature extraction, means for performing voice pattern analysis using a pre-trained AI model based on the features extracted by the server, means for determining the presence or absence of abnormal sounds or specific patterns based on the analysis results and generating a determination result, means for transmitting the determination result to the terminal and notifying the user, means for recognizing the user's emotional state using a camera and image recognition, and means for dynamically adjusting the notification method to the user depending on the emotional state. This makes it possible not only to accurately detect abnormal sounds from machines but also to provide appropriate notifications taking the user's emotional state into consideration.

[1690] A "terminal" is a device equipped with a highly sensitive microphone that collects audio data in real time.

[1691] The "server" is a device that receives voice data sent from the terminal, removes noise and extracts features, and analyzes the data to determine whether or not there are any abnormal sounds or specific patterns.

[1692] A "high-sensitivity microphone" is a microphone that can collect sound with high accuracy.

[1693] "Noise reduction" is the process of removing unnecessary background noise from collected audio data.

[1694] "Feature extraction" is the process of extracting important patterns and features from collected voice data.

[1695] An "AI model" is a model that uses artificial intelligence to analyze data and is trained to identify specific patterns.

[1696] "Voice pattern analysis" is a process that uses an AI model to analyze patterns in voice data and determine whether there are any abnormal sounds or specific patterns.

[1697] The "determination result" is a result regarding the presence of an abnormal sound or a specific pattern, which is generated based on the sound pattern analysis.

[1698] A "camera" is a device for taking still images and videos.

[1699] "Image recognition" is a technology that analyzes images taken with a camera and recognizes specific objects or situations.

[1700] "Emotional state" refers to the user's emotional state, such as stress level, excitement, or calmness.

[1701] "Notification method" refers to the means or format for conveying information or warnings to the user.

[1702] A specific system for implementing the present invention will now be described.

[1703] System Overview

[1704] This system is configured so that a device equipped with a highly sensitive microphone collects voice data in real time and sends it to a server. The server then removes noise from the voice data, extracts features, and analyzes the voice patterns using an AI model. Based on the analysis results, it determines whether there are any abnormal sounds or specific patterns, and dynamically adjusts the notification method taking into account the results of the analysis and the user's emotional state.

[1705] Specific processing method for the terminal

[1706] The device is equipped with a highly sensitive microphone that collects audio data in real time from specific sound sources, such as machinery sounds in a factory. The collected audio data is converted from analog to digital format and then filtered to reduce background noise and emphasize frequency bands that are useful for detecting abnormal sounds. After preprocessing, the audio data is packetized and sent to a server using a secure protocol.

[1707] Specific server processing method

[1708] The server receives the voice data sent from the device and temporarily stores it in storage. Next, it checks for errors while verifying the data's consistency. After that, it loads the trained AI model and extracts features (frequency spectrum, amplitude, phase, etc.) from the voice data. It performs voice pattern analysis based on the features and compares them with normal voice patterns and abnormal sound patterns to measure the degree of match, thereby determining whether there is an abnormality or the presence of a specific pattern. The judgment result is generated as "no abnormality," "abnormality present," or "specific abnormal sound pattern present." The server then uses a camera and image recognition technology to recognize the user's emotional state from facial expressions, etc. The recognized emotional state is combined with the judgment result to select the most appropriate notification method and send it to the device.

[1709] Specific user operations

[1710] The user can check the results of the assessment through their device. Notifications are dynamically adjusted according to the user's emotional state; for example, if the user is in a high stress state, a more pronounced warning sound or visual alert is displayed. Based on the analysis results and the evaluation of the emotional state, support information is provided to take prompt and appropriate action if the user is in a high stress state. The user can then perform maintenance or repairs on the machine as needed.

[1711] Hardware and software used

[1712] Hardware used:

[1713] High-sensitivity microphone

[1714] Terminal (for collecting voice data)

[1715] Server (for data analysis and judgment)

[1716] Camera (for emotion recognition)

[1717] Software used:

[1718] pyaudio (audio data collection)

[1719] socket (data transmission)

[1720] Keras (AI model for analyzing voice patterns)

[1721] dlib (image recognition and emotional state recognition)

[1722] Specific examples

[1723] For example, if this system is used to detect abnormal sounds from machinery operating in a factory, a highly sensitive microphone collects the sounds and sends the data to a server. The server analyzes the data, and if an abnormal sound is detected, the results are sent to the terminal. At the same time, the camera analyzes the operator's facial expression, and if the user is in a state of high stress, an exaggerated warning sound or visual alert is displayed. This notification method takes the user's emotional state into account, enabling a quick and appropriate response.

[1724] Prompt Sentence Examples

[1725] "Write a Python program to help a factory robot collect mechanical sounds and detect abnormal sounds. Use a highly sensitive microphone to collect the audio in real time, and then convert and filter the data digitally before sending it to a server."

[1726] "Write a Python program that uses an AI model to analyze voice data to detect anomalous patterns, and then notifies the user based on the results. The notification content should dynamically adjust based on the user's emotional state (stress level, calmness)."

[1727] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1728] Step 1:

[1729] The terminal collects machine sounds within the factory in real time using a highly sensitive microphone.

[1730] Specifically, a high-sensitivity microphone installed in the terminal collects analog audio data from specific sound sources within the factory, and this collected audio data becomes the input for the next processing step.

[1731] Step 2:

[1732] The analog voice data collected by the terminal is converted into digital format and filtered.

[1733] The input is analog audio data, which is converted to a digital signal, then filtered to reduce background noise and highlight abnormal sounds, and this digital audio data is the input for the next processing step.

[1734] Step 3:

[1735] The terminal converts the preprocessed voice data into packets and transmits them to the server.

[1736] The input is pre-processed digital audio data, which is then packetized and sent to the server using a secure protocol (e.g., TLS), where the packetized data is the input for processing.

[1737] Step 4:

[1738] The server receives the audio data, temporarily stores it in storage, and performs error checking.

[1739] The input is packetized voice data sent from the terminal, which the server receives and stores, and performs error checks to ensure data integrity. This temporarily stored data becomes the input for the next analysis step.

[1740] Step 5:

[1741] The server extracts features (frequency spectrum, amplitude, phase, etc.) from the stored audio data.

[1742] The input is stored digital audio data, and an AI model is used to extract features from the audio data, which then become the input for the next step of audio pattern analysis.

[1743] Step 6:

[1744] The server analyzes the audio pattern using an AI model based on the features and determines whether there are any abnormal sounds or specific patterns.

[1745] The input is the extracted features, which are then input into the AI ​​model for analysis. The analysis results determine whether there are any abnormal sounds or specific patterns, and a judgment result is generated. This judgment result becomes the input for the next notification adjustment step.

[1746] Step 7:

[1747] The server uses a camera to recognize the user's emotional state.

[1748] The input is image data of the user collected by a camera, and image recognition technology is used to analyze the user's emotional state (stress level, calmness, etc.). This analyzed emotional state becomes additional input for the next notification adjustment step.

[1749] Step 8:

[1750] The server dynamically adjusts the notification method based on the judgment result and the user's emotional state.

[1751] The inputs are the analyzed emotional state and the judgment result, and these are combined to select the optimal notification method. For example, if the user is in a high stress state, an enhanced warning sound or visual alert is selected. This adjusted notification method becomes the input for the next notification sending step.

[1752] Step 9:

[1753] The server transmits the adjusted notification method and the determination result to the terminal.

[1754] The input is the adjusted notification method and the judgment result, which is sent to the terminal. The sent notification becomes the input for the next user confirmation step.

[1755] Step 10:

[1756] The terminal notifies the user, and the user confirms the judgment result.

[1757] The input is the notification method and judgment result sent from the server, which the device notifies to the user. The user can take appropriate measures or actions based on the displayed judgment result and notification.

[1758] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1759] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1760] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1761] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1762] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1763] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1764] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1765] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1766] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1767] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1768] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1769] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1770] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1771] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1772] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1773] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1774] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1775] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1776] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, i...

Claims

1. A terminal equipped with a highly sensitive microphone that collects voice data in real time; a server that receives the voice data transmitted from the terminal and performs noise removal and feature extraction; A means for performing voice pattern analysis using a pre-trained AI model based on the feature values ​​extracted by the server; a means for determining whether or not there is an abnormal sound or a specific pattern based on the analysis result and generating a determination result; means for transmitting the determination result to the terminal and notifying the user; A system including:

2. 10. The system of claim 1, wherein the terminal further comprises means for converting collected audio data into a digital format and for pre-processing the data through filtering.

3. The system according to claim 1 , wherein the server further comprises means for checking criteria for determining abnormal sounds against criteria preset by an expert.

Images