Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a device for recording user voice; means for transmitting the recorded voice to a server; the device is used for analyzing the audio data in the server and generating a voice characteristic file; the device is used for converting the voice into the text based on the voice characteristic file; means for setting synthetic speech parameters; means for assigning a unique identifier to the audio data; means for generating synthetic speech; and means for a user to download the generated audio data.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] In existing technologies, when users experience voice abnormalities (such as illness or temporary loss of voice), they cannot effectively transmit personal voice information, and the uniqueness and security of voice data cannot be guaranteed. Furthermore, existing voice synthesis and management systems struggle to achieve personalized voice feature restoration and commercial utilization of voice data, posing a risk of illegal use or forgery. Therefore, there is an urgent need for a system capable of efficiently recording, personalized synthesis, anti-counterfeiting identification, and convenient use of user voice data to address these issues. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides an information processing system, comprising: a device for recording user speech; a device for sending the recorded speech to a server; a device for parsing audio data on the server and generating a speech feature profile; a device for transing speech into text based on the speech feature profile; a device for setting parameters for synthesized speech; a device for assigning a unique identifier to the audio data; a device for generating synthesized speech; and a device for allowing users to download the generated audio data. Through the synergistic effect of the above devices, efficient and personalized recording, synthesis, identification, anti-counterfeiting, and flexible utilization of user speech are achieved, effectively enhancing the security and commercial application value of speech data.

[0005] "User voice" refers to the original sound information input by the user through a terminal device.

[0006] "Recording" refers to the process of collecting and storing the voice signals emitted by a user through devices such as microphones.

[0007] A "server" refers to a computer system that has the functions of receiving, processing, storing, and managing audio data.

[0008] "Audio data" refers to sound data files that can be digitally processed and are recorded by users and acquired through terminal devices.

[0009] A “voice feature profile” refers to a set of parameters that reflect the characteristics of a user’s voice, extracted by analyzing audio data.

[0010] "Transcription" refers to the process of automatically converting speech content into text information.

[0011] "Synthetic speech" refers to audio data that simulates a user's voice, generated through speech synthesis technology based on text and speech feature parameters.

[0012] "Synthetic speech parameters" refers to the settings used to adjust the audio effects of synthesized speech, such as sound quality, pitch, and speech rate.

[0013] A "unique identifier" is a unique code or digital signature assigned to each piece of audio data for the identification, anti-counterfeiting, and management of audio data.

[0014] "Downloading" refers to the process by which a user obtains generated audio data from a server via the internet and stores it on their local device. Attached Figure Description

[0015] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0016] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0017] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0018] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0019] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0020] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0021] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0022] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0023] Figure 9 This represents an emotion map that maps multiple emotions.

[0024] Figure 10 This represents an emotion map that maps multiple emotions.

[0025] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0026] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0027] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0028] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0029] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0030] First, let me explain the terminology used in the following instructions.

[0031] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0032] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0033] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0034] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0036] First Implementation Method

[0037] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0038] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0039] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0040] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0041] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0042] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0043] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0044] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0045] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0046] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0047] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0048] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0049] Example 1

[0050] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0051] Existing audio data acquisition and processing systems often require cumbersome manual operations and specialized knowledge from users, failing to efficiently and automatically complete the entire process from audio acquisition, data parsing, text transcription, personalized audio synthesis, and security protection. This results in low user efficiency, insufficient customization, and a high risk of audio data tampering or illegal use. Furthermore, current technologies struggle to balance high security and flexible application in areas such as exclusive identification of audio content, ownership authentication, and secure transfer among multiple users, leaving users with limited ways to effectively utilize and monetize their audio data.

[0052] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0053] In this invention, the server includes a device for automatically parsing and extracting features from user audio signals; a device for automatically transcribing text based on audio features using natural language processing technology; a device for automatically setting synthesis parameters and generating synthesized audio content through an artificial intelligence model; a device for automatically generating and attaching identification information codes to audio data and synthesized audio content; a device for displaying the entire audio content processing flow and progress on the user interface; a device for audio content security management and evidence storage; and a control device for enabling content circulation and rights transfer among multiple users. This allows for fully automated, integrated processing of user audio data from acquisition, uploading, analysis, transcription, synthesis, security identification, downloading to subsequent management and application, greatly improving operational efficiency and the exclusivity and security of audio data, providing an innovative system platform for the efficient circulation and personalized use of audio content.

[0054] "User" refers to an individual or organization that uses this system to perform operations such as audio recording, uploading, management, generation, and downloading.

[0055] "Electronic device" refers to an information processing device that can collect, store, and transmit audio data and communicate with a server, including but not limited to smartphones, computers, tablets, etc.

[0056] "Audio signal" refers to digital sound data output by the user, collected by electronic devices, and used for subsequent processing.

[0057] "Information processing device" refers to a server or data processing platform that has functions such as audio parsing, data storage, synthesis processing, and access control.

[0058] "Audio feature information" refers to the set of feature parameters that reflect the characteristics of a user's voice, extracted from audio signals through automatic analysis.

[0059] Natural Language Processing (NLP) technology refers to software or algorithms that automatically convert speech content in audio signals into text information.

[0060] "Physical parameters, emotional parameters, and speed" refer to attribute variables used to customize the performance of synthesized audio content, including timbre, pitch, speech rate, and emotional type.

[0061] "Identification information code" refers to an identifier that is automatically generated for audio data and synthesized content and is used for unique identification and security authentication.

[0062] "Deep learning model" refers to artificial intelligence technology that uses a multi-layer neural network structure to achieve data analysis, audio feature modeling, and synthesis generation.

[0063] "Artificial intelligence technology" refers to the general term for technologies that use machine learning, deep learning, neural networks, and other methods to automatically process and generate audio signals.

[0064] "Audio synthesis content" refers to electronic audio data with a specific performance style generated by artificial intelligence models using original audio features and control parameters.

[0065] The "user interface" refers to the graphical and operable display interface through which a system provides users with interactive information such as operation entry points, process progress, and content display.

[0066] "Evidence storage" refers to the secure storage of the process and results of audio content processing and identification information in the storage system of an information processing device in the form of data records.

[0067] "Content circulation" refers to the process by which audio content is digitally transferred, downloaded, shared, or its rights transferred among multiple users under system control.

[0068] "Control device" refers to a combination of hardware and software modules used to implement functional process control, access control, information management, and user operation response.

[0069] The embodiments of the present invention are described below.

[0070] This system consists of information processing devices (servers), electronic devices (terminals), and users who operate these devices. The system is primarily used for the entire process of automatically acquiring, uploading, parsing, extracting audio features, transcribing text, setting synthesis parameters, generating identification identifiers, generating synthesized audio, downloading, and managing user audio data.

[0071] Users can capture audio using a dedicated recording application (such as "voice recording software") on a terminal (such as a smartphone, tablet, or personal computer). The terminal's built-in microphone or an external microphone is used to record the user's raw audio signal. After recording, the terminal temporarily saves the digital audio data (such as WAV or MP3 format) locally and uploads the audio file along with relevant metadata information to a designated data interface on the server via the HTTP protocol.

[0072] After receiving the audio signal, the server calls audio recognition software or service modules (such as general "speech recognition API", "cloud automatic speech recognition service", etc.) to automatically parse the audio data. The server automatically uses acoustic feature analysis algorithms (such as the "librosa" open-source audio processing library or "timbre analysis module") to extract audio feature parameters that reflect the user's unique voice characteristics from the audio signal and saves them as audio feature information.

[0073] The server further utilizes natural language processing technologies (such as "automatic speech-to-text software" or "natural language understanding modules") to automatically convert the audio content into corresponding text information. Simultaneously, the server automatically sets parameters such as sound quality, tone, speed, and emotion based on audio feature information. These parameters serve as crucial control variables for subsequent audio synthesis, enabling better simulation of the user's own voice style.

[0074] The server combines audio feature information and text content, employing deep learning models and audio synthesis technology (such as based on "AI speech synthesis services" or "cloud-based TTS" modules) to automatically generate synthesized audio content that simulates the user's voice characteristics. For each piece of audio data and its synthesized content, the server automatically generates a unique identifier code using an internally configured algorithm, attaching this code to the data and its corresponding metadata tag to enhance the uniqueness and security of the content. All data, synthesized content, processing logs, and evidence records are centrally stored in the server's database system.

[0075] Users can select and manage generated synthesized audio and initiate download requests through the terminal application's interface. The server automatically verifies permissions and returns the required audio content or operation result to the terminal. After receiving the synthesized audio, the terminal automatically saves it and can play it using a local player. It can also be used in various business scenarios, such as conference speaking, voice content creation, audio copyright trading, and customized voice services. Simultaneously, the system supports the transfer of audio content permissions and compliant circulation among multiple users.

[0076] In terms of hardware, the terminal can be a smartphone (such as an Android or iOS platform device), a personal computer, a tablet, etc.; the server can be a network server with computing, storage, and AI capabilities. Examples of software used include: general recording applications (such as "VoiceRecorder" type applications), audio recognition and transcription APIs (such as "cloud-based automatic speech recognition service"), artificial intelligence speech synthesis APIs (such as "cloud-based TTS" service), feature extraction tools (such as audio analysis libraries such as "librosa"), unique identifier generation algorithm modules, and database management systems (such as MySQL, PostgreSQL, etc.).

[0077] Specific usage examples:

[0078] For example, before a meeting, a user records their voice using a recording app on their phone and uploads it to the system. The server parses the audio, automatically transcribes it into text, extracts acoustic features, and generates a unique identifier. The user can then select text content and use the AI ​​speech synthesis module to generate an audio presentation that matches their own voice. During the meeting, even if the user is unable to speak in person, they can automatically play the pre-generated synthesized presentation on their device, enabling them to speak virtually.

[0079] Here are some examples of prompts for generative artificial intelligence models:

[0080] Hello, my name is Tanaka Taro. Please use my voice to synthesize the following text: "Welcome to today's press conference."

[0081] Please generate audio content for 'The next topic is technological innovation' based on the voice style I uploaded.

[0082] Mimic my voice and convert 'Generative AI Models Have a Bright Future' into an audio file.

[0083] Through this system, users can efficiently and securely manage and utilize personalized audio data, enabling automated and intelligent processing and multi-scenario applications, greatly enhancing the exclusive value and market potential of audio content.

[0084] use Figure 11 The processing procedure is explained.

[0085] Step 1:

[0086] Users launch the recording application on their devices and click the "Start Recording" button. The device then activates its microphone, capturing the user's voice signal as digital audio data in real time and displaying the recording progress during recording.

[0087] Input: The user's voice signal.

[0088] Data processing and calculation: The terminal performs digital sampling and encoding of the user's voice.

[0089] Output: Audio files generated locally in the terminal (such as WAV or MP3 format).

[0090] Step 2:

[0091] When the user clicks the "Stop Recording" button, the terminal completes the audio acquisition and saves the audio file, recording duration, creation time, and other metadata information to local storage, and then displays a message "Recording Complete".

[0092] Input: Audio stream data from the recording.

[0093] Data processing and calculation: The terminal encapsulates the audio stream data into an audio file and records metadata.

[0094] Output: Local audio files and related metadata.

[0095] Step 3:

[0096] The terminal automatically or according to user instructions combines the audio file and metadata into an HTTP request and sends it over the network to the data interface specified by the server.

[0097] Input: Locally saved audio files and metadata.

[0098] Data processing and calculation: The terminal formats the audio file and metadata into an HTTP POST request.

[0099] Output: An upload request containing audio data and metadata.

[0100] Step 4:

[0101] The server receives requests from terminals, performs validity checks on the request content (such as user authentication, file type and size checks), and temporarily stores the audio files and related data in the server's storage system.

[0102] Input: Audio data and metadata from the HTTP request.

[0103] Data processing and calculation: The server parses the request content, verifies the data integrity, and stores the original audio.

[0104] Output: A copy of the original audio file on the local server and upload logs.

[0105] Step 5:

[0106] The server calls audio recognition software or APIs (such as general speech recognition services) to analyze the received audio files, automatically transcribes them into text, and extracts feature parameters such as timbre, intonation, and speaking speed from the audio signal through acoustic feature extraction algorithms to form audio feature information.

[0107] Input: Audio files located on the server.

[0108] Data processing and computation: The server performs speech recognition, feature parameter extraction, and text generation.

[0109] Output: Corresponding text data and audio feature information.

[0110] Step 6:

[0111] Based on audio feature information, the server automatically sets the control parameters required for audio synthesis (such as timbre, tone, speech rate, emotional state, etc.), and generates a unique identifier code through a unique identifier generation algorithm, which is then bound and stored with the audio data.

[0112] Input: Audio feature information.

[0113] Data processing and calculation: The server calculates the synthesis parameters based on the algorithm and generates a unique identification code.

[0114] Output: A dataset with identification information codes and synthesis parameters.

[0115] Step 7:

[0116] The server uses deep learning models or artificial intelligence speech synthesis engines to generate synthesized audio content that mimics the user's voice style based on text content, audio features, and synthesis parameters. The synthesized audio file is then associated with an identification code and saved to a database.

[0117] Input: text data, audio feature parameters, synthesis control parameters, and recognition information code.

[0118] Data processing and computation: The server performs AI audio synthesis, data integration, and file storage.

[0119] Output: Synthesized audio file and corresponding recognition information data.

[0120] Step 8:

[0121] The user selects the desired synthesized audio item in the terminal application interface and clicks the "Download" button. The terminal sends a download request to the server, along with the required audio recognition information code.

[0122] Input: User's selections and operation commands on the terminal.

[0123] Data processing and calculation: The terminal organizes HTTP download requests and sends them to the server.

[0124] Output: A download request containing the audio identifier.

[0125] Step 9:

[0126] After verifying the validity of the download request and the corresponding permissions, the server retrieves the target synthesized audio file from the database, transmits it to the terminal over the network, and records the download operation log.

[0127] Input: Download request and audio recognition code.

[0128] Data processing and calculation: The server performs permission checks, file retrieval, and data transmission.

[0129] Output: Synthesized audio file data sent to the terminal.

[0130] Step 10:

[0131] The terminal receives the synthesized audio file transmitted from the server, stores it in a specified local directory, and updates the audio list and status in the application interface. The terminal allows users to listen to the file immediately or recall it in other scenarios later.

[0132] Input: Synthesized audio data returned by the server.

[0133] Data processing and computation: terminal file storage, list updates, and audio media preparation.

[0134] Output: Locally usable synthesized audio files and interface update results.

[0135] Application Example 1

[0136] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0137] Existing speech recognition and authentication systems have shortcomings in terms of accuracy, security, and comprehensive utilization of individual voice characteristics and emotional information. For example, current technologies struggle to provide high-precision feature analysis, emotion recognition, and synthetic audio generation tailored to the different voice characteristics and emotional states of users, resulting in inadequate protection of the security, uniqueness, and application diversity of voice data. How to achieve high-precision analysis, authentication, emotion recognition, and unique identification of user voice information, and to efficiently, securely, and flexibly apply voice data to various scenarios, has become a pressing technical challenge in this field.

[0138] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0139] In this invention, the server includes a device for receiving and storing user voice information, a device for extracting features and recognizing emotions from the received voice using a generative artificial intelligence model, a device for formulating synthesis parameters and generating synthesized audio based on audio features and emotional information, and a device for assigning a unique identifier to the audio data and supporting encryption and security management. This enables accurate analysis and emotion recognition of user voice data, improves the security and uniqueness of voice data, enhances voice-based identity authentication and personalized synthesis capabilities, and supports the efficient use of voice data in various scenarios such as personal information protection and commercial applications.

[0140] "Biological information acquisition device" refers to hardware or software equipment used to collect biometric data (such as voice, fingerprint, face, etc.) from users. In this invention, it mainly refers to a device used to collect user voice information.

[0141] “Voice information” refers to audio data collected by users through biometric acquisition devices and stored digitally, which includes personal voice characteristics and content information.

[0142] "Information and communication network" refers to wired or wireless communication systems that support data exchange and transmission between different devices or systems, including network infrastructure such as the Internet and local area networks.

[0143] "Information processing device" refers to an electronic computer or server system used to receive, process, store, parse, and synthesize collected data.

[0144] "Audio information analysis device" refers to a software module or hardware device used to analyze, process, extract features, and recognize speech information received.

[0145] "Audio feature information" refers to the set of data extracted from speech information that reflects the speaker's acoustic characteristics, including parameters such as timbre, pitch, and speech rate.

[0146] "Text information" refers to readable content such as characters and sentences obtained after converting speech information through an audio information analysis device.

[0147] "Emotional analysis device" refers to software or hardware devices used to analyze the emotional states (such as happiness, calmness, anger, etc.) contained in a user's speech.

[0148] "Synthetic audio information generation parameters" refers to various parameters set based on audio feature information and emotional information, used to guide the generation of synthesized audio, such as timbre, intonation, and speech rate.

[0149] "Information identifier" refers to a data tag used to uniquely identify a certain audio data, including code, strings, etc. generated by encryption algorithms or uniqueness guarantee algorithms.

[0150] "Synthetic audio information generation device" refers to a software module or hardware device that generates audio data simulating the user's timbre and emotions based on the above parameters and content information.

[0151] "Internal storage device" refers to a storage medium used in information processing devices to store various audio data, feature information, text data, etc., such as hard disks and solid-state storage.

[0152] "Terminal" refers to a personal computing device operated by a user for collecting, receiving, storing, and playing audio information, such as a smartphone, tablet, or personal computer.

[0153] "Generative artificial intelligence models" refer to artificial intelligence algorithm models that can achieve audio content recognition, feature extraction, or synthesis by performing deep learning on input audio data.

[0154] "Encryption algorithm" refers to a mathematical algorithm that converts raw data into a specific format to ensure data privacy, integrity and security.

[0155] "Uniqueness guarantee algorithm" refers to an algorithm used to generate a globally unique identifier for each audio data, such as hash algorithm, timestamp and user parameter combination algorithm, etc.

[0156] This invention can be implemented in the following manner. A user installs a dedicated application on a terminal (such as a smartphone, tablet, or personal computer) and uses the terminal's built-in biometric acquisition device (such as a microphone) to collect voice information. The terminal uses an audio acquisition module (such as Android AudioRecorder or AVFoundation) to collect and save the user's voice information as a digital audio file (e.g., WAV or MP3 format), and sends it to a server via an information communication network (such as the Internet) using an encrypted data transmission protocol (such as HTTPS).

[0157] As an information processing device, the server receives voice information from the terminal and analyzes the received data using audio information parsing devices (such as Google Cloud Speech-to-Text, VOSK, and other speech recognition software running on the server), extracting audio feature information (such as timbre, pitch, and speech rate). Simultaneously, the server uses sentiment analysis devices (such as Tencent AI Lab's sentiment recognition API or DeepSpectrum) to perform sentiment analysis, identifying emotional elements in the speech, such as happiness, calmness, and tension.

[0158] Based on audio feature information and sentiment analysis results, the server determines the parameters for generating synthesized audio information (such as timbre, pitch, and speech rate). The server then uses a synthesized audio information generation device (such as Amazon Polly or Google Text-to-Speech synthesizers) to generate synthesized speech containing the user's timbre and emotional characteristics. The synthesized audio file is assigned a unique identifier by the server (such as a hash value generated using the SHA-256 algorithm) and is securely stored on the server's internal storage device (such as a local disk or cloud storage).

[0159] When a user needs synthesized audio, they can send a request through the terminal. The server will then transmit the generated audio information to the terminal via the network. The terminal can save the received audio information locally or play it directly through a media player, enabling various application scenarios such as personal authentication, virtual assistants, intelligent customer service, and the commercialization of speech synthesis.

[0160] In software implementation, this system can utilize server-side applications developed using programming languages ​​such as Java, Python, or Node.js. Databases can include MySQL, PostgreSQL, or MongoDB, and must support efficient management and retrieval of audio files and feature information. For hardware, both the server and terminals can utilize commercially available general-purpose computer equipment, smart devices, and audio acquisition and playback hardware.

[0161] Here's a specific application example: Users record their own voice through a smartphone app, for example, saying, "My name is Zhang Wei, please allow me to log in." The terminal uploads the recording to a server via the network. The server analyzes the audio information, extracting audio features and emotional information, such as recognizing it as "happy." Subsequently, the server uses the user's unique timbre and emotion to generate a synthesized audio clip, such as "Welcome everyone to our press conference!" Users can download this audio for purposes such as speeches or identity verification. If users commercialize their audio, they can sell it on the voice platform and generate revenue.

[0162] Examples of prompts for generative artificial intelligence models include:

[0163] Please use my tone and emotion to synthesize a simulated voice from the text "Hello everyone, this is the user's press conference, thank you very much for coming!"

[0164] Please say, "My name is Wang Qiang, and I would like to log in to the member system," and generate a profile using your voiceprint and emotion data for security authentication.

[0165] Using my voice, generate a highly excited and emotionally charged voice announcement: "Our new product is officially launched today!"

[0166] Please upload a self-introduction audio with a relaxed tone, and we will automatically generate a marketable NFT audio resource for you.

[0167] Based on my recording data, the system automatically identifies emotions and synthesizes style-matched voice files, which are then downloaded and saved.

[0168] As described above, this invention can enhance the security, personalization, emotional expressiveness, and commercial value of voice data, and is widely applicable to various application scenarios such as identity authentication, virtual interaction, and voiceprint commercialization.

[0169] use Figure 12 The processing procedure is explained.

[0170] Step 1:

[0171] Users open a dedicated application on their terminal, click the "Start Recording" button, and record voice information using the terminal's microphone. The input is the user's raw audio signal. The terminal converts the analog signal captured by the microphone into a digital audio file (such as WAV or MP3 format) and temporarily stores it in local memory. The output is a digitized audio file.

[0172] Step 2:

[0173] After the user finishes recording, they click the "Stop Recording" button. The terminal then invokes the application to read the locally stored audio file and sends the audio data to the server's specified API address via encrypted HTTPS network protocol. The input is the locally stored audio file, and the output is a network transmission request containing the audio file.

[0174] Step 3:

[0175] The server receives upload requests from clients, extracts audio files from the request body, and saves them to local server storage or cloud storage. The input is a network request containing audio data. The server verifies the format validity and integrity of the audio file and stores the audio data in a dedicated directory. The output is the audio file saved on the server and returns the upload result (such as success or failure information) to the client.

[0176] Step 4:

[0177] The server invokes an audio information parsing device (such as a generative artificial intelligence model) to process the received audio file. The input is the audio file stored on the server. The server performs preprocessing on the audio data, such as noise reduction and frame segmentation, and then extracts audio feature data (such as voiceprint, timbre, spectrum, etc.) through the generative artificial intelligence model, organizing this feature data into an audio feature profile. The output is an audio feature profile representing the characteristics of that user.

[0178] Step 5:

[0179] The server uses an audio feature profile to call a text transcription model (such as an automatic speech recognition system) to transcribe the audio content. The input consists of the audio feature profile and the raw audio data. The server analyzes the audio data, generates corresponding text information, and saves the results to a database. The output is the corresponding text statement.

[0180] Step 6:

[0181] The server uses a sentiment analysis device to determine the sentiment characteristics of received audio files. The input consists of the audio file and the extracted audio features. The server uses a model to analyze the sentiment characteristics of the speech content, determining emotions such as happiness, calmness, or anxiety. The results are added to the audio feature profile. The output is an audio feature profile with sentiment labels.

[0182] Step 7:

[0183] The server sets the generation parameters (such as timbre, intonation, and speech rate) for synthesized audio information based on an audio feature profile with emotional tags. The input is the audio feature profile incorporating emotional information. The server generates a detailed parameter configuration table for subsequent speech synthesis. The output is a set of parameters used for audio synthesis.

[0184] Step 8:

[0185] The server uses a uniqueness algorithm (such as a hash algorithm) to generate unique identifiers for the original audio data and the future synthesized audio data. The input consists of the audio file and user-related identification information. The server generates a unique code (such as a hash code or encrypted symbol) and binds it to the user data and storage location. The output is a unique identifier.

[0186] Step 9:

[0187] The server utilizes an audio synthesis system (such as the TTS module of a generative artificial intelligence model) to combine pre-generated text, audio features, and emotional parameters to produce synthesized audio files that reflect the user's personal characteristics and emotions. The inputs are text content, audio feature profiles, and parameter configuration tables. After data fusion and audio synthesis algorithms, the server outputs the generated synthesized audio data file.

[0188] Step 10:

[0189] The server saves the generated synthesized audio files to a database or file storage system and updates the audio resource directory. The input is the synthesized audio data and related metadata. The output is the complete audio file path or URL in the storage system.

[0190] Step 11:

[0191] The user selects the "Download Synthetic Audio" function in the terminal application. The terminal sends a request to the server, with the audio query request data as input. The server locates the corresponding audio resource and returns the synthesized audio file to the terminal. The output is the synthesized audio file received and stored locally on the terminal, which can then use a player to play it or perform further operations.

[0192] Step 12:

[0193] Users can perform various operations using locally stored audio data, including authentication, voice playback, and audio commercialization. Input consists of local audio data and user commands. The terminal uses the audio data in practical applications, outputting authentication results, audio playback effects, or market transaction information.

[0194] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0195] Example 2

[0196] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0197] In existing technologies, synthesized speech based on user voice struggles to accurately reflect the user's emotional characteristics, resulting in a lack of naturalness and personalization in the generated speech. Furthermore, the illegal use and forgery of audio data are increasingly serious problems, and the lack of effective unique identification and traceability methods makes it difficult to guarantee data reliability. Existing systems also fall short in terms of convenient user download and efficient use of audio data, failing to meet diverse practical application needs. Therefore, a new system is needed that can achieve emotion recognition, synthesis quality optimization, secure identification management, and efficient terminal distribution.

[0198] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0199] In this invention, the server includes a device for parsing biometric information acquired and transmitted by an information input device, a device for generating and managing attribute information and identification information, a device for optimizing information generation conditions based on emotional information, a device for imbuing biometric information and synthetic information with identification information for anti-counterfeiting and traceability, a device for producing synthetic information with emotions consistent with the user through the information generation device, and a device for efficiently distributing the synthetic information to the user's information terminal. This enables personalized natural speech synthesis driven by emotion recognition, while ensuring the security, uniqueness, and traceability of audio data, and improving the user's convenient access to and diverse utilization experience of synthesized speech in various application scenarios.

[0200] "Information input device" refers to a hardware or software device used to acquire and record a user's physiological or behavioral information, such as a microphone or camera on a terminal device.

[0201] "Bioinformatics" refers to personal digital data that includes a user's voice, images, physiological signals, and other personal information.

[0202] "Information processing device" refers to a data processing unit used to analyze, process, and manage input biological information, including servers, cloud platforms, etc.

[0203] "Attribute information" refers to various types of data extracted from biometrics that can describe user characteristics, including information such as timbre, tone of voice, and emotional state.

[0204] "Information transformation processing" refers to the process of transforming, synthesizing, or interpreting data based on attribute information, such as text-to-speech and sentiment analysis.

[0205] "Information generation conditions" refers to the parameters and rules set when generating synthetic information, including adjustments such as sound quality, speech rate, and emotion.

[0206] "Identification information" refers to the identification code or related data used to uniquely identify and verify the authenticity of biological and synthetic information.

[0207] "Information generation device" refers to a technical device that can generate synthesized information based on input parameters and conditions, such as a speech synthesis engine.

[0208] "Synthetic information" refers to digital data generated by an information generation device based on set conditions and attribute information, including synthesized speech and text.

[0209] "Information terminal" refers to hardware or software devices that allow users to access, receive, and utilize synthetic information, including smartphones, computers, etc.

[0210] "Recording media" refers to any physical or electronic medium capable of storing biological and synthetic information, such as hard drives and cloud storage.

[0211] "Emotional information" refers to data about a user's emotions or mood state identified from biometric information.

[0212] "Information prompting device" refers to a human-computer interaction interface that allows users to input prompts into generative artificial intelligence models, thereby enabling customized information generation.

[0213] "Generative AI models" refer to AI programs that can generate corresponding text, speech, or other digital content based on inputs such as prompts.

[0214] "Prompt statements" refer to information instructions entered by the user, which are natural language descriptions used to guide generative artificial intelligence models to generate the required content.

[0215] This invention can be implemented in the following ways.

[0216] This system includes an information input device, an information processing device, an information generation device, an information terminal, and a recording medium. The system also utilizes generative artificial intelligence models and software resources such as a prompt-based input interface. Through collaboration among users, terminals, and servers, it achieves highly reliable synthetic information generation and distribution driven by emotion recognition.

[0217] Users can directly record biometric information, such as voice data, containing personal emotions, through applications on information terminals (such as smartphones, tablets, or computers). Common hardware includes built-in microphones and cameras in smartphones. Applications can be dedicated recording apps or multi-functional human-computer interaction software.

[0218] The terminal temporarily stores the recorded biometric information locally as digital files (e.g., WAV or MP3 format), along with basic metadata such as recording time and user ID. The terminal then uploads this data to the server via a network (e.g., Wi-Fi or cellular network) using HTTP requests. The server is a Linux server or an equivalent high-performance computing platform, equipped with a database management system (e.g., relational database, cloud storage platform).

[0219] After receiving the data, the server automatically parses the biometric information using modules such as speech recognition (e.g., general speech-to-text APIs, open-source speech libraries, etc.) to extract text and perform parametric audio feature analysis. The parametric audio features are used to generate attribute information, including timbre, intonation, and speech rate.

[0220] Subsequently, the server invokes emotion recognition algorithms (such as emotion analysis software, signal processing APIs, etc.) to further analyze the emotional information in the biometric data. The identified emotional information is integrated into the attribute information and used for setting the synthesis parameters in the next step.

[0221] In the information synthesis generation stage, the server sets speech synthesis parameters, such as synthesized timbre, emotional tone, and speech rate, based on the extracted audio attribute information, emotional information, and additional conditions input by the user (such as prompts related to the generative artificial intelligence model). The server can call third-party speech synthesis engines (such as general speech synthesis APIs, modular speech generators, etc.) to produce the final synthesized information, such as speech files, text, or multimodal information with specified emotional tone.

[0222] Throughout the entire processing flow, the server generates unique identification information (such as UUID, information hash, NFT technology, etc.) for each piece of biometric and synthetic information, and writes it to a database or blockchain to achieve data traceability, anti-counterfeiting, and data asset management, thereby improving information credibility. All original and generated information will be securely stored in recording media such as local storage or cloud storage.

[0223] Users can download, replay, edit, or forward generated synthetic information on demand through the interface on the information terminal. In some application scenarios, users can also achieve customized information generation based on natural language commands by using the prompt input function of the generative artificial intelligence model. For example, users can directly input prompts on the interface to allow the system to generate synthetic speech containing specific emotions, content, or styles.

[0224] The system can utilize hardware including smartphones, tablets, desktop computers, servers, and storage devices. Available software includes a dedicated recording app, a speech recognition API, a sentiment analysis module, a speech synthesis module, a database system, cloud storage services, and generative artificial intelligence models.

[0225] Specific application examples:

[0226] Users record "Hello everyone, I'm very happy today" on their smartphones via an app. After uploading the recording to the server, the system automatically determines the emotion as "happy" and synthesizes a new voice message with a bright tone for users to download.

[0227] In a presentation scenario, if a user's voice suddenly becomes abnormal, the system can generate multiple self-introductions or speeches with natural emotions in advance, store them on the terminal, and recall them at any time to complete the presentation.

[0228] Users can also publish their own voice templates on the platform, which can be purchased by others through legal contracts for use in innovative content and commercial purposes.

[0229] Examples of prompts for generative artificial intelligence models include:

[0230] "Please say 'Welcome to our event' in my voice and with a cheerful tone."

[0231] "If I lose my voice during a speech, please explain how I can use synthesized speech to play the prepared speech content."

[0232] How can I sell my voice on the digital audio marketplace and allow others to generate voice content online after purchasing it?

[0233] use Figure 13 The processing procedure is explained.

[0234] Step 1:

[0235] Users initiate the recording function through the recording application on the terminal. The input is the user's voice signal, and the output is a locally saved voice data file. The terminal activates the microphone, captures the user's speech, and saves the recorded file (such as WAV format) to local storage when the user clicks "stop".

[0236] Step 2:

[0237] The terminal generates an HTTP request containing the audio file and related metadata (such as recording time, user ID, etc.) and uploads it to the server. The input is the local audio file and metadata, and the output is the request packet sent to the server. The terminal detects network status and uploads the audio data packet via the internet automatically or after user confirmation.

[0238] Step 3:

[0239] The server receives audio data and metadata uploaded by the terminal and archives them in the storage system. The input is an HTTP request packet from the terminal, and the output is the audio file and metadata stored in the server's file system and database. The server parses the request content and stores the audio and additional information in the specified paths and database tables, respectively.

[0240] Step 4:

[0241] The server uses a speech recognition engine to analyze the acquired audio files, transcribe the audio content into text, and extract audio features to generate attribute information. The input is the original audio file, and the output is text information along with attribute parameters such as timbre and intonation. The server calls APIs such as speech-to-text to decode the audio and return the text, while simultaneously obtaining an audio profile through signal processing algorithms.

[0242] Step 5:

[0243] The server further performs sentiment analysis on the audio or text content using a sentiment recognition module, extracting the user's emotional characteristics in the speech. The input consists of audio attribute information and transcribed text; the output is attribute parameters containing sentiment tags. The server analyzes speech rhythm, pitch, etc., and combines this with the text content to determine the emotional state, such as "happy" or "sad."

[0244] Step 6:

[0245] The server sets the parameters for synthesized speech based on extracted emotional and attribute information, as well as user-input prompts. Inputs include emotional tags, audio attributes, and prompts; output is a set of synthesized speech parameters. The server adjusts synthesis parameters such as timbre and speech rate according to the emotional type and combines them with the prompts to generate the final configuration.

[0246] Step 7:

[0247] The server assigns a unique identifier to audio data and attribute information for anti-counterfeiting and data traceability. The input is audio data and attribute information; the output is a data record containing a unique identifier. The server calls an identifier generation algorithm to generate and append a unique identifier to each record in the database.

[0248] Step 8:

[0249] The server invokes the speech synthesis module to generate synthesized speech with corresponding emotions and personality according to the set parameters and attribute information. The input includes text content, voice attributes, and emotion parameters, and the output is a synthesized audio file. The server processes the audio using the speech synthesis API to output audio that simulates the user's style and emotions.

[0250] Step 9:

[0251] The server stores the generated synthesized speech, unique identifier, and all related data on a recording medium, ready for distribution. The input is synthesized audio and metadata; the output is a complete audio asset record within a database and storage system. The server completes file archiving and database writing, ensuring information traceability and secure storage.

[0252] Step 10:

[0253] Users request to download or use synthesized speech through a terminal application. The input is the user's download request, and the output is the synthesized audio file downloaded to the terminal. The terminal sends a download request based on the user's command. After verifying the user's identity, the server transmits the audio and saves it locally on the terminal. Users can play the audio through a player or use it in real-world scenarios (such as speeches, audio market transactions, etc.).

[0254] Step 11:

[0255] Users can input prompts into the generative AI model interface to customize synthesized speech content to meet their needs. The input is the user's natural language prompt, and the output is the corresponding audio synthesized and returned by the server on demand. For example, a user can type "Please introduce the product to me in a confident and gentle tone" through the application interface, and the system will automatically match the voice style with emotional parameters to generate audio and return it to the user.

[0256] Application Example 2

[0257] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0258] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0259] In this invention, the server includes a device for acquiring user information, a device for sending the acquired information to an information processing device, a device for parsing the information data and generating feature information in the information processing device, a device for converting the information data into symbolic information based on the feature information, a device for identifying the user's emotional state from the information data and setting generation data attributes according to the identified emotional state, a device for assigning a unique identifier to the information data, a device for generating the generated data, and a device for enabling the user to access the generated data. This allows for accurate identification of the user's emotional state and automatic generation of personalized and natural responses based on that emotional state, improving user experience and service quality while ensuring data uniqueness and security.

[0260] "User information" refers to data provided by users to express their intentions, emotions, or needs, including audio, text, images, etc.

[0261] "Information processing device" refers to computer equipment or system that can analyze, process, and store received information data.

[0262] "Information data" refers to various types of user-related data content that are transmitted, processed, and stored in the system.

[0263] "Feature information" refers to the set of parameters extracted from information data that can characterize the attributes, state, or characteristics of the data itself.

[0264] "Symbolic information" refers to text, code, or other symbolic data that expresses user intent or content and is derived from feature information.

[0265] "Emotional state" refers to the psychological or emotional state of a user during communication, which is identified by analyzing the user's input data.

[0266] "Generated data attributes" refer to parameters set based on the identified emotional state and used to control the system's response behavior or data representation.

[0267] A "unique identifier" is a code or number that can uniquely identify a piece of information data or generated data, used for identification, prevention of tampering, and tracking of data.

[0268] "Generated data" refers to new data content automatically generated by the system after processing user input information, including audio, text, etc.

[0269] To better understand the present invention, the system of the present invention will be described in detail below with reference to specific embodiments.

[0270] This invention relates to a system that, through collaboration between an information processing device and a terminal device, enables emotion recognition and response generation based on user input. The system collects user information, performs emotion analysis via a server, sets attributes for generated data based on the emotional state, and ultimately generates and provides personalized data content to the user.

[0271] In this invention, the terminal can be an information terminal device such as a smartphone, tablet computer, or personal computer. The server can be a general-purpose computer or cloud computing environment configured with a high-performance processor and storage device. The server can run a Linux-based operating system, and the terminal can be based on mainstream platforms such as Android or iOS.

[0272] Specifically, users input personal information on the terminal through an application (such as a custom-developed app), for example, recording voice messages using the recording function. The terminal uses local audio recording software (such as MediaRecorder on Android or AVAudioRecorder on iOS) to digitize the audio data and temporarily store it locally. Subsequently, the terminal automatically sends the audio data to a server via the network. This transmission can be based on HTTP or HTTPS protocols.

[0273] After receiving audio data, the server calls an audio recognition engine (such as a general speech recognition API) to parse the audio data in real time, extracting feature information including pitch, speech rate, and volume, and organizing it into an audio feature file. The server then combines the audio features with the speech-to-text results to automatically convert the user's speech content into text information. The server also uses emotion recognition and analysis tools (such as a speech emotion recognition API or a general emotion analysis engine) to identify the emotional state expressed by the user in the input information, such as happiness, anger, and anxiety.

[0274] Once the user's emotional state is identified, the server automatically sets the attributes of the generated data (such as timbre, speech rate, and tone). The server assigns a unique identifier to each piece of information data (such as using the UUID algorithm) to ensure the uniqueness and traceability of the data. Subsequently, the server uses a speech synthesis engine (such as a general speech synthesis API) to generate natural, fluent, and emotionally consistent synthesized speech data based on the identified text, emotional state, and corresponding parameters.

[0275] Once the generated data (such as synthesized audio) is stored on the server, users can access the application again through their terminal devices. By clicking the "Download" button, they can download the generated data from the server and play and use it on their terminals. All of the above processes can be transparently implemented between the user and the server, providing users with an emotionally intelligent interactive service experience and improving user satisfaction.

[0276] During data processing, the server can utilize database services (such as relational databases) or cloud storage services to store and manage data. The entire system can be implemented using programming languages ​​such as Python and Java, as well as conventional web development frameworks.

[0277] Specific examples are as follows:

[0278] A user encountered a delivery delay issue on a food delivery app and recorded a complaint through the app's customer support function: "My food delivery is two hours late, I'm very angry." The recording was uploaded to the server. The server recognized the audio content as "angry" and then generated a synthesized voice with a firm tone and a relatively fast speaking speed: "We are very sorry for the inconvenience caused. We will handle your problem as soon as possible." The user can listen to the voice message after clicking download in the app.

[0279] Here are some examples of prompts for generative artificial intelligence models:

[0280] Please analyze the audio recordings uploaded by users, identify their emotions based on acoustic features such as pitch, speech rate, and rhythm, and automatically set the tone, timbre, and speed parameters for the synthesized speech. For example, if the user's emotion is "joy," please generate speech using a bright timbre and a faster speech rate; if it is "anger," use a firm and slightly faster tone.

[0281] The above methods can efficiently realize the generation, transmission and personalized response of emotion recognition-based data, ensure the security and uniqueness of user information, and improve the overall experience and satisfaction in service scenarios.

[0282] use Figure 14 The processing procedure is explained.

[0283] Step 1:

[0284] The user opens the application on the terminal and clicks the "Start Recording" button, speaking into the terminal's microphone. The input is the user's voice, and the output is an audio file stored locally on the terminal. The terminal uses the audio recording module to convert the analog sound signal into digital audio data (such as a WAV file) and saves it to local storage.

[0285] Step 2:

[0286] The terminal automatically reads locally saved audio files and converts them into binary data. The input is the audio file, and the output is the binary data packet to be uploaded. The terminal constructs an HTTP POST request, using the audio data as the request body, and transmits it over the network to the specified interface address on the server, thus completing the data upload.

[0287] Step 3:

[0288] The server receives HTTP requests from the terminal, parses the audio binary data within the request body, and saves it to a local temporary storage directory on the server. The input is an audio data stream, and the output is an audio file stored on the server. The server calls the file writing module to save the received audio data to disk.

[0289] Step 4:

[0290] The server calls an audio analysis module (such as a general speech recognition API) to parse the stored audio files, extracting feature information including pitch, speech rate, and intensity, and generating an audio feature file. The input is the audio file, and the output is the audio feature information. The server uploads the audio file to the analysis engine, obtains the analysis results, and stores them in a structured manner.

[0291] Step 5:

[0292] The server uses a speech-to-text engine to convert speech data in audio feature files into text content. The input is audio feature information, and the output is text data. The server decodes and segments the signals in the audio to obtain readable text, which is then saved to the database.

[0293] Step 6:

[0294] The server analyzes text content and audio features through a sentiment recognition module to identify and determine the user's emotional state (such as anger, joy, etc.). The input consists of text data and audio features, and the output is a sentiment label and sentiment score. The server extracts and matches features from various data points to achieve a multi-dimensional comprehensive score.

[0295] Step 7:

[0296] The server sets the data attribute parameters of the synthesized speech based on the recognized emotion tags, including timbre, speech rate, and intonation. The input is the emotion tag, and the output is the synthesized speech parameter configuration. The server matches and adjusts the emotions and parameters according to preset logic.

[0297] Step 8:

[0298] The server assigns a unique identifier to the current audio data. The input is an audio file or data object, and the output is audio data with a unique ID. The server calls the unique identifier generation module to generate a unique number and write it to the data record, achieving data tracking and tamper-proofing.

[0299] Step 9:

[0300] The server invokes the speech synthesis module, taking text, synthesis parameters, and emotion tags as input to generate personalized synthesized audio that matches the user's emotions. The input consists of text, parameters, and emotion tags; the output is the generated synthesized audio file. The server utilizes the speech synthesis API to output audio with corresponding tone and style based on set parameters.

[0301] Step 10:

[0302] When a user clicks the "Download Synthesized Audio" button in the terminal application, the terminal sends an HTTP request with a unique ID to the server. The input is the user's request, and the output is the audio data request packet. The terminal communicates with the server through a standard interface.

[0303] Step 11:

[0304] After receiving a request, the server searches its local database and storage system to find the synthesized audio data with the corresponding unique ID and returns it to the terminal. The input is the user request, and the output is the synthesized audio file. The server performs file querying, reading, and data stream transmission.

[0305] Step 12:

[0306] After receiving the synthesized audio file, the terminal saves it to local storage and plays it using a multimedia player, allowing the user to directly hear the voice information generated through emotion recognition and intelligent synthesis. The input is the synthesized audio file, and the output is playable audio content. The terminal completes the file storage and audio playback operations.

[0307] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0308] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0309] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0310] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0311] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0312] Second Implementation Method

[0313] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0314] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0315] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0316] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0317] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0318] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0319] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0320] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0321] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0322] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0323] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. Reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0324] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0325] Example 1

[0326] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0327] Application Example 1

[0328] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0329] Example 2

[0330] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0331] Application Example 2

[0332] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0333] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0334] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0335] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0336] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0337] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0338] Third Implementation Method

[0339] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0340] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0341] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0342] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0343] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0344] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0345] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0346] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0347] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0348] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0349] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0350] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0351] Example 1

[0352] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0353] Application Example 1

[0354] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0355] Example 2

[0356] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0357] Application Example 2

[0358] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0359] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0360] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0361] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0362] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0363] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0364] Fourth Implementation Method

[0365] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0366] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0367] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0368] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0369] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0370] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0371] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0372] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0373] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0374] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0375] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0376] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0377] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0378] Example 1

[0379] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0380] Application Example 1

[0381] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0382] Example 2

[0383] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0384] Application Example 2

[0385] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0386] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input from the user representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0387] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0388] Furthermore, the processing of the data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0389] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0390] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0391] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph (see reference) which serves as a specific mapping. Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0392] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0393] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0394] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0395] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and battery level, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps, for example, can be based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Brain Physiological Signal Analysis Systems for Emotions, Tokushima University, Doctoral Dissertation: https: / /

[0396] The map is generated using the index / / ci.nii.ac.jp / naid / 500000375379. In the left half of the emotion map, emotions belonging to the "response" region, where sensation is dominant, are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the "situation" region, where situational cognition is dominant, are arranged.

[0397] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0398] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0399] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0400] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0401] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0402] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0403] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0404] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0405] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0406] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0407] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0408] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0409] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0410] In addition, the following notes are provided in response to the above explanation.

[0411] Example 1

[0412] (Note 1)

[0413] An information processing system includes: a device for recording a user's audio signal via an electronic device; a device for transmitting the recorded audio signal to an information processing device via a communication network; a device for automatically parsing the audio signal and extracting audio feature information in the information processing device; a device for automatically converting the audio signal into text information using natural language processing technology based on the extracted audio feature information; a device for automatically setting at least one feature control variable such as physical parameters, emotional parameters, and speed for audio synthesis; a device for attaching identification information codes to the audio signal and its products; a device for generating synthesized audio content using artificial intelligence technologies such as deep learning models based on audio signal features and identification information; a device for downloading the generated audio content from the information processing device to the electronic device according to the user's operation; a device for displaying and controlling the entire audio content processing progress and acquisition operation on a user interface; a device for storing evidence of the association between the generated audio content and the identification information in the internal memory of the information processing device to prevent illegal use and for authentication; and a control device for supporting the circulation or transfer of rights of audio content among multiple users.

[0414] (Note 2)

[0415] According to the information processing system described in Appendix 1, the information processing device employs an audio recognition computing device or an audio recognition software module when automatically parsing or converting audio signals into text information.

[0416] (Note 3)

[0417] According to the information processing system described in Appendix 1, the identification information code attached to the audio signal and the generated audio content by the information processing device is automatically generated by an identification information generation algorithm implemented within the information processing device.

[0418] Application Example 1

[0419] (Note 1)

[0420] An information processing system includes: a device for recording user voice information using a biometric acquisition device; a device for transmitting the recorded voice information to an information processing device via an information communication network; a device within the information processing device for analyzing the received voice information using an audio information parsing device and generating audio feature information; a device for generating text information corresponding to the voice information based on the generated audio feature information; a device for the information processing device to determine parameters for generating synthesized audio information using the audio feature information and emotional information obtained by an emotional analysis device; a device for assigning a unique identifier to the voice information; a device for generating audio information based on the user's audio features and emotional features using a synthesized audio information generation device; a device for storing the generated audio information in an internal storage device of the information processing device; and a device for transmitting the audio information to a user terminal via an information communication network when the user terminal requests the generated audio information, and for the terminal to save or play the audio information.

[0421] (Note 2)

[0422] According to the information processing system described in Appendix 1, in the audio information analysis, a generative artificial intelligence model is adopted as the audio information parsing device.

[0423] (Note 3)

[0424] According to the information processing system described in Appendix 1, when assigning an information identifier to audio information, the information identifier is generated using an encryption algorithm or a uniqueness guarantee algorithm.

[0425] Example 2

[0426] (Note 1)

[0427] An information processing system includes: a device for acquiring and recording biological information via an information input device; a device for transmitting the recorded biological information to an information processing device; a device for parsing the biological information and generating attribute information in the information processing device; a device for performing information conversion processing based on the attribute information; a device for setting information generation conditions according to the attribute information and additional information; a device for assigning identification information to the biological information; a device for generating synthetic information based on the attribute information, information generation conditions, and information conversion processing results using an information generation device; a device for providing the generated synthetic information to a user's information terminal; a device for saving the biological information and synthetic information to a recording medium; a device for correcting the information generation conditions based on emotional information contained in the attribute information; and a device for inputting prompt statements to a generative artificial intelligence model via an information prompt device and instructing the generation of synthetic information.

[0428] (Note 2)

[0429] According to the information processing system described in Appendix 1, the parsing of the biological information is performed using a speech recognition processing device.

[0430] (Note 3)

[0431] According to the information processing system described in Appendix 1, the information processing device employs identification information generation processing when assigning identification information to biological information and synthetic information.

[0432] Application Example 2

[0433] (Note 1)

[0434] An information processing system includes: a device for acquiring user information; a device for sending the acquired information to an information processing device; a device for parsing information data and generating feature information in the information processing device; a device for converting information data into symbolic information based on the feature information; a device for identifying the user's emotional state from the information data and setting attributes for generated data according to the identified emotional state; a device for assigning a unique identifier to the information data; a device for generating generated data; and a device for enabling the user to access the generated data.

[0435] (Note 2)

[0436] According to the information processing system described in Appendix 1, a device utilizing an information parsing engine is used during the information data parsing process.

[0437] (Note 3)

[0438] According to the information processing system described in Appendix 1, when the information processing device assigns a unique identifier to information data, it applies an identifier generation processing device.

Claims

1. An information processing system, characterized in that, include: A device for recording user voice; A device for sending recorded audio to a server; A means for parsing audio data and generating a speech feature profile in the server; A device for transcribing speech into text based on speech feature profiles; A device for setting parameters of synthesized speech; A device for assigning a unique identifier to audio data; A device for generating synthesized speech; A device for allowing users to download generated audio data.

2. The information processing system according to claim 1, characterized in that, It also includes devices that use a speech recognition engine during the audio data parsing process.

3. The information processing system according to claim 1, characterized in that, It also includes means for the server to use an identifier generation algorithm when assigning identifiers to audio data.

Images