Utterance information processing device and utterance information processing method

Abstract

An utterance information processing device 10 includes: an acquisition unit 20 for acquiring utterance information D2 indicating utterance content of a person and information related to non-language information D3 indicating demeanor of the person at the time of utterance related to the utterance content; and a processing unit 30 for executing processing regarding the utterance information D2 and the information related to the non-language information D3, in order to obtain utterance character information D5, which is text indicating the utterance content of the person based on the utterance information D2 and the information related to the non-language information D3, by a generative AI model L generated by machine learning.

Description

Speech information processing device and speech information processing method

[0001] The present invention relates to an utterance information processing device and an utterance information processing method.

[0002] Conventionally, there has been proposed an utterance information processing device that generates utterance information indicating the content of a person's utterance and emotion information indicating the emotion of the person when uttering the utterance content. For example, Patent Literature 1 describes an utterance information processing device that generates meeting minutes data that records the content of speech made by attendees at a meeting and emotion data of the attendees in response to the content of speech. The emotion data is data that indicates the type of emotion of the attendees at each time.

[0003] Japanese Patent Application Laid-Open No. 2005-277463

[0004] In the minutes data generated by the speech information processing device described in Patent Literature 1, speeches of meeting attendees are associated with numerical values ​​indicating the type of emotion each attendee had at the time of the speech. In such cases, a user who checks the minutes data cannot always easily understand the state of the attendees at the meeting. Therefore, it is desirable for the speech information processing device to enable a user to easily understand the state of people when they are speaking.

[0005] One embodiment of the present invention has been made in consideration of the above, and aims to provide a speech information processing device and a speech information processing method that allow a user to easily understand the state of a person when speaking.

[0006] In order to achieve the above-mentioned object, a speech information processing device according to one embodiment of the present invention comprises an acquisition unit that acquires speech information indicating the content of a human utterance and information regarding non-verbal information indicating the state of the person when uttering the speech content, and a processing unit that executes processing on the speech information and information regarding the non-verbal information to obtain speech text information, which is text indicating the content of a human utterance, based on the speech information and information regarding the non-verbal information using a first generation AI model generated by machine learning.

[0007] In an utterance information processing device according to an embodiment of the present invention, processing is performed on the utterance information and information related to the non-verbal information to obtain utterance text information, which is text indicating the content of a human utterance, based on the utterance information and information related to the non-verbal information, using a first generation AI model generated by machine learning. With this configuration, the first generation AI model can generate utterance text information that expresses in sentences the human's behavior in response to the utterance content. This allows a user to easily understand the human's behavior when speaking.

[0008] Incidentally, one embodiment of the present invention can be described not only as an apparatus invention as described above, but also as a method invention as described below. These are essentially the same inventions, just in different categories, and have similar functions and effects.

[0009] That is, an utterance information processing method according to one embodiment of the present invention includes an acquisition step of acquiring utterance information indicating the content of a human utterance and information regarding non-verbal information indicating the state of the human at the time of uttering the utterance content, and a processing step of executing processing on the utterance information and information regarding the non-verbal information to obtain utterance text information, which is text indicating the content of a human utterance, based on the utterance information and information regarding the non-verbal information using a first generation AI model generated by machine learning.

[0010] According to one embodiment of the present invention, it becomes possible for a user to easily understand the state of a person when speaking.

[0011] 1 is a diagram for explaining an overview of an utterance information processing device according to an embodiment. (a), (b), and (c) are diagrams showing an example of an utterance frame and non-verbal information included in utterance information. (a) is a diagram showing an example of a prompt. (b) is a diagram showing an example of utterance text information. (c) is a block diagram showing an information processing system according to an embodiment. (a) and (b) are diagrams showing an example of processing for information related to non-verbal information. (a) is a flowchart showing an example of processing for utterance information and information related to non-verbal information in the utterance information processing method according to an embodiment. (b) is a flowchart showing an example of processing for utterance information and information related to non-verbal information in the utterance information processing method according to an embodiment. (c) is a flowchart showing an example of processing for utterance information and information related to non-verbal information in the utterance information processing method according to an embodiment. (c) is a diagram showing a hardware configuration of an utterance information processing device according to an embodiment of the invention.

[0012] Hereinafter, an utterance information processing device and an utterance information processing method according to an embodiment of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals and redundant description will be omitted.

[0013] 1 is a diagram illustrating an overview of processing in a speech information processing device. The speech information processing device acquires human speech and generates input information to be input to a generative AI model based on the acquired speech. The speech information processing device may input the generated input information to the generative AI model and acquire speech text information, which is text indicating the content of the human speech.

[0014] In the example shown in FIG. 1 , the speech information processing device executes processing for generating speech text information D5 (minutes data) based on recorded information D1, which is a recording of the speech of each participant of the conference. Specifically, the speech information processing device acquires recorded information D1 indicating the speech of the conference participants. The speech information processing device generates speech information D2, which is a transcription of the speech content related to the speech, based on the recorded information D1. The speech information processing device generates information related to non-verbal information D3 (e.g., information indicating emotions in the speech, speech speed, and speech clarity (clarity)) based on the recorded information D1. The speech information processing device generates a prompt D4 to be input to the generation AI model L based on the speech information D2 and the information related to the non-verbal information D3. Note that the speech information processing device may acquire the minutes of the conference as speech text information D5 by inputting the generated prompt D4 to the generation AI model L. For example, the speech information processing device generates speech text information D5 (minutes data) including sentences that take into account information related to non-verbal information D3 using a generation AI model L. Hereinafter, an example of information related to non-verbal information D3 will be referred to as non-verbal information D3.

[0015] In this embodiment, the person is the person (speaker) who made the utterance related to the utterance information D2. For example, when the utterance information processing device generates meeting minutes data as utterance text information D5, the person is an attendee who is attending the meeting online or offline. Note that the person may be a person other than the speaker, for example, a person who can hear the utterance.

[0016] The recording information D1 is audio data representing the content of human speech. For example, the recording information D1 is audio data (audio or video file) containing speech from multiple people.

[0017] The speech information D2 is information indicating the content of a person's utterance. Specifically, the speech information D2 is data generated based on the recording information D1, and is character data indicating the content of a person's utterance. The speech information D2 is time-series information. For example, the speech information D2 is composed of multiple speech frames in chronological order. As an example, each speech frame is generated by dividing the speech information D2 into individual utterances. The order of the multiple speech frames corresponds to the time series of the multiple utterances in the utterance information D2. Each of the multiple speech frames is associated with the speaker who made the utterance corresponding to the utterance frame. In this way, the speech information D2 includes at least one or more speech frames (information indicating an utterance).

[0018] 2(a), 2(b), and 2(c) are diagrams illustrating an example of information related to utterance information D2 and non-verbal information D3. In the examples illustrated in FIGS. 2(a), 2(b), and 2(c), the utterance information D2 includes a plurality of utterance frames F1 to F3. As an example, in the utterance information D2, three utterance contents, "Today we will consider the introduction of a new IT system," "With the conventional system, managing financial settlements was time-consuming," and "With the new system, we will automate the processes required for quarterly financial settlements," correspond to three utterance frames F1 to F3, respectively. The utterance frame F1 is associated with the speaker "Manager," who made the utterance corresponding to the utterance frame F1. The two utterance frames F2 and F3 are associated with the speaker "Yamada," who made the utterances corresponding to the two utterance frames F2 and F3. The order set for the three utterance frames F1 to F3 indicates the chronological order in which each utterance content was spoken.

[0019] The non-verbal information D3 is information indicating the appearance of a person (speaker) when making an utterance related to the utterance content. The appearance of a person when making an utterance includes at least one of the person's emotions when making an utterance and the person's speaking style when making an utterance. The non-verbal information D3 is information for each time series of the utterance information D2. For example, the non-verbal information D3 is information for each speech frame included in the utterance information D2. In the example shown in Figures 2(a), 2(b), and 2(c), three pieces of non-verbal information D3 are associated with three speech frames F1 to F3, respectively.

[0020] Each piece of non-verbal information D3 includes information indicating the emotional value of a person at the time of utterance of the speech content, for each type of emotion. In the examples shown in FIGS. 2(a), 2(b), and 2(c), the types of emotions are "normal," "happy," "sad," "anger," and "excited." The emotional value indicates the strength (degree) of each emotion. As an example, the emotional value indicates the strength of each emotion as a value between 0 and 1.0. The larger the emotional value, the stronger the emotion. For example, when the emotional value of "anger" is close to 0, it indicates that the person is not very angry. When the emotional value of "anger" is close to 1.0, it indicates that the person is very angry.

[0021] The non-linguistic information D3 includes information indicating the speech rate (speech speed) and clarity of speech (clarity) as information indicating the manner in which a person speaks during speech. In the examples shown in Figures 2(a), 2(b), and 2(c), the information indicating the speech rate includes a numerical value indicating the number of characters spoken per second. The information indicating clarity of speech includes a numerical value indicating clarity of speech. As an example, the numerical value indicating clarity of speech is a numerical value between 0 and 1.0. A larger numerical value indicating clarity of speech indicates clearer speech.

[0022] Typically, the appearance of each person participating in a conference will remain unchanged for a long period of time (for example, they will remain emotionless for a long period of time), so the non-verbal information D3 corresponding to the conference-related speech information D2 will show small changes in the values. Therefore, the data associating each piece of non-verbal information D3 with every speech frame in the speech information D2 will be redundant data.

[0023] The information regarding the non-verbal information D3 is non-verbal information D3 such as information indicating numerical values ​​related to a person's appearance, but is not limited to this. For example, the information regarding the non-verbal information D3 may be an audio file such as a tone of voice or a video file such as a facial expression, or may be information such as a text file indicating the results of analyzing and verbalizing these files.

[0024] The generative AI model L is a model that can generate content in response to input of a prompt D4 containing input information, according to any one or a combination of the instructions, context, question, and output format indicated by the prompt D4, and return the content as response information. The generative AI model L generates response information targeted at the input information. The generative AI model L may be, for example, an interactive AI model that includes a large-scale language model (LLM) and a user interface (UI) for interacting with a user, enabling text chat or voice chat with the user. Examples of such generative AI models include ChatGPT, GPT (registered trademark)-3.5, GPT-4V, PaLM2, etc. Furthermore, although the above describes an example of a large-scale language model, other AI models may also be used.

[0025] A prompt is information indicating an instruction or question entered by a user in an interactive system, such as a dialogue with a generative AI model or a command line interface (CLI). The prompt uses text to express, for example, the command to be executed by the interactive AI model, the task to be executed by the interactive AI model, the background / context to be considered by the interactive AI model (e.g., role, condition), the question to be answered by the interactive AI model, and the output format of the response information from the interactive AI model. Input information to be used as the target of the command / task to be executed by the interactive AI may be added to the prompt D4. Examples of such input information include data files with file names including a predetermined extension, such as text data, image data, application-related data, audio data, video data, and still image data. Application-related data refers to data such as document data, table data, and graph data that can be processed by a default application program. In this embodiment, the prompt D4 includes, as input information, information obtained by adding information regarding non-verbal information D3 to speech information D2.

[0026] The prompt D4 includes an instruction statement for the generation AI model L. The instruction statement includes at least one of an explanation of the input information to be input to the generation AI model L and an explanation of the response information to be output by the generation AI model L. FIG. 3 is a diagram showing an example of the prompt D4. The example of the prompt D4 shown in FIG. 3 includes an instruction statement D41 that is an explanation of the input information and output information (spoken text information D5) to the generation AI model L. First, the instruction statement D41 explains what input information is to be generated and based on that input information. Then, the instruction statement D41 explains the characteristics of the input information and the advantages of the input information including those characteristics.

[0027] For example, the instruction D41 may include content indicating that output information, which is text indicating the content of a human utterance, is to be generated from the input information. The instruction D41 indicates that the generated output information includes the utterance information D2 itself included in the input information, or text (e.g., a summary) based on the utterance information D2. The instruction D41 may also indicate that the generated output information includes a sentence based on the non-verbal information D3 included in the input information. For example, the sentence indicates the manner of speech of a speaker indicated by the non-verbal information D3. The instruction D41 may also indicate that the format of a portion of the generated output information is to be changed according to a predetermined standard. For example, the instruction D41 may indicate that the format is to be changed to emphasize the content of the utterance before and after the numerical value of the speaker's emotion exceeds a predetermined value. As an example, the instruction D41 may indicate that the font size, etc., of a sentence indicating the content of the utterance is to be changed so that it stands out more than sentences indicating other content of the utterance.

[0028] Specifically, an example of prompt D4 includes an instruction D41 that reads, "I would like to create minutes using spoken text transcribed from audio recordings of a meeting conversation. This spoken text also includes accompanying information about the speaker's emotional distribution and speech rhythm. I would like to achieve the following benefits by utilizing this accompanying information..." This instruction D41 can instruct the generation AI model L on what output information to generate based on what input information. Instruction D41 then states, "1. The speaker's emotional distribution is reflected: The accompanying information makes it possible to grasp changes in the speaker's emotions and attitude. This makes it possible to accurately express the atmosphere of the meeting and the progress of the discussion. For example, if the speaker is dissatisfied, this can be explicitly stated, and problems and areas for improvement can be discussed in more detail.", "2. Changes in speaking rate are reflected: The accompanying information also includes changes in speaking rate, making it possible to accurately convey the speaker's intentions and important points. For example, by using an appropriate style and punctuation, the speaker can coordinate and effectively convey information that they want to emphasize or convey quickly to the reader.", and "3. The accuracy of communication is improved." The instruction D41 includes the following: "3. Improves the reader's comprehension: Minutes that reflect the speaker's emotions and speaking speed allow the reader to realistically sense the atmosphere of the meeting and signals of unreasonable arguments. This attracts the reader's interest and improves their understanding of the minutes as a whole. Furthermore, the vividness of the sentences ensures that the reader will not tire of reading through them for long periods of time." Such instruction D41 allows the generation AI model L to learn the significance of the input information format and how to utilize that format. This allows the generation AI model L to more reliably and effectively utilize the characteristics of the input information, thereby outputting the desired spoken text information D5.

[0029] The example of prompt D4 shown in FIG. 3 includes an instruction D42, which is an explanatory sentence explaining the format of the input information and output information (spoken text information D5). Specifically, the example of prompt D4 includes the following instruction D42: "This accompanying information is expressed in the following format: Emotion distribution (Happy: 0.2, Angry: 0.1, Excited: 0.1, Neutral: 0.4)" and "The accompanying information is attached to each utterance. Please summarize the minutes of the meeting in the following items, taking into account the benefits of the accompanying information: 1. Purpose of the meeting; 2. Meeting agenda; 3. Decisions; 4. Undecided matters; 5. Issues; 6. Action points; 7. Summary of the meeting using the accompanying information; 8. Sentences that have been processed from spoken text based on the accompanying information to make them easier for the reader to understand." Such instruction D42 allows the generation AI model L to specify the format of the input information and the format of the output information. This allows the generation AI model L to accurately understand the input information.

[0030] The example of the prompt D4 shown in Figure 3 includes an instruction D43, which is an explanatory sentence that explains the content of the input information to be input to the generation AI model L. Specifically, the example of the prompt D4 includes the instruction D43, "The following is the input sentence that we would like you to create for the minutes. Input sentence: ...". The instruction D43 includes utterance information D2 that includes multiple utterance frames and multiple pieces of non-verbal information D3 (input sentences in the above example). This instruction D43 allows input information based on information related to the utterance information D2 and the non-verbal information D3 to be input to the generation AI model L.

[0031] The spoken character information D5 is response information output by the generation AI model L. In this embodiment, the spoken character information D5 is character data that indicates, in sentences, the content of a person's utterance in a meeting and the person's behavior. For example, the spoken character information D5 is minutes data (response information) that is indicated in sentences that take into account the non-verbal information D3.

[0032] FIG. 4 is a diagram showing an example of spoken text information D5. In the example shown in FIG. 4, spoken text information D5 is a sentence that follows the format specified in instruction sentence D42 of prompt D4. Spoken text information D5 includes sentences D51 that express the state of a person in response to the content of the utterance. Sentences D51 express the speaker who uttered each utterance, the content of each utterance, and the state of the speaker.

[0033] For example, the speech information processing device can automatically generate (create) spoken text information D5 (meeting minutes) by inputting a prompt D4 (instruction sentence) including speech information D2 and non-verbal information D3 as input information to the generation AI model L. This allows a user to more easily understand the spoken text information D5 (meeting minutes data) generated by the generation AI model L, which has previously learned a wide range of knowledge. Furthermore, the speech information processing device automatically creates spoken text information D5 (meeting minutes) without human intervention, thereby significantly improving business efficiency. For example, the use of the speech information processing device reduces the operation and cost of tasks such as creating minutes. Furthermore, for example, the speech information processing device utilizes the prompt D4 including speech information D2 and non-verbal information D3 to generate spoken text information D5 including topics of high importance in a meeting. This provides advanced support for decision-making by a person who has reviewed the spoken text information D5.

[0034] In addition, by including non-verbal information D3 in the prompt D4 (instruction sentence), the prompt D4 including non-verbal information used by humans in communication in a meeting or the like (for example, various non-verbal information such as emotional expressions, speaking speed, and clarity) is input to the generation AI model L. As a result, it is possible to extract important topics, convert text into a form that makes it easy for the reader (user) to understand the meeting situation (the situation around the speaker), and create spoken text information D5 (minutes data) based on the impressions received by the meeting participants, leading to more advanced creation of spoken text information D5 (minutes data).

[0035] Next, the functions of the utterance information processing device 10 according to this embodiment will be described. Fig. 5 is a block diagram showing an information processing system according to this embodiment. As shown in Fig. 5, the information processing system 1 includes a terminal A, the utterance information processing device 10, and a server device 100.

[0036] Terminal A is a device used by a user who wishes to obtain various types of content (text, audio, images, videos, etc.) based on input information using interactive AI. Terminal A is, for example, a personal computer, smartphone, tablet terminal, feature phone, server device, game console, etc. Note that while only two terminals A are illustrated in FIG. 5 , the information processing system may include any number of terminals A greater than or equal to two.

[0037] The server device 100 is a device that enables the provision of content using the generated AI model L. In this embodiment, the server device 100 is capable of providing a content provision function using the generated AI model L. The generated AI model L may be stored within the server device 100, or may be stored in another device connected to the server device 100 via a network, and configured to enable information exchange with the speech information processing device 10 via the server device 100.

[0038] The utterance information processing device 10 according to this embodiment is, for example, a Retrieval-Augmented Generation (RAG) system. The knowledge database searched by the RAG system is implemented, for example, within the utterance information processing device 10 (not shown). The utterance information processing device 10 transmits a prompt D4 including input information based on information relating to utterance information D2 and non-verbal information D3 to the server device 100, and receives response information (utterance text information D5) from the server device 100 in response to the prompt D4. The utterance information processing device 10 includes an acquisition unit 20 and a processing unit 30. The functions of each functional unit of the utterance information processing device 10 will be described in detail below.

[0039] The acquisition unit 20 acquires recorded information D1 from terminal A. The acquisition unit 20 acquires speech information D2 based on the recorded information D1. Specifically, the acquisition unit 20 generates speech information D2 by converting the speech indicated by the acquired recorded information D1 into text. The process of converting speech into text is realized by various known methods, such as a known speech recognition method.

[0040] The acquisition unit 20 generates a plurality of speech frames based on the speech information D2. Specifically, the acquisition unit 20 generates a plurality of speech frames by dividing the speech information D2 into individual utterances. The generation of the speech frames may also be realized by a known method.

[0041] The acquisition unit 20 acquires information related to the non-language information D3. Specifically, the acquisition unit 20 acquires audio data related to the utterance information D2 and generates the non-language information D3 based on the acquired audio data. More specifically, the acquisition unit 20 generates the non-language information D3 based on the recorded information D1 and the utterance information D2. For example, the acquisition unit 20 extracts audio data related to an utterance corresponding to each utterance frame from the recorded information D1. The acquisition unit 20 generates the non-language information D3 based on the extracted audio data. As an example, the acquisition unit 20 generates the non-language information D3 based on the recorded information D1 for each utterance frame (at the granularity of one utterance) included in the utterance information D2. The acquisition unit 20 generates the appearance of a person at the time of utterance based on the audio data using a known model such as an emotion analysis model.

[0042] In the example shown in Figure 2(a), the acquisition unit 20 generates non-verbal information D3 including information such as "normal 0.6, happy 0.1, sad 0.1, angry 0.1, and excited 0.1," "speech rate 5 characters / second," and "clarity level 0.5" based on speech data related to the speech content, "Today, we will consider introducing a new IT system." In the examples shown in Figures 2(b) and 2(c), the acquisition unit 20 similarly generates non-verbal information D3. Note that the process of generating non-verbal information D3 based on speech data can be realized by various known methods.

[0043] The acquisition unit 20 does not have to generate information related to the utterance information D2 and the non-verbal information D3. For example, the acquisition unit 20 may acquire the utterance information D2 and the non-verbal information D3, rather than the recorded information D1, from terminal A. Furthermore, for example, the acquisition unit 20 may acquire the utterance information D2 and the non-verbal information D3 from another device that generates the utterance information D2 and the non-verbal information D3 based on the recorded information D1.

[0044] The processing unit 30 executes processing on the information relating to the speech information D2 and the non-verbal information D3. This processing is processing for obtaining speech text information D5, which is text indicating the content of a human utterance, based on the information relating to the speech information D2 and the non-verbal information D3, using a generation AI model L (first generation AI model) generated by machine learning. The processing unit 30 has a generation unit 31, a determination unit 32 (non-verbal information reduction unit), and a selection unit 33.

[0045] The generation unit 31 generates a prompt D4 to be input to the generation AI model L based on information related to the utterance information D2 and the non-verbal information D3. The generation unit 31 inputs the prompt D4, including input information based on the utterance information D2 and the non-verbal information D3, to the generation AI model L and acquires utterance text information D5. Specifically, the utterance information processing device 10 pre-stores a prompt. The generation unit 31 generates input information including the utterance information D2 and the non-verbal information D3 and includes the input information in a prompt (hereinafter referred to as a general-purpose prompt) stored by the utterance information processing device 10, thereby generating the prompt D4. In the example shown in FIG. 3 , the general-purpose prompt is an instruction statement including instruction statements D41 and D42. The generation unit 31 inputs input information including the utterance information D2 and the non-verbal information D3 into the input field of the general-purpose prompt. For example, the general-purpose prompt includes an explanatory sentence after the instruction statements D41 and D42 indicating that the following sentence is input information. The generation unit 31 generates a prompt D4 by adding input information including the utterance information D2 and the non-verbal information D3 after the explanatory sentence to form an instruction sentence D43. The input information includes sentences indicating the speaker and the content of the speaker's utterance, as well as sentences indicating the speaker's appearance (emotion distribution), for each utterance (for each utterance frame) in the order of the utterance. The general-purpose prompt may be generated by a user using the utterance information processing device 10, or may be prepared by a method other than the above. In this way, the processing unit 30 generates a prompt D4 to be input to the generation AI model L based on information related to the utterance information D2 and the non-verbal information D3. Therefore, the generation unit 31 acquires a prompt D4 based on information related to the utterance information D2 and the non-verbal information D3 for inputting the utterance information D2 and the non-verbal information D3 to the generation AI model L.

[0046] The generation unit 31 inputs the generated prompt D4 to the generation AI model L and acquires utterance text information D5. The prompt D4 is input to the generation AI model L by transmitting the prompt D4 to the server device 100. The generation unit 31 receives, from the server device 100, response information returned from the generation AI model L in response to the input of the prompt D4, and transmits the received response information to terminal A. Note that the generation unit 31 does not have to generate the prompt D4, and may acquire the prompt D4 generated in a device other than the utterance information processing device 10. In this case, the device other than the utterance information processing device 10 generates the prompt D4 based on the utterance information D2 and non-verbal information D3 using a general-purpose prompt stored in advance.

[0047] In this way, the processing unit 30 generates the spoken text information D5 based on the utterance information D2 and the non-verbal information D3 assigned to each speech frame of the utterance information D2. Here, the difference between the value of the non-verbal information D3 for a given speech frame and the value of the non-verbal information D3 for the immediately preceding speech frame may be small. Therefore, if non-verbal information D3 is assigned to all speech frames of the utterance information D2, a large portion of the prompt D4 generated by the processing unit 30, including the utterance information D2 and the non-verbal information D3, may be occupied by redundant data. Considering that the generation AI model L outputs the spoken text information D5 (the most accurate sentence) based on the prompt D4 (or the embedded vector generated from the prompt D4), it is important to shape the prompt D4 itself in order to obtain the desired output from the generation AI model L. Therefore, if the prompt D4 is input directly into the generation AI model L without being shaped, the accuracy of the spoken text information D5 output from the generation AI model L may be reduced (not shaping the prompt D4 may result in noise when the spoken text information D5 is output from the generation AI model L).

[0048] Taking the above into consideration, the processing unit 30 executes processing on the information related to the speech information D2 and the non-language information D3. For example, the processing unit 30 determines, from among the information related to the plurality of pieces of non-language information D3, the information related to the non-language information D3 to be used to obtain the speech text information D5. This makes it possible, for example, to associate the information related to each piece of non-language information D3 with the speech information D2 at an optimal granularity (in other words, to shape the information related to the speech information D2 and the non-language information D3 into an optimal format). As a result, it is possible to suppress a decrease in the accuracy of the speech text information D5.

[0049] Before the generation unit 31 generates the prompt D4, the determination unit 32 performs preprocessing to optimally shape the utterance information D2 and the information related to the non-verbal information D3. Specifically, the determination unit 32 determines, based on a predetermined criterion, the information related to the non-verbal information D3 at each time point, which non-verbal information D3 to use for obtaining the utterance text information D5, from the information related to the non-verbal information D3 at each time point, in accordance with a difference between a statistical value of the information related to the non-verbal information D3 across multiple time points and the information related to the non-verbal information D3 at each time point. For example, the determination unit 32 reduces the non-verbal information D3 assigned to the utterance information D2 for each utterance frame by using a certain deviation as a threshold. As a result, the determination unit 32 reduces the granularity of the non-verbal information D3 assigned to each utterance frame to such an extent that the generation AI model L can accurately reflect the non-verbal information D3 as a sentence in the utterance text information D5. The statistical value is, for example, an average value, but may also be a median or any other numerical value other than the average value.

[0050] Specifically, the determination unit 32 first calculates the average value of non-verbal information across a plurality of pre-set speech frames (e.g., all speech frames related to the recorded information D1 used to generate one piece of speech text information D5) for each speaker and for each type of appearance of the speaker. Next, the determination unit 32 calculates the difference between the calculated average value and the non-verbal information D3 of each speech frame corresponding to the speaker for each type of appearance of the speaker. The difference calculated by the determination unit 32 is an absolute value. For example, for a specific person (speaker), the determination unit 32 calculates the time average of the person's appearance throughout the entire conference for each type of appearance of the speaker by taking the average value of the non-verbal information D3 for each type of appearance of the speaker.

[0051] In the examples shown in Figures 2(a), 2(b), and 2(c), the determination unit 32 calculates the average value of the non-verbal information D3 across three speech frames F1 to F3 for each speaker and for each type of non-verbal information. As an example, the determination unit 32 calculates the average value of "normal" across multiple speech frames including the speech frame F1 corresponding to the speaker "Manager." The determination unit 32 similarly calculates the average values ​​for "happiness," "sadness," "anger," and "excitement," as well as "speech rate" and "clarity." The determination unit 32 calculates multiple average values ​​like this for each speaker. In this way, the determination unit 32 calculates the average value of each piece of non-verbal information for the entire duration of the meeting for which minutes data is to be generated.

[0052] In the example shown in FIG. 2( a), the determination unit 32 calculates the difference between the average value of "normal" corresponding to the speaker "Manager" and the numerical value of "normal" of "0.6." The determination unit 32 similarly calculates the above differences for the speaker "Manager" regarding "happiness," "sadness," "anger," "excitement," "speech rate," and "clarity." Furthermore, in the examples shown in FIGS. 2( b) and 2(c), the determination unit 32 calculates the above differences for the speaker "Yamada-kun." In this way, the determination unit 32 calculates the deviation of the non-verbal information D3 for each type of speaker's state for each speech frame (utterance unit).

[0053] Finally, the determination unit 32 determines, based on a preset threshold, information related to the non-language information D3 for each speech frame, which information is related to the non-language information D3 to be used to obtain the spoken text information D5. More specifically, the determination unit 32 compares the calculated difference with a preset threshold for each type of speaker's appearance. If the difference is equal to or greater than the threshold, the determination unit 32 determines the non-language information D3 of the speech frame corresponding to the difference as the non-language information D3 to be used to obtain the spoken text information D5. Therefore, if the difference between at least one type of numerical value of the speaker's appearance and an average value in the non-language information D3 of the speech frame is equal to or greater than the threshold, the determination unit 32 determines the non-language information D3 as the non-language information D3 to be used to obtain the spoken text information D5. If the differences between all types of numerical values ​​of the speaker's appearance and the average values ​​in all speech frames are below the threshold, the determination unit 32 determines not to use the non-language information D3 to obtain the spoken text information D5. In addition, the determination unit 32 may set other criteria besides the threshold value in advance and determine the non-verbal information D3 to be used to obtain the speech text information D5 from the non-verbal information D3 for each speech frame based on the criteria.

[0054] The threshold is a numerical value indicating a certain difference (deviation) from the average value of the overall non-verbal information D3 of the utterance information D2, as an index for the reduction process of the non-verbal information D3 described above. The threshold may be a value preset by a user or the like, or an optimal threshold stored in a process previously executed (described below). The threshold may be set for each meeting. For example, the threshold may be set based on the meeting format (online, offline, hybrid), the type of meeting (e.g., a meeting where a conclusion is already decided versus a meeting where a conclusion is yet to be reached through discussion), the participants of the meeting (e.g., whether a business partner is participating), or attribute information of the meeting participants (e.g., the organizer, facilitator, proposer, etc.).

[0055] 6(a) and 6(b) are diagrams showing an example of processing of non-language information D3. In FIGS. 6(a) and 6(b), for a specific speaker, multiple speech frames F11 to F17 and multiple pieces of non-language information D31 to D37 are arranged in chronological order. Multiple pieces of non-language information D31 to D37 are associated with the multiple speech frames F11 to F17, respectively. In FIG. 6(b), multiple graphs are shown for each type of speaker's appearance, with the horizontal axis representing time and the vertical axis representing the difference from the average value. The horizontal axis corresponds to the multiple speech frames F11 to F17.

[0056] In the example shown in Figure 6(b), the decision unit 32 determines whether the difference between the numerical value for the emotion "normal" in the non-language information D36 and the average value of the numerical values ​​for "normal" across multiple speech frames is equal to or greater than the threshold T1. Because the difference is equal to or greater than the threshold T1, the decision unit 32 decides that the non-language information D36 of the speech frame F36 is the non-language information D3 to be used for obtaining the speech text information D5. Note that the decision unit 32 similarly determines whether the difference between the numerical value for an emotion other than "normal" in the non-language information D36 and the average value of the numerical values ​​for emotions other than "normal" across multiple speech frames is equal to or greater than the threshold T1.

[0057] Similarly, the determination unit 32 determines the non-language information D34, D32 of speech frames F34, F32 as non-language information D3 to be used to obtain speech text information D5 because the differences between the numerical values ​​related to "speech rate" and "clarity" in the non-language information D34, D32 and the average values ​​of the numerical values ​​related to "speech rate" and "clarity" across multiple speech frames are equal to or greater than thresholds T2, T3. In this way, the determination unit 32 retains the non-language information D32, D34, D36 whose deviations are equal to or greater than the deviation threshold (absolute value). The determination unit 32 deletes the non-language information D31, D33, D35, D37 other than the retained non-language information D32, D34, D36.

[0058] In this way, when the difference between at least one type of numerical value of the speaker's appearance and the average value in the non-language information D3 of an utterance frame is equal to or greater than a threshold, the determination unit 32 determines the non-language information D3 as the non-language information D3 to be used to obtain the utterance text information D5. This allows the determination unit 32 to use the non-language information D3 when the speaker's appearance changes significantly in the entire utterance information D2 to obtain the utterance text information D5.

[0059] The determiner 32 may determine, among the non-language information D3 whose deviation is equal to or greater than a threshold (absolute value), the non-language information D3 corresponding to a change in the deviation as the non-language information D3 used to obtain the spoken text information D5. Specifically, the determiner 32 may determine only the non-language information D3 corresponding to the maximum value (peak) of the deviation during a period in which the deviation is equal to or greater than the threshold as the non-language information D3 used to obtain the spoken text information D5. The determiner 32 may further determine, as the non-language information D3 used to obtain the spoken text information D5, the non-language information D3 of the utterance frame corresponding to the start of the period in which the deviation is equal to or greater than the threshold (the change point at which the non-language information D3 becomes equal to or greater than the threshold), or may further determine, as the non-language information D3 used to obtain the spoken text information D5, the non-language information D3 of the utterance frame corresponding to the end of the period in which the deviation is equal to or greater than the threshold (the change point at which the non-language information D3 finally becomes equal to or greater than the threshold). Therefore, the determination unit 32 may assign non-language information D3 corresponding to the peaks and the transition points to each utterance frame of the utterance information D2.

[0060] After the preprocessing described above is completed in the determination unit 32, the generation unit 31 generates a prompt D4 based on the preprocessed utterance information D2 and non-verbal information D3. The generation unit 31 inputs the prompt D4 to the generation AI model L and acquires utterance character information D5.

[0061] The generation unit 31 may process the statistical values ​​so that the statistical values ​​are also input to the generation AI model L (second generation AI model). Specifically, the input information may further include information indicating statistical values ​​of the non-verbal information D3 in addition to information in which the non-verbal information D3 is assigned to the utterance information D2. More specifically, the input information may further include information indicating statistical values ​​of the non-verbal information D3 for each type of speaker's appearance in addition to information in which the non-verbal information D3 is assigned to the utterance information D2. In this case, the input information includes information indicating the value of the non-verbal information D3 that is not used to obtain the spoken text information D5. In this case, the prompt D4 includes an explanatory statement that a sentence based on the non-verbal information D3 (e.g., the speaker's appearance indicated by the statistical values) will be generated taking into account the statistical values ​​of the non-verbal information D3. For example, the explanatory statement may be included in the instruction statement D41. This allows the generation unit 31 to use the generation AI model L to generate spoken text information D5 that more accurately reflects the speaker's appearance. In the example shown in FIG. 4, the spoken text information D5 expresses in sentences the speaker who spoke each utterance, the content of each utterance, and the appearance of the speaker.

[0062] The generation unit 31 generates multiple pieces of spoken text information D5 for each of multiple preset criteria. Specifically, the determination unit 32 first performs preprocessing on the spoken information D2 and the non-verbal information D3 for each of multiple thresholds that are set in advance and stored, and generates input information for each threshold. Then, the generation unit 31 generates spoken text information D5 for each piece of input information using the generation AI model L. For example, the generation unit 31 generates minutes of a meeting for each threshold using the generation AI model L.

[0063] The selection unit 33 selects spoken character information D5 from the plurality of pieces of spoken character information D5 using a generation AI model L (second generation AI model) generated by machine learning. The selection unit 33 outputs the selected spoken character information D5 to the terminal A. Specifically, the selection unit 33 selects one piece of spoken character information D5 using the generation AI model L from the plurality of pieces of spoken character information D5 generated by the generation unit 31 and the determination unit 32 for each of a plurality of preset thresholds. More specifically, the selection unit 33 inputs the plurality of pieces of spoken character information D5 (minutes data) to the generation AI model L and selects the optimal piece of spoken character information D5 (minutes data). For example, the selection unit 33 includes the spoken character information D5 generated using the threshold value X1, the spoken character information D5 generated using the threshold value X2, ..., and the spoken character information D5 generated using the threshold value Xn in a prompt D4 for the user to select the spoken character information D5 that will most easily understand the speaker's state (for example, a prompt D4 including a statement such as "Which minutes can best be read by a human to grasp the atmosphere of the situation?"), and inputs this to the generation AI model L, thereby causing the generation AI model L to select the optimal spoken character information D5. Note that the selection unit 33 may allow the user to select the optimal minutes data from multiple minutes data.

[0064] The selection unit 33 stores the threshold value corresponding to the optimal utterance character information D5 as the optimal threshold value. The selection unit 33 determines the deviation threshold value required for reducing the non-verbal information D3 in the determination unit 32. The determination unit 32 may adopt the threshold value as the optimal threshold value when the utterance information processing device 10 next generates utterance character information D5. For example, the determination unit 32 may adopt the determined threshold value as the optimal threshold value for the next meeting and thereafter.

[0065] The generation of the plurality of pieces of spoken text information D5 using the plurality of thresholds in this manner may be used to determine the threshold used to reduce the non-verbal information D3. In this case, the generation of the plurality of pieces of spoken text information D5 is performed only when the threshold is determined, and thereafter, a single piece of spoken text information D5 is generated using the determined threshold. Furthermore, the plurality of pieces of spoken text information D5 used to determine the threshold do not need to be based on the same recorded information D1, and may be based on each of the plurality of pieces of recorded information D1 that are different from each other.

[0066] Next, an utterance information processing method, which is a process executed by the utterance information processing device according to this embodiment, will be described with reference to the flowcharts shown in Figures 7, 8, and 9. This process is performed when the utterance information processing device 10 receives recorded information D1.

[0067] First, the acquisition unit 20 acquires information related to the utterance information D2 and the non-verbal information D3 (S100: acquisition step). Specifically, the acquisition unit 20 first acquires the recorded information D1. Then, the acquisition unit 20 generates information related to the utterance information D2 and the non-verbal information D3 based on the recorded information D1.

[0068] Then, the processing unit 30 executes processing on the information related to the utterance information D2 and the non-verbal information D3 (S200: processing step). This processing is for obtaining, using the generation AI model L generated by machine learning, utterance character information D5, which is text indicating the content of a human utterance, based on the information related to the utterance information D2 and the non-verbal information D3. Specifically, the processing shown in the flowchart of FIG. 8 is executed.

[0069] In the process of Fig. 8, first, the processing unit 30 generates utterance character information D5 for each of a plurality of preset criteria (S210). Specifically, the process shown in the flowchart of Fig. 9 is executed. The following processes of S211 to S214 are repeatedly executed for each of a plurality of preset criteria.

[0070] 9 , first, the processing unit 30 calculates statistical values ​​of information related to non-language information D3 across multiple time points (S211). Next, the processing unit 30 determines information related to non-language information D3 to be used to obtain spoken text information D5 (S212). Specifically, the processing unit 30 determines, based on a preset criterion, the information related to non-language information D3 at each time point, which information is related to non-language information D3 to be used to obtain spoken text information D5, in accordance with the difference between the statistical values ​​of information related to non-language information D3 across multiple time points and the information related to non-language information D3 at each time point. Next, the processing unit 30 generates a prompt D4 to be input to the generation AI model L based on the spoken information D2 and the information related to non-language information D3 (S213).

[0071] Finally, the processing unit 30 inputs input information based on the information related to the speech information D2 and the non-verbal information D3 to the generated AI model L, and acquires the speech character information D5 (S214). For example, the processing unit 30 inputs the generated prompt D4 to the generated AI model L, and acquires the speech character information D5.

[0072] Then, the processing unit 30 selects utterance character information D5 from the plurality of pieces of utterance character information D5 (S220). In this case, the selection unit 33 determines the threshold value corresponding to the selected utterance character information D5 as the optimal threshold value. Thereafter, the processing unit 30 outputs the generated utterance character information D5 to the terminal A. This completes the utterance information processing method according to this embodiment.

[0073] Next, the effects of the speech information processing device 10 and speech information processing method according to this embodiment will be described. According to this embodiment, a generation AI model L (first generation AI model) generated by machine learning processes the speech information D2 and information related to the non-verbal information D3 to obtain speech character information D5, which is text indicating the content of a person's utterance, based on information related to the speech information D2 and the non-verbal information D3. With this configuration, the generation AI model L can generate speech character information D5 (see FIG. 4 ) that expresses the person's behavior in response to the utterance content in sentences. This allows the user to easily understand the person's behavior when speaking.

[0074] As in the above-described embodiment, the acquisition unit 20 may acquire voice data related to the speech information D2 and generate information related to the non-verbal information D3 based on the acquired voice data. According to this configuration, information related to the non-verbal information D3 indicating the manner of the person speaking is generated based on voice data related to the content of the utterance. This makes it possible to generate information related to the non-verbal information D3 with high accuracy based on, for example, the tone of voice of the speaker contained in the voice data. However, the acquisition unit 20 may acquire information related to the non-verbal information D3 by a method other than the above.

[0075] As in the above-described embodiment, the processing unit 30 may generate a prompt D4 to be input to the generation AI model L (first generation AI model) based on information related to the utterance information D2 and the non-verbal information D3. This configuration enables the generation AI model L to generate utterance text information D5 that expresses in sentences how a person responds to the utterance content. Furthermore, by adjusting the description of the prompt D4, the desired utterance text information D5 can be reliably obtained.

[0076] As in the above-described embodiment, the utterance information D2 may be time-series information. The information regarding the non-language information D3 may be information for each time point in the time series. The processing unit 30 may determine, based on a predetermined criterion, the information regarding the non-language information D3 for each time point, which information regarding the non-language information D3 to use to obtain the utterance text information D5, from the information regarding the non-language information D3 for each time point, in accordance with the difference between the statistical value of the information regarding the non-language information D3 across multiple time points and the information regarding the non-language information D3 for each time point. This configuration allows the generation AI model L to reduce the granularity of the information regarding the non-language information D3 assigned to each utterance frame to an extent that the generation AI model L can accurately reflect the information regarding the non-language information D3 in the utterance text information D5. This allows the information regarding each piece of non-language information D3 to be associated with the utterance information D2 with appropriate granularity, thereby suppressing a decrease in the accuracy of the utterance text information D5 output from the generation AI model L. Additionally, by reducing the information regarding the non-language information D3 associated with the utterance information D2, the length of the input text of the prompt D4 input to the generation AI model L is significantly reduced. This makes it possible to reduce the computational resources required for the generation AI model L when generating spoken text information D5 using the generation AI model L, and also to generate spoken text information D5 stably and accurately (improving the accuracy and stability of the generated minutes data).

[0077] However, the spoken information D2 does not have to be time-series information. The information related to the non-verbal information D3 does not have to be information for each point in time. In such a case, the spoken text information D5 may be information that describes the state of the speaker in text. In the example shown in FIG. 4, the spoken text information D5 includes the sentence "7. Summary of the meeting using accompanying information: Yamada reported on the progress and stated that there was not much progress. The department head reacted angrily and excitedly to Yamada's lack of progress." This sentence describes the state of the "department head" and "Yamada," so a user who reads this sentence can easily understand the state of the speaker.

[0078] Alternatively, the processing unit 30 may determine information about all of the non-language information D3 as information about the non-language information D3 to be used to obtain the spoken text information D5. In this case, the preprocessing by the determination unit 32 is not performed.

[0079] As in the above-described embodiment, the processing unit 30 may generate spoken text information D5 for each of a plurality of preset criteria and select spoken text information D5 from the plurality of pieces of spoken text information D5 using the generation AI model L (second generation AI model) generated by machine learning. This configuration allows the selection of spoken text information D5 generated based on the prompt D4 in which information about each piece of non-verbal information D3 is associated with the utterance information D2 at a more appropriate granularity. However, the process of selecting spoken text information D5 from the plurality of pieces of spoken text information D5 does not have to be performed by the speech information processing device 10. For example, the user may select the most appropriate piece of spoken text information D5 from the plurality of pieces of spoken text information D5.

[0080] As in the above-described embodiment, the processing unit 30 may process the statistical values ​​so that the statistical values ​​are also input to the generation AI model L (second generation AI model). With this configuration, the spoken text information D5 can be generated with higher accuracy by taking into account information about non-language information D3 that was deleted when information about each piece of non-language information D3 was associated with the utterance information D2 at an appropriate granularity and information about non-language information D3 that was not deleted. However, the processing unit 30 does not have to input the statistical values ​​to the generation AI model L. Even in this case, the spoken text information D5 can be generated with sufficient accuracy.

[0081] As in the above-described embodiment, the appearance of the person at the time of speaking may include the emotion of the person at the time of speaking. With this configuration, the emotion of the person at the time of speaking can be taken into consideration, and the spoken text information D5 can be generated with higher accuracy.

[0082] As in the above-described embodiment, the appearance of the person when speaking may include the way the person speaks when speaking. With this configuration, the way the person speaks when speaking can be taken into consideration, and the spoken text information D5 can be generated with higher accuracy.

[0083] As in the above-described embodiment, the speech information D2 includes information indicating at least one or more utterances, and the information regarding the non-verbal information D3 is information for each piece of information indicating an utterance, and the processing unit 30 may acquire information based on the speech information D2 and the information regarding the non-verbal information D3 to input to the generation AI model L. This allows information indicating the state of a person in each utterance to be input to the generation AI model L. As a result, the speech text information D5 can be generated with greater accuracy, taking into account the state of a person when speaking.

[0084] For example, the speech information D2 is information indicating at least one or more utterances and the speaker associated with each utterance. The information related to the non-verbal information D3 is a numerical value indicating the degree (for example, a score) of the speaker's emotion (for example, anger) in each utterance. By inputting information based on such speech information D2 and information related to the non-verbal information D3 into the generation AI model L, it is possible to input information indicating the degree of anger of the speaker at the timing of each utterance into the generation AI model L. As a result, the generation AI model L can generate speech text information D5 as content that sounds as if it were thought up by a human.

[0085] In this way, the process of the present disclosure, which generates content using the generation AI model L, differs from the conventional process of generating content using preset rules rather than the generation AI model L. Specifically, the conventional process distributes pre-prepared content based on information that is somewhat fixed, whereas the process of the present disclosure generates content by comprehensively considering multiple pieces of detailed information. As a result, the multiple pieces of content generated by the process of the present disclosure are completely different from each other.

[0086] The configuration of the speech information processing device 10 described in the above embodiment is an example, and various modifications can be made. Representative modifications will be described below.

[0087] In the above embodiment, the processing unit 30 inputs input information based on information related to the utterance information D2 and the non-verbal information D3 to the generation AI model L and acquires the spoken character information D5, but it is not necessary to acquire the spoken character information D5. In this case, the processing unit 30 generates a prompt D4 to be input to the generation AI model L based on information related to the utterance information D2 and the non-verbal information D3, and outputs the generated prompt D4 to another device (e.g., terminal A). The prompt D4 is transmitted to the server device 100 by another device and input to the generation AI model L. The server device 100 transmits the spoken character information D5 output from the generation AI model L to terminal A. The process of causing the generation AI model to generate the spoken character information D5 using the prompt D4 in this manner may be performed by the other device.

[0088] As in the above-described embodiment, the processing unit 30 may input input information based on information related to the utterance information D2 and the non-verbal information D3 to the generation AI model L and acquire the utterance text information D5. According to this configuration, the generation AI model L can acquire the utterance text information D5. In this embodiment, the processing unit 30 generates a prompt D4 from the input information so that the generation AI model L generates the utterance text information D5. However, in a configuration in which the processing unit 30 inputs the input information to the generation AI model L, the generation of the prompt D4 does not necessarily have to be performed by the utterance information processing device 10, but may be performed by another device, etc. In this case, the acquisition unit 20 may acquire the prompt D4 including the input information generated by another device, etc. Furthermore, the information input to the generation AI model L to generate the utterance text information D5 may be information other than the prompt D4, as long as it includes the above-described input information.

[0089] In the above embodiment, the utterance information processing device 10 is a server device separate from terminal A, but is not limited to this. For example, the utterance information processing device 10 may be a part of terminal A. As an example, the RAG system configured by the acquisition unit 20 and the processing unit 30 may be implemented in terminal A. In this case, the knowledge database searched by the RAG system may be implemented inside terminal A, may be implemented in the utterance information processing device 10, or may be implemented in another device on a network (e.g., the cloud).

[0090] In the above embodiment, the information processing system 1 includes a terminal A, an utterance information processing device 10, and a server device 100, but is not limited thereto. For example, the information processing system 1 of a modified example may include only the utterance information processing device 10 according to the modified example. In this case, the utterance information processing device 10 is a terminal held by a user, and is, for example, a device used by a user who wishes to obtain various types of content based on input information using interactive AI. The utterance information processing device 10 includes a RAG system composed of an acquisition unit 20 and a processing unit 30, and a generation AI model L. The configuration of the above modified example can be realized by installing an application that executes the functions of the generation AI model L in the utterance information processing device 10. In this way, the generation AI model L may be implemented on a network (e.g., a cloud) other than the server device 100. Note that in the information processing system 1 of the modified example, the knowledge database searched by the RAG system may be implemented within the utterance information processing device 10 or on another device on the network (e.g., a cloud).

[0091] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (e.g., wired, wireless, etc.) and these multiple devices. The functional block may also be realized by combining software with the single device or multiple devices.

[0092] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0093] For example, the speech information processing device 10 according to an embodiment of the present disclosure may function as a computer that performs information processing according to the present disclosure. Fig. 10 is a diagram showing an example of the hardware configuration of the speech information processing device 10 according to an embodiment of the present disclosure. The above-described speech information processing device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0094] In the following description, the term "device" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the speech information processing device 10 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

[0095] Each function in the speech information processing device 10 is realized by loading specified software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0096] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the processing unit 30 of the speech information processing device 10 may be realized by the processor 1001.

[0097] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the processing unit 30 of the speech information processing device 10 may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be made for other functional blocks. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0098] The memory 1002 is a computer-readable recording medium and may be configured by, for example, at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to an embodiment of the present disclosure.

[0099] The storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device. The storage medium provided in the speech information processing device 10 may be, for example, a database, a server, or other appropriate medium including at least one of the memory 1002 and the storage 1003.

[0100] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0101] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0102] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0103] The speech information processing device 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0104] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (New Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems, and next-generation systems enhanced based on these. Furthermore, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G, etc.) may also be applied.

[0105] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0106] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0107] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0108] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0109] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0110] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0111] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0112] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0113] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0114] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0115] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0116] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0117] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0118] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0119] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0120] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0121] The system, speech information processing device, and speech information processing method disclosed herein have the following configurations. [1] An utterance information processing device comprising: an acquisition unit that acquires utterance information indicating the content of a human utterance and information related to non-verbal information that indicates the state of the human when making the utterance related to the utterance content; and a processing unit that executes processing on the utterance information and information related to the non-verbal information to obtain, by a first generation AI model generated by machine learning, utterance text information that is text indicating the content of the human utterance based on the utterance information and the information related to the non-verbal information. [2] The utterance information processing device described in [1], wherein the processing unit generates a prompt to be input to the first generation AI model based on the utterance information and the information related to the non-verbal information. [3] The utterance information processing device described in [1], wherein the processing unit inputs input information based on the utterance information and the information related to the non-verbal information to the first generation AI model to acquire the utterance text information. [4] The speech information processing device according to any one of [1] to [3], wherein the speech information is time-series information, the information related to the non-language information is information for each time point in the time series, and the processing unit determines, based on a preset criterion, information related to the non-language information for each time point, information related to the non-language information to be used to obtain speech text information, from the information related to the non-language information for each time point, in accordance with a difference between a statistical value of the information related to the non-language information over a plurality of time points and the information related to the non-language information for each time point. [5] The speech information processing device according to [4], wherein the processing unit generates the speech text information for each of the plurality of preset criterions, and selects the speech text information from the plurality of pieces of speech text information using a second generation AI model generated by machine learning. [6] The speech information processing device according to [4] or [5], wherein the processing unit processes the statistical value so that the statistical value is also input to the second generation AI model. [7] The speech information processing device according to any one of [1] to [6], wherein the appearance of the person at the time of speaking includes the emotion of the person at the time of speaking. [8] The speech information processing device according to any one of [1] to [7], wherein the appearance of the person when speaking includes a way of speaking of the person when speaking.[9] The utterance information processing device according to any of [1] to [8], wherein the utterance information includes information indicating at least one or more utterances, the information related to the non-verbal information is information for each piece of information indicating the utterance, and the processing unit acquires a prompt based on the utterance information and the information related to the non-verbal information, for inputting the utterance information and the information related to the non-verbal information to the first generation AI model.

[10] A speech information processing method comprising: an acquisition step of acquiring utterance information indicating the content of a person's utterance and information related to non-verbal information indicating the state of the person when speaking related to the utterance content, and a processing step of executing processing on the utterance information and the information related to the non-verbal information, to obtain, by a first generation AI model generated by machine learning, utterance text information, which is text indicating the content of the person's utterance, based on the utterance information and the information related to the non-verbal information.

[0122] 10... speech information processing device, 20... acquisition unit, 30... processing unit, 1001... processor, 1002... memory, 1003... storage, 1004... communication device, 1005... input device, 1006... output device.

Claims

1. A speech information processing device comprising: an acquisition unit that acquires speech information indicating the content of a person's speech and information regarding non-verbal information that indicates the state of the person when speaking the speech content; and a processing unit that executes processing on the speech information and information regarding the non-verbal information to obtain speech text information, which is text indicating the content of the person's speech, based on the speech information and information regarding the non-verbal information using a first generation AI model generated by machine learning.

2. The speech information processing device according to claim 1, wherein the processing unit generates a prompt to be input to the first generative AI model based on the speech information and information related to the non-verbal information.

3. The speech information processing device according to claim 1, wherein the processing unit inputs input information based on the speech information and information related to the non-verbal information into the first generative AI model and acquires the speech text information.

4. The speech information processing device described in claim 1, wherein the speech information is time-series information, the information regarding the non-verbal information is information for each point in time in the time series, and the processing unit determines, based on a predetermined criterion, the information regarding the non-verbal information for each point in time, which information regarding the non-verbal information to be used to obtain speech text information, in accordance with the difference between the statistical value of the information regarding the non-verbal information over multiple points in time and the information regarding the non-verbal information for each point in time.

5. The speech information processing device according to claim 4, wherein the processing unit generates the speech text information for each of a plurality of the predetermined criteria, and selects the speech text information from the plurality of pieces of speech text information using a second generation AI model generated by machine learning.

6. The speech information processing device according to claim 4, wherein the processing unit performs processing on the statistical value so that the statistical value is also input to a second generation AI model.

7. The speech information processing device according to claim 1, wherein the appearance of the person when speaking includes the emotion of the person when speaking.

8. The speech information processing device according to claim 1, wherein the appearance of the person when speaking includes the way the person speaks when speaking.

9. The speech information processing device described in claim 1, wherein the speech information includes information indicating at least one or more utterances, the information regarding the non-verbal information is information for each piece of information indicating the utterance, and the processing unit obtains a prompt based on the speech information and the information regarding the non-verbal information for inputting the speech information and the information regarding the non-verbal information into the first generation AI model.

10. A method for processing speech information, comprising: an acquisition step of acquiring speech information indicating the content of a person's speech and information regarding non-verbal information indicating the state of the person when speaking the speech content; and a processing step of executing processing on the speech information and information regarding the non-verbal information to obtain speech text information, which is text indicating the content of the person's speech, based on the speech information and information regarding the non-verbal information using a first generation AI model generated by machine learning.