Voice conversation generation device, voice conversation system, voice conversation generation device control method, and control program
The voice conversation generation device addresses the emotional isolation of elderly individuals by periodically updating conversation information to reflect the speaker's voice changes, enhancing user engagement and connection through dynamic conversations.
Patent Information
- Application Number
- PCT/JP2024/014571
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2024-04-10
- Publication Date
- 2024-11-28
AI Technical Summary
Conventional voice conversation systems fail to meet the emotional and social needs of elderly individuals living alone, particularly in providing a sense of connection and companionship, as they cannot effectively reflect the changes in a speaker's voice over time, leading to a lack of engagement and potential isolation.
A voice conversation generation device that periodically acquires and synthesizes voice information from a specific speaker, generating conversation information using multiple synthesized voices to reflect changes in the speaker's voice over time, allowing users to feel more connected and engaged through dynamic conversations.
The system enhances the emotional connection by providing conversation information that reflects the speaker's current state, allowing users to feel the growth and changes in the speaker's voice, thereby reducing feelings of loneliness and isolation.
Smart Images

Figure JP2024014571_28112024_PF_FP_ABST
Abstract
Description
Voice conversation generation device, voice conversation system, and control method and control program for voice conversation generation device
[0001] The present disclosure relates to a voice conversation generation device, a voice conversation system, and a control method and control program for a voice conversation generation device.
[0002] A voice conversation system that realizes free conversation with a human using a terminal device such as an interactive robot or a smartphone is known. For example, Japanese Patent Application Laid-Open No. 2016-110151 describes a conversation voice providing system that uses synthesized human voice as the voice used for conversation with a human.
[0003] The voice conversation system described above is primarily used to provide users with desired information and to entertain them through dialogue. On the other hand, when used by users who have relatively few opportunities to converse with others, such as elderly people who live alone, such voice conversation systems can be expected to have further effects, such as reducing the chances of feeling lonely or preventing the onset of depression or dementia.
[0004] Meanwhile, elderly people who live alone as described above may have relatives, such as children or grandchildren, who live far away. Such elderly people often wish to have frequent conversations with their relatives or to feel closer to their grandchildren as they grow up. While conventional voice conversation systems can be expected to achieve the above-mentioned effects by enabling free conversation, they have not been able to meet the needs of elderly people as described above. Therefore, there is room for conversations using voice conversation systems to provide further added value in the above respects.
[0005] In view of the above, the present disclosure provides a voice conversation generation device, a voice conversation system, and a control method and control program for a voice conversation generation device that increase the added value of conversations held with users.
[0006] A voice conversation generation device according to a first aspect of the present disclosure includes: a voice information acquisition unit that acquires voice information including the voice of a specific speaker from a first terminal device every time a predetermined period of time elapses; a voice synthesis unit that generates a synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device; a conversation information generation unit that generates conversation information using one of a plurality of synthetic voices of the specific speaker generated by the voice synthesis unit at different times; and a conversation information transmission unit that transmits the generated conversation information to a second terminal device that is configured as a terminal device different from the first terminal device and is capable of engaging in free conversation with a user.
[0007] In this voice conversation generation device, voice information containing the voice of a specific speaker used to generate synthetic speech for use in conversation information is obtained from the first terminal device at predetermined intervals, making it possible to generate synthetic speech that reflects changes in the voice of the specific speaker over time. As a result, the conversation information generated by the conversation information generation unit reflects the state of the speaker at the time the synthetic speech was generated. Therefore, a user who listens to the conversation information generated via the second terminal device can not only easily listen to the voice of a desired speaker, such as a relative living far away, but also get a sense of the speaker's state, such as their level of growth.
[0008] A voice conversation generation device according to a second aspect of the present disclosure is the voice conversation generation device according to the first aspect of the present disclosure, wherein the conversation information generation unit generates conversation information using the latest synthetic voice among the synthetic voices of the multiple specific speakers generated by the voice synthesis unit.
[0009] In such a voice conversation generation device, conversation information is generated using the latest synthetic voice, so that a user who hears the conversation information can get a sense of the speaker's current state.
[0010] A voice conversation generation device according to a third aspect of the present disclosure is the voice conversation generation device according to the first aspect of the present disclosure, wherein the conversation information generation unit generates conversation information using a synthetic voice instructed by the first terminal device or the second terminal device from among the synthetic voices of the plurality of specific speakers generated by the voice synthesis unit.
[0011] In such a voice conversation generation device, the synthesized voice used for the conversation information can be selected by the terminal device, so that a user can choose a voice of their choice and reflect it in the conversation information, or a voice randomly selected from multiple voices can be reflected in the conversation information. Furthermore, by listening to synthesized voices collected at different times, changes in the speaker's state can be more clearly perceived.
[0012] A voice conversation generation device according to a fourth aspect of the present disclosure is the voice conversation generation device according to any one of the first to third aspects of the present disclosure, wherein the predetermined period is set to between 1 and 12 months.
[0013] In such a voice conversation generation device, the timing of acquiring voice information can be adjusted according to the age of the speaker, etc.
[0014] A voice conversation generation device according to a fifth aspect of the present disclosure is a voice conversation generation device according to any one of the first to fourth aspects of the present disclosure, wherein the voice conversation generation device further includes a voice information storage unit capable of storing a plurality of pieces of the voice information acquired from the first terminal device for each speaker, and the voice synthesis unit generates the synthetic voice for each of the plurality of speakers stored in the voice information storage unit.
[0015] Such a voice conversation generation device can generate synthetic voices for multiple speakers, and can accommodate cases where the user of the second terminal device wants to hear from multiple speakers.
[0016] A voice conversation generation device according to a sixth aspect of the present disclosure is the voice conversation generation device according to the fifth aspect of the present disclosure, wherein the conversation information generation unit generates conversation information using the synthetic voice of a speaker specified by the second terminal device from among the synthetic voices of multiple speakers generated by the voice synthesis unit.
[0017] In such a voice conversation generation device, conversation information can be easily generated in the voice of a desired speaker in response to an instruction from the terminal side.
[0018] A voice conversation generation device according to a seventh aspect of the present disclosure is the voice conversation generation device according to any one of the first to sixth aspects of the present disclosure, further including a voice information storage unit capable of storing a plurality of pieces of the voice information acquired from the first terminal device, and a voice information transmission unit that transmits specific voice information stored in the voice information storage unit to the first terminal device or the second terminal device based on a request from the first terminal device or the second terminal device.
[0019] In such a voice conversation generation device, the voice information stored in the voice conversation generation device can be acquired on the terminal side, so that the speaker's voice collected in the past can be listened to on the terminal side at any time.
[0020] A voice conversation generation device according to an eighth aspect of the present disclosure is a voice conversation generation device according to any one of the first to seventh aspects of the present disclosure, wherein the voice conversation generation device further includes a usage history collection unit that collects communication content between the voice conversation generation device and the second terminal device as usage history information, and the usage history information collected by the usage history collection unit can be obtained from the first terminal device.
[0021] In such a voice conversation generation device, the owner of the first terminal device can indirectly monitor whether anything unusual is happening to the user of the second terminal device by obtaining the history of conversations conducted on the second terminal device on the first terminal device.
[0022] A voice conversation system according to a ninth aspect of the present disclosure includes a first terminal device that acquires voice information including the voice of a specific speaker, a voice conversation generation device according to any one of the first to eighth aspects of the present disclosure, and a second terminal device that is configured as a terminal device different from the first terminal device and is capable of conducting a free conversation with a user using conversation information generated by the voice conversation generation device.
[0023] In such a voice conversation system, voice information containing the voice of a specific speaker used to generate synthetic speech for conversation information is acquired at predetermined intervals, making it possible to generate synthetic speech that reflects changes in the voice of the specific speaker over time. As a result, the conversation information generated by the conversation information generation unit reflects the state of the speaker at the time the synthetic speech was generated. Therefore, a user who listens to the generated conversation information can not only easily listen to the voice of a desired speaker, such as a relative living far away, but also get a sense of the speaker's state, such as their level of growth.
[0024] A control method for a voice conversation generation device according to a tenth aspect of the present disclosure includes the steps of: acquiring voice information including the voice of a specific speaker from a first terminal device; generating synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device; measuring the elapsed time since the synthetic voice was generated; repeatedly executing the above-mentioned series of steps each time the elapsed time reaches a predetermined period; generating conversation information using one of multiple synthetic voices of the specific speaker generated at different times; and transmitting the generated conversation information to a second terminal device configured as a terminal device different from the first terminal device and capable of engaging in free conversation with a user.
[0025] In this control method for a voice conversation generation device, the synthetic speech used in the conversation information is generated every predetermined period of time, making it possible to generate synthetic speech that reflects changes in the voice of a specific speaker over time. As a result, the conversation information generated by the conversation information generation unit reflects the state of the speaker at the time the synthetic speech was generated. Therefore, a user who listens to the generated conversation information can easily listen to the voice of a desired speaker, such as a relative living far away, and can also get a sense of the speaker's state, such as their level of growth.
[0026] A control program for a voice conversation generation device according to an eleventh aspect of the present disclosure causes at least one processor of a computer provided in the voice conversation generation device, which generates conversation information to be used in a second terminal device capable of conducting free conversation with a user, to execute the following processes: acquiring voice information including the voice of a specific speaker from a first terminal device configured as a terminal device different from the second terminal device; generating synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device; measuring the elapsed time since the synthetic voice was generated; repeatedly executing the above-mentioned series of steps each time the elapsed time reaches a predetermined period; generating the conversation information using one of multiple synthetic voices of the specific speaker generated at different times; and transmitting the generated conversation information to the second terminal device.
[0027] In the control program for such a voice conversation generation device, the synthetic speech used in the conversation information is generated at predetermined intervals, making it possible to generate synthetic speech that reflects changes in the voice of a specific speaker over time. As a result, the conversation information generated by the conversation information generation unit reflects the speaker's state at the time the synthetic speech was generated. This allows a user who listens to the generated conversation information to not only easily listen to the voice of a desired speaker, such as a relative living far away, but also to get a sense of the speaker's state, such as their level of growth.
[0028] According to the voice conversation generation device, voice conversation system, and control method and control program for a voice conversation generation device of the present disclosure, it is possible to increase the added value of conversations held with users.
[0029] FIG. 1 is a configuration diagram schematically illustrating an example of a voice conversation system according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example of the hardware configuration of a first terminal device of the voice conversation system shown in FIG. 1. FIG. 3 is a block diagram illustrating an example of the hardware configuration of a voice conversation generation device according to an embodiment of the present disclosure shown in FIG. 1. FIG. 4 is a functional block diagram illustrating an example of the functions of each component of the voice conversation system shown in FIG. 1. FIG. 5 is a flowchart illustrating an example of a control method for a voice conversation generation device according to an embodiment of the present disclosure, which is a schematic process for generating synthetic voice. FIG. 6 is a flowchart illustrating an example of a control method for a voice conversation generation device according to an embodiment of the present disclosure, which is a schematic process for transmitting conversation information.
[0030] This application is based on Japanese Patent Application No. 2023-086527, filed on May 25, 2023, in Japan, the contents of which are incorporated herein by reference. The present disclosure will become more fully understood from the following detailed description. Further scope of application of the present application will become apparent from the following detailed description. However, the detailed description and specific examples are preferred embodiments of the present disclosure and are described for illustrative purposes only. From this detailed description, various changes and modifications will be apparent to those skilled in the art within the spirit and scope of the present disclosure. The applicant does not intend to dedicate any of the described embodiments to the public, and the applicants also consider disclosed modifications and alternatives, even if not literally included within the scope of the claims, to be part of the invention under the doctrine of equivalents. Like reference numbers and names in the various drawings indicate like elements.
[0031] Hereinafter, each embodiment for carrying out the present disclosure will be described with reference to the drawings. Note that the scope necessary for the explanation to achieve the object of the present disclosure will be schematically shown below, and the scope necessary for explaining the relevant parts of the present disclosure will be mainly explained, and the parts for which explanation is omitted will be considered to be publicly known technologies.
[0032] <Voice Conversation System> The voice conversation system according to one embodiment described below includes a voice conversation generation device according to one embodiment of the present disclosure as a part thereof. The voice conversation system is a system for engaging in free conversation with a first user (an example of a user). The system generates the voice used in the free conversation based on voice information of a second user (an example of a specific speaker). Here, the first user in this embodiment is a grandfather and / or grandmother who lives alone or as a married couple, and the second user is a relative of the first user, more specifically, a grandchild who lives far away. This voice conversation system according to this embodiment aims to provide care for elderly people such as grandfathers and / or grandmothers through conversation. Note that the users in the voice conversation system of the present disclosure are not limited to those described above. For example, the first user could be a father or mother who is working away from home, and the second user could be a child who lives far away.
[0033] 1 is a schematic diagram illustrating an example of a voice conversation system according to an embodiment of the present disclosure. As shown in FIG. 1, the voice conversation system 1 according to this embodiment includes at least a first terminal device 2, a second terminal device 3, and a voice conversation generation device 4. These devices may be capable of communicating with each other via a network NW such as the Internet.
[0034] The first terminal device 2 is a terminal device for acquiring audio information including the voice of a specific speaker. This first terminal device 2 may be owned by a guardian of a grandchild who is a second user, for example, a child of a grandfather or grandmother who is a first user. The owner of the first terminal device 2 is not particularly limited, and may be the second user himself or herself.
[0035] The first terminal device 2 may be configured as a device capable of acquiring at least ambient sounds and performing data communication, such as a smartphone, a personal computer, a tablet terminal, a mobile phone, an interactive robot, or a smart speaker. In this embodiment, as shown in FIG. 1, a smartphone is used as the first terminal device 2.
[0036] Fig. 2 is a block diagram showing an example of the hardware configuration of the first terminal device 2 of the voice conversation system shown in Fig. 1. As shown in Fig. 2, the first terminal device 2 may mainly include a microphone 11, a speaker 12, a display unit 13, an operation unit 14, and a control device 15.
[0037] The microphone 11 can be configured as a microphone capable of collecting sounds generated around the first terminal device 2. The speaker 12 can be configured to notify various pieces of information by voice to users around the first terminal device 2. The specific configurations of the microphone 11 and the speaker 12 are not particularly limited, but for example, microphones built into a smartphone can be used.
[0038] The display unit 13 is an example of a user interface that can provide various information to a user or the like who uses the first terminal device 2. This display unit 13 can be a well-known display device such as a liquid crystal display (LCD) or an organic electroluminescent display (ELD).
[0039] The operation unit 14 is an example of a user interface that allows the owner of the first terminal device 2 to perform predetermined input operations. The operation unit 14 of this embodiment can be configured with a touch panel and one or more operation switches provided on a smartphone. The specific configuration of the operation unit 14 can be adjusted appropriately to suit the form of the first terminal device 2. For example, if the first terminal device 2 is a personal computer, the operation unit 14 may be a keyboard or a pointing device.
[0040] The control device 15 is a device capable of performing various controls of the first terminal device 2 and may be configured, for example, by a well-known computer. The control device 15 configured by a well-known computer may include, for example, at least one processor 21, a ROM (Read Only Memory) 22 and a RAM (Random Access Memory) 23 as examples of memory, a storage 24, a communication interface 25, and an input / output interface 26. These components may be connected to each other via an internal bus 27 so as to be able to communicate with each other.
[0041] The processor 21 may be configured, for example, as a CPU (Central Processing Unit) and may be capable of executing various programs and controlling each part. Specifically, the processor 21 may be capable of reading various control programs stored, for example, in the ROM 22 or the storage 24, and executing the programs using the RAM 23 as a work area. The processor 21 may be capable of controlling the operation of each component of the first terminal device 2, specifically the microphone 11, the speaker 12, etc., and performing various arithmetic processing in accordance with the programs.
[0042] The ROM 22 may be capable of storing various programs and various data, and the RAM 23 may be capable of temporarily storing programs or data as a working area.
[0043] The storage 24 can be configured with a recording medium such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage 24 may at least temporarily store various programs including an operating system and various data necessary to operate the first terminal device 2.
[0044] The communication interface 25 may be capable of transmitting and receiving predetermined data via wired communication using a communication cable or wireless communication based on a wireless communication standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). This communication interface 25 allows the control device 15 of the first terminal device 2 to communicate with other electronic devices, such as the voice conversation generation device 4, via the network NW.
[0045] The input / output interface 26 may be connected to various components of the first terminal device 2 and may be capable of transmitting and receiving various information, such as control signals for the components and audio signals collected by the microphone 11. Although not shown, the input / output interface 26 may further include a connector to which an external recording medium can be connected, and various drives such as a DVD drive.
[0046] The second terminal device 3 is a terminal device that conducts free conversation with the first user. Specifically, the second terminal device 3 may be a terminal device owned by a grandfather and / or grandmother, who are examples of the first user. This second terminal device 3 is configured as a terminal device different from the first terminal device 2. In this embodiment, by having the first terminal device 2 and the second terminal device 3 be different terminal devices, the added value of the synthesized speech output by the second terminal device 3 is improved.
[0047] The second terminal device 3 may be configured as a device capable of data communication, capable of at least acquiring surrounding sounds and outputting sounds. Specifically, the second terminal device 3 may be configured as a smartphone, a personal computer, a tablet terminal, a mobile phone, an interactive robot, or a smart speaker, similar to the first terminal device 2. Note that in this embodiment, as shown in FIG. 1 , an interactive robot is used as the second terminal device 3. This interactive robot may be installed, for example, in the home of the first user.
[0048] The second terminal device 3 may have the same basic configuration as the first terminal device 2. More specifically, as long as it includes at least a microphone, a speaker, and a control device, other components are not particularly limited. Therefore, the detailed structure of the second terminal device 3 will not be described here, as the description of the first terminal device 2 will be used.
[0049] FIG. 3 is a block diagram illustrating an example of the hardware configuration of the voice conversation generation device according to the embodiment of the present disclosure shown in FIG. 1 . The voice conversation generation device 4 according to the embodiment of the present disclosure is a device for generating conversation information to be used by the second terminal device 3. The voice conversation generation device 4 may be configured, for example, with one or more server computers. As shown in FIG. 3 , the voice conversation generation device 4 may include at least one processor 31, ROM 32 and RAM 33 as examples of memory, storage 34, a communication interface 35, and a timer 38. These various components may be communicatively connected to each other via an internal bus 37. Furthermore, these components, except for the timer 38, may perform substantially the same functions as the corresponding components of the control device 15 of the first terminal device 2 described above. Therefore, the details of these components will be omitted here, as the previous description will be reused. In addition to the above-described components, the voice conversation generation device 4 according to the present embodiment may further include other components, such as a user interface such as a display unit or an operation unit.
[0050] Timer 38 is a timing device that measures the passage of time, and in this embodiment, is capable of measuring the time that has elapsed since the synthetic speech was generated. Note that if there is no need to measure the time that has elapsed since the synthetic speech was generated on the voice conversation generation device 4 side, timer 38 may be omitted.
[0051] Fig. 4 is a functional block diagram showing an example of the functions of each component of the voice conversation system shown in Fig. 1. Below, the functions realized by the various components of the voice conversation system 1 having the above-mentioned hardware configuration will be described mainly with reference to Fig. 4.
[0052] 4, the voice conversation generation device 4 according to this embodiment includes at least a voice information acquisition unit 41 that acquires voice information of a specific speaker from the first terminal device 2, a voice synthesis unit 44 that generates synthetic voice from the voice information, a conversation information generation unit 46 that generates conversation information using the synthetic voice, and a conversation information transmission / reception unit 47 as an example of a conversation information transmission unit that transmits the conversation information to the second terminal device 3. In addition, the voice conversation generation device 4 according to this embodiment may include a voice information database 42 as an example of a voice information storage unit in which voice information is stored, a voice information analysis unit 43 that analyzes feature points of the voice information, and a synthetic voice database 45 in which synthetic voice is stored.
[0053] The voice information acquisition unit 41 acquires voice information including the voice of a specific speaker. Specifically, the voice information acquisition unit 41 of the present embodiment may acquire voice information transmitted from the first terminal device 2 by requesting the first terminal device 2 to transmit voice information of the specific speaker. This voice information acquisition unit 41 can be realized by the communication interface 35 or the like. In the present embodiment, an example is illustrated in which the voice information acquisition unit 41 requests voice information from the first terminal device 2. However, the first terminal device 2 may periodically collect voice information and transmit it via the voice conversation generation device 4. In this case, for example, by measuring the elapsed time since the first terminal device 2 transmitted voice information, the owner of the first terminal device 2 may voluntarily transmit voice information every time a predetermined period elapses.
[0054] The specific method of the voice information acquisition unit 41 making the request to the first terminal device 2 is not particularly limited, but examples include sending a script used to acquire the voice information by mail, or sending an email along with the text information of the script. Considering that the speaker may be a child, it is advisable to use a picture book or a simple story as the script. Using a picture book or the like as the script in this way is expected to encourage the speaker to actively participate in collecting the voice information.
[0055] It is particularly important to note that the voice information acquisition unit 41 acquires voice information from the first terminal device 2 at predetermined intervals. The reason for periodically acquiring voice information in this manner is to take into consideration that the voice tone, etc. of the speaker changes as the speaker grows. The change in voice tone, etc. will be more noticeable when the speaker is a child (e.g., a grandchild of the elderly first user) than when the speaker is an adult.
[0056] The predetermined period is preferably set to any period between 1 and 12 months, more preferably between 3 and 6 months. The reason for setting the predetermined period to several months is to take into account the period during which the voice tone of a child's voice changes as the child grows. The predetermined period is not limited to the aforementioned period. For example, when adults are included as speakers, the predetermined period can be set relatively long, for example, longer than 12 months, because the voice tone of an adult generally does not change much over a few months. Whether the predetermined period has elapsed can be determined by measuring the elapsed time from the time the synthetic speech was generated using the timer 38.
[0057] The voice information database 42 may be a storage means capable of storing voice information transmitted from the first terminal device 2 based on a request from the voice information acquisition unit 41. This voice information database 42 may be realized mainly by the storage 34. Voice information for each speaker may be stored in this voice information database 42. The voice information may also include information regarding the date and time of reception. In relation to this, the specific speaker acquired as voice information by the voice conversation generation device 4 may be one or more speakers. In relation to this, when multiple speakers are set, the timing of requesting voice information from the voice information acquisition unit 41 may be managed for each speaker.
[0058] The speech information analysis unit 43 may analyze feature points of the voice of a specific speaker contained in the speech information. Specifically, the feature points of the voice may include one or more of tone of voice, intonation, dialect (including accent), number of words, and phrasing. Extraction of these feature points may vary depending on the method of acquiring the speech information, and therefore may be determined taking into consideration the method of acquiring the speech information. For example, if the speech information to be analyzed is speech information collected by a specific speaker (e.g., a grandchild) reading aloud a script such as a picture book, the speech information analysis unit 43 may primarily extract tone of voice and intonation as feature points. On the other hand, if the speech information to be analyzed is speech information composed of, for example, a free conversation between a specific speaker and another person, the speech information analysis unit 43 may extract all of tone of voice, intonation, dialect, number of words, and phrasing as feature points.
[0059] The speech synthesis unit 44 generates synthetic speech of a specific speaker from speech information transmitted from the first terminal device 2 and stored in the speech information database 42. More specifically, the speech synthesis unit 44 may generate synthetic speech based on the analysis results of the speech information analysis unit 43. The synthetic speech generated by the speech synthesis unit 44 is generated as a speech that resembles the speaker's voice. The speech synthesis unit 44 can be mainly implemented by the processor 31. Various speech synthesis methods are known for use in the speech synthesis unit 44, and are not particularly limited. For example, a statistical model-based speech synthesis method using machine learning can be adopted. In the statistical model-based speech synthesis method, an acoustic model is generated by learning speech information and text information of a script corresponding to the speech information, and speech is inferred based on the output of the model. A hidden Markov model or a neural network model can be used as the acoustic model. Since the statistical model-based speech synthesis method is a well-known technology, a detailed description thereof will be omitted here.
[0060] The synthetic speech database 45 may be a storage means for storing information necessary for generating synthetic speech, such as the synthetic speech generated by the speech synthesis unit 44, specifically, a trained acoustic model when the speech synthesis unit 44 employs the statistical model-based speech synthesis method described above. This synthetic speech database 45, like the speech information database 42, can be realized mainly by the storage 34. Furthermore, the synthetic speech in the synthetic speech database 45 may be stored for each speaker. Furthermore, this synthetic speech may include information regarding the date and time of generation.
[0061] The conversation information generation unit 46 generates conversation information using one of multiple synthetic voices generated by the voice synthesis unit 44 at different times and stored in the synthetic voice database 45. The conversation information generation unit 46 can generate, as conversation information, information to be provided to the first user or a response to a question from the first user. For the text information constituting the conversation information, a sentence generation model (sometimes called an "AI chatbot"), such as Chat GPT, can be used. It is preferable to use a large-scale language model as this sentence generation model. Note that sentence generation models using large-scale language models are themselves well known, and therefore a detailed description thereof will be omitted.
[0062] The voice of the conversation information generated by the conversation information generation unit 46 is generated using one of the synthetic voices stored in the synthetic voice database 45. The speaker of the synthetic voice used for the conversation information may be a specific speaker selected in the first terminal device 2 or the second terminal device 3. Therefore, for example, if the voices of multiple grandchildren are registered as speakers, the voice of the desired grandchild can be heard at any time. It is preferable that the speaker of the synthetic voice used for the conversation information can be specified based on, for example, voice information acquired from the user by the second terminal device 3. Specifically, it is preferable that, for example, a grandparent who is the user can select the synthetic voice of a specific speaker by calling the name of the grandchild who is the specific speaker to the second terminal device 3.
[0063] In addition, the synthetic speech used for the conversation information may be the most recent synthetic speech among multiple synthetic speeches of a specific speaker. If the synthetic speech used to generate the conversation information is the most recent, the speech of the generated conversation information will change each time a new synthetic speech of a specific speaker is generated. Therefore, a user (e.g., a grandparent) who listens to the speech of the generated conversation information can naturally sense the growth of the specific speaker (e.g., a grandchild) from changes in the tone of voice, etc.
[0064] Furthermore, the synthetic voice used for the conversation information may be synthetic voice designated by the first terminal device 2 or the second terminal device 3. If the time when the synthetic voice was generated by the first terminal device 2 or the second terminal device 3 can be designated, changes in the speaker's tone of voice, etc., can be clearly confirmed. Furthermore, if the speaker is a grandchild, the voice of the grandchild when he or she was young can be heard at any time through a conversation with the interactive robot.
[0065] The conversation information transmitting / receiving unit 47 transmits the conversation information generated by the conversation information generating unit 46. In addition, the conversation information transmitting / receiving unit 47 may be capable of receiving conversation information transmitted from the second terminal device 3. This conversation information transmitting / receiving unit 47 can be realized mainly by the communication interface 35. The conversation information transmitted from the second terminal device 3 may be composed of, for example, a question voice from the user collected via a microphone of the second terminal device 3.
[0066] 4, the first terminal device 2 may include at least a voice information collecting unit 51 and an information transmitting / receiving unit 52. Of these, the voice information collecting unit 51 collects voice information including the voice of a specific speaker. The voice information collecting unit 51 can be realized mainly by the microphone 11 and the control device 15.
[0067] The speech information collected by the speech information collection unit 51 includes the speech of a specific speaker, specifically the first user's grandchild. The grandchild's speech is preferably composed of the grandchild reading aloud a specific script, such as a pre-prepared picture book or simple story. The content of the script may be adjusted in advance to accommodate the number of vocabulary words required for speech synthesis. Furthermore, requests from the speech information acquisition unit 41 are transmitted multiple times at predetermined intervals. However, the speech collected for each request does not need to be collected using the same script as long as the speaker is the same. Therefore, in the speech conversation system 1 according to this embodiment, different picture books may be provided to the speaker in response to requests for speech information. A speaker who is provided with new picture books on a regular basis is expected to cooperate with the collection of speech information.
[0068] The information transmission / reception unit 52 is mainly realized by the communication interface 25, and may be capable of at least receiving requests from the voice information acquisition unit 41 and transmitting the voice information collected by the voice information collection unit 51.
[0069] 4, the second terminal device 3 may include at least a conversation information collection unit 61 and an information transmission / reception unit 62. Of these, the conversation information collection unit 61 collects, as conversation information, the user's voice uttered to the second terminal device 3. This conversation information collection unit 61 can be realized mainly by a microphone and a control device.
[0070] The information transmitting and receiving unit 62 is mainly realized by a communication interface and may be capable of at least requesting conversation information from the voice conversation generation device 4 and receiving conversation information generated by the voice conversation generation device 4. The signal requesting conversation information from the voice conversation generation device 4, which is transmitted from the information transmitting and receiving unit 62, may include, but is not required to include, the user's conversation information collected by the conversation information collecting unit 61. Specifically, this may include cases where the second terminal device 3 voluntarily requests conversation information from the voice conversation generation device 4 at a specific timing, such as when it detects that a user is near the second terminal device 3 or when a predetermined time arrives.
[0071] As an option, the voice conversation generating device 4 of this embodiment may include a voice information transmitting unit 48 capable of transmitting voice information in the voice information database 42 to the first terminal device 2 or the second terminal device 3. The voice information transmitting unit 48 can transmit voice information acquired at a specific time to the first terminal device 2, for example, based on a request from the first terminal device 2. By employing this voice information transmitting unit 48, for example, the parent of a speaker who owns the first terminal device 2 can listen back to the speaker's past voice information, which can be used as a record of the child's growth.
[0072] As an option, the voice conversation system 1 of this embodiment may also have a monitoring function for monitoring the user of the second terminal device 3. Such a monitoring function is particularly effective when the user is elderly. Specifically, as one of the monitoring functions described above, the voice conversation generation device 4 of this embodiment may further include a usage history collection unit 49 that collects communication content of the conversation information transmission / reception unit 47 as usage history information of the voice conversation system 1.
[0073] The usage history collecting unit 49 may collect, as usage history information, among the communication contents of the conversation information transmitting and receiving unit, particularly conversation information acquired by the second terminal device 3. Furthermore, the usage history information collected by the usage history collecting unit 49 can be transmitted to the first terminal device 2 in response to a request from the first terminal device 2 or periodically. Therefore, a child who owns the first terminal device 2 can indirectly keep an eye on the safety of his or her elderly parent (grandparent for the grandchild) who uses the second terminal device 3.
[0074] It is known that elderly people may hesitate to install a device to monitor themselves at home, as this may feel like acknowledging their own aging. In this regard, it is conceivable that relatives, such as children, may hesitate to suggest to the elderly that they use a monitoring service out of consideration for the elderly. However, if an interactive robot that allows grandparents to feel the growth of their grandchildren through conversation is equipped with an additional monitoring function, as in the voice conversation system 1 according to the present embodiment, grandparents will hardly hesitate to install the system, and this is expected to lower the barrier to introducing a monitoring service. Furthermore, for children, an interactive robot that can converse in the voice of their grandchildren can be used to suggest the use of the system, including the monitoring function, without the aforementioned psychological considerations for their grandparents.
[0075] In the present embodiment, the monitoring function uses the usage history of the second terminal device 3 to monitor the safety of the user using the second terminal device 3, but other monitoring functions may also be provided. For example, other monitoring functions may include a function for detecting signs of dementia, brain disease, or the like in the user using the second terminal device 3 by analyzing the content of the user's speech to the second terminal device 3 and detecting whether the tone of voice or articulation has changed significantly from previous speech. Alternatively, a function for notifying the first terminal device 2 or calling an emergency service (e.g., 119) when conversation information indicating a user's request for help, such as "call an ambulance," may be obtained. Furthermore, if the usage history collection unit 49 has not recorded any usage history for several days, in addition to notifying the first terminal device 2, a function for transmitting conversation information for calling out to the user from the voice conversation generation device 4 to the second terminal device 3 may also be employed.
[0076] According to the voice conversation system 1 and voice conversation generation device 4 of the present embodiment, voice information of a specific speaker used to generate synthetic speech is periodically acquired at predetermined intervals, thereby generating synthetic speech that captures changes in the voice of a single speaker. The conversation information generated by the conversation information generation unit 46 uses synthetic speech that reflects the voice of a specific speaker at a specific point in time. Therefore, a user who hears conversation information including synthetic speech generated in this manner can recognize changes in the speaker's voice over time, enabling a conversation with a periodically changing voice. This allows for a conversation with higher added value than a conventional conversation with a monotonous voice.
[0077] Furthermore, the voice conversation system 1 and voice conversation generation device 4 having the above-described configuration can provide greater added value by, for example, using them as follows. Specifically, the second terminal device 3 is an interactive robot installed in the house where the grandparents, who are users, live, and the first terminal device 2 is a smartphone owned by a child (son or daughter) who lives far from the grandparents' house. The child then uses the smartphone to transmit the voice of the grandchild living with them as voice information to the voice conversation generation device 4 at predetermined intervals. The voice conversation system 1 according to this embodiment may be provided, for example, in the form of application software for the smartphone. Meanwhile, the grandparents set the voice of the interactive robot to that of their grandchild and converse with the interactive robot.
[0078] When the voice conversation system 1 and the voice conversation generation device 4 according to the present embodiment are used in the above-described manner, the conversation information of the interactive robot conversing with the grandparents is spoken in a synthetic voice created by the voice conversation generation device 4 based on the voice of the grandchild. As a result, the grandparents who hear the conversation information can feel as if they are conversing with the interactive robot as if they were actually having a conversation with their grandchild. Therefore, the conversation can be more enjoyable than with conventional interactive robots. In addition, if the synthetic voice used in the conversation information generation unit 46 is set to the latest synthetic voice of the grandchild, the grandparents can feel closer to the growth of their grandchild through the voice tone, etc. of the interactive robot.
[0079] Furthermore, in order to accommodate cases where grandparents have multiple grandchildren or wish to hear the voices of their children as well, it is advisable to allow multiple speakers of the synthetic voice to be registered in the voice conversation system 1 according to this embodiment. If multiple speakers of the synthetic voice are registered, the voice of the interactive robot can be switched to that of another relative depending on the grandparents' mood that day or how often they meet the speaker, or a voice randomly selected from the multiple registered voices can be reflected in the conversation information, allowing the grandparents to have more enjoyable conversations.
[0080] Furthermore, for children who own smartphones, they can periodically send voice information to inform their grandparents about the growth of their grandchildren. In addition, if the voice conversation generation device 4 includes the voice information sending unit 48, the voice information can be used as a growth record of their own child (grandchild from the grandparents' perspective). Furthermore, if the voice conversation system 1 has the above-mentioned monitoring function, it is possible to monitor the condition of the parent.
[0081] <Control Method of Voice Conversation Generation Device> Next, an example of a control method of the voice conversation generation device according to this embodiment will be described below. The following description illustrates a case in which a series of control methods are executed by the above-described voice conversation generation device 4. In this regard, the following description also serves as an explanation of the functions and effects of the above-described voice conversation generation device 4 and voice conversation system 1. The control method of the voice conversation generation device 4 described below may be implemented by executing a control program that causes at least one processor 31 of a computer constituting the voice conversation generation device 4 to perform a predetermined operation. This control program may be stored in a storage device such as the ROM 32 or the storage 34, or may be provided in the form of a computer-readable recording medium.
[0082] The control method for the voice conversation generation device according to this embodiment includes at least the steps of acquiring voice information including the voice of a specific speaker from a first terminal device 2 (step S11), generating synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device 2 (step S14), measuring the time elapsed since the synthetic voice was generated (step S16), repeatedly executing the above-described series of steps each time the elapsed time reaches a predetermined period (step S17, etc.), generating conversation information using one of multiple synthetic voices of the specific speaker generated at different times (step S23), and transmitting the generated conversation information to a second terminal device 3 configured as a terminal device different from the first terminal device 2 and capable of engaging in free conversation with a user (step S24). In the following description, the synthetic voice generation process executed between the first terminal device 2 and the conversation information generation process executed between the second terminal device 3 will be described separately.
[0083] 5 is a flowchart illustrating an example of a control method for a voice conversation generation device according to an embodiment of the present disclosure, outlining a process for generating synthetic speech. When the voice conversation generation device 4 according to this embodiment starts generating synthetic speech, the voice conversation generation device 4 first operates the voice information acquisition unit 41 to request voice information including the voice of a specific speaker from the first terminal device 2 (step S11). Upon receiving the request, the owner of the first terminal device 2 requests the specific speaker to read aloud a script previously sent by mail or email, and collects the voice of the specific speaker via the microphone 11. The voice of the specific speaker collected by the microphone 11 is converted into a voice signal by the control device 15 or the like and transmitted as voice information from the information transmission / reception unit 52 to the voice conversation generation device 4.
[0084] When the voice conversation generation device 4 acquires voice information of a specific speaker from the first terminal device 2 (Yes in step S12), it stores the voice information in the voice information database 42 in association with the speaker and the date and time of receipt (step S13), and generates synthetic voice using the voice information (step S14).
[0085] The synthetic speech generated in step S14 is stored in the synthetic speech database 45 in association with the speaker and the date and time of generation (step S15). The synthetic speech stored in the synthetic speech database 45 is available to the conversation information generation unit 46.
[0086] When the generation of the synthetic speech is completed, the timer 38 starts measuring the elapsed time since the synthetic speech was generated in order to determine when the next synthetic speech will be generated (step S16). Note that this measurement of the elapsed time is preferably performed for each speaker.
[0087] When the voice conversation generation device 4 detects that the measured elapsed time has reached a predetermined period (e.g., three months) (Yes in step S17), it returns to step S11 and repeats the series of synthetic voice generation processes.
[0088] By executing the above-described series of processes, the voice conversation generation device 4 can generate new synthetic speech using speech information containing the voice of a specific speaker at each predetermined time interval. Therefore, the synthetic speech generated by the voice conversation generation device 4 corresponds to changes in the speaker's voice over time.
[0089] 6 is a flowchart illustrating an example of a control method of a voice conversation generation device according to an embodiment of the present disclosure, which is a schematic diagram of a process for transmitting conversation information. The generation of conversation information by the voice conversation generation device 4 according to this embodiment may be initiated when a request for transmission of conversation information is received from the second terminal device 3. Note that the voice conversation generation device 4 may also generate conversation information on its own initiative, for example, in connection with the above-described monitoring function of the voice conversation generation device 4.
[0090] When the conversation information transmitting / receiving unit 47 of the voice conversation generation device 4 receives a request to transmit conversation information from the second terminal device 3 (Yes in step S21), the conversation information generation unit 46 generates conversation information based on the user's conversation information, etc., included in the transmission request. Specifically, first, a synthetic voice matching a preselected speaker and generation time is extracted from the synthetic voice database 45, and the synthetic voice to be used in generating the conversation information is determined (step S22). Next, a sentence generation model in the conversation information generation unit 46 is used to generate text information constituting the conversation information, and the conversation information is generated using the preselected synthetic voice (step S23).
[0091] The conversation information generated by the conversation information generation unit 46 is transmitted from the conversation information transmission / reception unit 47 to the second terminal device 3 (step S24). The second terminal device 3 outputs the conversation information received by the information transmission / reception unit 62 to the user using a speaker.
[0092] As described above, the control method and control program for the voice conversation generation device according to this embodiment can provide the same effects as those described above for the voice conversation system 1 according to this embodiment.
[0093] In the above embodiments, the term "processor" refers to a processor in a broad sense, and includes a general-purpose processor (e.g., a CPU, etc.) and a dedicated processor (e.g., a GPU: Graphics Processing Unit, an ASIC: Application Specific Integrated Circuit, an FPGA: Field Programmable Gate Array, a programmable logic device, etc.).
[0094] Furthermore, the operations of the processors in the above embodiments may be performed not only by a single processor but also by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processors is not limited to the order described in the above embodiments and may be changed as appropriate.
[0095] The present disclosure is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present disclosure. All of these modifications are included in the technical concept of the present disclosure. Furthermore, in the present disclosure, each component may be present in only one form or in two or more forms, provided that no contradiction occurs.
[0096] All references, including publications, patent applications, and patents, cited in this specification are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and set forth in its entirety herein.
[0097] The use of nouns and similar referents in connection with the description of this disclosure (particularly in connection with the claims that follow) shall be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The words "comprises," "has," "includes," and "comprises" shall be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise noted. The recitation of numerical ranges herein is merely intended to serve as a shorthand method for referring individually to each value falling within the range, unless otherwise indicated herein, and each value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. Any example or exemplary language used herein (e.g., "such as"), unless otherwise claimed, is intended merely to better illustrate the disclosure and does not pose a limitation on the scope of the disclosure. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the present disclosure.
[0098] Preferred embodiments of the disclosure are described herein, including the best mode known to the inventors for carrying out the disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. The inventor expects that skilled persons will apply such variations as appropriate, and intends to practice the disclosure otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, this disclosure includes any combination of the above-described elements in all variations thereof unless otherwise indicated herein or otherwise clearly contradicted by context.
Claims
1. A voice conversation generation device comprising: a voice information acquisition unit that acquires voice information including the voice of a specific speaker from a first terminal device every time a predetermined period of time elapses; a voice synthesis unit that generates a synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device; a conversation information generation unit that generates conversation information using one of a plurality of synthetic voices of the specific speaker generated by the voice synthesis unit at different times; and a conversation information transmission unit that transmits the generated conversation information to a second terminal device configured as a terminal device different from the first terminal device and capable of engaging in free conversation with a user.
2. The voice conversation generation device according to claim 1, wherein the conversation information generation unit generates conversation information using the latest synthetic voice among the multiple synthetic voices of the specific speaker generated by the voice synthesis unit.
3. The voice conversation generation device according to claim 1, wherein the conversation information generation unit generates conversation information using a synthetic voice instructed by the first terminal device or the second terminal device from among the synthetic voices of the multiple specific speakers generated by the voice synthesis unit.
4. The voice conversation generating device according to claim 1, wherein the predetermined period is set to a value between 1 and 12 months.
5. The voice conversation generation device according to claim 1, further comprising a voice information storage unit capable of storing a plurality of pieces of the voice information acquired from the first terminal device for each speaker, and the voice synthesis unit generates the synthetic voice for each of the plurality of speakers stored in the voice information storage unit.
6. The voice conversation generating device according to claim 5, wherein the conversation information generating unit generates conversation information using the synthetic voice of a speaker specified by the second terminal device among the synthetic voices of multiple speakers generated by the voice synthesis unit.
7. The voice conversation generating device according to claim 1, further comprising: a voice information storage unit capable of storing a plurality of pieces of the voice information acquired from the first terminal device; and a voice information transmission unit that transmits specific voice information stored in the voice information storage unit to the first terminal device or the second terminal device based on a request from the first terminal device or the second terminal device.
8. The voice conversation generation device according to claim 1, further comprising a usage history collection unit that collects communication content between the voice conversation generation device and the second terminal device as usage history information, and the usage history information collected by the usage history collection unit is obtainable from the first terminal device.
9. A voice conversation system comprising: a first terminal device that acquires voice information including the voice of a specific speaker; a voice conversation generation device as described in any one of claims 1 to 8; and a second terminal device that is configured as a terminal device different from the first terminal device and is capable of conducting a free conversation with a user using the conversation information generated by the voice conversation generation device.
10. A method for controlling a voice conversation generation device, comprising: acquiring voice information including the voice of a specific speaker from a first terminal device; generating a synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device; measuring the elapsed time since the synthetic voice was generated; repeatedly executing the above-mentioned series of steps each time the elapsed time reaches a predetermined period; generating conversation information using one of a plurality of synthetic voices of the specific speaker generated at different times; and transmitting the generated conversation information to a second terminal device configured as a terminal device different from the first terminal device and capable of engaging in free conversation with a user.
11. A control program for a voice conversation generation device that generates conversation information to be used in a second terminal device capable of engaging in free conversation with a user, the control program causing at least one processor of a computer provided in the voice conversation generation device to execute the following processes: acquire voice information including the voice of a specific speaker from a first terminal device configured as a terminal device different from the second terminal device; generate a synthetic voice of the specific speaker using the voice information including the voice of the specific speaker acquired from the first terminal device; measure the elapsed time since the synthetic voice was generated; repeat the above-mentioned series of steps each time the elapsed time reaches a predetermined period; generate the conversation information using one of multiple synthetic voices of the specific speaker generated at different times; and transmit the generated conversation information to the second terminal device.
Citation Information
Patent Citations
Voice synthesizer and voice synthesis program
JP2007163875A
Voice management server device, conversation voice provision method, and conversation voice provision system
JP2016110151A
Voice information replacement system and program
JP2021060484A
Voice interaction method and electronic device
US20230017274A1
