Voice recommendation device, voice recommendation method, and program

The voice recommendation device addresses the issue of individual differences in user reactions by calculating listening habits through SNS activity and recommending similar or dissimilar character voices, enhancing user engagement and absorption in content.

WO2024241541A1PCT designated stage expired Publication Date: 2024-11-28NT T INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/019380
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing voice recommendation technologies fail to account for individual differences in user reactions to their own voices, leading to discomfort and reduced engagement in content such as games, as they do not consider listening habits when recommending character voices.

Method used

A voice recommendation device that calculates the similarity between a user's voice and character voices, determining listening habits based on social networking service (SNS) activity, and recommends either similar or dissimilar character voices accordingly to enhance user experience.

Benefits of technology

This approach reduces negative reactions and increases user absorption in content by tailoring voice recommendations to individual listening habits, improving user engagement and reducing the likelihood of disengagement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023019380_28112024_PF_FP_ABST
    Figure JP2023019380_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A voice recommendation device according to one aspect disclosed herein recommends a character voice representing the voice of a character, and comprises: a similarity calculation unit that calculates the similarity between a user voice representing the voice of a user and a plurality of character voices; a determination unit that determines whether the user has a listening habit of listening to the user's own voice; and a recommendation unit that recommends a character voice similar to the user's voice when it is determined that the user has the listening habit, and recommends a character voice dissimilar to the user's voice when it is determined that the user does not have the listening habit.
Need to check novelty before this filing date? Find Prior Art

Description

Audio recommendation device, audio recommendation method, and program

[0001] The present disclosure relates to an audio recommendation device, an audio recommendation method, and a program.

[0002] Studies have been conducted on improving the user experience for content such as games by personalizing characters in the content. One of these studies claims that users can become more engrossed in a game by hearing their own voice during the game. For example, Non-Patent Document 1 shows that using a synthesized voice of the user's voice can increase the likelihood of the user becoming engrossed in the game, such as by playing the game for longer periods of time.

[0003] On the other hand, for example, Non-Patent Document 2 shows that some people feel uncomfortable when listening to a recording of their own voice.

[0004] Pucher, Michael, et al. "Influence of speaker familiarity on blind and visually impaired children's and young adults' perception of synthetic voices." Computer Speech & Language 46 (2017): 179-195. Holzman, Philip S., and Clyde Rousey. "The voice as a percept." Journal of Personality and Social Psychology 4.1 (1966): 79.

[0005] However, for example, when recommending the user's voice or a voice similar to it as the voice of a character in content to improve the user experience, individual differences in reactions to one's own voice, such as the fact that some users may find their own voice unpleasant, were not taken into consideration.

[0006] The present disclosure has been made in consideration of the above points, and provides a technology that can recommend audio to improve the user experience of content, taking into account individual differences in reactions to one's own audio.

[0007] An audio recommendation device according to one aspect of the present disclosure is an audio recommendation device that recommends character voices that represent the voice of a character, and includes: a similarity calculation unit that calculates the similarity between a user voice that represents the user's voice and each of a plurality of character voices; a determination unit that determines whether the user has a listening habit for their own voice; and a recommendation unit that recommends a character voice that is similar to the user voice if it is determined that the user has the listening habit, and recommends a character voice that is not similar to the user voice if it is not determined that the user has the listening habit.

[0008] The company will provide technology that can recommend audio to improve the user experience of content, taking into account individual differences in reactions to one's own voice.

[0009] 1 is a diagram illustrating an example of a hardware configuration of an audio recommendation device according to a first embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of an audio recommendation device according to a first embodiment. FIG. 3 is a diagram illustrating an example of a detailed functional configuration of a user information acquisition unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of a detailed functional configuration of an audio recommendation unit according to the first embodiment. FIG. 5 is a flowchart illustrating an example of a user information acquisition process according to the first embodiment. FIG. 6 is a flowchart illustrating an example of an audio recommendation process according to the first embodiment. FIG. 7 is a diagram illustrating an example of a detailed functional configuration of a user information acquisition unit according to a second embodiment. FIG. 8 is a flowchart illustrating an example of a user information acquisition process according to the second embodiment. FIG. 9 is a flowchart illustrating an example of a process for calculating a video posting frequency of a video including a user voice. FIG. 10 is a diagram illustrating an example of a detailed functional configuration of a user information acquisition unit according to a third embodiment. FIG. 11 is a diagram illustrating an example of a detailed functional configuration of an audio recommendation unit according to the third embodiment. FIG. 12 is a flowchart illustrating an example of a user information acquisition process according to the third embodiment. FIG. 13 is a flowchart illustrating an example of a process for extracting user voice and calculating a video posting frequency of a video including the user voice. FIG. 14 is a flowchart illustrating an example of an audio recommendation process according to the third embodiment.

[0010] Hereinafter, first to third embodiments will be described as embodiments of the present invention. In each of the following embodiments, a voice recommendation device 10 that can recommend voices for improving the user experience of content, taking into consideration individual differences in reactions to one's own voice, will be described.

[0011] First Embodiment First, a first embodiment will be described. One of the factors that influences differences in users' reactions to their own voice is their own listening habits. For example, people who have a certain level of habit of listening to their own recorded voice (e.g., at least once a week) are likely to have a positive reaction to their own recorded voice, such as feeling attracted to it or feeling a sense of closeness to it. On the other hand, people who do not have this habit are likely to have a negative reaction to their own recorded voice, such as feeling annoyed by it.

[0012] Therefore, the following describes a case where, when setting a voice character for content such as a game or advertisement, a user's own voice listening habits are taken into consideration to recommend (suggest) candidate voices for the content. Regarding the user's own voice listening habits, a user who links with a social networking service (SNS) account and posts videos to the SNS at a certain frequency or more is considered to have a "listening habit," while a user who does not is considered to have a "listening habit." Furthermore, regarding recommendations, a character voice similar to the user's voice from pre-recorded character voices (hereinafter also referred to as character voices) is recommended to a user with a listening habit, and a character voice not similar to the user's voice is recommended to a user without a listening habit.

[0013] This makes it possible to recommend character voices similar to a user's own voice only to users who have a positive reaction to their own voice. Therefore, compared to recommending character voices similar to a user's own voice to all users, for example, negative reactions are reduced, which is expected to result in users becoming more engrossed in the content.

[0014] <Example of Hardware Configuration of Audio Recommendation Device 10> An example of a hardware configuration of the audio recommendation device 10 according to the first embodiment is shown in Fig. 1. As shown in Fig. 1, the audio recommendation device 10 according to the first embodiment includes an input device 101, an output device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0015] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, an audio input device such as a microphone, etc. The output device 102 is, for example, a display, a display panel, an audio output device such as a speaker, etc.

[0016] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0017] The communication I / F 104 is an interface for connecting the audio recommendation device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is various arithmetic devices such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0018] 1 is an example, and the hardware configuration of the audio recommendation device 10 is not limited to this. For example, the audio recommendation device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0019] <Example of Functional Configuration of Audio Recommendation Device 10> An example of the functional configuration of the audio recommendation device 10 according to the first embodiment is shown in Fig. 2. As shown in Fig. 2, the audio recommendation device 10 according to the first embodiment has a user information acquisition unit 201 and an audio recommendation unit 202. These units are realized, for example, by processing in which one or more programs installed in the audio recommendation device 10 are executed by the processor 108 or the like. The audio recommendation device 10 according to the first embodiment also has a character audio DB 203. The character audio DB 203 is realized, for example, by the auxiliary storage device 107 or a storage device included in a database server connected via a communication network.

[0020] The user information acquisition unit 201 acquires user information in response to a content usage start instruction given to the audio recommendation device 10. Here, the content usage start instruction is an instruction indicating the start of content usage. This content usage start instruction starts setting the character voice for the content. The content usage start instruction is given to the audio recommendation device 10 by, for example, a user operation on the input device 101 or a terminal device connected via a communication network. Furthermore, the user information is information including the user's own voice (hereinafter also referred to as user voice) and the frequency of video posting for determining the user's listening habits. The user information acquisition unit 201 will be described in detail later.

[0021] The voice recommendation unit 202 recommends to the user one of the character voices stored in the character voice DB 203, using the user information acquired by the user information acquisition unit 201. Details of the voice recommendation unit 202 will be described later.

[0022] The character voice DB 203 stores the voices of multiple characters. In the following, it is assumed that N character voices are stored in the character voice DB 203. Here, the character voice refers to the voice of a specific character (speaker). In other words, the character voice DB 203 stores the character voices of N characters (speakers). It is desirable that N be, for example, several tens or more.

[0023] <<Detailed Functional Configuration Example of User Information Acquisition Unit 201>> A detailed functional configuration example of the user information acquisition unit 201 according to the first embodiment is shown in Fig. 3. As shown in Fig. 3, the user information acquisition unit 201 according to the first embodiment includes a voice recording unit 211, an SNS information acquisition unit 212, and a user information creation unit 213.

[0024] The voice recording unit 211 records the user's voice to generate user voice. The SNS information acquisition unit 212 uses the user's SNS account to calculate the video posting frequency of the SNS account. The user information creation unit 213 creates user information including the user voice generated by the voice recording unit 211 and the video posting frequency calculated by the SNS information acquisition unit 212.

[0025] <<Detailed Functional Configuration Example of Audio Recommendation Unit 202>> A detailed functional configuration example of the audio recommendation unit 202 according to the first embodiment is shown in Fig. 4. As shown in Fig. 4, the audio recommendation unit 202 according to the first embodiment includes a similarity calculation unit 221, a determination unit 222, and a recommendation unit 223.

[0026] The similarity calculation unit 221 calculates the similarity between the user voice included in the user information acquired by the user information acquisition unit 201 and each of the N character voices stored in the character voice DB 203. For example, the similarity calculation unit 221 may calculate, for each character voice, the cosine similarity between the speaker vector of the user voice and the speaker vector of the character voice. Here, the speaker vector is a vector that represents the characteristics (speaker information) of the speaker of the voice. For example, a speaker vector obtained by a neural network (e.g., x-vector (Reference 1)) may be used as the speaker vector.

[0027] The determination unit 222 determines whether the video posting frequency included in the user information acquired by the user information acquisition unit 201 is equal to or greater than a predetermined threshold. If the video posting frequency is equal to or greater than the predetermined threshold, it means that the user has a listening habit, and if not, it means that the user does not have a listening habit.

[0028] The recommendation unit 223 recommends to the user either a character voice similar to the user voice or a character voice dissimilar to the user voice, depending on the determination result by the determination unit 222. That is, the recommendation unit 223 recommends a character voice similar to the user voice if the determination result by the determination unit 222 is "the video posting frequency is equal to or greater than a predetermined threshold," and otherwise recommends a character voice dissimilar to the user voice.

[0029] <User Information Acquisition Process> An example of the user information acquisition process according to the first embodiment will be described below with reference to Fig. 5. In the following, it is assumed that a content usage start instruction is given to the audio recommendation device 10 by the user.

[0030] In response to the instruction to start using the content, the voice recording unit 211 of the user information acquisition unit 201 records the user's voice to generate user voice (step S101). For example, similar to Reference 2, the voice recording unit 211 may generate user voice by requesting the user to read out the specified utterance content for about 5 seconds as if talking to a character, and recording the user's voice in response to that. However, this is just one example, and the voice recording unit 211 may generate or acquire user voice by other methods.

[0031] The SNS information acquisition unit 212 of the user information acquisition unit 201 calculates the video posting frequency of the user's SNS account using the user's SNS account (step S102). For example, the SNS information acquisition unit 212 may extract the posting dates and times of videos in the user's SNS account by web scraping or the like, and calculate the video posting frequency from those posting dates and times. The video posting frequency may be, for example, the average number of videos posted per predetermined period (e.g., one week), the number of videos posted in a recent predetermined period (e.g., the most recent week), or the like. In the following, as an example, the video posting frequency is assumed to be the number of videos posted in the most recent week.

[0032] The user's SNS account may be specified by the user when issuing a command to start using the content, or may be acquired by any other method. The type of SNS is not particularly limited, and multiple SNS accounts may be specified or acquired. However, it is preferable that the SNS account be an SNS account that the user uses on a daily basis.

[0033] The user information creation unit 213 of the user information acquisition unit 201 creates user information including the user voice generated in the above step S101 and the video posting frequency calculated in the above step S102 (step S103). This allows the user information including the user voice and the video posting frequency to be obtained.

[0034] <Audio Recommendation Processing> An example of the audio recommendation processing according to the first embodiment will be described below with reference to Fig. 6. Hereinafter, the video posting frequency is denoted by f, and the user voice is denoted by a. u, character voices stored in the character voice DB 203 are a c,i (1≦i≦N).

[0035] The similarity calculation unit 221 of the voice recommendation unit 202 calculates the similarity between the user voice a u The speaker vector of the user voice a is calculated (step S201). u The speaker vector of v u Let's say.

[0036] The similarity calculation unit 221 of the voice recommendation unit 202 calculates the similarity of the character voice a stored in the character voice DB 203. c,i (1≦i≦N) c,i is acquired (step S202).

[0037] Next, the similarity calculation unit 221 of the voice recommendation unit 202 calculates the similarity between the character voice a acquired in step S202 and the character voice a c,i The speaker vector of the character voice a is calculated (step S203). c,i The speaker vector of v c,i Let's say.

[0038] Next, the similarity calculation unit 221 of the audio recommendation unit 202 calculates the speaker vector v u and the speaker vector v c,i (Step S204) calculates the similarity (for example, cosine similarity) between the speaker vector v u and the speaker vector v c,i The similarity between i Let's say.

[0039] Next, the similarity calculation unit 221 of the voice recommendation unit 202 calculates the similarity between the character voice a stored in the character voice DB 203 and the character voice a c,i (1≦i≦N) c,i It is determined whether there is a

[0040] In step S205, the character voice a c,i If it is determined that there is a character voice a, the similarity calculation unit 221 of the voice recommendation unit 202 returns to step S202.c,i The above steps S202 to S204 are repeated until the speaker vector v is acquired. u and each speaker vector v c,i The similarity S between each of i (1≦i≦N) is obtained.

[0041] On the other hand, the character voice a that has not been acquired in step S205 c,i If it is not determined that there is a video posting frequency f, the determination unit 222 of the audio recommendation unit 202 determines whether the video posting frequency f is equal to or greater than a predetermined threshold value F (step S206). The value of the threshold value F can be set arbitrarily, but for example, it is conceivable to set F=1, with "posting videos to SNS at least once a week" as the threshold value.

[0042] If it is determined in step S206 above that the video posting frequency f is equal to or greater than the threshold value F, the recommendation unit 223 of the audio recommendation unit 202 selects the video with the highest similarity S i Character voice a with a large c,i to the user (step S207). That is, the recommendation unit 223 recommends i←argmax{S i} as character voice a c,i In this way, for users who are deemed to have a "listening habit," character voices similar to the user's voice are recommended.

[0043] On the other hand, if it is not determined in step S206 that the video posting frequency f is equal to or greater than the threshold value F, the recommendation unit 223 of the audio recommendation unit 202 calculates the similarity S i Small character voice a c,i For example, the recommendation unit 223 recommends the item with the highest similarity S i Small character voice a c,i However, this is just an example, and the recommendation unit 223 may use other methods to recommend the similarity S i Small character voice a c,i Select the character voice a c,iFor example, the recommendation unit 223 may recommend a certain threshold value (e.g., similarity S i (1≦i≦N)) and the most similarity S i Character voice a with a large c,i Alternatively, for example, the recommendation unit 223 may recommend to the user the similarity S i is a certain threshold (e.g., similarity S i (1≦i≦N) c,i Randomly selected character voice a c,i In this way, a character voice that is not similar to the user voice is recommended to a user who is deemed to have "no listening habit."

[0044] [Second Embodiment] Next, a second embodiment will be described. In the first embodiment, a character voice similar to the user's voice is recommended if the user frequently posts videos to their SNS account, and otherwise a character voice dissimilar to the user's voice is recommended. However, because the videos posted by the user to their SNS account do not necessarily include the user's own voice, the first embodiment may not fully take into account the user's listening habits.

[0045] Therefore, in the second embodiment, we will explain a case where the user's listening habits are determined by taking into account whether or not the user's own voice is included in videos posted to the user's SNS account, and character voices are recommended.

[0046] In the second embodiment, differences from the first embodiment will be mainly described, and a description of components that may be the same as those in the first embodiment will be omitted. In other words, points that are not particularly mentioned in the second embodiment may be the same as those in the first embodiment.

[0047] <Example of Functional Configuration of Audio Recommendation Device 10> The audio recommendation device 10 according to the second embodiment differs from the first embodiment in the user information acquisition unit 201.

[0048] <<Detailed Functional Configuration Example of User Information Acquisition Unit 201>> A detailed functional configuration example of the user information acquisition unit 201 according to the second embodiment is shown in Fig. 7. As shown in Fig. 7, the user information acquisition unit 201 according to the second embodiment includes a voice recording unit 211, an SNS information acquisition unit 212, and a user information creation unit 213, similar to the first embodiment.

[0049] As in the first embodiment, the voice recording unit 211 records the user's voice to generate user voice. The SNS information acquisition unit 212 uses the user's SNS account to calculate the video posting frequency for that SNS account. At this time, the SNS information acquisition unit 212 calculates the video posting frequency for videos that include the user's own voice. As in the first embodiment, the user information creation unit 213 creates user information that includes the user voice generated by the voice recording unit 211 and the video posting frequency calculated by the SNS information acquisition unit 212.

[0050] <User Information Acquisition Process> An example of the user information acquisition process according to the second embodiment will be described below with reference to Fig. 8. In the following, it is assumed that a content usage start instruction is given to the audio recommendation device 10 by the user.

[0051] In response to the instruction to start using the content, the voice recording unit 211 of the user information acquisition unit 201 records the user's voice and outputs the user voice a, similar to step S101 in FIG. u is generated (step S301).

[0052] The SNS information acquisition unit 212 of the user information acquisition unit 201 calculates the video posting frequency f of videos including the user's voice using the user's SNS account (step S302). Note that the process of calculating the video posting frequency f in this step will be described in detail later.

[0053] The user information generation unit 213 of the user information acquisition unit 201 generates the user voice a generated in the above step S301 in the same manner as in step S103 of FIG. u and the video posting frequency f calculated in step S302 above (step S303).u and video posting frequency f are obtained.

[0054] <Process for Calculating Video Posting Frequency of Video Including User Voice> An example of the process for calculating the video posting frequency f of videos including user voice in step S302 of FIG. 8 will now be described with reference to FIG. 9 .

[0055] The SNS information acquisition unit 212 calculates the number of videos containing user voices in the most recent week as N s As N s = 0 (step S401).

[0056] The SNS information acquisition unit 212 uses the user's SNS account to acquire videos posted to the SNS account in the past week by web scraping or the like (step S402).

[0057] The SNS information acquisition unit 212 determines whether or not one or more videos have been acquired in step S402 (step S403).

[0058] If it is determined in step S403 that one or more videos have been acquired, the SNS information acquisition unit 212 acquires one video that has not yet been acquired from the videos acquired in step S402 (step S404). Hereinafter, the video acquired in step S402 is referred to as m. i (1≦i≦M), and one video m that has not been acquired in this step i is deemed to have been obtained.

[0059] Next, the SNS information acquisition unit 212 receives the user voice a u The speaker vector v u and the moving image m acquired in step S404 above. i and the speaker vector of the voice contained in the video m are calculated (step S405). i The speaker vector of the speech contained in i The SNS information acquisition unit 212 may calculate the speaker vector using the same method as the similarity calculation unit 221.

[0060] Next, the SNS information acquisition unit 212 calculates the speaker vector v uand the speaker vector w i (Step S406) calculates the similarity (for example, cosine similarity) between the speaker vector v u and the speaker vector w i The similarity between i Let's say.

[0061] Next, the SNS information acquisition unit 212 calculates the similarity s i is equal to or greater than a predetermined threshold (step S407). When cosine similarity is used as the similarity, the threshold can be set to any value equal to or greater than 0.5, for example.

[0062] In step S407, the similarity s i is determined to be equal to or greater than the threshold value, the SNS information acquisition unit 212 s (Step S408). For simplicity, in this embodiment, i It is assumed that there is only one audio section included in the video m. i If it is included in the video m i and a speaker vector of the speech of at least one speech interval of the user speech a u The speaker vector v u When the similarity with N is equal to or greater than a predetermined threshold value, s 1 can be added to

[0063] In step S407, the similarity s i If it is not determined that the threshold value is not less than the threshold value or after step S408, the SNS information acquisition unit 212 i (1≦i≦M) contains unobtained videos m i It is determined whether there is a

[0064] In step S409, the video m i If it is determined that there is a video m, the SNS information acquisition unit 212 returns to step S404. i The above steps S404 to S408 are repeated until the number of the acquired data is reached.

[0065] On the other hand, in step S409, the video m i If it is not determined that there is a video or if it is not determined that one or more videos have been acquired in step S403, the SNS information acquisition unit 212 s As a result, the video upload frequency f is output as the user voice (more precisely, the user voice a u The video posting frequency f of videos containing audio similar to the above is obtained.

[0066] [Third Embodiment] Next, a third embodiment will be described. In the second embodiment, a case where the video posting frequency f of videos including the user's own voice is calculated has been described. However, when recommending a character voice, the user voice a is used in the same way as in the first embodiment. u The speaker vector v u and each character's voice a c,i The speaker vector v c,i On the other hand, the similarity between the user voice a u In addition, it may be possible to calculate a more accurate speaker vector by using the user's own voice contained in videos posted to their SNS account.

[0067] Therefore, in the third embodiment, we will explain a case where the speaker vector of the user's voice is calculated using the user's own voice contained in videos posted to the user's SNS account, and character voices are recommended depending on whether or not the user has listening habits.

[0068] In the third embodiment, differences from the first and second embodiments will be mainly described, and a description of components that may be the same as those in the first or second embodiment will be omitted. In other words, points that are not particularly mentioned in the third embodiment may be the same as those in the first or second embodiment.

[0069] <Example of Functional Configuration of Audio Recommendation Device 10> The audio recommendation device 10 according to the third embodiment differs from the first and second embodiments in the user information acquisition unit 201 and the audio recommendation unit 202.

[0070] <<Detailed Functional Configuration Example of User Information Acquisition Unit 201>> A detailed functional configuration example of the user information acquisition unit 201 according to the third embodiment is shown in Fig. 10. As shown in Fig. 10, the user information acquisition unit 201 according to the third embodiment includes a voice recording unit 211, an SNS information acquisition unit 212, and a user information creation unit 213, similar to the first and second embodiments.

[0071] The voice recording unit 211 records the user's voice to generate user voice, as in the first and second embodiments. The SNS information acquisition unit 212 uses the user's SNS account to calculate the video posting frequency of videos containing the user's own voice, as in the second embodiment. The SNS information acquisition unit 212 also acquires (extracts) the user's own voice (user voice) included in the videos. The user information creation unit 213 creates user information that includes the user voice generated by the voice recording unit 211, the video posting frequency calculated by the SNS information acquisition unit 212, and the user voice acquired by the SNS information acquisition unit 212.

[0072] <<Detailed Functional Configuration Example of Audio Recommendation Unit 202>> A detailed functional configuration example of the audio recommendation unit 202 according to the third embodiment is shown in Fig. 11. As shown in Fig. 11, the audio recommendation unit 202 according to the second embodiment includes a similarity calculation unit 221, a determination unit 222, and a recommendation unit 223, similar to the first and second embodiments.

[0073] The similarity calculation unit 221 calculates the similarity (for example, cosine similarity) between the average of the speaker vectors of the user voices included in the user information acquired by the user information acquisition unit 201 and the speaker vectors of each character voice. Note that the determination unit 222 and the recommendation unit 223 are the same as those in the first and second embodiments.

[0074] <User Information Acquisition Process> An example of the user information acquisition process according to the third embodiment will be described below with reference to Fig. 12. Note that, in the following, it is assumed that a content usage start instruction is given to the audio recommendation device 10 by the user.

[0075] 5, the voice recording unit 211 of the user information acquisition unit 201 records the user's voice in response to the instruction to start using the content, and generates a user voice (step S501). Hereinafter, the user voice generated in this step will be referred to as a u (0) Let's say.

[0076] The SNS information acquisition unit 212 of the user information acquisition unit 201 uses the user's SNS account to extract user voices included in videos posted to the SNS account, and calculates the video posting frequency f of the videos including the user voices (step S502). Details of the process of extracting user voices and calculating the video posting frequency f in this step will be described later. Hereinafter, when user voices are extracted in this step, the user voices are referred to as a u (1) , ..., a u (K) However, it should be noted that there may be cases where the user's voice is not extracted in this step.

[0077] The user information generation unit 213 of the user information acquisition unit 201 receives the user voice a generated in step S501. u (0) the video posting frequency f calculated in step S502 above, and the user voice a if the user voice is extracted in step S502 above. u (1) , ..., a u (K) (Step S503) u (k) (k=0, . . . , K; where K is the number of user voices extracted in step S502 above) and video posting frequency f are obtained.

[0078] <Process for extracting user voice and calculating the video posting frequency of videos including the user voice> An example of the process for extracting user voice and calculating the video posting frequency of videos including the user voice in step S502 of FIG. 12 will now be described with reference to FIG. 13 .

[0079] The SNS information acquisition unit 212 calculates the number of videos containing user voices in the most recent week as N s As N s = 0 (step S601).

[0080] The SNS information acquisition unit 212 uses the user's SNS account to acquire videos posted to the SNS account in the past week by web scraping or the like (step S602).

[0081] The SNS information acquisition unit 212 determines whether or not one or more videos have been acquired in step S602 (step S603).

[0082] If it is determined in step S603 that one or more videos have been acquired, the SNS information acquisition unit 212 acquires one video that has not yet been acquired from the videos acquired in step S602 (step S604). Hereinafter, the video acquired in step S602 is referred to as m. i (1≦i≦M), and one video m that has not been acquired in this step i is deemed to have been obtained.

[0083] Next, the SNS information acquisition unit 212 acquires the video m acquired in step S604. i For simplicity, the audio section included in the video m is extracted (step S605). i In this step, L (i) speech segments T j (1≦j≦L (i) ) is assumed to be extracted.

[0084] Next, the SNS information acquisition unit 212 receives the user voice a u (0) The speaker vector v u (0) and the moving image m acquired in step S604 above. i Each voice section T j (1≦j≦L (i) ) and the speaker vector of the speech in the speech section T j The speaker vector of the speech is w ij (1≦j≦L(i) ) The SNS information acquisition unit 212 may calculate the speaker vector using the same method as the similarity calculation unit 221.

[0085] Next, the SNS information acquisition unit 212 calculates the speaker vector v u (0) and the speaker vector w ij (1≦j≦L (i) ) and calculate the similarity (for example, cosine similarity) between them (step S607). u (0) and the speaker vector w ij The similarity between ij Let's say.

[0086] Next, the SNS information acquisition unit 212 calculates the similarity s ij (1≦j≦L (i) ) with a similarity s equal to or greater than a predetermined threshold. ij It is determined whether or not there exists a cosine similarity (step S608). When the cosine similarity is used as the similarity, the threshold value can be set to any value equal to or greater than 0.5, for example.

[0087] In step S608, the similarity s ij If it is determined that there is a s (step S609).

[0088] Next, the SNS information acquisition unit 212 acquires the similarity s ij The speech interval T j The speech of the speech having the similarity s equal to or greater than the threshold value is output (step S610). ij If there are multiple similarities s ij For each of the ij The speech interval T j The audio to be output is:

[0089] Following step S610 or in step S608, the similarity s equal to or greater than the threshold is ijIf it is determined that the video m acquired in step S602 does not exist, the SNS information acquisition unit 212 i (1≦i≦M) contains unobtained videos m i It is determined whether there is a

[0090] In step S611, the video m i If it is determined that there is a video m, the SNS information acquisition unit 212 returns to step S604. i The above steps S604 to S610 are repeated until all the parameters have been acquired.

[0091] On the other hand, in step S611, the video m i If it is not determined that there is any video, or if it is not determined that one or more videos have been acquired in step S603, the SNS information acquisition unit 212 s It is determined whether or not is equal to or greater than 1 (step S612).

[0092] N in step S612 above s If it is determined that is 1 or more, the SNS information acquisition unit 212 converts each of the voices to be output in step S610 into user voice a u (k) (k=1, . . . , K) (step S613), where K is the total number of sounds targeted for output in S610.

[0093] Following step S613 or in step S612 above, N s If it is not determined that f is 1 or more, the SNS information acquisition unit 212 s As a result, the video upload frequency f is calculated as the user voice (more precisely, the user voice a u (0) The frequency f of videos containing similar audio to the video is obtained. s If ≧1, the user voice a u (k) (k=1, . . . , K) are also obtained.

[0094] <Audio Recommendation Processing> Hereinafter, an example of audio recommendation processing according to the third embodiment will be described with reference to FIG.

[0095] The similarity calculation unit 221 of the voice recommendation unit 202 calculates the similarity between the user voice a u (k) Calculate the average of the speaker vectors of (k=0, . . . , K) (step S701). u (k) The speaker vector of v u (k) Then, the similarity calculation unit 221 calculates the speaker vector v u (k) Calculate the average vector of v u Steps S702 to S708 are executed as follows: Steps S702 to S708 are similar to steps S202 to S208 in Fig. 6, respectively, and therefore a description thereof will be omitted.

[0096] [Summary] As described above, the audio recommendation device 10 according to the first to third embodiments determines whether a user has a listening habit based on the frequency of posting videos to a social media account, based on the knowledge that a user's listening habit affects individual differences in responses to that user's listening habit. As a result, the audio recommendation device 10 according to the first to third embodiments can recommend character voices similar to the user's voice to a user with a listening habit, while recommending character voices dissimilar to the user's voice to a user without a listening habit. Therefore, for example, with regard to content such as games, it is expected that users with a listening habit can be more engrossed in the content and become more interested in the content, while reducing the likelihood that users without a listening habit will feel uncomfortable and stop using the content.

[0097] The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0098] [References] Reference 1: David Snyder et al.: X-vectors: Robust DNN Embeddings for Speaker Recognition, 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 2018. Reference 2: Kao, Dominic, et al. "The effects of a self-similar avatar voice in educational games." Proceedings of the ACM on Human-Computer Interaction 5.CHI PLAY (2021): 1-28.

[0099] 10 Audio recommendation device 101 Input device 102 Output device 103 External I / F 103a Recording medium 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage device 108 Processor 109 Bus 201 User information acquisition unit 202 Audio recommendation unit 203 Character audio DB 211 Audio recording unit 212 SNS information acquisition unit 213 User information creation unit 221 Similarity calculation unit 222 Determination unit 223 Recommendation unit

Claims

1. A voice recommendation device that recommends a character voice representing a character's voice, comprising: a similarity calculation unit that calculates a similarity between a user voice representing a user's voice and each of a plurality of character voices; a determination unit that determines whether the user has a listening habit for his or her own voice; and a recommendation unit that recommends a character voice similar to the user voice if it is determined that the user has the listening habit, and recommends a character voice that is not similar to the user voice if it is not determined that the user has the listening habit.

2. The audio recommendation device of claim 1, wherein the judgment unit judges that the user has the listening habit if the user's video posting frequency to a social networking service is equal to or greater than a predetermined threshold, and does not judge that the user has the listening habit if the video posting frequency is less than the threshold.

3. The audio recommendation device according to claim 2, wherein the video posting frequency is the posting frequency of videos including the user's audio.

4. The voice recommendation device of claim 3, wherein the similarity calculation unit calculates a similarity between a voice representing an average of the user's voice and the user voice included in the video and each of the plurality of character voices, and the recommendation unit recommends a character voice similar to the voice representing the average if it is determined that the user has the listening habit, and recommends a character voice not similar to the voice representing the average if it is not determined that the user has the listening habit.

5. The audio recommendation device according to any one of claims 2 to 4, wherein the video posting frequency is an average number of videos posted by the user to the social networking service per week.

6. A voice recommendation method in which a voice recommendation device that recommends a character voice representing a character's voice executes the following steps: a similarity calculation step of calculating a similarity between a user voice representing a user's voice and each of a plurality of character voices; a determination step of determining whether the user has a listening habit for his or her own voice; and a recommendation step of recommending a character voice similar to the user voice if it is determined that the user has the listening habit, and recommending a character voice not similar to the user voice if it is not determined that the user has the listening habit.

7. A program for causing a computer to function as the audio recommendation device according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Computer program, server device and method

    JP2021092644A

  • Electronic apparatus and controlling method thereof

    US20220148562A1