Speaker anonymization device, speaker anonymization method, speaker anonymization program, learning device, learning method, and learning program
The speaker anonymization device uses a neural network to generate anonymized voice data by processing speaker vectors, addressing the vulnerabilities of existing technologies by maintaining voice data usability and robustness against re-identification attacks through effective anonymization.
Patent Information
- Application Number
- PCT/JP2024/018985
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-05-23
- Publication Date
- 2024-12-05
AI Technical Summary
Existing speaker anonymization technologies compromise voice data usability and are vulnerable to re-identification attacks, as they impair the recognition accuracy and analysis accuracy of voice data and fail to robustly protect against identification attempts.
A speaker anonymization device and method that decomposes voice data into pitch, utterance content, and speaker vector, using a trained neural network to generate a speaker anonymization vector, ensuring the anonymized voice data meets specific conditions to maintain usability and robustness against re-identification attacks.
The solution effectively enhances the robustness of voice data against re-identification attacks while maintaining its usability and convenience, ensuring the anonymized voice data is distinct from original and other speaker vectors, and has a human-like distribution, thus preventing speaker identification.
Smart Images

Figure JP2024018985_05122024_PF_FP_ABST
Abstract
Description
Speaker anonymization device, speaker anonymization method, speaker anonymization program, learning device, learning method, and learning program
[0001] The present disclosure relates to a speaker anonymization device, a speaker anonymization method, a speaker anonymization program, a learning device, a learning method, and a learning program.
[0002] One known "speaker anonymization" technology that processes voice data so that individual speakers (e.g., names) cannot be identified from the voice data is a technology that performs signal processing on the voice data and randomly converts the spectral envelope and formants.
[0003] However, the above-mentioned technology has problems such as the possibility of impairing the usability of voice data and vulnerability to re-identification attacks. The impairing usability of voice data means that when various processes such as recognition processing and emotion analysis processing are performed on voice data converted using speaker anonymization technology, the recognition accuracy and analysis accuracy are reduced compared to the recognition accuracy and analysis accuracy for the original voice data. A re-identification attack refers to an act in which a third party attempts to identify an individual speaker from publicly available voice data of a speaker who does not wish to be identified from the voice data.
[0004] Japanese Patent Application Laid-Open No. 2004-266746
[0005] In response to this, the applicant of the present application has focused on a method of generating anonymized voice data by decomposing voice data into pitch (fundamental frequency), speech content, and speaker vectors, and performing speaker anonymization processing on the speaker vectors, and is considering using this method to solve the above-mentioned problems.
[0006] The present disclosure provides a speaker anonymization technique that improves robustness against re-identification attacks while maintaining the usability of voice data.
[0007] According to one aspect, a speaker anonymization device for generating anonymized speech data includes a trained neural network that performs household transformation processing, and a generation unit that generates a speaker anonymization vector by inputting a speaker vector extracted from speech data to be anonymized, into the trained neural network that has been trained using a predetermined loss function.
[0008] According to the present disclosure, it is possible to provide a speaker anonymization technology that improves robustness against re-identification attacks while maintaining the convenience of voice data.
[0009] FIG. 1 is a diagram illustrating an application example of speaker anonymization technology. FIG. 2 is a diagram illustrating an example of the hardware configuration of a server device. FIG. 3 is a diagram for explaining an overview of speaker anonymization technology. FIG. 4 is a diagram illustrating an example of the functional configuration of a server device. FIG. 5 is a diagram illustrating details of the functional configuration of a learning unit. FIG. 6 is a diagram illustrating a specific example of processing by an NN unit. FIG. 7 is a diagram illustrating a specific example of processing by a loss function calculation unit. FIG. 8 is a diagram illustrating the relationship between a speaker vector and a speaker anonymization vector. FIG. 9 is an example of a flowchart illustrating the flow of learning processing by the learning unit. FIG. 10 is a diagram illustrating details of the functional configuration of a generation unit and a provision unit. FIG. 11 is a flowchart illustrating the flow of generation processing by the generation unit. FIG. 12 is a flowchart illustrating the flow of provision processing by the provision unit. FIG. 13 is a diagram illustrating an example of the hardware configuration of a terminal. FIG. 14 is a diagram illustrating an example of the functional configuration of a terminal. FIG. 15 is an example of a flowchart illustrating the flow of anonymized audio data generation processing by an anonymized audio data generation unit. FIG. 16 is a diagram illustrating characteristics of anonymized audio data. FIG. 17 is a diagram illustrating an overview of evaluation indices for anonymized audio data. Fig. 18 is a diagram showing a method of an experiment to verify the robustness of anonymized voice data against re-identification attacks. Fig. 19 is a diagram showing the results of an experiment to verify the robustness of anonymized voice data against re-identification attacks. Fig. 20 is a diagram showing a method of an experiment to verify the usability of anonymized voice data. Fig. 21 is a diagram showing the results of an experiment to verify the usability of anonymized voice data.
[0010] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.
[0011] [First embodiment] <Application example of speaker anonymization technology> First, an application example of speaker anonymization technology will be described. Fig. 1 is a diagram showing an application example of speaker anonymization technology. In the first embodiment, a network system 100 is an example of a system to which speaker anonymization technology is applied, and includes a server device 110. In the first embodiment, the speaker anonymization technology is provided via the server device 110.
[0012] The server device 110 is an example of a speaker anonymization device. As shown in Fig. 1 , various types of voice data are transmitted and received over a network 180 to which the server device 110 is connected. The server device 110 provides users with a speaker anonymization technology to prevent individual speakers from being identified from the voice data transmitted and received over the network 180.
[0013] As shown in FIG. 1, the following cases are possible for audio data to be transmitted and received via the network 180.
[0014] As an example, a case can be considered in which an SNS user 131 who uses an SNS (Social Networking Service) provided by the server device 120 uploads audio-accompanied videos to the server device 120. Specifically, the SNS user 131, who does not wish to be identified as an individual speaker, uploads audio-accompanied videos to the server device 120 via the terminal 130 (another example of a speaker anonymization device). In this case, the uploaded audio-accompanied videos will be viewed by an unspecified number of viewers (not shown) via the network 180.
[0015] In such a case, the server device 110 executes a providing process for providing the speaker anonymization technology to the user. Specifically, upon receiving a request from the terminal 130, the server device 110 provides an anonymized audio data generation program (described in detail below) to the terminal 130. This allows the SNS user to convert audio data included in the audio-accompanied video into anonymized audio data using the anonymized audio data generation program, and then upload the audio-accompanied video.
[0016] As another example, a case may be considered in which voice-attached personal information is leaked from the server device 140 that manages a voice-attached personal information database in which voice data of various registrants is associated with the personal information of the registrants. In this case, the leaked voice-attached personal information may be misused, for example, for a re-identification attack.
[0017] In preparation for such a case, the voice DB (Database) administrator 141 transmits a conversion request to the server device 110 to convert voice data included in the voice-attached personal information managed by the server device 140 into anonymized voice data in advance. As a result, the server device 110 executes a provision process to provide the speaker anonymization technology to the user. Specifically, the server device 110 executes an anonymized voice data generation program, converts the voice data included in the conversion request into anonymized voice data, and transmits it to the server device 140. As a result, the server device 140 becomes able to manage the voice-attached personal information database converted into anonymized voice data.
[0018] Another example is a case in which digital data for broadcasting is distributed from broadcasting equipment 160 in the broadcasting industry, and the digital data for broadcasting contains audio data of a speaker who does not want to be identified. In this case, the distributed digital data for broadcasting is viewed by an unspecified number of viewers (not shown) via network 180. Note that an example of a speaker whose audio data is included in the digital data for broadcasting and who does not want to be identified is a speaker who is being interviewed for a report.
[0019] In such a case, the server device 110 executes a providing process for providing the speaker anonymization technology to a user. Specifically, upon receiving a request from a terminal 150 (another example of a speaker anonymization device) of a broadcaster 151, the server device 110 provides an anonymized voice data generation program (described in detail below) to the terminal 150. This enables the broadcaster 151 to convert voice data included in digital data for broadcasting into anonymized voice data using the anonymized voice data generation program, and then distribute the digital data for broadcasting.
[0020] On the other hand, as shown in Fig. 1, there is a possibility that a terminal 170 of a re-identification attacker 171 is connected to the network 180. The re-identification attacker 171 is a person who carries out a re-identification attack, and in the case of Fig. 1, includes a person who attempts to identify the SNS user 131 based on audio-accompanied video that the SNS user 131 has uploaded to the server device 120. Alternatively, the re-identification attacker 171 includes a person who attempts to identify a speaker based on digital data for broadcasting distributed by the broadcast equipment 160.
[0021] It is assumed that the re-identification attacker 171 already has a database in which voice data is associated with personal information of the speaker of the voice data.
[0022] In such a situation, let us suppose that the SNS user 131 has not converted the audio data included in the audio-accompanied video uploaded to the server device 120 into anonymized audio data. In this case, the re-identification attacker 171 can easily identify the individual speaker from the audio data included in the audio-accompanied video. On the other hand, let us suppose that the SNS user 131 has converted the audio data included in the audio-accompanied video into anonymized audio data, as described above. In this case, the SNS user 131 can avoid the individual speaker being easily identified by a re-identification attack by the re-identification attacker 171.
[0023] Similarly, suppose that the broadcaster 151 did not convert the audio data contained in the distributed digital data for broadcasting into anonymized audio data. In this case, a re-identification attacker 171 could easily identify an individual speaker from the audio data contained in the distributed digital data for broadcasting. On the other hand, suppose that the broadcaster 151 converted the audio data contained in the digital data for broadcasting into anonymized audio data, as described above. In this case, it is possible to prevent a re-identification attack by the re-identification attacker 171 from identifying an individual speaker from the audio data contained in the distributed digital data for broadcasting.
[0024] It is assumed that the database held in advance by the re-identification attacker 171 is a database that has been illegally obtained from the server device 140 by illegally accessing the server device 140 .
[0025] Even in such a case, if the voice data included in the voice-accompanying personal information managed by server device 140 has been converted into anonymous voice data in advance, it is possible to prevent individual speakers from being identified based on the database. Specifically, even if the database is used, re-identification attacker 171 cannot identify individual speakers from the voice data included in the voice-accompanying video, nor can he identify individual speakers from the voice data included in the distributed digital data for broadcasting.
[0026] <Hardware Configuration of Server Apparatus> Next, a hardware configuration of the server apparatus 110 according to the first embodiment will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the server apparatus.
[0027] 2, the server device 110 includes a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a drive device 206. The hardware components included in the server device 110 are connected to each other via a bus 207.
[0028] The processor 201 has various computing devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 201 reads various programs (for example, a learning program, a generating program, a providing program, etc., which will be described later) into the memory 202 and executes them.
[0029] The memory 202 has a main storage device such as a read-only memory (ROM) or a random access memory (RAM). The processor 201 and the memory 202 form a so-called computer, and the processor 201 executes various programs read onto the memory 202, thereby realizing various functional units of the computer.
[0030] The auxiliary storage device 203 stores various programs and various information used when the various programs are executed by the processor 201. For example, a speech database storage unit 411, a trained model storage unit 412, and a program storage unit 413, which will be described later, are realized by the auxiliary storage device 203.
[0031] The I / F device 204 is a connection device that connects to an external device 210 (such as an operation device or a display device). The I / F device 204 accepts operations by an administrator (not shown) of the server device 110 via the operation device. The I / F device 204 also outputs the results of processing in the server device 110 and displays them to the administrator of the server device 110 via the display device.
[0032] The communication device 205 is a communication device for communicating with an external device 220 (another terminal, device, etc. within the network system 100) via the network 180. Specifically, the communication device 205 communicates with the terminal 130, the server device 140, the terminal 150, etc. via the network 180.
[0033] The drive device 206 is a device for loading a recording medium 230. The recording medium 230 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The recording medium 230 may also include semiconductor memory that records information electrically, such as a ROM or flash memory.
[0034] The various programs to be installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 230 in the drive device 206 and reading the various programs recorded on the recording medium 230 by the drive device 206. Alternatively, the various programs to be installed in the auxiliary storage device 203 may be installed by being downloaded from the network 180 via the communication device 205.
[0035] <Outline of Speaker Anonymization Technology> Next, an outline of the speaker anonymization technology provided to a user via the server device 110 according to the first embodiment will be described. Fig. 3 is a diagram for explaining the outline of the speaker anonymization technology. As shown in Fig. 3, the speaker anonymization technology provided to a user via the server device 110 according to the first embodiment is a technology for converting voice data into anonymized voice data by executing: a decomposition process 310; a speaker anonymization process 320; and a resynthesis process 330.
[0036] 3, the decomposition process 310 involves extracting pitch (fundamental frequency) by decomposing the audio data. The pitch extracted from the audio data is used to perform the resynthesis process 330.
[0037] 3, the decomposition process 310 includes a process of recognizing speech content by decomposing the voice data. The speech content recognized from the voice data is used when executing the resynthesis process 330.
[0038] 3, the decomposition process 310 includes a process of extracting speaker vectors by decomposing the speech data. The speaker vectors extracted from the speech data are used when executing the speaker anonymization process 320.
[0039] The speaker anonymization process 320 is a process for generating a speaker anonymization vector from the speaker vector extracted in the decomposition process 310. The speaker anonymization vector generated from the speaker vector is used when executing the resynthesis process 330.
[0040] Here, although it is clear that "speaker anonymization technology" is generally a technology for processing voice data so that individual speakers (e.g., names) cannot be identified from the voice data, the processing content is not defined from a technical point of view. Therefore, there is no correct answer data in the anonymized voice data, and there is also no correct answer vector among the speaker anonymization vectors generated by the speaker anonymization process 320.
[0041] On the other hand, speaker anonymization technology does not mean that any processing can be performed on the voice data as long as the individual speaker cannot be identified from the voice data. In the network system 100 to which speaker anonymization technology is applied, it is required that the usability of the anonymized voice data is maintained and that the robustness against re-identification attacks is strong.
[0042] For this reason, the applicant of the present application has considered this need from a technical perspective and defined the following three conditions to be met when generating a speaker anonymization vector in the speaker anonymization process 320: (Condition 1) The speaker vector after the speaker anonymization process (speaker anonymization vector) is different from the speaker vector before the speaker anonymization process. (Condition 2) The speaker vector after the speaker anonymization process (speaker anonymization vector) is different from any other speaker vector included in the speech database. Also, it is different from the speaker vector after the speaker anonymization process (other speaker anonymization vector) of any other speaker vector included in the speech database. (Condition 3) The distribution of the speaker vector after the speaker anonymization process (speaker anonymization vector) does not differ significantly from the distribution when all speaker vectors included in the speech database are used as the population. In other words, the speech resynthesized based on the speaker vector after the speaker anonymization process (speaker anonymization vector) is human speech.
[0043] The resynthesis process 330 resynthesizes the pitch extracted in the decomposition process 310, the speech content extracted in the decomposition process 310, and the speaker anonymization vector generated in the speaker anonymization process 320, to generate anonymized voice data.
[0044] As described above, the speaker anonymization technology provided via the server device 110 according to the first embodiment is a technology for generating anonymized voice data by executing the speaker anonymization process 320 so as to satisfy the above three conditions.
[0045] <Functional Configuration of Server Device> Next, the functional configuration of the server device 110 according to the first embodiment will be described. Fig. 4 is a diagram showing an example of the functional configuration of the server device. As described above, a learning program, a generation program, a provision program, etc. are installed in the server device 110, and by executing these programs, the server device 110 functions as a learning unit 401, a generation unit 402, and a provision unit 403.
[0046] The training unit 401 reads out the speech database stored in the speech database storage unit 411, and trains a neural network (NN) unit (details will be described later) so that the speaker anonymization process 320 is executed to satisfy the above three conditions. The trained NN unit trained by the training unit 401 is stored in the trained model storage unit 412.
[0047] The generation unit 402 reads out the trained NN unit stored in the trained model storage unit 412. The generation unit 402 also constructs an anonymized voice data generation program by combining a program for executing the decomposition process 310 and a program for executing the resynthesis process 330, which are stored in advance, with the read trained NN unit. The anonymized voice data generation program constructed by the generation unit 402 is stored in the program storage unit 413.
[0048] The providing unit 403 reads out the anonymized voice data generation program stored in the program storage unit 413 and transmits it to the terminal that made the request (terminal 130, terminal 150, etc. in the example of FIG. 1 ). Alternatively, the providing unit 403 reads out the anonymized voice data generation program stored in the program storage unit 413 and executes the read out anonymized voice data generation program by inputting the voice data included in the conversion request. The providing unit 403 also transmits the anonymized voice data generated by executing the anonymized voice data generation program to the device that made the conversion request (server device 140, etc. in the example of FIG. 1 ).
[0049] In this way, providing speaker anonymization technology to a user via server device 110 includes at least: - providing an anonymized voice data generation program to the requestor; - providing anonymized voice data generated by executing the anonymized voice data generation program to the requestor.
[0050] <Details of Learning Unit of Server Apparatus> Next, the learning unit 401 of the server apparatus 110 will be described in detail.
[0051] (1) Functional Configuration of the Training Unit First, a detailed description will be given of the functional configuration of the training unit 401. Fig. 5 is a diagram showing the detailed functional configuration of the training unit. As shown in Fig. 5, the training unit 401 includes a speaker vector extraction unit 501, an NN unit 502, and a loss function calculation unit 503.
[0052] The speaker vector extraction unit 501 sequentially reads out speech data (an example of speech data to be learned) stored in the speech database storage unit 411, and extracts speaker vectors by decomposing the read out speech data. The speaker vector extraction unit 501 notifies the extracted speaker vectors to the NN unit 502 and the loss function calculation unit 503.
[0053] The NN unit 502 outputs a speaker anonymization vector by receiving the speaker vector notified from the speaker vector extraction unit 501. The weight parameters of the NN unit 502 are updated based on the loss calculated by the loss function calculation unit 503.
[0054] The loss function calculation unit 503 includes a first loss function calculation unit 511 and a second loss function calculation unit 512 .
[0055] The first loss function calculation unit 511 calculates the loss using a function that indicates the degree to which condition 1 is satisfied. As described above, condition 1 is that "the speaker vector after the speaker anonymization process is different from the speaker vector before the speaker anonymization process." For this reason, the first loss function calculation unit 511 compares the speaker anonymization vector output from the NN unit 502 with the speaker vector extracted by the speaker vector extraction unit 501. Furthermore, the first loss function calculation unit 511 calculates the loss using a function that reduces the loss as the difference between the two vectors increases and reduces the loss as the difference between the two vectors decreases.
[0056] The second loss function calculation unit 512 calculates the loss using a function indicating the degree of satisfaction of condition 2. As described above, condition 2 is: the speaker vector after the speaker anonymization process is different from any other speaker vector included in the speech database, and the speaker vector after the speaker anonymization process is different from the speaker vector of any other speaker vector included in the speech database after the speaker anonymization process.
[0057] For this reason, the second loss function calculation unit 512 compares the speaker anonymization vector output from the NN unit 502 with each of the other speaker vectors extracted by the speaker vector extraction unit 501. Furthermore, the second loss function calculation unit 512 calculates the loss using a function that reduces the loss as the dissimilarity between the two vectors increases and reduces the loss as the dissimilarity between the two vectors decreases.
[0058] Furthermore, the second loss function calculation unit 512 compares the speaker anonymization vector output from the NN unit 502 with each of the other speaker anonymization vectors output from the NN unit 502. Furthermore, the second loss function calculation unit 512 calculates the loss using a function that reduces the loss as the dissimilarity between the two vectors increases and reduces the loss as the dissimilarity between the two vectors decreases.
[0059] The loss function calculation unit 503 updates the weight parameters of the NN unit 502 based on the sum of the loss calculated by the first loss function calculation unit 511 and all the losses calculated by the second loss function calculation unit 512.
[0060] The trained NN unit, which is the NN unit 502 with updated weight parameters, is stored in the trained model storage unit 412.
[0061] (2) Specific Example of Processing by NN Unit Next, a description will be given of a specific example of processing by the NN unit 502 of the learning unit 401. Fig. 6 is a diagram showing a specific example of processing by the NN unit.
[0062] The NN unit 502 is configured by a neural network that performs household transformation processing. ql , orthogonal transformation W l , the definition of the neural network W is given below.
[0063] As shown at 600, the Household transformation H ql is the identity matrix I and the weight parameter v ql The orthogonal transformation W l is the Household transformation H 1 ~H ql Furthermore, the neural network W is calculated by the product of the orthogonal transformation W 1 ~W L As described above, the orthogonal transform W 1 ~W L are the Household transformations H 1 ~H ql Since the orthogonal transformation W 1 ~W L The product of these is the Household transformation H 1 ~H ql In other words, the neural network W can be said to be a neural network that repeatedly performs household transformation processing.
[0064] As shown in the lower part of FIG. 6, the NN unit 502 uses a neural network W denoted by reference numeral 600 to generate a speaker vector x0 The speaker anonymization vector x a Convert to.
[0065] Specifically, the NN unit 502 calculates the speaker vector x 0 When the speaker vector is input, the following steps are performed: - pre-processing shown by reference numeral 610 is performed (a process of moving the speaker vector in a direction that makes it easier to perform the rotation process shown by reference numeral 621 or 622), - rotation processing shown by reference numeral 621 or 622 is performed on the speaker vector after the pre-processing, and - post-processing shown by reference numeral 630 is performed (a process of returning the speaker vector after the rotation process to its original direction), thereby obtaining the speaker anonymization vector x a Output.
[0066] In this way, the neural network W shown at 600 is used in the rotation process shown at 621 or 622. Note that this rotation process is configured to satisfy condition 3.
[0067] As described above, condition 3 is that the distribution of speaker vectors after speaker anonymization processing does not differ significantly from the distribution of a population of all speaker vectors (before speaker anonymization processing) included in the speech database. The NN unit 502 employs a neural network that repeatedly performs household transformation processing to perform the rotation processing indicated by reference numeral 621 or 622. In this way, the NN unit 502 ensures that the distribution of speaker anonymization vectors does not significantly deviate from the distribution of speaker vectors with respect to the rotation center.
[0068] In this way, the neural network employed in the trained NN unit generated by the training unit 401 is a neural network that has mathematical support for satisfying the above condition 3. In other words, the speaker anonymization vector generated by the trained NN unit can be said to be a vector that has mathematical support for satisfying the above condition 3.
[0069] In the NN unit 502 shown in FIG. 6, the rotation process indicated by the reference numeral 621 is performed by using the weight parameter v qlare randomly initialized. Therefore, according to the rotation process indicated by the reference numeral 621, even if training is performed using the same speech database, the trained NN unit generated will be different each time. In other words, even if the same speaker vector is input, the anonymized speaker vectors output from each trained NN unit will be different speaker anonymization vectors. Such a trained NN unit is effective in the following cases among the application examples shown in FIG. 1.
[0070] For example, this is effective in a case where the voice data of the SNS user 131 is registered in the server device 140. In such a case, suppose that the SNS user 131 converts the voice data included in the voice-accompanied video into anonymized voice data when uploading the voice-accompanied video. Also, suppose that the voice DB administrator 141 converts the voice data registered in the server device 140 into anonymized voice data. In this case, if different trained NN units are used, the following will result in different anonymized voice data: - the anonymized voice data into which the voice data included in the voice-accompanied video is converted; and - the anonymized voice data into which the voice data registered in the server device 140 is converted.
[0071] Therefore, even if personal information with voice registered in server device 140 is leaked, it is possible to avoid identifying individual speakers through a re-identification attack based on the leaked personal information with voice.
[0072] On the other hand, in the NN unit 502 shown in FIG. 6, the rotation process indicated by the reference numeral 622 is performed by using the weight parameter v ql are initialized to fixed values. Therefore, according to the rotation process indicated by reference numeral 622, when training is performed using the same speech database, the trained NN units generated will also be the same. In other words, when the same speaker vector is input, the anonymized speaker vectors output from each trained NN unit will be the same speaker anonymization vector. Such a trained NN unit is effective in the following cases among the application examples shown in FIG. 1.
[0073] For example, this is effective in a case where the SNS user 131 is interviewed anonymously and the voice of the SNS user 131 is distributed as digital data for broadcast. In such a case, suppose that the broadcaster 151 converts the voice data included in the digital data for broadcast into anonymized voice data when distributing the digital data for broadcast. In this case, if the same trained NN unit is used as the one used when the SNS user 131 converted the voice data included in the video with audio into anonymized voice data, the following will be the same anonymized voice data: - the anonymized voice data into which the voice data included in the video with audio is converted, and - the anonymized voice data into which the voice data included in the digital data for broadcast is converted.
[0074] Therefore, viewers viewing the audio-accompanied video of the SNS user 131 can listen to the audio of the SNS user 131 contained in the digital data for broadcast without feeling uncomfortable.
[0075] (3) Specific Example of Processing by Loss Function Calculation Unit Next, a description will be given of a specific example of processing by the loss function calculation unit 503 of the learning unit 401. Fig. 7 is a diagram showing a specific example of processing by the loss function calculation unit.
[0076] 7, reference numeral 700 denotes an example of a loss function used when calculating a loss in the loss function calculation unit 503. As described above, the loss function calculation unit 503 calculates a loss by using the first loss function calculation unit 511 (L S ) and the loss calculated by the second loss function calculation unit 512 (L in 700) C ) and the loss calculated by (weighted addition value using coefficient λ).
[0077] In the example of FIG. 7, the first loss function calculation unit 511 calculates the speaker vector x 0 and the speaker anonymization vector x a = f θ (x 0 ) and calculate the loss according to the difference, using the cosine distance.
[0078] In the example of FIG. 7, the second loss function calculation unit 512 calculates the speaker vector x0 Class G Ψ (x 0 ) and the speaker anonymization vector x a = f θ (x 0 ) class g Ψ (f θ (x 0 )) and another speaker vector y extracted by the speaker vector extraction unit 501. 0 The sum of the classification losses for each, and the speaker vector x 0 Class G Ψ (x 0 ) and the speaker anonymization vector x a = f θ (x 0 ) class g Ψ (f θ (x 0 )) and the other speaker anonymization vector y output from the NN unit 502 a When calculating the sum of the classification losses with each, it indicates that one of the following methods is used: "Cross-entropy loss", "AAM (Additive Angular Margin softmax) loss", "w-AAM (Weighted Additive Angular Margin softmax) loss", etc.
[0079] In this way, the trained NN unit generated by the training unit 401 is trained using a loss function that has mathematical support for satisfying the above conditions 1 and 2. In other words, it can be said that the speaker anonymization vector generated by the trained NN unit is a vector that has mathematical support for satisfying the above conditions 1 and 2.
[0080] (4) Relationship between Speaker Vector and Speaker Anonymization Vector Next, we will explain the relationship between the speaker vector before the speaker anonymization process and the speaker anonymization vector when the speaker anonymization process is performed using the trained NN unit and the speaker anonymization vector is generated from the speaker vector. Fig. 8 is a diagram showing the relationship between the speaker vector and the speaker anonymization vector.
[0081] In Fig. 8, black dots indicate positions of speaker vectors in a multidimensional space. Note that, for ease of explanation, the black dots in Fig. 8 are shown projected onto a predetermined two-dimensional plane. Also, in Fig. 8, green dots indicate positions in a multidimensional space of speaker anonymization vectors generated by performing speaker anonymization processing using a trained NN unit. Note that, for ease of explanation, the green dots in Fig. 8 are also shown projected onto a predetermined two-dimensional plane.
[0082] As shown in Figure 8, the green dots and the black dots do not overlap each other, which indicates that the speaker anonymization vector generated by performing speaker anonymization processing using a trained NN unit is different from any of the speaker vectors before the speaker anonymization processing. Note that the black dots appear to overlap each other because they are projected onto a predetermined two-dimensional plane, and are assumed to be located at positions separated from each other at least in the depth direction of the page. Similarly, the green dots appear to overlap each other because they are projected onto a predetermined two-dimensional plane, and are assumed to be located at positions separated from each other at least in the depth direction of the page. In other words, it indicates that the speaker anonymization vector generated by performing speaker anonymization processing using a trained NN unit is different from any of the other speaker anonymization vectors.
[0083] Furthermore, as shown in FIG. 8, it can be seen that the distribution of green dots does not deviate significantly from the distribution of black dots.
[0084] From the above, it can be said that the speaker anonymization vector generated by performing the speaker anonymization process using the trained NN unit is a vector that satisfies conditions 1 to 3.
[0085] (5) Flow of Learning Process by Learning Unit Next, a description will be given of the flow of learning process by the learning unit 401. Fig. 9 is an example of a flowchart showing the flow of learning process by the learning unit.
[0086] In step S 901 , the learning unit 401 reads out the voice database from the voice database storage unit 411 .
[0087] In step S902, the learning unit 401 extracts speaker vectors by decomposing each piece of speech data contained in the read speech database.
[0088] In step S903, the learning unit 401 acquires a speaker vector to be anonymized.
[0089] In step S904, the learning unit 401 inputs the acquired speaker vector to the NN unit 502, thereby generating anonymized speaker vectors from the speaker vectors.
[0090] In step S905, the learning unit 401 determines whether a predetermined number of speaker anonymization vectors have been generated. If it is determined in step S905 that the predetermined number of speaker anonymization vectors have not been generated (NO in step S905), the process returns to step S903. On the other hand, if it is determined in step S905 that the predetermined number of speaker anonymization vectors have been generated (YES in step S905), the process proceeds to step S906.
[0091] In step S906, the learning unit 401 inputs "1" to a counter i that counts speaker anonymization vectors.
[0092] In step S907, the learning unit 401 calculates a loss for the i-th speaker anonymization vector. Specifically, the learning unit 401 calculates the loss by comparing, among the speaker vectors extracted in step S902, a speaker vector corresponding to the i-th speaker anonymization vector with the i-th speaker anonymization vector. The learning unit 401 also calculates the loss by comparing, among the speaker vectors extracted in step S902, speaker vectors other than the speaker vector corresponding to the i-th speaker anonymization vector with the i-th speaker anonymization vector. The learning unit 401 also calculates the loss by comparing, among the speaker vectors generated in step S904, speaker vectors other than the i-th speaker anonymization vector with the i-th speaker anonymization vector.
[0093] In step S908, the learning unit 401 updates the weight parameters of the NN unit based on the calculated loss.
[0094] In step S909, the learning unit 401 determines whether or not the end condition for the learning process is satisfied.
[0095] If it is determined in step S909 that the conditions for ending the learning process are not met (NO in step S909), the process proceeds to step S910.
[0096] In step S910, the learning unit 401 increments the counter i, and then returns to step S907.
[0097] On the other hand, if it is determined in step S909 that the condition for ending the learning process is met (YES in step S909), the process proceeds to step S911.
[0098] In step S911, the learning unit 401 stores the trained NN unit in the trained model storage unit 412, and ends the learning process.
[0099] <Details of Generator and Provider of Server Apparatus> Next, the generator 402 and provider 403 of the server apparatus 110 will be described in detail.
[0100] (1) Functional Configuration of Generator and Provider First, the functional configuration of the generator 402 and provider 403 will be described in detail. Fig. 10 is a diagram showing the functional configuration of the generator and provider in detail. As shown in Fig. 10, the generator 402 has a decomposition program inserting unit 1001, a trained NN unit acquiring unit 1002, a resynthesis program inserting unit 1003, and an anonymized voice data generation program constructing unit 1004.
[0101] The decomposition program insertion unit 1001 inserts a decomposition program for executing the decomposition process 310, which decomposes the voice data and extracts the pitch, speech content, and speaker vector, into the anonymized voice data generation program construction unit 1004. Note that the decomposition program insertion unit 1001 is assumed to have the decomposition program stored therein in advance.
[0102] For example, the decomposition program insertion unit 1001 holds a program that applies a Prosody encoder as a program for extracting pitch. The decomposition program insertion unit 1001 also holds a program that applies HuBERT as a program for extracting speech content. The decomposition program insertion unit 1001 also holds a program that applies ECAPA-TDNN as a program for extracting speaker vectors. ECAPA-TDNN is an abbreviation for "Emphasized Channel Attention, Propagation and Aggregation in Time-Delay Neural Network."
[0103] The trained NN part acquisition unit 1002 reads out the trained NN part from the trained model storage unit 412 and inserts it into the anonymized voice data generation program construction unit 1004.
[0104] The resynthesis program insertion unit 1003 inserts a resynthesis program for executing the resynthesis process 330, which resynthesizes the pitch, the speech content, and the speaker anonymization vector, into the anonymized voice data generation program construction unit 1004. Note that the resynthesis program insertion unit 1003 is assumed to store the resynthesis program in advance.
[0105] For example, the resynthesis program inserting unit 1003 holds a program to which HiFi-GAN is applied as the resynthesis program.
[0106] The anonymized voice data generation program construction unit 1004 acquires: a decomposition program inserted by the decomposition program insertion unit 1001; a trained NN unit inserted by the trained NN unit acquisition unit 1002; and a resynthesis program inserted by the resynthesis program insertion unit 1003. The anonymized voice data generation program construction unit 1004 constructs an anonymized voice data generation program by combining the decomposition program, the trained NN unit, and the resynthesis program. The anonymized voice data generation program construction unit 1004 also stores the constructed anonymized voice data generation program in the program storage unit 413.
[0107] As shown in FIG. 10 , the providing unit 403 includes a request obtaining unit 1011 , a program executing unit 1012 , and a transmitting unit 1013 .
[0108] The request acquisition unit 1011 receives a request to download the anonymized voice data generation program from a user who uses the anonymized voice data generation program (for example, an SNS user 131 or a broadcaster 151) via the terminal 130 or 150. Alternatively, the request acquisition unit 1011 receives a request to convert voice data into anonymized voice data from a user who uses the anonymized voice data generation program (for example, a voice DB administrator 141).
[0109] When the request acquisition unit 1011 receives a request to download the anonymized voice data generation program, it notifies the transmission unit 1013 of the request. Furthermore, when the request acquisition unit 1011 receives a request to convert voice data into anonymized voice data, it notifies the program execution unit 1012 of the conversion request.
[0110] When a conversion request is notified from the request acquisition unit 1011, the program execution unit 1012 extracts the voice data included in the notified conversion request. The program execution unit 1012 also reads the anonymized voice data generation program from the program storage unit 413 and executes the anonymized voice data generation program by inputting the extracted voice data. The program execution unit 1012 also notifies the transmission unit 1013 of the anonymized voice data generated by executing the anonymized voice data generation program.
[0111] When the transmission unit 1013 receives a request from the request acquisition unit 1011 to download an anonymized voice data generation program, the transmission unit 1013 reads the anonymized voice data generation program from the program storage unit 413 and transmits it to the requestor (e.g., terminal 130, 150).
[0112] Furthermore, when the transmitting unit 1013 is notified of anonymized voice data by the program executing unit 1012, the transmitting unit 1013 transmits the anonymized voice data to the request source (for example, the server device 140).
[0113] (2) Flow of Generation Processing by the Generation Unit Next, a description will be given of the flow of generation processing by the generation unit 402. Fig. 11 is a flowchart showing the flow of generation processing by the generation unit.
[0114] In step S1101, the generation unit 402 acquires a trained NN unit from the trained model storage unit 412.
[0115] In step S1102, the generation unit 402 inserts the decomposition program into the anonymized voice data generation program.
[0116] In step S1103, the generation unit 402 inserts the trained NN unit into the anonymized voice data generation program.
[0117] In step S1104, the generation unit 402 inserts the resynthesis program into the anonymized voice data generation program.
[0118] In step S1105 , the generation unit 402 stores the anonymized voice data generation program in the program storage unit 413 .
[0119] (3) Flow of the Providing Process by the Providing Unit Next, a description will be given of the flow of the providing process of the speaker anonymization technology by the providing unit 403. Fig. 12 is a flowchart showing the flow of the providing process by the providing unit.
[0120] In step S1201, the providing unit 403 determines whether or not a request to download an anonymized voice data generation program or a request to convert voice data into anonymized voice data has been received from any terminal or device via the network 180.
[0121] If it is determined in step S1201 that no request has been received (NO in step S1201), the process waits until any request is received.
[0122] On the other hand, if it is determined in step S1201 that any request has been received (YES in step S1201), the process proceeds to step S1202.
[0123] In step S1202, the providing unit 403 determines the type of the received request. If it is determined in step S1202 that the received request is a request to convert voice data into anonymized voice data, the process proceeds to step S1211.
[0124] In step S1211, the providing unit 403 extracts the audio data included in the conversion request.
[0125] In step S1212, the providing unit 403 reads out the anonymized voice data generation program, and inputs the extracted voice data to execute the anonymized voice data generation program, thereby generating anonymized voice data.
[0126] In step S1213, the providing unit 403 transmits the generated anonymized voice data to the request source.
[0127] On the other hand, if it is determined in step S1202 that the received request is a request to download an anonymized voice data generation program, the process proceeds to step S1221.
[0128] In step S1221, the providing unit 403 reads out the anonymized voice data generation program and transmits it to the request source.
[0129] In step S1231, the providing unit 403 determines whether or not to end the providing process. If it is determined in step S1231 that the providing process should be continued (NO in step S1231), the process returns to step S1201. On the other hand, if it is determined in step S1231 that the providing process should be ended (YES in step S1231), the providing process of the speaker anonymization technology ends.
[0130] <Hardware configuration of terminal> Next, the hardware configuration of a terminal (e.g., terminals 130, 150, etc.) operated by a user (here, SNS user 131, broadcaster 151, etc.) who uses the anonymized voice data generation program will be described. Fig. 13 is a diagram showing an example of the hardware configuration of a terminal. Note that the hardware configuration of the terminals 130, 150 operated by SNS user 131, broadcaster 151, etc. is similar to the hardware configuration of server device 110 described using Fig. 2 , and therefore differences from the hardware configuration of server device 110 will be mainly described here.
[0131] The processor 1301 reads out and executes various programs, such as an anonymized voice data generation program, onto the memory 1302. The auxiliary storage device 1303 realizes, for example, a voice data storage unit 1440, which will be described later.
[0132] The I / F device 1304 accepts operations by the SNS user 131, the broadcaster 151, etc. via an operation device, which is an example of the external device 1310. The I / F device 1304 also outputs an operation screen when the SNS user 131, the broadcaster 151, etc. operates, and displays it to the SNS user 131, the broadcaster 151, etc. via a display device, which is an example of the external device 1310. Furthermore, in the case of the terminal 150, the I / F device 1304 outputs the processing result (anonymized audio data) to the broadcasting equipment 160, which is an example of the external device 1310.
[0133] The communication device 1305 communicates with the server device 110 , the server device 120 , and the like, which are examples of the external device 1320 , via the network 180 .
[0134] The anonymized voice data generation program to be installed in the auxiliary storage device 1303 is installed by being downloaded from the network 180 via the communication device 1305. Alternatively, in the case of the terminal 150, the anonymized voice data generation program to be installed in the auxiliary storage device 1303 may be installed by reading the anonymized voice data generation program recorded on the recording medium 1330 by the drive device 1306.
[0135] <Functional Configuration of Terminal> Next, the functional configuration of a terminal (e.g., terminals 130, 150, etc.) operated by a user (here, SNS user 131, broadcaster 151, etc.) who uses the anonymized voice data generation program will be described. Fig. 14 is a diagram showing an example of the functional configuration of a terminal. As described above, the anonymized voice data generation program is installed in the terminals 130, 150, and by executing this program, the terminals 130, 150, etc. function as anonymized voice data generation unit 1400.
[0136] As shown in FIG. 14 , the anonymized voice data generation unit 1400 includes a decomposition unit 1410 , a trained NN unit execution unit 1420 , and a resynthesis unit 1430 .
[0137] The decomposing unit 1410 further includes a pitch acquiring unit 1411 , a speech content acquiring unit 1412 , and a speaker vector extracting unit 1413 .
[0138] The pitch acquisition unit 1411 reads out audio data (an example of audio data to be anonymized) from the audio data storage unit 1440 and extracts pitch by decomposing the read audio data. The pitch acquisition unit 1411 extracts pitch, for example, by executing a program to which a prosody encoder is applied. The pitch acquisition unit 1411 notifies the resynthesis unit 1430 of the extracted pitch. It is assumed that the audio data storage unit 1440 stores audio data to be anonymized (for example, audio data included in audio-accompanied video, audio data included in digital data for broadcasting, etc.).
[0139] The speech content acquisition unit 1412 extracts the speech content by performing speech recognition processing on the speech data read from the speech data storage unit 1440. The speech content acquisition unit 1412 extracts the speech content by, for example, executing a program to which HuberT is applied. The speech content acquisition unit 1412 notifies the resynthesis unit 1430 of the extracted speech content.
[0140] The speaker vector extraction unit 1413 extracts speaker vectors from the speech data read out from the speech data storage unit 1440. The speaker vector extraction unit 1413 extracts speaker vectors by, for example, executing a program to which ECAPA-TDNN is applied. The speaker vector extraction unit 1413 notifies the trained NN unit execution unit 1420 of the extracted speaker vectors.
[0141] The trained NN unit execution unit 1420 executes the trained NN unit and generates a speaker anonymization vector by inputting the speaker vector notified by the speaker vector extraction unit 1413. The trained NN unit execution unit 1420 also notifies the resynthesis unit 1430 of the generated speaker anonymization vector.
[0142] The resynthesis unit 1430 resynthesizes the pitch notified by the pitch acquisition unit 1411, the speech content notified by the speech content acquisition unit 1412, and the speaker anonymization vector notified by the trained NN unit execution unit 1420, to generate anonymized speech data. The resynthesis unit 1430 generates anonymized speech data by, for example, executing a program to which HiFi-GAN is applied. The resynthesis unit 1430 stores the generated anonymized speech data in the speech data storage unit 1440.
[0143] <Flow of anonymized voice data generation process> Next, a flow of anonymized voice data generation process by the anonymized voice data generation unit 1400 will be described. Fig. 15 is an example of a flowchart showing the flow of anonymized voice data generation process by the anonymized voice data generation unit. The terminals 130, 150, etc. execute the anonymized voice data generation process shown in Fig. 15 by starting an anonymized voice data generation program.
[0144] In step S1501 , the anonymized voice data generation unit 1400 reads voice data from the voice data storage unit 1440 .
[0145] In step S1502, the anonymized voice data generating unit 1400 generates anonymized voice data from the read voice data.
[0146] In step S1503 , the anonymized voice data generating unit 1400 stores the generated anonymized voice data in the voice data storage unit 1440 .
[0147] In step S1504, the anonymized voice data generation unit 1400 determines whether or not to end the anonymized voice data generation process. If it is determined in step S1504 that the anonymized voice data generation process should be continued (NO in step S1504), the process returns to step S1501. On the other hand, if it is determined in step S1504 that the anonymized voice data generation process should be ended (YES in step S1504), the anonymized voice data generation process is ended.
[0148] <Characteristics of Anonymous Voice Data> Next, the characteristics of the anonymous voice data generated by executing the anonymized voice data generation program will be described.
[0149] Fig. 16 is a diagram showing characteristics of anonymized voice data. In Fig. 16, voice data or anonymized voice data (before speaker anonymization processing) are arranged on the horizontal axis and the vertical axis, respectively. The color of the area where any of the voice data or anonymized voice data (before speaker anonymization processing) on the horizontal axis intersects with any of the voice data or anonymized voice data (before speaker anonymization processing) on the vertical axis represents the similarity between: the voice data or anonymized voice data (before speaker anonymization processing) on the corresponding horizontal axis; and the voice data or anonymized voice data (before speaker anonymization processing) on the corresponding vertical axis. In the example of Fig. 16, the color is arranged so that the higher the similarity, the closer to yellow, and the lower the similarity, the closer to green.
[0150] 16, when comparing voice data, each voice data (before the speaker anonymization process) has a low degree of similarity with other voice data (before the speaker anonymization process) (see area 1610). Note that the yellow portion in area 1710 indicates the degree of similarity between the own voice data.
[0151] 16, when anonymized voice data are compared with each other, each anonymized voice data has a low degree of similarity to other anonymized voice data (see area 1640). Note that the yellow colored portion in area 1640 indicates the degree of similarity between the anonymized voice data.
[0152] 16, when comparing the speech data (before the speaker anonymization process) with the anonymized speech data, the results show that each anonymized speech data has a low degree of similarity to the speech data (see areas 1620 and 1630). Note that areas 1620 and 1630 are basically the same, except that the vertical and horizontal relationships are reversed.
[0153] In this way, each anonymized voice data generated by executing the anonymized voice data generation program can be said to have the following characteristics: - Low similarity to the corresponding own voice data (before the speaker anonymization process) (areas 1620, 1630), - Low similarity to other voice data other than the corresponding voice data (before the speaker anonymization process) (areas 1620, 1630), - Low similarity to other anonymized voice data (area 1640).
[0154] <Outline of evaluation indexes for anonymized voice data> Next, an outline of evaluation indexes for evaluating anonymized voice data generated by executing an anonymized voice data generation program will be described. Fig. 17 is a diagram showing an outline of evaluation indexes for anonymized voice data.
[0155] In the distribution diagram shown in FIG. 17, the horizontal axis represents convenience, and the vertical axis represents robustness against re-identification attacks. Convenience increases toward the right and decreases toward the left. Robustness against re-identification attacks increases toward the top and decreases toward the bottom. In this case, the anonymized voice data is more useful toward the top right.
[0156] Here, the anonymized voice data generation program provided by the server device 110 according to the first embodiment converts only the speaker vector of the voice data without changing the pitch or speech content of the voice data. Therefore, it can be said that the anonymized voice data generated by the anonymized voice data generation program ideally has the following characteristics compared to the original voice data: - convenience is maintained, and - robustness against re-identification attacks is improved.
[0157] In other words, in the distribution diagram shown in Figure 17, it is desirable that the anonymized voice data generated by the anonymized voice data generation program be plotted directly above the original voice data and at a position farther away from the original voice data.
[0158] <Evaluation of Anonymized Voice Data> Next, in order to evaluate the anonymized voice data generated by executing the anonymized voice data generation program based on the above evaluation index, robustness against re-identification attacks and convenience are verified.
[0159] (1) Verification of Robustness Against Re-identification Attacks First, we verify the robustness against re-identification attacks of the anonymized voice data generated by executing the anonymized voice data generation program.
[0160] (1-1) Method of Verification Experiment First, the method of the verification experiment will be described. Fig. 18 is a diagram showing the method of the verification experiment for the robustness of anonymized voice data against re-identification attacks.
[0161] As shown in Fig. 18, in the experiment to verify the robustness of anonymized voice data against re-identification attacks, three different levels of re-identification attackers were assumed as the re-identification attacker 171. In Fig. 18, the first re-identification attacker 171_1 is a re-identification attacker with a low skill level in re-identification attacks, the second re-identification attacker 171_2 is a re-identification attacker with a medium skill level in re-identification attacks, and the third re-identification attacker 171_3 is a re-identification attacker with a high skill level in re-identification attacks.
[0162] The example in Figure 18 assumes a case in which each re-identification attacker performs a re-identification attack on the anonymized voice data generated by executing an anonymized voice data generation program 1820 including a trained NN unit execution unit 1420 for verification voice data.
[0163] Specifically, the first re-identification attacker 171_1 attempts speaker authentication for anonymized voice data generated by executing the anonymized voice data generation program 1820, without using any anonymized voice data generation program. The speaker authentication unit 1831 is assumed to have been trained in advance using training voice data (before the speaker anonymization process) to extract features of the voice data. The speaker authentication unit 1831 performs speaker authentication by comparing the features of the anonymized voice data generated by executing the anonymized voice data generation program 1820 with the features of the voice data input by the first re-identification attacker 171_1. At this time, the evaluator calculates an EER (Equal Error Rate), which indicates the degree to which speaker authentication was successful.
[0164] Furthermore, the second re-identification attacker 171_2 attempts speaker authentication for the anonymized voice data generated by executing the anonymized voice data generation program 1820, using its own anonymized voice data generation program 1810. The speaker authentication unit 1832 is assumed to have been trained in advance using training voice data (before the speaker anonymization process) to extract features of the voice data. The speaker authentication unit 1832 compares the features of the anonymized voice data generated by executing the anonymized voice data generation program 1820 with the features of the anonymized voice data generated by executing the anonymized voice data generation program 1810. In this way, the speaker authentication unit 1832 performs speaker authentication. At this time, the evaluator calculates the EER.
[0165] Furthermore, the third re-identification attacker 171_3 attempts speaker authentication for the anonymized voice data generated by executing the anonymized voice data generation program 1820, using a more advanced, unique anonymized voice data generation program 1811. The speaker authentication unit 1833 is assumed to have been trained in advance using training anonymized voice data so as to extract features of the anonymized voice data. The speaker authentication unit 1833 compares the features of the anonymized voice data generated by executing the anonymized voice data generation program 1820 with the features of the anonymized voice data generated by executing the anonymized voice data generation program 1811. In this way, the speaker authentication unit 1833 performs speaker authentication. At this time, the evaluator calculates the EER.
[0166] As a result, according to the verification experiment method shown in FIG. 18, it is possible to verify the robustness of the anonymized voice data generated by executing the anonymized voice data generation program 1820 against re-identification attacks.
[0167] (1-2) Results of Verification Experiments Next, we will explain the results of verification experiments based on the method described in (1-1) above. Figure 19 shows the results of verification experiments on the robustness of anonymized voice data against re-identification attacks.
[0168] 19 also illustrates, as a reference example, the equivalent error rate when speaker authentication is performed on the voice data (before the speaker anonymization process) (when the anonymized voice data generation program 1820 in FIG. 18 is not executed). In FIG. 19, reference numeral 1910 indicates the equivalent error rate when speaker authentication is performed on the voice data (before the speaker anonymization process).
[0169] On the other hand, reference numeral 1920 indicates the equivalent error rate when the first re-identification attacker 171_1 to the third re-identification attacker 171_3 attempt speaker authentication for anonymized voice data. Note that the example of reference numeral 1920 also illustrates, as a comparative example, the equivalent error rate when a "conventional anonymization process" is used (when the anonymized voice data generation program 1820 in FIG. 18 is replaced with a conventional anonymization process). The conventional anonymization process is, for example, an anonymization process that randomly converts formants by signal processing voice data, and is an anonymization process that is commonly used in the broadcasting industry, etc.
[0170] According to the example of reference numeral 1920, when conventional anonymization processing is used, speaker authentication by the first re-identification attacker 171_1 and the second re-identification attacker 171_2 results in a high equivalent error rate (low authentication accuracy). In other words, the rate at which individual speakers are identified is kept low. On the other hand, speaker authentication by the third re-identification attacker 171_3 results in a low equivalent error rate (high authentication accuracy). In other words, the rate at which individual speakers are identified is high.
[0171] In this way, although conventional anonymization processes significantly change the sound of a voice to the ear, they are vulnerable to highly skilled re-identification attacks, and can be said to be processes that have a high possibility of identifying individual speakers.
[0172] In contrast, the equivalent error rate when the current anonymization process was used was high (resulting in low authentication accuracy). Note that the "equivalent error rate when the current anonymization process was used" refers to the equivalent error rate when speaker authentication was performed on anonymized speech data generated by executing the anonymized speech data generation program 1820 including the trained NN unit execution unit 1420.
[0173] In other words, when this anonymization process is used, the rate at which individual speakers are identified can be kept low regardless of which of the first re-identification attacker 171_1 to the third re-identification attacker 171_3 performs a re-identification attack.
[0174] In this way, the anonymized voice data generated by executing the anonymized voice data generation program 1820 including the trained NN unit execution unit 1420 can be said to be: - robust against re-identification attacks by a re-identification attacker of any level, and - stronger than the robustness against re-identification attacks of voice data before speaker anonymization processing, and further stronger than the robustness against re-identification attacks of anonymized voice data when conventional anonymization processing is used.
[0175] (2) Verification of Usability Next, the usability of the anonymized voice data generated by executing the anonymized voice data generation program will be verified.
[0176] (2-1) Method of Verification Experiment First, the method of the verification experiment will be described. Fig. 20 is a diagram showing the method of the verification experiment on the usability of anonymized voice data.
[0177] The example in Figure 20 shows a case where an anonymized voice data generation program 1820 including a trained NN unit execution unit 1420 is executed for the voice data for verification, voice recognition is performed on the generated anonymized voice data, and the data is verified by an evaluator calculating the error rate.
[0178] As shown in Fig. 20, two types of automatic speech recognition (ASR) systems (reference numerals 2010 and 2020) are used. Note that the first automatic speech recognition system indicated by reference numeral 2010 is a system trained using training speech data (before speaker anonymization processing), and the second automatic speech recognition system indicated by reference numeral 2020 is a system trained using anonymized training speech data.
[0179] The anonymized voice data generated by executing the anonymized voice data generation program 1820 is input to a first automatic voice recognition system indicated by reference numeral 2010 for voice recognition, and an evaluator calculates the error rate (WER: Word Error Rate) for the recognition result. The anonymized voice data is input to a second automatic voice recognition system indicated by reference numeral 2020 for voice recognition, and an evaluator calculates the error rate (WER: Word Error Rate) for the recognition result.
[0180] As a result, according to the method of the verification experiment shown in FIG. 20, it is possible to verify the usability of the anonymized voice data generated by executing the anonymized voice data generation program 1820.
[0181] (2-2) Results of the Verification Experiment Next, the results of the verification experiment using the method described in (2-1) above will be described. Fig. 21 shows the results of the verification experiment on the usability of anonymized voice data.
[0182] 21 also illustrates, as a reference example, the error rate when speech data (before speaker anonymization processing) is input to a first automatic speech recognition system and speech is recognized. Also, as a comparative example, the verification experiment results shown in FIG. 21 also illustrate the error rate when conventional anonymization processing is used. The "conventional anonymization processing" is, for example, an anonymization processing that randomly converts formants by signal processing speech data, and is an anonymization processing that is commonly used in the broadcasting industry, etc.
[0183] According to the example of reference numeral 2100, the error rate when the conventional anonymization process is used increases compared to the error rate of the speech data (before the speaker anonymization process) when the first speech recognition system is used. In contrast, the error rates when the current anonymization process is used are all comparable to the error rate of the speech data (before the speaker anonymization process). Note that the "error rate when the current anonymization process is used" refers to the error rate when speech recognition is performed on anonymized speech data generated by executing the anonymized speech data generation program 1820 including the trained NN unit execution unit 1420.
[0184] In this way, the anonymized voice data generated by executing the anonymized voice data generation program 1820 including the trained NN unit execution unit 1420 can be said to have the same level of convenience as voice data (before the speaker anonymization process).
[0185] (3) Evaluation Results of Anonymized Voice Data As described in (1-2) and (2-2) above, the anonymized voice data generated by executing the anonymized voice data generation program 1820 including the trained NN unit execution unit 1420: - Has the same level of convenience as the original voice data; - Has stronger robustness against re-identification attacks by a re-identification attacker of any level than the original voice data, and is stronger than anonymized voice data when conventional anonymization processing is used.
[0186] In other words, it was found that the anonymized voice data generated by executing the anonymized voice data generation program results in an evaluation result in which the anonymized voice data is plotted directly above the original voice data and at a position farther away from the original voice data in the distribution diagram of Figure 17.
[0187] <Summary> As is clear from the above description, the speaker anonymization device (server device 110, terminal 130, or terminal 150) executing the speaker anonymization program according to the first embodiment: performs speaker anonymization processing on speaker vectors extracted by decomposing speech data, and generates anonymized speech data by resynthesizing the resulting vectors; stores a trained neural network (a neural network for satisfying condition 3) that performs household transformation processing. The trained neural network is trained so that a vector output in response to an input speaker vector satisfies a predetermined loss function. Specifically, the trained neural network is trained based on a first loss function and a second loss function. The first loss function is a loss function for calculating a loss corresponding to the difference between an input speaker vector and an output anonymized speaker vector (a loss function for satisfying condition 1). The second loss function is a loss function for calculating a loss corresponding to the dissimilarity between the output anonymized speaker vector and a speaker vector other than the input speaker vector (a loss function for satisfying condition 2). - An anonymized speaker vector is generated by inputting the speaker vector to be anonymized into the trained neural network.
[0188] As a result, the server device 110 according to the first embodiment can generate anonymized speaker vectors that satisfy the following conditions: (Condition 1) the speaker vector after the speaker anonymization process is different from the speaker vector before the speaker anonymization process; (Condition 2) the speaker vector after the speaker anonymization process is different from any other speaker vector included in the speech database, and is also different from the speaker vector of any other speaker vector included in the speech database after the speaker anonymization process; and (Condition 3) the distribution of the speaker vector after the speaker anonymization process of the speaker vector does not differ significantly from the distribution of a population of all speaker vectors included in the speech database. In other words, the speech based on the speaker vector after the speaker anonymization process is human speech.
[0189] Furthermore, it has been verified that, according to the first embodiment, the anonymized speech data resynthesized using the anonymized speaker vector has the same level of convenience as the speech data (before the speaker anonymization process) and can improve robustness against re-identification attacks.
[0190] In this way, the speaker anonymization device (server device 110, terminal 130, or terminal 150) according to the first embodiment can provide a speaker anonymization technology that improves robustness against re-identification attacks while maintaining the convenience of speech data.
[0191] [Other Embodiments] In the above first embodiment, the learning unit 401 performs learning using two calculation units, the first loss function calculation unit 511 and the second loss function calculation unit 512, as the loss function calculation unit 503. However, the learning unit 401 may be configured to perform learning using either the first loss function calculation unit 511 or the second loss function calculation unit 512 as the loss function calculation unit 503. Alternatively, the learning unit 401 may be configured to perform learning using a calculation unit using a predetermined loss function other than the first loss function calculation unit 511 and the second loss function calculation unit 512 as the loss function calculation unit 503.
[0192] In the first embodiment, the anonymized voice data generation unit 1400 has been described as employing a trained neural network that performs household transformation processing. However, the anonymized voice data generation unit 1400 may employ a trained neural network other than a trained neural network that performs household transformation processing, as long as the trained neural network satisfies condition 3. However, when employing a trained neural network other than a trained neural network that performs household transformation processing, it is desirable that the trained neural network be trained using either the first loss function calculation unit 511 or the second loss function calculation unit 512.
[0193] In the first embodiment, the decomposing unit 1410 of the anonymized voice data generating unit 1400 is described as including the pitch acquiring unit 1411. However, the pitch acquiring unit 1411 is not an essential component, and the decomposing unit 1410 of the anonymized voice data generating unit 1400 may be configured not to include the pitch acquiring unit 1411.
[0194] In the first embodiment, an example has been given in which "Cosine similarity loss" is used as the first loss function calculation unit 511. However, any other loss function may be used as the first loss function calculation unit 511 as long as it satisfies Condition 1.
[0195] Furthermore, in the first embodiment, examples have been given in which the following functions are used as the second loss function calculation unit 512: "Cross-entropy loss", "AAM (Additive Angular Margin softmax) loss", "w-AAM (Weighted Additive Angular Margin softmax) loss", etc. However, any loss function other than these may be used as the second loss function calculation unit 512 as long as it satisfies Condition 2.
[0196] In the first embodiment, the speaker vector extracted from the speech data is subjected to the speaker anonymization process. However, the target of the anonymization process is not limited to the speaker vector extracted from the speech data. For example, the anonymization process may be performed on a face recognition vector extracted from the image data.
[0197] In the first embodiment, the server device 110 executes a learning program to generate a trained NN unit. However, the device that executes the learning program is not limited to the server device 110, and the learning program may be executed by a device (not shown) owned by a user who uses the anonymized voice data generation program. In this case, the server device 110 may be configured to provide the user who uses the anonymized voice data generation program with the anonymized voice data generation program including an untrained NN unit and the learning program.
[0198] In the first embodiment, the server device 110 is described as being implemented as an integrated device, but each functional unit included in the server device 110 may be implemented in a separate device. For example, the learning unit 401 may be implemented in a learning device separate from the server device 110. Furthermore, the generation unit 402 may be implemented in a generation device separate from the server device 110.
[0199] Furthermore, in the first embodiment, no mention is made of the type of language of the voice data, but the type of language of the voice data is arbitrary.
[0200] The present invention is not limited to the configurations described in the above embodiments, but may be combined with other elements, etc. These aspects can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form.
[0201] This application claims priority based on Japanese Patent Application No. 2023-088164, filed on May 29, 2023, the entire contents of which are incorporated herein by reference.
[0202] 100: Network system 110: Server device 130: Terminal 140: Server device 150: Terminal 401: Learning unit 402: Generation unit 403: Provision unit 501: Speaker vector extraction unit 502: NN unit 503: Loss function calculation unit 511: First loss function calculation unit 512: Second loss function calculation unit 1001: Decomposition program insertion unit 1002: Trained NN unit acquisition unit 1003: Re-synthesis program insertion unit 1004: Anonymized voice data generation program construction unit 1011: Request acquisition unit 1012: Program execution unit 1013: Transmission unit 1400: Anonymized voice data generation unit 1410: Decomposition unit 1411: Pitch acquisition unit 1412: Speech content acquisition unit 1413 : Speaker vector extraction unit 1420 : Trained NN execution unit 1430 : Re-synthesis unit
Claims
1. A speaker anonymization device that generates anonymized voice data, comprising: a generation unit that generates a speaker anonymization vector by inputting a speaker vector extracted from the voice data to be anonymized to a trained neural network that performs household transformation processing, the trained neural network being trained using a predetermined loss function.
2. The speaker anonymization device described in claim 1, wherein the predetermined loss function includes a first loss function that calculates a loss corresponding to the difference between a speaker vector extracted from the speech data to be trained and a speaker anonymization vector output by inputting the speaker vector into a neural network that performs household transformation processing.
3. The speaker anonymization device according to claim 2, wherein the predetermined loss function includes a second loss function that calculates a loss according to the dissimilarity of a speaker anonymization vector output by inputting a speaker vector extracted from the speech data to be trained into a neural network that performs household transformation processing, relative to other vectors.
4. A speaker anonymization device that generates anonymized voice data, comprising: a generation unit that generates a speaker anonymization vector by inputting a speaker vector extracted from the voice data to be anonymized to a trained neural network generated by training the neural network using a first loss function that calculates a loss corresponding to a difference between a speaker vector extracted from the voice data to be trained and a speaker anonymization vector output from the neural network by inputting the speaker vector, and / or a second loss function that calculates a loss corresponding to a dissimilarity of the output speaker anonymization vector to other vectors.
5. The speaker anonymization device according to claim 4, wherein the neural network performs household transformation processing on a speaker vector extracted from the speech data to be trained.
6. The speaker anonymization device of claim 3 or 5, wherein the dissimilarity to other vectors includes: a dissimilarity between a speaker anonymization vector output by inputting a speaker vector extracted from the speech data to be trained into a neural network performing household transformation processing, and a speaker vector other than the speaker vector; and a dissimilarity between a speaker anonymization vector output by inputting a speaker vector extracted from the speech data to be trained into a neural network performing household transformation processing, and another speaker anonymization vector output by inputting another speaker vector extracted from the speech data to be trained into a neural network performing household transformation processing.
7. A speaker anonymization device as described in claim 1 or 5, wherein the trained neural network that performs the household transformation processing includes at least one of a trained neural network trained with parameters initialized to fixed values and a trained neural network trained with parameters initialized randomly.
8. A speaker anonymization device as described in claim 1 or 4, further comprising: a speech content acquisition unit that acquires speech content from voice data to be anonymized; and a resynthesis unit that generates anonymized voice data by resynthesizing the acquired speech content with the generated speaker anonymization vector.
9. A speaker anonymization device as described in claim 8, further comprising a pitch acquisition unit that acquires pitch from the voice data to be anonymized, wherein the resynthesis unit generates the anonymized voice data by resynthesizing the acquired pitch, the acquired speech content, and the generated speaker anonymization vector.
10. The speaker anonymization device as described in claim 8, further comprising a transmission unit that transmits the anonymized voice data generated from the voice data to be anonymized that is included in a request acquired via a network to a request source.
11. A speaker anonymization method for generating anonymized voice data, the method comprising the steps of: generating a speaker anonymization vector by inputting a speaker vector extracted from the voice data to be anonymized to a trained neural network that performs a household transformation process, the trained neural network being trained using a predetermined loss function, the speaker anonymization method being executed by a computer.
12. A speaker anonymization method for generating anonymized voice data, the method comprising the steps of: generating a speaker anonymization vector by inputting a speaker vector extracted from the voice data to be anonymized to a trained neural network generated by training the neural network using a first loss function that calculates a loss corresponding to the difference between a speaker vector extracted from the voice data to be trained and a speaker anonymization vector output from the neural network by inputting the speaker vector; and / or a second loss function that calculates a loss corresponding to the dissimilarity of the output speaker anonymization vector to other vectors.
13. A speaker anonymization program for causing a computer of a speaker anonymization device that generates anonymized voice data to execute a generation process of generating a speaker anonymization vector by inputting a speaker vector extracted from the voice data to be anonymized to a trained neural network that performs household transformation processing, the trained neural network being trained using a predetermined loss function.
14. A speaker anonymization program for causing a computer of a speaker anonymization device that generates anonymized voice data to execute a generation process of generating a speaker anonymization vector by inputting a speaker vector extracted from the voice data to be anonymized to a trained neural network generated by training the neural network using a first loss function that calculates a loss corresponding to the difference between a speaker vector extracted from the voice data to be trained and a speaker anonymization vector output from the neural network by inputting the speaker vector, and / or a second loss function that calculates a loss corresponding to the dissimilarity of the output speaker anonymization vector to other vectors.
15. A learning device having a learning unit that generates a trained neural network by training a neural network that performs household transformation processing using: a first loss function that calculates a loss corresponding to the difference between a speaker vector extracted from speech data to be trained and a vector that is output when the speaker vector is input to the neural network; and a second loss function that calculates a loss corresponding to the dissimilarity of the output vector to other vectors.
16. A learning method performed by a computer, comprising: a learning process for generating a trained neural network by training a neural network that performs household transformation processing using: a first loss function that calculates a loss corresponding to the difference between a speaker vector extracted from speech data to be trained and a vector that is output when the speaker vector is input to the neural network; and a second loss function that calculates a loss corresponding to the dissimilarity of the output vector with respect to other vectors.
17. A learning program for causing a computer to execute a learning process for generating a trained neural network by training a neural network that performs household transformation processing using: a first loss function that calculates a loss corresponding to the difference between a speaker vector extracted from speech data to be trained and a vector that is output when the speaker vector is input to the neural network; and a second loss function that calculates a loss corresponding to the dissimilarity of the output vector with respect to other vectors.
Citation Information
Patent Citations
Image and voice communication system
JP2004266746A
Device, method, and program for predicting situation of diffusion of radioactive materials
JP2023088164A
Voice communication method and equipment, and voice communication program
JP2003157100A
Multi-faceted plate-type terminal of a super capacitor module
KR1020240029364A
