Agreement determination method, computer program and agreement determination system

Abstract

To provide an agreement determination method, a computer program and an agreement determination system, which determine success / failure of an agreement by a user.SOLUTION: An agreement determination method performed by a computer device having at least one processing unit which executes a computer readable command stored in a storage unit includes: a voice reception step for receiving uttered voice uttered by a user for acquirement of an agreement by the at least one processing unit; a voice recognition step for recognizing a content of utterance by the user as a language based on the uttered voice; an emotion analysis step for analyzing emotion of the user based on the uttered voice; and an agreement determination step for determining success / failure of the agreement by the user based on a recognition result in the voice recognition step and an analysis result in the emotion analysis step.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a consent decision method, a computer program, and a consent decision system. [Background technology]

[0002] In recent years, with the widespread use of devices equipped with cameras, such as smartphones and tablet terminals, video calling services have become available that allow users to enjoy calls while seeing the other person's face. Such video calling services are also used as a means of communication in services targeting elderly people, such as elderly care.

[0003] Patent Document 1 discloses a technology relating to a video call service that enables communication using a video call bot.

[0004] Patent Document 2 discloses a technology relating to a monitoring device and a monitoring system that can transmit fixed messages and carry out voice conversations. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-129296 [Patent Document 2] Japanese Patent Application Publication No. 2017-208657 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when verbally obtaining consent from a user remotely via video call for applying for a service or confirming the details of a contract, if the supporter makes a nod that sounds like they are agreeing with the flow of the conversation but does not intend to give consent, there is a risk that a mishearing or misunderstanding will occur, resulting in a discrepancy between the user's intention and the supporter's interpretation, and the user may be provided with services they do not want or support they did not intend.

[0007] An object of the present invention is to provide a novel technique for determining whether or not a user has given consent. [Means for solving the problem]

[0008] [1] A consent determination method executed by a computer device, the computing device includes at least one processing unit configured to execute computer-readable instructions stored in a memory unit; The consent determination method includes: a voice receiving step in which the at least one processing unit receives a voice uttered by a user for establishing consent; a speech recognition step in which the at least one processing unit recognizes the content of the user's speech as a language based on the speech; an emotion analysis step in which the at least one processing unit analyzes the emotion of the user based on the speech sound; a consent determination step in which the at least one processing unit determines whether consent has been given by the user based on a recognition result in the voice recognition step and an analysis result in the emotion analysis step; A consent determination method having the following. [2] The consent determination method further comprises: the at least one processing unit has an execution step of executing a support process based on the establishment of consent when it is determined that consent is established in the consent determination step; [1] The method for determining consent described above. [3] The consent determination method further comprises: the at least one processing unit has a display processing step of displaying a consent statement to be read out to the user to obtain consent, and displaying the consent statement to the user; In the speech recognition step, the content of the user's utterance is recognized as a language based on the speech voice, which is a voice uttered by the user reading out the consent sentence; In the display processing step, a speech portion uttered in the agreement sentence is identified based on the recognition result obtained by the speech recognition step and the agreement sentence, and the agreement sentence showing the identified speech portion is displayed. The method for determining consent described in [1] or [2]. [4] The at least one processing unit has a display processing step of displaying a consent statement to be read out to the user to obtain consent, and showing the consent statement to the user; In the display processing step, a portion to be read out in the consent sentence is determined based on the elapsed time, and the consent sentence indicating the timing of speech up to the determined portion to be read out is displayed. A consent determination method according to any one of [1] to [3]. [5] The computer device stores tolerance range information regarding a tolerance range of the emotion parameter in the storage unit, the analysis result includes one or more emotion parameters; In the consent determination step, if it is determined that the consent sentence has been uttered correctly based on the recognition result and the consent sentence, it is further determined whether one or more of the emotion parameters are within an acceptable range based on the analysis result and the acceptable range information, and the success or failure of the consent is determined. A consent determination method according to any one of [1] to [4]. [6] The emotion parameters include one or more parameters related to negative emotions. [5] The consent determination method described in [5]. [7] The consent determination method further comprises: a receiving step in which the at least one processing unit receives a selection of a determination subject from a determination subject list, which is a list of candidates for determination subjects for which the success or failure of consent is determined, by a supporter who confirms the success or failure of consent by the user; a registration step in which, when the consent determination step determines whether the consent has been granted, the at least one processing unit registers the selected determination subject and the determination result of the consent in association with user information; The consent determination method according to any one of [1] to [6], [8] A computer program that causes a computer device to execute the consent determination method described in any one of [1] to [7]. [9] A consent decision system, a voice receiving unit that receives a voice uttered by a user for the purpose of establishing consent; a speech recognition unit that recognizes the content of the user's speech as a language based on the speech; an emotion analysis unit that analyzes the emotion of the user based on the uttered voice; a consent determination unit that determines whether consent has been granted based on a speech recognition result by the speech recognition unit and an analysis result by the emotion analysis unit; A consent decision system comprising:

[0009] The invention of [1] makes it possible to determine whether consent has been established, such as whether the user truly consents, by utilizing the results of speech recognition and emotion analysis from the user's speech. For example, even if words that suggest consent are spoken, the user may not actually intend to consent, and it is possible to make a more detailed judgment of consent based on the user's intention, which can also handle cases where the user's intention is expressed through the nuances of the speech.

[0010] According to the invention of [2], once it is determined that consent has been established, the services provided by the supporter can be provided.

[0011] The invention according to [3] makes it easier for the user to understand how much of the consent sentence has been read out, and assists the user in reading out the consent sentence.

[0012] The invention according to [4] allows the user to easily determine the timing for reading out the consent statement.

[0013] [5] The invention of the present invention makes it possible to determine whether consent has been given or not, taking into account emotions not expressed in the user's words. For example, if the consent sentence is spoken correctly, and furthermore, by utilizing the emotional parameters included in the analysis results, consent will not be established if the user has spoken the word "agreement" but anxiety remains and the emotional parameter related to anxiety is outside the acceptable range.

[0014] [6] The invention of the present invention makes it possible to determine consent by referring to negative emotions, such as when anxiety is above a certain level, or when negative emotions reach a certain level, consent is not established.

[0015] According to the invention of [7], a supporter who provides services to a user can easily select a subject of judgment using a list. In addition, information about the selected subject of judgment can be easily registered in association with user information and the judgment result of the success or failure of consent. [Effects of the Invention]

[0016] The present invention can provide a novel technique for determining whether or not consent has been given by a user. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing the configuration of a system according to an embodiment. [Figure 2] FIG. 1 is a hardware configuration diagram of a system according to an embodiment. [Figure 3] FIG. 1 is a functional block diagram of a system according to an embodiment. [Figure 4] 10 is a flowchart illustrating a process for determining whether consent has been obtained in a system according to an embodiment. [Figure 5] 10 is an example of a screen display in a liquid system. [Figure 6] 10 shows an example of a screen display in the system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] The present invention will now be described more fully with reference to the accompanying drawings, in which preferred embodiments are shown, but which may be embodied in many different forms and are not limited to the embodiments set forth herein.

[0019] For example, in this embodiment, the configuration, operation, etc. of the consent determination system will be described, but similar effects can be achieved by an executed method, device, computer program, etc. The program may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server.

[0020] In this embodiment, a case will be described in particular where consent is confirmed between a user, such as a person with physical disabilities or the elderly, and a supporter, such as an operator, who provides services to the user or handles procedures on their behalf. Furthermore, users are not limited to the elderly; healthy people and young people without physical or cognitive disabilities can also be users. In this embodiment, the supporter who interacts with the user and provides support is a human. However, as long as the supporter is capable of interacting with the user and providing support, it may be a rule-based robot, such as a generative artificial intelligence (AI) or a chatbot, that can interact with the user. In this case, the supporter terminal 3 does not need to have a video call function.

[0021] Furthermore, in this embodiment, an example of confirming whether consent has been obtained in a non-face-to-face manner via a video call is described, but the consent determination system 0 may also be used when confirming consent in person rather than via a video call. Furthermore, in this embodiment, the user's consent is determined while having a conversation with the supporter, but the user may also determine consent using their own terminal device without having a conversation with the supporter. In this case, the consent determination system 0 may include only one terminal device 9 that has the functions of the user terminal 2 and the supporter terminal 3, and the user terminal 2 and the supporter terminal 3 may not have functional configurations for realizing video calls, such as video call processing units 21 and 31.

[0022] <System configuration> Fig. 1 is a block diagram showing the configuration of a system according to one embodiment. As shown in Fig. 1, the consent determination system 0 includes a consent determination device 1, a user terminal 2, and a supporter terminal 3, which are configured to be able to communicate with each other via a network 4. In this embodiment, the consent determination system 0 includes the user terminal 2 and the supporter terminal 3 in order to confirm agreement via a video call. However, if agreement is confirmed without face-to-face contact or conversation with a supporter, the system may include only one terminal device 9 that has the functions of the supporter terminal 3 and the user terminal 2.

[0023] One or more server devices, such as a general-purpose server or a personal computer, can be used as the consent determination device 1. In this embodiment, a video call between the user terminal 2 and the supporter terminal 3 is realized, for example, via a session conforming to the WebRTC (Web Real-Time Communication) standard, but may also be made by establishing a session conforming to standards such as HTTP or WebSocket. Furthermore, the consent determination device 1 may appropriately include a web server for screen display, a STUN server, a TURN server, a signaling server, etc. as resources for realizing the video call, but may also realize functions related to the video call by communicating with an external communication system. Hereinafter, explanation of the video call function will be omitted to the extent that it will not be unclear.

[0024] The user terminal 2 and the supporter terminal 3 can be terminal devices 9 such as personal computers, smartphones, tablet terminals, etc. The user terminal 2 and the supporter terminal 3 connect to the consent determination device 1 by executing a consent determination device utilization program, which will be described later, and perform processing related to determining whether consent has been granted. In this embodiment, the user terminal 2 and the supporter terminal 3 can make video calls by executing a video call program. In this embodiment, the consent determination device utilization program is a web browser application that is pre-installed or downloaded in advance to the terminal device 9, but it may also be a client application downloaded from a program providing device (not shown), etc.

[0025] In this embodiment, the network 4 is an IP (Internet Protocol) network, but there are no restrictions on the type of communication protocol, the type of network, etc.

[0026] <Hardware configuration> FIG. 2 is a hardware configuration diagram of the consent decision device 1 and the terminal device 9 (user terminal 2, supporter terminal 3).

[0027] Fig. 2(a) is a hardware configuration diagram of the consent determination device 1. As shown in Fig. 2(a), the consent determination device 1 has a processing unit 101, a storage unit 102, and a communication unit 103, which are used to perform the functions of each unit and each process.

[0028] The processing unit 101 has a processor such as a CPU that can execute an instruction set, and executes an OS, a consent decision program, and the like. The storage unit 102 includes a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS, a consent decision program, and the like. The communication unit 103 has an interface for physically connecting to a network, and controls communication with the network 4 to input and output information.

[0029] Fig. 2(b) is a hardware configuration diagram of the terminal device 9 (user terminal 2, supporter terminal 3). As shown in Fig. 2(b), the terminal device 9 has a processing unit 901, a memory unit 902, a communication unit 903, an input unit 904, and an output unit 905, which are used to perform the functions of each unit and each process.

[0030] The processing unit 901 has a processor such as a CPU that can execute an instruction set, and executes programs such as an OS and a program for using the consent decision device. The memory unit 902 has a volatile memory such as a RAM capable of storing an instruction set, an OS, a non-volatile recording medium such as an HDD or SSD capable of recording an application program (consent determination device utilization program) that can use multiple services including a consent determination device utilization program, etc. The communication unit 903 has an interface for physically connecting to a network, and controls communication with the network 4 to input and output information. The input unit 904 includes an operation input device capable of input processing such as a touch panel or keyboard, an audio input device capable of audio input such as a microphone, and an image input device capable of image input such as a camera. The output unit 905 includes a display device capable of display processing, such as a display, and an audio output device, such as a speaker.

[0031] <Functional configuration of consent decision system 0> FIG. 3 is a functional block diagram of the consent decision system 0. As shown in FIG. 3, the consent decision device 1 includes a voice recognition unit 11, an emotion analysis unit 12, a consent decision unit 13, and an execution unit 14. This is information processing by software (a consent decision program stored transiently or non-transiently in a storage unit 102) specifically realized by hardware (a processing unit 101, etc.). Although not shown in FIG. 3, the consent decision device 1 may further include a video call control unit that mediates the delivery of audio and images (including videos, etc.) related to video calls and controls processing related to video calls.

[0032] The user terminal 2 includes a video call processing unit 21, an audio reception unit 22, and a display processing unit 23. The supporter terminal 3 includes a video call processing unit 31, a reception unit 32, and a display processing unit 33. The functional configuration of these terminal devices 9 (user terminal 2, supporter terminal 3) is a specific implementation of information processing by software (such as a user or supporter consent determination device utilization program stored transiently or non-transiently in the storage unit 902, etc.) using hardware (such as the processing unit 901).

[0033] <Video call processing units 21 and 31> The video call processing units 21, 31 display a video call screen on the terminal device 9 (user terminal 2, supporter terminal 3) and execute video call processing. As part of the video call processing, the video call processing units 21, 31 transmit audio and images (including video) acquired via an audio input device and an image input device of the terminal device 9 (user terminal 2, supporter terminal 3) to the video call processing units 21, 31 of other terminal devices 9 (user terminal 2, supporter terminal 3) directly or via another consent determination device 1.

[0034] The supporter terminal 3 may also be equipped with a video call control unit, and may be able to handle from the supporter terminal 3 one or both of the controls for audio on / mute and video on / off input at the user terminal 2. Typically, the video call control unit in the supporter terminal 3 can be used to eliminate erroneous operations by the user.

[0035] <Reception Department 32> The reception unit 32 receives various inputs from the supporter via an input device such as a touch panel or a mouse, including the specification of a subject of judgment selected from candidates for subject of judgment displayed in the subject of judgment list and an instruction to start the judgment of the success or failure of consent. The subject of judgment is a subject for which the user's consent is judged, and includes, for example, the purchase of a product, application for a service, application for insurance or other procedures, consent to the contents of a contract document such as consent to the provision of personal information, etc. The user terminal 2 may also be provided with a reception unit 24 to receive various inputs from the user, such as the specification of a subject of judgment.

[0036] <Voice Reception Unit 22> The voice receiving unit 22 receives a voice (spoken voice) uttered by the user, acquired via a voice input device or the like in the user terminal 2, in order to obtain consent. The spoken voice is a voice uttered by the user for which consent is to be determined to be established, for the purpose of establishing consent, and in this embodiment, is a voice in which the user reads out a consent statement. The voice receiving unit 22 may pass the voice of the user reading out the consent statement to the end as a spoken sentence to the voice recognition unit 11 and the emotion analysis unit 12, or may pass streaming voice received in real time as the spoken voice. In this embodiment, the voice receiving unit 22 receives the spoken voice, which is voice data, and transmits it to the consent determination device 1, which then passes it to the voice recognition unit 11 and the emotion analysis unit 12.

[0037] In addition, when the user and the supporter have a conversation via video call and determine whether or not consent has been reached, the video call processing unit 21 functions as the voice receiving unit 22, transmits the received voice to the consent determination device 1, and passes the voice to the voice recognition unit 11 and the emotion analysis unit 12.

[0038] <Display Processing Unit 23, Display Processing Unit 33> The display processing unit 23 in the user terminal 2 displays a consent statement, which is a sentence read by the user, to determine whether the consent has been granted. The display processing unit 33 in the supporter terminal 3 displays a determination subject list, which is a list showing candidates for the determination subject.

[0039] <Speech Recognition Unit 11> The speech recognition unit 11 recognizes the content of the user's utterance as language based on the speech received by the speech receiving unit 22, performs processing (transcription) to output a recognition result that is linguistic information such as text, and passes the recognition result to the consent determination unit 13. In this embodiment, the speech recognition unit 11 performs processing related to speech recognition using a speech recognition model stored in the storage unit 102, but may also perform processing related to speech recognition using a speech recognition model managed by another system by using an API (Application Programming Interface). When the speech receiving unit 22 receives streaming speech being spoken in real time as the speech, the speech recognition unit 11 performs processing related to speech recognition in real time based on the received speech, and outputs the latest recognition result based on the speech acquired at that time. Furthermore, when an utterance of an incomplete consent sentence is received as the spoken voice, the voice recognition unit 11 outputs linguistic information up to the point where the utterance is completed (for example, "agree" up to the point where the utterance of the consent sentence "I agree") based on the received spoken voice, such as the voice up to the point where the user's utterance is completed, as the recognition result. Furthermore, when there are multiple pieces of linguistic information as candidates for the content of the user's utterance, the voice recognition unit 11 may pass the multiple pieces of linguistic information of the candidates as the recognition result to the consent determination unit 13.

[0040] <Emotion analysis section 12> The emotion analysis unit 12 performs a process of analyzing the user's emotion in the utterance from the intonation, volume, etc., based on the utterance voice received by the voice receiving unit 22, and outputs the analysis result. In this embodiment, the emotion analysis unit 12 passes the analysis result to the consent determination unit 13. In this embodiment, the emotion analysis unit 12 outputs the analysis result including, for example, parameters related to emotions (emotion parameters) such as "anxiety" and "confidence." The emotion parameters are parameters related to the user's emotion based on the utterance voice, and include parameters related to positive emotions such as happiness and enthusiasm, and parameters related to negative emotions such as anxiety and stress. In this embodiment, the emotion analysis unit 12 performs the emotion analysis process by using an emotion analysis model stored in the storage unit 102; however, the emotion analysis process may also be performed by using an emotion analysis model managed by another system by using an API (Application Programming Interface).

[0041] <Consent determination section 13> The consent determination unit 13 determines whether consent has been granted based on the recognition result from the voice recognition unit 11 and the analysis result from the emotion analysis unit 12. In this embodiment, the consent determination unit 13 determines whether the consent statement has been correctly uttered based on the recognition result and whether the emotion parameters based on the analysis result are within an allowable range, and then determines the final determination result regarding whether consent has been granted.

[0042] For example, when the consent sentence is "I agree," if the recognition result acquired as linguistic information is "I agree," the consent determination unit 13 determines that the user's utterance is correct if the recognition result acquired as linguistic information is "I agree," as well as the consent sentence. Furthermore, even if the recognition result and the consent sentence are not exactly identical, the consent determination unit 13 may determine that the consent sentence has been uttered correctly if the difference is within a predetermined range, such as a difference of one character or less, or a particularly important part (such as "agree") being the same and other parts being different.

[0043] When it is determined that the consent sentence has been correctly uttered, the consent determination unit 13 further determines whether one or more emotion parameters are within an acceptable range based on the analysis result and acceptable range information related to the acceptable range of the emotion parameter. At this time, the consent determination unit 13 determines whether the emotion parameter is within an acceptable range based on acceptable range information including thresholds and numerical ranges of emotion parameter values ​​stored in the storage unit 902, for example, determining that an emotion parameter of anxiety included in the analysis result is within an acceptable range if it is equal to or less than a threshold. In addition, an acceptable range for an emotion parameter may be set for each user, such that a person with a strong intonation in speech has a wider acceptable range. In this case, the consent determination unit 13 makes a determination based on the analysis result using the acceptable range information for each user associated with the user information.

[0044] <Executive Unit 14> When the consent determination unit 13 determines that consent has been established based on the recognition result and the analysis result, the execution unit 14 executes a support process based on the establishment of consent. The support process is a process related to the support provided by the supporter to the user and is a process related to the matter selected as the determination subject for the determination. For example, when a service application is selected as the determination subject, processes such as executing the service application and processing payment for the service fee are executed as the support process. At this time, the execution unit 14 executes a process corresponding to the determination subject selected by the supporter or user, such as product payment processing or insurance application processing, based on the selection of the determination subject accepted by the acceptance unit 32. In this embodiment, the support process is a process for purchasing goods, applying for insurance, applying for online medical care, transmitting personal information, etc. In addition, the execution unit 14 may perform a process for storing in the memory unit 902 a determination result indicating whether consent has been established, in association with the determination subject and user information.

[0045] <Registration Section 15> The registration unit 15 performs a process of storing the determination result of the consent determination unit 13, whether the consent has been granted, in the storage unit 102 in association with a determination subject selected by the user or supporter and user information of the user who has given consent, as a process of registering the determination result of the consent grant in association with the user information and the determination subject. At this time, the determination subject and the determination result of the consent grant (e.g., "determination subject: purchase agent for natural water", "determination result: consent granted", etc.) are registered in association with the user information. In addition, the process by the registration unit 15 may be executed as a process (support process) executed when the execution unit 14 grants consent.

[0046] <Consent determination method> FIG. 4 is a flowchart of a process for determining whether consent has been obtained using a video call.

[0047] <Video call screen display process> In step S101, a video call is started between the user terminal 2 and the supporter terminal 3. The video call processing units 21 and 31 display images (video) of the user and the supporter on the video call screens of the participants (user, supporter) of the video call.

[0048] <Video call screen> 5(a) is an example of a video call screen W1 displayed when the determination start button is pressed and the process of determining whether consent has been granted is started. The video call screen W1 is an example of a screen display when there are two participants. Note that any number of participants can participate in a video call, and more than two participants may be allowed to participate.

[0049] The video call screen W1 includes a supporter image W11, a consent statement display area W12, a user image W13, a shared screen display area W14, and an end call button W15. The video call screen W1 may also include buttons for other video call-related operations, such as an audio mute / unmute button and a video on / off button. The supporter image W11 and the user image W13 are screen displays displaying images of video call participants, including the supporter and the user, and may be arranged in a screen configuration other than that shown in FIG. 5(a). The consent statement display area W12 is a screen display displaying a consent statement processed by the display processing unit 23 or the display processing unit 33. In the example shown in FIG. 5(a), it is displayed superimposed on the supporter image W11, but may also be displayed superimposed on other screen displays on the video call screen, such as the user image W13 or the shared screen display area W14. The shared screen display area W14 is a screen display when the screen displayed on the supporter terminal 3 is shared. The shared screen display unit W14 is mainly a screen display that is shared by displaying content related to the judgment subject that determines whether consent has been obtained, but content related to the judgment subject may also be displayed based on the judgment subject selected by the supporter by the video call processing units 21, 31 or the display processing units 23, 33. In step S102, when the call end button W15 is pressed (YES in S102), the video call is ended.

[0050] <Judgment subject selection process> When end call is not selected (NO in step S102) and the call is continuing, if a display request for the determination subject list is sent from the supporter terminal 3 to the consent determination device 1, the display processing unit 33 of the supporter terminal 3 performs display processing of the determination subject list used to determine consent. At this time, the display processing unit 33 may send a display request for the determination subject list to the consent determination device 1 and perform display processing based on information about the determination subject (determination subject information) acquired from the consent determination device 1, or may perform display processing based on the determination subject information stored in the memory unit 902 of the supporter terminal 3. The determination subject information is information about the determination subject, includes the name of the determination subject, and is stored in the memory unit 102 or the memory unit 902 in association with information about the processing (support processing) that is executed when the establishment of consent is determined.

[0051] <Judgment subject list display screen> FIG. 5(b) is an example of the judgment subject list display screen W2. The judgment subject list display screen W2 is a screen for displaying a judgment subject list, which is a list of candidate judgment subjects for which the success or failure of consent is to be determined. It includes a judgment subject list W21 and a judgment start button W22. The judgment subject list W21 is a list of candidate judgment subjects. A judgment subject is selected from these candidate judgment subjects displayed in the list and pressed to select the judgment subject for which the success or failure of consent is to be determined. When the judgment start button W22 is pressed after the judgment subject is selected, an instruction to start the judgment of the success or failure of consent is transmitted from the supporter terminal 3, and the judgment process regarding the success or failure of consent is initiated. In addition, before the judgment subject list display screen W2 is displayed, a user for whom the success or failure of consent is to be determined is selected, and the judgment subject list display screen W2 includes display of information about the selected user (user information), such as the user ID and user name.

[0052] When a judgment subject is selected on the judgment subject list display screen W2, the reception unit 32 accepts the selection of the judgment subject (YES in step S103), and when the judgment start button W22 is pressed, an instruction to start judging the success or failure of consent is sent from the supporter terminal 3 to the user terminal 2 and / or the consent judgment device 1, and processing related to the judgment of consent is started.

[0053] <Agreement statement display process> When the consent determination is started, the display processing units 23 and 33 perform processing to display the consent statement (step S104). In this embodiment, when the start of consent determination is selected, the display processing units 23 and 33 perform processing to display the consent statement so that it is superimposed on the video call screen W1, as shown in Fig. 5(a). In the illustrated example, an instruction message instructing the user to read out the consent statement is displayed together with the consent statement.

[0054] Fig. 6 shows an example of how the display of the consent sentence displayed in the consent sentence display unit W12 in Fig. 5(a) changes. The consent sentence may be displayed to indicate the part of the speech based on the speech by the user, or may be displayed to indicate the timing and speed of speech, etc.

[0055] An example will be described in which the consent sentence is displayed to indicate the user's utterance. When the speech recognition unit 11 outputs a recognition result as linguistic information based on the spoken voice, the display processing units 23 and 33 identify the utterance portion, which is the portion of the consent sentence that has been correctly uttered, based on the recognition result and the consent sentence, and display the consent sentence with the utterance portion highlighted to indicate the identified utterance portion. At this time, the display processing units 23 and 33 repeatedly acquire the recognition result from the speech recognition unit 11, identify the utterance portion, and display the consent sentence with the identified utterance portion indicated until the user utters the end of the consent sentence, thereby identifying the utterance portion in accordance with the user's utterance and changing the display of the consent sentence.

[0056] In the example shown in FIG. 6, the display of the consent statement "I agree" changes based on the user's utterance. The consent statement indicating the utterance location is displayed by coloring the characters up to the utterance location in different colors for the consent statement "I agree" based on the utterance location. In this embodiment, the characters up to the utterance location that has already been uttered are colored in a highly visible color such as red or a bright color, and the characters after the utterance location that have not yet been uttered are colored in white or a low-brightness color, thereby displaying the consent statement with the utterance location highlighted. In the example shown in FIG. 6, the display processing units 23 and 33 display the consent statement indicating the utterance location by coloring the characters of the consent statement. However, the consent statement indicating the utterance location may also be displayed by pointing to the utterance location with an arrow or the like or by encircling the spoken characters. Furthermore, if the user does not correctly utter the consent statement to the end within a certain period of time from the start of utterance, the display processing units 23 and 33 may return the utterance location to the beginning and display the consent statement in a way that prompts the user to speak from the beginning.

[0057] Next, an example will be described in which the consent sentence is displayed to indicate the timing and speed of speaking. Even when the consent sentence indicates the timing of speaking, the consent sentence is displayed changing as shown in Fig. 6, just as when the display changes based on the user's speech. In this case, the storage unit 902 may store timing information regarding the timing of speaking in association with the consent sentence, and the display processing unit 23, 33 may display the consent sentence indicating the timing of speaking using this timing information.

[0058] The display processing unit 23, 33 indicates the portion of the consent sentence to be read aloud by the user by coloring the characters instructing speech in a color different from their original color as time passes based on the elapsed time, and displays the consent sentence indicating the timing of speech. At this time, the display processing unit 23, 33 determines the portion to be read aloud and, based on the determined portion to be read aloud, changes the portion to be read aloud after a certain time has passed for each character and colors the character to be the next portion to be read aloud. However, the display processing unit 23, 33 may also display each character determined as the portion to be read aloud in color when a predetermined time has passed since the start of display of the consent sentence, based on the timing of speech of each character included in the consent sentence and the elapsed time since the start of display of the consent sentence. Once the consent sentence has been colored to the end and the instruction for the timing of speech has been completed, the display processing unit 23, 33 determines whether the consent sentence was spoken correctly to the end based on the speech recognition result by the speech recognition unit 11. If it is determined that the consent sentence was not spoken correctly to the end, the consent sentence indicating the timing of speech may be displayed again from the beginning. In addition, in this embodiment, the display processing units 23 and 33 indicate the timing of speaking by displaying a consent sentence with colored characters, but it is also possible to display a consent sentence that indicates the timing of speaking by indicating the point to speak with an arrow or the like.

[0059] <Consent determination process> Hereinafter, a process for making an agreement decision based on the speech of the user who reads out the agreement sentence that has been processed for display by the display processing unit 23 will be described.

[0060] When the user speaks by reading out the displayed consent statement, the voice receiving unit 22 receives the spoken voice (step S105). Based on this spoken voice, the voice recognition unit 11 recognizes the spoken voice as language and outputs a recognition result, which is language information. The consent determination unit 13 determines whether the consent statement has been spoken correctly based on the recognition result, which is language information recognized based on the spoken voice, and the consent statement. When it is determined that the consent statement has been spoken correctly (YES in step S106), the emotion analysis unit 12 performs emotion analysis based on the spoken voice. The consent determination unit 13 further determines whether the emotion parameters included in the analysis result are within the allowable range based on the analysis result by the emotion analysis unit 12 and the allowable range information regarding the allowable range of the emotion parameters. When it is determined that the emotion parameters are within the allowable range (YES in step S107), the consent determination unit 13 determines that consent has been established (step S109).

[0061] When it is determined that the consent sentence was not spoken correctly (NO in step S106), or when it is determined that the emotion parameter is outside the allowable range (NO in step S107), a selection as to whether to make a judgment again is accepted from the user terminal 2 or the supporter terminal 3. When it is selected to make a judgment again (YES in step S108), the process of receiving the spoken voice (step S105) is executed again. When it is selected not to make a judgment again (NO in step S108), the consent judgment unit 13 determines the final judgment result of the success or failure of the consent as consent not being established (S110).

[0062] In this embodiment, the consent statement display process and the consent determination process are executed simultaneously. Also, in this embodiment, processes related to the video call and processes related to the consent determination, such as processes related to the consent determination, are executed again until the call end button is pressed.

[0063] After the process for determining whether or not consent has been performed (after step S109 or step S110), the execution unit 14 performs a support process and / or the registration unit 15 performs a process for registering the result of the determination on whether or not consent has been obtained.

[0064] <Modification> Furthermore, the consent decision device 1 may execute processing for setting an acceptable range of emotion parameters appropriate for each user. In this case, the consent decision device 1 may include a range setting unit, which may categorize users based on the results of the user emotion analysis, and the consent decision unit 13 may execute processing related to emotion parameter determination using the acceptable range for each category based on the classification results by the range setting unit. Furthermore, the method for setting the acceptable range of emotion parameters for each user is not limited to this, and the range setting unit may adjust the acceptable range based on the values ​​of emotion parameters included in the analysis results, such as the average, median, maximum, and minimum values ​​of emotion parameters in the entire user's utterances, such as by setting a higher threshold for a user with a high average value of emotion parameters.

[0065] Alternatively, the storage unit 902 may store a consent decision model for determining whether consent has been obtained, and the consent decision unit 13 may use this consent decision model to determine whether consent has been obtained. The consent decision model used in this case is a model that has been trained using one or more pieces of data such as the user's speech, recognition results, and emotion parameters as input data, and the success or failure of consent as output data.

[0066] When the supporter is an AI or the like that receives requests and the like from a user through dialogue, the reception unit 32 may receive the selection of a subject of determination based on the user's request acquired through dialogue with the user. At this time, the supporter generates a question asking which subject of determination to make based on the subject of determination information, and asks the user the question. Based on the content of the utterance obtained as the user's response to this question, the supporter may identify the subject of determination to be made, and may execute a process of transferring the identified subject of determination to the reception unit 32 as the selected subject of determination. [Explanation of symbols]

[0067] 0: Consent Decision System 1: Consent determination device 2: User terminal 3: Supporter terminal 4: Network 9: Terminal device 11: Voice recognition unit 12: Emotion analysis department 13: Consent determination department 14: Executive Department 15: Registration Department 21: Video Call Section 22: Voice reception section 23: Display processing section 31: Video call processing unit 32: Reception 33: Display processing section 101: Processing section 102: Storage section 103: Communications Department 901: Processing section 902: Storage section 903: Communications Department 904: Input section 905: Output section

Claims

1. A consent determination method executed by a computer device, comprising: the computing device includes at least one processing unit configured to execute computer-readable instructions stored in a memory unit; The consent determination method includes: a voice receiving step in which the at least one processing unit receives a voice uttered by a user for establishing consent; a speech recognition step in which the at least one processing unit recognizes the content of the user's speech as a language based on the speech; an emotion analysis step in which the at least one processing unit analyzes the emotion of the user based on the speech sound; a consent determination step in which the at least one processing unit determines whether consent has been given by the user based on a recognition result in the voice recognition step and an analysis result in the emotion analysis step; A consent determination method having the following.

2. The consent determination method further includes: the at least one processing unit has an execution step of executing a support process based on the establishment of consent when it is determined that consent is established in the consent determination step; The consent determination method according to claim 1 .

3. The consent determination method further includes: the at least one processing unit has a display processing step of displaying a consent statement to be read out to the user to obtain consent, and displaying the consent statement to the user; In the speech recognition step, the content of the user's utterance is recognized as a language based on the speech voice, which is a voice uttered by the user reading out the consent sentence; In the display processing step, a speech portion uttered in the agreement sentence is identified based on the recognition result obtained by the speech recognition step and the agreement sentence, and the agreement sentence showing the identified speech portion is displayed. The consent determination method according to claim 1 .

4. the at least one processing unit has a display processing step of displaying a consent statement to be read out to the user to obtain consent, and displaying the consent statement to the user; In the display processing step, a portion to be read out in the consent sentence is determined based on the elapsed time, and the consent sentence indicating the timing of speech up to the determined portion to be read out is displayed. The consent determination method according to claim 1 .

5. the computer device stores tolerance range information regarding tolerance ranges of emotion parameters in the storage unit; the analysis result includes one or more emotion parameters; In the consent determination step, if it is determined that the consent sentence has been uttered correctly based on the recognition result and the consent sentence, it is further determined whether one or more of the emotion parameters are within an acceptable range based on the analysis result and the acceptable range information, and the success or failure of the consent is determined. The consent determination method according to claim 1 .

6. the emotion parameters include one or more parameters related to negative emotions; The consent determination method according to claim 5 .

7. The consent determination method further includes: a receiving step in which the at least one processing unit receives a selection of a determination subject from a determination subject list, which is a list of candidates for determination subjects for which the success or failure of consent is determined, by a supporter who confirms the success or failure of consent by the user; a registration step in which, when the consent determination step determines whether the consent has been granted, the at least one processing unit registers the selected determination subject and the determination result of the consent in association with user information; The consent determination method according to claim 1 , comprising:

8. A computer program that causes a computer device to execute the consent determination method according to any one of claims 1 to 7.

9. A consent decision system, a voice receiving unit that receives a voice uttered by a user for the purpose of establishing consent; a speech recognition unit that recognizes the content of the user's speech as a language based on the speech; an emotion analysis unit that analyzes the emotion of the user based on the uttered voice; a consent determination unit that determines whether consent has been granted based on a speech recognition result by the speech recognition unit and an analysis result by the emotion analysis unit; A consent decision system comprising:

Images