Real-time conversation interpretation method and system using sensor-based language switching control

Abstract

According to one embodiment, a real-time conversation interpretation method using a terminal is disclosed. The method sets the first language of a first user as the current input language and the second language of a second user as the current output language, and then sorts voice data received from a microphone and sensor data received from a gyroscope sensor and an accelerometer sensor in chronological order. Subsequently, it calculates a silence time and a speech duration state value, and determines whether the movement of the terminal corresponds to a language switching candidate action based on the difference between the terminal's current direction angle and a reference direction angle corresponding to the first user's direction, angular velocity, change in angular velocity, change in acceleration, rotation duration, and rotation direction. If a language switching candidate action is confirmed, an unintended behavior judgment model analyzes the interrelationships between input tokens included in an integrated input sequence and calculates a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score. A processor calculates a single switching score based on the scores, and if the single switching score is greater than or equal to the switching reference score, it changes the current input language to the second language, changes the current output language to the first language, and changes the current speaker to the second user. In addition, if it is determined that the terminal has returned to a reference direction corresponding to the direction of the first user, the current input language, current output language, and current speaker can be reset to their initial state.

Description

Technology Field

[0001] The embodiments disclosed in this specification relate to real-time conversational interpretation technology, specifically to real-time conversational interpretation technology that switches between an input language and an output language between a first user's first language and a second user's second language using motion data obtained from a gyroscope sensor and an accelerometer sensor of a terminal. Background Technology

[0002] Real-time interpretation technology utilizing speech recognition, machine translation, and speech synthesis can be used for conversations between users speaking different languages. In such interpretation technology, processing can be performed to convert input speech into text, translate the converted text into the other party's language, and output the translated text as speech.

[0003] However, in a one-on-one conversation environment, the input language and output language may change depending on which user is the current speaker. For example, if the first user speaks in the first language, the first language may become the current input language and the second language may become the current output language, and if the second user speaks in the second language, the second language may become the current input language and the first language may become the current output language. If such language switching relies on button operation or screen touch, the user must perform a separate operation during the flow of conversation, which may result in operation delays or omissions.

[0004] Furthermore, when language switching is performed using only the orientation angle of the terminal, unintended movements such as hand tremors, shaking while moving, changes in the terminal's grip position, or changes in posture during speech may be recognized as language switching actions. In this case, the current input language and the current output language may be exchanged differently than intended, and an incorrect language state may persist for subsequent speech. Therefore, a technical configuration is required that utilizes the terminal's orientation switching action as an interpretation direction switching interface while distinguishing switching caused by unintended behaviors and stably managing the language state. The problem to be solved

[0005] According to one embodiment, a real-time conversation interpretation method is provided that can automatically exchange the current input language and the current output language based on the direction change operation of a terminal in a real-time conversation situation using the first language of a first user and the second language of a second user.

[0006] According to one embodiment, a real-time conversation interpretation method is provided that can suppress language switching caused by changes in grip or shaking during speech by aligning sensor data obtained from a gyroscope sensor and an accelerometer sensor of a terminal and voice data obtained from a microphone in chronological order, and analyzing the rotation state of the terminal and the speech sustainment state of the current speaker together.

[0007] According to one embodiment, a real-time conversational interpretation method is provided that determines a language switching candidate action based on a relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration, rotation direction, silence time, and speech duration state value, and determines a language switching control value by calculating a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score through an unintended behavior judgment model.

[0008] According to one embodiment, a real-time conversation interpretation method is provided that, even when the current input language, current output language, or current speaker is updated differently from the intention during the language switching process, the terminal can reset the current input language to the first language, the current output language to the second language, and the current speaker to the first user by utilizing an operation in which the terminal returns to a reference direction corresponding to the direction of the first user. means of solving the problem

[0009] According to one embodiment, a real-time conversation interpretation method using a terminal comprises: the terminal including an input unit, a microphone, a gyroscope sensor, an accelerometer sensor, a voice output unit, a memory, and one or more processors; by the processor, an operation of setting a first language of a first user as the current input language, setting a second language of a second user as the current output language, and setting the current speaker as the first user; by the processor, an operation of assigning time information according to the same time standard to voice data received from the microphone and sensor data regarding the movement of the terminal received from the gyroscope sensor and the accelerometer sensor, or by referring to the time information included in the voice data and the sensor data to sort the voice data and the sensor data in chronological order; and by the processor, an operation of calculating a speech continuation state value indicating a silent time and whether the current speaker is continuing to speak based on the voice data.The above processor determines whether the movement of the terminal corresponds to a language switching candidate operation based on a relative angle change amount representing the difference between a first current direction angle representing the current direction of the terminal and a first reference direction angle corresponding to the direction of the first user, an angular velocity representing the rotational speed of the terminal, a change amount of the angular velocity, a change amount of the acceleration of the terminal, a rotational duration of the terminal, a rotational direction of the terminal, a silence time, and a speech duration state value; and if the movement of the terminal corresponds to the language switching candidate operation, calculates a language switching control value indicating whether to exchange the current input language and the current output language based on the relative angle change amount, the angular velocity, the change amount of acceleration, the rotational duration, the rotational direction, the silence time, the speech duration state value, the current input language, the current output language, the current speaker, a direction switching history, and the difference between the first reference direction angle and the first current direction angle. The above processor includes, when the language switching control value satisfies a reference condition, an operation to change the current input language to the second language, translate the current output language to the first language, and change the current speaker to the second user.

[0010] The above method may further include: an operation by the processor to calculate a second direction angle representing the current direction of the terminal based on sensor data received from the gyroscope sensor and the accelerometer sensor after the current speaker is changed to the second user; an operation by the processor to determine whether the difference between the second direction angle and the reference direction angle is less than or equal to the return allowable angle; an operation by the processor to determine that the terminal has returned to a second reference direction corresponding to the direction of the first user if the state in which the difference between the current direction angle and the reference direction angle is less than or equal to the return allowable angle persists for a reference time or longer and the angular velocity of the terminal is less than or equal to the stable angular velocity; and an operation by the processor to change the current input language to the first language, change the current output language to the second language, and change the current speaker to the first user when it is determined that the terminal has returned to the second reference direction.

[0011] The operation of calculating the language switching control value comprises: an operation by the processor to determine whether the movement of the terminal corresponds to a rotational movement from the direction of the first user toward the direction of the second user, based on the relative angle change amount, the angular velocity, the change amount of the angular velocity, the acceleration change amount, the rotation duration, and the rotation direction; an operation by the processor to determine whether, if the movement of the terminal corresponds to the rotational movement, the rotational movement is a movement that occurred while the current speaker's speech is continuing, based on the silence time and the speech duration state value; and an operation by the processor to classify the rotational movement as a candidate for unintended action if the rotational movement is determined to be a movement that occurred while the current speaker's speech is continuing, and classify the rotational movement as a candidate for language switching intent if the rotational movement is determined to be a movement that occurred after the current speaker's speech has ended. The operation of converting the relative angle change amount, the angular velocity, the acceleration change amount, the rotation duration, the rotation direction, the silence time, the speech duration state value, the current input language, the current output language, the current speaker, the direction change history, and the difference between the first reference direction angle and the first current direction angle into input tokens, respectively by the processor, and arranging the input tokens in chronological order to generate an integrated input sequence; the operation of inputting the integrated input sequence into an unintended behavior judgment model by the processor; and the operation of analyzing the temporal relationship and intercorrelation between the input tokens included in the integrated input sequence through the unintended behavior judgment model by the processor.The operation of the processor to calculate, through the unintended behavior judgment model, the degree to which the rotation direction of the terminal aligns with the direction of the second user based on the relative angle change amount and the rotation direction, to calculate the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern based on the angular velocity and the rotation duration, to calculate the degree to which the movement of the terminal corresponds to shaking based on the acceleration change amount, and to calculate the degree to which the rotation motion corresponds to the speech alternation point based on the silence time and the speech duration state value; the operation of the processor to output a transition intention score, an unintended behavior score, a direction matching score, a speech alternation suitability score, and a judgment confidence score for the movement of the terminal through the unintended behavior judgment model; the operation of the processor to calculate a single transition score by reflecting the transition intention score, the direction matching score, the speech alternation suitability score, and the judgment confidence score as factors supporting the transition, and reflecting the unintended behavior score, the speech duration state value, the acceleration change amount, and the direction transition history as factors suppressing the transition; and the processor may include an operation of calculating the language switching control value as a value indicating switching allowance when the single switching score is greater than or equal to the switching threshold score, and calculating the language switching control value as a value indicating switching hold when the single switching score is less than the switching threshold score.;

[0012] The operation of analyzing the temporal relationship and intercorrelation among input tokens included in the integrated input sequence through the unintended behavior judgment model comprises, by the input alignment layer of the unintended behavior judgment model, aligning input tokens corresponding to the relative angle change amount, the angular velocity, the change amount of the angular velocity, the change amount of the acceleration, the rotation duration, the rotation direction, the silence time, the speech duration state value, the current input language, the current output language, the current speaker, and the direction change history included in the integrated input sequence within a common time interval based on the point in time when the language switching candidate action occurred; The operation of normalizing, by means of the sensor feature normalization layer of the unintended behavior judgment model, an input token corresponding to the relative angle change amount, an input token corresponding to the angular velocity, an input token corresponding to the change amount of the angular velocity, an input token corresponding to the acceleration change amount, an input token corresponding to the rotation duration, and an input token corresponding to the rotation direction with respect to the first reference direction angle, and acquiring sensor feature tokens representing the rotation magnitude, rotation speed, change in rotation speed, change in acceleration, rotation duration, and rotation directionality of the terminal; the operation of generating speech context tokens representing at least one of a state in which the current speaker's speech continues, a state immediately after the current speaker's speech ends, a state in which silence is maintained, and a state in which speech is unclear, based on the input token corresponding to the silence time and the input token corresponding to the speech duration state value, by means of the speech context encoding layer of the unintended behavior judgment model;The operation of converting the sensor feature tokens, the utterance context tokens, the input language state token corresponding to the current input language, the output language state token corresponding to the current output language, the speaker state token corresponding to the current speaker, and the history token corresponding to the direction change history into embedding vectors of the same dimension by means of the token embedding layer of the unintended behavior judgment model, and adding relative time information based on the time of occurrence of the language change candidate action to the embedding vectors; the operation of applying one of a plurality of attention heads to each of the embedding vectors to which the relative time information has been added by means of the multi-head attention layer of the unintended behavior judgment model; and the operation of generating a direction matching context vector representing the relationship between the rotation direction of the terminal and the direction of the second user by means of the direction matching head among the plurality of attention heads, by referencing the embedding vectors corresponding to the relative angle change amount and the rotation direction. An operation to generate a rotation pattern context vector representing the relationship between the change in rotation speed and the rotation duration of the terminal by referencing embedding vectors corresponding to the angular velocity, the amount of change in angular velocity, and the rotation duration by means of a rotation pattern head among the plurality of attention heads; an operation to generate a shake context vector for distinguishing whether the movement of the terminal is a rotational movement or a shake movement by referencing embedding vectors corresponding to the amount of change in acceleration and the amount of change in angular velocity by means of a shake detection head among the plurality of attention heads; an operation to generate a speech state context vector for distinguishing whether the movement of the terminal is a movement that occurred during the current speaker's speech or a movement that occurred after the end of speech by referencing embedding vectors corresponding to the silent time and the speech duration state value by means of a speech state head among the plurality of attention heads;The method may include: an operation to generate a history context vector indicating recent repetitive direction change candidate actions or instability of language states by referencing an embedding vector corresponding to the direction change history and an embedding vector corresponding to the current speaker by means of a history stability head among the plurality of attention heads; and an operation to generate an integrated context vector by combining the direction matching context vector, the rotation pattern context vector, the shake context vector, the utterance state context vector, and the history context vector by means of a context fusion layer of the unintended behavior judgment model, and applying normalization processing and non-linear transformation processing to the combined context vector.

[0013] The operation of calculating the degree to which the rotational motion corresponds to the speech alternation point comprises: an operation by the direction matching score calculation layer of the unintended behavior judgment model to calculate a direction matching score indicating the degree to which the direction at the point of rotation termination of the terminal is included in the direction range corresponding to the second user's direction, based on an embedding vector corresponding to the relative angle change amount, an embedding vector corresponding to the rotation direction, and direction range information corresponding to the second user's direction; an operation by the rotation pattern score calculation layer of the unintended behavior judgment model to calculate a transition intention score indicating the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern, based on an embedding vector corresponding to the angular velocity, an embedding vector corresponding to the rotation duration, and an embedding vector corresponding to the change amount of the angular velocity; and an operation by the shake score calculation layer of the unintended behavior judgment model to calculate an unintended behavior score indicating the degree to which the movement of the terminal corresponds to shake, based on an embedding vector corresponding to the acceleration change amount, an embedding vector corresponding to the change amount of the angular velocity, and an embedding vector corresponding to the rotation duration. And, by the speech alternation score calculation layer of the above unintended behavior judgment model, it may include an operation of calculating a speech alternation suitability score that indicates the degree to which the rotational motion corresponds to the speech alternation time based on an embedding vector corresponding to the silence time, an embedding vector corresponding to the speech duration state value, and time information corresponding to the time of occurrence of the rotational motion. Effects of the invention

[0014] According to one embodiment, a real-time conversation interpretation method using a terminal can exchange the current input language and the current output language between the first language of a first user and the second language of a second user according to the direction switching operation of the terminal, so that the user can set the interpretation direction suitable for the conversation partner without separate button operation or screen touch.

[0015] According to one embodiment, the real-time conversation interpretation method can analyze the relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration, and rotation direction of the terminal together, so it can limit language switching due to hand tremors or temporary changes in posture compared to when only simple changes in direction angle are used.

[0016] According to one embodiment, a real-time conversation interpretation method can determine whether the current speaker is continuing to speak by using a silence time and a speech duration state value together, thereby reducing situations where an action of changing the position of holding the terminal while the current speaker is speaking is processed as a language switch.

[0017] According to one embodiment, a real-time conversation interpretation method can calculate a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score using an unintended behavior judgment model, and calculate a language switching control value based thereon, thereby enabling a language switching judgment that reflects the temporal relationship between sensor data and speech state data.

[0018] According to one embodiment, a real-time conversation interpretation method can reset the language state based on the first language, the second language, and the first user by determining whether the terminal returns to a reference direction corresponding to the direction of the first user, even if the current input language, the current output language, or the current speaker is updated differently from the intention during the language switching process.

[0019] According to one embodiment, a real-time conversation interpretation method can convert a voice conversation between a first user and a second user into text according to the current input language, convert it into a translation text corresponding to the current output language, and output the translation text as voice, so that bidirectional voice interpretation can be continuously performed in a one-on-one conversation situation. Brief explanation of the drawing

[0020] FIG. 1 is a schematic diagram illustrating the configuration of a system according to one embodiment. FIG. 2 is a block diagram illustrating the hardware configuration of a real-time conversation interpretation system according to one embodiment. FIG. 3 is a flowchart exemplifying the overall operation flow of a real-time conversation interpretation method according to one embodiment. FIG. 4 is a block diagram exemplarily illustrating the layer configuration of an unintended behavior judgment model according to one embodiment. Specific details for implementing the invention

[0021] FIG. 1 is a schematic diagram illustrating the configuration of a system according to one embodiment.

[0022] The embodiments disclosed in this specification relate to real-time conversational interpretation technology, specifically to real-time conversational interpretation technology that switches between an input language and an output language between a first user's first language and a second user's second language using motion data obtained from a gyroscope sensor and an accelerometer sensor of a terminal.

[0023] Real-time interpretation technology utilizing speech recognition, machine translation, and speech synthesis can be used for conversations between users speaking different languages. In such interpretation technology, processing can be performed to convert input speech into text, translate the converted text into the other party's language, and output the translated text as speech.

[0024] However, in a one-on-one conversation environment, the input language and output language may change depending on which user is the current speaker. For example, if the first user speaks in the first language, the first language may become the current input language and the second language may become the current output language, and if the second user speaks in the second language, the second language may become the current input language and the first language may become the current output language. If such language switching relies on button operation or screen touch, the user must perform a separate operation during the flow of conversation, which may result in operation delays or omissions.

[0025] Furthermore, when language switching is performed using only the orientation angle of the terminal, unintended movements such as hand tremors, shaking while moving, changes in the terminal's grip position, or changes in posture during speech may be recognized as language switching actions. In this case, the current input language and the current output language may be exchanged differently than intended, and an incorrect language state may persist for subsequent speech. Therefore, a technical configuration is required that utilizes the terminal's orientation switching action as an interpretation direction switching interface while distinguishing switching caused by unintended behaviors and stably managing the language state.

[0026] According to one embodiment, a real-time conversation interpretation method is provided that can automatically exchange the current input language and the current output language based on the direction change operation of a terminal in a real-time conversation situation using the first language of a first user and the second language of a second user.

[0027] According to one embodiment, a real-time conversation interpretation method is provided that can suppress language switching caused by changes in grip or shaking during speech by aligning sensor data obtained from a gyroscope sensor and an accelerometer sensor of a terminal and voice data obtained from a microphone in chronological order, and analyzing the rotation state of the terminal and the speech sustainment state of the current speaker together.

[0028] According to one embodiment, a real-time conversational interpretation method is provided that determines a language switching candidate action based on a relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration, rotation direction, silence time, and speech duration state value, and determines a language switching control value by calculating a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score through an unintended behavior judgment model.

[0029] According to one embodiment, a real-time conversation interpretation method is provided that, even when the current input language, current output language, or current speaker is updated differently from the intention during the language switching process, the terminal can reset the current input language to the first language, the current output language to the second language, and the current speaker to the first user by utilizing an operation in which the terminal returns to a reference direction corresponding to the direction of the first user.

[0030] Referring to FIG. 1, a system (100) according to one embodiment may include a terminal (110), a first user (121), and a second user (123). The system (100) may further include a backend server (130) and a network (140).

[0031] Each operation described in the claim in the system (100) may be performed by one or more processors included in the terminal (110) and may be performed by one or more processors included in the backend server (130), and some operations may be distributed and performed between the terminal (110) and the backend server (130).

[0032] The system (100) may be configured to perform real-time conversational interpretation between the first language of the first user (121) and the second language of the second user (123). For example, the operation of setting the first language as the current input language and the second language as the current output language may be performed by the processor of the terminal (110), and at least some of speech recognition, translation, or speech synthesis may be performed by the processor of the backend server (130). Accordingly, the system (100) may selectively combine local processing and server processing depending on the computational resources, network status, or service policy of the terminal (110).

[0033] The system (100) can control the direction of interpretation based on voice data received from the microphone of the terminal (110) and sensor data received from the gyroscope sensor and the accelerometer sensor. The operation of sorting the voice data and sensor data in chronological order, the operation of calculating a speech duration state value, the operation of calculating a language switching control value, and the operation of changing the current input language and the current output language may be performed at the terminal (110), or may be performed at the backend server (130) after the terminal (110) transmits the necessary data to the backend server (130).

[0034] The terminal (110) may be a user-side device for interpreting conversation between a first user (121) and a second user (123). The terminal (110) may include an input unit, a microphone, a gyroscope sensor, an accelerometer sensor, a voice output unit, a memory, and one or more processors, and in some embodiments may further include a display, a communication unit, a storage device, or a sensor hub. The processor of the terminal (110) may be configured to directly perform at least some of the operations described in the claims.

[0035] The terminal (110) can serve as the center of a hardware configuration for collecting voice data and sensor data. The microphone of the terminal (110) receives speech from a first user (121) or a second user (123), and the gyroscope sensor and accelerometer sensor can generate sensor data regarding the movement of the terminal (110). The terminal (110) can process the generated voice data and sensor data in its own processor or transmit them to a backend server (130) via a network (140) so that a server-side processor can perform subsequent processing.

[0036] The terminal (110) can exchange the current input language and the current output language according to the result of calculating the language switching control value. In this case, the language switching control value may be calculated directly by the processor of the terminal (110), or may be calculated by the processor of the backend server (130) and provided to the terminal (110). The terminal (110) can update the current speaker, the current input language, and the current output language based on the received or calculated language switching control value.

[0037] The first user (121) may be a user who uses the first language. In one embodiment, the first language of the first user (121) may be set as the current input language at the start of a conversation, and the second language of the second user (123) may be set as the current output language. This language state setting may be performed by the processor of the terminal (110) or the processor of the backend server (130) based on language information entered through the input section of the terminal (110).

[0038] The first user (121) can rotate the terminal (110) from the direction of the first user (121) to the direction of the second user (123). At this time, the sensor data of the terminal (110) can be used to calculate the difference between the current direction angle and the first reference direction angle, the angular velocity, the amount of change in acceleration, the duration of rotation, and the direction of rotation. This calculation processing may be performed at the terminal (110), or it may be performed at the backend server (130) after the terminal (110) transmits the sensor data to the backend server (130).

[0039] When the utterance of the first user (121) is input, the system (100) can perform speech recognition based on the first language, translate it into the second language, and then output it as speech. At this time, speech recognition, translation, and speech synthesis may all be performed at the terminal (110) or at the backend server (130), and, for example, may be distributed such that speech recognition is performed at the terminal (110) and translation and speech synthesis are performed at the backend server (130).

[0040] The second user (123) may be a counterpart user who uses the second language. The system (100) may change the current input language to the second language and the current output language to the first language and change the current speaker to the second user (123) when the language switching control value satisfies the reference condition. In this state, the speech of the second user (123) may be recognized based on the second language and translated into the first language and output.

[0041] The direction of the second user (123) can be used as a criterion for determining whether the rotation direction of the terminal (110) corresponds to the actual interpretation direction change intention. For example, the system (100) can calculate the extent to which the direction at the end of the rotation of the terminal (110) falls within the direction range corresponding to the direction of the second user (123). This direction matching score can be calculated by the processor of the terminal (110) or the processor of the backend server (130).

[0042] When the speech of the second user (123) ends or the terminal (110) returns to the direction of the first user (121), the system (100) may reset the current input language and the current output language based on the first language and the second language. The determination of whether to return may be based on sensor data generated from the gyroscope sensor and the accelerometer sensor, and the decision-maker may be the processor of the terminal (110) or the processor of the backend server (130).

[0043] The backend server (130) may be a server device capable of communicating with the terminal (110) via a network (140). The backend server (130) may include one or more processors and memory and may process voice data, sensor data, speech state data, integrated input sequences, or language switching determination requests received from the terminal (110). The processor of the backend server (130) may be configured to perform at least some of the operations described in the claims.

[0044] The backend server (130) may execute at least one of a speech recognition model, a translation model, a speech synthesis model, or an unintended behavior determination model. For example, the backend server (130) may convert speech data received from the terminal (110) into text, convert the text into translation text corresponding to the current output language, and generate synthetic speech data corresponding to the translation text and return it to the terminal (110). In another embodiment, the backend server (130) may calculate only the language switching control value, and the interpretation output may be performed by the terminal (110).

[0045] The backend server (130) can calculate a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score using an unintended behavior judgment model. In this case, the terminal (110) can transmit at least some of the relative angle change amount, angular velocity, acceleration change amount, rotation duration, rotation direction, silence time, speech duration status value, and direction switching history to the backend server (130). The backend server (130) provides the calculated results to the terminal (110), and the terminal (110) can update the language state based on this.

[0046] The network (140) may be a communication path that supports data transmission and reception between the terminal (110) and the backend server (130). The network (140) may include a mobile communication network, a Wi-Fi network, a Bluetooth communication network, a short-range wireless communication network, a wired communication network, or a combination thereof. The network (140) may transmit voice data, sensor data, text data, translated text, synthesized speech data, model input data, model output data, or language switching control values.

[0047] In an embodiment using the network (140), the terminal (110) may adjust the transmission range of data to maintain real-time. For example, if the network delay is below a threshold value, the terminal (110) may request voice recognition, translation, voice synthesis, or unintended behavior determination from the backend server (130). Conversely, if the network delay exceeds the threshold value or the connection is unstable, the terminal (110) may perform at least some operations using its own processor.

[0048] The network (140) can provide a communication basis that enables the terminal (110) and the backend server (130) to be interconnected as a single system (100). Accordingly, the term “processor” described in the claim may be interpreted to include the processor of the terminal (110), the processor of the backend server (130), or a combination thereof, and the system (100) may perform each operation in an on-device manner, a server manner, or a hybrid manner.

[0049] According to one embodiment, a real-time conversation interpretation method using a terminal can exchange the current input language and the current output language between the first language of a first user and the second language of a second user according to the direction switching operation of the terminal, so that the user can set the interpretation direction suitable for the conversation partner without separate button operation or screen touch.

[0050] According to one embodiment, the real-time conversation interpretation method can analyze the relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration, and rotation direction of the terminal together, so it can limit language switching due to hand tremors or temporary changes in posture compared to when only simple changes in direction angle are used.

[0051] According to one embodiment, a real-time conversation interpretation method can determine whether the current speaker is continuing to speak by using a silence time and a speech duration state value together, thereby reducing situations where an action of changing the position of holding the terminal while the current speaker is speaking is processed as a language switch.

[0052] According to one embodiment, a real-time conversation interpretation method can calculate a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score using an unintended behavior judgment model, and calculate a language switching control value based thereon, thereby enabling a language switching judgment that reflects the temporal relationship between sensor data and speech state data.

[0053] According to one embodiment, a real-time conversation interpretation method can reset the language state based on the first language, the second language, and the first user by determining whether the terminal returns to a reference direction corresponding to the direction of the first user, even if the current input language, the current output language, or the current speaker is updated differently from the intention during the language switching process.

[0054] According to one embodiment, a real-time conversation interpretation method can convert a voice conversation between a first user and a second user into text according to the current input language, convert it into a translation text corresponding to the current output language, and output the translation text as voice, so that bidirectional voice interpretation can be continuously performed in a one-on-one conversation situation.

[0055] Definitions of terms are described below.

[0056] The current input language refers to the language that serves as the standard when converting voice data received from the microphone into text at a specific point in time. The current output language refers to the relative language that serves as the standard when text recognized as the current input language is translated and provided as voice or text. The current speaker refers to the user set as the speaker at a specific point in time and may be set as either the first user or the second user.

[0057] Voice data may include data regarding speech signals received through a microphone, periods of silence, voice intensity, the start time of speech, and the end time of speech. Sensor data refers to data regarding the orientation, rotation, angular velocity, acceleration, and attitude changes of the terminal received from the gyroscope and accelerometer sensors. The same time standard refers to a common time system established so that voice data and sensor data can be compared on the same time axis even if they are collected at different sampling periods.

[0058] Time information refers to a time stamp, sequence information, or relative time information indicating the point in time when voice data or sensor data was collected or generated. Silence time refers to the duration of an interval during which the current speaker's voice is not detected and can be used to determine whether a speech takeover is possible. The speech duration status value refers to a status value indicating whether the current speaker is continuing to speak, has ended the speech, is maintaining silence, or whether the speech status is unclear.

[0059] The first reference direction angle is a reference angle corresponding to the direction of the first user and represents a reference value for calculating the change in direction of the terminal. The first current direction angle is an angle value representing the current direction of the terminal at a specific point in time and is used to calculate the relative change in angle by comparing it with the first reference direction angle. The relative change in angle refers to the difference between the first reference direction angle and the first current direction angle, and can indicate how much the terminal has rotated from the direction of the first user.

[0060] A language switching candidate action refers to the movement of a terminal selected as a target for language switching judgment based on the terminal's relative angle change amount, angular velocity, rotation direction, rotation duration, silence duration, and speech duration status value. Direction switching history refers to records regarding previously occurred language switching candidate actions, language switching allowed, language switching suspended, and judgments regarding changing or returning the current speaker. The language switching control value is a control value indicating whether to exchange the current input language and the current output language, and can be calculated as a value indicating switching allowed or a value indicating switching suspended.

[0061] The criterion condition refers to the condition that must be satisfied for the language switching control value to be determined as a value indicating switching permission, and may include criteria for a single switching score or other judgment scores. The second direction angle refers to an angle value indicating the current direction of the terminal after the current speaker has changed to the second user. The reference direction angle refers to the direction angle used as a comparison criterion to determine whether the terminal has returned to the direction of the first user.

[0062] The allowable return angle refers to the range in which the difference between the terminal's current direction angle and the reference direction angle is recognized as a return. The reference time refers to the duration condition used to determine whether the terminal remains stably within the direction range corresponding to the reference direction angle. The stable angular velocity refers to the angular velocity reference value used to determine whether the terminal is continuously rotating or stably positioned in a specific direction.

[0063] The second reference direction refers to a directional state where the terminal is judged to have returned to the reference direction corresponding to the first user's direction after the current speaker has changed to the second user. Unintended behavior candidates refer to candidates classified as being close to changes in grip, hand tremors, shaking while moving, or temporary posture adjustments during the current speaker's speech, even if terminal movement is present. Language switching intention candidates refer to candidates where the terminal's rotational movement is classified as being close to an intentional change of direction to exchange the current input language with the current output language.

[0064] Input tokens refer to data converted into model processing units from input elements such as relative angle change, angular velocity, acceleration change, silence duration, utterance duration status, current input language, current output language, current speaker, or direction change history. An integrated input sequence refers to a sequence of tokens configured to arrange multiple input tokens in chronological order so that they can be input into an unintended behavior judgment model. An unintended behavior judgment model refers to a model that analyzes sensor, utterance, language status, and history information included in the integrated input sequence to determine whether the device's movement is an intentional language switching action or an unintended behavior.

[0065] Interrelationship refers to the temporal, state, or semantic relationship between input tokens included in an integrated input sequence, and may include, for example, the relationship between the point of rotation and the silence time, or the relationship between angular velocity and the change in acceleration. The direction match score refers to a score indicating the extent to which the direction at the terminal's point of rotation termination falls within the direction range corresponding to the second user's direction. The switching intention score refers to a score indicating the extent to which the terminal's rotation pattern corresponds to the intended rotation pattern for language switching.

[0066] The unintended behavior score refers to a score indicating the degree to which the device's movement responds to shaking, changes in grip, posture adjustments during speech, or impactful movements. The speech change point refers to the moment when the current speaker's speech ends or a silence period is formed, allowing the speech order to switch to the conversation partner. The speech change suitability score refers to a score indicating the degree to which the device's rotational movement responds to the speech change point.

[0067] Elements supporting transition refer to factors that reinforce the judgment of exchanging the current input language with the current output language, and may include a transition intention score, a direction match score, and an utterance alternation suitability score. Elements inhibiting transition refer to factors that reinforce the judgment of maintaining the current input language and the current output language, and may include an unintentional behavior score, an utterance persistence state value, a change in acceleration, and a direction change history. A single transition score refers to a single judgment score calculated by integrating the elements supporting transition and the elements inhibiting transition.

[0068] The transition threshold score refers to the criterion value used to determine whether a single transition score allows for language transition. A value indicating transition allowance refers to a language transition control value that swaps the current input language with the current output language. A value indicating transition deferral refers to a language transition control value that maintains the current input language, current output language, and current speaker.

[0069] The input alignment layer refers to a layer of the unintended behavior judgment model that aligns input tokens included in the integrated input sequence within a common time interval based on the point in time when a language switching candidate action occurs. The common time interval refers to a time interval established to analyze input tokens before, during, and after a rotation action within the same time range. The sensor feature normalization layer refers to a layer that normalizes input tokens corresponding to relative angle change, angular velocity, change in angular velocity, change in acceleration, rotation duration, and rotation direction based on a first reference direction angle.

[0070] Sensor feature tokens refer to normalized tokens representing the terminal's rotation magnitude, rotation speed, change in rotation speed, change in acceleration, rotation duration, and rotation direction. The utterance context encoding layer refers to a layer that generates tokens regarding the utterance state based on input tokens corresponding to silence time and utterance duration state values. Utterance context tokens refer to tokens representing the current speaker's utterance continuation state, the state immediately after utterance termination, the silence maintenance state, or a state where it is unclear whether utterance has occurred.

[0071] The input language state token refers to a token representing the current input language state as model input. The output language state token refers to a token representing the current output language state as model input. The speaker state token refers to a token representing whether the current speaker is set as User 1 or User 2 as model input.

[0072] History tokens refer to tokens that represent information as model input regarding direction transition history, the occurrence of recent transition candidates, the elapsed state since the previous transition, or the stability of the language state. Embedding vectors refer to representations obtained by converting input tokens into numerical vectors of the same dimension so that the unintended behavior judgment model can compute them. Relative time information refers to time information indicating at which point in time each input token corresponds relative to the time when the language transition candidate behavior occurred.

[0073] A multi-head attention layer refers to a layer that analyzes the interrelationships between input elements in parallel by applying multiple attention heads to the embedding vectors of an integrated input sequence. An attention head refers to a sub-processing unit of the multi-head attention layer that generates a context vector by focusing on specific decision elements among the embedding vectors. A direction matching head refers to an attention head that analyzes the relationship between the terminal's rotation direction and the second user's direction by referencing embedding vectors corresponding to relative angle changes and rotation directions.

[0074] The direction matching context vector is a context vector generated by the direction matching head, containing information regarding the degree to which the rotation direction of the terminal aligns with the direction of the second user. The rotation pattern head refers to an attention head that analyzes the rotation pattern of the terminal by referencing an embedding vector corresponding to angular velocity, the amount of change in angular velocity, and the rotation duration. The rotation pattern context vector is a context vector generated by the rotation pattern head, containing information regarding the relationship between the change in rotation speed of the terminal and the rotation duration.

[0075] The shake detection head refers to an attention head that distinguishes whether the terminal's movement is rotational or shaking by referencing embedding vectors corresponding to changes in acceleration and angular velocity. The shake context vector is a context vector generated by the shake detection head, containing information indicating whether the terminal's movement is close to shaking, a change in grip, or an impact-induced movement. The speech status head refers to an attention head that analyzes whether the terminal's movement occurred during speech or after the end of speech by referencing embedding vectors corresponding to silence time and speech duration status values.

[0076] The utterance state context vector is a context vector generated by the utterance state head, containing information indicating the relationship between rotation actions and the current utterance state of the speaker. The hysteresis stability head refers to an attention head that analyzes the instability of repetitive rotation candidate actions or language states by referencing the rotation history and the embedding vector corresponding to the current speaker. The hysteresis context vector is a context vector generated by the hysteresis stability head, containing information regarding the recent rotation history, the current speaker state, and the stability of the language state.

[0077] The context fusion layer refers to a layer that combines the direction matching context vector, rotation pattern context vector, wavering context vector, utterance state context vector, and history context vector. The integrated context vector refers to a comprehensive contextual representation generated by the context fusion layer by combining multiple context vectors and applying normalization or non-linear transformation processing. Direction range information refers to information regarding the allowable direction range or angle range corresponding to the direction of the second user.

[0078] The direction alignment score calculation layer refers to a layer that calculates a direction alignment score based on information regarding the change in relative angle, rotation direction, and direction range. The rotation pattern score calculation layer refers to a layer that calculates a transition intention score based on angular velocity, rotation duration, and the change in angular velocity. The sway score calculation layer refers to a layer that calculates an unintended behavior score based on the change in acceleration, the change in angular velocity, and rotation duration.

[0079] The utterance alternation score calculation layer refers to a layer that calculates an utterance alternation suitability score based on time information corresponding to the silent time, the utterance sustainment state value, and the timing of the occurrence of rotational motion.

[0081] FIG. 2 is a block diagram illustrating the hardware configuration of a real-time conversation interpretation system according to one embodiment.

[0082] According to one embodiment, the terminal (110) may include an input unit (201), a microphone (202), a gyroscope sensor (203), an accelerometer sensor (204), a voice output unit (205), a memory (206), and one or more processors (207). The terminal (110) may be an electronic device for performing real-time conversation interpretation between a first user and a second user, and each component may be interconnected for the collection of voice data, the collection of sensor data, the calculation of language switching control values, the updating of language status, and the output of translated voice.

[0083] The input unit (201) may be configured to receive the first language of the first user and the second language of the second user. The input unit (201) may be implemented as a touch panel, a physical button, a voice input interface, a settings screen, a gesture input unit, or a combination thereof, and allows the user to specify language information that serves as the standard for the current input language and the current output language before starting a conversation.

[0084] The input unit (201) can receive not only language information but also setting values ​​such as a conversation start command, interpretation mode selection, selection of on-device processing or server-linked processing, voice output method, speaker setting, return allowance angle or transition sensitivity. In some embodiments, the input unit (201) may provide language settings by calling up previous usage history or saved profiles without direct user input.

[0085] Information input through the input unit (201) can be stored in memory (206) or transmitted to one or more processors (207). The processor (207) can set the first language provided from the input unit (201) as the current input language, set the second language as the current output language, and set the current speaker as the first user. Accordingly, subsequent speech recognition, translation, and language switching judgments can be performed based on consistent criteria.

[0086] The microphone (202) may be configured to receive speech from a first user or a second user and generate voice data. The microphone (202) may be implemented as a single microphone, and in some embodiments, may be implemented as a microphone array including a plurality of microphones to provide information regarding the direction of speech, ambient noise, voice intensity, or difference in voice arrival time.

[0087] The microphone (202) can continuously receive the voice of the current speaker during the process of performing real-time interpretation. The voice data received from the microphone (202) can be aligned with sensor data with time information assigned to it, and the processor (207) can calculate a speech continuation state value indicating the silence time and whether the current speaker is continuing to speak based on the voice data.

[0088] Voice data generated from the microphone (202) can be used as input for voice recognition processing. A processor (207) or a processor of a backend server can convert the voice data into text based on the current input language, and the converted text can be converted into translated text corresponding to the current output language. In this way, the microphone (202) can function as a starting point for a real-time interpretation pipeline.

[0089] The gyroscope sensor (203) may be a sensor for detecting rotational movement of the terminal (110). The gyroscope sensor (203) may provide sensor data for calculating the direction angle, rotation direction, angular velocity, or amount of change of angular velocity of the terminal (110), and such sensor data may be used to determine whether the direction change of the terminal (110) is a candidate action for language change.

[0090] The gyroscope sensor (203) can provide basic data for calculating the difference between a first reference direction angle corresponding to the direction of the first user and a current direction angle indicating the current direction of the terminal (110). The processor (207) can calculate the relative angle change amount, rotation duration, and rotation direction using the output of the gyroscope sensor (203), and can reflect this in the calculation of the language switching control value.

[0091] The output of the gyroscope sensor (203) is not limited to determining whether the terminal (110) is tilted, but can also be used to determine whether a rotational motion has occurred from the first user toward the second user. In some embodiments, the sensor data of the gyroscope sensor (203) may be combined with the sensor data of the accelerometer sensor (204) to distinguish between intentional direction change and unintentional shaking.

[0092] The acceleration sensor (204) may be configured to generate sensor data regarding linear acceleration, gravity direction, attitude change, and shaking of the terminal (110). The acceleration sensor (204) can detect the amount of change in acceleration that occurs during the process of the terminal (110) being rotated by the user's hand, and the amount of change can be used to determine the nature of the language switching candidate action.

[0093] Sensor data from the accelerometer (204) can be used together with sensor data from the gyroscope (203) to help determine whether the movement of the terminal (110) is an intentional rotational motion or is closer to hand shaking or a change in grip. For example, even if there is a change in angle, if the change in acceleration is irregular or repeatedly reverses within a short period of time, the processor (207) can determine that the movement is closer to an unintentional action.

[0094] The acceleration sensor (204) can provide information to distinguish movements such as changing the position of holding the terminal (110) during speech, shaking while moving, impact movements, or temporary changes in posture. This information can be used as input to an unintended behavior judgment model and can be reflected as a factor that inhibits conversion in the process of calculating an unintended behavior score or a language conversion control value.

[0095] The voice output unit (205) may be configured to output a synthesized voice corresponding to the translated text. The voice output unit (205) may be implemented as a speaker built into the terminal (110), and in some embodiments, it may be implemented as an earphone output unit, a Bluetooth audio transmitter, an external speaker connection unit, or a combination thereof.

[0096] The voice output unit (205) can deliver a voice synthesis result corresponding to the current output language to the first user or the second user. For example, if the current input language is the first language and the current output language is the second language, the utterance of the first user can be output as synthesized voice in the second language. Conversely, after language switching, the utterance of the second user can be output as synthesized voice in the first language.

[0097] The voice output unit (205) can contribute to maintaining the flow of conversation by providing the interpretation result audibly without displaying it on a screen. In some embodiments, the voice output unit (205) may be controlled to adjust the output volume, output direction, tone, or output device according to the current speaker, output language, whether interpretation is complete, or language switching status.

[0098] The memory (206) may be configured to store data and commands necessary for the operation of the terminal (110). The memory (206) may store a first language, a second language, a current input language, a current output language, a current speaker, a first reference direction angle, a direction change history, a language change control value, voice data, and sensor data.

[0099] The memory (206) may store an unintended behavior judgment model, a speech recognition model, a translation model, a speech synthesis model, or some parameters thereof. In some embodiments, the entire model may be stored in the memory (206) and executed on the terminal (110), and in other embodiments, model identification information, API call information, or cache data for interacting with a backend server may be stored.

[0100] The memory (206) can also store intermediate data generated during the real-time processing. For example, voice data and sensor data arranged in chronological order, speech duration status values, relative angle change amounts, angular velocity, acceleration change amounts, integrated input sequences, direction matching scores, switching intention scores, unintended behavior scores, and speech alternation suitability scores can be temporarily stored in the memory (206).

[0101] One or more processors (207) may be configured to control each component of the terminal (110) and perform at least some operations of the real-time conversation interpretation method. The processors (207) may process voice data and sensor data in conjunction with the input unit (201), microphone (202), gyroscope sensor (203), accelerometer sensor (204), voice output unit (205), and memory (206).

[0102] The processor (207) can set the first language as the current input language and the second language as the current output language, and can sort voice data and sensor data in chronological order. Additionally, the processor (207) can calculate the silence time, speech duration state value, relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration time, and rotation direction, and determine whether the movement of the terminal (110) corresponds to a language switching candidate action.

[0103] When a language switching candidate action is identified, the processor (207) can execute an unintended action judgment model or request model inference from a backend server to calculate a language switching control value. When the language switching control value satisfies a reference condition, the processor (207) can swap the current input language with the current output language, swap the current speaker, and perform or control speech recognition, translation, and speech synthesis output according to the current input language.

[0105] FIG. 3 is a flowchart exemplifying the overall operation flow of a real-time conversation interpretation method according to one embodiment.

[0106] According to one embodiment, in operation (S301), the processor may set the first language of the first user as the current input language, set the second language of the second user as the current output language, and set the current speaker as the first user.

[0107] At this time, the processor can identify a first language and a second language based on language setting information received through the input unit, user selection information, or a user profile stored in advance. The first language can be set as the standard language for speech recognition at the start of a conversation, and the second language can be set as the standard language for outputting translated text and synthesized speech.

[0108] Additionally, the processor can set the current speaker as the first user so that subsequently received voice data is processed preferentially as corresponding to the first user's utterance. This setting can be used as an initial state to determine whether to exchange the current input language and the current output language in a subsequent operation, and can also be utilized for managing the speaker state before and after language switching.

[0109] According to one embodiment, in operation (S302), the processor may assign time information based on the same time standard to voice data received from a microphone and sensor data regarding the movement of a terminal received from a gyroscope sensor and an accelerometer sensor, or may arrange the voice data and the sensor data in chronological order by referring to the time information included in the voice data and the sensor data.

[0110] In this case, the voice data may include information regarding the speech signal received through the microphone, silent periods, voice intensity, the start time of speech, and the end time of speech. The sensor data may include the direction angle, angular velocity, and rotational direction obtained from the gyroscope sensor, and information on the amount of acceleration change or attitude change obtained from the accelerometer sensor.

[0111] The processor can align the occurrence order of the two data by applying the same time standard even when the sampling periods of the voice data and the sensor data are different. Accordingly, basic data can be generated to determine whether the rotational movement of the terminal occurred during the current speech, during a silent period after the speech ended, or as shaking unrelated to the speech state.

[0112] According to one embodiment, in operation (S303), the processor can calculate a speech continuation state value based on the voice data, indicating a silent time and whether the current speaker is continuing to speak.

[0113] The processor can distinguish between speech and silence intervals by utilizing the energy and frequency components of voice data, voice activity detection results, or intermediate voice recognition results. Silence time can be calculated as the length of the interval during which no speech is detected, and a silence interval lasting for a certain period after the end of a speech can be used to determine the likelihood of a change in the conversation partner.

[0114] The speech duration status value can indicate whether the speaker is currently continuing to speak, has ended the speech, is maintaining silence, or whether the speech status is unclear. This value can be used to determine whether the movement of the terminal is due to an actual intention to switch language or is an unintended movement resulting from a change in the terminal's grip during speech.

[0115] According to one embodiment, in operation (S304), the processor can determine whether the movement of the terminal corresponds to a language switching candidate operation based on a relative angle change amount representing the difference between a first reference direction angle representing the current direction of the terminal and a first reference direction angle corresponding to the direction of the first user, an angular velocity representing the rotational speed of the terminal, a change amount of the angular velocity, a change amount of the acceleration of the terminal, a rotational duration of the terminal, a rotational direction of the terminal, a silent time, and a speech duration state value.

[0116] The first reference direction angle may be a reference value corresponding to the direction of the first user, and the first current direction angle may be a value indicating the direction the terminal is facing at a specific point in time. The processor can determine whether the terminal is deviating from the direction of the first user and rotating toward the direction of the second user by using the difference between the first reference direction angle and the first current direction angle.

[0117] In addition, the processor can consider not only the relative angle change but also angular velocity, the amount of change in angular velocity, the amount of change in acceleration, the duration of rotation, and the direction of rotation. By combining this with the silence time and the speech duration state value, it is possible to distinguish whether the movement of the terminal is a change of direction accompanying speech alternation or a movement caused by hand tremor, shaking while moving, or a change in grip during speech.

[0118] According to one embodiment, in operation (S305), if the movement of the terminal corresponds to the language switching candidate operation, the processor can calculate a language switching control value indicating whether to exchange the current input language and the current output language based on the relative angle change amount, the angular velocity, the acceleration change amount, the rotation duration, the rotation direction, the silence time, the speech duration state value, the current input language, the current output language, the current speaker, the direction switching history, and the difference between the first reference direction angle and the first current direction angle.

[0119] According to one embodiment, the processor can determine whether the movement of the terminal corresponds to a rotational motion from the direction of the first user toward the direction of the second user, based on the relative angle change amount, angular velocity, the change amount of angular velocity, the change amount of acceleration, the rotation duration, and the rotation direction.

[0120] At this time, the relative angle change amount may represent the difference between a first reference direction angle corresponding to the direction of the first user and a first current direction angle representing the current direction of the terminal. The processor may determine together whether the relative angle change amount falls within a set candidate angle range, whether the rotation direction is directed toward the direction of the second user, and whether the angular velocity falls within a range that can be distinguished from a simple change in the terminal's attitude.

[0121] In addition, the processor can distinguish whether the movement of the terminal is a continuous rotational motion, a shaking that occurs for a short period of time, or an impact-type movement during motion by utilizing the change in angular velocity, the change in acceleration, and the rotation duration. Accordingly, even if there is a change in the orientation of the terminal, it may not be processed as a language conversion candidate if the rotation duration falls outside the standard range or if the direction of rotation is unclear.

[0122] According to one embodiment, when the processor determines that the movement of the terminal corresponds to a rotational movement, it can determine whether the rotational movement is a movement that occurred while the current speaker's speech was continuing, based on the silent time and speech duration state values.

[0123] The silence time may indicate the length of the interval during which the current speaker's voice is not detected, and the speech duration state value may indicate whether the current speaker is continuing to speak. The processor may determine whether the rotation of the terminal occurred within a speech alternation interval by comparing the time at which the rotation action occurred with the start time of the silence interval or the end time of the speech.

[0124] For example, if a rotational movement occurs during a period when the current speaker's utterance is continuing, the movement may be determined to be closer to a change in terminal grip, wrist movement, or a change in posture during speech rather than an intention to change the interpretation direction. Conversely, if a rotational movement occurs during a silent period formed after the utterance has ended, the processor may determine the movement as an action to point the terminal toward the conversation partner.

[0125] According to one embodiment, if the rotational motion is determined to be a movement that occurred while the current speaker's utterance is continuing, the processor classifies the rotational motion as a candidate for an unintended action, and if the rotational motion is determined to be a movement that occurred after the current speaker's utterance has ended, the processor classifies the rotational motion as a candidate for a language switching intention.

[0126] Candidates for unintended behavior may refer to movements that are not suitable for immediately exchanging the current input language and the current output language, even if there is a change in the orientation of the terminal. For example, if the current speaker rotates the terminal slightly while continuing to speak, adjusts the position of the terminal held in the hand, or shakes the terminal while moving, it may be classified as a candidate for unintended behavior.

[0127] A language switching intention candidate can correspond to a motion in which the terminal rotates toward the direction of the second user after the current speaker's utterance has ended. By reflecting the results of this candidate classification as input to a subsequent unintended behavior judgment model or in the process of calculating language switching control values, the processor can perform a direction switching judgment combined with the utterance context, rather than a simple change in angle.

[0128] According to one embodiment, the processor can convert a relative angle change amount, angular velocity, acceleration change amount, rotation duration, rotation direction, silence time, speech duration state value, current input language, current output language, current speaker, direction change history, and the difference between a first reference direction angle and a first current direction angle into input tokens, and generate an integrated input sequence by arranging the input tokens in chronological order.

[0129] Input tokens may be unit data converted from sensor features, utterance states, language states, and history states that the model can process. For example, relative angle change amounts and rotation directions can be converted into direction-related input tokens, angular velocity and rotation duration can be converted into rotation pattern-related input tokens, and silence time and utterance duration values ​​can be converted into utterance context-related input tokens.

[0130] The processor can add time information to input tokens to indicate the sequence in which rotation movements, silence periods, speech sustain states, and direction change histories occurred. The integrated input sequence can be provided to a subsequent unintended behavior determination model, and based on the sequence, the model can determine whether the terminal's movement is closer to a language switching intention or an unintended behavior.

[0131] According to one embodiment, the processor can analyze the interrelationships between input tokens included in an integrated input sequence through an unintended behavior judgment model.

[0132] The processor can align input tokens corresponding to the relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration, rotation direction, silence time, utterance duration state value, current input language, current output language, current speaker, and direction change history included in the integrated input sequence within a common time interval based on the time when the language switching candidate action occurred, through the input alignment layer of the unintended behavior judgment model.

[0133] In this case, the input alignment layer can rearrange sensor-related input tokens and voice-related input tokens generated at different points in time based on a single time standard. For example, the speech duration state prior to the point where rotation occurs, the rotation start time, the rotation end time, and the occurrence time of the silent period can be placed on the same time axis centered on the point where rotation occurs.

[0134] The input alignment layer can align input tokens through interpolation, interval representative value calculation, or time stamp matching, even when the sampling cycles of the gyroscope and accelerometer sensors differ from the processing cycles of the voice data. Accordingly, the unintended behavior detection model can analyze whether the terminal's movement occurred during speech, after the end of speech, or is temporally adjacent to the history of direction changes.

[0135] Additionally, the input alignment layer can select input tokens within a common time interval set based on the point in time when a language switching candidate action occurs. The common time interval may include the interval immediately before the rotation action, the interval during the rotation action, and the interval immediately after the rotation action, and the tokens within this interval can be used as basic data for analyzing temporal relationships and intercorrelations in subsequent layers.

[0136] The processor can obtain sensor feature tokens representing the rotation magnitude, rotation speed, change in rotation speed, change in acceleration, rotation duration, and rotation direction through the sensor feature normalization layer of the unintended behavior judgment model based on a first reference direction angle, and acquire sensor feature tokens representing the rotation magnitude, rotation speed, change in rotation speed, change in acceleration, rotation duration, and rotation direction of the terminal.

[0137] The sensor feature normalization layer can convert the current orientation and rotation direction of the terminal into relative coordinates based on a first reference direction angle corresponding to the direction of the first user. Accordingly, even if the gripping posture, initial tilt, and gripping habits of each user differ, the model can make a determination based on the change pattern from the direction of the first user to the direction of the second user, rather than absolute angle values.

[0138] The sensor feature normalization layer processes the relative angle change and angular velocity together to generate input that can determine whether the terminal has actually rotated in a specific direction. For example, if the relative angle change is large but the angular velocity or rotation duration does not match the reference range, the movement may be described as an input pattern closer to momentary shaking or a change in attitude rather than intentional rotation.

[0139] Additionally, the sensor feature normalization layer can generate feature tokens that distinguish whether the terminal's movement is a smooth rotation, irregular shaking, or impulsive motion based on changes in acceleration and angular velocity. These sensor feature tokens can be used by subsequent attention heads to perform orientation matching, rotation pattern detection, and shaking discrimination.

[0140] The processor can generate utterance context tokens representing at least one of a state in which the current speaker's utterance continues, a state immediately after the current speaker's utterance ends, a state in which silence is maintained, and a state in which it is unclear whether utterance is occurring, based on an input token corresponding to a silence time and an input token corresponding to an utterance duration state value, through the utterance context encoding layer of an unintended behavior judgment model.

[0141] The utterance context encoding layer can represent at what point in the conversation flow a rotational movement occurred by utilizing the silence time and utterance duration state value calculated from the voice data. For example, if the terminal moves during a section where the current speaker's voice is continuously detected, that movement can be represented by an utterance context token that may correspond to a change in grip or wrist movement during speech.

[0142] Conversely, if the terminal is rotated toward the second user after the current speaker's utterance has ended and a period of silence has been formed, the utterance context encoding layer can represent this situation as a state where utterance switching is possible. Such an utterance context token can support subsequent models in interpreting the terminal's rotation motion not as a simple sensor event, but as an action combined with the conversation sequence.

[0143] In addition, the utterance context encoding layer can separately represent states where it is unclear whether an utterance is occurring. For example, if it is difficult to clearly determine whether an utterance is continuing due to ambient noise, overlapping utterances, or reduced reliability of speech recognition, the state can be tokenized as an unclear state to suppress excessive switching in subsequent language switching decisions.

[0144] The processor can convert sensor feature tokens, utterance context tokens, an input language state token corresponding to the current input language, an output language state token corresponding to the current output language, a speaker state token corresponding to the current speaker, and a history token corresponding to the direction change history into embedding vectors of the same dimension through the token embedding layer of the unintended behavior judgment model, and add relative time information based on the time of occurrence of the language switching candidate action to the said embedding vectors.

[0145] The token embedding layer can convert input tokens of different formats into a single neural network processing space. Sensor feature tokens correspond to numerical motion information, utterance context tokens correspond to speech context information, and language state tokens and speaker state tokens can represent state information of the current interpretation direction.

[0146] In this context, converting into embedding vectors of the same dimension implies that each input token is transformed into a common representation format that can be compared and referenced by subsequent multi-head attention layers. Accordingly, the model can analyze the relationships between relative angle changes, utterance duration values, angular velocity, silence time, and the current speaker and direction change history within a single representation space.

[0147] Relative time information can indicate at what point in time each input token corresponds to the point in time when a language switching candidate action occurs. For example, by expressing whether a rotation occurred immediately after the end of an utterance, whether the silence time increased after the rotation, or whether a rotation candidate occurred again immediately after the previous direction change, the model's temporal judgment ability can be enhanced.

[0148] The processor can apply multiple attention heads to embedding vectors with relative time information added through the multi-head attention layer of the unintended behavior judgment model.

[0149] A multi-head attention layer can be a configuration for parallel analysis of a single input sequence from multiple perspectives. Even if each attention head receives the same embedding vectors, it can focus on specific decision elements through different weight structures; accordingly, it can distinguish and analyze orientation matching, rotation patterns, wavering, utterance state, and hysteresis stability.

[0150] For example, the relative angle change and the direction of rotation may be important to determine whether the terminal's rotation direction aligns with the second user's direction. On the other hand, to determine whether the rotation motion is a change in grip during speech, the silence time and speech sustainment state value may be more important. A multi-head attention layer can reflect these differences in the importance of input elements on a head-by-head basis.

[0151] Furthermore, the multi-head attention layer can handle complex situations that are difficult to judge based on a single sensor value alone. For example, if the change in angle supports speech switching but the utterance duration value suggests movement during speech and the change in acceleration suggests shaking, multiple attention heads can separate and analyze the relationships between each signal, and then synthesize them in a subsequent context fusion layer.

[0152] The processor can generate a direction matching context vector representing the relationship between the rotation direction of the terminal and the direction of the second user by referencing embedding vectors corresponding to the relative angle change amount and the rotation direction through a direction matching head among a plurality of attention heads.

[0153] The orientation matching head may focus on determining whether the rotation of the terminal is a change from the direction of the first user toward the direction of the second user. To this end, the orientation matching head may refer to the relative angle change amount, the direction of rotation, the direction at the point of rotation termination, and direction range information corresponding to the direction of the second user.

[0154] The orientation matching context vector may be a contextual representation indicating whether the terminal has rotated toward the direction where the conversation partner is located, rather than simply shaking or moving in a random direction. For example, if the relative angle change is sufficient and the direction of rotation converges to the direction range of the second user, the orientation matching context vector may include orientation information that supports language switching.

[0155] Conversely, if the terminal's current orientation deviates from the second user's orientation range, the rotation direction is unclear, or the orientation at the end of the rotation changes again within a short period of time, the orientation matching context vector may include information that weakens the transition judgment. Accordingly, the orientation matching head can reflect alignment with the conversation partner's direction rather than a simple change in angle.

[0156] The processor can generate a rotation pattern context vector representing the relationship between the change in rotation speed and the rotation duration of the terminal by referencing embedding vectors corresponding to the angular velocity, the amount of change in angular velocity, and the rotation duration through the rotation pattern head among a plurality of attention heads.

[0157] The rotation pattern head can analyze temporal movement patterns to determine whether the movement of the terminal is an intentional rotation by the user or a momentary shake. Intentional rotation proceeds continuously over a certain period, and changes in angular velocity may appear relatively consistently at the start and end points of the rotation.

[0158] The rotation pattern context vector can reflect not only the angular velocity itself but also the amount of change in angular velocity and the duration of rotation. For example, if the angular velocity increases rapidly only temporarily and the rotation duration is short, it can be represented as a pattern similar to shaking or grip changes, whereas if the rotation is maintained in a constant direction, it can be composed of a representation similar to an intentional rotation pattern.

[0159] Additionally, the rotation pattern head can reinforce the possibility of language switching from a different perspective than the direction matching head. Since it may be appropriate to withhold language switching if the rotation pattern is irregular or very short, even if the direction is similar to that of the second user, the rotation pattern context vector can be used as an important correction factor in the subsequent calculation of single switching scores.

[0160] The processor can generate a shake context vector to distinguish whether the movement of the terminal is a rotational motion or a shake motion by referencing embedding vectors corresponding to changes in acceleration and angular velocity through a shake detection head among a plurality of attention heads.

[0161] The shake detection head can analyze whether the device's movement is a controlled rotation or irregular shaking by referencing changes in both the gyroscope and accelerometer sensors. For example, if the amount of change in angular velocity repeatedly reverses over a short period of time and the amount of change in acceleration also increases irregularly, the movement can be described as being closer to shaking motion than an intentional change of direction.

[0162] Shaking context vectors may be contextual representations reflecting unintended behaviors such as hand tremors, impacts while moving, changes in device grip, or temporary shaking within a pocket or hand. These vectors can be used to increase unintended behavior scores in subsequent score calculations or to cause language switching controls to indicate a switching hold.

[0163] In addition, the shake detection head can perform the role of preventing the system from immediately interpreting changes in the terminal's orientation as an intention to switch languages. Accordingly, compared to a method that utilizes only changes in the terminal's angle, the system can distinguish more finely between the user's actual intention to switch interpretation directions and simple shaking.

[0164] The processor can generate a speech state context vector to distinguish whether the movement of the terminal occurred during the current speaker's speech or after the end of the speech by referencing embedding vectors corresponding to the silence time and speech duration state values ​​through the speech state head among a plurality of attention heads.

[0165] The speech state head can analyze at what point in the conversation flow the rotational movement occurred. If the terminal moves while the current speaker is still speaking, the movement may correspond to a change in posture or grip adjustment during speech, rather than an action to hand the terminal to the other party.

[0166] The speech state context vector can reflect whether a silence period has been formed, immediately after the end of a speech, or during the continuation of speech. For example, if a certain silence period exists after the end of a speech and the terminal is rotated toward the second user immediately thereafter, the speech state context vector may include contextual information that increases the possibility of speech alternation.

[0167] Conversely, if the utterance continuation state value indicates the current speaker's utterance continuation state, the utterance state context vector can be used as contextual information to suppress language switching. This allows the system to reduce unnecessary exchange between the current input language and the current output language even when the first user slightly moves the terminal while speaking.

[0168] The processor can generate a history context vector indicating recent iterative direction change candidate actions or instability of the language state by referencing an embedding vector corresponding to a direction change history and an embedding vector corresponding to the current speaker through a history stability head among a plurality of attention heads.

[0169] The hysteresis stability head can analyze the relationship between the current candidate action and the previous redirection history. For example, if multiple redirection candidate actions occur within a short period, or if a redirection candidate arises immediately after the current speaker changes, such situations may indicate a potential for language state instability.

[0170] Historical context vectors can reflect the number of recent transitions, the elapsed time since the previous transition, the current speaker state, and patterns of change in language state. This information can be reflected as a directional transition history risk value in the single transition score calculation process and can contribute to reducing repetitive mistransitions or state tangles.

[0171] In addition, the hysteresis stability head can provide auxiliary criteria for determining whether a language switching candidate behavior is a natural transition in the actual conversation flow or an unnecessary re-switch after a transition that has already occurred. Accordingly, the model can stably calculate language switching control values ​​by considering not only current sensor values ​​but also the state history accumulated over time.

[0172] The processor can generate an integrated context vector by combining the direction matching context vector, rotation pattern context vector, sway context vector, utterance state context vector, and history context vector through the context fusion layer of the unintended behavior judgment model, and applying normalization processing and non-linear transformation processing to the combined context vector.

[0173] The context fusion layer can combine context vectors generated from multiple attention heads into a single integrated representation. The direction matching context vector indicates the validity of the transition direction, the rotation pattern context vector indicates the possibility of intentional rotation, and the sway context vector indicates the possibility of unintentional movement.

[0174] Additionally, the utterance state context vector indicates whether a rotational movement occurred at the time of the utterance change, and the history context vector can indicate stability or risk based on recent direction change history. The context fusion layer can combine these context vectors to generate a comprehensive representation for determining whether the terminal's movement is close to the language switch intention or a situation where the switch should be deferred.

[0175] Normalization and non-linear transformation processing can be used to reliably combine context vectors of different sizes and distributions. The integrated context vector can be used in subsequent scoring layers to calculate direction matching scores, transition intention scores, unintentional behavior scores, and utterance alternation suitability scores, and can ultimately serve as the basis for calculating language transition control values.

[0176] According to one embodiment, the processor can calculate the degree to which the rotation direction of the terminal aligns with the direction of the second user based on the relative angle change amount and rotation direction through an unintended behavior judgment model, calculate the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern based on angular velocity and rotation duration, calculate the degree to which the movement of the terminal corresponds to shaking based on the acceleration change amount, and calculate the degree to which the rotation operation corresponds to the timing of speech alternation based on silent time and speech duration state value.

[0177] The direction matching score calculation layer of the unintended behavior judgment model can calculate a direction matching score indicating the degree to which the direction at the time of the terminal's rotation termination is included in the direction range corresponding to the second user's direction, based on an embedding vector corresponding to the relative angle change amount, an embedding vector corresponding to the rotation direction, and direction range information corresponding to the second user's direction.

[0178] The orientation matching score calculation layer may be a layer for determining whether the rotation of the terminal deviates from the direction of the first user and faces the direction of the second user. This layer can check how much the terminal has rotated from the reference direction through the relative angle change amount and analyze whether the rotation faces the direction of the second user through the direction of rotation.

[0179] Additionally, the direction matching score calculation layer can determine whether the terminal's direction at the time of rotation termination falls within the range actually facing the second user by referring to direction range information corresponding to the second user's direction. Accordingly, it is possible to distinguish between cases where the terminal simply moves in an arbitrary direction and cases where it is intentionally oriented toward the second user to deliver the interpretation result.

[0180] The orientation match score can be used as a supporting factor for the transition in the subsequent calculation of a single transition score. For example, the closer the terminal's rotation end direction is to the second user's orientation range, the more the orientation match score can be reflected as a value supporting the transition; conversely, the further it deviates from the orientation range, the more the language transition control value can be reflected in a direction that indicates a transition hold.

[0181] The rotation pattern score calculation layer of the unintended behavior judgment model can calculate a conversion intention score that indicates the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern, based on an embedding vector corresponding to angular velocity, an embedding vector corresponding to rotation duration, and an embedding vector corresponding to the amount of change of angular velocity.

[0182] The rotation pattern score calculation layer may be a layer for determining whether the movement of the terminal is an action in which the user intentionally rotated the terminal toward the direction of a second user. This layer analyzes not only the magnitude of the angular velocity but also the amount of change in angular velocity and the duration of the rotation together to verify whether the rotational motion occurred continuously for a certain period of time.

[0183] For example, an intentional rotation pattern may manifest as an increase in angular velocity at the start of the rotation, the maintenance of a constant directionality in the middle, and a decrease in angular velocity at the end. Conversely, if the rotation duration is excessively short or changes in angular velocity repeat irregularly, the movement may be evaluated as being closer to shaking or a change in grip rather than intentional rotation.

[0184] The switching intention score can indicate the degree to which the terminal's movement is close to a rotational motion for changing the interpretation direction. Therefore, if the switching intention score is calculated to be high, it can support the decision to exchange the current input language with the current output language, and if the switching intention score is calculated to be low, it can contribute to the decision to postpone language switching.

[0185] The shake score calculation layer of the unintended behavior judgment model can calculate an unintended behavior score indicating the degree to which the movement of the terminal corresponds to shake, based on an embedding vector corresponding to the change in acceleration, an embedding vector corresponding to the change in angular velocity, and an embedding vector corresponding to the rotation duration.

[0186] The shake score calculation layer may be a layer designed to determine whether the device's movement is closer to hand tremor, shaking while moving, impact-induced movement, or a change in device grip, rather than an intentional rotational motion. This layer can analyze whether the changes in acceleration and angular velocity increase irregularly or repeatedly reverse within a short period of time.

[0187] For example, when the terminal rotates stably toward the direction of the second user, the direction of rotation and the flow of changes in acceleration may be relatively consistent. Conversely, when the terminal moves while the user is walking or adjusting their grip, the amount of change in acceleration changes abruptly, and the amount of change in angular velocity may also appear inconsistent with the direction of rotation.

[0188] The unintended behavior score can be reflected as a factor that inhibits conversion in the subsequent single conversion score calculation. Accordingly, even if a change in the terminal's angle is present, if the shake score calculation layer determines that the movement is close to unintended behavior, there is a high probability that the language conversion control value will be calculated as a value indicating conversion suspension.

[0189] The utterance alternation score calculation layer of the unintended behavior judgment model can calculate an utterance alternation suitability score that indicates the degree to which a rotational motion corresponds to the utterance alternation point, based on an embedding vector corresponding to the silence time, an embedding vector corresponding to the utterance duration state value, and time information corresponding to the time of occurrence of the rotational motion.

[0190] The utterance alternation score calculation layer may be a layer for determining whether the terminal's rotation action occurred at the point where the utterance turn passes from the current speaker to the counterpart in the conversation flow. This layer can analyze whether a silence period has been formed, whether the current speaker is continuing to speak, and whether the rotation action is temporally adjacent to the end of the utterance.

[0191] For example, if a certain period of silence occurs after the current speaker finishes speaking, and the terminal is rotated toward the second user immediately thereafter, the speech alternation suitability score may increase. Conversely, if the terminal moves while the current speaker continues to speak, such movement may be evaluated as being closer to a change in grip or posture adjustment during speech rather than speech alternation.

[0192] The speech alternation suitability score can be used as a supporting factor for language switching, along with the direction matching score and the switching intention score. Accordingly, even if the terminal's rotation direction and rotation pattern support language switching, if they do not match the timing of speech alternation, the final language switching control value may be adjusted to indicate a switching hold.

[0193] The processor can calculate a single transition score by reflecting the transition intention score, direction matching score, and utterance alternation suitability score as factors supporting the transition, and reflecting the unintended behavior score, utterance duration state value, acceleration change amount, and direction change history as factors inhibiting the transition.

[0194] A single transition score may be a value that integrates multiple scores and status values ​​into a single judgment criterion. The transition intention score indicates whether the terminal's rotation pattern is intentional, the direction matching score indicates whether the rotation direction aligns with the second user's direction, and the utterance alternation suitability score indicates whether the rotation action is appropriate for the transition point in the conversation sequence.

[0195] Conversely, unintentional behavior scores, utterance duration values, acceleration changes, and direction change history can be reflected as factors that inhibit language switching. For example, if the current speaker is continuing to speak, if the acceleration change indicates a wavering pattern, or if there are recent repetitive candidate movements for direction change, the single switch score may be processed to be lowered.

[0196] In this way, a single switching score can collectively reflect rotational suitability on the sensor side, alternation suitability on the utterance side, the possibility of unintended behavior, and hysteresis stability. Therefore, the system can calculate a language switching control value by synthesizing various judgment factors, rather than simply exchanging the current input language and the current output language based solely on the amount of angle change.

[0197] The processor may calculate a language switching control value as a value indicating that switching is allowed when the single switching score is greater than or equal to the switching threshold score, and may calculate a language switching control value as a value indicating that switching is suspended when the single switching score is less than the switching threshold score.

[0198] The switching threshold score may be a reference value for calculating the language switching control value as an acceptable switching value. If a single switching score is greater than or equal to the switching threshold score, the processor may determine that the movement of the terminal is an intentional rotational motion toward the direction of the second user and is also appropriate at the time of speech alternation.

[0199] In this case, the language switching control value can be calculated as a value that swaps the current input language with the current output language. Accordingly, the current input language can be changed to a second language, the current output language can be changed to a first language, and the current speaker can be changed to a second user.

[0200] Conversely, if a single transition score is less than the transition threshold score, the processor may determine that the terminal's movement is insufficient to perform language transition or that there is a possibility of unintended behavior. In this case, the language transition control value is calculated as a value indicating a transition hold, and the current input language, current output language, and current speaker may be retained.

[0201] For example, the processor can calculate a single conversion score using the following mathematical formula 1.

[0202] [Mathematical Formula 1]

[0203]

[0204] Single conversion score is the conversion support score and conversion inhibition score It can be calculated using. Transition support score is the direction matching score , conversion intent score , utterance alternation suitability score , relative angle change amount , angular velocity Can reflect. Conversion inhibition score is the unintended behavior score , utterance persistence status value , change in acceleration , direction change history risk value It can reflect.

[0205] Transition support score It can be calculated as follows.

[0206] [Mathematical Formula 2]

[0207]

[0208] Conversion Inhibition Score It can be calculated as follows.

[0210]

[0211] The processor is a single transition point A conversion criteria score In the case of abnormalities, that is In this case, the language switching control value can be calculated as a value indicating that switching is allowed. Conversely, a single switching score A conversion criteria score In cases where it is less than, i.e. In this case, the language switching control value can be calculated as a value indicating a switching hold.

[0212] Here This can be a direction matching score that indicates the degree to which the direction at the end of the terminal's rotation is included in the direction range corresponding to the direction of the second user. is a switching intention score that can indicate the degree to which the rotation pattern of the terminal corresponds to the intended rotation pattern. This is a speech alternation suitability score that can indicate the degree to which rotational motion corresponds to the speech alternation point.

[0213] This is an unintended behavior score that can indicate the degree to which the movement of the terminal corresponds to shaking, changes in grip, or movement during speech. is a speech duration state value that can indicate the extent to which the current speaker is continuing to speak. is a change in acceleration that can indicate whether the movement of the terminal is close to shaking or impact movement. is a direction change history risk value that may indicate recent repetitive direction change candidate actions or instability of the language state.

[0214] , , , , can represent the direction alignment score, transition intention score, utterance alternation suitability score, and transition support weights for relative angle change and angular velocity, respectively. These coefficients represent the transition support scores of the corresponding elements. You can adjust the ratio reflected in it.

[0215] , , , may represent the conversion inhibition weights for the unintentional behavior score, utterance persistence state value, acceleration change amount, and direction change history risk value, respectively. These coefficients represent the conversion inhibition score of the corresponding element You can adjust the ratio reflected in it.

[0216] , , The transition support score when the direction alignment score, transition intention score, and utterance alternation suitability score each exceed the threshold value. This can mean an increasing sensitivity. , , , The unintentional behavior score, utterance persistence state value, acceleration change amount, and direction change history risk value are, respectively, the transition inhibition score. It can mean the sensitivity reflected in.

[0217] , , , , , , , can mean direction matching criterion value, transition intention criterion value, utterance alternation criterion value, angle criterion value, angular velocity criterion value, unintended behavior criterion value, acceleration change criterion value, and transition criterion score, respectively. The conversion inhibition score It is a stabilization coefficient to ensure the stability of the denominator, and is a single conversion score It can be a bias coefficient that adjusts the basic sensitivity of.

[0218] Transition support score at The part marked with is the direction matching score , conversion intent score and speech alternation suitability score α can be a non-linear incremental term that increases the conversion support effect as it exceeds each respective threshold value. Also, and It may be a saturated reinforcement term that increases the conversion support effect as the relative angle change amount and angular velocity approach the reference value, but prevents the score from increasing indefinitely due to excessive sensor values.

[0219] Conversion Inhibition Score at , , and may be a damping term that inhibits conversion as the risk of unintended behavior, speech persistence, shaky movement, and repeated conversions increases. Therefore, even if the terminal's orientation and rotation speed support conversion, if the current speaker continues speaking or the amount of acceleration change approaches a shaking pattern, the single conversion score It can decrease.

[0220] According to one embodiment, in operation (S306), if the language switching control value satisfies a reference condition, the processor may change the current input language to a second language, change the current output language to a first language, and change the current speaker to a second user.

[0221] At this time, the reference condition may be set based on at least one of a single transition score, a direction matching score, a transition intention score, a speech alternation suitability score, or an unintended behavior score. The processor may determine that the language switching control value satisfies the reference condition if it determines that the movement of the terminal corresponds to a rotational motion toward the direction of the second user and that speech alternation is possible after the current speaker's speech has ended.

[0222] If the language switching control value satisfies the reference condition, the processor can exchange the current input language and the current output language. Accordingly, when the second user speaks in the second language, the second language can be used as the reference language for speech recognition, and the first language can be used as the output language for translated text and synthesized speech. At the same time, the current speaker is changed to the second user, and the reference state for subsequent interpretation processing can be updated.

[0223] According to one embodiment, in operation (S307), the processor can calculate a second direction angle representing the current direction of the terminal based on sensor data received from the gyroscope sensor and the accelerometer sensor after the current speaker has been changed to a second user.

[0224] The second direction angle may be a value indicating which direction the terminal is facing after language switching. The processor may calculate the current direction of the terminal by using rotation angle or angular velocity information received from the gyroscope sensor and referring together with gravity direction or attitude change information received from the accelerometer sensor.

[0225] The operation (S307) may be an operation to continue tracking the direction state of the terminal even after language switching. In particular, if the terminal returns to the direction of the first user after the current speaker has changed to the second user, the return operation may be used as a basis for judgment to reset the current input language and the current output language to their initial state.

[0226] According to one embodiment, in operation (S308), the processor can determine whether the difference between the second direction angle and the reference direction angle corresponding to the first user's direction is less than or equal to the return allowance angle.

[0227] The reference direction angle may be a direction reference value corresponding to the direction of the first user, and the return allowance angle may represent an allowable range in which the terminal can be considered to have returned to the direction of the first user. The processor may calculate the difference between the second direction angle and the reference direction angle and compare whether the difference is less than or equal to the return allowance angle.

[0228] This judgment can be used as a primary condition to determine whether the terminal has actually returned to the direction of the first user. If the second direction angle simply approaches the reference direction angle instantaneously, it may not be concluded that it has returned yet, and the reference time and stable angular velocity conditions can be checked together in a subsequent operation.

[0229] According to one embodiment, in operation (S309), if the difference between the second direction angle and the reference direction angle is less than or equal to the return allowable angle and the state persists for a reference time or longer, and the angular velocity of the terminal is less than or equal to the stable angular velocity, the processor may determine that the terminal has returned to the second reference direction corresponding to the direction of the first user.

[0230] At this time, the reference time may be a time condition for determining whether the terminal's direction stably remains within a range corresponding to the first user's direction. The processor can determine whether a state in which the difference between the second direction angle and the reference direction angle is less than or equal to the return allowable angle has occurred temporarily or has been maintained for a certain period of time or longer.

[0231] Additionally, the stable angular velocity may serve as a reference value for determining whether the terminal is continuously rotating or is stably positioned in a specific direction. If the terminal's angular velocity is less than or equal to the stable angular velocity, the processor may determine that the terminal has stably returned toward the direction of the first user, rather than being in a shaking or moving state. Accordingly, erroneous return judgments caused by momentary shaking can be reduced.

[0232] According to one embodiment, in operation (S310), if the processor determines that the terminal has returned to the second reference direction, it may change the current input language to the first language, change the current output language to the second language, and change the current speaker to the first user.

[0233] The operation (S310) may be an operation to reset the interpretation status based on the first user when the terminal returns to the direction of the first user after language switching. Even if the current input language is set to the second language or the current output language is set to the first language, the processor may change the current input language to the first language and change the current output language to the second language based on the return determination result.

[0234] Additionally, the processor can change the current speaker to the first user, thereby ensuring that subsequent input voice data is processed to correspond to the first user's utterance. This return-based reset can reliably initialize the interpretation direction in cases where the language state is updated unintendedly or where the conversation flow returns to the first user after a previous language switch.

[0236] FIG. 4 is a block diagram exemplarily illustrating the layer configuration of an unintended behavior judgment model according to one embodiment.

[0237] Referring to FIG. 4, an unintended behavior judgment model (400) according to one embodiment may include an input alignment layer (401), a sensor feature normalization layer (402), a speech context encoding layer (403), a token embedding layer (404), a multi-head attention layer (405), a direction matching head (406), a rotation pattern head (407), a shake detection head (408), a speech state head (409), a hysteresis stability head (410), a context fusion layer (411), a direction matching score calculation layer (412), a rotation pattern score calculation layer (413), a shake score calculation layer (414), a speech alternation score calculation layer (415), and an output layer (416).

[0238] The unintended behavior judgment model (400) may be a model for determining whether the movement of the terminal is an intentional rotational motion for language switching, or an unintended behavior such as shaking, changing grip, or adjusting posture during speech. The model may receive sensor data, speech state data, language state, and direction change history together to generate judgment information used to calculate language switching control values.

[0239] The unintended behavior judgment model (400) can sequentially perform input alignment, sensor feature normalization, utterance context encoding, token embedding, multi-head attention, context fusion, and score calculation processing. Accordingly, the model can make a judgment by combining rotation direction, rotation pattern, utterance state, and history information, rather than making a judgment based on a single angle change alone.

[0240] The input alignment layer (401) may be a layer that aligns input tokens included in the integrated input sequence within a common time interval based on the time when a language switching candidate action occurs. This layer may align the temporal order of tokens corresponding to relative angle change amount, angular velocity, acceleration change amount, silence time, speech duration state value, and direction change history.

[0241] The input alignment layer (401) can align input tokens through time stamp matching, interpolation, or interval representative value calculation even when the sampling periods of the sensor data and the voice data are different. Accordingly, the subsequent layer can analyze whether the rotational motion occurred during the speech, after the end of the speech, or immediately after the previous transition.

[0242] The sensor feature normalization layer (402) can normalize input tokens corresponding to a relative angle change amount, angular velocity, change amount of angular velocity, change amount of acceleration, rotation duration, and rotation direction based on a first reference direction angle. This layer can reconstruct sensor features based on relative movement from the direction of the first user rather than the absolute posture of the terminal.

[0243] The sensor feature normalization layer (402) can obtain sensor feature tokens representing the rotation size, rotation speed, change in rotation speed, change in acceleration, rotation persistence, and rotation direction of the terminal. Accordingly, even if the gripping posture or initial tilt differs for each user, the model can distinguish between a rotation pattern toward the direction of the second user and an unintended shaking pattern.

[0244] The utterance context encoding layer (403) can generate utterance context tokens representing the current utterance state of the speaker based on the silence time and the utterance duration state value. The utterance context tokens may represent a state where the utterance continues, a state immediately after the utterance ends, a state where silence is maintained, or a state where it is unclear whether the utterance is being made.

[0245] The speech context encoding layer (403) enables the model to determine at what point in the conversation flow the rotation action occurred. For example, rotation during speech may indicate a possibility of changing the grip, and rotation during a silent period after the end of speech may be expressed in a context close to an action of handing over the interpretation direction to a second user.

[0246] The token embedding layer (404) can convert sensor feature tokens, utterance context tokens, input language state tokens, output language state tokens, speaker state tokens, and history tokens into embedding vectors of the same dimension. This layer can project different data formats into a common representation space suitable for subsequent attention processing.

[0247] The token embedding layer (404) can add relative time information based on the time of occurrence of the language switching candidate action to each embedding vector. Accordingly, the model can analyze together the time sequence in which angle change, silence time, speech duration state, and direction change history occurred.

[0248] The multi-head attention layer (405) may be a layer that applies multiple attention heads to embedding vectors to which relative time information is added. This layer may include an orientation matching head (406), a rotation pattern head (407), a shake detection head (408), a utterance state head (409), and a hysteresis stability head (410) as internal components.

[0249] The multi-head attention layer (405) can analyze the same integrated input sequence in parallel from multiple perspectives. Since direction, speed, acceleration, speech state, and transition history have different meanings, a complex context regarding the movement of the terminal can be extracted by configuring multiple heads to pay attention to each judgment element.

[0250] The direction matching head (406) can analyze the relationship between the rotation direction of the terminal and the direction of the second user by referring to embedding vectors corresponding to the relative angle change amount and the rotation direction. This head can be used to determine whether the terminal has deviated from the direction of the first user and has actually rotated toward the direction of the second user.

[0251] The direction matching head (406) can generate a direction matching context vector indicating how well the direction at the end of the rotation matches the direction range corresponding to the direction of the second user. The vector can be used as direction-related judgment information to support the transition in a subsequent direction matching score calculation layer.

[0252] The rotation pattern head (407) can analyze the relationship between the rotation speed change and rotation duration of the terminal by referring to embedding vectors corresponding to the angular velocity, the amount of change in angular velocity, and the rotation duration. This head can be used to distinguish whether the movement of the terminal is an intentional rotation pattern or a momentary movement.

[0253] The rotation pattern head (407) can represent changes in speed in the rotation start section, rotation progress section, and rotation end section as a single context. When rotation is maintained in a constant direction and the rotation duration falls within a reference range, the rotation pattern context vector can be composed of information supporting language switching intentions.

[0254] The shaking detection head (408) can distinguish whether the movement of the terminal is a rotational motion or a shaking motion by referring to embedding vectors corresponding to the change in acceleration and the change in angular velocity. This head can be used to determine unintended movements such as hand shaking, impact while moving, or changes in the grip of the terminal.

[0255] The shake detection head (408) can represent a pattern in which a rapid change in acceleration occurs within a short period of time or a change in angular velocity is repeatedly reversed as a shake context. The shake context vector can be used as a primary input to calculate an unintended behavior score in a subsequent shake score calculation layer.

[0256] The speech status head (409) can analyze whether the movement of the terminal occurred during the current speaker's speech or after the end of speech by referring to embedding vectors corresponding to the silence time and speech duration status values. This head can be used to determine the temporal relationship between rotational movements and the conversation sequence.

[0257] The utterance state head (409) can represent different contexts for cases where the current speaker continues speaking and cases where a silent period is formed after the utterance ends. The utterance state context vector can contribute to reducing mistranslations caused by changes in retention during utterance and reflecting the rotation of the actual utterance change point as a language transition candidate.

[0258] The history stability head (410) can analyze the instability of recent repetitive candidate actions or language states by referencing an embedding vector corresponding to the direction change history and an embedding vector corresponding to the current speaker. This head can determine how the current candidate action is related to the previous change history.

[0259] The history stability head (410) can generate a history context vector by reflecting the elapsed time since the previous conversion, the number of recent conversion candidate occurrences, and the current state of the speaker. The vector can be used as a conversion suppression factor in the subsequent single conversion score calculation process to reduce the possibility of repeated misconversions or tangles in the language state.

[0260] The context fusion layer (411) can combine the direction matching context vector, the rotation pattern context vector, the shaking context vector, the utterance state context vector, and the history context vector. This layer can perform the role of composing the judgment information extracted from each attention head into a single integrated context vector.

[0261] The context fusion layer (411) can apply normalization processing and non-linear transformation processing to the combined context vector. Accordingly, head-specific context information having different sizes and distributions is stably combined, and subsequent score calculation layers can calculate consistent judgment scores based on the integrated context vector.

[0262] The direction matching score calculation layer (412) can calculate a direction matching score based on an embedding vector corresponding to a relative angle change amount, an embedding vector corresponding to a rotation direction, and direction range information corresponding to the direction of the second user. The direction matching score can indicate the degree to which the terminal direction at the time of rotation termination is included in the direction range of the second user.

[0263] The direction matching score calculation layer (412) can distinguish between cases where the terminal simply moves in a random direction and cases where it is rotated toward the direction of the second user. The direction matching score can be reflected as a supporting factor in the subsequent language switching control value calculation process.

[0264] The rotation pattern score calculation layer (413) can calculate a switching intention score based on an embedding vector corresponding to angular velocity, an embedding vector corresponding to rotation duration, and an embedding vector corresponding to the amount of change of angular velocity. The switching intention score can indicate the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern.

[0265] The rotation pattern score calculation layer (413) can determine whether rotation occurs continuously for a certain period of time, whether the change in rotation speed is irregular, or whether the duration of rotation can be distinguished from a change in grip. The calculated transition intention score can be used as an indicator to show whether the movement of the terminal is close to an operation for language transition.

[0266] The shaking score calculation layer (414) can calculate an unintended behavior score based on an embedding vector corresponding to a change in acceleration, an embedding vector corresponding to a change in angular velocity, and an embedding vector corresponding to a rotation duration. The unintended behavior score can indicate the degree to which the movement of the terminal corresponds to shaking, a change in grip, or an impact movement.

[0267] The shaking score calculation layer (414) can analyze whether a change in the direction of the terminal is a stable rotation or an irregular movement, even if such change exists. The unintended behavior score is reflected as a factor that suppresses conversion in the subsequent single conversion score calculation process and can be used to reduce unnecessary language conversion.

[0268] The speech alternation score calculation layer (415) can calculate a speech alternation suitability score based on an embedding vector corresponding to a silent time, an embedding vector corresponding to a speech duration state value, and time information corresponding to the time of occurrence of a rotational motion. This score can indicate the degree to which the rotational motion corresponds to the time of speech alternation.

[0269] The utterance alternation score calculation layer (415) can determine whether a rotation occurred in a silent interval after the end of an utterance, whether a rotation occurred while the current speaker continues to speak, or whether a rotation occurred while the utterance status is unclear. The utterance alternation suitability score can be a factor that reflects the suitability of the conversational flow when determining whether language switching occurs.

[0270] The output layer (416) can output a direction matching score, a switching intention score, an unintended behavior score, and a speech alternation suitability score. The output layer (416) can pass each score to a subsequent language switching control value calculation operation so that it is used to determine whether to exchange the current input language and the current output language.

[0271] In some embodiments, the output layer (416) may output a judgment reason code, an event log, or a reason for holding the transition together with each score. This output information may be used for threshold correction based on the user's operation habits, retraining of an unintended behavior judgment model, or management of language transition history. Explanation of the symbols

[0273] 100: System

Claims

Claim 1 A method for real-time conversation interpretation using a terminal, wherein the terminal comprises an input unit, a microphone, a gyroscope sensor, an accelerometer sensor, a voice output unit, a memory, and one or more processors, and by the processor, an operation of setting a first user's first language as the current input language, setting a second user's second language as the current output language, and setting the current speaker as the first user; by the processor, an operation of assigning time information according to the same time standard to voice data received from the microphone and sensor data regarding the movement of the terminal received from the gyroscope sensor and the accelerometer sensor, or by referring to the time information included in the voice data and the sensor data to sort the voice data and the sensor data in chronological order; and by the processor, an operation of calculating a speech duration and a speech duration state value indicating whether the current speaker is continuing to speak based on the voice data.The above processor determines whether the movement of the terminal corresponds to a language switching candidate operation based on a relative angle change amount representing the difference between a first current direction angle representing the current direction of the terminal and a first reference direction angle corresponding to the direction of the first user, an angular velocity representing the rotational speed of the terminal, a change amount of the angular velocity, a change amount of the acceleration of the terminal, a rotational duration of the terminal, a rotational direction of the terminal, a silence time, and a speech duration state value; and if the movement of the terminal corresponds to the language switching candidate operation, calculates a language switching control value indicating whether to exchange the current input language and the current output language based on the relative angle change amount, the angular velocity, the change amount of acceleration, the rotational duration, the rotational direction, the silence time, the speech duration state value, the current input language, the current output language, the current speaker, a direction switching history, and the difference between the first reference direction angle and the first current direction angle; and if the language switching control value satisfies a reference condition, the current input language is changed to the second language, and the current A real-time conversation interpretation method comprising the operation of translating an output language into the first language and changing the current speaker to the second user.; Claim 2 A real-time conversation interpretation method according to claim 1, further comprising: an operation by which, after the current speaker is changed to the second user, the processor calculates a second direction angle representing the current direction of the terminal based on sensor data received from the gyroscope sensor and the accelerometer sensor; an operation by which, by which, the processor determines whether the difference between the second direction angle and the reference direction angle is less than or equal to the return allowance angle; an operation by which, if the state in which the difference between the current direction angle and the reference direction angle is less than or equal to the return allowance angle persists for a reference time or longer and the angular velocity of the terminal is less than or equal to the stable angular velocity, the processor determines that the terminal has returned to a second reference direction corresponding to the direction of the first user; and, by which, when the processor determines that the terminal has returned to the second reference direction, the current input language is changed to the first language, the current output language is changed to the second language, and the current speaker is changed to the first user. Claim 3 In claim 1, the operation of calculating the language switching control value comprises: an operation by the processor to determine whether the movement of the terminal corresponds to a rotational movement from the direction of the first user toward the direction of the second user, based on the relative angle change amount, the angular velocity, the change amount of the angular velocity, the change amount of the acceleration, the rotation duration, and the rotation direction; an operation by the processor to determine whether, if the movement of the terminal corresponds to the rotational movement, the rotational movement is a movement that occurred while the current speaker's speech continues, based on the silence time and the speech duration state value; an operation by the processor to classify the rotational movement as a candidate for unintended action if the rotational movement is determined to be a movement that occurred while the current speaker's speech continues, and classify the rotational movement as a candidate for language switching intent if the rotational movement is determined to be a movement that occurred after the current speaker's speech has ended; and an operation by the processor to determine the relative angle change amount, the angular velocity, the change amount of the acceleration, the rotation duration, the rotation direction, the silence time, and the speech An operation of converting a persistence state value, the current input language, the current output language, the current speaker, the direction change history, and the difference between the first reference direction angle and the first current direction angle into input tokens, respectively, and arranging the input tokens in chronological order to generate an integrated input sequence; an operation of analyzing the interrelationships between the input tokens included in the integrated input sequence through the unintended behavior judgment model by the processor;An operation by the processor to calculate a direction matching score indicating the degree to which the rotation direction of the terminal aligns with the direction of the second user based on the relative angle change amount and the rotation direction through the unintended behavior judgment model, to calculate a transition intention score indicating the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern based on the angular velocity and the rotation duration, to calculate an unintended behavior score indicating the degree to which the movement of the terminal corresponds to shaking based on the acceleration change amount, and to calculate a speech alternation suitability score indicating the degree to which the rotation motion corresponds to the speech alternation point based on the silence time and the speech duration state value; an operation by the processor to calculate a single transition score by reflecting the transition intention score, the direction matching score, the speech alternation suitability score, and the score as factors supporting the transition, and reflecting the unintended behavior score, the speech duration state value, the acceleration change amount, and the direction transition history as factors suppressing the transition; A real-time conversation interpretation method comprising the operation, wherein the processor calculates the language switching control value as a value indicating switching allowance when the single switching score is greater than or equal to the switching threshold score, and calculates the language switching control value as a value indicating switching hold when the single switching score is less than the switching threshold score. Claim 4 In paragraph 3, the operation of analyzing the interrelationships between input tokens included in the integrated input sequence through the unintended behavior judgment model comprises: an operation of aligning input tokens corresponding to the relative angle change amount, the angular velocity, the change amount of the angular velocity, the change amount of the acceleration, the rotation duration, the rotation direction, the silence time, the speech duration state value, the current input language, the current output language, the current speaker, and the direction change history included in the integrated input sequence within a common time interval based on the point in time when the language switching candidate operation occurred, by means of the input alignment layer of the unintended behavior judgment model; and, by means of the sensor feature normalization layer of the unintended behavior judgment model, normalizing the input token corresponding to the relative angle change amount, the input token corresponding to the angular velocity, the input token corresponding to the change amount of the angular velocity, the input token corresponding to the change amount of the acceleration, the input token corresponding to the rotation duration, and the input token corresponding to the rotation direction based on the first reference direction angle, and indicating the rotation magnitude, rotation speed, change in rotation speed, change in acceleration, rotation duration, and rotation directionality of the terminal An operation of acquiring sensor feature tokens; an operation of generating speech context tokens representing at least one of a state in which the current speaker's speech continues, a state immediately after the current speaker's speech ends, a state in which silence is maintained, and a state in which speech is unclear, based on an input token corresponding to the silence time and an input token corresponding to the speech duration state value, by the speech context encoding layer of the unintended behavior judgment model;By the token embedding layer of the unintended behavior judgment model, the sensor feature tokens, the utterance context tokens, the input language state token corresponding to the current input language, the output language state token corresponding to the current output language, the speaker state token corresponding to the current speaker, and the history token corresponding to the direction change history are converted into embedding vectors of the same dimension, and relative time information based on the occurrence time of the language change candidate action is added to the embedding vectors; by the multi-head attention layer of the unintended behavior judgment model, one of a plurality of attention heads is applied to each of the embedding vectors to which the relative time information is added; by the direction matching head among the plurality of attention heads, a direction matching context vector is generated by referencing the embedding vectors corresponding to the relative angle change amount and the rotation direction to generate a direction matching context vector representing the relationship between the rotation direction of the terminal and the direction of the second user; by the rotation pattern head among the plurality of attention heads, the embedding vectors corresponding to the angular velocity, the change amount of the angular velocity, and the rotation duration are referenced, the An operation to generate a rotation pattern context vector representing the relationship between the change in rotation speed and the rotation persistence of the terminal; an operation to generate a shake context vector for distinguishing whether the movement of the terminal is a rotational movement or a shake movement by referencing embedding vectors corresponding to the change in acceleration and the change in angular velocity by a shake discrimination head among the plurality of attention heads; an operation to generate a speech state context vector for distinguishing whether the movement of the terminal is a movement that occurred during the current speaker's speech or a movement that occurred after the end of speech by referencing embedding vectors corresponding to the silence time and the speech persistence state value by a speech state head among the plurality of attention heads.A real-time conversation interpretation method comprising: an operation to generate a history context vector indicating recent repetitive direction change candidate actions or instability of language states by referencing an embedding vector corresponding to the direction change history and an embedding vector corresponding to the current speaker by means of a history stability head among the plurality of attention heads; and an operation to generate an integrated context vector by combining the direction matching context vector, the rotation pattern context vector, the shake context vector, the utterance state context vector, and the history context vector by means of a context fusion layer of the unintended behavior judgment model, and applying normalization processing and non-linear transformation processing to the combined context vector. Claim 5 In paragraph 4, the operation of calculating the degree to which the rotational motion corresponds to the speech alternation point comprises: an operation of calculating a direction matching score indicating the degree to which the direction at the point of rotation termination of the terminal is included in the direction range corresponding to the second user's direction, based on an embedding vector corresponding to the relative angle change amount, an embedding vector corresponding to the rotation direction, and direction range information corresponding to the second user's direction, by means of the direction matching score calculation layer of the unintentional behavior judgment model; an operation of calculating a transition intention score indicating the degree to which the rotation pattern of the terminal corresponds to an intentional rotation pattern, based on an embedding vector corresponding to the angular velocity, an embedding vector corresponding to the rotation duration, and an embedding vector corresponding to the change amount of the angular velocity, by means of the rotation pattern score calculation layer of the unintentional behavior judgment model; and an unintentional behavior score indicating the degree to which the movement of the terminal corresponds to shaking, based on an embedding vector corresponding to the acceleration change amount, an embedding vector corresponding to the change amount of the angular velocity, and an embedding vector corresponding to the rotation duration, by means of the shaking score calculation layer of the unintentional behavior judgment model. A real-time conversation interpretation method comprising: a calculation operation; and a calculation operation by the speech alternation score calculation layer of the unintended behavior judgment model, wherein the calculation is based on an embedding vector corresponding to the silence time, an embedding vector corresponding to the speech duration state value, and time information corresponding to the occurrence time of the rotation operation, and a speech alternation suitability score indicating the degree to which the rotation operation corresponds to the speech alternation time.

Images