A location prior and multi-source redundant positioning emotionally coupled intent interaction method

Abstract

The application discloses a multimode terminal adaptive spatial voice intention recognition method and device. The application comprises: distinguishing fixed or portable mode according to equipment displacement posture and switching corresponding coordinate system, and constructing an identity position prior knowledge base for storing user identity, voiceprint, behavior and high-frequency intention. Multimodal signals are collected, acoustic, visual and external three-source positioning are fused to complete coordinate arbitration, and speaker authentication binding is realized by combining azimuth, face, voiceprint similarity weighted fusion. Space verification is completed relying on the equipment coordinate system, high-probability intention candidate set is pre-generated by calling the prior knowledge base, and the semantic retrieval range is reduced. Coupling reasoning is carried out by fusing four-dimensional constraints of single sentence intention, context, emotion and identity position, sentence ambiguity is resolved by means of position prior, and accurate user intention is output. The application takes into account equipment multi-working-condition adaptation and multi-source positioning fusion verification, candidate intention is predicted in advance, the recognition interval is narrowed, and the accuracy of voice intention recognition in complex scenes is improved by multi-dimensional constraint.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an emotion-coupled intention interaction method that combines location prior and multi-source redundant localization. Background Technology

[0002] Intelligent voice interaction, as an important means of natural human-computer interaction, is widely used in smart homes, mobile terminals, and service robots. In related technologies, a multi-speaker recognition and interaction system based on device-centric coordinates has been constructed through the collaborative operation of microphone arrays and cameras. Specifically, this system covers key aspects such as acoustic localization, visual localization, emotional feature extraction, and semantic intent parsing, and introduces external UWB / AOA localization assistance to enhance global spatial perception capabilities.

[0003] However, existing interaction methods directly adopt a fixed space determination mechanism centered on the device, without constructing an adaptive switching architecture between human-centered and device-centered dual coordinate systems. This may lead to the failure of interaction space rules after the portable mobile terminal is moved, or a sharp drop in the reliability of acoustic and visual positioning in harsh environments, resulting in problems such as identity drift, cross-reference of intent, and difficulty in resolving ambiguous references, which seriously affect the accuracy of personalized intent recognition and the robustness of the system in multiple scenarios. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose an emotion-coupled intention interaction method that combines location prior and multi-source redundant localization.

[0006] Another objective of this invention is to propose an emotion-coupled intention interaction device that combines location prior and multi-source redundant positioning.

[0007] The third objective of this invention is to provide a computer device.

[0008] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes an emotion-coupled intention interaction method based on location prior and multi-source redundant localization, comprising: S1 automatically determines the working mode of the device based on the displacement and attitude fluctuation data of the device. In the fixed terminal mode, the device center coordinate system is enabled, and in the portable mobile terminal mode, it is switched to the human-centered dynamic coordinate system. It also builds a priori knowledge base of identity and location for each registered user, which includes identity identifiers, voice features, behavioral preferences, and high-frequency intent sets in the corresponding scenarios. S2 collects multimodal signals and performs fusion arbitration through three-source redundant positioning: acoustic positioning, visual positioning, and external positioning. When the credibility of audio-visual positioning meets the standard, the audio-visual fusion location is adopted; otherwise, it automatically switches to external positioning coordinates as a reliable location source. It also combines orientation matching degree, face similarity, and voiceprint similarity for dynamic weight fusion to complete the speaker's identity confirmation and temporary identity binding. S3. Based on the coordinate system corresponding to the current working mode, perform adaptive spatial verification on the speaker after the rights are confirmed. After the verification is passed, query the prior knowledge base of the identity location and predict the set of high-probability candidate intentions in the current scene before semantic recognition is completed, so as to narrow the semantic retrieval range. S4. The initial intent of a single sentence, multi-round contextual logic constraints, user emotion correction items, and identity and location prior preference constraints are used as four-dimensional reasoning constraints to perform deep coupling reasoning on the candidate intent set, and the location prior is used to resolve ambiguities of vague referential statements and output the final intent.

[0010] In one embodiment of the present invention, S1 includes: The device's displacement and attitude fluctuation data are collected in real time by IMU attitude sensor. When the device coordinate fluctuation is very small and the displacement does not exceed the preset threshold, it is automatically determined to be a fixed terminal mode and the device center coordinate system is enabled. When the device continues to move and the attitude changes frequently, it is automatically determined to be a portable mobile terminal mode and switched to human-centered dynamic coordinate system. Collect the voice features and personal habit features of each registered user, generate a unique and tamper-proof user identity identifier through hash encryption, and establish a correspondence between the user identity identifier and the legal location and distance range around the device in fixed mode, the preset adaptive parameters of the personal interaction space in mobile mode, and the behavioral preferences in different motion states, and store them locally at the edge. The user's identity identifier, location, or movement state are associated with the corresponding high-frequency intent set to construct an identity location prior knowledge base containing static prior branches and dynamic prior branches. The static prior branch stores the mapping between the user's identity identifier and the location to the high-frequency intent set of that location, while the dynamic prior branch stores the mapping between the user's identity identifier and the movement state to the dynamic candidate intent pool.

[0011] In one embodiment of the present invention, S2 includes: Echo cancellation, noise reduction, and speech start and end point detection are performed on the mixed speech of multiple people captured by the microphone. The overlapping speech is split into single-person clean speech by using a separation algorithm combining Transformer and CNN with attention mechanism. The location and distance of the sound source are calculated by TDOA or SRP-PHAT algorithm to obtain acoustic localization results and acoustic confidence value. When the camera is in normal use, it recognizes faces and extracts facial features, calculates the spatial angle of the person relative to the device, obtains visual positioning results, and fuses acoustic positioning results and visual positioning results to obtain audio-visual fusion position and comprehensive credibility. The device uses its built-in IMU, microphone array, camera, or Bluetooth ranging and direction finding to calculate the user's local coordinates relative to the portable mobile device in real time. Combined with the device's own real-time position obtained through the terminal's internal positioning system module, the user's global real-time coordinates are calculated. The user with the optimal dynamic relative position is identified as the main interactive object and used as the dynamic origin of the human-centered system. At the same time, the device receives the global coordinates of the person, the tag identification, and the positioning reliability output by the external positioning tag in real time. When there is sufficient light, no obstruction, and little echo, and the overall credibility of audio-visual positioning meets the standard, the audio-visual fusion location is used as the main location. When the lens is blocked, the light is dim, or the room has severe echo, and the overall credibility of audio-visual positioning is low, the external positioning coordinates are automatically switched as a reliable location source. A comprehensive scoring model is constructed, the weights of location matching degree, face similarity and voiceprint similarity are dynamically adjusted, the comprehensive matching score is calculated, a temporary user identity is initially generated, and Kalman smoothing is performed on continuous positions to suppress jitter. When a face moves out of the frame or is obscured, the system relies on historical motion trajectories combined with voiceprint clustering to maintain uninterrupted identity and smoothly resume the session after the face reappears. When the identity of the user is confirmed to be reliable by the external location tag, the current user's voice clip and face image are stored in a buffer as backup samples for subsequent model updates.

[0012] In one embodiment of the present invention, the method further includes: calculating the local coordinates of the user relative to the portable mobile device in real time through the device's built-in IMU, microphone array, camera, or mobile phone Bluetooth ranging and direction finding, and determining the user with the optimal dynamic relative position as the main interaction object as the dynamic origin of the human-centered system, so as to determine how close the person is to the device and in which direction, thereby determining that the person closest to the device is most likely to give instructions.

[0013] In one embodiment of the present invention, S3 includes: In fixed terminal mode, a user is considered a legitimate user if both the voiceprint matching criteria are met and the speaking location is within the device's preset interaction range. In portable mobile terminal mode, the user's real-time relative position coordinates are calculated based on the device's real-time location and the user's relative coordinates to determine the current main interaction user and identify the current valid user. When the sound source is in the near area and the voiceprint is confirmed to be the device owner, the highest intent prior is given and the main interaction lock is entered directly without wake-up. When the relative distance is greater than the preset threshold, even if a registered user is identified, the interaction intensity is pre-judged as extremely low and classified as background dialogue and discarded. When another registered user is perceived to be continuously approaching the device from another room through external positioning, their user profile is pre-loaded to increase the prior probability of their appearance in the relative area. After identity verification is completed but before the entire sentence has been fully recognized, the prior knowledge base of the identity location is queried to list the set of high-probability candidate intentions in the current scenario of the user in order to narrow the scope of semantic retrieval. Temporary user identifiers are assigned to voices that do not belong to registered users, and a separate signal processing channel and session space are opened for them. All temporary recordings and feature data are automatically deleted after the session ends.

[0014] In one embodiment of the present invention, S4 includes: The initial literal intent of a single sentence, the logical constraints of multiple rounds of context, the user's current emotional correction items, and the prior preference constraints of identity and location are used as four-dimensional reasoning constraints. TSFL learning is used to mine the inherent relationship between the four, and deep coupling reasoning is performed on the candidate intent set to correct the one-sided problem caused by simple literal understanding. For ambiguous referential statements, in fixed scenarios, the final intent is selected from multiple candidate intents based on the behavioral preference scores corresponding to the location, while in portable mobile scenarios, the final intent is selected from multiple candidate intents based on the behavioral preference scores corresponding to the motion state. When encountering ambiguous instructions, personalized questions are generated based on users' common habits, and ambiguous handling samples are recorded to continuously optimize the relationship between the prior library and sentiment intent mapping. For urgent instructions, basic operations are performed first, and then details are confirmed.

[0015] In one embodiment of the present invention, it further includes: Each valid user identity is created with its own independent session space and context cache, and the chat history, to-do tasks and pronouns of different users are completely isolated; the frequency of user speech is counted, the context storage time is extended when there is frequent use, and memory resources are automatically released when there is no use for a long time. When two users take turns speaking and switch identities, the conversation progress of the previous user is automatically saved and can be seamlessly resumed when switching back; When a user pauses for a long time and then speaks again, the previous conversation context is automatically restored.

[0016] To achieve the above objectives, a second aspect of the present invention provides an emotion-coupled intention interaction device based on location prior and multi-source redundant localization, comprising: The mode coordinate system switching module is used to automatically determine the working mode of the device based on the displacement and attitude fluctuation data of the device. In the fixed terminal mode, the device center coordinate system is enabled, and in the portable mobile terminal mode, it is switched to the human-centered dynamic coordinate system. It also builds a priori knowledge base of identity and location for each registered user, which includes identity identifiers, voice features, behavioral preferences, and high-frequency intent sets in the corresponding scenarios. The multi-source positioning and identity verification module is used to collect multimodal signals and perform fusion arbitration through three redundant positioning sources: acoustic positioning, visual positioning, and external positioning. When the credibility of audio-visual positioning meets the standard, the audio-visual fusion location is adopted; otherwise, it automatically switches to external positioning coordinates as a reliable location source. It also combines orientation matching degree, face similarity, and voiceprint similarity for dynamic weight fusion to complete the speaker's identity verification and binding of temporary identity identifier. The spatial verification intent pre-screening module is used to perform adaptive spatial verification of the speaker after the rights have been confirmed according to the coordinate system corresponding to the current working mode. After the verification is passed, the module queries the prior knowledge base of the identity location and predicts the set of high-probability candidate intents in the current scene before semantic recognition is completed, so as to narrow the semantic retrieval range. The four-dimensional coupled intent reasoning module is used to perform deep coupled reasoning on the candidate intent set by taking the initial intent of a single sentence, multi-round contextual logic constraints, user emotion correction items, and identity and position prior preference constraints as four-dimensional reasoning constraints, and using position priors to resolve ambiguities in vague referential statements and output the final intent.

[0017] This invention discloses an emotion-coupled intent interaction method and apparatus based on location prior and multi-source redundant positioning, which is compatible with fixed and mobile terminals and enables universal interaction across all scenarios. It improves the stability of identity recognition in harsh environments through three-source redundant positioning and enhances the accuracy of personalized intent recognition by using an identity and location prior knowledge base to pre-judge intent and resolve ambiguities.

[0018] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the method described in the first aspect embodiment.

[0019] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] Figure 1 This is a flowchart of an emotion-coupled intention interaction method based on location prior and multi-source redundant localization according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall interactive business process according to an embodiment of the present invention; Figure 3 This is a block diagram of the three-source redundant positioning arbitration logic according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the mapping relationship of the dual-branch identity location prior knowledge base according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the four-dimensional coupled intention reasoning logic according to an embodiment of the present invention; Figure 6 This is a closed-loop block diagram of an online self-updating model driven by trusted identity of location tags according to an embodiment of the present invention; Figure 7 This is a structural diagram of an emotion-coupled intention interaction device based on location prior and multi-source redundant positioning according to an embodiment of the present invention. Figure 8 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following description, with reference to the accompanying drawings, describes an embodiment of the present invention that provides an emotion-coupled intention interaction method and apparatus based on location prior and multi-source redundant positioning.

[0025] Figure 1 This is a flowchart of an emotion-coupled intention interaction method based on location prior and multi-source redundant localization according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1 automatically determines the working mode of the device based on the displacement and attitude fluctuation data of the device. In the fixed terminal mode, the device center coordinate system is enabled, and in the portable mobile terminal mode, it is switched to the human-centered dynamic coordinate system. It also builds a priori knowledge base of identity and location for each registered user, which includes identity identifiers, voice features, behavioral preferences, and high-frequency intent sets in the corresponding scenarios. S2 collects multimodal signals and performs fusion arbitration through three-source redundant positioning: acoustic positioning, visual positioning, and external positioning. When the credibility of audio-visual positioning meets the standard, the audio-visual fusion location is adopted; otherwise, it automatically switches to external positioning coordinates as a reliable location source. It also combines orientation matching degree, face similarity, and voiceprint similarity for dynamic weight fusion to complete the speaker's identity confirmation and temporary identity binding. S3. Based on the coordinate system corresponding to the current working mode, perform adaptive spatial verification on the speaker after the rights are confirmed. After the verification is passed, query the prior knowledge base of the identity location and predict the set of high-probability candidate intentions in the current scene before semantic recognition is completed, so as to narrow the semantic retrieval range. S4. The initial intent of a single sentence, multi-round contextual logic constraints, user emotion correction items, and identity and location prior preference constraints are used as four-dimensional reasoning constraints to perform deep coupling reasoning on the candidate intent set, and the location prior is used to resolve ambiguities of vague referential statements and output the final intent.

[0026] This invention proposes another emotion-coupled intent interaction method based on location prior and multi-source redundant localization, such as... Figure 2 As shown, it includes: The system performs power-on initialization, device dynamic and static identification, dual coordinate system configuration, and dual-branch prior feature library construction. Hardware and communication initialization includes: device power-on self-test of microphone array, camera, and IMU attitude sensor, and unified audio, video, and positioning data timestamps; real-time acquisition of device displacement and attitude fluctuation data, and automatic determination of working mode: device coordinates with minimal long-term fluctuations and displacement not exceeding the threshold → fixed terminal mode, enabling the device center coordinate system; device with continuous displacement and frequent attitude changes → portable mobile terminal mode, switching to the human-centered (user-center) dynamic coordinate system; the mode switching is automatic throughout the process, requiring no manual settings.

[0027] Furthermore, multi-dimensional user profiles are created and bound: voice features (voiceprint template, intonation curve, speech rate patterns) and personal habit features (frequently used commands, speaking style, emotional expression characteristics) of each registered user are collected, and a unique and tamper-proof UserID is generated through hash encryption; fixed mode: the legal location and distance range of the user around the device are stored; mobile mode: preset adaptive parameters for the user's interactive space and behavioral preferences in different states of walking / stationary / transportation are set; a one-to-one correspondence is established between the location coordinates of the mobile mode and the UserID inside the system, all data is stored locally at the edge, can run normally when offline, and asynchronously synchronize data with the cloud when connected to the network to protect privacy.

[0028] Furthermore, a dynamic and static dual-branch identity-location prior knowledge base is constructed: Static prior (fixed mode): User ID + location → set of high-frequency intents for that location. Examples: When a user is in the living room, high-frequency intents include turning on the lights, turning on the TV, adjusting the volume, or related questions and answers; when a user is in the bedroom, high-frequency intents include turning off the lights, setting an alarm clock, playing bedtime audio, etc. Dynamic prior (mobile mode): User ID + movement state → dynamic candidate intent pool. Examples: When walking, the user tends to use navigation queries and schedule announcements; when stationary and holding a device, the user tends to use casual conversation and system settings. The prior knowledge base is used to predict intents in advance and resolve semantic ambiguities.

[0029] In one embodiment of the present invention, multimodal signal acquisition, three-source redundant positioning fusion arbitration, and preliminary speaker identity confirmation are performed as follows: Figure 3 As shown: Speech preprocessing and speech separation: The microphone collects the mixed speech of multiple people, and performs echo cancellation, noise reduction, and speech start and end point detection; the Transformer+CNN separation algorithm with attention mechanism is used to split the overlapping speech of multiple people into the pure speech of a single person; the TDOA / SRP-PHAT algorithm is used to calculate the location and distance of the sound source to obtain the acoustic localization result and the confidence value.

[0030] Furthermore, visual positioning calculation: when the camera is in normal use, it recognizes faces, extracts facial features, calculates the spatial angle of the person relative to the device, and obtains visual positioning results; it integrates acoustic and visual positions to obtain audio-visual fusion position and overall credibility.

[0031] Furthermore, external redundant positioning access and arbitration logic: The device's built-in IMU, microphone array, camera, or mobile phone Bluetooth ranging / direction finding calculates the user's local coordinates relative to the portable mobile device in real time; the device's own real-time position is obtained through the positioning system module within the terminal; the user's global real-time coordinates are calculated in real time by combining the user's local position relative to the device and the device's own position, and the user with the optimal dynamic relative position is determined as the primary interaction, serving as the dynamic origin of the human-centered system. "How close is the person to the device, and in which direction?" determines "the person closest is most likely to give instructions."

[0032] Furthermore, the system receives the global coordinates, tag ID, and location reliability of personnel from the positioning tags in real time; the system automatically determines the primary location to use: if the lighting is sufficient, there are no obstructions, and the echo is small, the audio-visual positioning reliability meets the standard → adopt the audio-visual fusion location; if the lens is obstructed, the lighting is dim, and the room has severe echo, the audio-visual positioning reliability is low → automatically switch to external positioning coordinates as a reliable location source.

[0033] Furthermore, the system incorporates a scenario-adaptive dynamic weight fusion identity binding and visual failure fallback ID renewal mechanism: It constructs a comprehensive scoring model, dynamically adjusts the weights of location matching degree, face similarity, and voiceprint similarity to adapt to scenarios such as face occlusion, multiple people crowding together, and long-distance voice transmission, calculates a comprehensive matching score, and initially binds and generates a temporary UserID; it performs Kalman smoothing on continuous positions to suppress jitter; when a face moves out of the frame or is occluded, it relies on historical motion trajectory + voiceprint clustering to maintain uninterrupted identity, and smoothly connects the conversation after the face reappears without resetting the dialogue.

[0034] Furthermore, sample caching preparation: if the identity of the current location tag is confirmed to be trustworthy, the voice clips and facial images of the current person are automatically stored in the buffer as backup samples for subsequent model updates.

[0035] In one embodiment of the present invention, adaptive spatial secondary verification, identity-location prior intent prediction, and temporary visitor isolation and control are implemented, such as... Figure 4 As shown, adaptive spatial dual identity verification is implemented: Fixed device mode: Users are identified as legitimate if they simultaneously meet the criteria of "voiceprint matching meets the standard" and "speaking location is within the device's preset interaction range." Mobile mode: The system determines the user currently in primary interaction based on the device's real-time location and the user's relative coordinates, thus confirming the current valid UserID. Location determination automatically uses audio-visual positioning or external positioning results, ensuring the verification logic remains functional even in harsh environments. Furthermore, in mobile mode, relative distance and orientation are inherently strong intent characteristics: Automatic near-field triggering: When the sound source is located in the "near-field zone" and the voiceprint is confirmed to belong to the device owner, the system immediately provides the highest intent prior, allowing direct entry into primary interaction lock without wake-up. For example, when the owner speaks to the phone at close range, it almost certainly means they are issuing a command to the device. Far-field suppression: Voice at a relative distance greater than a threshold (e.g., 1.5 meters), even if a registered user is identified, has extremely low interaction intensity prior. Strong visual gaze or a calling word is required to increase the intent probability; otherwise, it is directly classified as background dialogue and discarded. Absolute positioning assistance: If the system senses that another registered user is continuously approaching the device from another room through UWB or other means, even if the user has not spoken yet, it can preload the user profile to increase the prior probability that the user will appear in the relative area, thus enabling interaction preparation.

[0036] Furthermore, intent prediction is performed in advance: when identity verification is completed but the entire sentence has not yet been fully recognized, the system queries the corresponding prior knowledge base to list the user's most likely intent in the current scenario, thus narrowing the scope of recognition and improving recognition speed.

[0037] Furthermore, the system isolates unfamiliar users: if the voice does not belong to a registered user, a temporary UserID is assigned, and a separate signal processing channel and session space are created to avoid interfering with conversations of registered users; all temporary recordings and feature data are automatically deleted when the session ends; temporary tags can be registered as registered users with one click.

[0038] In one embodiment of the present invention, virtualized isolation context management and multi-round temporal semantic construction based on UserID are implemented, and independent session isolation is achieved: each valid UserID has its own independent session space and context cache, and different users' chat records, to-do tasks, and pronouns are completely separated, thus preventing A's instructions from being identified as B's content from the source.

[0039] Furthermore, context duration adaptive management: it counts the frequency of user speech, and extends the context retention time if the user speaks frequently; it automatically releases memory resources if the user does not speak for a long time; when two people take turns speaking and switch roles, it automatically saves the previous person's conversation progress and seamlessly resumes when switching back without having to repeat the previous content; if a user pauses for a long time and then speaks again, it automatically restores the previous conversation context and does not start from the beginning.

[0040] Furthermore, semantic parsing and temporal analysis: An optimized BERT model is used to parse the basic meaning of a single sentence and store it in the corresponding context; the logic, referential relationships, and unfinished tasks within multi-turn dialogues are analyzed to provide temporal constraints for subsequent intent reasoning. Example: First ask "How far have you progressed in listening to the audiobook?", then say "Fast forward a bit," and the system can recognize that it refers to an action related to listening to the audiobook.

[0041] In one embodiment of the present invention, single-user multimodal emotion extraction and personalized emotion computation are bound to the context to extract two types of emotion features: speech prosody features: speaking speed, tone fluctuation, volume, and pause frequency; and text semantic features: sentence praise or criticism tendency, emotional keywords, and characteristics of modal particles; additionally, the number of repeated questions and speaking speed can be counted as auxiliary judgment criteria.

[0042] Furthermore, personalized emotion fusion calculation: retrieve the user's exclusive emotion model. Since everyone has different emotional expression habits, the weight of each feature is automatically adjusted. Use the CSS cross-space collaborative fusion algorithm to comprehensively calculate emotions and output: emotion types such as calm, joy, anger, urgency, anxiety, and fatigue, as well as emotional intensity values ​​from 0 to 100.

[0043] Furthermore, special features are included: the mode for minors improves the sensitivity of recognizing negative emotions such as anger, grievance, and anxiety, while restricting large-amount spending and operation of high-risk devices; the mode for the elderly reduces misjudgments caused by slight fluctuations in speech speed and volume, resulting in more stable recognition.

[0044] Furthermore, the emotion type and intensity are bound to the user context as a fourth-dimensional reasoning constraint.

[0045] In one embodiment of the present invention, four-dimensional coupled depth intention reasoning, with position priors participating in ambiguity resolution, such as... Figure 5 As shown, a four-dimensional reasoning constraint is constructed: the initial literal intent of a single sentence; the logical constraints of multiple rounds of context; the user's current emotional correction; the prior preference constraint of identity and location; and the use of TSFL to learn and explore the inherent connections among the four to correct the one-sidedness caused by simple literal understanding.

[0046] Furthermore, here's an example of location adjudication in ambiguous scenarios: When a user gives the vague command "turn it off," in a fixed scenario: in the living room, the priority is to turn off the TV; in the bedroom, the priority is to turn off the bedside lamp. In a mobile handheld scenario: when walking, the priority is to turn off the navigation voice; when stationary, the priority is to turn off the portable music player. When multiple candidate intents appear, the final result is output based on the behavioral preference score corresponding to the location, thus resolving the issue of unclear referencing.

[0047] Furthermore, personalized disambiguation processing: when encountering ambiguous commands, instead of using a uniform template, questions are asked based on the user's common habits; for example, when listening to audiobooks, the question is asked, "Do you want to fast forward to the current book?"; at the same time, ambiguity processing samples are recorded to continuously optimize the mapping relationship between the prior library and sentiment intent; for urgent commands, basic operations can be performed first, and then details can be confirmed, balancing speed and accuracy.

[0048] In one embodiment of the present invention, hierarchical permission verification, multi-instruction concurrent conflict resolution, intent execution and context update are implemented. Permission verification distinguishes between administrators, ordinary users, and restricted minor users, and automatically blocks unauthorized operations, control of high-power dangerous electrical appliances, excessive entertainment time for minors, privacy queries, and other restricted instructions. Actions are only executed after verification is passed.

[0049] Furthermore, the system handles conflicts arising from multiple people speaking simultaneously, prioritizing users according to the following rules: Emergency equipment control > Information retrieval > Entertainment operations. Execution is prioritized based on user level and emotional intensity, with high-priority users responding immediately and low-priority users waiting in a queue. Users are also encouraged to manually adjust priorities.

[0050] Furthermore, the execution result is populated: the operation is successful / failed, the execution details are written to the user's context, the task progress is updated, and the current interaction loop is completed.

[0051] In one embodiment of the present invention, the online iteration of the location tag trusted identity-driven model and the global privacy closed-loop optimization are as follows: Figure 6 As shown, the validity of the samples is determined as follows: the external location tag identity has absolute credibility, and the voice and face samples with clear UserID in the buffer are used as valid training data. Incremental learning is used to update the user's voiceprint template and face model. The update is completed seamlessly in daily chat, which offsets the recognition degradation caused by changes in human voice, facial features and environmental echoes.

[0052] Furthermore, personalized local optimization: based on user feedback such as correcting recognition errors, supplementing instructions, and providing negative answers, the system automatically adjusts the weights of the personal sentiment model, the correspondence between sentiment intent and context, the duration strategy, and the prior probability distribution of location, making it increasingly more in line with individual usage habits.

[0053] Furthermore, federated learning optimizes privacy globally: it does not upload original audio recordings, facial data, or location privacy data, but only uploads model gradient parameters to perform multi-terminal collaborative optimization of the global basic model; it retains each person's personalized parameters locally, balancing privacy compliance, personalized adaptation, and overall recognition accuracy, forming a long-term self-iterative complete closed loop.

[0054] The embodiments of this invention also have the following technical effects: This invention constructs an automatic switching architecture with a human-centric and device-centric dual coordinate system, simultaneously compatible with both fixed and mobile modes, achieving universal interaction across all scenarios. It establishes a three-source redundant fault-tolerant positioning system based on acoustics, vision, and external positioning, automatically switching external positioning as a reliable location benchmark under harsh perception environments, improving the stability of identity recognition and spatial determination, and solving the problems of identity drift and intent cross-referencing. It constructs a dynamic and static dual-branch identity-location prior knowledge base, endowing location information with the dual functions of pre-judging intent and resolving ambiguities, narrowing the semantic retrieval range, resolving ambiguities in vague referential statements, and improving the accuracy of personalized intent recognition.

[0055] To achieve the above embodiments, such as Figure 7 As shown, this embodiment also provides an emotion-coupled intention interaction device 10 with location prior and multi-source redundant localization, including: The mode coordinate system switching module 100 is used to automatically determine the working mode of the device based on the displacement and attitude fluctuation data of the device. In the fixed terminal mode, the device center coordinate system is enabled, and in the portable mobile terminal mode, it is switched to the human-centered dynamic coordinate system. It also builds an identity and location prior knowledge base for each registered user, which includes identity identifiers, voice features, behavioral preferences, and high-frequency intent sets in the corresponding scenarios. The multi-source positioning identity confirmation module 200 is used to collect multimodal signals and perform fusion arbitration through three-source redundant positioning of acoustic positioning, visual positioning and external positioning. When the credibility of audio-visual positioning meets the standard, the audio-visual fusion position is adopted; otherwise, it automatically switches to external positioning coordinates as a reliable position source. It also combines orientation matching degree, face similarity and voiceprint similarity for dynamic weight fusion to complete the speaker's identity confirmation and binding of temporary identity identifier. The spatial verification intent pre-screening module 300 is used to perform adaptive spatial verification of the speaker after the rights have been confirmed according to the coordinate system corresponding to the current working mode. After the verification is passed, the module queries the prior knowledge base of the identity location and predicts the set of high-probability candidate intents in the current scene before semantic recognition is completed, so as to narrow the semantic retrieval range. The four-dimensional coupled intent reasoning module 400 is used to perform deep coupled reasoning on the candidate intent set by taking the initial intent of a single sentence, multi-round contextual logic constraints, user emotion correction items, and identity and position prior preference constraints as four-dimensional reasoning constraints, and to use position priors to make ambiguity decisions on vague referential statements and output the final intent.

[0056] This invention discloses an emotion-coupled intent interaction device based on location prior and multi-source redundant positioning, which is compatible with fixed and mobile terminals and enables universal interaction across all scenarios. It improves the stability of identity recognition in harsh environments through three-source redundant positioning and enhances the accuracy of personalized intent recognition by using an identity and location prior knowledge base to pre-judge intent and resolve ambiguities.

[0057] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 8 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0058] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0059] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A location-priority and multi-source redundant localization-coupled emotion-intention interaction method, characterized in that, include: S1 automatically determines the working mode of the device based on the displacement and attitude fluctuation data of the device. In the fixed terminal mode, the device center coordinate system is enabled, and in the portable mobile terminal mode, it is switched to the human-centered dynamic coordinate system. It also builds a priori knowledge base of identity and location for each registered user, which includes identity identifiers, voice features, behavioral preferences, and high-frequency intent sets in the corresponding scenarios. S2 collects multimodal signals and performs fusion arbitration through three-source redundant positioning: acoustic positioning, visual positioning, and external positioning. When the credibility of audio-visual positioning meets the standard, the audio-visual fusion location is adopted; otherwise, it automatically switches to external positioning coordinates as a reliable location source. It also combines orientation matching degree, face similarity, and voiceprint similarity for dynamic weight fusion to complete the speaker's identity confirmation and temporary identity binding. S3. Based on the coordinate system corresponding to the current working mode, perform adaptive spatial verification on the speaker after the rights are confirmed. After the verification is passed, query the prior knowledge base of the identity location and predict the set of high-probability candidate intentions in the current scene before semantic recognition is completed, so as to narrow the semantic retrieval range. S4. The initial intent of a single sentence, multi-round contextual logic constraints, user emotion correction items, and identity and location prior preference constraints are used as four-dimensional reasoning constraints to perform deep coupling reasoning on the candidate intent set, and the location prior is used to resolve ambiguities of vague referential statements and output the final intent.

2. The method as described in claim 1, characterized in that, S1 includes: The device's displacement and attitude fluctuation data are collected in real time by IMU attitude sensor. When the device coordinate fluctuation is very small and the displacement does not exceed the preset threshold, it is automatically determined to be a fixed terminal mode and the device center coordinate system is enabled. When the device continues to move and the attitude changes frequently, it is automatically determined to be a portable mobile terminal mode and switched to human-centered dynamic coordinate system. Collect the voice features and personal habit features of each registered user, generate a unique and tamper-proof user identity identifier through hash encryption, and establish a correspondence between the user identity identifier and the legal location and distance range around the device in fixed mode, the preset adaptive parameters of the personal interaction space in mobile mode, and the behavioral preferences in different motion states, and store them locally at the edge. The user's identity identifier, location, or movement state are associated with the corresponding high-frequency intent set to construct an identity location prior knowledge base containing static prior branches and dynamic prior branches. The static prior branch stores the mapping between the user's identity identifier and the location to the high-frequency intent set of that location, while the dynamic prior branch stores the mapping between the user's identity identifier and the movement state to the dynamic candidate intent pool.

3. The method as described in claim 1, characterized in that, The S2 includes: Echo cancellation, noise reduction, and speech start and end point detection are performed on the mixed speech of multiple people captured by the microphone. The overlapping speech is split into single-person clean speech by using a separation algorithm combining Transformer and CNN with attention mechanism. The location and distance of the sound source are calculated by TDOA or SRP-PHAT algorithm to obtain acoustic localization results and acoustic confidence value. When the camera is in normal use, it recognizes faces and extracts facial features, calculates the spatial angle of the person relative to the device, obtains visual positioning results, and fuses acoustic positioning results and visual positioning results to obtain audio-visual fusion position and comprehensive credibility. The device uses its built-in IMU, microphone array, camera, or Bluetooth ranging and direction finding to calculate the user's local coordinates relative to the portable mobile device in real time. Combined with the device's own real-time position obtained through the terminal's internal positioning system module, the user's global real-time coordinates are calculated. The user with the optimal dynamic relative position is identified as the main interactive object and used as the dynamic origin of the human-centered system. At the same time, the device receives the global coordinates of the person, the tag identification, and the positioning reliability output by the external positioning tag in real time. When there is sufficient light, no obstruction, and little echo, and the overall credibility of audio-visual positioning meets the standard, the audio-visual fusion location is used as the main location. When the lens is blocked, the light is dim, or the room has severe echo, and the overall credibility of audio-visual positioning is low, the external positioning coordinates are automatically switched as a reliable location source. A comprehensive scoring model is constructed, the weights of location matching degree, face similarity and voiceprint similarity are dynamically adjusted, the comprehensive matching score is calculated, a temporary user identity is initially generated, and Kalman smoothing is performed on continuous positions to suppress jitter. When a face moves out of the frame or is obscured, the system relies on historical motion trajectories combined with voiceprint clustering to maintain uninterrupted identity and smoothly resume the session after the face reappears. When the identity of the user is confirmed to be reliable by the external location tag, the current user's voice clip and face image are stored in a buffer as backup samples for subsequent model updates.

4. The method as described in claim 3, characterized in that, The method further includes: calculating the user's local coordinates relative to the portable mobile device in real time through the device's built-in IMU, microphone array, camera, or mobile phone Bluetooth ranging and direction finding, and determining the user with the optimal dynamic relative position as the main interaction object as the dynamic origin of the human-centered system, in order to determine how close the person is to the device and in which direction, thereby determining that the person closest to the device is most likely to give instructions.

5. The method as described in claim 1, characterized in that, The S3 includes: In fixed terminal mode, a user is considered a legitimate user if both the voiceprint matching criteria are met and the speaking location is within the device's preset interaction range. In portable mobile terminal mode, the user's real-time relative position coordinates are calculated based on the device's real-time location and the user's relative coordinates to determine the current main interaction user and identify the current valid user. When the sound source is in the near area and the voiceprint is confirmed to be the device owner, the highest intent prior is given and the main interaction lock is entered directly without wake-up. When the relative distance is greater than the preset threshold, even if a registered user is identified, the interaction intensity is pre-judged as extremely low and classified as background dialogue and discarded. When another registered user is perceived to be continuously approaching the device from another room through external positioning, their user profile is pre-loaded to increase the prior probability of their appearance in the relative area. After identity verification is completed but before the entire sentence has been fully recognized, the prior knowledge base of the identity location is queried to list the set of high-probability candidate intentions in the current scenario of the user in order to narrow the scope of semantic retrieval. Temporary user identifiers are assigned to voices that do not belong to registered users, and a separate signal processing channel and session space are opened for them. All temporary recordings and feature data are automatically deleted after the session ends.

6. The method as described in claim 1, characterized in that, The S4 includes: The initial literal intent of a single sentence, the logical constraints of multiple rounds of context, the user's current emotional correction items, and the prior preference constraints of identity and location are used as four-dimensional reasoning constraints. TSFL learning is used to mine the inherent relationship between the four, and deep coupling reasoning is performed on the candidate intent set to correct the one-sided problem caused by simple literal understanding. For ambiguous referential statements, in fixed scenarios, the final intent is selected from multiple candidate intents based on the behavioral preference scores corresponding to the location, while in portable mobile scenarios, the final intent is selected from multiple candidate intents based on the behavioral preference scores corresponding to the motion state. When encountering ambiguous instructions, personalized questions are generated based on users' common habits, and ambiguous handling samples are recorded to continuously optimize the relationship between the prior library and sentiment intent mapping. For urgent instructions, basic operations are performed first, and then details are confirmed.

7. The method as described in claim 1, characterized in that, The method further includes: Each valid user identity is created with its own independent session space and context cache, and the chat history, to-do tasks and pronouns of different users are completely isolated; the frequency of user speech is counted, the context storage time is extended when there is frequent use, and memory resources are automatically released when there is no use for a long time. When two users take turns speaking and switch identities, the conversation progress of the previous user is automatically saved and can be seamlessly resumed when switching back; When a user pauses for a long time and then speaks again, the previous conversation context is automatically restored.

8. A location-priority and multi-source redundant localization-based emotion-coupled intention interaction device, characterized in that, include: The mode coordinate system switching module is used to automatically determine the working mode of the device based on the device's displacement and attitude fluctuation data. In the fixed terminal mode, the device's central coordinate system is enabled, and in the portable mobile terminal mode, it is switched to the human-centered dynamic coordinate system. It also builds an identity and location prior knowledge base for each registered user, which includes identity identifiers, voice features, behavioral preferences, and high-frequency intent sets in the corresponding scenarios. The multi-source positioning identity verification module is used to collect multimodal signals and perform fusion arbitration through three redundant positioning sources: acoustic positioning, visual positioning, and external positioning. When the credibility of audio-visual positioning meets the standard, the audio-visual fusion location is adopted; otherwise, it automatically switches to external positioning coordinates as a reliable location source. It also combines orientation matching degree, face similarity, and voiceprint similarity for dynamic weight fusion to complete the speaker's identity verification and binding of temporary identity identifier. The spatial verification intent pre-screening module is used to perform adaptive spatial verification of the speaker after the rights have been confirmed according to the coordinate system corresponding to the current working mode. After the verification is passed, the module queries the prior knowledge base of the identity location and predicts the set of high-probability candidate intents in the current scene before semantic recognition is completed, so as to narrow the semantic retrieval range. The four-dimensional coupled intent reasoning module is used to perform deep coupled reasoning on the candidate intent set by taking the initial intent of a single sentence, multi-round contextual logic constraints, user emotion correction items, and identity and position prior preference constraints as four-dimensional reasoning constraints, and using position priors to resolve ambiguities in vague referential statements and output the final intent.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images