Multi-modal ai assistant behavior control method and system based on meeting scenario recognition

Abstract

The application discloses a kind of multi-modal AI assistant behavior control method and system based on conference scene identification, system constructs the gradient participation system from shadow mode, salon mode to host mode, and is matched with pause mode as privacy protection state, wherein the salon mode is subdivided into 4 levels of participation hierarchy, and the AI intervention depth of different conference scenes is matched;System acquires conference audio through AI hardware terminal, extracts acoustic and semantic multidimensional scene features, compares and calculates comprehensive conference deviation degree with preset threshold matrix;After determining that the state deviates and continues to exceed time, mute generation mode switching suggestion is pushed to authorized mobile terminal through encrypted channel, and after double-factor verification manual confirmation, progressive mode switching is executed, public and private domain output decoupling and full-link software and hardware collaborative control are realized.Non-invasive intervention logic and asymmetric architecture of public input and private output are used, and the natural rhythm of meeting, AI auxiliary efficiency and privacy security are considered.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, intelligent conference equipment, human-computer interaction and data security technology, and in particular to a multimodal AI assistant behavior control method and system based on conference scene recognition. Background Technology

[0002] With the development of speech recognition and big data technology, products such as smart voice recorders, meeting transcription software, and smart meeting terminals have been widely used in various meeting scenarios. Their core functions are concentrated in audio acquisition, speech-to-text conversion, meeting minutes generation, and task extraction.

[0003] The existing technology has the following obvious drawbacks: First, AI intervention methods are limited and highly intrusive. Some meeting systems with agenda control functions directly issue public voice prompts through speakers when they detect anomalies such as timeouts or going off-topic. This disrupts the meeting's rhythm and interferes with participants' train of thought, making it unsuitable for scenarios such as creative discussions and business negotiations where human control needs to be preserved.

[0004] Second, there is a lack of output isolation mechanisms adapted to sensitive scenarios. Existing meeting AI either starts collecting data and broadcasting it aloud simultaneously, or stops collecting data altogether, with no intermediate state. In highly sensitive negotiations and decision-making meetings, the moderator needs AI to provide fact-checking, background information, and summary of viewpoints, but does not want the AI ​​to speak publicly or for the auxiliary information to be known by other participants. Existing technology cannot meet this need.

[0005] Third, mode switching lacks intelligent and secure verification. Existing devices mostly use manual binary switching (record / pause), which cannot automatically provide mode adjustment suggestions based on the real-time status of the meeting; at the same time, mode switching lacks an authentication mechanism, making it easy for accidental touches or unauthorized personnel to operate, posing a risk of privacy leaks.

[0006] Fourth, the granularity of hardware and software collaborative control is coarse. The existing equipment only controls the recording function on and off, and does not achieve decoupled control of multiple channels such as audio acquisition, public output, private domain display, and light indication, making it unable to finely adapt to different types of meeting scenarios.

[0007] Therefore, there is an urgent need in this field for a multimodal AI assistant behavior control solution that can dynamically identify meeting status, provide non-intrusive AI assistance, achieve public-private domain output isolation, and balance ease of use and privacy security. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal AI assistant behavior control method and system based on meeting scenario recognition. Through the non-intrusive control logic of "meeting deviation recognition - private domain suggestion push - manual confirmation execution", it avoids AI from publicly interrupting the meeting. Through the asymmetric distribution architecture of "public domain input, private domain output" in shadow mode, it meets the AI ​​assistance needs and privacy protection needs in sensitive scenarios, while realizing multi-mode fine-grained software and hardware collaborative control and hierarchical security switching.

[0009] The objective of this invention is achieved through the following technical solution: A multimodal AI assistant behavior control method based on meeting scenario recognition includes the following steps: S1. Collect audio data from the meeting venue through AI hardware terminals and extract meeting scene features; S2. Compare the meeting scene features with the feature threshold matrix corresponding to the preset meeting type to determine whether the current meeting status deviates from the preset meeting type; S3. If it is determined that there is a deviation and the duration exceeds a preset threshold, a mode switching suggestion is generated; the audio output module of the AI ​​hardware terminal is controlled to remain silent and not publicly play the mode switching suggestion, and the mode switching suggestion is pushed to the authorized mobile terminal through an encrypted channel; S4. After receiving confirmation information from the authorized mobile terminal, voice control command, or physical input module, switch the behavior mode of the AI ​​hardware terminal; the behavior mode includes shadow mode, salon mode, host mode, and pause mode, wherein the salon mode includes multiple participation sub-levels; S5. After switching to the corresponding behavior mode, execute the matching behavior logic: When switching to shadow mode, the audio acquisition module of the AI ​​hardware terminal continues to acquire audio data, while blocking the public audio output path of the AI ​​hardware terminal. The meeting assistance information generated by the AI ​​engine is sent only to the authorized mobile terminal for display through a secure directional channel. When switching to salon mode, the corresponding triggering and output logic is executed according to the permission rules of the current sub-level; When switching to host mode, grant permissions to the audio output module, and the AI ​​will proactively and publicly output meeting control prompts based on the meeting agenda and speaking duration recognition results. When switching to pause mode, audio acquisition or audio data upload is stopped, and the queue of pending sensitive data is cleared. S6. Based on the switched behavior mode, coordinate the control of the audio acquisition module, audio output module, light indicator module of the AI ​​hardware terminal, and the display content of the authorized mobile terminal.

[0010] As a further improvement of the present invention, the meeting scene features include at least one of the following: speaking frequency, single-person speaking duration, tone fluctuation amplitude, sound power spectral density, frequency of multiple people speaking overlapping, silence duration, semantic divergence, keyword deviation, emotional intensity, topic matching degree, and factual deviation degree.

[0011] As a further improvement of the present invention, step S2, determining whether the current meeting status deviates from the preset meeting type, specifically includes: Obtain the meeting topic vector V_topic and the preset participation level corresponding to the preset meeting type; Semantically encode the speech content within the current sliding time window to obtain the current semantic vector V_current; Calculate the semantic divergence F_semantic=1-cosine_similarity(V_topic,V_current), and extract multiple deviation components of the acoustic dimension. The weighted average deviation of each dimension is used to calculate the overall meeting deviation degree D. The calculation formula is as follows: D=w1F_speech+w2F_pitch+w3F_semantic+w4F_overlap+w5*F_silence Where F_speech is the abnormal speech frequency component, F_pitch is the tone fluctuation or emotion intensity component, F_semantic is the semantic divergence component, F_overlap is the overlapping speech of multiple people component, F_silence is the silence duration component, and w1 to w5 are configurable weights. When the deviation of the overall meeting exceeds a preset threshold and the duration exceeds a preset time threshold, it is determined that the current meeting status deviates from the preset meeting type.

[0012] As a further improvement of the present invention, the salon mode is divided into four sub-levels according to the depth of AI participation from low to high, and the execution logic of each level is as follows: Level 1 roll call response: Only responds to explicit triggering actions of participants. After being triggered, it provides a public voice response to the question and returns to silence after the response is completed. Level 2: Raise your hand to speak: AI identifies valuable points in the meeting content in real time. When the trigger threshold is reached, a public domain prompt signal is issued through the light indicator module. At the same time, the speech draft is pushed to authorized mobile terminals. After manual confirmation, the speech is then output to the public domain. Level 3 Conditional Interruption: AI monitors the meeting status in real time according to preset interruption rules. When preset trigger conditions are met, no real-time confirmation is required, and a public voice prompt is given directly. Level 4: Free Participation: As an equal participant, AI autonomously judges the pace of discussion and the timing of its speech, naturally outputting its content in the public domain during dialogue breaks and participating in the meeting discussion.

[0013] As a further improvement of the present invention, blocking the public audio output path of the AI ​​hardware terminal in step S5 specifically includes at least one of the following methods: The software layer sets the audio output gain to zero, preventing all TTS audio and prompts from entering the speaker playback queue; Intercept all common external commands within the state machine logic; Turn off the audio power amplifier; The signal path of the speaker is physically disconnected by analog switches or relays.

[0014] As a further improvement of the present invention, in step S4, when switching to shadow mode, pause mode or high-privilege participation level, two-factor authentication must be satisfied, where both the identity verification condition and the operation confirmation condition are met simultaneously. The identity verification conditions include at least one of voiceprint recognition matching the preset controller's identity and authorized mobile terminal account verification. The operation confirmation conditions include at least one of authorized mobile terminal touch confirmation and AI hardware terminal physical button operation; Mode switching follows a gradual adjustment principle. Cross-mode switching prioritizes a gradual transition between Shadow Mode, Salon Mode, and Host Mode. Within Salon Mode, sub-levels support smooth bidirectional adjustment in ascending and descending order.

[0015] A multimodal AI assistant behavior control system based on meeting scene recognition includes an AI hardware terminal, an AI engine, an authorized mobile terminal, a mode control module, and a data security distribution module; The AI ​​hardware terminal is used to collect audio data at the meeting venue and perform audio output, light indication, and physical input interaction. The AI ​​engine is used to extract features from audio data to obtain meeting scene features, compare and judge the deviation of the meeting status, and generate mode switching suggestions and meeting assistance information. The mode control module is used to compare the meeting scene features with the feature threshold matrix of the preset meeting type. When it is determined that the meeting state deviates and continues to time out, it generates a mode switching suggestion and controls the audio output module of the AI ​​hardware terminal to remain silent and not publicly play the mode switching suggestion. After receiving confirmation information, it switches the behavior mode and coordinates the control of the display content of each module of the AI ​​hardware terminal and the authorized mobile terminal. The behavior modes include shadow mode, salon mode, host mode and pause mode, wherein the salon mode contains multiple levels of participation sub-levels. The data security distribution module is used to manage data flow permissions in different modes: in the lower sub-levels of shadow mode and salon mode, it keeps the audio acquisition channel of the AI ​​hardware terminal open, blocks the public audio output channel of the AI ​​hardware terminal, and sends the meeting assistance information generated by the AI ​​engine only to authorized mobile terminals for display through a secure directional channel; in the higher sub-levels of host mode and salon mode, it opens the public audio output channel permission; in pause mode, it completely cuts off audio acquisition and data transmission. The authorized mobile terminal is used to receive and display mode switching suggestions and meeting assistance information, receive user confirmation, rejection and adjustment operations and feed them back to the mode control module.

[0016] As a further improvement of the present invention, the AI ​​hardware terminal includes a control processing unit, an audio acquisition module, an audio output module, a light indicator module, a physical input module, a communication module, and a storage module. The audio acquisition module consists of a single microphone or a multi-microphone array and is used to acquire audio data at the meeting. The audio output module is used to publicly play audio content, and its output is blocked by the data security distribution module in the lower sub-levels of shadow mode, pause mode and salon mode. The light indicator module is used to indicate the current behavior pattern, AI's willingness to speak, and warning status through different lighting effects and flashing rhythms; The physical input module is used to receive user mode switching, pause triggering, and level adjustment operations; The control processing unit is used for local audio preprocessing, device status control, and edge AI inference.

[0017] As a further improvement of the present invention, the AI ​​engine includes a speech recognition module, a feature extraction module, a deviation judgment module, a content generation module, and a voiceprint recognition module. The speech recognition module is used to convert audio data into text content and supports multi-person voiceprint separation and role labeling; The feature extraction module is used to extract acoustic and semantic features to obtain multi-dimensional meeting scene features; The deviation judgment module is used to compare the meeting scene features with the feature threshold matrix in a weighted manner to calculate the comprehensive meeting deviation. The content generation module is used to generate mode switching suggestions, hierarchical adjustment suggestions, and meeting auxiliary information; The voiceprint recognition module is used to verify the identity of the speaker of the voice control commands and to ensure access to mode switching and advanced functions.

[0018] As a further improvement of the present invention, the mode control module has a built-in multi-mode state machine, which stores configuration parameters for four modes: hosting mode, salon mode, shadow mode, and pause mode. Each mode corresponds to independent audio acquisition permissions, public output permissions, private domain distribution permissions, light indicator parameters, and authorization verification level. The salon mode has a built-in 4-level sub-state machine, which corresponds to four participation levels: roll call, raising hands to speak, conditional interruption, and free participation. Each sub-state machine corresponds to different AI trigger permissions, public domain output permissions, and interaction logic.

[0019] The above technical solution has the following beneficial effects: 1. This invention breaks through the traditional technical logic of "publicly prompting when an anomaly is identified" and adopts a control path of "seamless identification - private domain suggestion - manual confirmation". AI only plays an auxiliary role to provide suggestions to the host without directly interrupting the public space of the meeting. This not only gives full play to the real-time perception capability of AI, but also fully preserves the meeting leadership of the participants, and is suitable for most meeting scenarios.

[0020] 2. It pioneered an asymmetric data flow mechanism of "public domain audio input and private domain targeted output". In shadow mode, the public domain output is completely shut down through a dual blocking scheme of software gain zeroing and hardware path disconnection. At the same time, AI continuously collects and analyzes the meeting content in full, and pushes auxiliary information such as fact correction, opinion summary, and agenda suggestions to authorized terminals only through encrypted channels, which solves the technical contradiction of "requiring AI assistance but not wanting AI to speak publicly and leak information" in highly sensitive meetings.

[0021] 3. Construct a tiered participation system from shadow mode (zero public domain participation), salon mode (4 levels of detailed participation) to host mode (active control throughout the process), with a pause mode as a privacy protection state. It covers all scenarios from silent private domain assistance, passive response, rule-based interruption to active control. The AI ​​participation depth can be dynamically matched according to the meeting type, taking into account the efficiency requirements of formal meetings with strong control and the flexibility requirements of discussion scenarios with weak control.

[0022] 4. By calculating the overall meeting deviation through acoustic and semantic multi-dimensional features, the system automatically identifies the meeting status and intelligently recommends mode / level adjustments, reducing manual operation costs. Mode switching adopts a two-factor authentication mechanism of identity verification and operation confirmation to avoid accidental touches and unauthorized operations, reducing the risk of privacy leakage. At the same time, it follows the principle of gradual switching, prioritizing smooth transitions between adjacent modes / levels to avoid abrupt changes in participation and to match the interactive rhythm of real participants.

[0023] 5. Breaking away from the coarse-grained limitations of traditional devices that only control the recording switch, it achieves independent and decoupled control of multiple channels such as audio acquisition, public audio output, private domain display, and light indication. The permissions of each channel can be configured independently in different modes, which can be finely adapted to different meeting scenarios. At the same time, the status of public domain lights and private domain terminals are synchronized in real time, and participants and hosts can intuitively obtain the current mode and permission status. Attached Figure Description

[0024] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0025] The structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0026] Figure 1 This is a schematic diagram of the hierarchical structure of the system of the present invention; Figure 2 This is a flowchart illustrating the multimodal AI assistant behavior control method of the present invention; Figure 3 A schematic diagram of the meeting scene recognition and pattern suggestion process of this invention; Figure 4 This is a schematic diagram of the public-private domain asymmetric distribution under the shadow mode of the present invention; Figure 5 This is a schematic diagram of the data stream truncation process in the pause mode of the present invention; Figure 6 This is a schematic diagram of the AI ​​hardware terminal structure of the present invention; Figure 7 This is a schematic diagram of the mobile terminal interaction process authorized by the present invention; Figure 8 This is a schematic diagram of the switching process between voiceprint and physical button trigger modes in this invention; Figure 9 This is a schematic diagram of the hierarchical participation control process for the salon mode of the present invention.

[0027] In the picture: 1. Site level; 2. Hardware terminal layer; 21. Control processing unit; 22. Audio acquisition module; 23. Audio output module; 24. Light indicator module; 25. Physical input module; 26. Power management module; 27. Communication module; 28. Storage module; 3. AI processing and control layer; 4. Private domain interaction layer. Detailed Implementation

[0028] In this invention, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this invention.

[0029] A multimodal AI assistant behavior control system based on meeting scenario recognition focuses on human-like intelligent perception and progressive intelligent switching as its core optimization directions. It breaks away from the traditional rigid "on or off" intervention mode of meeting AI, making the AI ​​assistant's participation behavior closer to the interaction logic of a real person attending a meeting. The system constructs a tiered mode system from "completely private assistance" to "fully proactive control" based on the AI ​​assistant's level of public participation: Shadow mode has the lowest participation level, providing implicit support only as the host's personal assistant; Host mode has the highest participation level, capable of independently controlling the meeting and advancing the process; Salon mode has a moderate participation level and is further subdivided into four levels based on the depth of intervention, achieving precise matching between AI participation and meeting scenario, maximizing the value of AI assistance while ensuring the natural rhythm of the meeting.

[0030] like Figure 1 As shown, the system is divided into four layers: field layer 1, hardware terminal layer 2, AI processing and control layer 3, and private domain interaction layer 4. The overall system takes the AI ​​processing and control layer as the core of intelligent decision-making, the hardware terminal layer as the public domain interaction carrier, and the private domain interaction layer as the authorized control entry point.

[0031] Floor 1 is a public space for meetings, serving as the physical setting for participants to speak, discuss, and conduct meeting activities, as well as the physical environment for AI audio acquisition and status output.

[0032] The hardware terminal layer includes AI hardware terminals, which are deployed at the site layer and are responsible for audio acquisition, audio output, lighting indication, and physical input interaction at the meeting site. They are the physical entities of the AI ​​assistant in public meeting spaces.

[0033] The hardware terminal layer 2 includes AI hardware terminals, which are deployed on the field layer 1 to collect audio data from the meeting site and perform audio output, light indication, and physical input interaction.

[0034] The AI ​​processing and control layer 3 includes an AI engine, a mode control module, and a data security distribution module. The AI ​​engine is used to extract features from audio data to obtain meeting scene features, compare and judge the deviation of the meeting status, and generate mode switching suggestions and meeting assistance information.

[0035] The mode control module compares the meeting scene features with a preset meeting type feature threshold matrix. When it determines that the meeting status deviates and continues to time out, it generates a mode switching suggestion and controls the audio output module of the AI ​​hardware terminal to remain silent and not publicly play the mode switching suggestion. After receiving confirmation information, it switches the behavior mode and coordinates the control of the display content of each module of the AI ​​hardware terminal and the authorized mobile terminal. This mode control module follows the principle of "gradual adjustment" and prioritizes a smooth transition between adjacent levels / adjacent modes to avoid the abruptness caused by the jump in participation and to fit the interactive rhythm of real participants.

[0036] The data security distribution module is responsible for data flow access control under different modes: • In Shadow Mode and Salon Mode (lower levels), keep the audio acquisition path of the AI ​​hardware terminal open, block the public audio output path, and send the AI-generated meeting assistance information only to authorized mobile terminals for display via a secure directional channel.

[0037] • In the host mode and salon mode at the higher level, open public audio output channel permissions to support public domain playback of AI voice content.

[0038] • In pause mode, audio acquisition and data transmission are completely cut off, and sensitive local data is cleared.

[0039] The private domain interaction layer 4 includes authorized mobile terminals, which can only be viewed and used by the host or authorized personnel. It serves as a carrier for AI to conduct private interactions with the host, and is used to receive and display mode switching suggestions, level adjustment suggestions, and meeting assistance information. It also receives user confirmation, rejection, and adjustment operations and feeds them back to the mode control module.

[0040] Authorized mobile terminals can be display devices such as smartphones, tablets, smartwatches, and PC clients that are bound to the user. They support functions such as receiving mode suggestions, displaying AI private domain prompts, granting mode / level authorization, viewing meeting summaries and action items, and displaying privacy protection status.

[0041] The on-site layer 1, hardware terminal layer 2, AI processing and control layer 3, and private domain interaction layer 4 are interconnected through three links, namely: Figure 1 Specifically, D1, D2, and D3 in the text are: D1 Main Data Link: Meeting Public Environment → AI Hardware Terminal → AI Engine → Authorized Mobile Terminal, responsible for raw data collection, intelligent analysis and private domain information distribution.

[0042] D2 Authorization Confirmation Link: After the authorized mobile terminal confirms the authorization, it feeds back the user authorization result to AI Processing and Control Layer 3, which is responsible for the feedback of human decision-making and permission verification.

[0043] D3 Control Synchronization Link: AI Processing and Control Layer 3 synchronizes the mode control results to Hardware Terminal Layer 2, controlling data acquisition, external playback, lighting, and button response, and is responsible for the implementation of intelligent decisions.

[0044] AI hardware terminals can be desktop smart hardware, conference auxiliary equipment, portable intercom-style AI interactive devices, smart recording devices, or other terminals with audio acquisition and status output capabilities.

[0045] Combination Figure 6 As shown, the AI ​​hardware terminal includes a control processing unit 21, an audio acquisition module 22, an audio output module 23, a light indicator module 24, a physical input module 25, a power management module 26, a communication module 27, and a storage module 28. Among them: The audio acquisition module 22 consists of a single microphone or multi-microphone array distributed on the field layer 1, used to acquire audio data at the conference site; for example, a 4-microphone ring array is used to support 360° omnidirectional sound pickup and has beamforming, noise reduction and dereverberation capabilities.

[0046] The audio output module 23 consists of speakers, amplifiers or directional audio output devices distributed on the field layer 1, including full-range speakers and audio power amplifiers, supports volume adjustment and hardware-level on / off control, and is used to publicly play audio content. When in shadow mode, low-level salon mode and pause mode, the output is blocked by the data security distribution module. The light indicator module 24 uses devices such as RGB LEDs to indicate the current behavior mode and AI's willingness to speak through different lighting effects and flashing rhythms. For example, constant light corresponds to stable mode, slow flashing corresponds to AI requesting to speak, and fast flashing corresponds to warning prompts. The physical input module 25 uses buttons, knobs, or touch areas to receive user mode switching and pause trigger operations; for example, it uses a single multi-function knob button to support single-click, double-click, long-press, and rotation operations, which correspond to different mode switching and function control.

[0047] Communication module 27: Supports Wi-Fi, Bluetooth 5.0 or cellular communication modules for data transmission with the cloud and mobile devices.

[0048] Storage module 28: Used to store local configuration, temporary audio cache, and firmware program.

[0049] Power management module 26: Supports Type-C power supply and built-in battery power supply, ensuring portable use.

[0050] The control processing module 21 uses a chip such as MCU, SoC, DSP or edge AI, and adopts an MCU+NPU architecture to be responsible for local audio preprocessing, device status control and edge AI inference, ensuring low latency response.

[0051] Furthermore, the AI ​​engine can be deployed on cloud servers, local servers, local hardware terminals, local mobile terminals, and combinations thereof. It is used to extract multi-dimensional features from audio data, accurately identify meeting scenario states, calculate meeting state deviations, and generate mode switching suggestions and meeting assistance information. Its built-in modules include: Speech recognition module: Converts audio data into text content, supports multi-person voiceprint separation and role labeling.

[0052] Feature extraction module: Extracts acoustic and semantic features to obtain multi-dimensional meeting scene features, including speaking frequency, tone fluctuation amplitude, sound power spectral density, frequency of multiple people speaking overlapping, silence duration, semantic divergence, keyword deviation, emotional intensity, topic matching degree, etc.

[0053] Deviation judgment module: compares the meeting scene features with the feature threshold matrix of the preset meeting type, calculates the comprehensive meeting deviation, and judges the degree of matching between the current meeting status and the preset target.

[0054] Content generation module: Generates content such as mode switching suggestions, meeting support information, speech drafts, and correction prompts based on the meeting context.

[0055] Voiceprint recognition module: Verifies the identity of the speaker when issuing voice control commands, and ensures access to mode switching and advanced functions. Furthermore, the authorized mobile terminal can be a smartphone, tablet, smartwatch, PC client, or other display device bound to the user. It is used for: receiving mode suggestions; displaying AI privacy prompts; granting mode authorization; viewing meeting summaries, action items, risk warnings, and corrective suggestions; and displaying privacy protection status.

[0056] Furthermore, the mode control module incorporates a multi-mode state machine, which stores configuration parameters for four modes: Host Mode, Salon Mode, Shadow Mode, and Pause Mode. Each mode corresponds to independent audio acquisition permissions, public output permissions, private domain distribution permissions, light indicator parameters, and authorization verification levels. The Salon Mode, in particular, incorporates a four-level sub-state machine, corresponding to different levels of AI participation.

[0057] The multimodal AI assistant behavior control system of the present invention can be mapped to a user client, and the relationship between the two is as follows:

[0058] The multimodal AI assistant behavior control system of this invention takes the degree of direct participation of the AI ​​assistant in the public domain of the meeting as the core dimension and constructs a four-level gradient mode system, from low to high: shadow mode (lowest participation), salon mode (medium participation, including 4 sub-levels), host mode (highest participation), and a pause mode as a privacy protection state.

[0059] 1.1 Shadow Mode: Lowest level of participation In shadow mode, the AI ​​assistant remains completely invisible in public meeting spaces, providing implicit support as the host's personal assistant.

[0060] Audio capture: Continuously enabled, capturing all audio from the meeting to ensure AI fully understands the meeting content.

[0061] Public domain output: Completely blocked, using a combination of software gain zeroing and hardware path disconnection, resulting in no sound output from the public speaker.

[0062] Lighting status: Low-visibility lighting effects (such as yellow breathing lights) are used to indicate only the equipment's operating status and do not convey meeting-related prompts.

[0063] AI behavior: All auxiliary information is sent to authorized mobile terminals only through encrypted targeted channels, including fact correction, historical minutes links, data source prompts, risk warnings, suggested questions, agenda advancement suggestions, and summaries of the other party's viewpoints. The host can choose whether to adopt these suggestions and when to make them public.

[0064] Applicable scenarios: Sensitive meetings, business negotiations, management decisions, and other scenarios that require AI assistance but do not want participants to be aware of AI intervention.

[0065] 1.2 Moderation: Highest level of participation, proactive control of the meeting. In the moderator mode, the AI ​​assistant has the highest public participation authority, can take the initiative to advance the process and publicly intervene in the meeting status as the meeting moderator, and has the highest participation rate.

[0066] Audio capture: Continuously enabled, real-time recognition of speech content, duration, and speaker's role.

[0067] Public domain output: Completely open, AI can publicly output voice prompts through speakers.

[0068] Lighting status: Use distinctive lighting effects (such as solid blue) to clearly indicate that the AI ​​is currently in an active control state.

[0069] AI behavior: Based on the preset agenda, speaking time, and meeting objectives, it proactively issues public prompts, including detecting speaking timeouts, prompting to move to the next topic, extracting action items, identifying unresolved issues, and guiding the direction of discussion, without requiring secondary human confirmation to intervene.

[0070] Applicable scenarios: Formal meetings with clear agendas and time constraints, requiring strong control, such as weekly meetings, project reviews, and performance review meetings.

[0071] 1.3 Salon Mode: Moderate participation level, dynamically adapted to different levels. Salon mode is suitable for meetings with weak control, such as discussion sessions and creative meetings. The AI ​​assistant's participation level is moderate and is subdivided into 4 levels from low to high based on the level of intervention. It can be automatically recommended and switched through intelligent perception, or it can be manually adjusted by the host to achieve dynamic matching between AI participation and meeting atmosphere.

[0072] Level 1: Roll Call Response - Manually Triggered, Passive Response Triggering entity: People; triggered only when participants take active action.

[0073] Triggering method: Participants can explicitly call on the AI ​​to ask questions through physical buttons, wake words, or other explicit operations.

[0074] AI behavior: It only provides answers to the current question, stops once the answer is given, does not actively extend the discussion, does not interfere with the meeting process, and responds completely passively.

[0075] Output format: The question is answered directly via public voice, without any additional prompts.

[0076] Example scenario: A participant presses a button to ask "What were the action items in the previous meeting?" The AI ​​directly reads the corresponding content and then returns to silence.

[0077] Level 2: Raise your hand to speak – AI proposes, human confirms. Triggering entity: AI, which actively determines the necessity of speaking.

[0078] Triggering method: When the AI ​​detects a valuable supplementary point, correction point, or guiding point, it will indicate this by flashing a light (such as a slow flashing green light) as if raising its hand. The AI ​​can only speak publicly after the host confirms permission through the terminal or a button.

[0079] AI behavior: It only generates speaking suggestions and will not speak publicly without human confirmation; the host always has the right to veto and can ignore or reject the AI's speaking requests.

[0080] Output format: Public domain light prompt application + private domain push speech draft, public domain voice output after confirmation.

[0081] Example scenario: AI detects factual discrepancies in the discussion content, flashes a light slowly to alert the host, and simultaneously pushes corrective content to the terminal. After the host confirms, AI publicly corrects the information.

[0082] Level 3: Conditional Intervention – Rules Pre-set, Proactive Intervention Within Rules Triggering entity: AI, which actively triggers the event within the scope of manually preset rules.

[0083] Triggering method: The host predefines the scenario rules that allow AI to interrupt (such as going off-topic for more than 3 minutes, silence for more than 60 seconds, speaking timeout, factual errors, etc.). AI detects the meeting status in real time, and when the preset conditions are met, it can directly interrupt and intervene in the public domain without real-time confirmation.

[0084] AI behavior: Intervention is strictly performed within the preset rules, without actively expanding the topic beyond the rule boundaries, and the listening state is immediately restored after intervention.

[0085] Output format: When the conditions are met, a public voice prompt is given, and the terminal simultaneously displays the prompt information.

[0086] Example scenario: With a preset "remind if discussion deviates from the topic for more than 3 minutes", once the AI ​​detects that the discussion has deviated from the original topic, it will directly prompt with a voice message: "The current discussion has deviated from the original topic. Do you want to return to the main line?"

[0087] Level 4: Free Participation – Independent Judgment and Equal Participation Triggering entity: AI, which autonomously determines when to speak.

[0088] Triggering method: As an equal participant, AI autonomously judges the pace of the discussion and speaks naturally, asks questions, adds viewpoints, and extends the discussion during the dialogue, just like a real person, without the need for preset rules or manual confirmation.

[0089] AI behavior: It can proactively ask questions, supplement background information, summarize discussion viewpoints, and propose new ideas, with a level of participation consistent with that of real participants, making it suitable for open discussion scenarios.

[0090] Output format: The public voice naturally participates in the discussion, and the terminal synchronously displays the content of the speech and the reference.

[0091] Example scenario: During creative brainstorming sessions, AI proactively proposes new creative directions or supplements relevant case studies during breaks in the discussion, integrating itself into the overall discussion rhythm.

[0092] 2.4 Pause Mode: Privacy Protection Status The pause mode is a privacy-protected state with no AI involvement; its core function is to cut off all sensitive data streams.

[0093] Audio Acquisition: Stop acquisition, or stop uploading audio data to the AI ​​engine and the cloud.

[0094] Public Area Output: Speakers completely off.

[0095] Light status: Solid red, clearly indicating that the current mode is under privacy protection.

[0096] AI Action: Stop all analysis and output, clear the local pending audio queue, temporary text and auxiliary content to eliminate the risk of data residue.

[0097] Applicable scenarios: Scenarios where highly sensitive topics such as salaries, personnel, and trade secrets are discussed.

[0098] The switching states for the four modes—hosting mode, salon mode, shadow mode, and pause mode—are as follows:

[0099] like Figure 2 As shown, a multimodal AI assistant behavior control method based on meeting scene recognition includes the following steps: S1. Multi-dimensional intelligent perception to extract meeting scene features Audio data from the meeting is collected by an AI hardware terminal, and the AI ​​engine extracts multi-dimensional features to obtain complete meeting scene characteristics.

[0100] The characteristics of the meeting scene cover two major dimensions: acoustics and semantics. Acoustic features include: speaking frequency, single-person speaking duration, pitch fluctuation amplitude, sound power spectral density, frequency of multiple people speaking overlapping, and duration of silence, as well as semantic features such as semantic divergence, keyword deviation, emotional intensity, topic matching degree, and factual deviation degree.

[0101] S2. Intelligent comparison and judgment to identify the degree of deviation in meeting status. The extracted meeting scenario features are compared with the feature threshold matrix corresponding to the preset meeting type to determine the degree of deviation between the current meeting status and the preset meeting goal, and at the same time, the suitability of the current AI participation is evaluated.

[0102] The specific judgment logic includes: 1. Obtain the meeting topic vector V_topic and the preset participation level corresponding to the preset meeting type; 2. Semantically encode the speech content within the current sliding time window to obtain the current semantic vector V_current; 3. Calculate the semantic divergence F_semantic=1-cosine_similarity(V_topic,V_current), and simultaneously calculate the deviation components of the acoustic dimension; 4. Calculate the overall meeting deviation degree D by weighting the deviation components of each dimension: D=w1*F_speech+w2*F_pitch+w3*F_semantic+w4*F_overlap+w5*F_silence in: • F_speech: Represents abnormal speech frequency components; • F_pitch: Represents the pitch fluctuation or emotional intensity component; • F_semantic: Represents the semantic divergence component; • F_overlap: Indicates multiple people speaking at the same time or overlapping speech. • F_silence: Represents the duration of silence; • w1 to w5 are weights, which can be adjusted according to the meeting template; 5. Based on the overall deviation, meeting stage, and atmosphere characteristics, determine whether the current AI participation mode / level is suitable and generate adjustment suggestions.

[0103] S3. Generate switching suggestions and silently push private domain messages. If it is determined that the current mode / level is not compatible with the meeting status and the deviation duration exceeds a preset threshold, a corresponding mode switching or level adjustment suggestion is generated; at the same time, the audio output module of the AI ​​hardware terminal is kept silent and does not make public sounds through the hardware speaker, and the suggestion is pushed to the authorized mobile terminal through an encrypted channel for the host to make a decision.

[0104] S4. Receive confirmation information and perform intelligent switching. After receiving confirmation information from an authorized mobile terminal, voice control command, or physical input module, switch the behavior mode or sub-level of the AI ​​hardware terminal.

[0105] • Cross-mode switching follows a gradual principle, prioritizing a step-by-step transition between Shadow Mode, Salon Mode, and Host Mode; • Salon mode supports smooth adjustment of ascending and descending order when switching between internal levels; • When switching to shadow mode, pause mode, or higher-level permissions, two-factor authentication and operation confirmation conditions must be met: authentication includes voiceprint matching and terminal account verification; operation confirmation includes terminal touch and physical button operation.

[0106] S5. Execute the corresponding behavioral logic according to the target pattern. After the switch is complete, each module will run according to the permission rules of the target mode: 1. Shadow Mode: Keeps audio acquisition running continuously, blocks public audio output channels, and sends conference support information only to authorized mobile terminals for display via a secure directional channel.

[0107] 2. Hosting Mode: With the audio output module open, AI proactively outputs meeting control prompts based on the meeting agenda and speaking duration to advance the meeting process.

[0108] 3. Salon Mode: Executes the corresponding interaction logic based on the current sub-level: Level 1: Waiting for manual roll call to trigger; once triggered, the response will stop. • Level 2: AI continuously monitors the meeting content and, when it determines that someone has something to say, it signals them by raising their hand with a light. After human confirmation, the speaker can then speak publicly. • Level 3: AI monitors triggering conditions according to preset rules and proactively prompts for public domain interaction when the conditions are met; Level 4: AI autonomously determines when to speak and naturally participates in the discussion as an equal participant.

[0109] 4. Pause Mode: Stops audio acquisition or data upload, clears the queue of pending sensitive data, and displays the privacy protection status synchronously with the terminal via lights.

[0110] S6, end-to-end collaborative control, state synchronization Based on the switched behavior pattern, the audio acquisition module, audio output module, and light indicator module of the AI ​​hardware terminal, as well as the display content and function access of the authorized mobile terminal, are coordinated and controlled to ensure that the status of public domain hardware and private domain terminal are synchronized in real time.

[0111] Figure 3 A meeting scene recognition and pattern suggestion process is disclosed, and its specific steps are as follows: S201: Start the meeting support process. The user selects the corresponding meeting template (such as weekly meeting template, creative salon template, negotiation template) on the authorized mobile terminal. The system loads the feature threshold matrix, default working mode and default participation level corresponding to the template.

[0112] S202: Read the parameters corresponding to the preset meeting type, including topic keywords, feature threshold, time window size, deviation duration threshold, default participation level, adjustable level range, etc.

[0113] S203: The AI ​​hardware terminal's multi-microphone array continuously collects audio data from the meeting, which is then pre-processed by noise reduction and de-reverberation before being uploaded to the AI ​​engine.

[0114] S204: The AI ​​engine performs multi-dimensional intelligent perception: at the acoustic level, it extracts features such as speaking frequency, single-person speaking duration, frequency of multiple people speaking overlapping, tone fluctuation, and duration of silence; at the semantic level, it uses speech recognition to transcribe text and calculates features such as semantic divergence, keyword deviation, emotional intensity, and factual deviation, integrating them into multi-dimensional meeting scene features.

[0115] S205: The deviation judgment module calculates the comprehensive meeting deviation by weighting the current scene features with the preset threshold matrix, and at the same time evaluates the matching degree between the current AI participation and the meeting status.

[0116] S206: Determine if mode / level adjustment is needed: If the overall deviation exceeds the threshold and the duration exceeds the target, or the participation matching degree is lower than the threshold, generate corresponding adjustment suggestions; otherwise, return to continuous data collection and analysis.

[0117] S207: The mode control module issues a command to keep the speakers silent and not play any prompts in public spaces to avoid interrupting the meeting.

[0118] S208: Generating mode / level switching suggestions (such as "It is detected that the meeting is going off-topic more and more, so it is recommended to upgrade the salon mode to level 3 and enable automatic reminders for going off-topic"), which are pushed to authorized mobile terminals through an encrypted channel.

[0119] S209: Wait for user confirmation. If an authorized terminal confirmation command, voiceprint matching command, or physical button confirmation is received, update the state machine and synchronously control the switching between hardware and terminal. If the user refuses or there is no response after timeout, maintain the current mode.

[0120] like Figure 9 The salon model tiered participation control process is shown below: S601: The system is in salon mode. Initialize the corresponding permissions and triggering logic according to the currently set level.

[0121] S602: The AI ​​engine continuously analyzes meeting audio in real time and identifies triggering conditions. Level 1: Listens for wake words and key presses, and only responds to explicit triggers; Level 2: Real-time identification of valuable points in speech (factual bias, awkward silence, signs of going off-topic) to determine whether the "raise hand" threshold has been reached; Level 3: Match preset interruption rules and determine whether the triggering conditions are met; Level 4: Analyze the pace and direction of the discussion, and independently determine when to speak.

[0122] S603: Upon reaching the corresponding trigger condition, the output is executed according to the hierarchical rules: Level 1: Directly generate response content and output public domain voice; Level 2: Control the light to flash slowly to send a "raise hand" signal, and push the speech draft to the terminal for human confirmation; if confirmed, output to the public domain; otherwise, cancel. Level 3: Directly generate prompt content, output public domain voice, and simultaneously record on the terminal; Level 4: Generate speech content and output it naturally in the public domain during dialogue breaks, integrating it into the discussion rhythm.

[0123] S604: After outputting, resume listening mode, continuously monitor the meeting status, and dynamically recommend adjustments to the hierarchy.

[0124] like Figure 4 The shadow mode public-private domain asymmetric distribution process shown: In the shadow mode, an asymmetric data flow of "public input - centralized processing - private domain output" is formed, realizing the decoupling of AI capabilities and public output, which is specifically divided into three stages: 1. Public domain input stage: The multi-microphone array of the AI ​​hardware terminal remains on, continuously collecting all speaking audio in the public space of the meeting. After preprocessing, it is transmitted to the AI ​​engine to ensure that the AI ​​has complete environmental perception capabilities.

[0125] 2. AI Processing Stage: The AI ​​engine performs real-time speech recognition, semantic understanding, opinion extraction, and fact verification on the input audio. Combining the knowledge base and historical meeting data, it generates corresponding meeting support information, including but not limited to: fact correction prompts, historical minutes links, data source verification, risk point prompts, question suggestions, agenda progression suggestions, and summaries of the other party's viewpoints.

[0126] 3. Output control: The data security distribution module implements two levels of blocking: • Software-level blocking: Set the speaker output gain to 0, prohibit all TTS audio and prompt tones from entering the playback queue, and the state machine logic intercepts all common external playback commands.

[0127] • Hardware-level blocking: Turn off the audio power amplifier, or physically disconnect the speaker's signal path using an analog switch / relay to completely prevent sound leakage at the hardware level.

[0128] At the same time, an end-to-end encrypted directional channel is established to send auxiliary information only to authorized mobile terminals, which can only be viewed by the host.

[0129] The data path in shadow mode is: input from a public meeting environment → AI analysis and processing → authorized mobile terminal private domain output; public output from local speakers is blocked.

[0130] like Figure 5 A pause mode data stream truncation process is disclosed, with the following specific steps: S301: Receives pause mode trigger command, including long press of hardware button, pause on mobile device, voice command trigger, and automatic trigger when the system detects highly sensitive keywords.

[0131] S302: The mode control module updates the state machine to pause mode and synchronously sends truncation commands to each module.

[0132] S303: Controls the audio acquisition module to stop working, or stops uploading audio data to the AI ​​engine and the cloud, cutting off the data source.

[0133] S304: Clear the pending audio queue, temporary text data and generated auxiliary content in the local cache to eliminate the risk of local data residue.

[0134] S305: Turn off the audio output module, disable the speakers, and stop all public domain outputs.

[0135] S306: Controls RGB lighting to switch to exclusive lighting effects such as constant red, providing a clear indication of the current privacy protection status in the physical space.

[0136] S307: Authorized mobile terminals will simultaneously display a message stating "Data stream has been interrupted, privacy protection in progress," clearly informing the host that no data is currently being collected or uploaded.

[0137] S308: Continuously waits for the recovery command. After receiving the recovery command verified by two factors, it exits the pause mode and restores to the working mode selected by the user.

[0138] Figure 7 The authorization process for mobile terminal interaction is shown below, with the specific steps as follows: S401: Receive the mode switching suggestion data packet sent by the mode control module, which includes the basis for meeting deviation, suggested mode and expected effect; S402: Display suggested content in a pop-up window on the authorized mobile terminal interface, which can trigger a vibration alert simultaneously; S403: The user confirms or rejects the action via touch. If the user rejects the action or does not perform any action within the preset timeout period, the system will maintain the current mode. S404: After user confirmation, the terminal sends an encrypted authorization command with identity verification; S405: After receiving the update receipt, the terminal switches to the corresponding mode display interface to show the function entry and status indicators.

[0139] The authorized mobile terminal can be equivalently replaced by a smartwatch, tablet, AR glasses, etc., and the interaction method can be replaced by voice confirmation or gesture confirmation, all of which are within the protection scope of this invention.

[0140] Figure 8 The process for switching between voiceprint and physical button trigger modes is shown below: S501 receives the mode switching trigger signal; S502 determines the trigger source, which is divided into two categories: voice command and physical button; if it is a voice trigger, S503 collects the command, S504 extracts the voiceprint and compares it with the preset controller template, and after the match is successful, S505 parses the target mode; if it is a button trigger, S506 receives the button code, S507 parses the corresponding operation command; S508 updates the behavior mode state machine parameters; S509 synchronously sends control commands to the hardware terminal and mobile terminal to realize the synchronous switching of audio path, lighting effects and APP interface.

[0141] The aforementioned physical buttons can be equivalently replaced by touch buttons, toggle switches, knob arrays, and other input structures, all of which fall within the equivalent scope of this invention.

[0142] Example 1: Automatic Pattern Suggestion in Project Weekly Meeting Scenario A project team started a weekly meeting. The host selected a "weekly meeting template" in the mobile app, and the system defaulted to salon mode level 2 (raising hands to speak). The green light on the AI ​​hardware terminal remained on.

[0143] Twenty minutes into the meeting, the discussion gradually deviated from the pre-set agenda, with an increased frequency of multiple speakers interfering with each other. The AI ​​engine calculated a semantic divergence of 0.72, exceeding the threshold of 0.6, and this divergence lasted for 4 minutes, resulting in an overall deviation exceeding the threshold.

[0144] The system does not emit sound through the speaker, but only pushes a notification to the host's mobile phone: "The meeting has been detected to have deviated from the current topic and the discussion among multiple people has become rambling. It is recommended to upgrade to salon level 3 and enable automatic reminders for going off-topic, or switch to host mode to bring the agenda back on track."

[0145] The host selects to switch to the host mode. After clicking to confirm, the system completes the switch, the hardware lights turn solid blue, and the AI ​​announces through the speaker: "The discussion time for the current topic is up. Please move on to the next topic: Version Iteration Plan." The agenda progress and identified action items are displayed simultaneously in the App.

[0146] Example 2: Application of Shadow Mode in Business Negotiation Scenarios A company was conducting business negotiations. The host activated the shadow mode in the app beforehand. After voiceprint verification and button confirmation, the host entered the shadow mode. The yellow breathing light on the hardware terminal remained on, and the speaker was physically disconnected.

[0147] During the negotiations, the other party's representative mentioned that "the industry growth rate was about 15% last year." The AI ​​engine compared the data with its internal knowledge base in real time and identified that the data did not match the official statistics of 12.8%, and then generated a correction prompt.

[0148] The notification was sent only to the host's mobile app via an encrypted channel and displayed as: "The industry growth rate data mentioned by the other party is questionable. The official statistics show that the industry growth rate in 2025 was 12.8%. You can click to view the data source."

[0149] After reviewing the data, the host naturally corrected it in subsequent remarks. The AI ​​remained silent throughout the process, and the other participants were unaware of its intervention. This approach ensured both the negotiating advantage and avoided the embarrassment of publicly correcting the error.

[0150] Example 3: Silent Assistance in a Creative Salon Setting A design team is conducting a creative brainstorming session. The system defaults to salon mode level 2 (raising hands to speak).

[0151] • Initial silence: If the AI ​​detects an 80-second silence, reaching the threshold for raising hands to speak, it will slowly flash RGB lights to indicate this, and simultaneously generate three draft suggestions for guiding the discussion in the host's tablet app. After seeing the light prompts, the host will check the app, select one of the suggestions, and verbally guide the discussion; the AI ​​will not speak publicly directly.

[0152] • Active discussion begins midway through the meeting: The moderator manually upgrades the level to level 4 (free participation). As an equal participant, the AI ​​actively supplements relevant design cases and proposes creative extensions during the discussion breaks, naturally integrating into the discussion rhythm.

[0153] • The discussion started to stray off-topic: The AI ​​detected that the topic was gradually deviating from the creative theme. Since we were currently at level 4, it directly prompted us with a voice message, "We can return to the core design direction of this discussion and continue the discussion," guiding us back to the main thread.

[0154] Example 4: Conditional Interruption Application in Technical Review Scenarios A team is conducting a technical solution review. The moderator sets up a salon mode at level 3 (conditional interruption), with preset rules such as "reminding if speaking for more than 10 minutes or if a factual error is found".

[0155] During the review process, a participant made a discrepancy in their description of a technical indicator. The AI ​​detected the factual error in real time and directly prompted through the loudspeaker: "For further explanation, the industry standard value for this parameter is XX, which may be used for reference."

[0156] Subsequently, when a speaker's presentation on a single topic reached 10 minutes, the AI ​​automatically prompted: "Your current presentation time has reached 10 minutes. Would you like to allow time for other colleagues to supplement?" Throughout the process, the AI ​​intervened strictly within the preset rules, ensuring both review efficiency and avoiding excessive disruption to the discussion's pace.

[0157] Example 5: Pause Mode Protection for Sensitive Issues During the meeting, highly sensitive topics such as employee salary adjustments were discussed. The host pressed and held the physical button on the AI ​​hardware terminal for 3 seconds. After the system underwent both button operation and account identity verification, it switched to pause mode.

[0158] The hardware indicator light immediately turns solid red, the microphone stops capturing audio, the local cache is cleared, the speaker is turned off, and the mobile app displays "Privacy protection in progress, all audio capture and uploading have been stopped."

[0159] Participants can clearly see that there is no recording when they see the red light, and they can safely discuss sensitive topics. After the discussion ends, the host clicks "restore" in the app. After voiceprint verification, the system reverts to the previous salon mode level and continues normal meeting support.

[0160] In summary, the embodiments of the present invention achieve the following technical effects: The multimodal AI assistant behavior control solution proposed in this invention, based on meeting scene recognition, breaks through the inherent technical logic of traditional meeting AI's "public intervention upon identifying anomalies." It pioneers a non-intrusive control path of "unobtrusive meeting status recognition - private domain push switching suggestions - intervention after manual confirmation," fundamentally preventing AI from publicly interrupting the meeting rhythm and fully preserving the participants' control over the meeting. At the same time, it innovatively constructs an asymmetric data stream distribution architecture of "public domain audio input and private domain directional output" in shadow mode, breaking the binary limitation of existing devices' "synchronous switching of acquisition and public playback," and solving the technical contradiction of balancing AI assistance needs and privacy protection needs in highly sensitive meeting scenarios. The solution's accompanying four-mode decoupled state machine, hierarchical two-factor mode switching verification mechanism, and full-link data stream truncation privacy protection mechanism further achieve a balance between refined hardware and software collaborative control and security and ease of use, and comprehensively expand the scenario adaptation boundaries of meeting AI devices.

[0161] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0162] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0163] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal AI assistant behavior control method based on meeting scene recognition, characterized in that, Includes the following steps: S1. Collect audio data from the meeting venue through AI hardware terminals and extract meeting scene features; S2. Compare the meeting scene features with the feature threshold matrix corresponding to the preset meeting type to determine whether the current meeting status deviates from the preset meeting type; S3. If it is determined that there is a deviation and the duration exceeds a preset threshold, a mode switching suggestion is generated; the audio output module of the AI ​​hardware terminal is controlled to remain silent and not publicly play the mode switching suggestion, and the mode switching suggestion is pushed to the authorized mobile terminal through an encrypted channel; S4. After receiving confirmation information from the authorized mobile terminal, voice control command, or physical input module, switch the behavior mode of the AI ​​hardware terminal; the behavior mode includes shadow mode, salon mode, host mode, and pause mode, wherein the salon mode includes multiple participation sub-levels; S5. After switching to the corresponding behavior mode, execute the matching behavior logic: When switching to shadow mode, the audio acquisition module of the AI ​​hardware terminal continues to acquire audio data, while blocking the public audio output path of the AI ​​hardware terminal. The meeting assistance information generated by the AI ​​engine is sent only to the authorized mobile terminal for display through a secure directional channel. When switching to salon mode, the corresponding triggering and output logic is executed according to the permission rules of the current sub-level; When switching to host mode, grant permissions to the audio output module, and the AI ​​will proactively and publicly output meeting control prompts based on the meeting agenda and speaking duration recognition results. When switching to pause mode, audio acquisition or audio data upload is stopped, and the queue of pending sensitive data is cleared. S6. Based on the switched behavior mode, coordinate the control of the audio acquisition module, audio output module, light indicator module of the AI ​​hardware terminal, and the display content of the authorized mobile terminal.

2. The multimodal AI assistant behavior control method based on meeting scene recognition according to claim 1, characterized in that, The meeting scenario features include at least one of the following: speaking frequency, single-person speaking duration, tone fluctuation amplitude, sound power spectral density, frequency of multiple people speaking in overlapping speech, duration of silence, semantic divergence, keyword deviation, emotional intensity, topic matching degree, and factual deviation degree.

3. The multimodal AI assistant behavior control method based on meeting scene recognition according to claim 1, characterized in that, Step S2 determines whether the current meeting status deviates from the preset meeting type, specifically including: Obtain the meeting topic vector V_topic and the preset participation level corresponding to the preset meeting type; Semantically encode the speech content within the current sliding time window to obtain the current semantic vector V_current; Calculate the semantic divergence F_semantic=1-cosine_similarity(V_topic,V_current), and extract multiple deviation components of the acoustic dimension. The weighted average deviation of each dimension is used to calculate the overall meeting deviation degree D. The calculation formula is as follows: D=w1F_speech+w2F_pitch+w3F_semantic+w4F_overlap+w5*F_silence Where F_speech is the abnormal speech frequency component, F_pitch is the tone fluctuation or emotion intensity component, F_semantic is the semantic divergence component, F_overlap is the overlapping speech of multiple people component, F_silence is the silence duration component, and w1 to w5 are configurable weights. When the deviation of the overall meeting exceeds a preset threshold and the duration exceeds a preset time threshold, it is determined that the current meeting status deviates from the preset meeting type.

4. The multimodal AI assistant behavior control method based on meeting scene recognition according to claim 1, characterized in that, The salon mode is divided into four sub-levels according to the depth of AI participation, from low to high. The execution logic of each level is as follows: Level 1 roll call response: Only responds to explicit triggering actions of participants. After being triggered, it provides a public voice response to the question and returns to silence after the response is completed. Level 2: Raise your hand to speak: AI identifies valuable points in the meeting content in real time. When the trigger threshold is reached, a public domain prompt signal is issued through the light indicator module. At the same time, the speech draft is pushed to authorized mobile terminals. After manual confirmation, the speech is then output to the public domain. Level 3 Conditional Interruption: AI monitors the meeting status in real time according to preset interruption rules. When preset trigger conditions are met, no real-time confirmation is required, and a public voice prompt is given directly. Level 4: Free Participation: As an equal participant, AI autonomously judges the pace of discussion and the timing of its speech, naturally outputting its content in the public domain during dialogue breaks and participating in the meeting discussion.

5. The multimodal AI assistant behavior control method based on meeting scene recognition according to claim 1, characterized in that, Step S5 involves blocking the public audio output path of the AI ​​hardware terminal, specifically including at least one of the following methods: The software layer sets the audio output gain to zero, preventing all TTS audio and prompts from entering the speaker playback queue; Intercept all common external commands within the state machine logic; Turn off the audio power amplifier; The signal path of the speaker is physically disconnected by analog switches or relays.

6. The multimodal AI assistant behavior control method based on meeting scene recognition according to claim 1, characterized in that, In step S4, when switching to shadow mode, pause mode or high-privilege participation level, two-factor authentication must be performed where both the identity verification condition and the operation confirmation condition are met simultaneously. The identity verification conditions include at least one of voiceprint recognition matching the preset controller's identity and authorized mobile terminal account verification. The operation confirmation conditions include at least one of authorized mobile terminal touch confirmation and AI hardware terminal physical button operation; Mode switching follows a gradual adjustment principle. Cross-mode switching prioritizes a gradual transition between Shadow Mode, Salon Mode, and Host Mode. Within Salon Mode, sub-levels support smooth bidirectional adjustment in ascending and descending order.

7. A multimodal AI assistant behavior control system based on meeting scene recognition, characterized in that, This includes AI hardware terminals, AI engines, authorized mobile terminals, mode control modules, and data security distribution modules; The AI ​​hardware terminal is used to collect audio data at the meeting venue and perform audio output, light indication, and physical input interaction. The AI ​​engine is used to extract features from audio data to obtain meeting scene features, compare and judge the deviation of the meeting status, and generate mode switching suggestions and meeting assistance information. The mode control module is used to compare the meeting scene features with the feature threshold matrix of the preset meeting type. When it is determined that the meeting state deviates and continues to time out, it generates a mode switching suggestion and controls the audio output module of the AI ​​hardware terminal to remain silent and not publicly play the mode switching suggestion. After receiving confirmation information, it switches the behavior mode and coordinates the control of the display content of each module of the AI ​​hardware terminal and the authorized mobile terminal. The behavior modes include shadow mode, salon mode, host mode and pause mode, wherein the salon mode contains multiple levels of participation sub-levels. The data security distribution module is used to manage data flow permissions in different modes: in the lower sub-levels of shadow mode and salon mode, the audio acquisition channel of the AI ​​hardware terminal is kept open, the public audio output channel of the AI ​​hardware terminal is blocked, and the meeting assistance information generated by the AI ​​engine is sent only to authorized mobile terminals for display through a secure directional channel; in the higher sub-levels of host mode and salon mode, the public audio output channel permission is opened. In pause mode, audio acquisition and data transmission are completely cut off from the entire link. The authorized mobile terminal is used to receive and display mode switching suggestions and meeting assistance information, receive user confirmation, rejection and adjustment operations and feed them back to the mode control module.

8. The multimodal AI assistant behavior control system based on meeting scene recognition according to claim 7, characterized in that, The AI ​​hardware terminal includes a control processing unit, an audio acquisition module, an audio output module, a light indicator module, a physical input module, a communication module, and a storage module. The audio acquisition module consists of a single microphone or a multi-microphone array and is used to acquire audio data at the meeting. The audio output module is used to publicly play audio content, and its output is blocked by the data security distribution module in the lower sub-levels of shadow mode, pause mode and salon mode. The light indicator module is used to indicate the current behavior pattern, AI's willingness to speak, and warning status through different lighting effects and flashing rhythms; The physical input module is used to receive user mode switching, pause triggering, and level adjustment operations; The control processing unit is used for local audio preprocessing, device status control, and edge AI inference.

9. The multimodal AI assistant behavior control system based on meeting scene recognition according to claim 7, characterized in that, The AI ​​engine includes a speech recognition module, a feature extraction module, a deviation judgment module, a content generation module, and a voiceprint recognition module. The speech recognition module is used to convert audio data into text content and supports multi-person voiceprint separation and role labeling; The feature extraction module is used to extract acoustic and semantic features to obtain multi-dimensional meeting scene features; The deviation judgment module is used to compare the meeting scene features with the feature threshold matrix in a weighted manner to calculate the comprehensive meeting deviation. The content generation module is used to generate mode switching suggestions, hierarchical adjustment suggestions, and meeting auxiliary information; The voiceprint recognition module is used to verify the identity of the speaker of the voice control commands and to ensure access to mode switching and advanced functions.

10. The multimodal AI assistant behavior control system based on meeting scene recognition according to claim 7, characterized in that, The mode control module has a built-in multi-mode state machine, which stores configuration parameters for four modes: host mode, salon mode, shadow mode, and pause mode. Each mode corresponds to independent audio acquisition permissions, public output permissions, private domain distribution permissions, light indicator parameters, and authorization verification level. The salon mode has a built-in 4-level sub-state machine, which corresponds to four participation levels: roll call, raising hands to speak, conditional interruption, and free participation. Each sub-state machine corresponds to different AI trigger permissions, public domain output permissions, and interaction logic.

Images