Method and device for implementing offline AI assistant based on video conference terminal and medium

Abstract

The application provides an offline AI assistant implementation method and device based on a video conference terminal, equipment and a medium, comprising deploying a lightweight artificial intelligence large model and an auxiliary model locally on the video conference terminal, the auxiliary model comprising a speech recognition model and an image recognition model; collecting user input information when the video conference terminal is in an offline state, the user input information comprising at least one of speech, text and images; extracting the content of the user input information to obtain text content, and using the lightweight artificial intelligence large model locally to understand the semantics of the text content and generate response content; and converting the generated response content into speech output through the auxiliary model, and / or presenting the answer content through the video conference terminal, to solve the problems of offline unavailability, high data security risk and large interaction delay caused by the dependence of traditional video conference terminals on cloud AI services.

Description

Technical Field

[0001] This invention relates to the field of intelligent interactive technology for video conferencing terminals, and in particular to a method, apparatus, device, and medium for implementing an offline AI assistant based on a video conferencing terminal. Background Technology

[0002] With the rapid development of artificial intelligence and large language model technology, AI assistants have been widely applied in scenarios such as voice interaction, intelligent control, content generation, knowledge Q&A, and office collaboration. In the field of video conferencing terminals, users have placed higher demands on the intelligence level of devices, expecting terminals to be able to complete operations such as meeting control, information retrieval, and content summarization through natural language interaction, in order to improve meeting efficiency and user experience.

[0003] Currently, mainstream AI assistant solutions for video conferencing terminals generally adopt a cloud architecture. Under this architecture, the terminal device is only responsible for collecting the user's voice or text commands, uploading the data to a cloud-based AI service platform via the network, where a large cloud model performs inference and calculations, and then returning the results to the terminal for execution or display. However, this solution has significant drawbacks in practical applications. First, the AI ​​function heavily relies on public network connectivity. In scenarios such as government and enterprise intranets, confidential conference rooms, private network environments, and weak or no network access, cloud services are unavailable, rendering the AI ​​assistant function completely unusable or severely degrading the user experience. Second, sensitive data such as conference audio, images, and control commands need to be uploaded to third-party cloud servers for processing, posing risks of enterprise data leakage, leakage of confidential information, and user privacy, making it difficult to meet the data security compliance requirements of industries such as government, finance, and military. Furthermore, due to reliance on network transmission and cloud queuing processing, there is a significant delay in command response, resulting in choppy voice interaction and affecting the real-time dialogue experience. In addition, traditional video conferencing terminals themselves only possess audio and video encoding / decoding and network communication capabilities, lacking intelligent capabilities such as local semantic understanding and local inference and decision-making, resulting in a low level of terminal intelligence.

[0004] Therefore, there is an urgent need for an offline AI assistant implementation method based on video conferencing terminals to solve the technical problems of offline unavailability, high data security risks, and large interaction latency caused by traditional video conferencing terminals relying on cloud AI services. Summary of the Invention

[0005] To overcome the problems existing in the related technologies, this disclosure provides a method, apparatus, device and medium for implementing an offline AI assistant based on a video conferencing terminal, so as to solve the technical problems of offline unavailability, high data security risks and large interaction delays caused by the reliance of traditional video conferencing terminals on cloud AI services.

[0006] This specification provides one or more embodiments of an offline AI assistant implementation method based on a video conferencing terminal, including the following steps: A lightweight artificial intelligence big model and auxiliary model are deployed locally on the video conferencing terminal. The auxiliary model includes a speech recognition model and an image recognition model. When the video conferencing terminal is offline, user input information is collected, and the user input information includes at least one of voice, text and image. The user input information is extracted to obtain text content, and the lightweight artificial intelligence model is used locally to perform semantic understanding on the text content to generate response content. The generated response content is converted into speech output through the auxiliary model, and / or the response content is presented through the video conferencing terminal.

[0007] Preferably, the step of extracting text content from the user input information specifically includes the following steps: When the user input information is voice, the voice recognition model is used to extract the text content from the voice.

[0008] When the user input information is an image, the image recognition model is used to perform text recognition and / or visual feature extraction on the image to obtain text content.

[0009] Preferably, the step of using the lightweight AI model locally to perform semantic understanding of the text content and generate response content specifically includes the following steps: When the user's intent is a knowledge-based question and answer, the local knowledge base of the video conferencing terminal is retrieved to obtain relevant text fragments, and the answer content is generated by combining the relevant text fragments with the lightweight artificial intelligence big model. When the user's intent is device control, the lightweight artificial intelligence model generates device control commands, which are then used to directly control the hardware of the video conferencing terminal through its control interface.

[0010] Preferably, the step of performing semantic understanding of the text content locally using the lightweight artificial intelligence large model further includes the following steps: Maintain a dialogue history cache and merge the current text content with the text content of the historical context to achieve multi-turn continuous semantic interaction.

[0011] Preferably, the step of retrieving relevant text fragments from the local knowledge base of the video conferencing terminal specifically includes the following steps: The text content is vectorized, and semantic retrieval is performed in a pre-built vectorized knowledge base index to recall multiple relevant document fragments that rank highly, which are then used as enhanced context input to the lightweight artificial intelligence model.

[0012] Preferably, the method further includes the following steps: The speech recognition model is used to transcribe the conference audio in real time and generate conference subtitles for display. After the meeting, the lightweight AI big data model is triggered to analyze the meeting content and automatically generate a meeting summary, to-do list and / or key points of the speeches.

[0013] Preferably, before deploying the lightweight AI large model and auxiliary model locally on the video conferencing terminal, the following steps are also included: The hardware resources of the video conferencing terminal are detected, and a suitable quantization model specification is dynamically selected and loaded based on the hardware resources. The hardware resources include a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and available memory.

[0014] This specification provides one or more embodiments of an offline AI assistant implementation device based on a video conferencing terminal, including: The model deployment module is used to deploy a lightweight artificial intelligence large model and auxiliary models locally on the video conferencing terminal. The auxiliary models include a speech recognition model and an image recognition model. The information acquisition module is used to collect user input information when the video conferencing terminal is offline, and the user input information includes at least one of voice, text and image. The intent parsing module is used to extract text content from the user input information, and to perform semantic understanding of the text content locally using the lightweight artificial intelligence big model to generate response content. The result output module is used to convert the generated answer content into speech output through the auxiliary model, and / or present the answer content through the video conferencing terminal.

[0015] This specification provides one or more embodiments of a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the offline AI assistant implementation method based on a video conferencing terminal as described above.

[0016] This specification provides one or more embodiments of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the offline AI assistant implementation method based on a video conferencing terminal as described above.

[0017] This disclosure provides a method, apparatus, device, and medium for implementing an offline AI assistant based on a video conferencing terminal. Its advantages lie in that by pre-deploying a pruned and quantized lightweight large model locally on the video conferencing terminal and integrating dedicated speech and image recognition auxiliary models, the terminal can independently complete multimodal information processing without cloud dependence. This significantly reduces response latency, ensures that meeting data remains on the device throughout the entire process, and avoids service unavailability due to network interruptions in offline scenarios. When there is no network connection, the terminal's microphone, camera, and text input interface capture the user's voice, text, or image commands in real time, ensuring complete acquisition of user intent even in offline or weak network environments. Furthermore, the acquisition process does not involve any external transmission, eliminating the risk of privacy leaks at the source. The collected raw information undergoes noise filtering and key text extraction. Then, a local lightweight large model is used to perform semantic parsing and logical reasoning to quickly generate responses or operation suggestions for the meeting scenario. This process is completed entirely on the terminal chip, avoiding the uncertain latency caused by cloud queuing and ensuring inference efficiency through lightweight model design, so that the response generation speed meets the needs of real-time interaction. The generated text response is converted into natural and fluent speech in real time by a speech synthesis model. At the same time, the response content is displayed in the form of text, graphics and other forms on the terminal screen or indicator lights to achieve audiovisual feedback. This output link does not require network access to call external services, which ensures output stability and improves the efficiency of information reception and the naturalness of interaction for users in offline scenarios through multimodal presentation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an offline AI assistant implementation method based on a video conferencing terminal, provided for one or more embodiments of this specification; Figure 2 A schematic diagram of the structure of an offline AI assistant implementation device based on a video conferencing terminal, provided for one or more embodiments of this specification; Figure 3 This is a schematic diagram of the structure of a computer device provided for one or more embodiments of this specification. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this invention.

[0021] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0022] Method Implementation Examples According to embodiments of the present invention, an offline AI assistant implementation method based on a video conferencing terminal is provided, such as... Figure 1 The diagram shown illustrates the flowchart of the offline AI assistant implementation method based on a video conferencing terminal provided in this embodiment. The video conferencing terminal can be an Android conferencing terminal, a Linux conferencing terminal, an all-in-one conferencing device, a smart conferencing tablet, or a video collaboration terminal. Specifically, it includes a camera module, a microphone array module, a speaker module, a display module, a codec module, a network communication module, and a system control module. The offline AI assistant implementation method based on a video conferencing terminal according to this embodiment includes the following steps: S110. A lightweight AI large-scale model and auxiliary models are deployed locally on the video conferencing terminal to enable local AI inference capabilities. The auxiliary models include speech recognition and image recognition models, specifically including a large language model (LLM), an ASR speech recognition model, a TTS speech synthesis model, an NLP semantic understanding model, a multimodal understanding model, an image recognition model, a translation model, and a vector retrieval model, completing the inference engine warm-up. Simultaneously, a pre-built vectorized knowledge base index, including enterprise documents, FAQs, product information, system configurations, and historical meeting records, is loaded and mounted to the RAG retrieval engine, completing the initialization of the enhanced inference channel. The inference hardware supports CPU inference, GPU inference, NPU inference, and heterogeneous accelerated inference.

[0023] S120. When the video conferencing terminal is completely offline—without a public network connection—the microphone array module continuously collects user input information in low-power mode. User input includes at least one of voice, text, and image. A local lightweight wake-word detection model processes the input in real time, activating the main inference channel upon detecting a wake-word (such as "little fish"). In addition to voice, the system simultaneously receives text commands from the touchscreen, image frames captured by the camera, and system operating metrics, constructing a multimodal input stream.

[0024] S130. Extract text content from user input information. Specifically, the input is processed according to its form: speech is transcribed into text using an ASR model; image frames undergo OCR text recognition, scene recognition, and face detection using an image recognition model, and the results are converted into text descriptions; text input is directly used as text content; system status data is converted into text descriptions after log parsing and anomaly indicator extraction, ultimately obtaining text content suitable for understanding by a lightweight AI model. A local NLP semantic understanding model uses a lightweight AI model to perform semantic understanding of the text content, obtaining intent classifications such as device control, knowledge question answering, meeting functions, and maintenance diagnosis, as well as extracting key slots such as target devices, operation types, and parameter values. Enhanced context is then passed to the local LLM inference engine to generate response content such as completed answers, summary extraction, translation, and action item extraction. The entire process is completed on the terminal side without data transmission.

[0025] S140: The generated response content is converted into natural language output through an auxiliary model and broadcast via a speaker, supporting multilingual and multi-tone configurations, and / or presented via video conferencing terminals in the form of cards, subtitles, pop-ups, etc., displaying subtitles, Q&A results, device status, etc. All interaction logs, inference records, and device operation history are stored locally on the terminal in encrypted form, supporting audit queries and problem tracing, without uploading any data to the cloud.

[0026] The method provided in this embodiment pre-deploys a pruned and quantized lightweight large model locally on the video conferencing terminal and integrates dedicated speech recognition and image recognition auxiliary models. This allows the terminal to independently complete multimodal information processing without cloud reliance, thereby significantly reducing response latency, ensuring that meeting data remains on the device throughout the entire process, and avoiding service unavailability due to network interruptions in offline scenarios. When there is no network connection, the method captures the user's voice, text, or image commands in real time through the terminal's microphone, camera, and text input interface, ensuring that user intent can still be fully acquired even in offline or weak network environments. Furthermore, the acquisition process does not involve any external transmission, eliminating the risk of privacy leaks at the source. The acquired raw information is then subjected to noise filtering and... Key text is extracted, and then semantic parsing and logical reasoning are performed using a local lightweight large model to quickly generate responses or operation suggestions for the meeting scenario. This process is completed entirely on the terminal chip, which avoids the uncertainty and latency caused by cloud queuing, and ensures inference efficiency through lightweight model design, so that the response generation speed meets the needs of real-time interaction. The generated text response is converted into a natural and fluent speech in real time by a speech synthesis model, and the response content is displayed in the form of text, graphics and other forms in combination with the terminal screen or indicator lights to achieve audiovisual feedback. This output link does not require network access to call external services, which ensures output stability and improves the efficiency of information reception and the naturalness of interaction for users in offline scenarios through multimodal presentation.

[0027] In one embodiment, extracting text content from user input information includes the following steps: When a user inputs information in speech, a speech recognition model extracts the text content from the speech. For example, if a user says "Start the meeting" in a meeting, the microphone array picks up the voice command. The speech recognition model (ASR) in the auxiliary model transcribes this speech into the text "Start the meeting" in real time. This text content is then fed into a lightweight AI model for semantic understanding. Similarly, voice commands such as "Turn up the volume," "Switch to presentation mode," "Turn on subtitles," and "Summarize the discussion" are all transcribed into corresponding text by the ASR model before being understood and responded to by the larger model. The ASR model runs on a local inference engine, and the transcription process does not rely on the network, ensuring voice interaction capabilities in offline environments.

[0028] When a user inputs an image, an image recognition model is used to perform text recognition and / or visual feature extraction to obtain the text content. In Scenario 1: During a meeting, the user points to a whiteboard in the shared screen content, and the AI ​​assistant captures an image frame of the whiteboard using a camera. The image recognition model in the auxiliary model performs OCR text recognition on the image to extract the text content on the whiteboard, while simultaneously performing visual feature extraction to recognize the graphic structure on the whiteboard. The recognition and extraction results are converted into a text description, such as "The whiteboard states that the sales target for this quarter is 50 million, and a bar chart compares the performance of each region." This text description is then fed into the larger model, where the user can further ask, "Summarize the content on the whiteboard." The larger model combines OCR text recognition with visual features to perform cross-modal semantic understanding and generate an accurate answer. In Scenario 2: During a meeting, the AI ​​assistant performs image recognition and OCR text extraction on the shared PPT content to obtain the text and chart information from the slides. The user asks, "What is the core idea of ​​that PPT slide?" The system performs semantic understanding and provides an answer based on the image recognition results. In scenario three: The system uses a camera to perform facial recognition and scene recognition to determine the identity of the attendees and the type of meeting scenario, and converts the recognition results into text descriptions for use by the large model, thus achieving intelligent scene perception.

[0029] The method provided in this embodiment, through a multimodal content extraction mechanism, can call a speech recognition model for voice input and an image recognition model for image input to perform text recognition or visual feature extraction, thereby uniformly converting different types of user input into structured text content, significantly improving the compatibility and flexibility of information processing, and is suitable for diverse human-computer interaction scenarios.

[0030] In one embodiment, a lightweight AI model is used locally to perform semantic understanding of the text content and generate response content, specifically including the following steps: When a user's intent is a knowledge-based question and answer, the system triggers a local knowledge base retrieval enhancement generation process. This process retrieves relevant text fragments from the video conferencing terminal's local knowledge base and generates the answer content by combining these fragments with a lightweight AI model.

[0031] In one scenario, a user asks during a meeting, "What is our company's sales strategy for Q3?" The system vectorizes the query text and performs semantic retrieval in a pre-built vectorized knowledge base index, recalling several top-ranked relevant document fragments (such as corporate strategy documents, sales department meeting minutes, etc.). These fragments are then appended as enhanced context to the original query and input into the LLM inference engine to generate an accurate answer that is relevant to the company's actual situation.

[0032] When a user's intent is device control, the AI ​​assistant parses the intent, generates device control commands using a lightweight AI model, verifies authorization via a permission verification module, and then calls the terminal device control API to directly control the video conferencing terminal's hardware through its control interface. For example, if a user says "mute microphone," the AI ​​assistant parses the intent as device control, the target device as the microphone, and the operation type as mute. After verifying the user's permission, it calls the microphone control interface to mute the device. Controllable hardware devices include, but are not limited to: cameras, microphones, speakers, display mode switching modules, and meeting control modules. Execution results are confirmed via a callback mechanism, and AI-powered automatic fault analysis is triggered in case of anomalies.

[0033] The method provided in this embodiment uses a local lightweight large model to perform semantic understanding of text, generates answers based on local knowledge base retrieval for knowledge-based question-and-answer intent, and directly generates control commands and operates hardware for device control intent. This achieves fast, secure, and cloud-free intelligent response, significantly improving the interaction efficiency and autonomy of video conferencing terminals.

[0034] In one embodiment, performing semantic understanding of text content locally using a lightweight AI large model further includes the following steps: By maintaining a dialogue history cache, the system fuses and infers the current text content with the historical context, enabling multi-turn continuous semantic interaction. For example, if a user says, "Turn the volume up a bit," the system will increase the volume. If the user then says, "A little higher," the system understands that "a little higher" means "increase the volume again," and the target is volume, so it increases the volume further. If the user then says, "Never mind, let's go back to the previous level," the system will retrieve the original volume value from the dialogue history and restore the setting. This multi-turn dialogue mechanism enables the AI ​​assistant to have continuous semantic interaction capabilities, improving the natural fluency of the user experience.

[0035] The method provided in this embodiment extracts speech and images into text in a unified manner through multimodal content extraction, and uses a local lightweight large model for semantic understanding: for knowledge question answering, it retrieves answers from the local knowledge base; for device control, it generates instructions to directly operate the hardware; and at the same time, it maintains a dialogue history cache to achieve multi-round continuous interaction, thereby providing fast, secure, and autonomous intelligent response locally, significantly improving the interaction efficiency and user experience of video conferencing terminals.

[0036] In one embodiment, retrieving relevant text fragments from the local knowledge base of the video conferencing terminal includes the following steps: First, the text content corresponding to the user query is vectorized, that is, the natural language query is encoded into a high-dimensional semantic vector through a vector retrieval model. Then, semantic retrieval is performed on a pre-built vectorized knowledge base index, which covers various knowledge sources such as company documents, product information, system configurations, historical meeting minutes, FAQ data, and user knowledge data. Semantic retrieval is based on vector similarity calculation, recalling the top-ranked relevant document fragments (Top-K). Finally, the recalled document fragments are used as enhanced context, concatenated with the original user query, and input into a lightweight AI model. The model performs reasoning based on the enhanced context, generating an accurate answer incorporating company-specific knowledge.

[0037] For example, if a user asks, "What is our after-sales service process?", the system will vectorize the query and retrieve relevant snippets from the knowledge base, such as after-sales service manuals and process documents. The LLM then generates a complete process description based on these snippets. If the user confirms the answer is correct, they can choose to incrementally write the question-and-answer pair to the local knowledge base vector index to continuously optimize subsequent search results.

[0038] The method provided in this embodiment can quickly recall multiple relevant document fragments ranked at the top by vectorizing the text and performing semantic retrieval in the local knowledge base index. These fragments serve as input to the large model to enhance the context, thereby significantly improving the accuracy of the answer and the knowledge coverage. At the same time, the retrieval process is completed locally, ensuring data security and low latency.

[0039] In one embodiment, the following steps are also included: During the meeting, the speech recognition model in the auxiliary model is used to transcribe the meeting audio in real time, continuously converting the speakers' voices into text to generate meeting subtitles for display. The subtitle content is displayed as a subtitle bar at the bottom of the screen through the display module of the video conferencing terminal, enhancing the readability and traceability of the meeting content.

[0040] For meeting scenarios requiring cross-language communication, the text transcribed by ASR is translated in real time by a translation model, and the translation results are displayed as subtitles, supporting multilingual translation.

[0041] By combining voiceprint feature extraction and facial recognition technologies, the system can identify the current speaker and annotate the speaker's information in subtitles or meeting minutes.

[0042] After the meeting concludes, an automated AI processing flow is triggered. A lightweight AI model analyzes the meeting content, automatically generating a meeting summary, to-do lists, and / or key points of each speaker. It automatically identifies and extracts explicit action items and to-do lists from the meeting, generating a to-do list; it also automatically summarizes the key points of each topic, generating a summary of key points. The generated content is displayed through a display module and stored locally on the terminal.

[0043] When the system detects abnormal logs or operational alarms, the AI ​​automatically initiates a log analysis pipeline: after feature extraction, the LLM performs diagnostic reasoning, generating fault cause analysis and handling suggestions, which are then communicated to the administrator via voice broadcast or screen display. For example, if a user asks "Why can't the terminal join the meeting?", "Analyze recent abnormal logs?", or "What is the reason for the current high CPU usage?", the AI ​​assistant analyzes the local logs and system status and provides operational suggestions.

[0044] The method provided in this embodiment extracts speech and images into text in a unified manner through multimodal content extraction, uses a local lightweight large model for semantic understanding, supports multi-turn continuous dialogue, enhances responses by combining local knowledge base retrieval, and generates device control commands; it also has the functions of real-time conference subtitle transcription and automatic generation of summaries, to-do lists, and key points of speeches after the meeting, thereby realizing fast, safe and intelligent conference assistance and terminal control locally, and comprehensively improving interaction efficiency and conference management effectiveness.

[0045] In one embodiment, before deploying the lightweight AI large model and auxiliary model locally on the video conferencing terminal, the following steps are also included: The hardware resources of the video conferencing terminal are detected, and the appropriate quantization model specification is dynamically selected and loaded based on these resources. A high-end all-in-one conference tablet is equipped with a high-performance GPU and 16GB of available memory. After system evaluation, it can load a lightweight large model with FP16 accuracy and a large number of parameters to obtain the best inference effect and response quality. A certain Android conferencing terminal is equipped with only a mid-range CPU and 4GB of memory. After system evaluation, it automatically selects a model version with INT8 quantization accuracy and a smaller number of parameters to ensure smooth operation under limited resources. A Linux conference terminal is equipped with a dedicated NPU neural network processor. The system enables the NPU inference channel for model inference, achieving low-power and high-efficiency AI computing. A terminal has both a GPU and an NPU. The system can enable heterogeneous accelerated inference, which can allocate different sub-models to the most suitable hardware for parallel operation, such as LLM running on the GPU and ASR running on the NPU.

[0046] Hardware resources include the model and computing power of the central processing unit (CPU), the video memory and computing units of the graphics processing unit (GPU), the availability of the neural network processor (NPU), and the available system memory and storage space.

[0047] The AI ​​task scheduling module is also responsible for AI task distribution, dynamic model loading, inference resource scheduling, power consumption control, and memory management, supporting multi-model collaborative operation. Simultaneously initialized auxiliary models include speech recognition models, image recognition models, and TTS speech synthesis models, all of which are loaded with adapted versions based on hardware resource evaluation results and enter standby mode after the inference engine has warmed up.

[0048] The method provided in this embodiment deploys a lightweight large model and auxiliary model locally on the video conferencing terminal. Before deployment, it dynamically detects the CPU, GPU, NPU, and available memory, and adaptively selects the optimal quantization specification for loading, ensuring compatibility and efficient operation. The solution supports multimodal input to text conversion, multi-turn continuous dialogue, enhanced responses through local knowledge base vector retrieval, and device command control. It also provides real-time conference subtitle transcription and automatic generation of summaries, to-do lists, and key points after the meeting, thereby comprehensively improving the efficiency and intelligence of conference interaction while ensuring data security and low latency.

[0049] Device Examples According to embodiments of the present invention, an offline AI assistant implementation device based on a video conferencing terminal is provided, such as... Figure 2 The diagram shown is a structural schematic of the offline AI assistant implementation device based on a video conferencing terminal provided in this embodiment. The offline AI assistant implementation device based on a video conferencing terminal according to this embodiment includes: The model deployment module 21 is used to deploy lightweight artificial intelligence large models and auxiliary models locally on the video conferencing terminal. The auxiliary models include speech recognition models and image recognition models.

[0050] The information acquisition module 22 is used to collect user input information when the video conferencing terminal is offline. The user input information includes at least one of voice, text and image.

[0051] The intent parsing module 23 is used to extract text content from user input information, and to perform semantic understanding of the text content locally using a lightweight artificial intelligence model to generate response content.

[0052] The result output module 24 is used to convert the generated answer content into speech output through an auxiliary model, and / or present the answer content through a video conferencing terminal.

[0053] The device provided in this embodiment pre-deploys a pruned and quantized lightweight large model locally on the video conferencing terminal through the model deployment module 21, and integrates dedicated speech recognition and image recognition auxiliary models. This allows the terminal to independently complete multimodal information processing without cloud dependence, thereby significantly reducing response latency, ensuring that meeting data does not leave the device throughout the entire process, and avoiding service unavailability due to network interruptions in offline scenarios. When there is no network connection, the information acquisition module 22 captures the user's voice, text, or image commands in real time through the terminal's microphone, camera, and text input interface, ensuring that the user's intent can still be fully obtained in the absence of a network or weak network environment. Moreover, the acquisition process does not involve any external transmission, eliminating the risk of privacy leakage from the source. The intent parsing module 23 processes the acquired raw information. Noise filtering and key text extraction are performed, and then a local lightweight large model is used to perform semantic parsing and logical reasoning to quickly generate responses or operation suggestions for the meeting scenario. This process is completed entirely on the terminal chip, which avoids the uncertainty and latency caused by cloud queuing, and ensures inference efficiency through lightweight model design, so that the response generation speed meets the needs of real-time interaction. The result output module 24 uses a speech synthesis model to convert the generated text response into a natural and fluent speech in real time. At the same time, the response content is displayed in the form of text, graphics and other forms on the terminal screen or indicator lights to achieve audiovisual feedback. This output link does not need to call external services through the network, which ensures output stability and improves the user's information reception efficiency and interaction naturalness in offline scenarios through multimodal presentation.

[0054] The embodiments of the present invention are device embodiments corresponding to the above method embodiments. The specific operations of each module processing step can be understood with reference to the description of the method embodiments, and will not be repeated here.

[0055] like Figure 3 As shown, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the offline AI assistant implementation method based on a video conferencing terminal in the above embodiments, or when the computer program is executed by a processor, it implements the offline AI assistant implementation method based on a video conferencing terminal in the above embodiments.

[0056] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0057] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention are known to those skilled in the art.

Claims

1. A method for implementing an offline AI assistant based on a video conferencing terminal, characterized in that, Includes the following steps: A lightweight artificial intelligence big model and auxiliary model are deployed locally on the video conferencing terminal. The auxiliary model includes a speech recognition model and an image recognition model. When the video conferencing terminal is offline, user input information is collected, and the user input information includes at least one of voice, text and image. The user input information is extracted to obtain text content, and the lightweight artificial intelligence model is used locally to perform semantic understanding on the text content to generate response content. The generated response content is converted into speech output through the auxiliary model, and / or the response content is presented through the video conferencing terminal.

2. The offline AI assistant implementation method based on a video conferencing terminal as described in claim 1, characterized in that, The process of extracting text content from the user input information specifically includes the following steps: When the user input information is voice, the voice recognition model is used to extract the text content from the voice. When the user input information is an image, the image recognition model is used to perform text recognition and / or visual feature extraction on the image to obtain text content.

3. The offline AI assistant implementation method based on a video conferencing terminal as described in claim 1, characterized in that, The step of using the lightweight AI model locally to perform semantic understanding of the text content and generate response content specifically includes the following steps: When the user's intent is a knowledge-based question and answer, the local knowledge base of the video conferencing terminal is retrieved to obtain relevant text fragments, and the answer content is generated by combining the relevant text fragments with the lightweight artificial intelligence big model. When the user's intent is device control, the lightweight artificial intelligence model generates device control commands, which are then used to directly control the hardware of the video conferencing terminal through its control interface.

4. The offline AI assistant implementation method based on a video conferencing terminal as described in claim 3, characterized in that, The step of retrieving relevant text fragments from the local knowledge base of the video conferencing terminal specifically includes the following steps: The text content is vectorized, and semantic retrieval is performed in a pre-built vectorized knowledge base index to recall multiple relevant document fragments that rank highly, which are then used as enhanced context input to the lightweight artificial intelligence model.

5. The offline AI assistant implementation method based on a video conferencing terminal as described in claim 1, characterized in that, The step of performing semantic understanding of the text content locally using the lightweight artificial intelligence model further includes the following steps: Maintain a dialogue history cache and merge the current text content with the text content of the historical context to achieve multi-turn continuous semantic interaction.

6. The offline AI assistant implementation method based on a video conferencing terminal as described in claim 1, characterized in that, It also includes the following steps: The speech recognition model is used to transcribe the conference audio in real time and generate conference subtitles for display. After the meeting, the lightweight AI big data model is triggered to analyze the meeting content and automatically generate a meeting summary, to-do list and / or key points of the speeches.

7. The offline AI assistant implementation method based on a video conferencing terminal as described in claim 1, characterized in that, Before deploying the lightweight AI large model and auxiliary model locally on the video conferencing terminal, the following steps are also included: The hardware resources of the video conferencing terminal are detected, and a suitable quantization model specification is dynamically selected and loaded based on the hardware resources. The hardware resources include a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and available memory.

8. An offline AI assistant implementation device based on a video conferencing terminal, characterized in that, include: The model deployment module is used to deploy a lightweight artificial intelligence large model and auxiliary models locally on the video conferencing terminal. The auxiliary models include a speech recognition model and an image recognition model. The information acquisition module is used to collect user input information when the video conferencing terminal is offline, and the user input information includes at least one of voice, text and image. The intent parsing module is used to extract text content from the user input information, and to perform semantic understanding of the text content locally using the lightweight artificial intelligence big model to generate response content. The result output module is used to convert the generated answer content into speech output through the auxiliary model, and / or present the answer content through the video conferencing terminal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the offline AI assistant implementation method based on a video conferencing terminal as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the offline AI assistant implementation method based on a video conferencing terminal as described in any one of claims 1 to 7.

Images