Intelligent conference auxiliary method and system

Abstract

The invention provides an intelligent conference auxiliary method and system, and belongs to the field of network communication. The method comprises the following steps: excluding target application window data from shared stream data in a process of generating the shared stream data based on collected screen shared data, and sending the target application window data to a conference terminal for visual interface feedback; acquiring voice data of the conference terminal in real time, and transliterating the voice data to generate voice text stream data; in response to the screenshot instruction, performing screenshot analysis on the selected screen image to generate image text stream data; and inputting the generated voice text stream data and / or image text stream data into a pre-training language model, fusing the context information of the current session, and generating a structured response content conforming to a preset rule. According to the intelligent conference assisting method and system, it can be ensured that a user selectively hides a specific application window in screen sharing, and intelligent assistance can be provided for a video conference of the user, so that the personalized and scene-aware interaction experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of network communications, and in particular to an intelligent conference assistance method and system. Background Art

[0002] In the current digital office and remote collaboration environment, video conferencing systems with screen sharing functions have become an indispensable tool.

[0003] However, existing video conferencing systems still have many shortcomings. For example, the screen sharing function in related technologies is usually unable to selectively hide specific application windows, which may cause participants to see all the content on the sharer's screen, including sensitive information and business secrets, posing a serious risk of leakage; and the video conferencing system lacks the support of intelligent auxiliary tools during user use, making it difficult to provide personalized and scene-aware interactive experience, resulting in a poor user experience.

[0004] In view of the above, this application is hereby filed. Summary of the Invention

[0005] The present application provides an intelligent conference assistance method and system to solve or alleviate at least one technical problem existing in the above-mentioned prior art.

[0006] According to a first aspect of the present application, there is provided an intelligent conference assistance method, comprising:

[0007] In the process of generating shared stream data based on the collected screen sharing data, the target application window data is excluded from the shared stream data;

[0008] Sending the shared stream data excluding the target application window data to the conference terminal for visual interface feedback; and

[0009] Acquire the voice data from the conference terminal in real time, transcribe the voice data, and generate voice text stream data;

[0010] Responding to the screenshot command, the screenshot of the screen selection image is parsed to generate image text stream data;

[0011] The generated speech text stream data and / or image text stream data are input into the pre-trained language model, integrated with the current conversation context information, and structured response content that conforms to preset rules is generated.

[0012] In some embodiments, the present invention further comprises:

[0013] Read and parse the resume text uploaded by the user, and extract the target information data in the resume text;

[0014] The extracted target information data and the voice text stream data or the image text stream data are input into the pre-trained language model to generate structured response content for the resume content.

[0015] In some implementations, the extracted target information data is further cached.

[0016] In some embodiments, the present invention further comprises:

[0017] Based on the preset prompt words, the behavioral pattern of the structured response content that meets the preset rules is determined;

[0018] When the response content that complies with the preset rules meets the prompt word, the behavior pattern is triggered and the behavior pattern adaptation operation is performed, including dynamically adjusting the response strategy and content recommendation priority of the pre-trained language model and / or performing parameter adjustments.

[0019] In some implementations, excluding target application window data from the shared stream data during the generation of shared stream data based on the collected screen sharing data includes:

[0020] During the screen sharing initialization phase, the multimedia collaborative processing interface at the bottom layer of the operating system is called to mark the target application window as a privacy isolation object;

[0021] Based on the flag, pixel data of the target application window is skipped in the shared stream generation pipeline.

[0022] In some embodiments, the method specifically includes:

[0023] Get the process identifier and graphics surface handle of the target application window;

[0024] Based on the obtained process identifier and graphics surface handle, register the privacy identifier with the multimedia collaborative processing interface and associate it with the graphics rendering pipeline, and establish a mapping relationship between the identifier and the pixel area;

[0025] When a shared stream is generated, the pixel region carrying the identifier is skipped based on the mapping relationship.

[0026] In some embodiments, the present invention further comprises:

[0027] Before the shared stream is activated, a preview interface is generated through a simulated rendering mechanism;

[0028] The preview interface marks the target application window area to be excluded with a highlighted frame and receives a user confirmation instruction.

[0029] In some implementations, excluding target application window data from the shared stream data during the generation of shared stream data based on the collected screen sharing data includes:

[0030] Intercept the rendering data stream of the target application window through the virtual rendering layer and perform transparency processing on its pixel area;

[0031] Specifically, the method includes: locating the position of the target application window in the screen coordinate system, and setting the RGBA transparency channel value of the corresponding area to zero to achieve complete transparency.

[0032] In some implementations, performing screenshot analysis on the screen selection image in response to the screenshot instruction includes:

[0033] Identify programming code within an image of a selected area of ​​the screen and associate it with the context of a technical question; and / or

[0034] Recognizes natural language text in images of selected screen areas and associates them with meeting discussion context.

[0035] In some embodiments, generating structured response content that complies with preset rules includes generating structured response content that complies with the STAR principle, wherein the STAR principle includes the elements of scenario, task, action, and result;

[0036] The target information data extracted from the resume text includes at least one of educational background information data, work experience information data or skill expertise information data.

[0037] According to a second aspect of the present application, an intelligent conference assistance system is provided, which is used for the intelligent conference assistance method as described above, including:

[0038] The privacy isolation layer is deployed between the operating system rendering layer and the screen capture layer. It is used to exclude the target application window data from the shared stream data generated based on the collected screen sharing data, and send the shared stream data excluding the target application window data to the conference terminal for visual interface feedback.

[0039] The multimodal input module includes: a speech recognition submodule for acquiring speech data from the conference terminal in real time, transcribing the speech data, and generating speech-to-text stream data; a screenshot recognition submodule for responding to screenshot commands to perform screenshot analysis on the screen selection image and generate image-to-text stream data;

[0040] The intelligent question-answering engine is connected to the multimodal input module and is used to call the pre-trained language model to fuse the generated voice text stream data and / or image text stream data with the current conversation context information to generate structured response content that conforms to preset rules.

[0041] In some embodiments, the present invention further comprises:

[0042] The resume parsing module is connected to the intelligent question-answering engine to read and parse the resume text uploaded by the user and extract the target information data from the resume text;

[0043] A behavior adaptation module, connected to the intelligent question-answering engine, is used to dynamically adjust the answer strategy and content recommendation priority of the pre-trained language model and / or adjust user preference parameters;

[0044] Among them, the intelligent question-answering engine is also used to call the pre-trained language model to fuse the extracted target information data with the voice text stream data or image text stream data to generate interview simulation questions and answer suggestions for the resume content; and based on preset prompt words, it determines the behavioral patterns of structured response content that meets the preset rules.

[0045] In some embodiments, the privacy isolation layer, multimodal input module, resume parsing module, intelligent question-and-answer engine, and behavior adaptation module are connected via an internal message bus for asynchronous communication; wherein the speech recognition submodule is connected to the speaker input interface to obtain the voice data of the conference terminal in real time via the internal message bus.

[0046] According to a third aspect of the present application, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; and when the processor executes the computer program instructions, any one of the above-mentioned intelligent conference assistance methods is implemented.

[0047] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, any of the above-mentioned intelligent conference assistance methods is implemented.

[0048] In summary, the intelligent conference assistance method and system provided by this application have at least the following beneficial effects:

[0049] In the intelligent conference assistance method of the present application, in the process of generating shared stream data based on the collected screen sharing data, the target application window data is excluded from the shared stream data, and the shared stream data excluding the target application window data is sent to the conference terminal for visual interface feedback. This can prevent the target application window from being captured by the shared stream and only display it on the local interface, thereby effectively protecting user privacy and commercial secrets in video conferencing or screen sharing scenarios and eliminating the risk of sensitive information leakage;

[0050] By acquiring voice data from conference terminals in real time, the voice data is transcribed to generate voice-to-text stream data; in response to a screenshot command, a screenshot of the screen selection image is taken and analyzed to generate image-to-text stream data; the user can complete the generation of voice-to-text stream and image-to-text stream without switching applications, thereby shortening the information collection time and saving operation steps; the generated voice-to-text stream data and / or image-to-text stream data are further input into a pre-trained language model, and the current session context information is integrated to generate structured response content that conforms to preset rules. The user does not need to manually provide context to achieve efficient and natural interaction, thereby improving the user's personalized and scene-aware interactive experience when using the video conferencing system.

[0051] Therefore, the intelligent conference assistance method and system provided by this application can ensure that users can selectively hide specific application windows during screen sharing, and can provide intelligent assistance for user video conferencing, thereby significantly improving the user experience.

[0052] Other features and advantages of the intelligent conference assistance method and system provided in this application will be elaborated in detail in the subsequent specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 A flowchart of an intelligent conference assistance method provided in an embodiment of the present application;

[0055] Figure 2 A structural diagram of an intelligent conference assistance system provided in an embodiment of the present application;

[0056] Figure 3 A structural diagram of a multimodal input module provided in an embodiment of the present application;

[0057] Figure 4 A logical block diagram of an intelligent conference assistance system provided in an embodiment of the present application;

[0058] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the above and other features and advantages of the present application more clear, the present application is further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0060] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it will be apparent to those skilled in the art that it is not necessary to adopt the specific details to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid obscuring the present application.

[0061] refer to Figure 1-Figure 5 As shown, the intelligent conference assistance method provided in the embodiment of the present application can be executed by the intelligent conference assistance system 100 provided in the embodiment of the present application, and the intelligent conference assistance system 100 can be configured in the electronic device 400.

[0062] refer to Figure 1 , an embodiment of the present application provides an intelligent conference assistance method, the method comprising:

[0063] S100: excluding target application window data from the shared stream data during the process of generating shared stream data based on the collected screen sharing data;

[0064] S200, sending the shared stream data excluding the target application window data to the conference terminal for visual interface feedback;

[0065] S300, acquiring voice data from the conference terminal in real time, transcribing the voice data, and generating voice text stream data;

[0066] S400, responding to the screenshot instruction, performing screenshot analysis on the screen selection image to generate image text stream data;

[0067] S500: Input the generated speech text stream data and / or image text stream data into a pre-trained language model, integrate the current conversation context information, and generate structured response content that complies with preset rules.

[0068] It should be noted that step S100 mainly uses visibility isolation technology to exclude the target application window that does not want to be shared between the operating system screen capture layer and the application rendering layer, ensuring that the target application window is only displayed locally and is not captured by the shared stream; step S200 sends the shared stream data excluding the target application window data to the conference terminal for visual interface feedback. After the shared stream data excluding the target application window data is shared on the conference terminal, the visual interface may not contain traces of the target application window, so as to effectively protect user privacy and business secrets in video conferencing and screen sharing scenarios, and eliminate the risk of sensitive information leakage.

[0069] It should be understood that in the traditional screen sharing process, the screen content is captured by relying on the operating system screen capture layer API (such as Windows' GDI or DirectX interface), shared stream data is generated, and the rendering layer is applied to perform decoding and display to perform screen sharing on the conference terminal. There are usually two modes: full-screen sharing and window sharing. If the user chooses full-screen sharing, the system will capture all the content on the screen (including sensitive application windows that are not minimized), resulting in the risk of privacy leakage. Even if window sharing is selected, the target window can only be located through the window handle, and other windows cannot be dynamically excluded (for example, if the user mistakenly places a sensitive window on top of the shared window, its content will still be captured). This application directly excludes the target application window between the operating system screen capture layer and the rendering layer. The shared stream data does not contain the target application window data. Even if the user mistakenly operates the target application window later, it will not be shared to the conference terminal for visual interface feedback.

[0070] Furthermore, in traditional conference systems, voice transcription text and screenshot content often exist as independent data streams, resulting in information fragmentation. The intelligent conference assistance method of the present application supports multimodal integrated data input. Specifically, step S300 obtains the voice data of the conference terminal in real time, transcribes the voice data, and generates voice-to-text stream data. The voice data of the conference terminal comes from the online conference audio or the local voice output terminal, rather than being collected from the local voice input terminal, so as to avoid interference with the voice data output during the user meeting; and the transcription of the voice data can be transcribed into voice-to-text stream data through a local or cloud-based voice recognition service, providing data support for the determination of the problem during the subsequent response; step S400 responds to the screenshot instruction to perform a screenshot analysis on the screen selection image to generate image-to-text stream data. For example, when taking a screenshot, the user selects an area at any position on the screen and generates a screenshot instruction. By responding to the screenshot instruction, the screen selection image is screenshot analyzed and the target content in the image is generated into text stream data. Combined with step S300, the user can complete the voice and image-to-text conversion directly in one interface without switching applications, thereby improving the recognition speed and accuracy, thereby improving user efficiency and experience.

[0071] In step S500, the generated speech-to-text stream data and / or image-to-text stream data are input into a pre-trained language model, integrated with the current conversation context information, and structured responses that conform to pre-set rules are generated. The conversation context information primarily refers to historical conversation information. Pre-trained language models include, but are not limited to, pre-trained large language models from Alibaba Cloud and Magic Square Quantization, which can be implemented through local deployment or cloud-based API access. By integrating the speech-to-text stream data and image-to-text stream data obtained through speech transcription with the historical conversation context, the large language model generates structured responses that conform to pre-set rules. This improves the accuracy and coherence of questions and answers, enabling efficient and natural interaction without the user having to manually provide context.

[0072] To exclude the target application window from the shared stream data, in some embodiments, step S100 includes:

[0073] Step S110: During the screen sharing initialization phase, the multimedia collaborative processing interface of the underlying operating system is called to mark the target application window as a privacy isolation object;

[0074] Step S120: Based on the mark, skip the pixel data of the target application window in the shared stream generation pipeline.

[0075] In this embodiment, during the screen sharing initialization phase, a privacy identifier is registered for the target application window through the multimedia control interface at the bottom layer of the operating system, so that it is recognized by the system as a privacy isolation object that needs to be excluded. When the operating system generates shared stream data, the pixel data is dynamically filtered according to the privacy identifier to ensure that the target window content does not enter the encoding and transmission link, so as to exclude the target application window from the shared stream data. Among them, the multimedia co-processing interface can be an adjusted audio and graphics co-processing interface (such as WASAPI of the Windows operating system), which binds the privacy identifier to the graphics rendering surface by extending its stream control function, so as to control the privacy attributes of the graphics stream through the audio interface.

[0076] Furthermore, step S110 and step S120 specifically include: obtaining the process identifier and graphics surface handle of the target application window; based on the obtained process identifier and graphics surface handle, registering the privacy identifier to the multimedia collaborative processing interface and associating it with the graphics rendering pipeline, establishing a mapping relationship between the identifier and the pixel area; when the shared stream is generated, skipping the pixel area carrying the identifier based on the mapping relationship.

[0077] Among them, the process identifier of the target application window can be obtained by traversing the current active window through the window management interface provided by the operating system, and the target application process can be located through the operating system API (such as Windows' GetProcessId or Linux's pidof) to accurately locate the target application window, avoid misfiltering or missed filtering, and call the graphics interface (such as DirectX's IDXGISurface object) through the graphics surface handle of the target application window to obtain the video memory address corresponding to the window to locate the physical location of the pixel data in the video memory, which is conducive to the subsequent modification of the underlying pixel properties. Furthermore, based on the obtained process identifier and graphics surface handle, the extension function of the multimedia collaborative processing interface is called to register the privacy identifier, and the privacy identifier is associated with the synthesis stage or rasterization stage in the graphics rendering pipeline to establish a mapping relationship between the identifier and the pixel area; by generating a privacy identifier and binding it to the graphics surface data of the target window, during the shared stream generation process, the rendering pipeline uses the privacy identifier to identify the pixel area to be excluded; based on the mapping relationship, the filtering logic is inserted into the rendering pipeline of the graphics processing unit to skip the pixel data carrying the privacy identifier and retain only the screen content of the non-sensitive area. As a result, the graphics processing unit graphics driver layer can discard pixels according to the identifier, and the pixels of the target application window do not enter the network transmission layer at all, thereby achieving the exclusion of the target application window from the shared stream data.

[0078] In some other embodiments, in the process of generating shared stream data based on the collected screen sharing data, excluding the target application window data from the shared stream data includes:

[0079] Step S130 , intercepting the rendering data stream of the target application window through the virtual rendering layer, and performing transparency processing on its pixel area; specifically, including: locating the position of the target application window in the screen coordinate system, and setting the RGBA transparency channel value of the corresponding area to zero to achieve full transparency.

[0080] In an embodiment of the present application, during the screen sharing data generation stage, the rendering data stream of the target application window is intercepted by the virtual rendering layer. First, the boundary coordinates of the window in the screen coordinate system are located, and then the RGBA transparency channel values ​​(Alpha) of all pixels in the rectangular area are forced to zero, so that the target application window enters the shared stream encoding process in a completely transparent state, thereby eliminating sensitive content at the data source and achieving pixel-level privacy protection. Compared with the above-mentioned graphics driver layer discarding pixels according to identifiers, it is more compatible.

[0081] To determine whether to hide the target application window, in some embodiments, the intelligent conference assistance method further includes:

[0082] Before the shared stream is activated, a preview interface is generated through a simulated rendering mechanism;

[0083] The preview interface marks the target application window area to be excluded with a highlighted frame and receives a user confirmation instruction.

[0084] For example, before activating the shared stream, the graphics rendering context can be cloned to generate a simulated shared frame. In this simulated shared frame, the excluded area of ​​the target application window is visually highlighted (such as a highlighted border). In response to the user's confirmation instruction on the preview interface, the actual shared stream output is activated. Cloning the graphics rendering context includes copying the current frame buffer data to an isolated memory space, retaining the privacy filtering logic but disabling the network transmission module, and injecting a visual marker layer into the simulated frame. This allows users to intuitively perceive the automatic hiding of the target application window, avoiding accidental operations.

[0085] In some embodiments, when the intelligent conference assistance method is applied to an interview scenario, the intelligent conference assistance method further includes:

[0086] S600: Read and parse the resume text uploaded by the user, and extract target information data from the resume text;

[0087] S700: Input the extracted target information data and the voice text stream data or the image text stream data into a pre-trained language model to generate structured response content for the resume content.

[0088] In step S600, the resume text file uploaded by the user (supporting formats such as PDF / DOCX) is read, and the resume text structure is parsed using natural language processing technology. Key information is located through named entity recognition (NER) and keyword extraction algorithms, such as the name of the school, major and graduation time in the educational background, the company name, position, project results in the work experience, as well as target information data such as skill certificates and language proficiency. Then, in step S700, the resume target information extracted in step S600 is multimodally combined with the voice text stream (such as the interviewer's questions) or image text stream (such as the shared interview question bank screenshot) generated in real time during the meeting. Data fusion uses a pre-trained language model to understand contextual semantics. The model combines the educational background information, work experience information, or skill expertise information in the resume to generate targeted questions (for example, based on the "Intelligent Customer Service System Development" project in which the candidate participated, it generates "Please explain in detail the specific measures for optimizing the NLP module you are responsible for in this project"). At the same time, it provides answer suggestions based on the industry knowledge graph and common interview question library (such as recommending the "STAR Rule" structured answer template for technical positions). The final output includes structured response content containing simulated interview questions and corresponding answer suggestions, assisting interviewers in asking in-depth questions or helping candidates prepare answers.

[0089] In some embodiments, when the intelligent conference assistance method is applied to an interview scenario, the generation of structured response content that complies with preset rules in step 500 includes generating structured response content that complies with the STAR principle (Situation-Task-Action-Result), wherein the STAR principle includes the elements of scenario, task, action, and result, that is, generating interview questions and answer suggestions that comply with the STAR principle (situation / task / action / result).

[0090] In some embodiments, the target information data extracted from the resume text includes at least one of educational background information data, work experience information data, or skill expertise information data.

[0091] In some embodiments, the intelligent conference assistance method also includes caching the extracted target information data so that the pre-trained language model can obtain the cached data in real time. When the user conducts an interview simulation, there is no need to re-parse the resume text. Instead, the relevant work experience data is directly extracted from the cache and combined with the current voice transcription text to generate a STAR principle structured answer, which significantly shortens the response delay.

[0092] To dynamically adjust the response style and recommendation priority to enhance the user's personalized experience, in some embodiments, the intelligent conference assistance method further includes:

[0093] S800: Based on the preset prompt words, determine the behavior pattern of the structured response content that meets the preset rules;

[0094] S900. When the response content that complies with the preset rules meets the prompt word, trigger the behavior pattern and perform the behavior pattern adaptation operation, including dynamically adjusting the response strategy and content recommendation priority of the pre-trained language model and / or performing parameter adjustment.

[0095] It should be understood that prompt words can be customized for users to achieve personalized interaction, continuously optimize user experience and improve satisfaction. For example, after the pre-trained language model generates structured response content that complies with the STAR principle, it uses preset prompt words (such as "users frequently revise answers", "questions beyond the scope of the resume" and other semantic features) to determine the user's interaction behavior pattern in real time, and automatically trigger the adaptation operation when the threshold conditions are met. Specifically: during the conversation, the matching degree between the structured response content and the user's actual interaction is continuously monitored (for example, when the answer suggested by the system is manually modified by the user more than 3 times, or the user's skip rate for a certain type of question is >60%), and such behavior is marked as a high deviation pattern based on a preset machine learning classifier (such as SVM support vector machine), thereby activating the behavior pattern adaptation engine. The core operations performed by this engine include: dynamic adjustment of response strategies: reducing the priority weight of the current answer style while increasing the recommendation strength of alternative strategies; reconstruction of content recommendation logic: based on the user's historical rejection records, marking low-matching question types in the recommended question library, so that subsequent conversations automatically filter such questions; real-time fine-tuning of model parameters: through lightweight incremental learning, injecting user preference parameters into the pre-trained language model (for example, when the user repeatedly requests to shorten the answer, the model output length is automatically compressed by 40%, switching between Chinese and English, and behavioral mode switching). In this way, an adaptive interactive system is built through a closed loop of behavioral data to achieve personalized meeting assistance that becomes more accurate with use.

[0096] In some embodiments, step S400 includes:

[0097] S410, identifying the programming code in the screen selection area image and associating it with the technical question and answer context; and / or

[0098] S420: Recognize natural language text in the screen selection area image and associate it with the conference discussion context.

[0099] It can be understood that step S400 achieves accurate parsing and contextual association of the screen selection content through subdivision operations, specifically including the following two parallel processing paths: the first is step S410, which first captures an image of the screen area selected by the user, and then calls a deep learning-based code recognition model. This model has been trained with a large number of programming language syntax samples and can accurately identify the code structure, keywords, and comments in the image. After recognition, the code snippet is associated with the contextual information of the current technical question and answer. For example, when abnormal indentation is detected in the code, it is automatically associated with the previously discussed "code standard optimization" topic. The second is step S420, which extracts text content from natural language text (such as document content or chat window messages) in the screen selection, analyzes the text topic using a semantic understanding model, and then associates it with the real-time context of the meeting discussion. For example, when the selected text is identified as containing "project delay risk", it is automatically associated with the "progress management" module in the meeting agenda. Ultimately, through the collaboration of steps S410 and S420, auxiliary functions such as code explanation, error detection, or summary of discussion points are provided to users when they attend the meeting.

[0100] refer to Figure 2-Figure 4 Another embodiment of the present application provides an intelligent conference assistance system 100 for use in the intelligent conference assistance method described above, including:

[0101] The privacy isolation layer 110 is deployed between the operating system rendering layer 20 and the screen capture layer 30 and is used to exclude the target application window data from the shared stream data when generating the shared stream data based on the collected screen sharing data, and send the shared stream data excluding the target application window data to the conference terminal 10 for visual interface feedback;

[0102] The multimodal input module 120 includes: a speech recognition submodule 121 for acquiring speech data from the conference terminal in real time, transcribing the speech data, and generating speech-to-text stream data; a screenshot recognition submodule 122 for performing screenshot analysis on the screen selection image in response to a screenshot instruction and generating image-to-text stream data;

[0103] The intelligent question-answering engine 130 is connected to the multimodal input module 120 and is used to call the pre-trained language model, fuse the generated voice text stream data and / or image text stream data with the current conversation context information, and generate structured response content that complies with preset rules.

[0104] It should be understood that the starting point of screen sharing is the screen capture layer 30 at the operating system level, which captures the original pixel data of the screen or specified window through the underlying API (such as DirectX of Windows and Quartz Composer of macOS), and the format is usually uncompressed RGB / RGBA image; after the conference terminal 10 receives the shared stream data, it performs decoding and display through the rendering layer 30.

[0105] In some embodiments, the intelligent conference assistance system 100 further includes:

[0106] The resume parsing module 140 is connected to the intelligent question-answering engine 130 and is used to read and parse the resume text uploaded by the user and extract the target information data in the resume text;

[0107] The behavior adaptation module 150 is connected to the intelligent question-answering engine 130 and is used to dynamically adjust the answer strategy and content recommendation priority of the pre-trained language model of the intelligent question-answering engine 130 and / or adjust user preference parameters;

[0108] Among them, the intelligent question-answering engine 130 is also used to call the pre-trained language model to fuse the target information data extracted by the resume parsing module 140 with the voice text stream data or the image text stream data to generate interview simulation questions and answer suggestions for the resume content; and based on preset prompt words, perform behavioral pattern judgment on the structured response content that meets the preset rules.

[0109] The resume parsing module 140 in this embodiment of the present application is integrated with a file selection control for users to upload resume text; in addition, the intelligent question and answer engine 130 in this embodiment of the present application is integrated with a context manager, which is used to cache received data and data generated by the pre-trained language model.

[0110] The intelligent question and answer engine 130 generates answer content that complies with preset rules and undergoes behavioral pattern judgment. If the prompt word is met, the behavioral pattern of the behavior adaptation module 150 is triggered and the behavioral pattern adaptation operation is performed. The intelligent question and answer engine 130 is adjusted through feedback from the internal message bus 101, including dynamically adjusting the answer strategy and content recommendation priority of the pre-trained language model of the intelligent question and answer engine 130 and / or performing parameter adjustments; in addition, the intelligent question and answer engine 130 generates answer content that complies with preset rules and undergoes behavioral pattern judgment. If the prompt word is not met, it is directly input into the internal message bus 101 and then passed through the privacy isolation layer 110 for interface feedback.

[0111] In some embodiments, the privacy isolation layer 110, the multimodal input module 120, the resume parsing module 140, the intelligent question-and-answer engine 130, and the behavior adaptation module 150 are connected through an internal message bus 101 to achieve asynchronous communication, ensuring decoupling and efficient processing; wherein the speech recognition submodule 121 is connected to the speaker input interface to directly capture the conference audio voice data of the conference terminal in real time through the internal message bus 101 to avoid interference with the user's voice input.

[0112] The answer content output by the intelligent question-answering engine 130 in this embodiment can be fed back in a visual interface at the conference terminal via the internal message bus 101 .

[0113] It should be understood that the specific features, operations, and details described hereinabove with respect to the method of the present application may also be similarly applied to the apparatus and system of the present application, or vice versa. In addition, each step of the method of the present application described above may be performed by a corresponding component or unit of the apparatus or system of the present application.

[0114] It should be understood that the various modules / units of the apparatus of the present application may be implemented in whole or in part by software, hardware, firmware, or a combination thereof. Each module / unit may be embedded in the processor of the electronic device in the form of hardware or firmware or may be independent of the processor, or may be stored in the memory of the electronic device in the form of software for the processor to call to execute the operation of each module / unit. Each module / unit may be implemented as an independent component or module, or two or more modules / units may be implemented as a single component or module.

[0115] like Figure 5 As shown, the present application provides an electronic device 400, which includes a processor 401 and a memory 402 storing computer program instructions. When the processor 401 executes the computer program instructions, each step of the above-described intelligent conference assistance method is implemented. The electronic device 400 can be broadly defined as a server, a terminal, or any other electronic device with the necessary computing and / or processing capabilities.

[0116] It should be understood that the screen capture layer 30 , the rendering layer 30 , the conference terminal 10 and / or the intelligent conference assistance system 100 may be configured in the same or different electronic devices 400 .

[0117] In one embodiment, the electronic device 400 may include a processor, memory, network interface, communication interface, etc. connected via a system bus. The processor of the electronic device 400 may be used to provide necessary computing, processing, and / or control capabilities. The memory of the electronic device 400 may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system, computer programs, etc. The internal memory may provide an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface and communication interface of the electronic device 400 may be used to connect to and communicate with external devices via a network. When the computer program is executed by the processor, the steps of the method of the present application are performed.

[0118] The present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the above-mentioned intelligent conference assistance method is implemented.

[0119] Those skilled in the art will appreciate that the method steps of the present application can be performed by instructing relevant hardware such as the electronic device 400 or a processor through a computer program, and the computer program can be stored in a non-transitory computer-readable storage medium, which causes the steps of the present application to be performed when the computer program is executed. Depending on the circumstances, any reference to memory, storage or other media herein may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0120] The various technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification as long as such combination does not conflict.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent conference assistance method, characterized in that: include: In the process of generating shared stream data based on the collected screen sharing data, the target application window data is excluded from the shared stream data; Sending the shared stream data excluding the target application window data to the conference terminal for visual interface feedback; and Acquire voice data from conference terminals in real time, transcribe the voice data, and generate voice text stream data; Responding to the screenshot command, the screenshot of the screen selection image is parsed to generate image text stream data; The generated speech text stream data and / or the image text stream data are input into a pre-trained language model, and the current conversation context information is integrated to generate structured response content that complies with preset rules.

2. The intelligent conference assistance method according to claim 1, characterized in that: Also includes: Read and parse the resume text uploaded by the user, and extract the target information data in the resume text; The extracted target information data and the voice text stream data or the image text stream data are input into a pre-trained language model to generate structured response content for the resume content.

3. The intelligent conference assistance method according to claim 2, characterized in that: Also includes: Based on the preset prompt words, the behavior pattern of the structured response content that meets the preset rules is determined; When the response content that complies with the preset rules meets the prompt word, the behavior pattern is triggered and the behavior pattern adaptation operation is performed, including dynamically adjusting the response strategy and content recommendation priority of the pre-trained language model and / or performing parameter adjustments.

4. The intelligent conference assistance method according to claim 1, characterized in that: The step of excluding the target application window data from the shared stream data during the process of generating the shared stream data based on the collected screen sharing data includes: During the screen sharing initialization phase, the multimedia collaborative processing interface at the bottom layer of the operating system is called to mark the target application window as a privacy isolation object; Based on the flag, pixel data of the target application window is skipped in the shared stream generation pipeline.

5. The intelligent conference assistance method according to claim 4, characterized in that: Specifically include: Get the process identifier and graphics surface handle of the target application window; Based on the obtained process identifier and graphics surface handle, register the privacy identifier with the multimedia collaborative processing interface and associate it with the graphics rendering pipeline, and establish a mapping relationship between the identifier and the pixel area; When a shared stream is generated, the pixel region carrying the identifier is skipped based on the mapping relationship.

6. The intelligent conference assistance method according to claim 1, characterized in that: The step of excluding the target application window data from the shared stream data during the process of generating the shared stream data based on the collected screen sharing data includes: Intercept the rendering data stream of the target application window through the virtual rendering layer and perform transparency processing on its pixel area; Specifically, the method includes: locating the position of the target application window in the screen coordinate system, and setting the RGBA transparency channel value of the corresponding area to zero to achieve complete transparency.

7. The intelligent conference assistance method according to claim 1, characterized in that: The screenshot analysis of the screen selection area image in response to the screenshot instruction includes: Identify programming code within an image of a selected area of ​​the screen and associate it with the context of a technical question; and / or Recognizes natural language text in images of selected screen areas and associates them with meeting discussion context.

8. The intelligent conference assistance method according to claim 2, characterized in that: Generating structured response content that complies with preset rules includes generating structured response content that complies with the STAR principle, wherein the STAR principle includes scenario, task, action, and result elements; In the target information data extracted from the resume text, the target information data includes at least one of educational background information data, work experience information data or skill expertise information data.

9. An intelligent conference assistance system, characterized in that: The intelligent conference assistance method according to any one of claims 1 to 8 comprises: The privacy isolation layer is deployed between the operating system rendering layer and the screen capture layer. It is used to exclude the target application window data from the shared stream data generated based on the collected screen sharing data, and send the shared stream data excluding the target application window data to the conference terminal for visual interface feedback. The multimodal input module includes: a speech recognition submodule for acquiring speech data from the conference terminal in real time, transcribing the speech data, and generating speech-to-text stream data; a screenshot recognition submodule for performing screenshot analysis on the screen selection image in response to the screenshot instruction, and generating image-to-text stream data; An intelligent question-answering engine, connected to the multimodal input module, is used to call a pre-trained language model, fuse the generated voice-text stream data and / or the image-text stream data with the current conversation context information, and generate structured response content that complies with preset rules.

10. The intelligent conference assistance system according to claim 9, characterized in that: Also includes: A resume parsing module, connected to the intelligent question-answering engine, is used to read and parse the resume text uploaded by the user and extract target information data from the resume text; A behavior adaptation module, connected to the intelligent question-answering engine, for dynamically adjusting the answer strategy and content recommendation priority of the pre-trained language model and / or adjusting user preference parameters; The intelligent question-answering engine is further configured to call a pre-trained language model to fuse the extracted target information data with the voice text stream data or the image text stream data to generate simulated interview questions and answer suggestions for the resume content; And based on the preset prompt words, the behavior pattern of the structured response content that meets the preset rules is determined.

Images