Intelligent conference multi-mode cooperative control and intelligent processing method
Through the interconnection of PC hosts and conference all-in-ones, combined with private real-time communication protocols and AI technology, the problem of insufficient intelligence in traditional conference mode is solved, remote control and intelligent screen are realized, and meeting efficiency and convenience are improved.
Patent Information
- Application Number
- CN202411028692.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
The traditional conference model is not intelligent and convenient, which makes the conference process cumbersome and difficult to meet actual needs.
Through the interconnection of PC host and conference all-in-one, private real-time communication protocol MSC is adopted to realize remote control, intelligent same-screen and AI empowerment, and combine multi-person face recognition and voice recognition technology to generate meeting minutes to improve the intelligence of conference management. and efficient.
It realizes the remote control function of large-screen devices by PC and mobile devices, improves server throughput, reduces response delay, saves system resources, simplifies the meeting process, and significantly improves work efficiency.
Smart Images

Figure CN118917819A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of collaborative control and relates to a method for multi-modal collaborative control and intelligent processing of intelligent conferences. Background Art
[0002] Traditional meeting models are often limited by their lack of intelligence and convenience, and are unable to meet the actual needs of work. Summary of the invention
[0003] In order to make up for the deficiencies of the prior art, the present invention provides a method for multi-modal collaborative control and intelligent processing of intelligent conferences, which solves the pain points of users in actual meetings, such as cumbersome processes and insufficient intelligence. Through the interconnection of the PC host and the all-in-one conference machine, remote control, intelligent screen sharing and AI empowerment can be achieved, providing a new intelligent and convenient way for meetings at work. The present invention is implemented through the following technical solutions: The present invention provides a method for multi-modal collaborative control and intelligent processing of intelligent conferences, which includes the following steps: Step 1: Private real-time communication protocol MSC; By integrating the open source framework Netty and performing secondary encapsulation on its basis, the JSON format commonly used in data transmission is adopted, and only two types of data, type and message, are defined; By defining different type values, different control types are distinguished; message uses the Object data type, and control content information can be transmitted when processing different control types; Step 2: Use the PC host to control the content displayed on the conference all-in-one machine in real time, and control multiple screens at the same time; Step 3: Convert different meeting files in doc, xls, ppt, and txt formats and efficiently transmit them to front-end devices for display to ensure seamless connection of meeting materials; Step 4: Remotely control the all-in-one conference machine to collect and upload photos and audio of the conference site in real time; the PC sends different MSC messages and distinguishes the operation type through the type field; after issuing the operation instruction, the all-in-one conference machine uploads the data collection result corresponding to the operation instruction through the HTTP interface; Step 5: Analyze the reported conference pictures in real time for face recognition; integrate compareFace with a multi-person face algorithm and encapsulate it through docker to provide API services to the outside world, providing face input and face recognition interface entrances; after the all-in-one conference machine collects and uploads the face picture, the back-end program calls the face interface for comparison; after the comparison is completed, the result is returned to the PC host and the all-in-one conference machine meeting page in real time; Step 6: Remotely control the conference all-in-one machine to collect on-site audio, and integrate the Whisper algorithm for speech recognition; Step seven: Use the Kimi large language model to summarize and polish the speech recognition content, summarize the recognized text, and generate meeting minutes with one click.
[0004] The beneficial effects of the present invention are: A method for realizing multi-modal collaborative control and intelligent processing of intelligent conferences based on the MSC protocol, realizing the remote control function of PC and mobile devices for large-screen devices. While providing two-way interaction, it improves server throughput, reduces response delay, saves system resources, and reduces unnecessary memory copies.
[0005] The present invention solves the pain points of users in actual meetings, such as cumbersome processes and lack of intelligence. By interconnecting the PC host and the all-in-one conference machine, remote control, intelligent screen sharing and AI empowerment can be achieved, providing a new intelligent and convenient way for meetings at work.
[0006] The present invention innovatively combines the cutting-edge technology of the all-in-one smart conference machine with the extensive connectivity of the cloud platform, and relies on the high-performance network programming framework Netty to achieve seamless intelligent linkage between the small screen on the PC and the large screen on the all-in-one conference machine.
[0007] By deeply integrating the rapidly developing AI technology, we have not only injected unprecedented intelligence and flexibility into the product, but also greatly simplified the originally cumbersome and complicated process in the meeting, and realized the intelligent, automated and efficient management of the meeting. This innovative solution aims to reshape the meeting experience, so that every meeting can be conducted in a faster, more convenient and intelligent way, thereby significantly improving work efficiency and pushing work to a new level. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The present invention will be further described below in conjunction with the accompanying drawings; Figure 1 is a flow chart of the overall method of the present invention; Figure 2 Configuration file for starting the MSC protocol long link service; Figure 3 Configuration file for starting the LibreOffice API service; Figure 4 Configuration file for operating the conference control page. DETAILED DESCRIPTION
[0009] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.
[0010] The accompanying drawing is a specific embodiment of a method for multi-modal collaborative control and intelligent processing of an intelligent conference according to the present invention.
[0011] A method for multi-modal collaborative control and intelligent processing of intelligent conferences, comprising the following steps: Step 1: Private real-time communication protocol MSC (Message Send Control); by integrating the open source framework Netty, it retains the excellent features of the original framework, such as high performance and high throughput, high scalability, rich protocol support, robust network communication capabilities, memory management mechanism, and asynchronous and event-driven. It is then repackaged on this basis, using the JSON format commonly used in data transmission, and only defines two types of data, type and message; different types of control are distinguished by defining different type values; message uses the Object data type, and can transmit control content information when processing different control types.
[0012] Step 2: Use the PC host to control the content displayed on the all-in-one conference machine in real time, and control multiple screens on the same screen. In step 2, the PC host provides a unified entrance to the meeting. After entering the meeting, all meeting files and data links are displayed; by defining different operation types, the private real-time communication protocol MSC is used to pass messages to the all-in-one conference machine; the content displayed on the PC host will be displayed on the all-in-one conference machine synchronously, so as to achieve the same-screen display; the MAC address of the all-in-one conference machine is used as the unique identifier for remote control.
[0013] Step 3: Convert different meeting files in doc, xls, ppt, and txt formats, and efficiently transmit them to front-end devices for display to ensure seamless connection of meeting materials. Integrate the open source LibreOffice all-purpose office software into the step 3 project, repackage it, and provide API interface services to the outside world; and automatically start and destroy the LibreOffice API service when the project is started and closed.
[0014] Step 4: Remotely control the all-in-one conference machine to collect and upload photos and audio of the conference site in real time; the PC sends different MSC messages and distinguishes the operation type through the type field; after issuing the operation instruction, the all-in-one conference machine uploads the data collection result corresponding to the operation instruction through the HTTP interface.
[0015] Step 5: Analyze the reported conference pictures in real time for face recognition; integrate compareFace with a multi-person face algorithm, encapsulate it through docker, provide API services to the outside, and provide face input and face recognition interface entrances; after the all-in-one conference machine collects the face picture and uploads it, the back-end program calls the face interface for comparison; after the comparison is completed, the result is returned to the PC host and the all-in-one conference machine meeting page in real time.
[0016] Step 6: Remotely control the conference machine to collect on-site audio, and integrate the Whisper algorithm for voice recognition. In step 6, voice recognition integrates the open source Whisper algorithm, which is then encapsulated into a Whisper-UI image to provide page operations while exposing HTTP API to the outside world; after the conference machine reports the audio, real-time voice recognition is performed.
[0017] Step seven: Use the Kimi large language model to summarize and polish the speech recognition content, summarize the recognized text, and generate meeting minutes with one click.
[0018] The method of multi-modal collaborative control and intelligent processing of intelligent conferences includes the following specific steps: (1) Server startup; a) Start the MSC protocol long link service; b) Start the LibreOffice API service; c) Start the CompareFace Docker image and provide API services; d) Start the Whisper-UI Docker image and provide API services; (2) After the user uploads the meeting file, determine whether it is a PDF file. If it is, directly call the LibreOffice API service to convert it into an image; other files are converted into PDF and then converted into images; (3) After the user logs in to the platform and enters the conference control page, he / she selects the corresponding control method, such as starting to play the file content, using the MAC address as the unique identifier of the control device, and sending the control signaling to the smart conference all-in-one machine through the MSC message; the PC plays the file content for operator control, while the all-in-one machine displays synchronously at the same time; (4) After the user clicks to take a photo, the MSC signaling is sent. After the all-in-one device receives it, it uploads the photo. After the server receives the image file, it calls the CompareFace API interface for face recognition. After the recognition result is returned, the backend service modifies the check-in status of the person. (5) After the user clicks to start or end the recording operation, the platform sends out recording-related signals; after the all-in-one device receives the signal, it will collect the on-site sound and finally generate an MP3 file and upload it to the platform; after the platform receives the audio file, it passes the file to the Whisper UI algorithm for inference; after returning the corresponding text content, the platform generates a txt file and saves it locally; and provides it to the PC user for viewing and modification; (6) Users click on the one-click function to generate meeting minutes, and the platform uploads the generated txt file to the Kimi large language model; using its text content analysis and multimodal capabilities, the meeting content is summarized; and finally the generated meeting minutes are returned to the platform, which then generates a txt file again, saves it locally, and provides it to platform users for viewing and modification.
[0019] Specific embodiments of the present invention include: (1) Integrate the high-performance network programming framework Netty to establish the server required for communication. It is based on Java NIO (New Input / Output) to support the rapid development of reliable network applications. Compared with BIO (Blocking I / O), its concurrency performance has been greatly improved. It is then encapsulated again to form the private real-time communication protocol MSC. This protocol uses the JSON format commonly used in data transmission. It only defines two types of data in the data content: type and message. Type is the message type, and message stores specific business data. This realizes the function of private business data definition.
[0020] (2) By defining a series of type identifiers with specific values, the system can distinguish and perform different control operations. These operations are designed to achieve flexible display and navigation of file content, while supporting cross-device display functions such as PC and all-in-one conference machine.
[0021] The specific control operations are defined as follows: 1. Start playing the file (CONTROL_FILE_START, value "2007"); When receiving this command, the system will parse the attached message to obtain the image URL address; The PC then loads and displays the image content pointed to by the URL; At the same time, the all-in-one conference machine will also synchronously display the same picture content to ensure seamless transmission of information and consistency of display.
[0022] 2. File page turning control: Previous page (CONTROL_FILE_LAST, value "2008"); Next page (CONTROL_FILE_NEXT, value "2009").
[0023] 3. File playback ends (CONTROL_FILE_END, value "2010"); When this command is received, the system will stop the current file playback process, and the presentation on both the PC and the all-in-one conference machine will end synchronously.
[0024] 4. Unified equipment management; The system uses the MAC address of the conference all-in-one machine as a unique identifier to achieve accurate identification and management of the device. This mechanism ensures that the control command can accurately point to the target device, achieving efficient screen directional control and same-screen display functions.
[0025] (3) After the user uploads the conference file, the system first uses the self-encapsulated LibreOffice API to perform an efficient file processing process. The core of this process is to seamlessly convert non-PDF format files into PDF format, and then further convert the PDF document into a sequence of images. This design cleverly avoids the direct dependence of the front-end device on multiple file formats, eliminating the tedious installation and maintenance of third-party plug-ins. It also significantly improves the response speed when displaying files, especially in file switching and page turning operations, reducing waiting time and ensuring the smoothness and efficiency of the same-screen control of the meeting. Through this series of automated conversions, not only is the technical implementation simplified, but the user experience is also greatly improved.
[0026] (4) When the PC sends the photo taking command CONTROL_TAKE_PHOTO_SIGN("2017") to the all-in-one conference machine, the all-in-one conference machine immediately captures the on-site picture and reports the picture to the server through the HTTP interface. The picture is not only displayed in real time on the PC page, but also saved as a photo of the conference site. Similarly, the PC triggers the all-in-one conference machine to record the on-site sound by sending the start recording command CONTROL_AUDIO_START("2014"). During the recording process, the all-in-one conference machine continues to capture audio data; once it receives the stop recording command CONTROL_AUDIO_STOP("2015"), the all-in-one conference machine immediately ends the recording and reports the complete recording file to the server through the HTTP interface for subsequent processing or storage.
[0027] (5) Use Docker to package the face recognition algorithm compareFace into an image package, which is convenient for deployment and exposes the API interface service. Simplify the operations of face input and face recognition during the calling process; compare the collected images through the API calling algorithm to finally obtain the face recognition results.
[0028] (6) The ASR speech recognition algorithm Whisper is exported to the image package Whisper-UI through Docker, which is convenient for deployment and exposes the API interface service. The collected audio is recognized through the API call algorithm and a text record is formed.
[0029] (7) Connect to Kimi's large language model, access the platform through the API Key, and call its dialogue interface. After the user clicks one button to generate the meeting minutes, the text content after the previous speech recognition is summarized and the AI capabilities of the large language model are used to finally generate the meeting minutes.
Claims
1. A method for multi-modal collaborative control and intelligent processing of intelligent conferences, characterized in that: The following steps are included: Step 1: Private real-time communication protocol MSC; By integrating the open source framework Netty and performing secondary encapsulation on its basis, the JSON format commonly used in data transmission is adopted, and only two types of data, type and message, are defined; By defining different type values, different control types are distinguished; message uses the Object data type, and control content information can be transmitted when processing different control types; Step 2: Use the PC host to control the content displayed on the conference all-in-one machine in real time, and control multiple screens at the same time; Step 3: Convert different meeting files in doc, xls, ppt, and txt formats and efficiently transmit them to front-end devices for display to ensure seamless connection of meeting materials; Step 4: Remotely control the all-in-one conference machine to collect and upload photos and audio of the conference site in real time; the PC sends different MSC messages and distinguishes the operation type through the type field; after issuing the operation instruction, the all-in-one conference machine uploads the data collection result corresponding to the operation instruction through the HTTP interface; Step 5: Analyze the reported conference pictures in real time for face recognition; integrate compareFace with a multi-person face algorithm and encapsulate it through docker to provide API services to the outside world, providing face input and face recognition interface entrances; after the conference all-in-one machine collects and uploads the face picture, the back-end program calls the face interface for comparison; After the comparison is completed, the results will be returned to the PC host and the conference page of the all-in-one conference machine in real time; Step 6: Remotely control the conference all-in-one machine to collect on-site audio, and integrate the Whisper algorithm for speech recognition; Step seven: Use the Kimi large language model to summarize and polish the speech recognition content, summarize the recognized text, and generate meeting minutes with one click.
2. The method for multi-modal collaborative control and intelligent processing of intelligent conference according to claim 1 is characterized in that: In step 2, the PC host provides a unified entrance to the meeting. After entering the meeting, all meeting files and data links are displayed. By defining different operation types, the private real-time communication protocol MSC is used to pass messages to the all-in-one conference machine. The content displayed on the PC host will be displayed on the all-in-one conference machine synchronously, so as to achieve same-screen display. Among them, the MAC address of the all-in-one conference machine is used as the unique identifier for remote control.
3. The method of multi-modal collaborative control and intelligent processing of intelligent conference according to claim 1 is characterized in that: Integrate the open source LibreOffice all-purpose office software into the project in step 3, repackage it, and provide API interface services to the outside world; and automatically start and destroy the LibreOffice API service when the project is started and closed.
4. The method for multi-modal collaborative control and intelligent processing of intelligent conference according to claim 1 is characterized in that: In step 6, the speech recognition integrates the open source Whisper algorithm, which is then encapsulated into a Whisper-UI image to provide page operations while exposing the HTTP API to the outside world. After the all-in-one conference machine reports the audio, real-time speech recognition is performed.
5. The method of multi-modal collaborative control and intelligent processing of intelligent conference according to claim 1 is characterized in that: The specific steps are: (1) Server startup; a) Start the MSC protocol long link service; b) Start the LibreOffice API service; c) Start the CompareFace Docker image and provide API services; d) Start the Whisper-UI Docker image and provide API services; (2) After the user uploads the meeting file, determine whether it is a PDF file. If it is, directly call the LibreOffice API service to convert it into an image; other files are converted into PDF and then converted into images; (3) After the user logs in to the platform and enters the conference control page, he / she selects the corresponding control method, uses the MAC address as the unique identifier of the control device, and sends the control signaling to the smart conference all-in-one machine through the MSC message; the PC plays the file content for operator control, while the all-in-one machine displays synchronously at the same time; (4) After the user clicks to take a photo, the MSC signaling is sent. After the all-in-one device receives it, it uploads the photo. After the server receives the image file, it calls the CompareFace API interface for face recognition. After the recognition result is returned, the backend service modifies the check-in status of the person. (5) After the user clicks to start or end the recording operation, the platform sends out recording-related signals; after receiving the signals, the all-in-one machine will collect the on-site sound and finally generate an MP3 file and upload it to the platform; After receiving the audio file, the platform passes the file to the Whisper UI algorithm for inference; After returning the corresponding text content, the platform will save the generated txt file locally and provide it to PC users for viewing and modification; (6) The user clicks on the one-click function to generate meeting minutes, and the platform uploads the generated txt file to the Kimi large language model; summarizes the meeting content; and finally generates the meeting minutes and returns them to the platform, which then generates a txt file again and saves it locally for platform users to view and modify.
Citation Information
Cited By
Conference live broadcast method and system
CN120602691A
Conference live broadcast method and system
CN120602691B
Information display visualization system applied to conference tablet computer
CN120803382A
