Multi-device cooperation system for conference system
By embedding a unified hardware abstraction interface and a dynamic priority allocation algorithm into the Linux operating system kernel layer of the conferencing equipment, the problem of poor multi-device collaboration is solved, achieving efficient audio and video synchronization and a smooth conferencing experience, suitable for high-resolution video and low-latency audio transmission.
Patent Information
- Application Number
- CN202511993027.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-03-17
AI Technical Summary
Traditional conferencing systems suffer from poor multi-device coordination, leading to problems such as audio-visual asynchrony, sound delay, and video stuttering, resulting in a poor experience, especially in remote office scenarios with high-resolution video and real-time interaction.
By embedding a unified Hardware Abstraction Interface (HAL) into the Linux operating system kernel layer of the conference equipment, unified instruction scheduling and data synchronous transmission are achieved. Combined with dynamic priority allocation algorithms and hardware acceleration chips, resource allocation and communication latency are optimized.
It enables efficient collaboration among multiple devices, improves audio and video synchronization and smoothness, supports high-resolution video and low-latency audio transmission, and is suitable for remote office and multi-person collaboration scenarios with high real-time requirements.
Smart Images
Figure CN121691591A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of integrated communication technology, in particular to a multi-device coordination system for conference systems. BACKGROUND
[0002] When using multiple devices (such as cameras, microphones, and speakers) in traditional conference systems, these devices often operate independently, with poor coordination. This can easily lead to problems such as audio and video being out of sync, sound delay, and video stuttering during the conference, especially in remote office and multi-person collaboration scenarios that require high-resolution video and real-time interaction, resulting in a poor experience. SUMMARY
[0003] The present application provides a solution that is embedded in the Linux operating system core layer of the conference device. This system provides a unified "language" and "instruction set" (Hardware Abstraction Interface, HAL) for all different types of hardware devices, thereby achieving unified scheduling of instructions and synchronized transmission of data.
[0004] The present application provides a multi-device coordination system for conference systems, including an application layer, a core abstraction layer, a driver layer, a kernel layer, and a hardware layer. The application layer runs conference application software and sends business logic call instructions to the core abstraction layer. The core abstraction layer is a unified hardware abstraction interface layer that provides standardized device interfaces for the application layer. The core abstraction layer receives business logic call instructions from the application layer and sends them to the driver layer and the kernel layer. The driver layer and the kernel layer provide hardware drivers, dynamic priority allocation algorithms, interrupt response and memory sharing mechanisms, and CPU / co-processor resource scheduling. They send resource management and optimization instructions optimized by the dynamic priority allocation algorithm to the hardware layer, providing drivers and controls for the hardware layer devices for the core abstraction layer. The dynamic priority allocation algorithm dynamically allocates priority to each task based on the task's characteristics and the system's current state, intelligently allocates computing resources, ensures that high-priority tasks are processed first, and ensures smooth conference flow. The hardware layer includes an acceleration chip that sends raw data and signals after hardware-accelerated encoding and decoding and protocol processing to the driver layer and the kernel layer.
[0005] Preferably, the dynamic priority allocation algorithm combines the task importance factor with the initial priority P0 to dynamically adjust the current priority P of the task, with the expression as follows: The current priority P = clamp (P0 + Fi - Fr + Fd, a, b), wherein clamp means limiting the result in the interval a~b, Fi is a task importance factor, Fr is a resource occupation factor, Fd is a deadline urgency factor, P0 is an initial priority, a is a lower limit of value, and b is an upper limit of value.
[0006] Preferably, the hardware abstraction interface encapsulates a unified interface on the functions of different platforms, masks the differences of underlying hardware, is responsible for translating the control instructions of different devices into unified standard commands, and coordinates their synchronous work.
[0007] Preferably, the conference application software involves an audio transmission task, a video transmission task, a data sharing task and a system management task, the real-time requirements of the tasks are reduced in turn, and the initial priorities set are also reduced in turn.
[0008] Preferably, the dynamic priority allocation algorithm dynamically adjusts the importance factor Fi of the task according to the actual situation of the conference.
[0009] Preferably, the dynamic priority allocation algorithm dynamically adjusts the resource occupation factor Fr of the task according to the resource occupation situation of the task.
[0010] Preferably, the dynamic priority allocation algorithm dynamically adjusts the deadline urgency factor Fd of the task according to the deadline situation of the task.
[0011] Preferably, the hardware layer includes a camera, a microphone and a sound, sound and image data are collected through the microphone and the camera respectively, the camera, the microphone and the sound are single or array, the camera is used to collect image data, the microphone is used to collect sound data, and the sound is used to play sound data.
[0012] Preferably, the acceleration chip is an FPGA acceleration chip, which is used as a special acceleration engine to efficiently process audio and video coding and decoding and network transmission protocols, and to free the main CPU.
[0013] Compared with the prior art, the present application has the following beneficial effects: 1. Efficient cooperation: by constructing a unified hardware abstraction interface (HAL) in the kernel layer, the unified scheduling of instructions and synchronous transmission of data of heterogeneous devices such as cameras, microphones and array speakers are realized, and the cooperation problem of multiple devices is solved.
[0014] 2. Hardware acceleration: the FPGA chip is used on the hardware side to accelerate the hardware of the computationally intensive audio and video coding and decoding and network transmission protocols, which significantly improves the processing efficiency and reduces the burden of the main CPU.
[0015] 3. Resource intelligent scheduling: the driver layer optimizes the resource occupancy rate of CPU and hardware coprocessor through a dynamic priority allocation algorithm, and effectively reduces the multi-thread communication delay by combining the interrupt response and memory sharing mechanism, thereby ensuring the smoothness of the system.
[0016] 4. Experience improvement: The architecture aims to improve the multi-device collaboration efficiency of the conference machine in real-time video collaboration scenarios, supports high-resolution video streaming and low-latency audio transmission, reduces audio-visual synchronization errors, and is suitable for remote office and multi-person collaboration scenarios with high real-time requirements. BRIEF DESCRIPTION OF DRAWINGS
[0017] Fig. 1 The overall architecture diagram of the multi-device collaboration system for the conference system according to the present application; Fig. 2 The working data flow diagram of the multi-device collaboration system for the conference system according to the present application. DETAILED DESCRIPTION
[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0019] As shown in Figs. 1-2 The present application provides a multi-device collaboration system for a conference system, which includes an application layer, a core abstraction layer, a driver layer, a kernel layer and a hardware layer. The application layer runs conference application software, such as zoom and Tencent conference software. The conference application software sends business logic call instructions to the core abstraction layer, which sends them to the hardware layer layer by layer, calls hardware drivers such as cameras, microphones or speakers, obtains the sound data of the participants in the conference, the image data of the conference site, and calls the sound data of the remote participants and displays the image data of the remote participants on the display. When the conference application software is running, it involves audio transmission tasks, video transmission tasks, data sharing tasks and system management tasks. The audio transmission task has very high real-time requirements, and any delay may cause the sound to be incoherent, affecting the conference experience. The initial priority P0 is set to high priority, and the priority value range is a-b, where a is the lower limit of the value and b is the upper limit of the value. For example, the initial priority P0 value of the audio transmission task is set to b. The video transmission task also has high real-time requirements, but compared with the audio transmission, a certain degree of delay or quality loss is acceptable, and the initial priority P0 is set to a medium priority, the priority value range is a-b, a is the lower limit of the value, and b is the upper limit of the value, and for example, the initial priority P0 value is set to 80% (b-a). The data sharing task includes file sharing, screen sharing and the like, and has relatively low real-time requirements, and resources can be allocated after the audio and video tasks, the initial priority P0 is set to a low priority, the priority range is a-b, and for example, the initial priority P0 value is set to 60% (b-a). The system management task includes system monitoring, log recording and the like, and has the lowest real-time requirements, the initial priority P0 is set to a lowest priority, the priority value range is a-b, a is the lower limit of the value, and b is the upper limit of the value, and for example, the initial priority P0 value is set to 40% (b-a).
[0020] The core abstraction layer is a unified hardware abstraction interface layer, and the hardware abstraction interface (HAL) is above functions of different platforms, encapsulates a unified interface, shields the difference of the underlying hardware, is responsible for translating control instructions of different devices into a unified standard command, and coordinates synchronous work of the devices, to provide a standardized device interface for the application layer. The core abstraction layer receives a service logic calling instruction from the application layer, and sends the service logic calling instruction to the driver layer and the kernel layer. The driver layer and the kernel layer provide a hardware driver, a dynamic priority allocation algorithm, an interrupt response and a memory sharing mechanism, and CPU / correlation processing resource scheduling; the resource management and optimization instructions optimized by the dynamic priority allocation algorithm are sent to the hardware layer, to provide a driver and control of a hardware layer device for the core abstraction layer. The hardware driver includes hardware involved in running of the conference application software, including a camera, a microphone and a sound device, and the devices can be single or in an array.
[0021] The dynamic priority allocation algorithm dynamically allocates a priority for each task according to characteristics of the task and a current state of the system, and intelligently allocates a computing resource, to ensure that a high-priority task (such as audio transmission) is preferentially processed, and to guarantee conference fluency. The dynamic priority allocation algorithm dynamically adjusts a task importance factor according to an actual situation of the conference, for example, in a discussion link of the conference, importance of the audio transmission task needs to be further improved; and in a presentation link, importance of the video transmission and data sharing tasks can be increased. Therefore, the importance factor is set for the task according to different conference scenes, for example, in the discussion link, the value of the importance adjustment factor Fi of the audio transmission task is increased, to increase the current priority value P of the audio transmission task.
[0022] The dynamic priority allocation algorithm dynamically adjusts task priorities based on resource consumption. When a task consumes too many resources, causing other high-priority tasks to malfunction, the priority of that task is appropriately reduced by increasing the resource consumption factor Fr corresponding to that task. For example, if a video transmission task consumes too much bandwidth, causing audio transmission stuttering, the priority of the video transmission task needs to be reduced. For instance, this can be achieved by increasing the Fr value corresponding to the video transmission task, thereby lowering its current priority value P.
[0023] The dynamic priority allocation algorithm dynamically adjusts task priorities based on task deadlines by adjusting the value of the deadline urgency factor Fd. Tasks nearing their deadlines require higher priorities. For example, if an audio transmission task's next frame must be sent within 10ms, and the current time is only 2ms away from the deadline, then the task's priority needs to be raised to the highest level. For instance, this can be achieved by increasing the value of Fd to adjust the current priority P of the audio transmission task.
[0024] The dynamic priority allocation algorithm combines the above task importance factors with the initial priority P0. The formula for calculating the current priority P is: P= clamp(P0 + Fi - Fr + Fd, a, b) (1) Here, `clamp` limits the priority result to the range a~b. The values of Fi, Fr, and Fd can be configured according to the scenario, with 'a' being the lower limit and 'b' being the upper limit. For example, Fi can be 0~20, Fr 0~15, Fd 0~25, 'a' can be 0, 'b' can be 100, and P can be between 0 and 100. This formula ensures that the priority change of the task after the factors are superimposed is controllable, avoiding the situation where a single factor excessively amplifies or excessively suppresses critical tasks.
[0025] Memory sharing and interrupt response mechanisms provide the system with interrupt response mechanisms and memory sharing technology for multiple processes or threads, reducing the waiting time for communication between devices, making data exchange faster and with lower latency.
[0026] The hardware layer includes a camera, microphone, speakers, and an FPGA acceleration chip. Sound and image data are collected by the microphone and camera respectively. After the FPGA acceleration chip accelerates the encoding and decoding and protocol processing, the raw data and signals are sent to the driver layer and kernel layer. The camera, microphone, and speakers can be set up individually or in an array. The camera is used to capture image data of local participants, the microphone is used to capture sound data of local participants, and the speakers are used to play sound data of remote participants.
[0027] FPGA acceleration chip as a dedicated acceleration engine, to accelerate the processing of local camera and microphone device data collection, efficient processing of audio and video codec and network transmission protocol, liberate the CPU.
[0028] Working principle As shown in Fig. 2 The local camera and microphone device collects the sound and image data of the participants, and after FPGA hardware acceleration codec and protocol processing, the original data / signal follows the memory sharing and interrupt response management mechanism, calls the audio transmission task and the video transmission task, and the audio transmission task and the video transmission task output the audio and video data to the conference application through the unified hardware abstraction interface HAL; at the same time, the dynamic priority allocation algorithm adjusts the priority of the current task and schedules the resource allocation according to the characteristics of the task and the current state of the system, and feeds back the instruction to optimize the resource occupation to the hardware layer, and the hardware layer allocates resources according to the feedback instruction to optimize the resource occupation.
Claims
1. A multi-device collaboration system for a meeting system, the system comprising: The application layer, the core abstraction layer, the driver layer and the kernel layer and the hardware layer are included. The application layer runs conference application software and sends a service logic calling instruction to the core abstraction layer. The core abstraction layer is a unified hardware abstraction interface layer and provides a standardized device interface for the application layer. The core abstraction layer receives the service logic calling instruction from the application layer and sends it to the driver layer and the kernel layer. The driver layer and the kernel layer provide hardware driving, a dynamic priority allocation algorithm, an interrupt response and a memory sharing mechanism and CPU / co-processor resource scheduling, send resource management and optimization instructions optimized by the dynamic priority allocation algorithm to the hardware layer, and provide driving and control of the hardware layer device for the core abstraction layer. The dynamic priority allocation algorithm dynamically allocates priority to each task according to the characteristics of the task and the current state of the system, intelligently allocates computing resources, ensures that high-priority tasks are processed first, and guarantees conference fluency. The hardware layer includes an acceleration chip, which sends raw data and signals after hardware acceleration coding and protocol processing to the driver layer and the kernel layer.
2. The multi-device collaboration system for meeting systems of claim 1, wherein: The dynamic priority allocation algorithm combines the task importance factor and the initial priority P0 to dynamically adjust the current priority P of the task, and the expression is as follows: current priority P = clamp(P0 + Fi - Fr + Fd, a, b), wherein clamp represents limiting the result to the interval a~b, Fi is the task importance factor, Fr is the resource occupation factor, Fd is the deadline urgency factor, P0 is the initial priority, a is the lower limit of the value, and b is the upper limit of the value. The hardware abstraction interface is above the functions of different platforms, encapsulates a unified interface, shields the differences of the underlying hardware, is responsible for translating the control instructions of different devices into unified standard commands, and coordinates their synchronous work.
3. The multi-device collaboration system for meeting systems of claim 2, wherein: The conference application software involves audio transmission tasks, video transmission tasks, data sharing tasks and system management tasks, and the requirements for real-time performance of the tasks decrease in turn, and the initial priorities set also decrease in turn.
4. The multi-device collaboration system for meeting systems of claim 3, wherein: The dynamic priority allocation algorithm dynamically adjusts the importance factor Fi of the task according to the actual situation of the conference.
5. The multi-device collaboration system for meeting systems of claim 4, wherein: The dynamic priority allocation algorithm dynamically adjusts the resource occupation factor Fr of the task according to the resource occupation situation of the task.
6. The multi-device collaboration system for meeting systems of claim 5, wherein: The dynamic priority allocation algorithm dynamically adjusts the deadline urgency factor Fd of the task according to the deadline situation of the task.
7. The meeting system oriented multi-device collaboration system of claim 6, wherein: The hardware layer includes a camera, a microphone and a sound, which respectively collect sound and image data through the microphone and the camera, and the camera, the microphone and the sound are single or array, the camera is used to collect image data, the microphone is used to collect sound data, and the sound is used to play sound data.
8. The multi-device collaboration system for meeting systems of claim 7, wherein: The acceleration chip is an FPGA acceleration chip, which is used as a special acceleration engine, efficiently processes audio and video coding and network transmission protocols, and frees the main CPU.
9. The multi-device collaboration system for meeting systems of claim 8, wherein: