Software conference system based on a honk operating system

Abstract

The application relates to the technical field of video conference software, in particular to a software conference system based on a Hongmeng operating system, which comprises a scheduling platform module arranged at a user terminal, the scheduling platform module is used for completing the registration and login of users, and is also used for completing call connection to complete the establishment of a conference; a Hongmeng terminal module arranged at the user terminal and connected with the scheduling platform module at each user terminal, the Hongmeng terminal module is used for initiating a conference to establish a point-to-point conference or a multi-party conference. The conference system of the application arranges the scheduling platform module at the user terminal, adopts a distributed design, solves the scheduling disaster recovery and bandwidth bottleneck problems through decentralization, so that regional video conferences can still be established under any node failure, bandwidth limitation and the like.

Description

Technical Field

[0001] This invention relates to the technical field of video conferencing software, and specifically to a video conferencing system based on the HarmonyOS operating system. Background Technology

[0002] With the development of audio and video technology, chip technology, mobile networks, and satellite communications, the market demand for video conferencing terminals in the civilian sector is continuously growing. As remote work becomes more widespread, businesses are increasingly demanding flexible conferencing solutions, especially for facilitating communication on the go during business trips and improving time efficiency. The development of 5G networks makes video conferencing connections more stable and faster, enhancing the user experience. It can be effectively applied in enterprise business, education and training, government, and public services.

[0003] Existing conferencing systems connect parties by designing a control center (or server). As the number of meetings increases, a control center solution will encounter problems such as scheduling, disaster recovery, and bandwidth bottlenecks, making it impossible to establish new video conferences. Therefore, it is necessary to provide a new solution. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a software conferencing system based on the HarmonyOS operating system, which solves the problems of scheduling disaster recovery and bandwidth bottlenecks in existing conferencing systems that rely on a control center for conferencing connections.

[0005] The technical solution to achieve the above objectives is:

[0006] This invention provides a software conferencing system based on the HarmonyOS operating system, comprising:

[0007] The scheduling platform module is located at the user terminal. The scheduling platform module is used to complete user registration and login, and also to complete call connection to establish a conference.

[0008] The HarmonyOS terminal module is located at the user terminal and establishes a connection with the scheduling platform module at each user terminal. The HarmonyOS terminal module is used to initiate meetings to facilitate the establishment of point-to-point meetings or multi-party meetings.

[0009] A further improvement of the software conferencing system based on the HarmonyOS operating system in this invention is that the scheduling platform module includes data synchronization service and SIP service;

[0010] The data synchronization service is used to complete user login authentication. After successful authentication, it returns SIP account information and data synchronization service address of other user terminals to the HarmonyOS terminal module at the same user terminal.

[0011] The SIP service is used to store user information and to implement call control to establish, maintain and release calls, facilitating the establishment of communication connections.

[0012] A further improvement of the software conferencing system based on the HarmonyOS operating system of the present invention is that the HarmonyOS terminal module includes a conferencing application and a media exchange process;

[0013] The conferencing application connects to the data synchronization service at the same user terminal. After receiving the SIP account information returned by the corresponding data synchronization service, the conferencing application registers all media exchange processes to the SIP service corresponding to the SIP account information. It is also used to establish a connection with the data synchronization service of other user terminals based on the data synchronization service address of the other user terminals received.

[0014] The media exchange process is used to establish call connections with other participants.

[0015] A further improvement of the software conferencing system based on the HarmonyOS operating system in this invention is that when the conferencing application at user terminal A initiates a point-to-point call conference, it directly registers with the SIP service at the called user terminal B.

[0016] After successful registration, the conferencing application on user terminal A initiates a SIP invitation with the called number of user terminal B. Once user terminal B accepts the SIP invitation, a point-to-point conference is established.

[0017] A further improvement of the software conferencing system based on the HarmonyOS operating system in this invention is that when the conferencing application on user terminal A initiates a multi-party conference, the conferencing application on user terminal A allocates a corresponding number of media exchange processes according to the number of participants.

[0018] The conferencing application of user terminal A requests the data synchronization service of user terminal A to establish a meeting;

[0019] User terminal A's data synchronization service sends meeting invitation notifications to the participants;

[0020] After the participants' terminals accept the request, the conference application on the participants' terminals allocates an appropriate number of media exchange processes according to the number of participants, and sends the list of allocated media exchange processes to the data synchronization service of user terminal A.

[0021] The data synchronization service of user terminal A allocates the media exchange process between the caller and the called party according to the order of joining the meeting, and notifies the calling party's conference application of the caller-called relationship.

[0022] The calling party's conference application controls the corresponding media exchange process to initiate a call. After registering with the SIP service of the called party, the multi-party conference is established upon completion of the call.

[0023] A further improvement of the software conferencing system based on the HarmonyOS operating system in this invention is that, when the participants include user terminal B and user terminal C, if user terminal B joins the meeting before user terminal C, the calling and called relationship formed by the media exchange process is as follows:

[0024] The first media switching process of user terminal A calls the first media switching process of user terminal B.

[0025] The second media switching process of user terminal A calls the first media switching process of user terminal C;

[0026] User terminal B's second media switching process calls user terminal C's second media switching process.

[0027] A further improvement of the software conferencing system based on the HarmonyOS operating system of the present invention is that the conferencing application is also used to access the corresponding third-party conference through the SIP service based on the information of the third-party conference.

[0028] A further improvement of the software conferencing system based on the HarmonyOS operating system of the present invention is that the conferencing application also includes a conferencing management module;

[0029] The meeting management module provides an operation interface for users to operate, and also provides functions for initiating, joining, leaving, and recording meetings.

[0030] A further improvement of the software conferencing system based on the HarmonyOS operating system of the present invention is that the conferencing application also includes a conferencing recording module;

[0031] The meeting recording module is used to record the meetings that users have participated in for easy retrieval.

[0032] A further improvement of the software conferencing system based on the HarmonyOS operating system in this invention is that the scheduling platform also includes a network detection service, which is used to send probe data packets to each user terminal in order to perceive and display the prior status of each user terminal.

[0033] The beneficial effects of the software conferencing system based on the HarmonyOS operating system of this invention are as follows:

[0034] The conferencing system of this invention places the scheduling platform module at the user terminal and adopts a distributed design. It solves the problems of scheduling disaster recovery and bandwidth bottlenecks through decentralization, so that regional video conferences can still be established even if any node fails or bandwidth is limited.

[0035] The conference system of the present invention includes a scheduling platform module and a HarmonyOS terminal module, which can perform point-to-point video calls and multi-person video conferencing.

[0036] The scheduling platform of the conference system of the present invention includes SIP service, which enables it to conduct video conferences with other standard SIP terminals, that is, it can directly connect to third-party conferences (as long as the third-party conference also uses the standard protocol SIP). Attached Figure Description

[0037] Figure 1 This is an architecture diagram showing the connection between multiple user terminals and a third-party conferencing platform in the software conferencing system based on the HarmonyOS operating system of this invention.

[0038] Figure 2 This is a schematic diagram of the communication connection during the initial registration of the software conferencing system based on the HarmonyOS operating system of this invention.

[0039] Figure 3 This is a timing diagram of point-to-point calls in the software conferencing system based on the HarmonyOS operating system of this invention.

[0040] Figure 4 This is a timing diagram of a multi-party meeting in the software conferencing system based on the HarmonyOS operating system of this invention.

[0041] Figure 5 This is a schematic diagram illustrating the interaction between modules in a multi-party conference using the software conferencing system based on the HarmonyOS operating system of this invention.

[0042] Figure 6 This is a timing diagram of user login in the software conferencing system based on the HarmonyOS operating system of this invention.

[0043] Figure 7 This is a schematic diagram of the user login interface in the software conferencing system based on the HarmonyOS operating system of this invention.

[0044] Figure 8 This is a sequence diagram of contact management in the software conferencing system based on the HarmonyOS operating system of this invention.

[0045] Figure 9 This is a schematic diagram of the contact management interface in the software conferencing system based on the HarmonyOS operating system of this invention.

[0046] Figure 10 This is a schematic diagram of the interface for initiating a meeting in the software conferencing system based on the HarmonyOS operating system of this invention.

[0047] Figure 11 This is a schematic diagram of the interface for joining a meeting in the software conferencing system based on the HarmonyOS operating system of this invention.

[0048] Figure 12 This is a schematic diagram of the interface for multi-party conferencing in the software conferencing system based on the HarmonyOS operating system of this invention.

[0049] Figure 13This is a timing diagram of meeting records in the software conferencing system based on the HarmonyOS operating system of this invention.

[0050] Figure 14 This is a schematic diagram of the meeting record list interface in the software conferencing system based on the HarmonyOS operating system of this invention.

[0051] Figure 15 This is a schematic diagram of a meeting detail in the software conferencing system based on the HarmonyOS operating system of this invention.

[0052] Figure 16 This is a timing diagram of adjusting the camera in the software conferencing system based on the HarmonyOS operating system of this invention.

[0053] Figure 17 This is a schematic diagram of the gimbal control interface in the software conferencing system based on the HarmonyOS operating system of this invention.

[0054] Figure 18 This is a schematic diagram of the terminal settings in the software conferencing system based on the HarmonyOS operating system of this invention.

[0055] Figure 19 This is a schematic diagram of the interface settings in the software conferencing system based on the HarmonyOS operating system of this invention.

[0056] Figure 20 This is a schematic diagram of the network detection interface in the software conferencing system based on the HarmonyOS operating system of this invention.

[0057] Figure 21 This is an architecture diagram of the HarmonyOS system, which is the basis of the software conferencing system of the present invention.

[0058] Figure 22 This is a flowchart illustrating the U-Boot customization and optimization of the software conferencing system based on the HarmonyOS operating system of this invention.

[0059] Figure 23 This is a diagram of the driver framework architecture for the software conferencing system based on the HarmonyOS operating system of this invention.

[0060] Figure 24 This is a flowchart of the USB bus driver for the software conferencing system based on the HarmonyOS operating system of this invention.

[0061] Figure 25 This is a framework diagram of the SDK IF for the USB bus driver of the software conferencing system based on the HarmonyOS operating system of this invention.

[0062] Figure 26 This is an architecture diagram of the UART bus driver for the software conferencing system based on the HarmonyOS operating system of this invention.

[0063] Figure 27This invention relates to the architecture diagram of the audio driver for a software conferencing system based on the HarmonyOS operating system. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0065] See Figure 1 This invention provides a software conferencing system based on the HarmonyOS operating system, designed to meet the needs of high-quality, stable, and reliable video conferencing in various environments. The software conferencing system of this invention possesses multiple functions, enabling multi-party audio and video conferencing, multi-screen display, audio and video recording, and other operations, while meeting requirements such as independent controllability, high performance, and low power consumption. In particular, this invention integrates a distributed scheduling platform module into each user terminal, solving problems such as scheduling disaster recovery and bandwidth bottlenecks through decentralization, enabling regional video conferences to be established even in the event of node failure or bandwidth limitations. The software conferencing system based on the HarmonyOS operating system of this invention will be described below with reference to the accompanying drawings.

[0066] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0067] See Figure 1 This diagram illustrates the architecture of the software conferencing system based on the HarmonyOS operating system, connecting multiple user terminals to a third-party conferencing platform. The following section, in conjunction with... Figure 1 The present invention describes a software conferencing system based on the HarmonyOS operating system.

[0068] like Figure 1 As shown, the software conferencing system based on the HarmonyOS operating system of the present invention includes a scheduling platform module and a HarmonyOS terminal module. Both the scheduling platform module and the HarmonyOS terminal module are located at the user terminal. The scheduling platform module is used to complete user registration and login, and also to complete call connection to establish a conference. The HarmonyOS terminal module establishes a connection with the scheduling platform module at each user terminal. The HarmonyOS terminal module is used to initiate a conference to facilitate the establishment of point-to-point conferences or multi-party conferences.

[0069] The software conferencing system of the present invention can realize point-to-point video calls between terminals, multi-person video conferencing functions, etc. It can also connect to third-party conferencing systems, such as Huawei conferencing platform, and can also conduct video conferencing with other standard SIP terminals.

[0070] Compared to traditional conferencing systems, the key feature of this invention is that the scheduling platform module is distributed and built into each terminal. By decentralizing the solution, problems such as scheduling disaster recovery and bandwidth bottlenecks are solved, enabling regional video conferences to be established even if any node fails or bandwidth is limited.

[0071] Furthermore, the scheduling platform module is deployed on a single board based on the RK3588 chip and running the Open Euler operating system. The functions performed by the scheduling platform module include: supporting address book management, registration and call connection for each terminal, network NAT traversal, assisting terminals in QoS network quality assurance, and interfacing with the superior scheduling system.

[0072] The HarmonyOS terminal module is deployed on a single board based on the RK3588 chip, which runs the open-source HarmonyOS operating system (the adaptation design for the HarmonyOS system will be described in detail later). It implements conferencing functionality based on the standard SIP protocol. Core functions include: establishing point-to-point or multi-party conferences, conference control (adding or removing participants, muting / unmute, etc.), multi-screen layout, and audio / video processing.

[0073] In one specific embodiment of the present invention, the scheduling platform module of the present invention includes a data synchronization service and a SIP service; the data synchronization service is used to complete the user's login authentication, and after the authentication is successful, it returns the SIP account information and the data synchronization service address of other user terminals to the HarmonyOS terminal module at the same user terminal; the SIP service is used to store user information, and is also used to implement call control to establish, maintain and release calls, so as to facilitate the establishment of communication connections.

[0074] Furthermore, the Data Synchronization Service (DSS) is responsible for functions such as user authentication, terminal status maintenance, and signaling message relay. This service adopts the mesh control protocol (a self-developed protocol) to provide user terminals with services such as message forwarding, meeting management, meeting control, and meeting member status synchronization.

[0075] The functions of SIP services include: user registration and management, registering users and storing user information so that other users can find them; call control, responsible for establishing, maintaining and releasing calls to ensure smooth communication; address resolution, converting users' SIP addresses into IP addresses to establish communication connections; session management, managing the entire process of sessions, including session establishment, maintenance and release; and media negotiation, negotiating parameters such as the transmission method and codec of media streams to ensure communication quality.

[0076] Furthermore, the scheduling platform also includes a network detection service, which sends probe data packets to each user terminal to detect and display the prior state of each user terminal. The network detection service detects the prior state of each user terminal for other terminals to query.

[0077] In one specific embodiment of the present invention, the HarmonyOS terminal module includes a conferencing application (APP) and a media exchange process (MSP). The conferencing application is connected to the data synchronization service at the same user terminal. After receiving the SIP account information returned by the corresponding data synchronization service, the conferencing application is used to register all media exchange processes to the SIP service corresponding to the SIP account information. It is also used to establish a connection with the data synchronization service of other user terminals based on the data synchronization service address received from the other user terminals. The media exchange process is used to establish a call connection with other participants.

[0078] The conferencing application is responsible for functions such as user interface and media session management, while the MSP is responsible for functions such as establishing calls and audio / video transmission with other participants. In point-to-point calls, the conferencing application also undertakes the functions of establishing calls and audio / video transmission (at which time the MSP does not need to participate).

[0079] like Figure 2 As shown, during initial registration, after the terminal starts up, the APP authenticates with the DSS service of this terminal. After successful authentication, the DSS returns the SIP account information and the DSS service addresses of all terminals in the system to the APP. After receiving the information, the APP instructs all MSPs to register with the SIP service (red line in the figure) and establishes long WebSocket connections to the DSS services of all terminals (green line in the figure).

[0080] Furthermore, such as Figure 3 As shown, when a conference application at user terminal A initiates a point-to-point call conference, it directly registers with the SIP service at the called user terminal B.

[0081] After successful registration, the conferencing application on user terminal A initiates a SIP invitation with the called number of user terminal B. Once user terminal B accepts the SIP invitation, a point-to-point conference is established.

[0082] The point-to-point call process is as follows: The user initiates a point-to-point call on terminal A; Terminal A determines that it is a point-to-point call and does not require the participation of an MSP, so it directly registers with the SIP service of terminal B; after successful registration, it initiates a SIP INVITE with the called number of terminal B; the subsequent process is the standard SIP call process, which will not be described in detail here.

[0083] Furthermore, such as Figure 4 As shown, when a conference application on user terminal A initiates a multi-party conference, the conference application on user terminal A allocates a corresponding number of media exchange processes based on the number of participants.

[0084] The conferencing application of user terminal A requests the data synchronization service of user terminal A to establish a meeting;

[0085] User terminal A's data synchronization service sends meeting invitation notifications to the participants;

[0086] After the participants' terminals accept the request, the conference application on the participants' terminals allocates an appropriate number of media exchange processes according to the number of participants, and sends the list of allocated media exchange processes to the data synchronization service of user terminal A.

[0087] The data synchronization service of user terminal A allocates the media exchange process between the caller and the called party according to the order of joining the meeting, and notifies the calling party's conference application of the caller-called relationship.

[0088] The calling party's conference application controls the corresponding media exchange process to initiate a call. After registering with the SIP service of the called party, the multi-party conference is established upon completion of the call.

[0089] The multi-party conference initiation process is as follows: 1. After selecting participants (B, C) on terminal A, the user initiates the conference; 2. The app module of terminal A allocates the corresponding number of MSPs (Media Exchange Processors) based on the number of participants; 3. The app requests the establishment of the conference from the DSS (Data Synchronization Service) of this terminal; 4. The DSS sends conference invitation notifications to the participants; 5. After user B accepts the invitation, the app allocates the corresponding number of MSPs based on the number of participants; 6. The MSP module list is sent to A's DSS; 7. The DSS allocates the calling and called MSPs according to the order of joining (A-MSP1 calls B-MSP1); 8. The DSS notifies the calling app of the calling party's calling party relationship; 9. The calling app instructs the corresponding MSP to initiate the call; 10. After the MSP registers with the SIP service of the called party, the call is completed (standard SIP call process, the same as point-to-point call, not described here); 11. The subsequent process for user C to join the conference is similar to steps 5-10 above.

[0090] When the participants include user terminal B and user terminal C, if user terminal B joins the meeting before user terminal C, the calling and called relationship formed by the media switching process is as follows:

[0091] The first media switching process of user terminal A calls the first media switching process of user terminal B.

[0092] The second media switching process of user terminal A calls the first media switching process of user terminal C;

[0093] User terminal B's second media switching process calls user terminal C's second media switching process.

[0094] Furthermore, such as Figure 5As shown, after the meeting is successfully established, the interaction between the modules is as follows: the initiator DSS interacts with the APPs of all participating terminals, and the user distributes control signaling (green line part in the figure); the APP interacts with the MSP, receiving audio and video streams from the other terminal and sending its own audio and video streams to the other terminal (blue line part in the figure); each terminal MSP communicates with each other, receiving and receiving audio and video (red line part in the figure).

[0095] Furthermore, the conferencing application is also used to access corresponding third-party conferences via SIP services based on information from those conferences.

[0096] The underlying communication protocol of the conferencing system is SIP / RTP, a standard protocol widely adopted in the industry and supported by the vast majority of manufacturers, ensuring cross-system interoperability. When connecting to a third-party conferencing system, you only need to configure the system's server address, assigned number, password, and other necessary information on the app to access the system as a standard SIP terminal and participate in the meeting.

[0097] In one specific embodiment of the present invention, the meeting application further includes a meeting management module; the meeting management module is used to provide an operation interface for users to operate, and the meeting management module also provides functions for initiating a meeting, joining a meeting, leaving a meeting, and recording a meeting.

[0098] The meeting management module provides users with the functions of initiating, joining, leaving, and recording video conferences. It supports no fewer than 16 video conferences and manages various meeting attributes, such as meeting topic and participants. It also controls the attributes of participants, such as muting, turning off microphones and cameras, and canceling meetings.

[0099] The meeting management module also has data processing capabilities, capable of storing meeting information, including meeting time, location, participant list, etc., and handling user operations such as initiating, joining, and leaving meetings. Figures 10 to 12 As shown, the meeting management module has a user-friendly interface, including a meeting list, start a meeting, and join buttons. For security reasons, meeting access permissions are controlled to ensure that only invited participants can join.

[0100] Furthermore, such as Figure 13 and Figure 14 As shown, the meeting application also includes a meeting recording module; this module is used to record the meetings that the user has participated in for easy retrieval.

[0101] The meeting minutes module displays the user's past meeting history, including ended meetings. It supports viewing details of past meetings, including participant information, meeting time, duration, and recorded video recordings.

[0102] The meeting record interaction process is as follows: The user submits a meeting record query request to the meeting management service with their user information; the meeting management service queries the database service for meeting record information based on the user's IP address and the meeting record; the meeting management service returns the meeting record query results; the user submits a meeting record deletion request to the meeting management service with the meeting record; the meeting management service queries the database service for meeting record information based on the user's IP address and the meeting record; if the meeting record exists, the service requests deletion from the database service based on the meeting record ID; the meeting management service returns a success message for deletion.

[0103] Module Interaction: Interacts with the user interface module to display a list of meeting history and provide the ability to view details.

[0104] Data processing: Query the meeting records that users participated in, including ended meetings and future meeting schedules.

[0105] Interface Design: Provides a user-friendly meeting history list interface, including meeting minutes and details viewing functions. For example... Figure 14 and Figure 15 As shown.

[0106] For security reasons, the privacy and security of meeting records must be protected, ensuring that only authorized users can access them.

[0107] In one specific embodiment of the present invention, the system further includes a login module, which can be set in a conference application. The login module provides authentication functions for users to log in to the system, using username and password authentication, and is responsible for verifying the user's identity to ensure that only authorized users can access the system.

[0108] like Figure 6 and Figure 7 As shown, the authentication process is as follows: The terminal sends a request to the authentication and authorization service with the username and password; the authentication and authorization service queries the database service based on the username and password; if the user exists and the password is correct, a JWT credential (token) is generated; if the user does not exist or the password is incorrect, the corresponding error message is returned; the token is returned to the terminal. Module interaction: It interacts with the user interface module to receive authentication information input by the user. It interacts with the user management module to verify the user's identity. The user enters their username to log in. The password is encrypted and stored to ensure data security. Data processing: It receives the authentication information input by the user, verifies it, and returns the authentication result. It handles abnormal situations during the user authentication process, such as incorrect passwords or account lockout. Interface design: It provides a user-friendly login interface, including username and password input boxes, a login button, etc. Security considerations: It adopts secure authentication methods, such as encrypted transmission and password hash storage, to protect user information security.

[0109] Furthermore, such as Figure 8 and Figure 9 As shown, it also includes a contact management module, allowing users to manage their contact list, including functions such as adding, deleting, and editing contact information. It provides the following functions: (1) Contact username first letter sorting function, which makes it convenient for users to find specific contacts. (2) Provides contact online status detection function, which can show whether the user is online in real time on the contact list. (3) Provides direct call function to the contact, which allows users to have a video conference with the user by clicking the call button in the contact list. (4) Provides ping packet test function, which allows users to directly enter the IP address to initiate a network ping packet and check whether the IP network where the user is located is normal.

[0110] Contact creation process: The user submits the contact name and IP address to the contact management service; the contact management service queries the database service for information based on the IP address; if the IP address exists, the service returns a message to the end user indicating that the contact already exists; otherwise, the service submits the contact name and IP address for database entry; once the contact is successfully entered into the database, the service returns a message to the end user indicating that the contact has been created successfully.

[0111] Module Interaction: Interacts with the user interface module to display the contact list and respond to user actions. Interacts with the meeting management module to use contact information to initiate meeting invitations.

[0112] Data processing: Stores contact information, including name, contact information, etc. Handles user operations such as adding, deleting, modifying, and querying contacts.

[0113] Interface design: Provides a user-friendly contact management interface, including a contact list, add / delete buttons, etc.

[0114] Security considerations: Protect the privacy and security of contact information, ensuring that only authorized users can access the contact list.

[0115] Furthermore, it includes a pan-tilt control module, allowing users to remotely control the camera pan-tilt unit via control buttons or gestures on the video conferencing terminal interface. It supports up / down and left / right rotation, as well as zoom and focus functions, allowing users to intuitively adjust the camera's angle and focus. Users can adjust camera image parameters, including brightness, contrast, and saturation, through the system interface according to the actual shooting scenario and needs. The adjusted parameters will be immediately reflected in the camera's output image, helping users obtain clearer and more realistic video footage.

[0116] like Figure 16 and Figure 17As shown, the PTZ control interaction process is as follows: The user client initiates a request to acquire the PTZ camera image to the instant messaging service; the instant messaging service calls the system tool (V4L2) to acquire the image in real time; the instant messaging service notifies the user client that image acquisition is in progress; the user client reads the image frames written to memory by the instant messaging service through memory sharing technology; the user client initiates a request to control the PTZ camera to the instant messaging service; the instant messaging service forwards the control commands to the PTZ camera through the serial port protocol; the PTZ camera responds to the serial port message and executes the control commands.

[0117] Module Interaction: Interacts with the user interface module to display the pan-tilt camera image and control the pan-tilt camera to perform relevant commands and actions.

[0118] Data processing: Storage PTZ control.

[0119] Interface design: Provides a user-friendly meeting management interface, including controls for pan-tilt rotation, screen brightness adjustment, etc.

[0120] Security considerations: Provide access control functionality to ensure that only authorized personnel can operate the PTZ.

[0121] Furthermore, such as Figures 18 to 20 As shown, it also includes a system settings module, which primarily allows users to configure various system parameters, including the current conference terminal's name, login password, microphone, camera, and speaker settings. It performs local network monitoring, including CPU load, memory load, network bandwidth (uplink, downlink), latency, and packet loss rate.

[0122] Module interaction: Interact with user interface modules to display system settings options and respond to user actions.

[0123] Data processing: Processes user modifications to system settings and saves user preferences and settings options.

[0124] Interface design: Provides a user-friendly system settings interface, including various settings options and a save button.

[0125] Security considerations: Ensure that users' personal preferences and settings are protected to prevent unauthorized access and modification.

[0126] The software conferencing system based on the HarmonyOS operating system of this invention is designed based on the adaptation design of the HarmonyOS operating system. The adaptation design of the HarmonyOS operating system is responsible for the loading and operation support of the system.

[0127] The software conferencing system of this invention uses the open-source HarmonyOS 5.0 system and follows a layered design, from bottom to top as follows:

[0128] The system architecture consists of a kernel layer, a system service layer, a framework layer, and an application layer. System functions are arranged hierarchically according to "system > subsystem > component," and in multi-device deployment scenarios, it supports the removal of unnecessary components based on actual needs. The open-source HarmonyOS technical architecture is as follows: Figure 21 As shown:

[0129] (1) System kernel layer

[0130] Kernel Subsystem: Employs a multi-kernel design (Linux kernel or LiteOS), supporting the selection of a suitable OS kernel for different resource-constrained devices. Kernel Abstraction Layer (KAL): By shielding the differences between multiple kernels, it provides basic kernel capabilities to upper layers, including process / thread management, memory management, file system, network management, and peripheral device management.

[0131] Driver Subsystem: The Driver Framework (HDF) is the foundation for the open system hardware ecosystem, providing unified peripheral access capabilities and a framework for driver development and management.

[0132] (2) System service layer

[0133] The system service layer is the core capability set of the open-source HarmonyOS, providing services to applications through the framework layer. This layer includes the following components:

[0134] The system's basic capability subsystem set provides the foundational capabilities for the operation, scheduling, and migration of distributed applications across multiple devices. It consists of subsystems such as distributed soft bus, distributed data management, distributed task scheduling, common basic library, multi-modal input, graphics, security, and AI.

[0135] Basic Software Service Subsystem Set: Provides common and general software services, consisting of subsystems such as event notification, telephone, multimedia, and DFX (Design for X).

[0136] Enhanced Software Service Subsystem Assembly: Provides differentiated capability-enhancing software services for different devices, consisting of subsystems such as smart screen proprietary services, wearable proprietary services, and IoT proprietary services.

[0137] Hardware service subsystem set: Provides hardware services, consisting of subsystems such as location services, user IAM, wearable proprietary hardware services, and IoT proprietary hardware services.

[0138] Depending on the deployment environment of different device types, the basic software service subsystem set, the enhanced software service subsystem set, and the hardware service subsystem set can be tailored at the subsystem granularity, and each subsystem can be tailored at the functional granularity.

[0139] (3) Framework layer

[0140] The framework layer provides user program frameworks and Ability frameworks in multiple languages ​​such as C / C++ / JS for application development, the ArkUI framework suitable for JS, and various multi-language framework APIs exposed to the outside world for various software and hardware services. The APIs supported by the device will vary depending on the degree of componentization of the system.

[0141] (4) Application layer

[0142] The application layer includes system applications and third-party non-system applications. An application consists of one or more Feature Abilities (FAs) or Particle Abilities (PAs). FAs have a user interface (UI) and provide the ability to interact with the user; while PAs do not have a UI but provide the ability to run background tasks and a unified data access abstraction. Applications developed based on FAs / PAs can implement specific business functions, support cross-device scheduling and distribution, and provide users with a consistent and efficient application experience.

[0143] After the video conferencing software starts, it first activates the meeting service, which is responsible for meeting scheduling and control. Then, it activates the self-developed streaming media distribution service, Fsagent Server, which handles the forwarding, encoding / decoding, resolution, and bitrate adjustment of audio and video streams. As a streaming media server, it ensures that audio and video data in the video conference can be transmitted smoothly between various terminals.

[0144] Furthermore, the bootloader uses the U-Boot open-source program. The main design idea for this part is to base it on the open-source code, and then tailor and customize it according to the characteristics of the single board itself, focusing on the following four parts: a) CPU initialization program; b) DDR controller configuration program; c) eMMC initialization program; d) Linux Kernel loader.

[0145] The device uses the Rockchip RK3588M chip, which integrates eight ARM cores (four single-core Cortex-A76 and four single-core Cortex-A55). After the device is powered on, the AP-side Cortex-A76 processor enters the boot state first. The U-Boot stage first configures the CPU's operating status, including CPU exception vector configuration, operating frequency, voltage, operating clock, and boot mode.

[0146] CPU boot mode configuration is mainly achieved by reading the status pin signals boot_ctl[1:0] (corresponding to pins BOOTCTL1 and BOOTCTL0).

[0147] The RK3588M is equipped with a dual-channel external memory controller, memctrl, to enable data transfer between the on-chip bus and external memory. External memory is mapped to the chip's internal address space. When the on-chip bus operates on this address space, memctrl translates the bus operation into an operation on the external memory, supporting a maximum of 32 GB. memctrl connects to external LPDRAM memory with a 32-bit data width and a 2-bit chip select.

[0148] The eMMC initialization program primarily initializes the eMMC chip controller and detects the driver on the device, preparing for subsequent reading of the eMMC and loading of the kernel. The eMMC driver adopts the Linux MMC general driver architecture, divided into the host driver layer, the core layer, and the MMC card block device layer. The focus is on the MMC card block device layer, which completes the eMMC initialization and device information registration.

[0149] After the U-Boot bootloader completes the initialization of the basic hardware modules, it will enter the kernel loader to load the kernel image. This part adopts the standard U-Boot processing method.

[0150] Since our solution is a large-capacity storage solution using the RK3588M platform with SPI NOR + PCIe SSD, we need to optimize and modify the uboot. The boot process is as follows: Figure 22 As shown, note that the NOR file contains the loader and uboot. The RK3588M loads the front-end loader from the NOR file, and the front-end loader then boots uboot. During the uboot stage, the SSD is initialized and the main system is booted. The main system's kernel and system firmware are stored in the SSD and are booted by uboot.

[0151] Furthermore, the open-source HarmonyOS system adopts a multi-kernel design (Linux kernel or LiteOS), supporting system deployment on devices with varying resource capacities. When the same hardware is deployed with different kernels, how to enable smooth migration of device drivers between different kernels, eliminating the burden of driver code porting, adaptation, and maintenance, is a crucial issue that the open-source HarmonyOS driver subsystem needs to address. It supports a component-based driver model, providing more granular driver management, allowing for component-based driver decomposition, and also includes some pre-built templated driver model components.

[0152] HarmonyOS Driver Architecture (HDF) provides driver developers with driver framework capabilities, including driver loading, driver service management, and driver messaging mechanisms. It aims to build a unified driver architecture platform, providing driver developers with a more precise and efficient development environment, striving to achieve "develop once, deploy across multiple systems."

[0153] The HDF driver architecture is built using a C language object-oriented programming model. Through platform decoupling and kernel decoupling, it achieves compatibility with different kernels and a unified platform foundation. The HDF driver framework architecture is as follows: Figure 23 As shown.

[0154] Main components of the HDF driver architecture:

[0155] HDI layer: Provides a unified and stable hardware device operation interface for the system through standardized device interface standards.

[0156] HDF driver framework: Provides unified hardware resource management, driver loading management, device node management, device power management and driver service model, and needs to include modules such as device management, service management, DeviceHost, PnPManager.

[0157] Unified configuration interface: Supports abstract description of hardware resources, hides hardware differences, enables developers to create generic driver code that is not bound to configuration information, improves development and migration efficiency, and can quickly generate configuration files through tools such as HC-Gen.

[0158] Operating System Abstraction Layer: Provides unified encapsulated kernel operation-related interfaces, shielding the differences in operation between different systems, including interfaces for memory, locks, threads, semaphores, etc.

[0159] Platform Driver: Provides a unified interface for operating Board hardware (such as I2C / SPI / UART bus and other platform resources) for peripheral drivers, and provides a unified adaptation interface abstraction for Board hardware operation to facilitate migration to different platforms.

[0160] Peripheral driver model: For peripheral drivers, it provides common driver abstraction models, mainly to achieve two purposes: to provide standardized device drivers, so that developers do not need to develop independently, but can complete the deployment of drivers through configuration; and to provide driver model abstraction, shielding the interaction between drivers and different system components, making drivers more universal.

[0161] The kernel drivers mainly include bus drivers and chip peripheral drivers. Their primary function is to provide low-level driver support for upper-layer applications, completing the low-level driver functions for the bus and chips. The kernel software mainly includes bus drivers and chip peripheral drivers. The bus driver focus is on the frequently used USB bus driver and UART bus driver. The chip peripheral driver focus is on the codec audio chip driver and display driver.

[0162] Furthermore, the USB Universal Serial Bus comprises a host and a device. The host is responsible for data transmission and port management within the USB bus, while the device can connect to various peripherals. Therefore, USB driver development is divided into host driver development and device driver development.

[0163] The HarmonyOS open-source USB module supports the development of USB services, provides USB-related functions, provides USB device data read and write interfaces for user-space third-party function drivers, and provides functions such as creating and deleting USB devices, obtaining and opening / closing interface events, piped synchronous and asynchronous read and write communication, and setting custom USB properties.

[0164] The USB DDK is a USB driver development kit provided by the HDF driver framework for developers, consisting of two parts: USB HostDDK and USB Device DDK.

[0165] The USB Host DDK provides developers with the ability to develop host-side USB drivers, which are divided into three main categories according to their functions: DDK initialization classes, interface object operation classes, and request object operation classes.

[0166] like Figure 24 As shown, the USB Interface Pool is responsible for USB Interface management. It provides the ability to request and reclaim USB Interface objects, which are used to record device port information and resources. The USB Interface Pool categorizes and manages USB Interfaces according to their USB ports. Additionally, this module provides the USBDDK API, facilitating USB data read and write operations for developers.

[0167] The USB Protocol Layer provides USB protocol encapsulation, translates and parses device I / O / control commands according to the USB protocol, and is responsible for managing device descriptors. It matches the corresponding descriptor based on the enumeration information reported by the USB Device, constructs the corresponding USB Interface object, and adds it to the USB Interface Pool for management.

[0168] The Device IO Manager is responsible for managing USB IO requests and provides synchronous and asynchronous IO management mechanisms. For asynchronous IO, the IO Manager is responsible for recording the request and then processing the IO requests to be sent sequentially through the interface provided by the Raw API Library. After receiving the processing result of the USB controller's response, the IO receiving thread is responsible for parsing and reporting the processing result to the upper-layer caller.

[0169] The Raw API Library abstracts the underlying OS capabilities, defines a unified OS capability interface, and provides the USBRAW API to facilitate developers in implementing more complex driver functions.

[0170] The OS Adapter encapsulates platform-specific (Linux and LiteOS) operations, compiling platform-specific wrapper interfaces according to different platform configurations. On the Linux platform, all operations for accessing the USB FS are encapsulated in this module; while on the LiteOS platform, all device access operations based on the FreeBSD USB framework are also encapsulated in this module.

[0171] PNP Notify is used to dynamically monitor USB status changes, updating device information when a new device is added or removed. Simultaneously, all USB device information is reported via KHDF to the PNP Notify Manager module on the UHDF side to handle the loading / unloading of third-party function drivers.

[0172] The USB Device DDK provides developers with the capability to develop USB drivers for devices. For example, it offers dynamic registration and deregistration of USB ports, allowing developers to dynamically add and combine USB ports; dynamic instantiation capabilities, supporting the creation of device instances and transmission channels based on dynamically assigned device, configuration, interface, and endpoint descriptors; user-space data sending and receiving capabilities, supporting sending and receiving data in user space; and composite device capabilities, supporting multiple logical devices on a single physical device, achieving isolation between multiple logical devices, and allowing different logical devices to be accessed simultaneously by different application processes.

[0173] like Figure 25 As shown, the SDK IF is responsible for logically dividing USB devices according to devices, interfaces, and pipes, and encapsulating configuration management, device management, and IO management. This module also provides developers with interfaces for device-side driver development capabilities such as device creation, interface acquisition, receiving event data, and sending and receiving data.

[0174] Configuration Manager is responsible for parsing the USB descriptor information described in the HCS file. The obtained USB descriptor information is used for device creation. The module also provides operations such as reading, creating, deleting, and modifying custom attributes.

[0175] Device Manager is responsible for parsing USB descriptors based on the configuration module and creating devices based on the USB descriptors. It is also responsible for acquiring devices, deleting devices, obtaining device status, and retrieving interface information from the devices.

[0176] The IO Manager is responsible for reading and writing data, including receiving events and events after data reading and writing is completed. It supports synchronous and asynchronous data reading and writing modes.

[0177] The Adapter IF mainly encapsulates the operation of composite device configuration drivers and general function drivers for device nodes, providing a unified device management interface for the upper layer.

[0178] The RK3588M integrates both a USB 3.0 controller and a USB 2.0 controller. Driver porting requires configuring and registering the RK3588M for USB devices to enable USB bus hot-plug communication. The main registration information includes:

[0179] a) Register the USB controller base address;

[0180] b) Register the USB controller interrupt signal;

[0181] c) Registration of USB HUB chip-related device and driver information;

[0182] d) Register USB HSIC device driver information;

[0183] e) Register USB OTG device driver information.

[0184] USB device-side drivers mainly include UDC drivers, Gadget Function APIs, and GadgetFunction drivers. These drivers are primarily implemented by the USB devices themselves; the system USB bus driver is not a concern.

[0185] Furthermore, such as Figure 26 As shown, the RK3588M integrates nine external serial ports and one system debugging serial port, making it an important bus for peripheral modules and system debugging.

[0186] In the HDF framework, the UART interface adaptation mode adopts an independent service model. In this model, each device object independently publishes a device service to handle external access. After receiving an API access request, the device manager extracts the request parameters to invoke the corresponding internal methods of the actual device object. The independent service model can directly leverage the service management capabilities of the HDF device manager, but it requires configuring a separate device node for each device, increasing memory usage.

[0187] In standalone service mode, the core layer does not publish a single service for use by the upper layers. Therefore, in this mode, the driver must publish a service for each controller, specifically as follows:

[0188] When adapting the driver, you need to implement the Bind hook function of HdfDriverEntry to bind the service.

[0189] In the device_info.hcs file, the policy field of deviceNode must be 1 or 2, and cannot be 0.

[0190] Functions of each layer in the UART module:

[0191] The interface layer provides interfaces for opening the UART device, reading data of a specified length from the UART device, writing data of a specified length to the UART device, setting the UART device baud rate, getting the UART device baud rate, setting UART device attributes, getting the UART device baud rate, setting the UART device transmission mode, and closing the UART device.

[0192] The core layer primarily provides the ability to create, remove, and manage UART controllers, and interacts with the adaptation layer through hook functions.

[0193] The adapter layer mainly instantiates the functionality of hook functions to implement specific functionalities.

[0194] Furthermore, such as Figure 27 As shown, the audio driver framework in HDF mainly consists of the following parts.

[0195] (1) HDI adapter:

[0196] Implements the Audio HAL layer driver (HDI interface adapter), providing the necessary audio hardware driver capabilities to Audio services (frameworks). It includes interface objects such as Audio Manager, Audio Adapter, Audio Control, AudioCapture, and Audio Render.

[0197] (2) Audio Interface Lib:

[0198] It works in conjunction with the Audio Driver Model in the kernel to control audio hardware, read recording data, and write playback data. It includes the Stream_ctrl_common general layer, primarily for interfacing with the upper-level AudioHDI Adapter layer.

[0199] (3) ADM (Audio Driver Model):

[0200] The audio driver framework model serves the multimedia audio subsystem, enabling system developers to more easily develop applications tailored to specific scenarios. It also serves specific device manufacturers, allowing codec and DSP device manufacturers to adapt their driver code using the unified interface provided by the ADM module, thus achieving rapid development and compatibility with the HarmonyOS open-source system.

[0201] (4)Audio Control Dispatch:

[0202] It receives control commands from the lib layer and distributes them to the driver layer.

[0203] (5) Audio Stream Dispatch:

[0204] It receives data from the lib layer and distributes the data to the driver layer.

[0205] (6) Card Manager:

[0206] Multi-sound card management module. Each sound card contains Dai, Platform, Codec, DSP, and SAPM modules.

[0207] (7) Platform Drivers:

[0208] Driver adaptation layer.

[0209] (8) SAPM (Smart Audio Power Manager):

[0210] The power management module optimizes the power consumption strategy for the entire ADM power supply.

[0211] The device's audio module uses the CJC8988 chip, which supports main microphone, secondary microphone, headphone earpiece, and headset. The audio chip interfaces with the CPU primarily through PCM and I2S interfaces, enabling functions such as audio chip control, voice signal acquisition, and audio data transmission.

[0212] When the system starts, the Platform, Codec, DSP, and Dai drivers of the Audio module are loaded first. Each driver obtains configuration information from its own private configuration file and saves the obtained configuration information into its Data structure.

[0213] The audio driver design mainly includes the CPU audio interface bus driver and the audio module's own driver design, detailed as follows:

[0214] a) Audio bus driver

[0215] The audio module and CPU interface bus mainly include I2S and PCM. The system software registers the base address, address space size, and interrupt pins of the controllers for these two buses respectively; at the same time, it registers the controller device information and driver information to the Linux kernel, which then completes the matching of devices and drivers, and finally completes the control of the I2S and PCM buses.

[0216] b) Audio module driver

[0217] The audio driver is based on the driver provided by the chip manufacturer and mainly completes the registration of audio information, the setting of audio data encoding format, and the registration of audio data stream.

[0218] Furthermore, the Ethernet module in the video conferencing terminal is a single-channel Gigabit Ethernet solution, which is implemented by connecting an external Gigabit Ethernet controller chip through an RK3588M PCIE2.0 interface, and the network card chip driver uses a standard driver provided by the Linux Kernel.

[0219] When adapting to the HarmonyOS system, first add driver support for PCIE in the HarmonyOS source code, and then add driver support for the Ethernet module.

[0220] Furthermore, the video conferencing system uses an external HDMI display for output. In the HDF framework, the HDMI module interface adaptation mode will adopt an independent service mode. In this mode, each device object will independently publish a device service to handle external access. After receiving an API access request, the device manager extracts the parameters of the request to call the corresponding internal methods of the actual device object. The independent service mode can directly leverage the service management capabilities of the HDF device manager, but it requires configuring a separate device node for each device, increasing memory usage.

[0221] In standalone service mode, the core layer does not publish a single service for use by the upper layers. Therefore, in this mode, the driver must publish a service for each controller, specifically as follows:

[0222] Driver adapters need to implement the Bind hook function of HdfDriverEntry to bind services.

[0223] In the device_info.hcs file, the policy field of deviceNode must be 1 or 2, and cannot be 0.

[0224] Functions of each layer in the HDMI module:

[0225] The interface layer provides functions such as opening the HDMI device, starting HDMI transmission, stopping HDMI transmission, setting audio and image blanking, setting color depth, getting color depth, setting video properties, getting video properties, setting HDR properties, reading raw EDID data from the Sink end, registering the HDMI hot-plug detection callback function, unregistering the HDMI hot-plug detection callback function, and closing the HDMI device interface.

[0226] The core layer primarily provides the ability to turn the HDMI controller on, off, and manage it, interacting with the adaptation layer through hook functions.

[0227] The adapter layer mainly instantiates the functionality of hook functions to implement specific functionalities.

[0228] The present invention has been described in detail above with reference to the accompanying drawings and embodiments. Those skilled in the art can make various modifications to the present invention based on the above description. Therefore, certain details in the embodiments should not be construed as limiting the present invention, and the scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A software conferencing system based on the HarmonyOS operating system, characterized in that, include: The scheduling platform module is located at the user terminal. The scheduling platform module is used to complete user registration and login, and also to complete call connection to establish a conference. The HarmonyOS terminal module is located at the user terminal and establishes a connection with the scheduling platform module at each user terminal. The HarmonyOS terminal module is used to initiate meetings to facilitate the establishment of point-to-point meetings or multi-party meetings.

2. The software conferencing system based on the HarmonyOS operating system as described in claim 1, characterized in that, The scheduling platform module includes data synchronization service and SIP service; The data synchronization service is used to complete user login authentication. After successful authentication, it returns SIP account information and data synchronization service address of other user terminals to the HarmonyOS terminal module at the same user terminal. The SIP service is used to store user information and to implement call control to establish, maintain and release calls, facilitating the establishment of communication connections.

3. The software conferencing system based on the HarmonyOS operating system as described in claim 2, characterized in that, The HarmonyOS terminal module includes conferencing applications and media exchange processes; The conferencing application connects to the data synchronization service at the same user terminal. After receiving the SIP account information returned by the corresponding data synchronization service, the conferencing application registers all media exchange processes to the SIP service corresponding to the SIP account information. It is also used to establish a connection with the data synchronization service of other user terminals based on the data synchronization service address of the other user terminals received. The media exchange process is used to establish call connections with other participants.

4. The software conferencing system based on the HarmonyOS operating system as described in claim 3, characterized in that, When a conference application at user terminal A initiates a point-to-point call conference, it directly registers with the SIP service at the called user terminal B. After successful registration, the conferencing application on user terminal A initiates a SIP invitation with the called number of user terminal B. Once user terminal B accepts the SIP invitation, a point-to-point conference is established.

5. The software conferencing system based on the HarmonyOS operating system as described in claim 3, characterized in that, When a conference application on user terminal A initiates a multi-party conference, the conference application on user terminal A allocates a corresponding number of media exchange processes based on the number of participants. The conferencing application of user terminal A requests the data synchronization service of user terminal A to establish a meeting; User terminal A's data synchronization service sends meeting invitation notifications to the participants; After the participants' terminals accept the request, the conference application on the participants' terminals allocates an appropriate number of media exchange processes according to the number of participants, and sends the list of allocated media exchange processes to the data synchronization service of user terminal A. The data synchronization service of user terminal A allocates the media exchange process between the caller and the called party according to the order of joining the meeting, and notifies the calling party's conference application of the caller-called relationship. The calling party's conference application controls the corresponding media exchange process to initiate a call. After registering with the SIP service of the called party, the multi-party conference is established upon completion of the call.

6. The software conferencing system based on the HarmonyOS operating system as described in claim 5, characterized in that, When the participants include user terminal B and user terminal C, if user terminal B joins the meeting before user terminal C, the calling and called relationship formed by the media switching process is as follows: The first media switching process of user terminal A calls the first media switching process of user terminal B. The second media switching process of user terminal A calls the first media switching process of user terminal C; User terminal B's second media switching process calls user terminal C's second media switching process.

7. The software conferencing system based on the HarmonyOS operating system as described in claim 3, characterized in that, The meeting application is also used to access the corresponding third-party meeting through the SIP service based on the information from the third-party meeting.

8. The software conferencing system based on the HarmonyOS operating system as described in claim 3, characterized in that, The meeting application also includes a meeting management module; The meeting management module provides an operation interface for users to operate, and also provides functions for initiating, joining, leaving, and recording meetings.

9. The software conferencing system based on the HarmonyOS operating system as described in claim 3, characterized in that, The meeting application also includes a meeting recording module; The meeting recording module is used to record the meetings that users have participated in for easy retrieval.

10. The software conferencing system based on the HarmonyOS operating system as described in claim 1, characterized in that, The scheduling platform also includes a network detection service, which is used to send detection data packets to each user terminal in order to perceive and display the prior status of each user terminal.

Images