An optimization method and system based on audio and video fusion communication technology

Abstract

This invention discloses an optimized method and system based on audio and video converged communication technology, comprising: adopting a centralized deployment scheme, with a converged communication server platform deployed at the central node; establishing a two-way transmission channel for local and external telephone lines, and initiating audio and video conferences; synchronizing contact information to the converged communication server; and allowing headquarters devices to access the converged communication server and branch devices to access the headquarters system. This invention innovates the interconnection and interoperability between various types of communication, possessing advantages such as openness and compatibility, interoperability between multiple system types, and rich access methods. It supports the integration of voice systems, video systems, IM systems, etc., on a unified platform, allowing users to fulfill all business communication needs on a single system; it is compatible with users' existing analog phones and supports rich access methods such as video conferencing terminals and live streaming; and it provides single sign-on authentication for unified entry.

Description

Technical Field

[0001] This invention relates to the field of converged communication technology, specifically to an optimization method and system based on audio and video converged communication technology. Background Technology

[0002] As internal communication scales and business systems continue to expand, collaboration between employees and between teams is becoming increasingly important. Due to the growing regionalization and mobility of current communication needs, these new office trends present new challenges to enterprise internal communication. The current state of many internal communication systems is characterized by independent subsystems. For example, achieving instant messaging requires a separate IM chat system, voice system, video system, and business office system, leading to fragmentation and isolated subsystems that cause inconvenience for users and maintenance personnel. For instance, current enterprise internal communication practices require a separate telephone system for making calls, a different conferencing system for holding meetings, and other systems for internal document collaboration. This inherently adds obstacles to system communication, and such fragmented communication systems limit the development of collaborative applications within the enterprise. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is that existing enterprise internal communication systems are fragmented, with each subsystem forming an independent island, causing inconvenience to users and maintenance personnel. This adds obstacles to system communication and also limits the development of enterprise collaborative applications.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: an optimization method based on audio-video fusion communication technology, comprising:

[0007] A centralized deployment scheme is adopted, with a converged communication server platform deployed at the central node;

[0008] Establish a two-way transmission channel for local and external telephone calls, and initiate audio and video conferences;

[0009] Synchronize contact information to the converged communication server;

[0010] Headquarters devices access the converged communications server for connection, while branch devices access the converged communications server to connect to the headquarters system.

[0011] As a preferred embodiment of the optimized method based on audio-video fusion communication technology described in this invention, the bidirectional transmission channel for local and external telephone lines includes:

[0012] Connect to the voice gateway via SIP trunk link;

[0013] The converged communication server transmits outbound calls for local calls to the voice gateway.

[0014] The voice gateway transmits the external telephone call to the operator.

[0015] When a local call makes an outside call, the call is routed back through the original operator.

[0016] As a preferred embodiment of the optimized method based on audio-video fusion communication technology described in this invention, the process of synchronizing contact information to the fusion communication server includes:

[0017] The converged communication server interfaces with the LDAP server;

[0018] The converged communication server transmits authentication information to the LDAP server;

[0019] After authentication by the LDAP server, the address book information is synchronized to the unified communications server.

[0020] As a preferred embodiment of the optimization method based on audio-video fusion communication technology described in this invention, the audio-video fusion process of the fusion communication server includes:

[0021] Determine the type of connected communication devices: When the connected communication devices are of the same type, they are grouped into a set of similar devices. No translation or encoding / decoding is required; the signal is directly output and the media stream is directly distributed to each communication device through the converged communication server. When the connected communication devices are of different types, they are grouped into sets of different types of devices, and translation is performed according to the audio and video encoding / decoding of different communication devices.

[0022] Establish communication channels in different types of device sets and determine whether the connected communication devices support video capabilities: when the connected communication devices do not support video capabilities, the communication devices that do not support video capabilities are divided into a set of call-only devices, and the communication devices that support audio and video capabilities are divided into a set of call and video devices.

[0023] For a set of call-only devices, the video stream in the media stream is removed, and the audio and video signals are decoded using audio and video codecs specific to different devices. The media stream is then distributed to the connected communication devices through the converged communication server.

[0024] For a set of call and video devices, a weighted average method is used to weight and fuse the audio and video signals in the media stream. Then, the audio and video signals are decoded using audio and video codecs specific to different devices. The media stream is then distributed to the connected communication devices through the fusion communication server.

[0025] Determine network signal status: When the network signal is good, it is classified as a smooth network state, and the audio and video signals are output at maximum power; when the network signal is average, it is classified as a good network state, and the weight of audio and video signals is adjusted evenly; when the network signal is poor, it is classified as a network latency state, and the network adaptive mode is enabled, and the weight of audio and video signals is adjusted, increasing the weight of audio signals and prioritizing audio calls.

[0026] As a preferred embodiment of the optimization method based on audio-video fusion communication technology described in this invention, the audio-video signal fusion method based on the time domain includes:

[0027]

[0028] Where t represents time, G(t) represents the fused audio and video signals, and X... i (t) and Y i (t) represent the i-th audio signal and the i-th video signal, respectively, w i v i Let be the weight of the i-th signal in the fusion process.

[0029] The audio and video signals are fused based on the frequency domain:

[0030]

[0031] Where f represents frequency, X(f) and Y(f) represent the power spectra of the fused audio and video signals, respectively, Xi(f) and Yi(f) represent the power spectra of the i-th audio signal and the i-th video signal, respectively, and w i v i Let be the weight of the i-th signal in the fusion process.

[0032] As a preferred embodiment of the optimization method based on audio-video fusion communication technology described in this invention, the signal fusion method for establishing a two-way transmission channel for local and external telephone lines includes:

[0033] The audio signals are fused using a weighted average method based on the following formula:

[0034]

[0035] Where n represents the position on the time axis, y(n) is the output audio signal, and x i (n) represents the i-th input audio signal, w i (n) represents the weight value of the i-th audio signal at time n. In implementing the weighted average method, it is typically necessary to calculate the corresponding weighting coefficients based on the signal-to-noise ratio and power of different audio signals.

[0036] The audio signal is fused using a spectral subtraction algorithm.

[0037]

[0038] Where k represents the position on the frequency axis, Y(k) is the power spectrum of the output audio signal, and X... i (k) represents the power spectrum of the i-th audio signal, E i (k) represents the noise power spectrum estimated from the i-th audio signal, and N is the number of audio channels.

[0039] As a preferred embodiment of the optimization method based on audio-video fusion communication technology described in this invention, the signal fusion process for establishing a two-way transmission channel for local and external telephone lines includes:

[0040] The formula for an audio signal fusion algorithm based on weighted averaging is as follows:

[0041]

[0042] Where n represents the position on the time axis, y(n) is the output audio signal, and x i (n) represents the i-th input audio signal, w i (n) represents the weight value of the i-th audio signal at time n. In implementing the weighted average method, it is typically necessary to calculate the corresponding weighting coefficients based on the signal-to-noise ratio and power of different audio signals.

[0043] The formula for an audio signal fusion algorithm based on spectral subtraction is as follows:

[0044]

[0045] Where k represents the position on the frequency axis, Y(k) is the power spectrum of the output audio signal, and X... i (k) represents the power spectrum of the i-th audio signal, E i (k) represents the noise power spectrum estimated from the i-th audio signal, and N is the number of audio channels.

[0046] As a preferred embodiment of the optimized system based on audio-video fusion communication technology described in this invention, it includes:

[0047] The meeting service module is used for applying for media resources and controlling meetings, as well as providing private deployment solutions;

[0048] The call service module is used to obtain SIP extension numbers, register the extension numbers obtained by the terminal, apply for relevant resources, connect to the voice gateway to realize PSTN inbound and outbound calls, and perform intelligent voice menu broadcasting.

[0049] The interactive collaboration service module is used to support whiteboard collaboration and whiteboard annotation.

[0050] The IM status service module is used to handle the display and updating of users' online status;

[0051] The IM session caching service module is used to save sessions in the cache, as well as message notifications, message synchronization across multiple devices, and to automatically push historical messages to new group members.

[0052] The IM file service module allows users to configure local or distributed storage as needed, ensuring data security, and supporting instant transfer and resume interrupted transfer. It also supports automatic file cleanup settings and group chat file management.

[0053] The IM business processing service module is used for mobile notification push, allows access to third-party push, and supports electronic voting, electronic announcement permission management and viewing.

[0054] As a preferred embodiment of the optimized system based on audio and video fusion communication technology described in this invention, the conference module includes: conference control service, conference scheduling service, SFU media forwarding service, interactive live streaming service, live streaming push service, and recording service.

[0055] The call service module includes: user center, voice service, voice media service, and voice load service.

[0056] The IM file service module adopts on-demand settings for local or distributed storage to ensure data security; file and image transmission is achieved through instant transmission and breakpoint resume transmission, with automatic file cleanup settings and group chat file management.

[0057] A computer device includes: a memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method described in any one of the present invention.

[0058] A computer-readable storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a processor, it implements the steps of the method described in this invention.

[0059] The beneficial effects of this invention are as follows: The optimized method and system based on audio-video fusion communication technology provided by this invention innovates the interconnection and interoperability between various types of communication, possessing advantages such as openness and compatibility, interconnection and interoperability of multiple system types, and rich access methods. It supports the integration of voice systems, video systems, IM systems, etc., on a unified platform, allowing users to meet all their business communication needs with a single system; it is compatible with users' existing analog phones, IP phones, and traditional SIP / H.323 video conferencing terminals; it supports rich access methods such as computer clients, mobile apps, H5 clients, analog phones, IP phones, monitoring equipment, video conferencing terminals, and live streaming; it provides single sign-on authentication, and the H5 method integrates with third-party systems, allowing for quick access to the H5 of various systems and achieving a unified entry point. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0061] Figure 1 A flowchart illustrating an optimization method based on audio-video fusion communication technology, provided in the first embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of a centralized deployment scheme for an optimization method based on audio and video fusion communication technology provided in the first embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram illustrating a centralized deployment scheme optimization method based on audio and video fusion communication technology, provided in the third embodiment of the present invention. Detailed Implementation

[0064] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0065] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0066] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0067] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0068] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0069] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0070] Example 1

[0071] Reference Figure 1-2 The first embodiment of the present invention provides an optimization method based on audio and video fusion communication technology, comprising:

[0072] S1: A centralized deployment scheme is adopted, with a converged communication server platform deployed at the central node;

[0073] Furthermore, branch offices and mobile users can access the headquarters platform via the Internet.

[0074] Furthermore, each branch office can choose from various equipment options, such as IP (Internet Protocol) phones, video conferencing terminals, and software clients.

[0075] It should be noted that in current common technical architectures, IP phones can only make calls between devices of the same type and cannot be combined with video conferencing because the encoding and decoding between devices are inconsistent and they cannot communicate with each other.

[0076] Furthermore, determine the type of communication device being connected:

[0077] When the connected communication devices are of the same type, they are grouped into a set of similar devices. No translation or encoding / decoding is required; the signal is directly output and the media stream is directly distributed to each communication device through the converged communication server.

[0078] When the connected communication devices are of different types, they are divided into different types of device sets, and the audio and video encoding and decoding of different communication devices are translated.

[0079] Furthermore, communication channels are established across different types of devices to determine whether the connected communication devices support video capabilities.

[0080] When the connected communication devices do not support video capabilities, the communication devices that do not support video capabilities are divided into a set of call-only devices, and the communication devices that support audio and video capabilities are divided into a set of call and video devices.

[0081] Furthermore, for a set of call-only devices, the video stream in the media stream is removed, and the audio and video signals are decoded using audio and video codecs specific to different devices. The media stream is then distributed to the connected communication devices through the converged communication server.

[0082] Furthermore, for a set of call and video devices, a weighted average method is used to weight and fuse the audio and video signals in the media stream, and then the audio and video signals are decoded using audio and video codecs specific to different devices. The media stream is then distributed to the connected communication devices through the fusion communication server.

[0083] Furthermore, the system assesses network signal status: when the network signal is strong, it is classified as a smooth network state, and audio and video signals are output at maximum power; when the network signal is moderate, it is classified as a good network state, and the weights of audio and video signals are balanced; when the network signal is weak, it is classified as a network latency state, and network adaptive mode is enabled, while the weights of audio and video signals are adjusted, with an increased weight for audio signals to prioritize audio calls.

[0084] Furthermore, time-domain based fusion methods for audio and video signals include:

[0085]

[0086] Where t represents time, G(t) represents the fused audio and video signals, Xi(t) and Yi(t) represent the i-th audio signal and the i-th video signal, respectively, and w i v i Let be the weight of the i-th signal in the fusion process.

[0087] Furthermore, a frequency-domain-based fusion method for audio and video signals:

[0088]

[0089] Where f represents frequency, X(f) and Y(f) represent the power spectra of the fused audio and video signals, respectively, Xi(f) and Yi(f) represent the power spectra of the i-th audio signal and the i-th video signal, respectively, and w i v i Let be the weight of the i-th signal in the fusion process.

[0090] For example, video conferencing terminals use H.264HP (a video codec standard named after the H.26x series by the ITU Telecommunication Standards Bureau) and G.711 (a voice compression standard customized by the ITU Telecommunication Standards Bureau) for audio and video encoding and decoding, while IP phones use OPUS (a voice coding format standardized by the Internet Engineering Task Force) for audio encoding and decoding, which does not support video encoding and decoding. The inconsistent encoding and decoding between the two devices prevents them from communicating with each other. Through the optimization of audio-video converged communication technology, the media nodes of the converged communication server establish independent connections with both devices using corresponding encoding and decoding formats. The converged communication server then forwards the corresponding audio and video streams, transcoding the IP phone's audio stream into G.711 format and sending it to the video conferencing terminal, while removing the video stream from the video conferencing terminal and transcoding the audio stream into OPUS and sending it to the IP phone, thus enabling communication between the video conferencing terminal and the IP phone. Similarly, in a conference, different similar devices can communicate with each other. When a device needs to receive a media stream, the converged communication server can integrate and transmit the audio and video streams of multiple other devices. If the device does not support video capabilities, the video stream in the media stream will be removed.

[0091] This embodiment also provides a computing device, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the automatic verification method for monitoring information of a conventional substation as proposed in the above embodiment.

[0092] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements an automatic verification method for conventional substation monitoring information as proposed in the above embodiments.

[0093] The storage medium proposed in this embodiment belongs to the same inventive concept as the automatic verification method for conventional substation monitoring information proposed in the above embodiment. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0094] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory, magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory, magnetic variable memory, ferroelectric memory, phase change memory, graphene memory, etc. Volatile memory can include random access memory or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory or dynamic random access memory, etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include blockchain-based distributed databases, etc., and are not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited thereto.

[0095] S2: Establish a two-way transmission channel for local and external telephone calls, and initiate audio and video conferences;

[0096] Furthermore, by connecting to the voice gateway via a SIP trunk link, the converged communication server transmits local calls to the voice gateway, and then to the operator.

[0097] It should be noted that SIP stands for Session Initialization Protocol.

[0098] Furthermore, the voice gateway supports E1 (a telecommunications standard) line access.

[0099] Furthermore, when a local call makes an E1 line call, the operator also transmits the call to the voice gateway, which then forwards it to the converged communications server, enabling local call access and PSTN (Public Switched Telephone Network) call functions to meet the demand for local calls;

[0100] Furthermore, during a meeting, local calls can be invited to join the meeting through the meeting control module, forming an audio call. Users can also directly make external calls from their mobile phones to participate in the meeting.

[0101] Furthermore, once the local call enters the conference, the transmitted audio stream enters the conference and, combined with the video feeds from other devices in the conference (video equipment, telephone equipment, etc.), forms a unified screen layout, becoming a complete audio and video media stream, which is then transmitted to the receiving device. At the same time, the converged communication server also integrates the media streams from other devices and removes the video stream portion before transmitting it to the local call.

[0102] Furthermore, the formula for the audio signal fusion algorithm based on the weighted average method is as follows:

[0103]

[0104] Where n represents the position on the time axis, y(n) is the output audio signal, xi(n) represents the i-th input audio signal, and wi(n) is the weight value of the i-th audio signal at time n. In the implementation of the weighted average method, it is usually necessary to calculate the corresponding weight coefficients based on the signal-to-noise ratio and power of different audio signals.

[0105] Furthermore, the formula for the fusion algorithm based on spectral subtraction is as follows:

[0106]

[0107] Where k represents the position on the frequency axis, Y(k) is the power spectrum of the output audio signal, and X... i (k) represents the power spectrum of the i-th audio signal, E i Y(k) represents the estimated noise power spectrum of the i-th audio signal, and N is the number of audio channels. If the value of Y(k) is greater than 0, the output power spectrum is Y(k); if the value of Y(k) is less than 0, the output power spectrum is 0.

[0108] It should be noted that traditional video conferencing does not support access via standard local calls.

[0109] S3: Synchronize contact information to the unified communications server;

[0110] Furthermore, authentication is performed via an LDAP server, after which the address book information is synchronized to the unified communications server.

[0111] It should be noted that the converged communications server supports integration with LDAP (Lightweight Directory Access Protocol) servers. After authentication via the LDAP server, address book information is synchronized to the converged communications server, but passwords and other information are not stored; only the user's login name is saved. When a user needs to log in to the converged communications server, the server transmits authentication information to the LDAP server and awaits the authentication result. This enables automatic synchronization of enterprise organizational structure and contact information, meeting the need for managing only one enterprise address book across multiple systems.

[0112] Furthermore, this solution can meet the needs of enterprises for convenient management of employee accounts. Administrators can quickly add, delete, and edit user accounts, and users only need to record one account information.

[0113] It should be noted that traditional communication system account systems are independent of each other. Different accounts and passwords need to be recorded for different systems, and administrators also need to manage multiple account systems separately.

[0114] S4: Headquarters devices access the converged communications server for connection, while branch devices access the headquarters system via the converged communications server.

[0115] Furthermore, desktop phones at headquarters, video terminals in conference rooms, and desktop computers of users can access the converged communications server through the headquarters' internal network and complete identity authentication through their accounts.

[0116] Furthermore, branch users can access the headquarters system via the enterprise private network / Internet and the converged communications server, and complete identity authentication through their accounts to achieve interconnection between the headquarters and the branch.

[0117] Furthermore, external business travelers can access the converged communications system via 3G (third-generation mobile communication technology), 4G (fourth-generation mobile communication technology), 5G (fifth-generation mobile communication technology), the Internet, or wireless LAN on computers, mobile devices, or web pages. They can complete identity authentication through their accounts, enabling mobile office scenarios.

[0118] Example 2

[0119] A second embodiment of the present invention provides an optimized system based on audio and video converged communication technology, including: a conference service module, a call service module, an interactive collaboration service module, an IM (Instant Messenger) status service module, an IM session caching service module, an IM file service module, and an IM business processing service module.

[0120] Furthermore, the conference service module also includes: conference control service, conference scheduling service, SFU (Selective Forwarding Unit) media forwarding service, interactive live streaming service, live streaming push service, and recording service.

[0121] After registering the extension number, users can initiate video conferences and apply for media resources and control the conference through the conference control and scheduling services. A private deployment solution is provided, supporting up to 3+N master-slave servers. Automatic migration occurs within 3 minutes of any node failure, ensuring seamless and continuous conference recovery and minimizing impact on business continuity. It supports both SIP (Session Initialization Protocol) and H.323 (part of the ITU multimedia communication standard H.32x) dual-standard protocols, ensuring broad compatibility with mainstream video conferencing terminal brands.

[0122] It should be noted that Yealink's unified communication and collaboration platform has advantages such as high concurrency, cluster disaster recovery, smooth expansion, and flexible deployment, meeting the converged communication needs of various enterprises, especially large government and enterprise multi-level organizations, reducing costs and increasing efficiency.

[0123] Furthermore, the IP-PBX (Internet Protocol-Private Branch Exchange) call service module also includes a user center, voice service (registration, gateway mediation, interactive voice response service), voice media service, and voice load service.

[0124] Furthermore, through the user center, converged communication terminals can obtain a SIP (Session Initialization Protocol) extension number after logging in; through the voice service, the extension number obtained by the terminal can be registered. When the extension number conducts P2P (Point to Point) calls or initiates video conferences, it can apply for relevant resources from the voice media service through the voice service; when there are calls involving internal and external networks, intelligent bypassing is possible. When an extension calls an external landline or mobile number, it can connect to the voice gateway through the voice service (gateway intermediary service) to realize PSTN (Public Switched Telephone Network) incoming and outgoing calls; when an external number calls into the voice gateway, it will use the IVR (Interactive Voice Response) service of the voice service to provide intelligent voice menu playback, and can be transferred to an internal extension according to the voice prompts.

[0125] Furthermore, the interactive collaboration service module provides collaboration-related services to support whiteboard collaboration and annotation. It supports sending and receiving auxiliary whiteboards, multiple people writing annotations simultaneously, and erasing annotations, facilitating simultaneous collaboration among multiple people during meetings.

[0126] It should be noted that the IM status service module is used to handle the display and updating of users' online status. This module can determine the user's real-time online status by detecting various information such as the user's login status, network connection status, and whether the application is running in the foreground, and then display it on the user list, chat window, and other interfaces.

[0127] It should be noted that the IM session caching service module supports point-to-point and group messages, saves sessions in the cache, supports tens of thousands of concurrent chat messages, allows messages to be recalled within one minute, provides message notifications, and synchronizes messages across multiple devices. It can also automatically push historical messages to new group members.

[0128] It should be noted that the IM file service module supports on-demand local or distributed storage to ensure data security. File and image transfers support instant uploads and resume interrupted uploads, automatic file cleanup is configured periodically, and group chat file management is available.

[0129] It should be noted that the IM business processing service module can support mobile notification push by processing IM-related business, allow access to third-party push, and support electronic voting, electronic announcement permission management and viewing, etc.

[0130] Example 3

[0131] Reference Figure 3This is the third embodiment of the present invention. In order to verify the beneficial effects of the present invention, this embodiment is scientifically demonstrated through benefit calculation and simulation experiments.

[0132] First, the method described in the above embodiments is compared with a deployment method (before optimization) in a converged communication system that isolates audio, video, and IM (Instant Messenger) systems.

[0133] Operating environment: Network: Intranet and Internet; Operating system: CentOS 7.5; Number of CPUs (Central Processing Units): 2; Total number of CPU cores: 16; Total number of threads: 32; Memory: 64GB; Disk space: 500GB

[0134] Test Method: The optimized converged communication system of this patent is deployed on a single server. It connects to the operator's telephone line via SIP (Session Initialization Protocol) trunk mode and E1 (a telecommunications standard) network. It also connects to an analog gateway via SIP trunk mode to connect analog phones. It directly connects to video conferencing terminals, IP phones, and computers / mobile phones via IP (Internet Protocol) network and configures Internet mapping for external network connections. The test verifies the specific interconnection effect by testing different types of devices such as IP phones, computer / mobile phone clients, and video conferencing terminals to make calls to each other.

[0135] This experiment primarily verifies the connectivity between devices, aiming to enable audio or video calls, without presenting any actual data.

[0136] Experimental results: Through the optimization of this patent, the converged communication system can realize audio / video calls between analog phones, video conferencing terminals, IP phones, and computer / mobile phone clients, and the software client supports IM (Instant Messenger) communication function.

[0137] This allows users to freely choose devices such as IP phones, computer / mobile clients, and video conferencing terminals for communication, avoiding the drawbacks of traditional communication modes that require specifying communication devices, and improving communication efficiency by more than 50%.

[0138] Compared with traditional methods that isolate audio, video, and IM systems, this patent can improve the types of devices that the system can access for audio and video calls. It improves the situation from the original isolation of audio, video, and IM systems to support interconnection. Users can choose any device and method to conduct business communication as needed, thereby improving office communication efficiency.

[0139] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory, magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory, magnetic variable memory, ferroelectric memory, phase change memory, graphene memory, etc. Volatile memory can include random access memory or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory or dynamic random access memory, etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include blockchain-based distributed databases, etc., and are not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited thereto.

[0140] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An optimization method based on audio-video fusion communication technology, characterized in that, include: A centralized deployment scheme is adopted, with a converged communication server platform deployed at the central node; Establish a two-way transmission channel for local and external telephone calls, and initiate audio and video conferences; Synchronize contact information to the converged communication server; Devices in the headquarters system access the converged communication server for connection, while devices in the branch offices access the headquarters system through the converged communication server. The bidirectional transmission channel for local and external telephone lines includes: Connect to the voice gateway via SIP trunk link; The converged communication server transmits outbound calls for local calls to the voice gateway. The voice gateway transmits the external telephone call to the operator. When a local call makes an outside call, the call is routed back through the aforementioned operator. The process of synchronizing the contact information to the converged communication server includes: The converged communication server interfaces with the LDAP server; The converged communication server transmits authentication information to the LDAP server; After authentication by the LDAP server, the address book information is synchronized to the unified communications server. The audio and video fusion process of the fusion communication server includes: Determine the type of connected communication devices: When the connected communication devices are of the same type, they are grouped into a set of similar devices. No translation or encoding / decoding is required; the signal is directly output and the media stream is directly distributed to each communication device through the converged communication server. When the connected communication devices are of different types, they are grouped into sets of different types of devices, and translation is performed according to the audio and video encoding / decoding of different communication devices. Establish communication channels in different types of device sets and determine whether the connected communication devices support video capabilities: when the connected communication devices do not support video capabilities, the communication devices that do not support video capabilities are divided into a set of call-only devices, and the communication devices that support audio and video capabilities are divided into a set of call and video devices. For a set of call-only devices, the video stream in the media stream is removed, and the audio and video signals are decoded using audio and video codecs specific to different devices. The media stream is then distributed to the connected communication devices through the converged communication server. For a set of call and video devices, a weighted average method is used to weight and fuse the audio and video signals in the media stream. Then, the audio and video signals are decoded using audio and video codecs specific to different devices. The media stream is then distributed to the connected communication devices through the fusion communication server. Determine network signal status: When the network signal is good, it is classified as a smooth network state, and the audio and video signals are output at maximum power; when the network signal is average, it is classified as a good network state, and the weight of audio and video signals is adjusted evenly; when the network signal is poor, it is classified as a network latency state, and the network adaptive mode is enabled, and the weight of audio and video signals is adjusted, increasing the weight of audio signals and prioritizing audio calls. The signal fusion method for establishing a two-way transmission channel for local and external telephone lines includes: The audio signals are fused using a weighted average method based on the following formula: Where n represents the position on the time axis, y(n) is the output audio signal, and x i (n) represents the i-th input audio signal, w i (n) represents the weight value of the i-th audio signal at the n-th time. In the implementation of the weighted average method, the corresponding weight coefficients need to be calculated based on the signal-to-noise ratio and power of different audio signals. The audio signal is fused using a spectral subtraction algorithm. Where k represents the position on the frequency axis, Y(k) is the power spectrum of the output audio signal, and X... i (k) represents the power spectrum of the i-th audio signal, E i (k) represents the noise power spectrum estimated from the i-th audio signal, and N is the number of audio channels.

2. The optimization method based on audio and video fusion communication technology as described in claim 1, characterized in that: The time-domain fusion method for audio and video signals includes: Where t represents time, G(t) represents the fused audio and video signals, and X... i (t) represents the i-th audio signal, Y i (t) and the i-th video signal respectively, w i v i Let i be the weight of the i-th signal in the fusion process; The audio and video signals are fused based on the frequency domain: Where f represents frequency, X(f) and Y(f) represent the power spectra of the fused audio and video signals, respectively. i (f) represents the power spectrum of the i-th audio signal, Y i (f) represents the power spectrum of the i-th video signal, w i v i The definition and value are the same The same applies in the formula, where represents the weight of the i-th signal in the fusion process.

3. An optimized system based on audio and video fusion communication technology, characterized in that... The method described in claim 1 or 2 is characterized by comprising: The meeting service module is used for applying for media resources and controlling meetings, as well as providing private deployment solutions; The call service module is used to obtain SIP extension numbers, register the extension numbers obtained by the terminal, apply for relevant resources, connect to the voice gateway to realize PSTN inbound and outbound calls, and perform intelligent voice menu broadcasting. The interactive collaboration service module is used to support whiteboard collaboration and whiteboard annotation. The IM status service module is used to handle the display and updating of users' online status; The IM session caching service module is used to save sessions in the cache, as well as message notifications, message synchronization across multiple devices, and to automatically push historical messages to new group members. The IM file service module allows users to configure local or distributed storage as needed, ensuring data security, and supporting instant transfer and resume interrupted transfer. It also supports automatic file cleanup settings and group chat file management. The IM business processing service module is used for mobile notification push, allows access to third-party push, and supports electronic voting, electronic announcement permission management and viewing.

4. The optimized system based on audio and video fusion communication technology as described in claim 3, characterized in that: The meeting service module includes: meeting control service, meeting scheduling service, SFU media forwarding service, interactive live streaming service, live streaming push service, and recording service; The call service module includes: user center, voice service, voice media service, and voice load service; The IM file service module adopts on-demand settings for local or distributed storage to ensure data security; file and image transmission is achieved through instant transmission and breakpoint resume transmission, with automatic file cleanup settings and group chat file management.

5. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method described in claim 1 or 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method described in claim 1 or 2.

Images