Conference control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a conference control method and device, electronic equipment and a storage medium, and the method comprises the steps that a first server side can receive a first video stream obtained by a conference management system from a second conference terminal of a spokesman in a first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate; the first server side decodes the first video stream into first picture data and sends the first picture data to a second server side, so that the second server side adopts a spokesman face image corresponding to the second conference terminal to replace a face in the first picture data to obtain second picture data; and the first server receives the second picture data sent by the second server, synthesizes a second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal. Through the embodiment of the invention, the definition of the video frame is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of conference control, and in particular to a conference control method and device, electronic equipment, and storage medium. Background Art

[0002] When in a low-bandwidth environment, the video stream has severe packet loss, the video resolution is reduced, the clarity is reduced, and the speaker's image becomes blurry and unclear to the participants, affecting the normal playback of the video. Summary of the invention

[0003] In view of the above problems, a conference control method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A conference control method, involving a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the method comprises:

[0005] The first server receives a first video stream obtained by the conference management system from a second conference terminal of a speaker in the first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate;

[0006] The first server decodes the first video stream into first picture data, and sends the first picture data to the second server, so that the second server replaces the face in the first picture data with the speaker face image corresponding to the second conference terminal to obtain second picture data;

[0007] The first server receives the second picture data sent by the second server, and synthesizes a second video stream according to the second picture data;

[0008] The first server sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal.

[0009] Optionally, the first server decodes the first video stream into first picture data, including:

[0010] The first server decodes the first video stream according to a preset picture format to obtain first picture data corresponding to the first video stream.

[0011] Optionally, the first network conference further includes a control terminal for controlling the first network conference, and further includes:

[0012] The first server responds to the network access request sent by the controller after the anti-packet loss mode is turned on and the packet loss rate of the first conference terminal is detected to be higher than the preset packet loss rate, and sends the virtual terminal number used by the first server to join the first network conference in the form of a virtual terminal to the controller, so that the controller sends the virtual terminal number to the conference management system and controls the conference management system to add the virtual terminal to the first network conference according to the virtual terminal number.

[0013] Optionally, the conference management system includes hierarchically connected autonomous servers and core servers, the first conference terminal, the second conference terminal and the first server are connected to the same upper-level core server, and the upper-level server of the core server is an autonomous server.

[0014] A conference control method, involving a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the method comprises:

[0015] When the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate, the conference management system acquires a first video stream from the second conference terminal speaking in the first network conference;

[0016] The conference management system sends the first video stream to the first server, so that the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, receives the second picture data sent by the second server, synthesizes a second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, wherein the second picture data is generated by the second server using the spokesperson face image corresponding to the second conference terminal to replace the face in the first picture data.

[0017] Optionally, the first network conference further includes a control terminal for controlling the first network conference, and further includes:

[0018] The conference management system receives a virtual terminal number sent by the control terminal when the packet loss rate of the first conference terminal is higher than a preset packet loss rate, wherein the virtual terminal number is obtained from the server terminal when the controller detects that the packet loss rate of the first conference terminal is higher than the preset packet loss rate after the anti-packet loss mode is turned on, and is used by the server terminal to join the first network conference in the form of a virtual terminal;

[0019] The conference management system adds the virtual terminal to the first network conference according to the virtual terminal number, so that the virtual terminal joins the conference.

[0020] A conference control device, relating to a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the device comprises:

[0021] A first video stream receiving module, configured to receive, through the first server, a first video stream acquired by the conference management system from a second conference terminal of a speaker in the first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate;

[0022] A first picture data sending module, configured to decode the first video stream into first picture data through the first server, and send the first picture data to the second server, so that the second server replaces the face in the first picture data with the speaker face image corresponding to the second conference terminal to obtain second picture data;

[0023] A second video stream synthesis module, configured for the first server to receive the second picture data sent by the second server, and to synthesize a second video stream according to the second picture data;

[0024] The second video stream forwarding module is used for the first server to send the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal.

[0025] A conference control device, relating to a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the device comprises:

[0026] A first video stream acquisition module, configured to acquire a first video stream from a second conference terminal speaking in the first network conference through the conference management system when a packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate;

[0027] The first video stream sending module is used for the conference management system to send the first video stream to the first server, so that the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, receives the second picture data sent by the second server, and synthesizes the second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, wherein the second picture data is generated by the second server using the spokesperson face image corresponding to the second conference terminal to replace the face in the first picture data.

[0028] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the conference control method as described above when executed by the processor.

[0029] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the conference control method described above is implemented.

[0030] The embodiments of the present invention have the following advantages:

[0031] In an embodiment of the present invention, a first video stream obtained by a conference management system from a second conference terminal of a speaker in a first network conference when a packet loss rate of a first conference terminal in a first network conference is higher than a preset packet loss rate can be received through a first server end; then, the first server end decodes the first video stream into first picture data, and sends the first picture data to a second server end, so that the second server end replaces the face in the first picture data with the speaker's face image corresponding to the second conference terminal to obtain second picture data; the first server end receives the second picture data sent by the second server end, synthesizes a second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, and replaces the face in the video stream image frame through the second server end in the second network, thereby improving the clarity of the video frame, so that the conference terminal can still play a clear picture normally even in a serious packet loss state. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0033] Figure 1is a flowchart of a conference control method provided by an embodiment of the present invention;

[0034] Figure 2 is a flowchart of another conference control method provided by an embodiment of the present invention;

[0035] Figure 3a This is a schematic diagram of a video network conference architecture in an embodiment of the present invention;

[0036] Figure 3b This is a schematic diagram of a conference video stream processing process in an embodiment of the present invention;

[0037] Figure 4 is a structural diagram of a conference control device provided by an embodiment of the present invention;

[0038] Figure 5 It is a structural diagram of a conference control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Reference Figure 1 , shows a flowchart of the steps of a conference control method provided by an embodiment of the present invention, involving a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of a conference on the first network, and specifically may include the following steps:

[0041] Step S101, the first server receives a first video stream obtained by a conference management system from a second conference terminal of a speaker in a first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate;

[0042] In actual applications, when it is detected that the packet loss rate of the first conference terminal in the first network conference is higher than the preset packet loss rate, it can be determined that the current first conference terminal has serious packet loss, and the video stream currently received by the first conference terminal may have an unclear problem, which can trigger the first server in the embodiment of the present invention to participate in the conference as a virtual terminal and call the second server in the first network to perform image processing, so that the first conference terminal can display a clear video picture.

[0043] Specifically, during the first network conference, the conference management system can receive the video stream of the conference terminal where the speaker is located and transfer it to other conference terminals. When the packet loss rate of the conference terminal is higher than the preset packet loss rate, the conference management system can obtain the first video stream from the second conference terminal of the speaker in the current conference, and forward the first video stream to the first server, so that the first server can call the image processing technology of the second server in the second network to perform video image repair to obtain clear video data. Among them, the first server in the embodiment of the present invention can participate in the first network conference in the form of a virtual terminal, and then the conference management terminal can schedule the video stream between the first server (virtual terminal) and the speaker's second conference terminal.

[0044] In an embodiment of the present invention, the first network may be a visual network or other local area network, the first network conference is a conference conducted on the first network, and the second network is a network other than the first network. When the first network is a visual network, the first network conference is a visual network conference, and the second network may be an Internet network.

[0045] Among them, the meeting management system can realize the overall management of the first network conference, and the management process includes the scheduling management of the video stream in the first network conference. The meeting management system can specifically provide meeting management web services and conference scheduling services. The meeting management web services are responsible for the users, audits, system configurations, scheduled meetings, real-time meetings, etc. of the conference management system. The conference scheduling service is responsible for the internal logic processing of conference operations, such as: speaker switching, meeting start / stop, dynamic addition and deletion of participants, etc. The program is divided into the meeting management front-end WEB management and the background scheduling program. The scheduling program is divided into two sub-programs, CMS and DMS. CMS is responsible for interacting with the WEB interface, and DMS is responsible for the underlying meeting and control logic.

[0046] The meeting management system may include cascaded servers. Specifically, the meeting management system may include hierarchically connected autonomous servers and core servers. The conference management service system can be connected to the autonomous servers. The autonomous servers can control the visual network core servers to perform conference business operations such as holding and controlling meetings through the visual network protocol. The lower layer of the autonomous servers can be connected to one or more core servers. The audio and video streams of the conference terminals can be forwarded through the core servers. Each core server can manage the video stream scheduling of one or more conference terminals. The core server in the embodiment of the present invention can be a 2K server within the visual network autonomous domain.

[0047] In one example, the first conference terminal, the second conference terminal, and the first server are connected to the same upper-level core server, and the upper-level server of the core server is an autonomous server. In actual applications, a first server can be configured under each core server, and the first server is used to implement video stream processing for other conference terminals under the core server to ensure that the video images of the conference terminals are clear during the meeting. By ensuring that the first conference terminal, the second conference terminal, and the first server are connected to the same upper-level core server, the first conference terminal, the second conference terminal, and the first server can be transmitted within the same network domain, avoiding cross-domain transmission.

[0048] In an embodiment of the present invention, the first server end can transmit data with the conference terminal in the first network conference in the first network, and can also transmit data with the second server end in the second network. The first server end can be a service gateway, such as an AI service gateway.

[0049] The first server can be used to call services in the second network during the first network conference to make up for the defect that the terminals in the first network conference do not have computing power.

[0050] In one embodiment of the present invention, the first network conference also includes a control terminal for controlling the first network conference, and also includes: the first server terminal responds to the network access request sent by the controller after the anti-packet loss mode is turned on and the packet loss rate of the first conference terminal is detected to be higher than the preset packet loss rate, and sends the virtual terminal number used by the first server to join the first network conference in the form of a virtual terminal to the controller, so that the controller sends the virtual terminal number to the conference management system, and controls the conference management system to add the virtual terminal to the first network conference according to the virtual terminal number.

[0051] In actual applications, the controller can be used to control the first network conference. During the conference, if the controller starts the anti-packet loss mode, the controller can periodically monitor the packet loss rate of the conference terminal in the first network conference, by obtaining the packet loss rate information of the conference terminal in the first network conference and comparing the packet loss rate information with the preset packet loss rate. When the packet loss rate of the first conference terminal is less than the preset packet loss rate, the current first conference terminal can continue to schedule the video stream according to the original video stream scheduling method, that is, the conference management terminal directly forwards the first video stream of the second video terminal to the first video terminal; when the packet loss rate of the first conference terminal is greater than or equal to the preset packet loss rate, the first conference terminal suffers from severe packet loss, and then the conference management system can forward the video stream received by the second conference terminal to the virtual terminal (that is, the first server) in the first network conference, and the first server calls the second server in the second network to perform image processing. Among them, the controller in the embodiment of the present invention can be Pamir, which is a scheduling software in video network conferences.

[0052] When the controller detects that the packet loss rate of the first conference terminal is higher than the preset packet loss rate, it sends a network access request to the first server end, and the first server end can then send its corresponding virtual terminal number for joining the first network conference in the form of a virtual terminal to the controller, and the controller can send the virtual terminal number to the conference management system, and the conference management system adds the virtual terminal number to the first network conference, and then the first server end can enter the first network conference using the virtual terminal number as a virtual terminal.

[0053] Step S102, the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, so that the second server replaces the face in the first picture data with the speaker face image corresponding to the second conference terminal to obtain second picture data;

[0054] After receiving the first video stream, the first server may decode the first video stream to obtain first image data, wherein the decoded first image data is image data that can be processed by the second server in the second network.

[0055] The first server can send one or more decoded first image data to the second server, and the second server can pre-store facial images of important participants or speakers. The second server can then recognize the faces in the first image data and replace the speaker's facial image with the face in the first image data to obtain the second image data, and the second server can then transmit the second image data back to the first server.

[0056] In an embodiment of the present invention, the first server decodes the first video stream into first picture data, including: the first server decodes the first video stream according to a preset picture format to obtain the first picture data corresponding to the first video stream. The preset picture format may be JPG format.

[0057] The second server end in the embodiment of the present invention may be an image processing end in the second network. Specifically, the second server end may be an AI algorithm server end that applies face recognition technology and AI face-changing technology.

[0058] Step S103, the first server receives the second picture data sent by the second server, and synthesizes a second video stream according to the second picture data;

[0059] The first server can receive the second image data after the face replacement processing by the second server, and then can combine the plurality of second image data into a second video stream. After the face replacement by the second server, a clear second video stream can be synthesized.

[0060] Step S104: The first server sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal.

[0061] The first is that after receiving the second video stream, the server can send the second video stream to the conference management system, so that the conference management can forward the second video stream to the first conference terminal to display the emotional image in the first conference terminal.

[0062] In one embodiment of the present invention, the first server can join the visual network in four virtual terminal modes, and the codec of each virtual terminal is shown in Table 1 to Table 4:

[0063] Table 1 shows the meaning of the coding channels of the main speaker AI virtual terminal:

[0064]

[0065] Table 2 shows the meaning of the coding channel of the speaker-AI virtual terminal:

[0066]

[0067] Table 3 Meaning of the coding channel of the speaker 2 AI virtual terminal:

[0068]

[0069]

[0070] Table 4 Meaning of the encoding channel of the speaker three AI virtual terminal (reserved, speaker three has no video and does not need to be converted):

[0071]

[0072] Table 5 represents the meaning of encoding and decoding channels for ordinary participants:

[0073]

[0074] The flow adjustment rules based on the above table are as follows:

[0075] (1) When the main speaker's viewing party and speaker 1 are not on the same terminal, the main speaker's AI virtual terminal decodes channel 0 and tunes to the channel with encoding index 1 on the viewing party's terminal. The main speaker's AI virtual terminal decodes channel 4 and tunes to the sound of the ordinary terminal being viewed, encoding channel index 4. The main speaker's physical terminal decodes channel 0 for the ordinary participant and tunes to the video stream of the main speaker's AI virtual terminal with encoding index 0. The main speaker's physical terminal decodes channel 4 for the ordinary participant and tunes to the audio stream of the main speaker's AI virtual terminal with encoding index 4.

[0076] The flow survey of the main speaker's audience -> the flow survey of the main speaker's AI virtual terminal -> the main speaker's terminal

[0077] Adjust the stream to the main speaker AI virtual terminal: Note: [Main speaker AI virtual terminal decoding channel]: [Which terminal to watch]: [The encoding channel of the terminal to watch]

[0078] 0:[Main speaker audience]:[1] / / The 1 in the last [] represents the main speaker audience small stream encoding channel

[0079] 1:-1 2:-1 3:-1

[0080] 4:[Main speaker audience]:[4] / / The 4 in the last [] represents the audio encoding channel of the main speaker audience

[0081] 5:-1 6:-1 7:-1

[0082] Adjust the flow to the main speaker terminal:

[0083] 0:[AI virtual terminal]:[0] / / The 1 in the last [] represents the main speaker AI virtual terminal encoding channel

[0084] 1:-1 2:-1 3:-1

[0085] 4:[AI virtual terminal]:[4] / / The 4 in the last [] represents the sound encoding channel of the main speaker AI virtual terminal

[0086] 5:-1 6:-1 7:-1

[0087] (2) When there is a speaker, the speaker AI virtual terminal decoding index 1 channel is tuned to the physical speaker terminal encoding index 1 channel. When there is a supplementary stream, the decoding index 0 channel is tuned to the physical speaker terminal encoding index 3 channel. For the ordinary participant terminals watching speakers 1 and 2, the decoding of channels 0, 1, and 4 is tuned to the speaker AI virtual terminal index 0, 2, and 4 encoding channels.

[0088] The flow survey of the speaker's terminal is -> the flow survey of the speaker's AI virtual terminal is -> the ordinary participants (watching the speaker's terminal)

[0089] Give the speaker an AI virtual terminal to adjust the flow:

[0090] 0:[Speaker 1]:[1] / / The 1 in the last [] represents the main stream encoding channel of the small stream of the speaker 1 terminal

[0091] 1:[Speaker 1]:[3] / / The 3 in the last [] represents the encoding channel of the small stream auxiliary stream of the speaker 1 terminal

[0092] 2:-1 3:-1

[0093] 4:[Speaker 1]:[4] / / The 4 in the last [] represents the audio coding channel of the speaker 1 terminal

[0094] 5:-1 6:-1 7:-1

[0095] Adjust the stream for the terminals of ordinary participants who are watching speaker 1:

[0096] 0:[AI virtual terminal]:[0] / / The 1 in the last [] represents the mainstream encoding channel of the speaker AI virtual terminal

[0097] 1:[AI virtual terminal]:[2] / / The 2 in the last [] represents the speaker's AI virtual terminal auxiliary stream encoding channel

[0098] 2:-1 3:-1

[0099] 4:[AI virtual terminal]:[4] / / The 4 in the last [] represents the speaker's AI virtual terminal sound encoding channel

[0100] 5:-1 6:-1 7:-1

[0101] (3) When the speaker 2 is included, the speaker 2 AI virtual terminal decoding index channel is tuned to the physical speaker 2 terminal coding index 1 channel, and the speaker 2 virtual terminal coding 0 channel is sent to the participants watching the 2nd terminal.

[0102] The flow survey of the speaker's terminal is -> the flow survey of the speaker's AI virtual terminal is -> the ordinary participants (watching the speaker's terminal)

[0103] Adjust the traffic for the second speaker AI virtual terminal:

[0104] 0:[Speaker 2]:[1] / / The 1 in the last [] represents the main stream encoding channel of the small stream of the speaker 2 terminal

[0105] 1:-1 2:-1 3:-1

[0106] 4:[Speaker 2]:[4] / / The 4 in the last [] represents the audio coding channel of the speaker 2 terminal

[0107] 5:-1 6:-1 7:-1

[0108] Adjust the stream for the terminals of ordinary participants who are watching speaker 2:

[0109] Note: [Decoding channel of the speaker 2 ordinary participant terminal]: [Which terminal to watch]: [Encoding channel of the terminal to watch]

[0110] 0:[AI virtual terminal]:[0] / / The 1 in the last [] represents the mainstream encoding channel of the speaker 2 AI virtual terminal

[0111] 1:-1 2:-1 3:-1

[0112] 4:[AI virtual terminal]:[4] / / The 4 in the last [] represents the sound encoding channel of the speaker 2 AI virtual terminal

[0113] 5:-1 6:-1 7:-1

[0114] In an embodiment of the present invention, a first video stream obtained by a conference management system from a second conference terminal of a speaker in a first network conference when a packet loss rate of a first conference terminal in a first network conference is higher than a preset packet loss rate can be received through a first server end; then, the first server end decodes the first video stream into first picture data, and sends the first picture data to a second server end, so that the second server end replaces the face in the first picture data with the speaker's face image corresponding to the second conference terminal to obtain second picture data; the first server end receives the second picture data sent by the second server end, synthesizes a second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, and replaces the face in the video stream image frame through the second server end in the second network, thereby improving the clarity of the video frame, so that the conference terminal can still play a clear picture normally even in a serious packet loss state.

[0115] Reference Figure 2 , shows a flowchart of the steps of another conference control method provided by an embodiment of the present invention, involving a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and specifically may include the following steps:

[0116] Step S201, when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate, the conference management system obtains a first video stream from the second conference terminal speaking in the first network conference;

[0117] In one embodiment of the present invention, the first network conference also includes a control terminal for controlling the first network conference, and also includes: a conference management system receives a virtual terminal number sent by the control terminal when the packet loss rate of the first conference terminal is higher than a preset packet loss rate, the virtual terminal number being obtained from the server when the controller detects that the packet loss rate of the first conference terminal is higher than the preset packet loss rate after turning on the anti-packet loss mode, and used by the server to join the first network conference in the form of a virtual terminal; the conference management system adds the virtual terminal to the first network conference according to the virtual terminal number to enable the virtual terminal to join the conference.

[0118] In step S202, the conference management system sends the first video stream to the first server, so that the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, receives the second picture data sent by the second server, synthesizes the second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, wherein the second picture data is generated by the second server using the speaker's face image corresponding to the second conference terminal to replace the face in the first picture data.

[0119] In the embodiment of the present invention, the second server in the second network performs face replacement in the video stream image frame to improve the clarity of the video frame, so that the conference terminal can still play clear images normally even in a serious packet loss state.

[0120] Reference Figure 3a , is a schematic diagram of a visual network conference architecture in an embodiment of the present invention. The architecture includes a conference management service system, an autonomous service, a 2k core server 1, a 2k core server 2, pamir, an AI algorithm server, a conference terminal, and an AI gateway service.

[0121] Among them, the 2k core server 1 and the 2k core server 2 upper servers are autonomous services. The 2k core server 1 is used to schedule conference terminal 1, conference terminal 2 and AI gateway service 1, and the 2k core server 2 is used to schedule conference terminal 3, conference terminal 4 and AI gateway service 2.

[0122] AI Gateway Service 1 and AI Gateway Service 2 are connected to the AI ​​algorithm service, and can call upon the computing power of the AI ​​algorithm service to realize face recognition and replacement of images in the visual network, thereby improving the clarity of the images.

[0123] Reference Figure 3b , is a schematic diagram of a conference video stream processing process in an embodiment of the present invention. Specifically, the process includes the following steps:

[0124] (1) Before the meeting starts, photos of important participants or speakers can be saved in the AI ​​algorithm library;

[0125] (2) AI gateway service enters the visual network in the form of a virtual terminal of the visual network: Pamir configures a group of AI gateway service numbers. One AI gateway service can be deployed on a 2K server. A terminal under a 2K uses the same AI gateway service. In the low bandwidth and anti-packet loss scenario, Pamir starts a meeting;

[0126] When the bandwidth is insufficient and the terminal detects serious packet loss, the terminal reports the packet loss rate to Pamir through the transparent transmission interface. Pamir determines whether AI face swapping is needed based on the packet loss rate. Pamir obtains 4 AI virtual terminals from the AI ​​gateway service. The conference manager and Pamir add new participant roles: AI virtual terminal, encoding channel: 5, decoding channel: 8; add them to the conference in the order of the main speaker, speaker 1, speaker 2, and speaker 3.

[0127] (3) After the AI ​​virtual terminal joins the conference, the conference manager will adjust the speaker's audio and video stream for the AI ​​virtual terminal. Pamir notifies the conference manager to let the physical terminal with severe packet loss watch the AI ​​virtual terminal. The conference manager adjusts the stream for the physical terminal with severe packet loss to watch the audio and video stream of the AI ​​virtual terminal.

[0128] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0129] Reference Figure 4 , shows a schematic diagram of the structure of a conference control device provided by an embodiment of the present invention, involving a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores the face image of the speaker of the first network conference, and specifically may include the following modules:

[0130] The first video stream receiving module 401 is used to receive, through the first server, a first video stream obtained by the conference management system from a second conference terminal of a speaker in the first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate;

[0131] The first picture data sending module 402 is used to decode the first video stream into first picture data through the first server, and send the first picture data to the second server, so that the second server replaces the face in the first picture data with the speaker face image corresponding to the second conference terminal to obtain second picture data;

[0132] A second video stream synthesis module 403, configured for the first server to receive the second picture data sent by the second server, and to synthesize a second video stream according to the second picture data;

[0133] The second video stream forwarding module 404 is used for the first server to send the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal.

[0134] In an embodiment of the present invention, the first picture data sending module 402 may include:

[0135] The first picture data acquisition submodule is used to decode the first video stream according to a preset picture format using the first server to obtain first picture data corresponding to the first video stream.

[0136] In an embodiment of the present invention, the first network conference further includes a control terminal for controlling the first network conference, and the device further includes:

[0137] A virtual terminal number sending module is used to use the first server to respond to the network access request sent by the controller after the anti-packet loss mode is turned on and the packet loss rate of the first conference terminal is detected to be higher than the preset packet loss rate, and send the virtual terminal number used by the server to join the first network conference in the form of a virtual terminal to the controller, so that the controller sends the virtual terminal number to the conference management system and controls the conference management system to add the virtual terminal to the first network conference according to the virtual terminal number.

[0138] In one embodiment of the present invention, the conference management system includes hierarchically connected autonomous servers and core servers, the first conference terminal, the second conference terminal and the first server are connected to the same upper-level core server, and the upper-level server of the core server is an autonomous server.

[0139] Reference Figure 5, shows a schematic diagram of the structure of a conference control device provided by an embodiment of the present invention, involving a conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores the face image of the speaker of the first network conference, and specifically may include the following modules:

[0140] A first video stream acquisition module 501 is used to acquire a first video stream from a second conference terminal speaking in the first network conference through the conference management system when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate;

[0141] The first video stream sending module 502 is used for the conference management system to send the first video stream to the first server, so that the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, receives the second picture data sent by the second server, and synthesizes a second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, wherein the second picture data is generated by the second server using the spokesperson face image corresponding to the second conference terminal to replace the face in the first picture data.

[0142] In an embodiment of the present invention, the first network conference further includes a control terminal for controlling the first network conference, and the device further includes:

[0143] A virtual terminal number receiving module, used for receiving, through the conference management system, a virtual terminal number sent by the control terminal when the packet loss rate of the first conference terminal is higher than a preset packet loss rate, wherein the virtual terminal number is obtained from the server when the controller detects that the packet loss rate of the first conference terminal is higher than the preset packet loss rate after the anti-packet loss mode is turned on, and is used by the server to join the first network conference in the form of a virtual terminal;

[0144] The virtual terminal joining module is used for the conference management system to add the virtual terminal to the first network conference according to the virtual terminal number, so that the virtual terminal joins the conference.

[0145] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above conference control method when executed by the processor.

[0146] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the conference control method described above is implemented.

[0147] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0149] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0150] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0154] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0155] The above is a detailed introduction to a conference control method and device, electronic device, and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A conference control method, characterized in that: A conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the method comprises: The first server receives a first video stream obtained by the conference management system from a second conference terminal of a speaker in the first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate; The first server decodes the first video stream into first picture data, and sends the first picture data to the second server, so that the second server replaces the face in the first picture data with the speaker face image corresponding to the second conference terminal to obtain second picture data; The first server receives the second picture data sent by the second server, and synthesizes a second video stream according to the second picture data; The first server sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal.

2. The method according to claim 1, characterized in that The first server decodes the first video stream into first picture data, including: The first server decodes the first video stream according to a preset picture format to obtain first picture data corresponding to the first video stream.

3. The method according to claim 1, characterized in that The first network conference also includes a control terminal for controlling the first network conference, and further includes: The first server responds to the network access request sent by the controller after the anti-packet loss mode is turned on and the packet loss rate of the first conference terminal is detected to be higher than the preset packet loss rate, and sends the virtual terminal number used by the first server to join the first network conference in the form of a virtual terminal to the controller, so that the controller sends the virtual terminal number to the conference management system and controls the conference management system to add the virtual terminal to the first network conference according to the virtual terminal number.

4. The method according to claim 1, characterized in that: The conference management system includes hierarchically connected autonomous servers and core servers. The first conference terminal, the second conference terminal and the first server are connected to the same upper-level core server. The upper-level server of the core server is an autonomous server.

5. A conference control method, characterized in that: A conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the method comprises: When the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate, the conference management system acquires a first video stream from the second conference terminal speaking in the first network conference; The conference management system sends the first video stream to the first server, so that the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, receives the second picture data sent by the second server, synthesizes a second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, wherein the second picture data is generated by the second server using the spokesperson face image corresponding to the second conference terminal to replace the face in the first picture data.

6. The method according to claim 5, characterized in that The first network conference also includes a control terminal for controlling the first network conference, and further includes: The conference management system receives a virtual terminal number sent by the control terminal when the packet loss rate of the first conference terminal is higher than a preset packet loss rate, wherein the virtual terminal number is obtained from the server terminal when the controller detects that the packet loss rate of the first conference terminal is higher than the preset packet loss rate after the anti-packet loss mode is turned on, and is used by the server terminal to join the first network conference in the form of a virtual terminal; The conference management system adds the virtual terminal to the first network conference according to the virtual terminal number, so that the virtual terminal joins the conference.

7. A conference control device, characterized in that: A conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the device comprises: A first video stream receiving module, configured to receive, through the first server, a first video stream acquired by the conference management system from a second conference terminal of a speaker in the first network conference when the packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate; A first picture data sending module, configured to decode the first video stream into first picture data through the first server, and send the first picture data to the second server, so that the second server replaces the face in the first picture data with the speaker face image corresponding to the second conference terminal to obtain second picture data; A second video stream synthesis module, configured for the first server to receive the second picture data sent by the second server, and to synthesize a second video stream according to the second picture data; The second video stream forwarding module is used for the first server to send the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal.

8. A conference control device, characterized in that: A conference management system for scheduling conference video streams in a first network, a first server end for connecting the first network with a second network, and a second server end for image processing in the second network, wherein the second server end stores a face image of a speaker of the first network conference, and the device comprises: A first video stream acquisition module, configured to acquire a first video stream from a second conference terminal speaking in the first network conference through the conference management system when a packet loss rate of the first conference terminal in the first network conference is higher than a preset packet loss rate; The first video stream sending module is used for the conference management system to send the first video stream to the first server, so that the first server decodes the first video stream into first picture data, and sends the first picture data to the second server, receives the second picture data sent by the second server, and synthesizes the second video stream according to the second picture data, and sends the second video stream to the conference management system, so that the conference management system forwards the second video stream to the first conference terminal, wherein the second picture data is generated by the second server using the spokesperson face image corresponding to the second conference terminal to replace the face in the first picture data.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the conference control method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the conference control method according to any one of claims 1 to 6 is implemented.

Images