Video conference implementation method, platform, equipment and storage medium

Abstract

The invention provides a video conference implementation method, platform and device and a storage medium, the method is implemented by adopting a video conference platform, and the video conference platform comprises decoders corresponding to various sending ends and encoders corresponding to various receiving ends. The method comprises the following steps: a target decoder corresponding to a target sending end receives a video code stream sent by the target sending end, and decodes the video code stream to obtain a plurality of video frames; the target decoder judges whether a preset frame loss condition is met currently or not, if yes, the target decoder discards part of video frames and sends the video frames which are not discarded to a target encoder corresponding to a target receiving end, and if not, the target decoder sends the video frames to the target encoder; and the target encoder encodes the received video frame to obtain an encoded code stream and sends the encoded code stream to the target receiving end. Through adoption of the method and the device, the problems of image transmission delay and image jamming in the video conference are solved, and the conference quality is improved.

Description

Background Art

[0002] In a full-encoding and full-decoding video conference, the implementation mode of the video conference is as follows: a codec will be assigned to each terminal that joins the conference, and the video conference platform will perform secondary decoding on the video code streams uploaded by all speaking terminals, and then perform secondary encoding based on the decoding capabilities of the audience terminals. In this mode, terminals with high video capabilities do not need to be backward compatible with terminals with low capabilities, which is conducive to improving the overall image quality of the conference. A single-level conference can be implemented through a video conference platform, or multiple video conference platforms can be combined to form a large cascade conference, with the top-level conference being the main control conference. Channel cascading technology is used between conferences to achieve mutual transmission of code streams between conferences, which can solve problems such as insufficient bandwidth of a single conference line and insufficient capacity of a single conference port. In the process of video conference code stream transmission, since each video conference platform needs to go through a complete transmission link of receiving, decoding, encoding, and sending, the video code stream processing time is relatively long, resulting in image transmission delay problems, which affects the experience of the audience terminal when watching the video. Summary of the invention

[0003] In view of the problems in the prior art, the purpose of this application is to provide a video conferencing implementation method, platform, device and storage medium to solve the problem of image transmission delay in video conferencing.

[0004] The embodiment of the present application provides a method for implementing a video conference, which is implemented using a video conference platform. The video conference platform includes a decoder corresponding to each sending end and an encoder corresponding to each receiving end. The method includes the following steps:

[0005] The method is implemented by using a video conferencing platform, which includes a decoder corresponding to each sending end and an encoder corresponding to each receiving end. The method includes the following steps:

[0006] The target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, and decodes it to obtain multiple video frames;

[0007] The target decoder determines whether the preset frame loss condition is currently met. If yes, the target decoder discards some video frames and sends the undiscarded video frames to the target encoder corresponding to the target receiving end. If no, the target decoder sends the video frames to the target encoder;

[0008] The target encoder encodes the received video frames, obtains the encoded bitstream and sends it to the target receiving end.

[0009] In some embodiments, after receiving the video code stream sent by the target sending end, the target decoder corresponding to the target sending end stores the video code stream in a cache, and decodes the video code stream in the cache to obtain multiple video frames;

[0010] The preset frame loss condition includes that the size of the video stream in the buffer is greater than a preset buffer data amount threshold.

[0011] In some embodiments, the video conferencing platform also includes a media network receiving module. Before the target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, it also includes: based on the H320 transmission protocol, the media network receiving module receives the video code stream sent by the target sending end and sends it directly to the target decoder.

[0012] In some embodiments, the video conferencing platform further includes a media network sending module and a media network receiving module. Before the target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, the following steps are also included:

[0013] Determine a target decoder corresponding to a target transmitting end and a target encoder corresponding to a target receiving end;

[0014] Compare the video parameters of the target decoder and the video parameters of the target encoder to see if they are consistent;

[0015] If they are consistent, it is determined that the transparent transmission mode is adopted, and a transparent transmission link from the media network receiving module to the media network sending module is established. The media network receiving module receives the video code stream sent by the target sending end and sends it to the media network sending module through the transparent transmission link. The media network sending module sends the video code stream to the target receiving end;

[0016] If they are inconsistent, it is determined to adopt the secondary encoding and decoding mode, bind the target decoder to the target encoder, and the media network receiving module receives the video code stream sent by the target sending end and sends it to the target decoder.

[0017] In some embodiments, after establishing a transparent transmission link from the media network receiving module to the media network sending module, the following steps are also included:

[0018] Receiving a video parameter change notification from a target transmitting end and / or a target receiving end;

[0019] After the video parameters of the target decoder and / or the target encoder are changed, the video parameters of the target decoder and the target encoder are compared, and if they are inconsistent, the secondary encoding and decoding mode is switched.

[0020] In some embodiments, the following steps are also included:

[0021] The first video conferencing platform receives the code stream transmission signaling, determines the first speaking terminal and the audience terminal for code stream transmission, determines the first speaking terminal as the first target sending terminal and the audience terminal as the first target receiving terminal, and binds the first decoder corresponding to the first target sending terminal and the first encoder corresponding to the first target receiving terminal;

[0022] The first video conferencing platform receives a first switching notification to switch the first speaking terminal to the second speaking terminal, and determines a second video conferencing platform to which the second speaking terminal is connected;

[0023] The first video conferencing platform sends a second switching notification that needs to be transmitted to the second speaking terminal code stream to the second video conferencing platform, and the second video conferencing platform is configured to bind the second decoder corresponding to the second speaking terminal as the second target transmitting end with the second encoder corresponding to the first video conferencing platform as the second target receiving end after receiving the second switching notification;

[0024] In the first video conferencing platform, the third decoder corresponding to the second video conferencing platform as the third target sending end is bound to the first encoder, and the first decoder is unbound from the first encoder.

[0025] In some embodiments, after the first video conferencing platform sends the second switching notification that needs to be transmitted to the second speaking terminal code stream to the second video conferencing platform, the following steps are also included:

[0026] In the first video conferencing platform, determining whether the third decoder receives a code stream key frame from the second video conferencing platform;

[0027] If so, the first video conferencing platform binds the third decoder to the first encoder, and unbinds the first decoder from the first encoder.

[0028] The embodiment of the present application also provides a video conferencing platform, which is applied to the above-mentioned video conferencing implementation method. The platform includes a decoder corresponding to each sending end and an encoder corresponding to each receiving end; the decoder and the encoder are configured to perform the following steps:

[0029] The target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, and decodes it to obtain multiple video frames;

[0030] The target decoder determines whether the preset frame loss condition is currently met. If yes, the target decoder discards some video frames and sends the undiscarded video frames to the encoder corresponding to the target receiving end. If no, the target decoder sends the video frames to the target encoder corresponding to the target receiving end.

[0031] The target encoder encodes the received video frames, obtains the encoded bitstream and sends it to the target receiving end.

[0032] The embodiment of the present application also provides a video conferencing implementation device, including:

[0033] processor;

[0034] a memory storing executable instructions of the processor;

[0035] Wherein, the processor is configured to perform the steps of the video conferencing implementation method by executing the executable instructions.

[0036] An embodiment of the present application also provides a computer-readable storage medium for storing a program, which implements the steps of the video conferencing implementation method when executed by a processor.

[0037] By adopting the video conferencing implementation method, platform, device and storage medium of the present application, the following beneficial effects are achieved:

[0038] By adopting the video conferencing implementation method of the present application, after receiving the video code stream from the target sending end, the target decoder corresponding to the target sending end decodes to obtain the video frame and sends it to the target encoder corresponding to the target receiving end. When sending the video frame to the target encoder, the target decoder executes the frame loss strategy, firstly judging whether the preset frame loss condition is met, if the frame loss condition is met, the target decoder discards some video frames and sends the non-discarded video frames to the target decoder. Since the target decoder sends the encoder a raw code stream, the target encoder does not need other dependencies for encoding, and can directly encode, which will not affect the normal transmission of the video screen, and the data transmission rate from the target decoder to the target encoder can be improved through frame loss processing, the number of video frames received by the target encoder end is reduced, the encoding speed is also accelerated, and the speed at which the code stream encoded by the target encoder is sent to the target receiving end is also faster, thereby improving the transmission speed of the video code stream from the target sending end to the target receiving end, reducing the transmission time consumption, and reducing the video conferencing delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0040] Figure 1 is a flow chart of a method for implementing a video conference according to an embodiment of the present application;

[0041] Figure 2 is a structural block diagram of a video conferencing platform according to an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of the code stream link of a cascade conference;

[0043] Figure 4 This is a schematic diagram of the code stream link of the cascade conference after adding the transparent transmission link;

[0044] Figure 5 It is a schematic diagram of the code stream link switching process when switching the speaking terminal in a cascade conference in the prior art;

[0045] Figure 6It is a schematic diagram of optimizing the code stream link switching process when switching the speaking terminal in a cascade conference;

[0046] Figure 7 It is a structural diagram of a video conferencing implementation device according to an embodiment of the present application;

[0047] Figure 8 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.

[0049] like Figure 1 As shown, in one embodiment, the present application provides a method for implementing a video conference, which is implemented using a video conference platform. The video conference platform includes a decoder corresponding to each sending end and an encoder corresponding to each receiving end. The method includes the following steps:

[0050] S100: a target decoder corresponding to a target transmitting end receives a video code stream sent by the target transmitting end, and decodes the video code stream to obtain multiple video frames;

[0051] For the video conferencing platform, the target sending end may be a terminal (such as a speaking terminal) or another video conferencing platform. The other video conferencing platform may be a platform accessed by the speaking terminal or a platform transferred in the middle of a cascade conference. The target sending end here may be any sending end, which here refers to the sending end of the current code stream transmission. The target receiving end refers to the receiving end that receives the code stream sent by the target sending end. The target receiving end may be one or more.

[0052] When a terminal or another platform is connected to the video conferencing platform, the video conferencing platform will allocate a decoder and an encoder to the terminal / platform. The decoder is used to decode the video code stream sent by the terminal / platform, and the encoder is used to encode the video code stream to be sent to the terminal / platform. A terminal / platform can be used as both a transmitter and a receiver. Here, a code stream transmission is taken as an example. Therefore, corresponding to a video conferencing platform, the transmitter of this code stream is called the target transmitter, the decoder corresponding to the target transmitter is called the target decoder, the receiver of this code stream is called the target receiver, and the encoder corresponding to the target receiver is the target encoder. The corresponding target receiver can be one or more, and the target encoder can be one or more.

[0053] After determining the code stream transmission direction, on a video conferencing platform, a target decoder corresponding to a target transmitting end is bound to a target encoder corresponding to a target receiving end; the target decoder receives the video code stream sent from the target transmitting end, decodes it, and then sends it to the target encoder, which encodes it and then sends it to the target receiving end. There may be one or more target encoders, and the target decoder may be bound to one or more target encoders at the same time;

[0054] S200: The target decoder determines whether a preset frame loss condition is currently met;

[0055] If yes, then S300: the target decoder discards some video frames and sends the undiscarded video frames to the target encoder corresponding to the target receiving end;

[0056] Therefore, when the preset frame loss condition is met, the target decoder discards some video frames to reduce the number of video frames transmitted to the target encoder, thereby increasing the video frame transmission speed and reducing the transmission delay of the video code stream in the video conferencing platform;

[0057] If not, then S400: the target decoder sends the video frame to the target encoder;

[0058] That is, when the preset frame loss condition is not met, that is, when frame loss is not required, the target decoder does not perform a frame loss operation, and sends all decoded video frames to the target encoder;

[0059] S500: The target encoder encodes the received video frame to obtain an encoded code stream and sends it to the target receiving end.

[0060] If the target receiving end is the audience terminal, it can decode and play after receiving the code stream. If the target receiving end is another platform, the other platform serving as the target receiving end will forward the encoded code stream to the audience terminal. During the forwarding process, the decoder that receives the encoded code stream in the other platform can also adopt the above-mentioned frame loss method. When the preset frame loss condition is met, some video frames are discarded and the non-discarded video frames are sent to the encoder corresponding to the audience terminal. The encoder corresponding to the audience terminal encodes them and then sends them to the audience terminal, and the audience terminal can decode and play them.

[0061] By adopting the video conferencing implementation method of the present application, through step S100, after receiving the video code stream of the target sending end, the target decoder corresponding to the target sending end decodes to obtain the video frame and then sends it to the target encoder corresponding to the target receiving end. When sending the video frame to the target encoder, the target decoder executes the frame loss strategy through steps S200 to S400. First, it is determined whether the preset frame loss condition is met. If the frame loss condition is met, the target decoder discards some video frames and sends the non-discarded video frames to the target decoder. Since the target decoder sends the target encoder a raw code stream, the target encoder does not need other dependencies for encoding, and can directly encode, which will not affect the normal transmission of the video picture. In addition, the data transmission rate from the target decoder to the target encoder can be improved through frame loss processing, the number of video frames received by the target encoder end is reduced, the encoding speed is also accelerated, and the speed at which the code stream encoded by the target encoder is sent to the target receiving end is also faster, thereby improving the transmission speed of the video code stream from the target sending end to the target receiving end, reducing the transmission time consumption, and reducing the video conference delay.

[0062] The video conferencing implementation method can be applied to a scenario with only one video conferencing platform, and the target sending end and the target receiving end are the speaking terminal and the audience terminal, respectively. The video conferencing implementation method can also be applied to a scenario with multiple video conferencing platforms cascaded. For a code stream transmission of a video conferencing platform, the target sending end is the speaking terminal or another platform, and the target receiving end is the audience terminal or another platform. When the video conferencing implementation method is applied to a scenario with multiple video conferencing platforms cascaded, the decoders of all or part of the video conferencing platforms can adopt the cache smoothing processing and frame loss processing strategy of the target decoder in the video conferencing implementation method. Moreover, when the method is applied to a cascaded conference scenario, when the code stream is transmitted across platforms between the target sending end and the target receiving end, the decoders and encoders in each video conferencing platform can adopt the processing strategy of the method. For the code stream transmission of a long code stream link, the optimization effect in terms of transmission delay and audio and video synchronization is more obvious.

[0063] The video conferencing implementation method and video conferencing platform can be used in scenarios that support 4K resolution video based on the H320 transmission protocol. Since the higher the resolution, the higher the frame rate, the more data needs to be transmitted, the larger the transmission bandwidth needs to be, and the code stream fluctuation may be more obvious. Currently, many devices support 4K resolution. The H320 transmission protocol is implemented based on narrow bandwidth, and its line transmission bandwidth is fixed, and the transmission rate is also fixed. If you need to support higher resolutions, you need to merge the bandwidth of multiple lines to increase the transmission bandwidth. When some lines are abnormal / recovered, corresponding speed increase and decrease processing is required to ensure normal images.

[0064] like Figure 2 FIG. 1 shows a structural block diagram of a video conferencing platform according to an embodiment of the present application. Figure 2 As shown, the video conferencing platform includes an encoder and a decoder assigned to each terminal and / or platform accessing the platform, each decoder corresponds to a transmitting end, and each encoder corresponds to a receiving end. The figure shows n encoders (ENC) and n decoders (DEC) by way of example, where n>1. The video conferencing platform is the MCU (media server, as a multipoint control unit, used to control the interaction of audio and video streams and data of each terminal in the conference) in the video conferencing scenario. The video conferencing platform also includes a media network sending module, a media network receiving module, and a control module. The media network receiving module is used to receive media data from a terminal or other platform, and the media network sending module is used to send media data to a terminal or other platform. The media network receiving module and the media network sending module are also used to implement functions such as packet loss retransmission. The decoder is used to decode the video stream uploaded by the corresponding terminal or platform to obtain a video frame. The encoder is used to encode the video frame to be sent to the corresponding terminal or platform to obtain an encoded stream. In a video stream transmission, the decoder and the encoder are used to execute steps S100 to S400 in the above-mentioned video conferencing implementation method.

[0065] The control module is used to manage decoders and encoders. When a terminal or another platform accesses the video conferencing platform, the control module allocates an encoder and a decoder to the terminal or the platform. When the video conferencing platform receives a code stream transmission instruction, the control module determines the target decoder corresponding to the target sending end and the target encoder corresponding to the target receiving end according to the code stream transmission instruction, and establishes a binding between the target decoder and the target encoder. After the code stream transmission starts, after the media network receiving module receives the video code stream from the target sending end, it sends it to the target decoder for decoding. After the target decoder decodes and obtains the video frame, the frame loss strategy is executed using the method of steps S200 to S400 above, and then it is sent to the target encoder corresponding to the target receiving end. After the target encoder encodes, the encoded code stream is obtained and sent to the media network sending module. The media network sending module sends the encoded code stream to the target receiving end.

[0066] In the existing full-encoding and full-decoding conference, after the target transmitter sends the video code stream to the media network receiving module of the platform, the media network receiving module sends the video code stream to the target decoder corresponding to the target transmitter. After decoding, the target decoder sends the video frame to the target encoder corresponding to the target receiving end. The target encoder encodes and sends it to the media network sending module. The media network sending module sends the encoded code stream to the target receiving end. In the case of cascading multiple video conferencing platforms, each video conferencing platform needs to be transmitted through the transmission link, which takes a long time to transmit. In addition, the decoding time and encoding time of the target decoder and the target encoder further increase the transmission time, which easily leads to image transmission delay. In video conferencing, since audio and images are transmitted separately, the amount of audio data is smaller than that of image data, and the transmission speed is faster. Compared with audio, the image has a larger data volume, and in the scenario of using the H320 transmission protocol with 4K high resolution and fixed transmission bandwidth, the transmission speed of the image in the existing solution is more susceptible to code flow fluctuations and network fluctuations. When the network bandwidth cannot meet the transmission requirements or code flow / network fluctuations occur, the transmission of the image will be delayed, and it will not be able to reach the target receiving end synchronously with the audio with a faster transmission speed. When watching the video at the target receiving end, the transmission delay will cause obvious audio and video asynchrony. Especially in the video conferencing scenario of three-level cascade or more levels of cascade, the code stream link is lengthened, the transmission time consumption is further increased, and there is obvious audio and video asynchrony. By adopting the video conferencing implementation method of the present application, when the preset frame loss condition is met, the amount of video frame data sent by the target decoder to the target encoder is reduced, the transmission time in the transmission link is increased, and it is beneficial to reduce the encoding time of the target encoder, thereby helping to reduce the image transmission delay, so that the image can basically keep synchronization with the audio to reach the target receiving end, thereby reducing the audio and video asynchrony.

[0067] Figure 3 It is a schematic diagram of the code stream link of a cascade conference. Here, a two-level video conferencing platform is taken as an example. In order to distinguish, it is defined as the first video conferencing platform and the second video conferencing platform. Platform peripherals refer to a collection of a series of processes in the MCU, that is, a collection of processes that implement the above-mentioned network terminal sending module, network terminal receiving module, decoder, encoder, and control module. Figure 3 The control module is omitted. In a cascade conference, the top-level conference is the master conference. Channel cascading technology is used between conferences to achieve mutual transmission of code streams between conferences. It can be used to solve the problem of insufficient bandwidth of a single conference line and insufficient capacity of a single conference port. It can also be used in scenarios where multiple regions need to hold joint meetings, which can easily solve the problem of regional management. In actual applications, different types of cascade modes such as two-level, three-level, and four-level can be set in cascade conferences. Figure 3 For example, the speaking terminal is connected to the first video conferencing platform, and the audience terminal is connected to the second video conferencing platform. For the first video conferencing platform, when executing steps S100 to S400, the target sending end is the speaking terminal, the target decoder is the decoder corresponding to the speaking terminal, the target receiving end is the second video conferencing platform, and the target encoder is the encoder corresponding to the second video conferencing platform. For the second video conferencing platform, when executing steps S100 to S400, the target sending end is the first video conferencing platform, the target decoder is the decoder corresponding to the first video conferencing platform, the target receiving end is the audience terminal, and the target encoder is the encoder corresponding to the audience terminal. The speaking terminal includes a camera, an encoder and a sending module. The camera is used to collect the speaker's video. After being encoded by the encoder of the speaking terminal, a video stream is obtained, which is sent to the media network receiving module of the first video conferencing platform through the sending module of the speaking terminal. This application mainly solves the problem of image transmission delay during video stream transmission. Therefore, the following mainly describes the transmission process of the video stream. It is understandable that the video conferencing platform is also used to forward audio data between different terminals. Since the amount of audio data is small, the transmission delay is smaller than the video code stream transmission delay. By reducing the video code stream transmission delay, the synchronization of audio and video can be improved. The audience terminal includes a receiving module, a decoder and a display module. The receiving module of the audience terminal receives the encoded code stream from the media network sending module of the second video conferencing platform and decodes it through the decoder of the audience terminal, and then displays it through the display module of the audience terminal.

[0068] When the media network receiving module sends the video code stream from the target sender to the target decoder, there are two processing modes: speed mode and quality mode. The speed mode means that the media network receiving module directly sends the video code stream to the target decoder after receiving it. However, it will be affected by the code stream fluctuation, and once the network loses packets, it needs to request the data packet again, and wait until the retransmission packet is sent before sending it to the target decoder. Therefore, the video code stream received at the target decoder is not sent at a uniform speed. The quality mode means that after the media network receiving module receives the video code stream, it first caches it, and then controls the sending speed to the target decoder, and sends it to the target decoder at a relatively stable speed to ensure that the image can be played smoothly.

[0069] During the transmission of video conferencing bitstreams, there is a problem of fluctuations in the amount of data in a single frame, which causes the transmitted bitstream to fluctuate and easily causes playback to be too fast or too slow. In order to ensure the quality of the video conferencing display, in the existing solution, the media network receiving module of the video conferencing platform adopts a quality mode for image transmission, that is, the media network receiving module first caches the bitstream after receiving it, and then controls the sending speed to the decoding module to ensure that the image can be played smoothly. When the bitstream fluctuation is serious, the media network receiving module needs to further smooth the process, and on the basis of the quality mode, further increase the amount of cached data, thereby further increasing the transmission delay.

[0070] In this embodiment, in order to solve the above problems, the media network receiving module adopts a speed mode, a quality mode is adopted at the target decoder end (that is, a cache smoothing optimization strategy is adopted at the decoding end), and a frame loss strategy is adopted at the target decoder end. This solution can avoid the impact of code stream fluctuations on video code stream transmission and improve image quality. At the same time, when the preset frame loss condition is met, the amount of data transmitted to the target encoder is reduced, the data transmission efficiency is improved, and the delay is reduced. Specifically, the target decoder adopts the quality mode, including: after the target decoder receives the video code stream sent by the target sending end, the video code stream is stored in the cache, and the video code stream in the cache is decoded to obtain multiple video frames. Further, the media network receiving module adopts a speed mode, specifically including: before step S100, based on the H320 transmission protocol, the media network receiving module receives the video code stream sent by the target sending end and sends it directly to the target decoder.

[0071] In this embodiment, the preset frame loss condition includes that the size of the video code stream in the cache is greater than the preset cache data volume threshold. If the media network receiving module transmits a large number of video code streams, the target decoder will store the video code stream in the cache faster, so that the amount of data in the cache is increasing. When the amount of video code stream data stored in the cache is too large, the target decoder will obtain more video frames after decoding. If it is still sent to the target encoder at a uniform speed, many video frames will not be sent in time, resulting in a large transmission delay. If the target decoder sends all the video frames obtained after decoding to the target encoder at a faster speed, it will cause the target receiving end to play too fast or too slow, affecting the image quality. Therefore, when too many video code streams are stored in the cache, the target decoder can quickly reduce the number of video frames obtained after decoding by discarding some video frames, thereby controlling the number of video frames sent to the target encoder within a certain reasonable range, and can send the video frames to the target encoder at a stable speed, thereby reducing network delay, avoiding the phenomenon of playing too fast or too slow, and improving image quality. The size of the preset cache data volume threshold can be determined according to the network quality. For example, when the network quality is good, a smaller threshold can be set to control the amount of video code stream data in the cache within a smaller range to achieve a smooth optimization effect. When the network quality is poor, a larger threshold can be set. The preset frame loss condition is not limited to being determined according to the amount of video code stream data in the cache. In another embodiment, the preset frame loss condition includes, for example, that the number of video frames decoded by the target decoder within a specific time range is greater than a preset decoding number threshold.

[0072] In step S300, the target decoder discards some video frames in a manner that the target decoder randomly takes out some video frames from the cache and discards them, or after the target decoder takes out the video frames from the cache, it discards 1 frame of video frames every x frames. For example, the target decoder discards 1 frame every 5 frames, or discards 1 frame every 10 frames. The value of x can be set in advance by the staff, or determined according to the network quality. For example, when the network quality is good, the value of x can be set to a larger value, and when the network quality is poor, the value of x can be set to a smaller value. Since the target decoder sends a raw code stream to the target encoder, the target encoder does not need other dependencies when encoding, and can directly encode, and discarding some video frames will not have a significant impact on the picture. The raw code stream (or original code stream) refers to the YUV data before any additional encapsulation, multiplexing or other processing.

[0073] In this embodiment, while the target decoder uses cache smoothing to improve image quality and increases data transmission speed by dropping frames, a dynamic link switching strategy executed by the control module is introduced, which can reduce code stream links in a specified scenario, further save transmission time, and improve image quality and code stream transmission speed. The dynamic link switching strategy includes the following steps for dynamic link management and switching before step S100:

[0074] Determine a target decoder corresponding to a target transmitting end and a target encoder corresponding to a target receiving end;

[0075] After receiving the code stream transmission instruction, the control module of the video conferencing platform determines the target decoder corresponding to the target sending end and the target encoder corresponding to the target receiving end, and can determine the video parameters of the target decoder and the video parameters of the target encoder; when the target sending end is connected to the video conferencing platform, the control module will create a target decoder according to the video parameters of the sending end, so the video parameters of the target decoder are the same as the video parameters of the terminal or platform serving as the target sending end, and when the target receiving end is connected to the video conferencing platform, the control module will create a target encoder according to the video parameters of the target receiving end, so the video parameters of the target encoder are the same as the video parameters of the terminal or platform serving as the target receiving end;

[0076] Compare the video parameters of the target decoder and the video parameters of the target encoder to see if they are consistent;

[0077] The video parameters include, for example, resolution, frame rate, bit rate, and other parameters; the video parameters of the target decoder correspond to the uplink (video transmission) capability of the target transmitter, and the video parameters of the target encoder correspond to the downlink (video reception) capability of the target receiver;

[0078] After determining the target decoder and the target encoder, the control module triggers the process of binding the target decoder and the target encoder. In this embodiment, before binding the target encoder and the target decoder, the control module first compares whether the video parameters of the two are consistent;

[0079] If the video parameters of the target decoder and the target encoder are consistent, it is determined to adopt the transparent transmission mode, establish a transparent transmission link from the media network receiving module to the media network sending module, the media network receiving module receives the video code stream sent by the target sending end and sends it to the media network sending module through the transparent transmission link, and the media network sending module sends the video code stream to the target receiving end; in the transparent transmission mode, the target decoder and the target encoder are not bound, and the media network receiving module does not send the video code stream sent by the target sending end to the target decoder after receiving it, and steps S100 to S400 are no longer executed, so the video code stream is directly transmitted through the transparent transmission link in the video conferencing platform, effectively reducing the time consumption in the transmission link and the time consumption of secondary decoding and encoding, thereby further reducing the image transmission delay; by adopting the transparent transmission link, in a specified scenario, the code stream link is reduced and the data transmission speed is improved, and the specified scenario here refers to the scenario where the video parameters of the target decoder corresponding to the target sending end and the video parameters of the target encoder corresponding to the target receiving end are consistent; when adopting the transparent transmission link, the media network receiving module adopts the quality mode to ensure the image quality, because the transmission link is greatly reduced and there is no need for secondary decoding and encoding on the platform side, the image transmission delay can also be reduced;

[0080] Figure 4 The example shows that a transparent transmission link from a media network receiving module to a media network sending module is established in a first video conferencing platform. The transparent transmission link is only for a target decoder and a target encoder with consistent video parameters. Since the video parameters of the terminal / platform connected to the media network sending module and the media network receiving module are the same, the encoded video code stream received by the media network receiving module from the terminal / platform as the target sending end can be directly decoded by another terminal / platform as the target receiving end, and there is no need for the intermediate target decoder and target encoder to perform secondary encoding. Since the secondary encoding process is omitted, the code stream link is reduced, the code stream transmission speed is greatly improved, which is conducive to reducing delay and improving audio and video synchronization.

[0081] If the video parameters of the target decoder and the target encoder are inconsistent, it is determined to use the secondary encoding and decoding mode, the target decoder is bound to the target encoder, the media network receiving module receives the video code stream sent by the target sending end and sends it to the target decoder, and then continues to perform the above steps S100 to S400. Among them, when the media network receiving module receives the video code stream sent by the target sending end, the media network receiving module does not use cache smoothing processing, but the media network receiving module directly sends the video code stream to the target decoder, that is, the media network receiving module uses the speed mode.

[0082] In this embodiment, after the transparent transmission link is established, if the video parameters of the target transmitting end or the target receiving end are subsequently changed so that the two are inconsistent, it is still necessary to use the secondary encoding process of the target decoder and the target encoder to make the target transmitting end and the target receiving end compatible with each other. Therefore, the dynamic link switching strategy also includes the step of switching from the transparent transmission mode back to the secondary encoding mode when the video parameters of the target decoder corresponding to the target transmitting end and the target encoder corresponding to the target receiving end are no longer consistent. Among them, after the transparent transmission link from the media network receiving module to the media network sending module is established, the following steps are also included:

[0083] Receiving a video parameter change notification from a target transmitting end and / or a target receiving end;

[0084] After the video parameters of the target decoder and / or target encoder are changed according to the video parameter change notification, the video parameters of the target decoder and the target encoder are compared. If they are inconsistent, the secondary encoding and decoding mode is switched to bind the target decoder corresponding to the target sending end with the target encoder corresponding to the target receiving end. After the media network receiving module receives the video code stream from the sending end, steps S100 to S400 are continued.

[0085] In the video conferencing platform of the prior art, when the speaking terminal switches, the code stream received by the audience terminal changes. For the video conferencing platform, its target sending terminal changes, and the target decoder corresponding to the target sending terminal also changes. If the alternation process between the code stream of the old speaking terminal and the code stream of the new speaking terminal lasts for a long time, the effect presented on the audience terminal is that the image freezes for a period of time. Especially in the multi-level cascade scenario, since the code stream link is very long, the alternation process between the old and new code streams lasts longer, the freeze phenomenon is aggravated, seriously affecting the user experience. Figure 5 The figure shows the code stream link switching process when switching the speaking terminal in the cascade conference in the prior art. In order to more clearly show the binding and unbinding process of the encoder and decoder, Figure 5 The media network sending module and the media network receiving module are omitted. In the cascade scenario, the upper-level video conferencing platform is the main control conferencing platform. If the lower-level terminal is operated to perform the service, the signaling needs to be transmitted from the upper-level video conferencing platform to the lower-level video conferencing platform. Figure 5As shown, the second video conferencing platform is a lower-level platform of the first video conferencing platform. In the first video conferencing platform, the control module pre-creates a first decoder corresponding to the speaking terminal A, a first encoder corresponding to the audience terminal C, and a third decoder corresponding to the second video conferencing platform. In the second video conferencing platform, the control module pre-creates a second encoder corresponding to the first video conferencing platform and a second decoder corresponding to the speaking terminal B. Taking the switch from speaking terminal A to speaking terminal B as an example, before changing the speaking terminal, the control module of the first video conferencing platform binds the first decoder to the first encoder, and the video code stream of the speaking terminal A is transmitted to the audience terminal C through the first video conferencing platform. When the first video conferencing platform receives the notification of switching the speaking terminal to the speaking terminal B, the first video conferencing platform first notifies the control module on the platform peripheral side to execute the switch binding, that is, execute Figure 5 In step (1), the binding between the first decoder and the first encoder is released, and the third decoder is bound to the first encoder. Then the first video conferencing platform switches the signaling of switching the speaking terminal to the second video conferencing platform, and the second video conferencing platform executes step (2) to bind the second decoder to the second encoder. Therefore, after step (1) is executed, the video stream of the speaking terminal B has not been transmitted to the third decoder, and the image of the viewer terminal will be stuck. If one or more levels of video conferencing platforms are added between the speaking terminal B and the viewer terminal C, the stuck time will be longer, which will have a great impact on the user experience.

[0086] In order to solve this technical problem, the present application further provides a smooth switching strategy for optimizing image stuttering in the speaking terminal switching scenario: before the new speaking terminal code stream is ready, the audience terminal is maintained to receive the old speaking terminal code stream, and the binding is switched only after the new speaking terminal code stream is ready. Figure 6 This is a schematic diagram of the optimized code stream link switching process when switching the speaking terminal in a cascade conference. Figure 6 The smooth switching strategy is introduced in detail.

[0087] In this embodiment, the video conference implementation method further includes the following steps:

[0088] The first video conferencing platform receives the code stream transmission signaling, determines the first speaking terminal and the audience terminal for code stream transmission, determines the first speaking terminal as the first target sending terminal and the audience terminal as the first target receiving terminal, and binds the first decoder corresponding to the first target sending terminal and the first encoder corresponding to the first target receiving terminal;

[0089] like Figure 6As shown, the first video conferencing platform receives the code stream transmission signaling, and determines according to the code stream transmission signaling that the first speaking terminal for code stream transmission is the speaking terminal A, and the audience terminal is the audience terminal C, then determines the speaking terminal A as the first target sending end, determines the audience terminal C as the first target receiving end, and binds the first decoder corresponding to the first target sending end and the first encoder corresponding to the first target receiving end;

[0090] The first video conferencing platform receives a first switching notification to switch the first speaking terminal to the second speaking terminal, and determines a second video conferencing platform to which the second speaking terminal is connected;

[0091] The first video conferencing platform sends a second switching notification that needs to be transmitted to the second speaking terminal code stream to the second video conferencing platform, and the second video conferencing platform is configured to bind the second decoder corresponding to the second speaking terminal as the second target transmitting end with the second encoder corresponding to the first video conferencing platform as the second target receiving end after receiving the second switching notification;

[0092] like Figure 6 As shown, the second video conference platform is a lower-level platform of the first video conference platform, the second speaking terminal is speaking terminal B, the first switching notification and the second switching notification correspond to switching the code stream transmission from speaking terminal A to audience terminal C to code stream transmission from speaking terminal B to audience terminal C; the second video conference platform pre-creates a second decoder corresponding to speaking terminal B and a second encoder corresponding to the first video conference platform, and the second video conference platform first executes Figure 6 In step (1), the second decoder is bound to the second encoder so that the video code stream of the speaking terminal B can be transmitted to the first video conferencing platform through the second decoder and the second encoder;

[0093] After the second decoder and the second encoder are bound, the first video conferencing platform can receive the video code stream of the second speaking terminal, and then in the first video conferencing platform, the third decoder corresponding to the second video conferencing platform as the third target sending end is bound to the first encoder, and the first decoder is unbound from the first encoder;

[0094] like Figure 6 As shown, the first video conferencing platform pre-creates a third decoder corresponding to the second video conferencing platform. After the second decoder and the second encoder are bound, the third decoder can receive the video code stream of the speaking terminal B sent by the second video conferencing platform. At this time, the first video conferencing platform executes Figure 6In step (2), the binding is switched, that is, the third decoder corresponding to the third target transmitter is bound to the first encoder, and the binding of the first decoder to the first encoder is released. Thus, the video code stream of the speaking terminal B can be output to the audience terminal C through the second decoder, the second encoder, the third decoder, and the first encoder. Therefore, after receiving the first switching notification, the first video conferencing platform does not switch the binding immediately, but first notifies the subordinate platform to bind. After the subordinate platform can normally input the code stream of the speaking terminal after the switch, the first video conferencing platform switches the binding. For the audience terminal C, it has been receiving the video code stream of the speaking terminal A without interruption before the switch. When switching, it immediately switches to receiving the video code stream of the speaking terminal B. There is no freeze in the middle, which ensures smooth image switching and realizes smooth switching of the speaking terminal. That is, it solves the problem of image freeze when switching the speaking terminal, and improves the conference quality.

[0095] Combination of the above Figure 6 Taking the switching of two-level video conferencing platforms as an example, when switching speaking terminals involving three-level or more-level video conferencing platforms, for a video conferencing platform, if it receives a switching notification and finds that a new speaking terminal is connected to its lower-level video conferencing platform, it is necessary to first notify the lower-level video conferencing platform to bind the corresponding decoder and encoder, and then execute the binding of the corresponding decoder and encoder in its own platform, so that the bit stream can be input normally after the switching of the lower level, and then switch its own binding relationship, to ensure that the bit stream transmission from the original speaking terminal to the audience terminal can be maintained without interruption before the switch.

[0096] In this embodiment, after the first video conferencing platform sends the second switching notification that needs to be transmitted to the second speaking terminal code stream to the second video conferencing platform, the following steps are also included:

[0097] In the first video conferencing platform, determining whether the third decoder receives a key frame of a bit stream from the second video conferencing platform;

[0098] If the third decoder receives the code stream key frame from the first video conference platform, it means that the third decoder can start decoding the video code stream of the second speaking terminal, which means that the video code stream of the second speaking terminal can be pushed to the audience terminal through the first encoder. Then the first video conference platform binds the third decoder to the first encoder and unbinds the first decoder from the first encoder. That is, for the first video conference platform, the evaluation criterion for whether the code stream of the new speaking terminal is ready is whether the third decoder receives the code stream key frame from the second video conference platform.

[0099] An embodiment of the present application also provides a video conferencing implementation device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the video conferencing implementation method by executing the executable instructions.

[0100] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a platform, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "platform" herein.

[0101] Refer to the following Figure 7 The electronic device 600 according to this embodiment of the present application is described. Figure 7 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0102] like Figure 7 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0103] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present application described in the above-mentioned video conferencing implementation method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0104] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0105] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating platform, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0106] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0107] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID platforms, tape drives, and data backup storage platforms, etc.

[0108] By adopting the video conferencing implementation device provided in the present application, the processor executes the video conferencing implementation method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned video conferencing implementation method.

[0109] The embodiment of the present application also provides a computer-readable storage medium for storing a program, which implements the steps of the video conferencing implementation method when executed by a processor. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present application described in the above-mentioned video conferencing implementation method section of this specification.

[0110] refer to Figure 8 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution platform, an apparatus or a device.

[0111] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor platform, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0112] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution platform, device, or device or for use in combination with the program. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0113] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0114] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the video conferencing implementation method when executed, thereby obtaining the beneficial effects of the above-mentioned video conferencing implementation method.

[0115] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. A method for implementing a video conference, characterized in that: The method is implemented by using a video conferencing platform, wherein the video conferencing platform includes a decoder corresponding to each transmitting end and an encoder corresponding to each receiving end. The method includes the following steps: The target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, and decodes it to obtain multiple video frames; The target decoder determines whether a preset frame loss condition is currently met. If yes, the target decoder discards some video frames and sends the undiscarded video frames to the target encoder corresponding to the target receiving end. If no, the target decoder sends the video frames to the target encoder. The target encoder encodes the received video frame to obtain an encoded code stream and sends it to the target receiving end.

2. The method for implementing a video conference according to claim 1, characterized in that: After receiving the video code stream sent by the target sending end, the target decoder corresponding to the target sending end stores the video code stream in a cache, and decodes the video code stream in the cache to obtain multiple video frames; The preset frame loss condition includes that the size of the video code stream in the cache is greater than a preset cache data amount threshold.

3. The video conferencing implementation method according to claim 1, characterized in that: The video conferencing platform also includes a media network receiving module. Before the target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, it also includes: based on the H320 transmission protocol, the media network receiving module receives the video code stream sent by the target sending end and directly sends it to the target decoder.

4. The method for implementing a video conference according to claim 1, characterized in that: The video conferencing platform further includes a media network sending module and a media network receiving module. Before the target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, the following steps are also included: Determine the target decoder corresponding to the target transmitting end and the target encoder corresponding to the target receiving end; Comparing whether the video parameters of the target decoder and the video parameters of the target encoder are consistent; If they are consistent, it is determined that the transparent transmission mode is adopted, and a transparent transmission link from the media network receiving module to the media network sending module is established. The media network receiving module receives the video code stream sent by the target sending end and sends it to the media network sending module through the transparent transmission link. The media network sending module sends the video code stream to the target receiving end; If they are inconsistent, it is determined to adopt the secondary encoding and decoding mode, the target decoder is bound to the target encoder, and the media network receiving module receives the video code stream sent by the target sending end and sends it to the target decoder.

5. The method for implementing a video conference according to claim 4, characterized in that: After the transparent transmission link from the media network receiving module to the media network sending module is established, the following steps are also included: Receiving a video parameter change notification from the target sending end and / or the target receiving end; After changing the video parameters of the target decoder and / or the target encoder, the video parameters of the target decoder and the target encoder are compared, and if they are inconsistent, switching to the secondary encoding and decoding mode is performed.

6. The method for implementing a video conference according to claim 1, characterized in that: The following steps are also included: The first video conferencing platform receives the code stream transmission signaling, determines the first speaking terminal and the audience terminal for code stream transmission, determines the first speaking terminal as the first target sending terminal and the audience terminal as the first target receiving terminal, and binds the first decoder corresponding to the first target sending terminal and the first encoder corresponding to the first target receiving terminal; The first video conferencing platform receives a first switching notification to switch the first speaking terminal to the second speaking terminal, and determines a second video conferencing platform to which the second speaking terminal is connected; The first video conferencing platform sends a second switching notification that needs to be transmitted to the second speaking terminal code stream to the second video conferencing platform, and the second video conferencing platform is configured to bind the second decoder corresponding to the second speaking terminal as the second target sending end with the second encoder corresponding to the first video conferencing platform as the second target receiving end after receiving the second switching notification; In the first video conferencing platform, the third decoder corresponding to the third target sending end of the second video conferencing platform is bound to the first encoder, and the first decoder is unbound from the first encoder.

7. The method for implementing a video conference according to claim 6, characterized in that: After the first video conferencing platform sends the second switching notification that needs to be transmitted to the second speaking terminal code stream to the second video conferencing platform, the following steps are also included: In the first video conferencing platform, determining whether the third decoder receives a code stream key frame from the second video conferencing platform; If so, the first video conferencing platform binds the third decoder to the first encoder, and unbinds the first decoder from the first encoder.

8. A video conferencing platform, characterized in that: The video conferencing implementation method applied to any one of claims 1 to 7, wherein the platform includes a decoder corresponding to each transmitting end and an encoder corresponding to each receiving end; the decoder and the encoder are configured to perform the following steps: The target decoder corresponding to the target sending end receives the video code stream sent by the target sending end, and decodes it to obtain multiple video frames; The target decoder determines whether a preset frame loss condition is currently met. If yes, the target decoder discards some video frames and sends the undiscarded video frames to the encoder corresponding to the target receiving end. If no, the target decoder sends the video frames to the target encoder corresponding to the target receiving end. The target encoder encodes the received video frame to obtain an encoded code stream and sends it to the target receiving end.

9. A video conferencing implementation device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the video conferencing implementation method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the video conferencing implementation method according to any one of claims 1 to 7 are implemented.

Images