Encoding method and apparatus, decoding method and apparatus, device, storage medium, and program product

Abstract

The present application relates to the technical field of image compression, and discloses an encoding method and apparatus, a decoding method and apparatus, a device, a storage medium, and a program product. The encoding method comprises: respectively performing region division on a plurality of images to be encoded, to obtain a plurality of image regions, wherein said plurality of images correspond to different bitstreams, and at least two images to be encoded among said plurality of images comprise the same image region (901); selecting, from among the plurality of image regions, image regions different from each other (902); acquiring a plurality of sub-image bitstreams obtained by respectively encoding the selected image regions (903); and on the basis of the plurality of sub-image bitstreams, performing bitstream concatenation according to the image regions respectively comprised in said plurality of images, to obtain global image bitstreams respectively corresponding to said plurality of images (904). In the present application, when encoding a plurality of images to be encoded, only image regions that are different from each other need to be encoded, that is, the same image region in said plurality of images only needs to be encoded once, thereby reducing the consumption of encoding resources and encoding computational power.

Description

Coding and decoding method, device, equipment, storage medium and program product

[0001] The present application claims priority to the Chinese patent application No. 202410905291.2, filed on July 5, 2024, and entitled "Coding and decoding method, device, equipment, storage medium and program product", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of image compression technology, and in particular, to a coding and decoding method, device, equipment, storage medium and program product. BACKGROUND

[0003] A video conference system mainly includes a video conference terminal, a video conference server, a transmission network and related accessories. Among them, the video conference server includes a multi-point control unit (MCU) and a network management device, the MCU as the core device of the video conference system, is responsible for processing and managing video, audio and data streams from various video conference terminals. In the scene of multi-party video conference, the MCU is used to decode the respective participant picture code streams of each participant, and send the combined conference room picture code streams of other participants to each participant's video conference terminal, so that each participant's terminal can present the corresponding conference room picture.

[0004] In the related art, referring to FIG. 1, taking a four-party video conference between a participant A, a participant B, a participant C and a participant D as an example, assuming that each participant watches a conference room picture with a 2*2 interface layout, the process of the MCU transmitting code streams in the four-party video conference is as follows: each participant's video conference terminal uploads the participant's own participant picture code stream to the MCU, the MCU decodes the participant picture code stream uploaded by each participant to obtain the participant picture of each participant, and based on the participant pictures of the four participants, combines a corresponding conference room picture for each participant, i.e., the A conference room picture of the participant A, the B conference room picture of the participant B, the C conference room picture of the participant C and the D conference room picture of the participant D, as shown in FIG. 1, each participant's conference room picture includes the participant pictures of the other three participants in the four-party video conference. Then, the MCU encodes each participant's conference room picture, and sends the encoded conference room picture code stream to the video conference terminal of the corresponding participant, so that each participant's video conference terminal can decode and display the corresponding conference room picture.

[0005] However, in the above method, the MCU needs to encode the conference scene picture code stream corresponding to each participant, and when the conference scene pictures watched by each participant are different, a large number of conference scene picture code streams need to be encoded, resulting in large encoding computing power and resource consumption of the MCU; and as the number of participants increases, the encoding computing power and resources consumed by the MCU in encoding the conference scene picture code streams of multiple participants also increase. SUMMARY

[0006] The present application provides a coding and decoding method, device, equipment, storage medium and program product, which can reduce the consumption of encoding resources and encoding computing power when encoding multiple to-be-encoded images. The technical solution is as follows:

[0007] In a first aspect, an encoding method is provided, which includes:

[0008] The multiple to-be-encoded images are respectively divided into regions to obtain multiple image regions; wherein the multiple to-be-encoded images correspond to different code streams, and at least two to-be-encoded images in the multiple to-be-encoded images include the same image region; mutually different image regions are selected from the multiple image regions; multiple sub-image code streams obtained after the selected image regions are encoded are obtained; and based on the multiple sub-image code streams, code stream splicing is performed according to the image regions included in the multiple to-be-encoded images to obtain global image code streams corresponding to the multiple to-be-encoded images.

[0009] Wherein, after the region division of the multiple to-be-encoded images, each to-be-encoded image is divided into at least one image region, at this time, at least two to-be-encoded images in the multiple to-be-encoded images include the same image region, that is, there may be one same image region or multiple same image regions in the image regions corresponding to the at least two to-be-encoded images, which is not limited by the present application.

[0010] Wherein, each to-be-encoded image corresponds to a code stream, that is, when generating global image code streams corresponding to the multiple to-be-encoded images, the code streams corresponding to each to-be-encoded image are different.

[0011] In a possible implementation, multiplexing analysis is performed on the multiple image regions to determine which image regions are different and which image regions are the same, so as to select mutually different image regions from the multiple image regions.

[0012] Therefore, when encoding the plurality of to-be-encoded images, the plurality of to-be-encoded images are first divided into a plurality of image regions respectively; then the multiplexability of each image region is analyzed to filter out image regions that are different from each other from the plurality of image regions. When encoding, only the image regions that are filtered out and different from each other (i.e., not repeated, differentiated) need to be encoded, and the same image regions in the plurality of to-be-encoded images only need to be encoded once. In this way, compared with the manner of directly encoding the entire to-be-encoded image in the related art, the consumption of encoding resources and encoding computing power in the encoding process is reduced.

[0013] In other words, after the image regions are divided for the plurality of to-be-encoded images, if there are the same image regions in the plurality of to-be-encoded images, the technical solution of the present application can effectively avoid repeated encoding of the same image regions and reduce the consumption of encoding resources and encoding computing power. Moreover, as the number of to-be-encoded images or the number of the same image regions in the to-be-encoded images increases, the technical solution provided by the present application can greatly reduce the consumption of encoding resources and encoding computing power, and the benefits achieved by the overall solution will be better and better.

[0014] In a possible implementation, the obtaining the plurality of sub-image code streams obtained after the filtered image regions are encoded respectively includes: encoding the filtered image regions respectively to obtain the plurality of sub-image code streams.

[0015] The filtered image regions and the sub-image code streams are one-to-one corresponding, and the number of the filtered image regions is the same as that of the sub-image code streams. That is, each filtered image region is independently encoded to obtain a sub-image code stream corresponding to the image region.

[0016] That is, for the video distribution device or the encoding device, after the image regions that are different from each other are filtered out, the video distribution device or the encoding device can independently complete image region encoding.

[0017] In a possible implementation, the encoding the filtered image regions respectively to obtain the plurality of sub-image code streams includes: determining boundary restriction information of the filtered image regions; the boundary restriction information includes time domain motion vector (MV) restriction information and space domain MV restriction information; and encoding the filtered image regions respectively based on the boundary restriction information to obtain the plurality of sub-image code streams.

[0018] It is considered that although the tiles in H.265 can be independently encoded, they cannot be completely independent in decoding. The tiles cross the tile boundary in the inter prediction, loop filtering and other processing processes, and use the encoded information of any tile in the encoded image. In order to ensure that the tiles can be independently encoded and decoded, H.265 can protect the boundaries of the tiles when encoding and decoding the tiles.

[0019] Therefore, when encoding the selected image regions, the boundary restriction information is set to ensure the independence of the image regions in the encoding and decoding processes, so as to fully utilize the computing power of the processor and improve the efficiency and performance of video encoding.

[0020] In a possible implementation, before the selected image regions are encoded based on the boundary restriction information, the method further includes: adjusting a quantization parameter of an encoded block at a boundary of the selected image regions to reduce a quantization step of the encoded block at the boundary of the selected image regions.

[0021] Since a smaller QP value corresponds to finer quantization and can retain more image details, the QP value at the boundary can be reduced to reduce the quantization step, so as to retain the detail information at the boundary of the image regions and reduce distortion, effectively prevent the quality of the image regions from being reduced and the flicker phenomenon from occurring, and thus improve the encoding and decoding quality of the overall image.

[0022] In a possible implementation, the multiple sub-image code streams obtained after the selected image regions are encoded include: sending an encoding request to a target encoding device, the encoding request being used to request the target encoding device to encode the selected image regions; and receiving the multiple sub-image code streams sent by the target encoding device.

[0023] That is, the selected image regions can also be encoded by the target encoding device to obtain the multiple sub-image code streams, and the multiple sub-image code streams are sent to the video distribution device.

[0024] In a possible implementation, the multiple image regions obtained by performing region division on the multiple to-be-encoded images include: performing region division on the multiple to-be-encoded images according to a division manner of an image slice, a tile or a patch to obtain the multiple image regions.

[0025] Since the Slice, Tile or Patch can be independently coded, after the to-be-coded image is divided into a plurality of image regions in the above manner, the image regions can be independently coded.

[0026] In a possible implementation, the code stream splicing according to the image regions included in the plurality of to-be-coded images based on the plurality of sub-image code streams to obtain the global image code streams corresponding to the plurality of to-be-coded images respectively includes: splicing the sub-image code streams of the image regions included in each to-be-coded image in the plurality of to-be-coded images according to the positions of the image regions included in each to-be-coded image to obtain the global image code streams corresponding to the plurality of to-be-coded images respectively.

[0027] In a possible implementation, the global image code stream carries position information of each image region included in the corresponding to-be-coded image.

[0028] The position information of each image region can include a starting point and an offset, and the specific position of the image region in the corresponding to-be-coded image can be determined based on the two parameters.

[0029] In a possible implementation, the to-be-coded image is any frame of conference scene picture viewed by a target participant in a multi-party video conference, and the target participant is any participant in the multi-party video conference; the method further includes: receiving a participant picture code stream of each participant in the multi-party video conference; parsing the participant picture code stream of each participant to obtain a participant picture of each participant; and splicing the conference scene picture of each participant based on the participant picture of each participant, wherein the conference scene picture of each participant includes participant pictures of other participants in the multi-party video conference.

[0030] That is, based on the participant picture of each participant, a plurality of participant pictures are spliced according to the conference scene picture requirement of each participant to generate the conference scene picture of each participant. Based on the conference scene picture of each participant, the technical solution of the present application is executed to code a plurality of conference scene pictures.

[0031] The conference scene picture of each participant can further include the participant picture of the participant himself / herself and the participant pictures of other participants in the multi-party video conference. In this case, each participant can not only clearly understand the participant state of other participants through the displayed conference scene picture, but also intuitively understand the participant state of himself / herself.

[0032] In a possible implementation, after the global image code streams corresponding to the plurality of to-be-coded images are obtained, the method further includes: sending the global image code stream of the conference scene picture corresponding to each participant to the participant.

[0033] Thus, after receiving the global image code stream, each participant's video conference terminal can decode the global image code stream by using the decoding method provided in the present application, and obtain the reconstructed conference scene picture by splicing the plurality of reconstructed image regions.

[0034] In a second aspect, a decoding method is provided, and the method comprises:

[0035] receiving a global image code stream, wherein the global image code stream comprises a plurality of image region corresponding sub-image code streams; parsing the plurality of image region corresponding sub-image code streams to obtain a plurality of reconstructed image regions; and splicing the plurality of reconstructed image regions to obtain a reconstructed image.

[0036] In the method, the global image code stream received by the client is not a code stream directly encoded from an entire image, but a global image code stream obtained by splicing a plurality of image region corresponding sub-image code streams in the image.

[0037] In the method, the plurality of image regions are determined according to a division manner of image slices, image tiles or image patches.

[0038] In a possible implementation, the global image code stream carries position information of the plurality of image regions; and the splicing the plurality of reconstructed image regions to obtain the reconstructed image comprises: splicing the plurality of reconstructed image regions based on the position information to obtain the reconstructed image.

[0039] Since the plurality of to-be-encoded images can reuse the already encoded image regions when being encoded, the efficiency of encoding images is higher, so that the corresponding global image code stream can be received by the client device more quickly and smoothly. Moreover, when receiving the global image code stream comprising a plurality of image region corresponding sub-image code streams, the client device can parse the plurality of sub-image code streams in parallel to obtain the plurality of reconstructed image regions, thereby improving the decoding efficiency.

[0040] In a third aspect, an encoding apparatus is provided, which has a function of implementing the behavior of the encoding method in the first aspect. The encoding apparatus comprises at least one module for implementing the encoding method provided in the first aspect.

[0041] In a fourth aspect, a decoding apparatus is provided, which has a function of implementing the behavior of the decoding method in the second aspect. The decoding apparatus comprises at least one module for implementing the decoding method provided in the second aspect.

[0042] In a fifth aspect, an encoding device is provided, which includes a processor and a memory. The memory is configured to store a computer program for implementing the encoding method of the first aspect. The processor is configured to execute the computer program stored in the memory to implement the encoding method of the first aspect.

[0043] In a possible implementation, the encoding device can further include a communication bus for establishing a connection between the processor and the memory.

[0044] In a sixth aspect, a decoding device is provided, which includes a processor and a memory. The memory is configured to store a computer program for implementing the decoding method of the second aspect. The processor is configured to execute the computer program stored in the memory to implement the decoding method of the second aspect.

[0045] In a possible implementation, the decoding device can further include a communication bus for establishing a connection between the processor and the memory.

[0046] In a seventh aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed on a computer or a processor, the computer or the processor is caused to perform the steps of the encoding method of the first aspect or the steps of the decoding method of the second aspect.

[0047] In an eighth aspect, a computer program product is provided, which includes computer instructions. When the computer instructions are executed on a computer or a processor, the computer or the processor is caused to perform the steps of the encoding method of the first aspect or the steps of the decoding method of the second aspect. Alternatively, a computer program is provided. When the computer program is executed on a computer or a processor, the computer or the processor is caused to perform the steps of the encoding method of the first aspect or the steps of the decoding method of the second aspect.

[0048] In a ninth aspect, a codec system is provided, which includes an encoding device and a decoding device. The encoding device is configured to implement the steps of the encoding method of the first aspect. The decoding device is configured to implement the steps of the decoding method of the second aspect.

[0049] In a tenth aspect, an encoded bitstream is provided, which is generated according to the encoding method of the first aspect.

[0050] In an eleventh aspect, a computer-readable storage medium is provided, which stores a bitstream generated by the encoding method according to the first aspect.

[0051] In a twelfth aspect, an apparatus for storing a bitstream is provided, which comprises a receiver configured to receive the bitstream generated by the encoding method according to the first aspect, and at least one storage medium configured to store the bitstream.

[0052] In a thirteenth aspect, an apparatus for transmitting a bitstream is provided, which comprises a receiver configured to receive the bitstream generated by the encoding method according to the first aspect, and a transmitter configured to transmit the bitstream to an end-side device via a transmission medium.

[0053] In a fourteenth aspect, an apparatus for transmitting a bitstream is provided, which comprises a transmitter and at least one storage medium configured to store the bitstream generated by the encoding method according to the first aspect, the transmitter being configured to acquire the bitstream from the storage medium and transmit the bitstream to an end-side device via a transmission medium.

[0054] In a fifteenth aspect, a system for distributing a bitstream is provided, which comprises at least one storage medium configured to store the bitstream generated by the encoding method according to the first aspect, and a streaming device configured to acquire a target bitstream from the at least one storage medium and transmit the target bitstream to an end-side device, wherein the streaming device comprises a content server or a content distribution server.

[0055] The technical effects obtained by the third aspect to the fifteenth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect and the second aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0056] FIG. 1 is a schematic diagram of bitstream transmission of a four-party video conference in a related technology according to an embodiment of the present application;

[0057] FIG. 2 is a schematic diagram of division of an SS according to an embodiment of the present application;

[0058] FIG. 3 is a schematic diagram of division of a Tile according to an embodiment of the present application;

[0059] FIG. 4 is a schematic diagram of bitstream transmission of another four-party video conference in a related technology according to an embodiment of the present application;

[0060] FIG. 5 is a schematic diagram of an implementation environment according to an embodiment of the present application;

[0061] FIG. 6 is a schematic diagram of another implementation environment according to an embodiment of the present application;

[0062] FIG. 7 is a schematic diagram of another implementation environment according to an embodiment of the present application;

[0063] FIG. 8 is a schematic diagram of a structure of a client device according to an embodiment of the present application;

[0064] FIG. 9 is a schematic diagram of a flow of an encoding method according to an embodiment of the present application;

[0065] FIG. 10 is a schematic diagram of filtering image regions through multiplexing analysis according to an embodiment of the present application;

[0066] FIG. 11 is a schematic diagram of a flow of a decoding method according to an embodiment of the present application;

[0067] FIG. 12 is a schematic diagram of an encoding and decoding flow of a five-party video conference according to an embodiment of the present application;

[0068] FIG. 13 is a schematic diagram of a flow of generating a global image code stream according to an embodiment of the present application;

[0069] FIG. 14 is a schematic diagram of a structure of an encoding apparatus according to an embodiment of the present application;

[0070] FIG. 15 is a schematic diagram of a structure of a decoding apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0072] To facilitate understanding, before the encoding and decoding method according to the embodiments of the present application is explained and described, the terms, application background and implementation environment involved in the embodiments of the present application are introduced.

[0073] First, the terms involved in the embodiments of the present application are introduced.

[0074] 1. Multi-point control unit (MCU)

[0075] MCU, also known as multi-point controller, is a key device of a multi-point video conference system. MCU is essentially a multi-media information switch, which performs multi-point call and connection, and realizes functions such as video broadcast, video selection, audio mixing, data broadcast, etc., to complete the switching and connection of signals of various video conference terminals.

[0076] The MCU differs from the switch in that the switch completes point-to-point connection of signals, while the MCU completes multipoint-to-multipoint switching, convergence or broadcasting. In a video conference system, the MCU mainly processes three kinds of data: video stream, audio stream and data stream, for decoding the participant picture code stream uploaded by each participant, and obtaining the conference hall picture of each participant by combining the participant pictures of multiple participants, and then encoding the conference hall picture of each participant and sending it to the video conference terminal of the corresponding participant.

[0077] 2. H.26x coding

[0078] H.26x coding is a commonly used video coding format and a block-oriented, motion-compensated video coding standard. For example, advanced video coding (AVC, also known as H.264) and high efficiency video coding (HEVC, also known as H.265).

[0079] 3. Coding tree unit (CTU)

[0080] In H.265, a frame of image is divided into multiple rectangular blocks, the sizes of which can be 16*16, 32*32 and 64*64, etc. These rectangular blocks and corresponding coding information are referred to as CTU. The CTU in H.265 is similar to the macroblock in H.264, but its size can be specified by the encoder and can be larger than the traditional macroblock.

[0081] 4. Slice (also referred to as image slice, coding slice, segment, strip, piece, etc.)

[0082] The slice in H.264 and H.265 is composed of an integer number of CTUs, and can be independently coded. This is achieved by terminating the context-based adaptive binary arithmetic coding (CABAC, an entropy coding method) code stream at the end of each slice, and removing the CTU dependency between different slices. However, multiple slice headers (used to describe the coding parameters and context model of the corresponding slice), and the removal of the CTU dependency between different slices, can cause the code rate to rise.

[0083] During encoding and decoding, different slices cannot reference each other, i.e., the encoding and decoding process within one slice does not rely on the information of other slices. In other words, when a slice is being encoded or decoded, it does not take into account any information in other slices, especially when the edge of one slice is adjacent to another slice, the pixels between these adjacent slices are not considered "available", i.e., the pixels across the slice boundary are inaccessible in the current slice encoding or decoding process.

[0084] Among them, the main purposes of dividing slices are as follows:

[0085] (1) Error robustness

[0086] By dividing the image into smaller slices, error robustness can be obtained by resynchronizing the decoding and parsing process in the case of data loss.

[0087] (2) Matching the maximum transmission unit (MTU)

[0088] The size of the slice can match the MTU size in the internet protocol (IP) network, so only the maximum number of bits contained in the slice needs to be controlled, without considering the size of the encoded image, and the network is more adaptable.

[0089] (3) Parallel processing

[0090] Since the slice can be independently encoded and decoded, the encoding and decoding of multiple slices can be performed in parallel to improve the overall encoding and decoding efficiency of the image.

[0091] Further, in order to process the encoding and decoding process more finely, the slice can be further divided into multiple slice segments (SS). In other words, the SS is a subdivided part of the slice, which can be part or all of the content in the slice. In the encoding process, the slice is divided into one or more SS, and each SS can be independently encoded and transmitted. This has the advantage of improving the flexibility of encoding, and different encoding strategies can be used for different SS according to specific scenarios and needs to achieve better encoding results.

[0092] In H.264, each Slice can have one or more SSs, and these SSs can be organized in different ways. Among them, the first SS of a Slice is called an independent SS, and its encoding does not depend on other SSs and can be decoded independently; while the subsequent SSs are called dependent SSs, and their encoding depends on the previous SSs and cannot be decoded independently.

[0093] As an example, see FIG. 2, which gives an exemplary SS division diagram. Among them, each Slice includes an independent SS and several dependent SSs, each Slice starts with an independent SS, and the dependent SSs after it only need very small amount of Slice header, and the SSs in the same Slice can refer to each other.

[0094] 5、Tile(Also known as tile, image block, coding block, block, etc.)

[0095] In H.265, Tile is a data unit obtained by dividing the image in space. As shown in FIG. 3, an image can be divided into multiple Tiles, i.e., Tile 1-Tile 9. Each Tile is an independent rectangular region, which can be independently coded and decoded without reference to each other. That is, the introduction of Tile improves the concurrency in video processing, which is beneficial to fully utilize the computing power of modern multi-core processors.

[0096] Among them, H.264 does not have the concept of Tile, and the Tile in H.265 is similar to Slice. When coding and decoding, different Tiles cannot refer to each other, that is, the coding and decoding process within a Tile will not depend on the information of other Tiles. In other words, when a Tile is being coded or decoded, it will not consider any information in other Tiles, especially when the edge of a Tile is adjacent to another Tile, the pixels between these adjacent Tiles will not be considered as "available", that is, the pixels across the Tile boundary are not accessible in the current Tile coding or decoding process.

[0097] It should be noted that when encoding, Tile modifies the scanning order of CTU, from image-based raster scanning to Tile-based raster scanning. Moreover, there can be multiple Tiles in a Slice, and multiple Slices in a Tile. However, the following two conditions must be met:

[0098] (1) All CTUs in a Slice belong to the same Tile, or all CTUs in a Tile belong to the same Slice;

[0099] (2) All CTUs in an SS belong to the same Tile, or all CTUs in a Tile belong to the same SS.

[0100] In addition, the divided Tile must be an integer multiple of the CTU except that the last column and the last row are not an integer multiple of the maximum CTU (64*64).

[0101] Secondly, the related background of the embodiments of the present application is introduced.

[0102] The video conference can be divided into an audio visual control system (AVC) based conference and a scalable video coding (SVC) based conference. Among them, the MCU in the AVC conference is mainly used to decode the video code stream uploaded by the conference participant, combine to generate the conference scene picture of each conference participant, and then re-encode the conference scene picture and send it to the video conference terminal of the corresponding conference participant; the MCU in the SVC conference is mainly used for forwarding the video code stream, but also for adapting multiple clients, and the MCU also needs to encode and decode in this process.

[0103] At present, the stock of MCU in the market is very large, and the number of new MCUs added every year is also very large. Each MCU supports dozens to hundreds of 1080P@30FPS or 4K@30FPS video encoding capabilities. If the number of encoding of each MCU is increased, a large amount of cost can be saved and the product competitiveness can be improved.

[0104] Among them, 1080P@30FPS refers to the resolution of the encoded video being 1920*1080 pixels, which is one of the standards of high-definition video, and the frame rate of the video is 30 frames per second, that is, the video updates 30 pictures per second. In 4K@30FPS, 4K is the standard of ultra-high-definition video, and its resolution is usually 3840*2160 pixels, which is four times that of 1080P. Similarly, 30FPS represents the frame rate of the video as 30 frames per second.

[0105] Next, the transmission process of the video code stream in the above two video conferences is introduced.

[0106] Please continue to refer to FIG. 1. The MCU in the AVC conference usually needs to encode a video code stream of a conference scene picture for each video conference terminal. When the conference scene pictures watched by each conference participant are different, a large number of video code streams need to be encoded. Taking a four-party video conference as an example, assuming that each conference participant watches a conference scene picture with a 2*2 interface layout, the process of the MCU transmitting the video code stream in the four-party video conference includes the following steps (1)-(4).

[0107] (1) Each participant's video conference terminal uploads the participant's own 4K@30FPS video stream to the MCU;

[0108] (2) The MCU decodes each participant's video stream to obtain each participant's participant picture.

[0109] (3) The MCU generates a corresponding conference picture for each participant based on each participant's participant picture by splicing the participant pictures.

[0110] (4) The MCU encodes each conference picture at 4K@30FPS and sends the encoded video stream to the corresponding participant's video conference terminal.

[0111] Therefore, the MCU needs a total of 4*4K@30FPS decoding capability and 4*4K@30FPS encoding capability. Since the computing power consumed by the decoding process is generally a few tenths of the encoding process, the decoding computing power consumption can be ignored, but the video encoding consumes a lot of computing power, and as the number of participants increases, the MCU consumes more encoding computing power and resources when encoding the conference picture streams of multiple participants. Moreover, since the above encoding process depends on the encoding capability of the hardware platform, whether it is a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated encoding hardware, the encoding specifications are limited by the performance of the hardware itself, for example, some hardware only supports 50-way 4K@30FPS encoding capability, which is only suitable for multi-party video conferences with less than 50 participants, and some CPUs can only encode a few ways of 4K@30FPS video stream, which is suitable for smaller multi-party video conferences.

[0112] In an SVC conference, the forwarding of video streams mainly depends on the routing and forwarding functions of the selective forwarding unit (SFU). When a cloud video conference adopts an SVC architecture, it can generate multiple rate video streams from one encoding, thereby adapting to the bandwidth requirements of different terminals in the system.

[0113] The SFU video stream forwarding process includes the following steps (1)-(4).

[0114] (1) The SFU receives the video stream of each participant's participant picture uploaded by each participant's video conference terminal.

[0115] These video streams are compressed data obtained by SVC encoding technology, containing video information of different quality levels and resolutions.

[0116] (2) SFU analyzes each video code stream received, determines its encoding structure, resolution, quality level and other information, so as to determine how to forward the video code stream, and ensure that each participant can display the corresponding conference picture.

[0117] (3) SFU forwards the video code stream to the participants who need to receive according to the analysis result. In the forwarding process, SFU will ensure that each participant only receives the video code stream matched with the terminal capability and network condition of the participant. Specifically, if the network condition of a participant is good and the terminal capability of the participant is strong, SFU will forward the video code stream of high quality level and high resolution to the participant; on the contrary, if the network condition of a participant is poor and the terminal capability of the participant is weak, SFU will forward the video code stream of low quality level and low resolution to the participant.

[0118] (4) After receiving the video code stream, the video conference terminal of each participant will decode and display the picture by itself. Since SFU has forwarded the video code stream of different quality level and resolution according to the actual situation of each participant, each participant can display the corresponding conference picture, and the quality and fluency of the picture are guaranteed.

[0119] As an example, as shown in FIG. 4, also taking a four-party video conference as an example, the process of SFU transmitting video code stream in the four-party video conference is as follows: the video conference terminal of each participant uploads the video code stream of the participant himself to SFU, SFU decodes the video code stream of each participant to obtain the participant picture of each participant, and then sends the corresponding participant picture to the video conference terminal of each participant, and each participant's video terminal receives the corresponding multiple participant pictures, combines the pictures by itself, and displays the corresponding conference picture.

[0120] Taking participant A as an example, it sends its own participant picture to SFU, and receives the participant picture of participant B, the participant picture of participant C and the participant picture of participant D sent by SFU, then combines the participant pictures of participant B, participant C and participant D to obtain the corresponding A conference picture, and displays the A conference picture.

[0121] As can be seen, in the above SVC conference, SFU does not do any encoding and decoding, and adopts a multi-stream based time domain SVC video encoding mode, so that the bandwidth occupancy of SFU in transmitting code stream in the multi-party video conference is more than 2 times of that in the AVC conference.

[0122] For the above two conference modes, at present, most video conferences are mainly AVC conferences, and cloud video conferences are mainly SVC conferences, and the two modes penetrate each other.

[0123] In view of the fact that in an AVC conference, the MCU encodes a large number of conference scene picture code streams, resulting in large encoding computing power and resource consumption of the MCU, the embodiment of the present application provides a coding and decoding method. After the MCU generates conference scene pictures of a plurality of conference participants based on conference pictures of the plurality of conference participants, the plurality of conference scene pictures are first divided into a plurality of image regions. Then, multiplexing analysis is performed on the plurality of image regions to screen out image regions that are different from each other from the plurality of image regions, and the screened image regions are independently encoded to obtain sub-image code streams corresponding to the plurality of image regions, respectively. Further, based on the plurality of sub-image code streams, code stream splicing is performed according to image regions included in the plurality of conference scene pictures to obtain global image code streams of the plurality of conference scene pictures. Finally, the MCU sends the global image code streams of the conference scene pictures to corresponding conference participants.

[0124] As can be seen, in the technical solution provided in the embodiment of the present application, when encoding a plurality of conference scene pictures, the MCU only needs to encode a plurality of image regions that are different from each other (i.e., not repeated, differentiated). For the same image region in each conference scene picture, only one encoding is required. Compared with the way of directly encoding the entire conference scene picture in the related art, the consumption of encoding resources and encoding computing power is reduced. In other words, for a plurality of conference scene pictures, as long as there are the same image regions in the plurality of conference scene pictures, the technical solution of the present application can avoid repeated encoding of the same image regions, thereby reducing the consumption of encoding resources and encoding computing power. Moreover, as the number of conference scene pictures increases or the number of the same image regions in the conference scene pictures increases, the technical solution provided in the embodiment of the present application can greatly reduce the consumption of encoding resources and encoding computing power, and the benefits achieved by the overall solution will be better and better.

[0125] Finally, the implementation environment of the embodiment of the present application is introduced.

[0126] Referring to FIG. 5, FIG. 5 is a schematic diagram of an example environment provided by this disclosure. The example environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. The source device 10 can generate encoded video, i.e., a bitstream. Thus, the source device 10 can also be referred to as an encoding device. The destination device 20 can decode the bitstream generated by the source device 10. Thus, the destination device 20 can also be referred to as a decoding device. The link 30 can receive the encoded video generated by the source device 10 and can transmit the encoded video to the destination device 20. The storage device 40 can receive the encoded video generated by the source device 10 and can store the encoded video, where the destination device 20 can retrieve the encoded video directly from the storage device 40. Alternatively, the storage device 40 can correspond to a file server or another intermediate storage device that stores the encoded video generated by the source device 10, where the destination device 20 can access the encoded video stored by the storage device 40 via streaming or download.

[0127] The source device 10 and the destination device 20 can each include one or more processors and a memory coupled to the one or more processors, which can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer, and the like. For example, the source device 10 and the destination device 20 can each include a phone, a smart phone, a personal digital assistant (PDA), a wearable device, a pocket personal computer (PPC), a tablet computer, a smart car, a smart television, a smart speaker, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called “smart” phone, a television, a camera, a display device, a digital media player, a video gaming console, an in-car computer, or the like.

[0128] Link 30 can include one or more media or devices that enable transmission of the encoded video from source device 10 to destination device 20. In one possible implementation, link 30 can include one or more communication media to enable source device 10 to transmit encoded video directly to destination device 20 in real-time. In embodiments of the application, source device 10 can modulate the encoded video based on a communication standard to include wireless communication protocols, and transmit the modulated video to destination device 20. The one or more communication media can include wireless and / or wired communication media, such as the one or more communication media can include a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet, among others. The one or more communication media can include routers, switches, base stations, or other equipment that facilitates communication from source device 10 to destination device 20, which embodiments of the application are not limited to.

[0129] In one possible implementation, storage device 40 can store the received encoded video transmitted by source device 10, and destination device 20 can access the encoded video directly from storage device 40. In this regard, storage device 40 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray disc, digital versatile disc (DVD), compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing a code stream, among others.

[0130] In one possible implementation, the storage device 40 can correspond to a file server, or another intermediate storage device that can hold the code stream generated by the source device 10, and the destination device 20 can access the stored image via streaming or download from the storage device 40. The file server can be any type of server capable of storing encoded video and transmitting that encoded video to the destination device 20. In one possible implementation, the file server can include a web server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. The destination device 20 can access the encoded video data from the file server via any standard data connection, including an Internet connection. The

[0131] The implementation environment shown in FIG. 5 is merely one possible implementation, and the techniques of embodiments of the present application can be applied to other devices that can encode video and devices that can decode a code stream, which are not shown in FIG. 5, without specific limitations on the present application.

[0132] In some embodiments, the source device 10 in the implementation environment shown in FIG. 5 can include a data source 120, an encoder 100, and an output interface 140. The output interface 140 can include a modulator / demodulator (modem) and / or a transmitter, which can also be referred to as a transmitter; the data source 120 can include a video capture device (e.g., a video camera, etc.), an archive containing previously captured video, a feed interface for receiving video from a video content provider, and / or a computer graphics system for generating video, or a combination of these sources of video.

[0133] The data source 120 can send video to the encoder 100, which can encode the video received from the data source 120 to obtain encoded video. The encoder can send the encoded video to the output interface. In some embodiments, the source device 10 sends the encoded video directly to the destination device 20 via the output interface 140. In other embodiments, the encoded video can also be stored onto a storage device 40 for later access by the destination device 20 for decoding and / or display.

[0134] In some embodiments, destination device 20 in the implementation environment of FIG. 5 can include an input interface 240, a decoder 200, and a display device 220. Input interface 240, which can include a receiver and / or a modem, can receive encoded video from, for example, link 30 and / or storage device 40 and can send the encoded video to decoder 200, which can decode the received encoded video to produce decoded video. Decoder 200 can send the decoded video to display device 220. Display device 220 can be integrated with or can be external to destination device 20. In general, display device 220 displays the decoded video. Display device 220 can be any of a variety of display devices such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0135] Although not shown in FIG. 5, in aspects, encoder 100 and decoder 200 can each be integrated with an encoder and a decoder, respectively, and can include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software, for encoding of both audio and video in a common data stream or separate data streams. In some embodiments, the MUX-DEMUX units can conform to the ITU H.223 multiplexer protocol, or other protocols, such as the user datagram protocol (UDP), as appropriate.

[0136] The encoder 100 and the decoder 200 can each be any of one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuitry, hardware, or any combinations thereof. If the techniques of this disclosure are implemented partially in software, a device can store instructions for the software in a suitable, non- volatile computer-readable storage medium and execute the instructions in hardware using one or more processors to implement the techniques of this disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be considered to be one or more processors. Each of the encoder 100 and the decoder 200 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device.

[0137] Embodiments of the disclosure can generally refer to the encoder 100 as "signaling" or "sending" certain information to another device, such as the decoder 200. The term "signaling" or "sending" can generally refer to the communication of syntax elements and / or other data used for decoding the compressed video. This communication can occur in real-time or near real-time. Alternatively, this communication can occur after a period of time, such as when syntax elements are stored in a computer-readable storage medium in an encoded bitstream at the time of encoding, which a decoding device can then retrieve at any time after the syntax elements are stored to this medium.

[0138] Referring to FIG. 6, FIG. 6 is a schematic diagram of another implementation environment provided by embodiments of the disclosure, which includes a video distribution device and a plurality of client devices. The video distribution device and the plurality of client devices can communicate with each other in a wired or wireless manner, and the number of client devices is not limited by embodiments of the disclosure.

[0139] In a possible implementation, the video distribution device can be a network device capable of realizing point-to-point communication of the plurality of client devices, such as a switch, a gateway, an MCU, etc., or a cluster composed of a plurality of network devices, and embodiments of the disclosure do not limit this.

[0140] In the implementation of the coding method provided in the embodiments of the present application, the video distribution device determines a plurality of to-be-encoded images based on the requirements of the client devices, each to-be-encoded image corresponding to a code stream, i.e., being sent to a client device. For the plurality of to-be-encoded images, the plurality of to-be-encoded images are first divided into a plurality of image regions, and there is at least one same image region in any two to-be-encoded images in the plurality of to-be-encoded images. Then, the video distribution device performs multiplexing analysis on the plurality of image regions to screen out image regions that are different from each other in the plurality of image regions, and obtains a plurality of sub-image code streams obtained after the screened image regions are encoded. Further, the video distribution device performs code stream splicing based on the plurality of sub-image code streams according to the image regions included in the plurality of to-be-encoded images, i.e., splicing the sub-image code streams corresponding to the image regions included in each to-be-encoded image to obtain a global image code stream corresponding to each to-be-encoded image.

[0141] In some embodiments, if the application scenario is a multi-party video conference, the video distribution device can be an MCU, and the client device can be a video conference terminal of each participant. The MCU first determines a conference scene picture of each participant based on the participation pictures of the participants, and takes the conference scene pictures of the participants as to-be-encoded images to perform the coding method provided in the embodiments of the present application, i.e., to perform image region division, analyze the multiplexing of the image regions, encode the screened image regions to obtain a plurality of sub-image code streams, and splice the sub-image code streams to obtain a global image code stream corresponding to the conference scene picture of each participant. Then, the MCU sends the global image code streams corresponding to the participants to the video conference terminals of the corresponding participants.

[0142] The video conference terminal of each participant receives the global image code stream and performs the decoding method provided in the embodiments of the present application to parse the sub-image code streams corresponding to the plurality of image regions in the global image code stream, obtain a plurality of reconstructed image regions, and perform splicing processing on the plurality of reconstructed image regions to obtain a reconstructed image, i.e., a conference scene picture corresponding to the participant.

[0143] The video conference terminal of each participant receives the global image code stream and performs the decoding method provided in the embodiments of the present application to parse the sub-image code streams corresponding to the plurality of image regions in the global image code stream, obtain a plurality of reconstructed image regions, and perform splicing processing on the plurality of reconstructed image regions to obtain a reconstructed image, i.e., a conference scene picture corresponding to the participant.

[0144] In a possible implementation, since the video distribution device needs to not only divide the image region and analyze the multiplexing of the image region, but also encode the screened image regions, and considering that the encoding needs to consume certain resources and computing power, the video analysis apparatus can be used to divide the image region and analyze the multiplexing of the image region, so as to screen the image regions; the target encoding apparatus is used to encode the screened image regions, so as to obtain the sub-image code streams, and the sub-image code streams are fed back to the video analysis apparatus, so that the video analysis apparatus continues to perform the code stream splicing and distribution operations.

[0145] The number of target encoding apparatuses can be one or multiple, and the embodiments of the present application do not limit this.

[0146] Based on this, if the application scenario is a multi-party video conference, as shown in FIG. 7, the video distribution device can include a video analysis apparatus and a target encoding apparatus. The video analysis apparatus can be an MCU, and the client device can be a video conference terminal of each participant. The MCU determines the conference picture of each participant based on the participation pictures of the participants, takes the conference pictures of the participants as to-be-encoded images, performs the encoding method provided in the embodiments of the present application to divide the to-be-encoded images into image regions, analyzes the multiplexing of the image regions, screens the image regions that are different from each other, and sends an encoding request to the target encoding apparatus based on the screened image regions. The encoding request is used to request the target encoding apparatus to encode the screened image regions respectively. The target encoding apparatus encodes the screened image regions respectively in response to the encoding request, to obtain sub-image code streams, and then sends the sub-image code streams to the MCU. Further, the MCU splices the sub-image code streams based on the sub-image code streams to obtain global image code streams corresponding to the conference pictures of the participants according to the image regions included in the conference pictures, and sends the global image code streams corresponding to the participants to the video conference terminals of the participants.

[0147] Similarly, after receiving the global image code streams, the video conference terminals of the participants perform the decoding method provided in the embodiments of the present application to analyze the sub-image code streams corresponding to the image regions in the global image code streams, obtain reconstructed image regions, and splice the reconstructed image regions to obtain a reconstructed image, that is, a conference picture corresponding to the participant.

[0148] The coding method provided by the embodiments of the present application can be applied to various scenes, and the images coded in various scenes can be images included in an image file or images included in a video file. It should be noted that, in combination with the implementation environment shown in FIG. 5, any of the encoding methods below can be performed by the encoder 100 in the source device 10, which is implemented by software, hardware or a combination of both to become part or all of the video distribution device in the embodiments of the present application. Any of the decoding methods below can be performed by the decoder 200 in the destination device 20, which is implemented by software, hardware or a combination of both to become part or all of the client device in the embodiments of the present application.

[0149] Please refer to FIG. 8, which is a structural schematic diagram of a client device according to an embodiment of the present application. The client device includes at least one processor 801, a communication bus 802, a memory 803 and at least one communication interface 804, and has an image decoding and reconstruction function.

[0150] The processor 801 can be a general central processing unit (CPU), a network processor (NP), a microprocessor, or can be one or more integrated circuits for implementing the solutions of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0151] The communication bus 802 is used to transmit information between the above-mentioned components. The communication bus 802 can be divided into an address bus, a data bus, a control bus and the like. For the sake of representation, only one thick line is shown in FIG. 8, but it does not mean that there is only one bus or only one type of bus.

[0152] The memory 803 can be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disk (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory 803 can exist independently, and is connected to the processor 801 through the communication bus 802. The memory 803 can also be integrated with the processor 801.

[0153] The communication interface 804 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 804 includes a wired communication interface and can also include a wireless communication interface. The wired communication interface can be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, a wired interface, or a combination thereof. The wireless communication interface can be, for example, a wireless local area networks (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0154] In a specific implementation, as an example, the processor 801 can include one or more CPUs, such as CPU0 and CPU1 as shown in FIG. 8.

[0155] In a specific implementation, as an example, the client device can include multiple processors, such as the processor 801 and the processor 805 as shown in FIG. 8. Each of these processors can be a single-core processor or a multi-core processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0156] In a specific implementation, as an embodiment, the client device can further include an output device and an input device. The output device is in communication with the processor 801 and can display information in a variety of ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device is in communication with the processor 801 and can receive user input in a variety of ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device, etc.

[0157] In some embodiments, the memory 803 is configured to store program code 810 for implementing the solutions of the present application, and the processor 801 can execute the program code 810 stored in the memory 803. The program code 810 can include one or more software modules, and the client device can implement the decoding method provided in the embodiment of FIG. 11 by means of the processor 801 and the program code 810 in the memory 803.

[0158] It should be noted that the application scenarios and implementation environments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It can be known by those skilled in the art that, as the application scenarios and implementation environments evolve, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0159] Next, the coding and decoding solutions provided by the embodiments of the present application will be explained in detail.

[0160] FIG. 9 is a flowchart of an encoding method provided by an embodiment of the present application. The method is applied to an encoding end, and can be applied to a video distribution device at the encoding end. Please refer to FIG. 9, the method includes the following steps.

[0161] Step 901: performing region division on each of a plurality of to-be-encoded images to obtain a plurality of image regions; wherein the plurality of to-be-encoded images correspond to different bitstreams, and there are at least two to-be-encoded images including the same image region in the plurality of to-be-encoded images.

[0162] For the plurality of to-be-encoded images, after region division, each to-be-encoded image is divided into at least one image region. At this time, there are at least two to-be-encoded images including the same image region in the plurality of to-be-encoded images, that is, there can be one same image region or multiple same image regions in the image regions corresponding to the at least two to-be-encoded images, which is not limited by the embodiments of the present application.

[0163] As an example, the to-be-encoded images include image M, image N and image P, the image M is divided into m1 image region and m2 image region, the image N is divided into n1 image region and n2 image region, and the image P is divided into p1 image region, p2 image region and p3 image region, and there is at least one image region that is the same among the m1 image region, the m2 image region, the n1 image region, the n2 image region, the p1 image region, the p2 image region and the p3 image region. For example, the m1 image region is the same as the p1 image region, or the m2 image region, the n2 image region and the p2 image region are the same.

[0164] In the embodiment of the present application, each to-be-encoded image corresponds to a code stream, that is, a global image code stream corresponding to each to-be-encoded image is generated in the subsequent step 904, and the code streams corresponding to the plurality of to-be-encoded images are different.

[0165] In the multi-party video conference scene, the plurality of to-be-encoded images can be the conference scene pictures of a plurality of participants, and the conference scene picture is obtained by splicing the participant pictures sent by the video conference terminals of different participants. After the encoding scheme provided in the embodiment of the present application is executed, a global image code stream corresponding to each to-be-encoded image is obtained, and each global image code stream indicates a conference scene picture of a participant.

[0166] For any participant, the participant picture of the participant includes face information, body shape information and environmental background of the participant, and the participant picture can be obtained by photographing the environment in which the participant is located by the video conference terminal of the participant; the conference scene picture of the participant includes the participant pictures of other participants in the multi-party video conference, that is, a combined picture obtained by splicing the participant pictures uploaded by other participants. For example, the participant picture of participant A shown in FIG. 1 can be a face image of A, and the conference scene picture of participant A is a picture obtained by splicing the participant picture of participant B (i.e., the face image of participant B), the participant picture of participant C (i.e., the face image of participant C) and the participant picture of participant D (i.e., the face image of participant D).

[0167] It should be noted that for any two to-be-encoded images in the plurality of to-be-encoded images, there can be one same image region or a plurality of same image regions. As the number of same image regions in the plurality of to-be-encoded images increases, the number of mutually different image regions screened out in the following step 902 is smaller, and thus the encoding computing power and encoding resources consumed in the step 903 of encoding the screened image regions are also smaller.

[0168] In a possible implementation, to improve the encoding efficiency of the plurality of to-be-encoded images as a whole, a smaller division granularity can be used to divide each to-be-encoded image into a larger number of image regions. In this way, when performing the subsequent step 902, a larger number of repeated image regions can be determined, and a smaller number of image regions can be filtered out, so as to reduce the encoding computing power and encoding resources consumed when encoding the filtered image regions in the following step 903.

[0169] In the step 901, each to-be-encoded image is divided into at least one image region, and thus the number of the plurality of image regions obtained after the division is greater than or equal to the number of the plurality of to-be-encoded images.

[0170] It should be noted that the contents of the image regions obtained after the division of each to-be-encoded image can be the same or different. In other words, after the division of each to-be-encoded image, a plurality of image regions with the same content or a plurality of image regions with different contents can be obtained, which depends on the image information of the to-be-encoded image, and the embodiments of the present application do not limit this. Moreover, the sizes of the plurality of image regions obtained after the division can be the same or different. In other words, after the division of each to-be-encoded image, a plurality of image regions with the same size or a plurality of image regions with different sizes can be obtained. The embodiments of the present application do not limit this.

[0171] In a possible implementation, the implementation process of the step 901 can be that each to-be-encoded image is divided into a plurality of image regions according to a division manner of an image slice, an image tile or an image patch.

[0172] In a possible implementation, before the division of the plurality of to-be-encoded images into image regions, each to-be-encoded image can be analyzed to determine the division size information of each to-be-encoded image, that is, the size of the slice, tile or patch; and then, based on the determined division size information, each to-be-encoded image is divided into a plurality of image regions corresponding to the to-be-encoded image.

[0173] It should be understood that different encoders and different encoding methods can have multiple region division manners. In addition to the slice defined in H. 264 and H. 265, the tile defined in H. 265, the tile concept in AV1 (a new type of video encoding format developed by the alliance for open media (AOMedia)), the tile and slice concepts in versatile video coding (VVC, also known as H. 266), and the patch concept in the third generation of digital audio and video coding standard (AVS3), the embodiments of the present application do not limit the region division manners introduced in different video encoding formats.

[0174] In some embodiments, taking the scenario of a multi-party video conference as an example, the to-be-encoded image can be any frame of conference scene picture watched by a target participant in the multi-party video conference. The target participant is any participant in the multi-party video conference.

[0175] Based on this, before performing the above step 901, the technical solution provided by the embodiments of the present application further includes the following steps (1)-(3).

[0176] (1) receiving a participant picture code stream of each participant in the multi-party video conference.

[0177] For each participant, the participant picture code stream is obtained by encoding the participant picture of the environment in which the participant is located by the video conference terminal after collecting the participant picture, and sending the participant code stream to the video distribution device.

[0178] As an example, the participant picture code stream can be a 4K@30FPS code stream, and the embodiments of the present application do not limit this.

[0179] (2) analyzing the participant picture code stream of each participant to obtain the participant picture of each participant.

[0180] After receiving the participant picture of each participant, the video distribution device can perform scaling processing on multiple pictures to reduce the size of each participant picture, thereby facilitating the subsequent step (3) of splicing.

[0181] As an example, after scaling processing, the resolution of each participant picture can be reduced to 1080P.

[0182] (3) based on the participant picture of each participant, splicing into a conference picture of each participant, and the conference picture includes the participant picture of other participants in the multi-party video conference.

[0183] That is, the video distribution device splices the multiple participant pictures according to the participant picture requirements of each participant based on the participant picture of each participant, to generate the conference picture of each participant.

[0184] In a possible implementation, the conference picture of each participant can further include the participant picture of the participant itself and the participant pictures of other participants in the multi-party video conference. In this case, each participant can not only clearly understand the participation status of other participants, but also intuitively understand the participation status of the participant itself through the displayed conference picture.

[0185] In a possible implementation, the participant picture included in the conference picture of each participant can also be determined by the participant itself, that is, the participant can interact with the video distribution device through the video conference terminal of the participant itself, to instruct the video distribution device to splice the participant picture of the specified object according to the requirements of the participant, to obtain the conference picture of the participant.

[0186] As an example, taking the A conference picture shown in FIG. 1 as an example, after receiving the participant pictures of participant A, participant B, participant C and participant D, the video distribution device splices the participant picture of participant B, the participant picture of participant C and the participant picture of participant D according to a 2*2 interface layout, to generate the A conference picture of participant A. In the A conference picture, the B image area is the participant picture of participant B, the C image area is the participant picture of participant C, and the D image area is the participant picture of participant D.

[0187] It should be noted that the A conference picture is taken as an example here, but in actual application, the A conference picture can further include the participant picture of participant A, or only include the participant pictures of part of the participants. Of course, the layout of the A conference picture in FIG. 1 is only one example in the embodiments of the present application, and in actual application, the technical solution of the present application does not limit the layout and position of each participant picture in the conference picture.

[0188] Step 902: screening out image areas different from each other from the multiple image areas.

[0189] In a possible implementation, the implementation process of step 902 can be: performing multiplicity analysis on the multiple image areas, to determine which image areas are different and which image areas are the same, so as to screen out image areas different from each other from the multiple image areas.

[0190] In other words, for the same image region in the plurality of image regions, only one image region is filtered out, and the sub-image code stream of the image region after subsequent encoding can be spliced into the global image code stream of the plurality of to-be-encoded images. In this way, through multiplexing analysis, the same image region needs to be encoded only once in the technical solution of the present application, and does not need to be repeatedly encoded in each to-be-encoded image.

[0191] It should be understood that, in the case that each to-be-encoded image is divided into at least one image region, and any two to-be-encoded images in the plurality of to-be-encoded images have at least one same image region, a plurality of image regions will be filtered out after the multiplexing analysis of step 902.

[0192] In a possible implementation, the multiplexing analysis includes image content multiplexing analysis and image size multiplexing analysis. The content multiplexing analysis refers to analyzing whether the pixel content of each image region in the plurality of image regions is the same, and the image size multiplexing analysis refers to analyzing whether the size of each image region in the plurality of image regions is the same, i.e., whether the resolution is the same. Based on this, the mutually different image regions filtered out refer to the image regions filtered out that are different in size and content.

[0193] That is, for the above step 902, after the multiplexing analysis of the plurality of image regions, a plurality of image regions can be filtered out, and any two image regions in the plurality of filtered-out image regions are different in size and / or content.

[0194] As an example, as shown in FIG. 10, it is assumed that the plurality of to-be-encoded images includes the A conference screen, the B conference screen, the C conference screen and the D conference screen shown in FIG. 1, and the region division is performed on the plurality of to-be-encoded images to obtain a plurality of image regions, including the B image region, the C image region, the D image region, the E1 image region and the F1 image region divided based on the A conference screen; the A image region, the C image region, the D image region, the E2 image region and the F2 image region divided based on the B conference screen; the A image region, the B image region, the D image region, the E3 image region and the F3 image region divided based on the C conference screen; and the A image region, the B image region, the C image region, the E4 image region and the F4 image region divided based on the D conference screen. Since the interface layouts of the four conference screens are all 2*2, the sizes of the A image region, the B image region, the C image region and the D image region obtained after the region division are all the same, and the image contents of the regions corresponding to the same letters are also the same. Therefore, after the multiplexing analysis of the A image region, the B image region, the C image region and the D image region included in the four conference screens, the filtered-out image regions include the A image region, the B image region, the C image region and the D image region.

[0195] If the A conference picture, the B conference picture, the C conference picture and the D conference picture are respectively divided into regions, the sizes and contents of the obtained E image regions (i.e., E1 image region-E4 image region) are also completely same, and only one of them is selected. Similarly, if the A conference picture, the B conference picture, the C conference picture and the D conference picture are respectively divided into regions, the sizes and contents of the obtained F image regions (i.e., F1 image region-F4 image region) are also completely same, and only one of them is selected.

[0196] In this way, the multiple image regions corresponding to the A conference picture, the B conference picture, the C conference picture and the D conference picture are selected from the image regions after the multiplexing analysis, and the selected image regions include the A image region, the B image region, the C image region, the D image region, the E image region (for example, E1 image region) and the F image region (for example, F1 image region).

[0197] Therefore, after the multiplexing analysis, the embodiment of the present application does not need to completely encode the four conference pictures, but only needs to encode the A image region, the B image region, the C image region, the D image region, the E image region and the F image region, i.e., the six image regions, once, without repeatedly encoding the A image region, the B image region, the C image region, the D image region, the E image region and the F image region in each conference picture. In this way, the consumption of encoding resources and encoding computing power is reduced, and the encoding efficiency is improved.

[0198] In some embodiments, if the to-be-encoded image is divided into multiple Slices or multiple Tiles, there may be some differences in the multiplexing analysis of the two types of image regions. Since the division of the Slice may require it to cover all macroblocks or CTUs in the entire row in the vertical direction, the size and position of the Slice may be limited by the row boundary. Unlike the Slice, the Tile can be divided in the horizontal and vertical directions, and therefore the Tile may have higher flexibility in multiplexing.

[0199] As an example, taking the four conference pictures shown in FIG. 1 as the to-be-encoded image, if the type of the image region obtained after the division of the to-be-encoded image is Tile, for the C conference picture and the D conference picture, the A image region and the B image region in the first row are two Tiles, and the two Tiles can be multiplexed in the C conference picture and the D conference picture, and the A image region and the B image region need to be encoded once.

[0200] As another example, taking the four conference pictures shown in FIG. 1 as the to-be-encoded images, if the type of the image region obtained after region division of the four to-be-encoded images is Slice, then for the C conference picture and the D conference picture, the A image region and the B image region in the first row are a Slice as a whole, and this Slice can be reused in the C conference picture and the D conference picture, and only one encoding is required for the entire region containing the A region and the B region.

[0201] It should be noted that Slice and Tile have their own advantages in video encoding. Slice is mainly used to limit the spread of errors and improve encoding efficiency, while Tile is more used to improve parallel processing capability and reduce encoding delay. Although Slice must be full of each row to improve decoding efficiency, this may limit its flexibility in reuse; Tile may have higher flexibility in reuse, but it may not be as effective as Slice in limiting error spread and improving encoding efficiency. Therefore, when choosing to use Slice or Tile, a trade-off can be made according to the specific application scenario and requirements.

[0202] Step 903: Obtain a plurality of sub-image code streams obtained after the screened image regions are encoded respectively.

[0203] Among them, the screened image regions and the sub-image code streams are one-to-one correspondence, and the number of the two is exactly the same. That is, each screened image region is independently encoded to obtain the sub-image code stream corresponding to the image region.

[0204] As described above, for the video distribution device, steps 901-904 in the embodiments of the present application can be independently executed by the video distribution device, that is, step 903 is also independently completed by the video distribution device. Of course, for the video distribution device, the above steps 901, 902 and 904 can be implemented by an image analysis device, and step 903 is executed by a dedicated target encoding device, and the image analysis device only needs to interact with the target encoding device to obtain the plurality of sub-image code streams returned by the target encoding device.

[0205] Next, the implementation of the two kinds of sub-image code stream acquisition methods will be introduced respectively.

[0206] In a possible implementation, in the case that the video distribution device independently completes the image region encoding, the implementation process of the above step 503 can be: the video distribution device encodes the screened image regions respectively to obtain a plurality of sub-image code streams.

[0207] In a possible implementation, if the image region type obtained by the division is Slice or Tile, since the encoding of Slice and Tile is independent, the video distribution device can use multiple processors or processing units to simultaneously encode multiple Slices or Tiles, thereby realizing parallel processing to significantly improve the encoding speed of the video distribution device.

[0208] In another possible implementation, in the case where the video distribution device comprises a video analysis apparatus and a target encoding apparatus, the implementation process of step 503 can be: the video analysis apparatus sends an encoding request to the target encoding apparatus, the encoding request being used to request the target encoding apparatus to respectively encode the filtered image regions; and the video analysis apparatus receives the multiple sub-image code streams sent by the target encoding apparatus.

[0209] That is, in the case where the video analysis apparatus filters the image regions that need to be encoded from the multiple image regions, the target encoding apparatus performs the encoding operation, that is, encodes the filtered image regions to obtain the multiple sub-image code streams, and sends the multiple sub-image code streams to the video analysis apparatus.

[0210] Of course, since Slice or Tile can be independently encoded, the target encoding apparatus can also use a parallel manner to encode the filtered image regions to improve the encoding efficiency.

[0211] Taking the image region type Tile as an example, each image region (that is, each Tile) filtered can be encoded using the conventional encoding technology of H.265, including the steps of prediction (intra prediction and inter prediction), transformation, quantization and entropy encoding. Among them, the encoding between Tiles is independent, that is, the encoding of one Tile will not be affected by other Tiles. The encoding manner of the image region is not limited in the embodiments of the present application.

[0212] In some embodiments, considering that although the Tiles in H.265 can be independently encoded in the encoding process, they cannot be completely independent when being decoded, and the Tiles will cross the Tile boundary and use the encoding information of any Tile in the encoded image in the processes of inter prediction and loop filtering. Based on this, in order to ensure that the Tiles can be independently encoded and decoded, H.265 can protect the boundary of the Tiles when encoding and decoding the Tiles.

[0213] In a possible implementation, the implementation process of respectively encoding the filtered image regions can be: determining boundary restriction information of the filtered image regions, the boundary restriction information comprising time domain motion vector (MV) restriction information and space domain MV restriction information; and respectively encoding the filtered image regions based on the boundary restriction information to obtain the multiple sub-image code streams.

[0214] In some embodiments, the time domain MV restriction is a restriction on the search range of a reference frame. For example, a rule can be set such that a pixel block in a current Tile does not search for a best matching block in a reference frame beyond a fixed region in the reference frame corresponding to the location of the current Tile. In this way, the decoding of the current Tile is ensured to be independent of the decoding results of neighboring Tiles. The spatial domain MV restriction is a restriction on the magnitude and direction of a motion vector of a pixel block in a current frame. For example, a rule can be set such that a motion vector of a pixel block in a current Tile does not point to a region of a neighboring Tile. In this way, the decoding of the current Tile is ensured to be independent of the neighboring Tiles.

[0215] In one possible implementation, to ensure that a Tile can be independently coded, the concept of motion constrained tile sets (MCTS) is introduced in H.265, which restricts the search range of a motion vector by modifying the encoder to ensure that each Tile does not rely on the data of other Tiles when encoding. In this way, each Tile can be treated as an independent coding unit, thereby improving the parallelism of encoding.

[0216] Further, to ensure the independence of a Tile, H.265 also restricts the use of temporal motion vector prediction (TMVP) to be within the boundary of a Tile, which uses the motion vector of a pixel block in a neighboring frame at the same or a neighboring location to predict the motion vector of a current block. However, in Tile encoding, due to the data isolation between Tiles, the motion vector of a neighboring Tile cannot be simply used for prediction. Therefore, H.265 restricts the use of TMVP to ensure that it is used within a Tile, thereby avoiding the dependence between Tiles.

[0217] That is, when independently encoding a Tile, the use of MCTS and TMVP can be restricted to ensure the independence of a Tile in the encoding and decoding processes, thereby fully utilizing the computing power of a processor and improving the efficiency and performance of video encoding.

[0218] In some embodiments, in video encoding, especially when using Tile encoding techniques, each Tile is independently encoded, but when decoding, it can be affected by neighboring Tiles. That is, this independence can lead to quality degradation at the Tile boundaries, resulting in the phenomenon of boundary flickering. Based on this, when encoding the selected image area, the quality of the periphery of each Tile can also be enhanced by adjusting the mapping of the quantization parameter (QP), that is, QP_Map, to prevent flickering.

[0219] In a possible implementation, based on the above boundary restriction information, before encoding the selected image area, the quantization parameter of the coding block at the boundary of the selected image area also needs to be adjusted to reduce the quantization step of the coding block at the boundary of the selected image area.

[0220] Wherein, QP is a key parameter in video encoding for controlling the quantization step, the quantization step determines the compression degree of data in the encoding process, a larger QP value will lead to a longer quantization step, thereby introducing more distortion, but also reducing the code rate. Therefore, by adjusting the QP value, a balance between code rate and distortion can be achieved.

[0221] QP_Map is a map used to represent the QP values of different regions in an image. In Tile encoding, a QP_Map can be created for each Tile, and the QP value at the Tile boundary can be adjusted as needed.

[0222] As an example, taking the image area as a Tile, the quantization parameter of the coding block at the boundary can be adjusted in the following three ways, including but not limited to, to enhance the image quality at the boundary of each Tile.

[0223] Method one: reduce the QP value at the boundary.

[0224] Since a smaller QP value corresponds to a more refined quantization, it can retain more image details, so the QP value at the boundary can be reduced to reduce the quantization step, to retain the detail information at the Tile boundary and reduce distortion, effectively prevent the quality degradation and flickering phenomenon in these areas, thereby improving the overall video quality.

[0225] Method two: smooth QP transition.

[0226] At the Tile boundary, sudden changes in QP values can lead to discontinuity in quality and flickering. Therefore, the effect can be reduced by smoothing the QP transition. For example, gradually changing QP values can be set at the Tile boundary to ensure the continuity of the quality.

[0227] Method three: QP adjustment based on content.

[0228] In addition to simply increasing the QP value at the Tile boundary, the QP value can also be adjusted according to the image content. For example, if the Tile boundary contains important edge information or texture details, the QP value of these areas can be reduced to preserve more detailed information.

[0229] Step 904: Based on the plurality of sub-image code streams, the sub-image code streams of the image regions included in each of the plurality of to-be-encoded images are spliced according to the positions of the image regions included in each of the plurality of to-be-encoded images, to obtain the global image code streams corresponding to the plurality of to-be-encoded images respectively.

[0230] In a possible implementation, the implementation process of step 904 can be as follows: the sub-image code streams of the image regions included in each of the plurality of to-be-encoded images are spliced according to the positions of the image regions included in each of the plurality of to-be-encoded images, to obtain the global image code streams corresponding to the plurality of to-be-encoded images respectively.

[0231] Continuing to take the A conference picture, the B conference picture, the C conference picture, and the D conference picture shown in FIG. 10 as an example, for the multiple image regions A image region, the B image region, the C image region, the D image region, the E image region, and the F image region that are different from each other and are screened out, although these image regions exist in each conference picture, the positions of these image regions in different conference pictures can be different. Therefore, when the global image code streams are generated by splicing, the positions of the image regions included in each of the plurality of to-be-encoded images need to be combined, and the sub-image code streams corresponding to the image regions are spliced, to generate the global image code streams corresponding to each of the plurality of to-be-encoded images.

[0232] Based on this, after obtaining the sub-picture code streams corresponding to the A image region, the B image region, the C image region, the D image region, the E image region and the F image region respectively through the step 903, the sub-picture code stream of the B image region, the sub-picture code stream of the C image region, the sub-picture code stream of the E image region, the sub-picture code stream of the D image region and the code stream of the F image region are sequentially spliced according to the interface layout of the A conference picture to obtain the global image code stream of the A conference picture. Similarly, the sub-picture code stream of the A image region, the sub-picture code stream of the C image region, the sub-picture code stream of the E image region, the sub-picture code stream of the D image region and the code stream of the F image region are sequentially spliced according to the interface layout of the B conference picture to obtain the global image code stream of the B conference picture; the sub-picture code stream of the A image region, the sub-picture code stream of the B image region, the sub-picture code stream of the E image region, the sub-picture code stream of the D image region and the code stream of the F image region are sequentially spliced according to the interface layout of the C conference picture to obtain the global image code stream of the C conference picture; and the sub-picture code stream of the A image region, the sub-picture code stream of the B image region, the sub-picture code stream of the E image region, the sub-picture code stream of the C image region and the code stream of the F image region are sequentially spliced according to the interface layout of the D conference picture to obtain the global image code stream of the D conference picture.

[0233] In a possible implementation, for each to-be-encoded image, in order to facilitate subsequent decoding to reconstruct a reconstructed image with the same picture layout, the generated global image code stream carries position information of each image region included in the to-be-encoded image.

[0234] The position information of each image region can include a starting point and an offset, and based on the two parameters, the specific position of the image region in the corresponding to-be-encoded image can be determined.

[0235] As an example, the starting point of each image region can be denoted as entry_point_offset_minus1, and the offset can be denoted as offset.

[0236] In some embodiments, taking a multi-party video conference as an example, if the to-be-encoded image is any frame conference picture viewed by a target participant in the multi-party video conference, and the target participant is any participant in the multi-party video conference, after generating the global image code stream corresponding to each to-be-encoded image through the steps 901-904, the video distribution device can further send the global image code stream of the corresponding conference picture to each participant's video conference terminal.

[0237] In this way, after receiving the global image code stream, each participant's video conference terminal can decode the global image code stream by using the decoding method provided in the embodiments of the present application, and obtain the reconstructed conference picture by splicing a plurality of reconstructed image regions.

[0238] In summary, in the embodiment of the present application, when encoding multiple to-be-encoded images, the multiple to-be-encoded images are first divided into multiple image regions; then the multiplexing of each image region is analyzed to filter out image regions that are different from each other from the multiple image regions. When encoding, only the image regions that are filtered out and different from each other (i.e., not repeated and differentiated) need to be encoded, and the same image regions in the multiple to-be-encoded images only need to be encoded once. In this way, compared with the manner of directly encoding the entire to-be-encoded image in the related art, the consumption of encoding resources and encoding computing power in the encoding process is reduced.

[0239] In other words, after the multiple to-be-encoded images are divided into image regions, if there are the same image regions in the multiple to-be-encoded images, the technical solution of the present application can effectively avoid repeated encoding of the same image regions and reduce the consumption of encoding resources and encoding computing power. Moreover, as the number of to-be-encoded images or the number of the same image regions in the to-be-encoded images increases, the technical solution provided in the embodiment of the present application can greatly reduce the consumption of encoding resources and encoding computing power, and the benefits achieved by the overall solution will be better and better.

[0240] FIG. 11 is a flow diagram of a decoding method provided in an embodiment of the present application, which is applied to a decoding end and can be applied to the client device shown in FIG. 8.

[0241] Step 1101: receiving a global image code stream, the global image code stream including sub-image code streams corresponding to multiple image regions.

[0242] The multiple image regions are determined according to the division manners of image slices, image tiles, or image patches.

[0243] That is, in the embodiment of the present application, the global image code stream received by the client is not a code stream directly obtained by encoding the entire image, but a global image code stream obtained by splicing sub-image code streams corresponding to multiple image regions in the image.

[0244] Step 1102: analyzing the sub-image code streams corresponding to the multiple image regions to obtain multiple reconstructed image regions.

[0245] Each image region corresponds to a sub-image code stream, and decoding the corresponding sub-image code stream can reconstruct the image region, thereby obtaining the corresponding reconstructed image region. In other words, the image region, the sub-image code stream, and the reconstructed image region are in one-to-one correspondence.

[0246] In a possible implementation, since the image regions of the Tile, Slice and the like types can be independently coded, i.e., without dependence on other Tiles, Slices, and the like, the client device can also decode the multiple sub-image bitstreams in parallel after receiving the sub-image bitstreams corresponding to the multiple image regions, thereby improving the decoding rate.

[0247] Step 1103: Splicing the multiple reconstructed image regions to obtain a reconstructed image.

[0248] In a possible implementation, the global image bitstream carries position information of the multiple image regions, and the position information is used to indicate positions of the multiple image regions in the reconstructed image.

[0249] In a possible implementation, the implementation process of step 1103 can be that the multiple reconstructed image regions are spliced based on the position information of the multiple image regions to obtain the reconstructed image. In other words, the multiple reconstructed image regions are spliced in sequence based on the position information of each image region to obtain the reconstructed image.

[0250] In summary, in the embodiments of the present application, since the video distribution device can reuse the coded image regions when encoding the multiple to-be-encoded images, the efficiency of the coded images is higher, so that the client device can receive the corresponding global image bitstream more quickly and smoothly. Moreover, when the client device receives the sub-image bitstreams corresponding to the multiple image regions, the client device can parse the multiple sub-image bitstreams in parallel to obtain the multiple reconstructed image regions, thereby improving the decoding efficiency.

[0251] For ease of understanding, the encoding and decoding method provided in the embodiments of the present application is exemplarily explained and described below taking a multi-party video conference scenario as an example.

[0252] Taking a five-party video conference as an example, the video distribution device can be an MCU, or the video distribution device includes an MCU and a target encoding apparatus; and the client device can be a video conference terminal of each participant. The video conference terminal of each participant is in communication connection with the MCU.

[0253] As shown in FIG. 12, the video conference terminal of each participant sends a participant picture bitstream of itself to the video distribution device. The video distribution device decodes the multiple participant picture bitstreams by using a corresponding decoder to obtain corresponding participant pictures, and then scales the multiple participant pictures according to a preset picture layout size, and generates a conference picture of each participant according to the picture requirement of each participant.

[0254] As an example, according to the picture requirement of the participant 1, the participant 2's participant picture is determined as the participant 1's conference picture (i.e., the first conference picture); according to the picture requirement of the participant 2, the participant 1's, participant 3's, participant 4's and participant 5's participant pictures are spliced together as the participant 2's conference picture (i.e., the second conference picture); according to the picture requirement of the participant 3, the participant 1's, participant 2's, participant 4's and participant 5's participant pictures are spliced together as the participant 3's conference picture (i.e., the third conference picture); according to the picture requirement of the participant 4, the participant 1's, participant 2's, participant 3's and participant 5's participant pictures are spliced together as the participant 4's conference picture (i.e., the fourth conference picture); according to the picture requirement of the participant 5, the participant 1's, participant 2's, participant 3's and participant 4's participant pictures are spliced together as the participant 5's conference picture (i.e., the fifth conference picture).

[0255] The first conference picture to the fifth conference picture obtained above are taken as to-be-encoded pictures, and the video distribution device performs region division on the first conference picture to the fifth conference picture by using the encoding method shown in the embodiment of FIG. 9, to obtain a plurality of image regions. After the region division, the first conference picture obtains the second image region, the second conference picture obtains the first image region, the third image region, the fourth image region and the fifth image region, the third conference picture obtains the first image region, the second image region, the fourth image region and the fifth image region, the fourth conference picture obtains the first image region, the second image region, the third image region and the fifth image region, and the fifth conference picture obtains the first image region, the second image region, the third image region and the fourth image region. That is, after the region division on the five conference pictures, 17 image regions are obtained.

[0256] Then, the multiplexing analysis is performed on the 17 image regions, and different image regions are screened out, i.e., image regions with different content and size are screened out. As shown in FIG. 12, the image regions screened out after the multiplexing analysis include: the 2nd image region included in the 1st conference picture, the 1st image region included in the 2nd conference picture / 3rd conference picture, the 3rd image region included in the 2nd conference picture, the 2nd image region included in the 3rd conference picture, the 4th image region included in the 2nd conference picture / 3rd conference picture, the 5th image region included in the 2nd conference picture / 3rd conference picture, the 1st image region included in the 4th conference picture / 5th conference picture, the 2nd image region included in the 4th conference picture / 5th conference picture, the 3rd image region included in the 4th conference picture / 5th conference picture, the 5th image region included in the 4th conference picture, and the 4th image region included in the 5th conference picture. That is, 11 different image regions are screened out from the 17 image regions, and the 11 image regions can be multiplexed in different conference pictures.

[0257] Further, the video distribution device acquires sub-image code streams obtained after the screened multiple image regions are encoded, and then performs code stream splicing on the multiple sub-image code streams based on the multiple conference pictures, to generate a global image code stream corresponding to each conference picture, and sends the generated global image code stream to a video conference terminal of a corresponding participant.

[0258] In the process of acquiring the sub-image code streams obtained after the multiple image regions are encoded, the video distribution device can perform the encoding operation through the MCU, or perform the encoding operation through a target encoding device and send the encoded sub-image code streams to the MCU.

[0259] Referring to FIG. 13, when the MCU screens the image regions and encodes through the processor of the MCU, the implementation process can be: for the multiple image regions obtained after the multiple to-be-encoded images are divided into image regions, the MCU performs multiplexing analysis on the multiple image regions to screen out different image regions; then the screened image regions are encoded respectively by using the boundary limit information and reducing the QP value at the boundary, to obtain multiple sub-image code streams; the MCU generates a global image code stream corresponding to each to-be-encoded image by splicing the sub-image code streams based on the multiple sub-image code streams, and modifies the global code stream information to carry the position information of each image region included in the to-be-encoded image in the global code stream information.

[0260] In a possible implementation, when the MCU interacts with the target coding device to obtain the plurality of sub-image code streams, the implementation process can be as follows: for the plurality of image regions obtained after the image region division of the plurality of to-be-coded images, the MCU performs multiplexing analysis on the plurality of image regions to filter out different image regions; then the MCU sends a coding request to the target coding device, and the coding request carries information of the plurality of image regions. The target coding device codes the pluralityed image regions to obtain the plurality of sub-image code streams, and returns the plurality of sub-image code streams to the MCU. After receiving the plurality of sub-image code streams, the MCU generates a global image code stream corresponding to each to-be-coded image by splicing the sub-image code streams, and modifies the global code stream information to carry position information of each image region included in the to-be-coded image.

[0261] The target coding device codes the plurality of image regions in a manner similar to the MCU, and thus is not described herein again.

[0262] For each video conference terminal of a participant, after receiving the global image code stream corresponding to the conference scene picture, the video conference terminal can analyze the plurality of sub-image code streams in the global image code stream according to the decoding method shown in the embodiment of FIG. 11 to obtain corresponding reconstructed image regions, splice the plurality of reconstructed image regions to obtain a reconstructed image, that is, a reconstructed conference scene picture, and display the reconstructed conference scene picture to the participant.

[0263] In summary, for the scenario of multi-party video conference, when the video distribution device codes the plurality of conference scene pictures, only the plurality of image regions that are different from each other (that is, not repeated and differentiated) need to be coded. For the same image region in each conference scene picture, only one coding is needed. Compared with the manner of directly coding the entire conference scene picture in the related art, the consumption of coding resources and coding computing power is reduced. In other words, for the plurality of conference scene pictures, as long as there is a same image region in the plurality of conference scene pictures, the technical solution of the present application can avoid repeated coding of the same image region, and reduce the consumption of coding resources and coding computing power. Moreover, as the number of conference scene pictures increases or the number of same image regions in the conference scene pictures increases, the technical solution provided in the embodiment of the present application can greatly reduce the consumption of coding resources and coding computing power, and the yield of the overall solution will be better and better.

[0264] FIG. 14 is a structural schematic diagram of an encoding device provided in an embodiment of the present application. The encoding device can be realized by software, hardware, or a combination of both to become part or all of an encoding device, which can be the video distribution device described above. Referring to FIG. 14, the device includes an image division module 1401, a region filtering module 1402, a code stream acquisition module 1403, and a code stream splicing module 1404.

[0265] The image division module 1401 is configured to divide a plurality of to-be-encoded images respectively to obtain a plurality of image regions; wherein the plurality of to-be-encoded images correspond to different bitstreams, and at least two to-be-encoded images in the plurality of to-be-encoded images include the same image region.

[0266] The region screening module 1402 is configured to screen mutually different image regions from the plurality of image regions.

[0267] The bitstream acquisition module 1403 is configured to acquire a plurality of sub-image bitstreams obtained after the screened image regions are encoded respectively.

[0268] The bitstream splicing module 1404 is configured to splice the bitstreams based on the plurality of sub-image bitstreams according to the image regions included in the plurality of to-be-encoded images respectively to obtain global image bitstreams corresponding to the plurality of to-be-encoded images respectively.

[0269] In a possible implementation, the bitstream acquisition module 1403 includes:

[0270] The encoding unit is configured to encode the screened image regions respectively to obtain the plurality of sub-image bitstreams.

[0271] In a possible implementation, the encoding unit is further configured to:

[0272] determine boundary restriction information of the screened image regions; wherein the boundary restriction information includes temporal motion vector (MV) restriction information and spatial MV restriction information;

[0273] encode the screened image regions respectively based on the boundary restriction information to obtain the plurality of sub-image bitstreams.

[0274] In a possible implementation, the encoding unit is specifically configured to:

[0275] adjust a quantization parameter of a coding block at a boundary of the screened image region to reduce a quantization step of the coding block at the boundary of the screened image region.

[0276] In a possible implementation, the bitstream acquisition module 1403 includes:

[0277] The sending unit is configured to send an encoding request to a target encoding device, the encoding request being used to request the target encoding device to encode the screened image regions respectively.

[0278] The bitstream receiving unit is configured to receive the plurality of sub-image bitstreams sent by the target encoding device.

[0279] In a possible implementation, the image division module 1401 is specifically configured to:

[0280] According to a division manner of the image slice, the image tile or the image patch, the plurality of to-be-encoded images are respectively divided into regions to obtain a plurality of image regions.

[0281] In a possible implementation, the code stream splicing module 1404 is specifically configured to:

[0282] According to positions of the image regions included in each of the plurality of to-be-encoded images, the sub-image code streams of the image regions included in each of the plurality of to-be-encoded images are spliced to obtain global image code streams corresponding to the plurality of to-be-encoded images respectively.

[0283] In a possible implementation, the global image code stream carries position information of the image regions included in the corresponding to-be-encoded image.

[0284] In a possible implementation, the to-be-encoded image is any frame of conference scene picture watched by a target participant in the multi-party video conference, and the target participant is any participant in the multi-party video conference; and the encoding apparatus further includes:

[0285] a code stream receiving module configured to receive participant picture code streams of each participant in the multi-party video conference;

[0286] a picture reconstruction module configured to parse the participant picture code streams of each participant to obtain participant pictures of each participant;

[0287] a picture splicing module configured to splice the participant pictures of each participant to obtain conference pictures of each participant, and the conference pictures include participant pictures of other participants in the multi-party video conference.

[0288] In a possible implementation, the encoding apparatus further includes:

[0289] a code stream sending module configured to send global image code streams of the conference pictures corresponding to each participant to each participant.

[0290] In the embodiments of the present application, when encoding the plurality of to-be-encoded images, the plurality of to-be-encoded images are first divided into regions to obtain a plurality of image regions, and then the multiplexing of the image regions is analyzed to screen out image regions that are different from each other from the plurality of image regions. When encoding, only the image regions that are different from each other (i.e., not repeated, differentiated) screened out need to be encoded, and the same image regions in the plurality of to-be-encoded images only need to be encoded once. In this way, compared with the manner of directly encoding the entire to-be-encoded image in the related art, the consumption of encoding resources and encoding computing power in the encoding process is reduced.

[0291] In other words, for a plurality of to-be-encoded images, if there are the same image regions in the plurality of to-be-encoded images, the technical solution of the present application can effectively avoid repeated encoding of the same image regions, and reduce the consumption of encoding resources and encoding computing power. Moreover, as the number of to-be-encoded images increases, or the number of the same image regions in the to-be-encoded images increases, the technical solution provided in the embodiments of the present application can greatly reduce the consumption of encoding resources and encoding computing power, and the overall scheme will achieve better and better benefits.

[0292] It should be noted that the encoding device provided in the above embodiments is only used as an example to illustrate the division of the above functional modules when encoding a plurality of to-be-encoded images. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the encoding device and the encoding method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0293] FIG. 15 is a structural schematic diagram of a decoding device provided in an embodiment of the present application. The decoding device can be realized by software, hardware or a combination of both to become part or all of a decoding device, which can be the client device described above. Referring to FIG. 15, the device includes a bitstream receiving module 1501, a region reconstruction module 1502 and a region splicing module 1503.

[0294] The bitstream receiving module 1501 is configured to receive a global image bitstream, and the global image bitstream includes a plurality of image region corresponding sub-image bitstreams.

[0295] The region reconstruction module 1502 is configured to parse the plurality of image region corresponding sub-image bitstreams to obtain a plurality of reconstructed image regions.

[0296] The region splicing module 1503 is configured to splice the plurality of reconstructed image regions to obtain a reconstructed image.

[0297] In a possible implementation, the global image bitstream carries position information of the plurality of image regions, and the region splicing module 1503 is specifically configured to:

[0298] Splice the plurality of reconstructed image regions based on the position information to obtain the reconstructed image.

[0299] In a possible implementation, the plurality of image regions are determined according to a division manner of image slices, image tiles or image patches.

[0300] In the embodiments of the present application, since the video distribution device can multiplex the encoded image regions when encoding multiple to-be-encoded images, the efficiency of the encoded images is higher, so that the decoding device can receive the corresponding global image code stream faster and more smoothly. Moreover, when receiving the sub-image code streams corresponding to the multiple image regions, the decoding device can analyze the multiple sub-image code streams in parallel to obtain the multiple reconstructed image regions, thereby improving the decoding efficiency.

[0301] It should be noted that the decoding device provided in the above embodiments is only used as an example to illustrate the division of the above functional modules when decoding the global image code stream to obtain the reconstructed image. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the decoding device and the decoding method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0302] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the encoding method shown in the above embodiments or executes the steps of the decoding method shown in the above embodiments.

[0303] The embodiments of the present application also provide a computer program product, which contains computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the steps of the encoding method shown in the above embodiments or executes the steps of the decoding method shown in the above embodiments. In other words, a computer program is provided. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the encoding method shown in the above embodiments or executes the steps of the decoding method shown in the above embodiments.

[0304] The embodiments of the present application also provide a coding system, which includes an encoding device and a decoding device. The encoding device is used to implement the steps of the encoding method shown in the above embodiments, and the decoding device is used to implement the steps of the decoding method shown in the above embodiments.

[0305] The embodiments of the present application also provide an encoded code stream, which is generated according to the encoding method shown in the above embodiments.

[0306] In a possible implementation, the code stream includes a plurality of sub-picture code streams and a global picture code stream; the plurality of sub-picture code streams are obtained by respectively encoding the filtered image regions, the filtered image regions are mutually different image regions in a plurality of image regions, and the plurality of image regions are obtained by dividing a plurality of to-be-encoded pictures, and at least two to-be-encoded pictures in the plurality of to-be-encoded pictures include the same image region.

[0307] The embodiment of the present application further provides a computer readable storage medium, which stores the code stream generated according to the encoding method shown in the above embodiment.

[0308] The embodiment of the present application further provides a device for storing a code stream, which includes a receiver and at least one storage medium, the receiver is used to receive the code stream generated according to the encoding method shown in the above embodiment, and the at least one storage medium is used to store the code stream.

[0309] The embodiment of the present application further provides a device for transmitting a code stream, which includes a transmitter and a receiver, the receiver is used to receive the code stream generated according to the encoding method shown in the above embodiment, and the transmitter is used to transmit the code stream to an end-side device through a transmission medium.

[0310] The embodiment of the present application further provides a device for transmitting a code stream, which includes a transmitter and at least one storage medium, the at least one storage medium is used to store the code stream generated according to the encoding method shown in the above embodiment, and the transmitter is used to acquire the code stream from the storage medium and transmit the code stream to an end-side device through a transmission medium.

[0311] The embodiment of the present application further provides a system for distributing a code stream, which includes at least one storage medium, the at least one storage medium is used to store the code stream generated according to the encoding method shown in the above embodiment, and a streaming media device is used to acquire a target code stream from the at least one storage medium and transmit the target code stream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.

[0312] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example: floppy disk, hard disk, magnetic tape), optical media (for example: digital versatile disc (DVD)) or semiconductor media (for example: solid state disk (SSD)) and the like. It should be noted that the computer-readable storage medium mentioned in the embodiments of the present application can be a non-volatile storage medium, in other words, it can be a non-transitory storage medium.

[0313] It should be understood that "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" herein only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, "first", "second" and the like are used to distinguish the same items or similar items with basically the same function and role in the embodiments of the present application. Those skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different.

[0314] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the conference picture code stream, the conference picture, the conference picture and the like involved in the embodiments of the present application are obtained under sufficient authorization.

[0315] The above describes the embodiments provided by the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An encoding method characterized by comprising: The method comprises: dividing a plurality of to-be-encoded images into regions respectively to obtain a plurality of image regions; wherein the plurality of to-be-encoded images correspond to different code streams, and at least two to-be-encoded images in the plurality of to-be-encoded images include the same image region; selecting mutually different image regions from the plurality of image regions; obtaining a plurality of sub-image code streams obtained after the selected image regions are encoded respectively; based on the plurality of sub-image code streams, splicing code streams according to the image regions included in the plurality of to-be-encoded images respectively to obtain global image code streams corresponding to the plurality of to-be-encoded images respectively.

2. The method of claim 1, wherein, The method further comprises: encoding the selected image regions respectively to obtain the plurality of sub-image code streams.

3. The method of claim 2, wherein, The method further comprises: determining boundary restriction information of the selected image regions; wherein the boundary restriction information comprises time domain motion vector (MV) restriction information and space domain MV restriction information; based on the boundary restriction information, encoding the selected image regions respectively to obtain the plurality of sub-image code streams.

4. The method of claim 3, wherein, The method further comprises: adjusting a quantization parameter of a coding block at a boundary in the selected image regions to reduce a quantization step of the coding block at the boundary in the selected image regions.

5. The method of claim 1, wherein, The method further comprises: sending an encoding request to a target encoding device, the encoding request being used to request the target encoding device to encode the selected image regions respectively; receiving the plurality of sub-image code streams sent by the target encoding device.

6. The method according to any one of claims 1 to 5, wherein, The method further comprises: dividing the plurality of to-be-encoded images into regions according to a division mode of an image slice, an image tile or an image patch to obtain the plurality of image regions.

7. The method according to any one of claims 1 to 6, wherein The method further comprises: based on the plurality of sub-image code streams, splicing code streams according to the image regions included in the plurality of to-be-encoded images respectively to obtain global image code streams corresponding to the plurality of to-be-encoded images respectively.

8. The method of claim 7, wherein, The method further comprises:

9. The method according to any one of claims 1 to 8, wherein, splicing sub-image code streams of image regions included in each to-be-encoded image according to positions of the image regions included in the each to-be-encoded image to obtain global image code streams corresponding to the plurality of to-be-encoded images respectively. The global image code stream carries position information of each image region included in a corresponding to-be-encoded image. The to-be-encoded image is any frame of conference scene picture watched by a target participant in a multi-party video conference, and the target participant is any participant in the multi-party video conference; the method further comprises: receiving a participant picture code stream of each participant in the multi-party video conference; parsing the participant picture code stream of the each participant to obtain a participant picture of the each participant; Splice into the conference scene picture of each participant based on the conference scene picture of each participant, the conference scene picture includes the conference scene picture of other participants in the multi-party video conference.

10. The method of claim 9, wherein, After obtaining the global image code stream corresponding to each of the plurality of to-be-encoded images, the method further comprises: sending the global image code stream corresponding to the conference scene picture of each participant to the participant.

11. A decoding method, comprising: The method comprises: receiving a global image code stream, the global image code stream comprising a plurality of image regions corresponding to sub-image code streams; parsing the sub-image code streams corresponding to the plurality of image regions to obtain a plurality of reconstructed image regions; splicing the plurality of reconstructed image regions to obtain a reconstructed image.

12. The method of claim 11, wherein, The global image code stream carries position information of the plurality of image regions. The splicing processing of the plurality of reconstructed image regions to obtain a reconstructed image comprises: based on the position information, splicing the plurality of reconstructed image regions to obtain the reconstructed image.

13. The method of claim 11 or 12, wherein, The plurality of image regions are determined according to the division mode of image slices, image tiles or image patches.

14. An encoding apparatus, comprising: The device comprises: an image division module for dividing a plurality of to-be-encoded images into a plurality of image regions, wherein the plurality of to-be-encoded images correspond to different code streams, and at least two to-be-encoded images in the plurality of to-be-encoded images include the same image region; a region screening module for screening mutually different image regions from the plurality of image regions; a code stream acquisition module for acquiring a plurality of sub-image code streams obtained after the screened image regions are encoded; a code stream splicing module for splicing the plurality of sub-image code streams according to the image regions included in the plurality of to-be-encoded images to obtain a global image code stream corresponding to each of the plurality of to-be-encoded images.

15. A decoding apparatus, characterized by comprising: The device comprises: a code stream receiving module for receiving a global image code stream, the global image code stream comprising a plurality of image regions corresponding to sub-image code streams; a region reconstruction module for parsing the sub-image code streams corresponding to the plurality of image regions to obtain a plurality of reconstructed image regions; a region splicing module for splicing the plurality of reconstructed image regions to obtain a reconstructed image.

16. An encoding device, comprising: The encoding device comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program to implement the steps of the encoding method according to any one of claims 1-10.

17. A decoding device, comprising: The decoding device comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program to implement the steps of the decoding method according to any one of claims 11-13.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program runs on a computer or a processor, the computer or the processor executes the steps of the encoding method according to any one of claims 1-10, or executes the steps of the decoding method according to any one of claims 11-13.

19. A computer program product, characterised in that, The computer program product comprises computer instructions which, when executed by a computer or a processor, cause the steps of the encoding method according to any one of claims 1-10 to be performed, or the steps of the decoding method according to any one of claims 11-13 to be performed.

20. A coded bitstream, comprising: The code stream is generated according to the method of any one of claims 1-10.

21. A coded bitstream, comprising: The code stream comprises a plurality of sub-picture code streams and a global picture code stream. The plurality of sub-picture code streams are obtained by respectively encoding the selected image regions, the selected image regions are mutually different image regions in the plurality of image regions, and the plurality of image regions are obtained by dividing a plurality of to-be-encoded pictures, and at least two to-be-encoded pictures in the plurality of to-be-encoded pictures comprise the same image region.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores the code stream according to any one of claims 1-10.

23. An apparatus for storing a bitstream, characterized by The apparatus comprises a receiver and at least one storage medium, The receiver is configured to receive the code stream according to any one of claims 1-10. The at least one storage medium is configured to store the code stream.

24. An apparatus for transmitting a code stream, the apparatus comprising: The apparatus comprises a transmitter and a receiver, The receiver is configured to receive the code stream according to any one of claims 1-10. The transmitter is configured to transmit the code stream to an end-side device via a transmission medium.

25. An apparatus for transmitting a code stream, the apparatus comprising: The apparatus comprises a transmitter and at least one storage medium, The at least one storage medium is configured to store the code stream according to any one of claims 1-10. The transmitter is configured to obtain the code stream from the storage medium and transmit the code stream to an end-side device via a transmission medium.

26. A system for distributing a code stream, the system comprising: The system comprises: At least one storage medium configured to store at least one code stream according to any one of claims 1-10; A streaming media device configured to obtain a target code stream from the at least one storage medium and transmit the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

Images