Image data transmission method and device

Abstract

The embodiment of the invention provides an image data transmission method and device, and relates to the technical field of data processing. The image data comprises a plurality of tiles; each tile comprises at least one slice; the plurality of tiles comprise a first tile and a second tile; the method comprises the following steps: acquiring a plurality of code stream groups; the plurality of code stream groups comprise a first code stream group and a second code stream group, the first code stream group comprises a code stream obtained by slice coding of a first tile, and the second code stream group comprises a code stream obtained by slice coding of a second tile; transmitting the bit stream to a receiving device; the bit streams comprise a plurality of code streams belonging to a plurality of code stream groups, and in the bit streams, at least one code stream belonging to a second code stream group is included between two adjacent code streams belonging to a first code stream group. Therefore, the processing time delay can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to an image data transmission method and device. BACKGROUND

[0002] Video and image are widely used in real life and work. The processing of video and image data can include acquisition, encoding, decoding, and display, etc. A sending device acquires an image, encodes image data to obtain a bitstream, and sends the bitstream to a receiving device; the receiving device receives the bitstream, decodes the bitstream to obtain image data, and displays the image. With the increasing requirements of resolution, frame rate (fps), quality, etc. of video and image, the performance requirements of devices for processing video and image are also increasing. The processing capability of a single chip or a single processor core chip is difficult to meet the performance requirements of the chip. At present, image data can be processed in parallel through a multi-die chip, or a multi-processor core chip, or multiple chips. For example, an image can be divided into multiple regions, and each region is processed in parallel by a different processor core to achieve the effect of doubling the overall processing performance. However, the prior art has the problem of high processing delay. SUMMARY

[0003] The present application provides an image data transmission method and device, which solves the problem of high processing delay in the prior art.

[0004] To achieve the above object, the present application adopts the following technical solutions:

[0005] In a first aspect, an image data transmission method is provided, the image data comprising a plurality of tiles; each tile comprising at least one slice; the plurality of tiles comprising a first tile and a second tile; the method comprising: obtaining a plurality of bitstream groups; the plurality of bitstream groups comprising a first bitstream group and a second bitstream group, the first bitstream group comprising bitstreams obtained by slice encoding of the first tile, and the second bitstream group comprising bitstreams obtained by slice encoding of the second tile; transmitting a bitstream to a receiving device; the bitstream comprising a plurality of bitstreams belonging to the plurality of bitstream groups, and in the bitstream, at least one bitstream belonging to the second bitstream group is included between two adjacent bitstreams belonging to the first bitstream group.

[0006] In the above technical solution, at least one bitstream belonging to the second bitstream group is included between two adjacent bitstreams belonging to the first bitstream group, that is, bitstreams in the first bitstream group and the second bitstream group are transmitted in advance, and the receiving device can decode the bitstreams in advance. In this way, the constraint of the coding and decoding standard is broken, processing is performed on smaller granularity slices, and the decoding processing delay can be reduced. Further, if the bitstream belongs to a bitstream group that should be transmitted last, the image frame can also be started to output in advance, further reducing the processing delay of image frame display.

[0007] In a possible implementation of the first aspect, the plurality of tiles further includes a third tile, and the plurality of bitstream groups further includes a third bitstream group; the third bitstream group includes bitstreams obtained by slice encoding of the third tile; and in the bitstream, between two adjacent bitstreams belonging to the first bitstream group, there are M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group; M and N are both integers greater than or equal to 1. In the possible implementation, between the two adjacent bitstreams belonging to the first bitstream group, there are the M bitstreams belonging to the second bitstream group and the N bitstreams belonging to the third bitstream group, that is, at least one bitstream exists in the first bitstream group, the second bitstream group, and the third bitstream group, which is transmitted in advance. The receiving device can decode the at least one bitstream in advance, so that the processing delay can be reduced. Further, if the at least one bitstream belongs to a bitstream group that should be transmitted last, the image frame can also be output in advance, further reducing the processing delay.

[0008] In a possible implementation of the first aspect, M = N. In the possible implementation, the order of the bitstreams in the bitstream is relatively simple, and the process of the receiving device for recovering the order of the bitstream groups from the bitstream is relatively simple.

[0009] In a possible implementation of the first aspect, in the bitstream, the transmission order of each bitstream of the plurality of bitstream groups in the bitstream is determined according to the encoding time order of each bitstream. In the possible implementation, the transmission order of each bitstream is determined according to the encoding time order of each bitstream, so that the time for which a bitstream that has been encoded waits for transmission can be reduced. Therefore, the processing delay can be reduced.

[0010] In a possible implementation of the first aspect, the bitstream includes indication information, and the indication information is used to indicate a tile in which a slice corresponding to a bitstream is located. In the possible implementation, the transmission order of each bitstream is determined according to the encoding time order of each bitstream, and although the transmission process at the sending device side is relatively simple, the receiving device cannot determine which bitstream corresponds to which tile when decoding. By setting the indication information in the bitstream, the receiving device can recover the bitstream into a plurality of bitstream groups according to the indication information, and place the bitstreams belonging to the same bitstream group into a corresponding decoding buffer. Then the receiving device can decode the bitstreams to obtain slices, and place the slices into image positions of tiles corresponding to the slices.

[0011] In one possible implementation of the first aspect, each slice in the multiple tiles includes multiple Maximum Coding Units (LCUs); the indication information includes the LCU start address carried in the bitstream; or, the bitstream includes a Network Abstraction Layer (NAL), and the indication information includes tile identification information in the NAL. In the above possible implementations, the form of the indication information is quite diverse. Existing LCU start addresses can be used as indication information. Indication information can also be added to the NAL of the bitstream. By setting indication information, the accuracy of image data transmission can be improved.

[0012] In one possible implementation of the first aspect, the bitstream is encapsulated in a NAL. In the above possible implementations, indicator information is added to the existing encapsulated bitstream's NAL, eliminating the need to create a new NAL, thus simplifying the bitstream implementation.

[0013] In one possible implementation of the first aspect, each slice in the plurality of tiles includes P maximum coding units (LCUs); P is an integer greater than or equal to 1. In the above possible implementations, each slice includes the same number of LCUs, and the size of the image region corresponding to each slice is the same (e.g., all with 4 LCUs, or all with 1 LCU row). This simplifies the complexity of slice division and encoding / decoding.

[0014] In one possible implementation of the first aspect, the number of P is determined based on the network transmission speed. In the above possible implementations, considering the size of the slice based on the network transmission speed can reduce transmission latency.

[0015] In one possible implementation of the first aspect, the bitstream corresponding to each slice in the multiple tiles has the same byte size. In the above possible implementations, the bitstream size after encoding each slice is the same byte size, which makes the processing latency of slice encoding, transmission, and decoding more stable and improves network transmission efficiency.

[0016] In one possible implementation of the first aspect, the byte size of the bitstream is determined based on the network transmission unit size. In the above possible implementations, considering the size of the slice division based on the network transmission unit size can reduce transmission latency and simplify the transmission process.

[0017] In one possible implementation of the first aspect, the method is applied to a transmitting device comprising multiple encoders for encoding slices of tiles to obtain a bitstream; the number of tiles is equal to the number of encoders. In this possible implementation, the number of tiles equals the number of encoders, allowing one encoder to encode one slice of a tile, thus enabling parallel processing of multiple tiles.

[0018] In one possible implementation of the first aspect, each encoder is used to encode a slice of a tile; the size of the tile is determined according to the encoding speed of the encoder that encodes the tile. In the above possible implementation, the size of the tile is determined according to the encoding speed of the encoder that encodes the tile. This allows the performance and latency of each tile encoding to be the same or close, resulting in optimal overall image encoding performance and latency.

[0019] In a second aspect, an image data transmission method is provided, wherein the image data includes multiple tiles; each tile includes at least one slice; the multiple tiles include a first tile and a second tile; the method includes: receiving a bit stream transmitted by a transmitting device; the bit stream includes multiple bit streams belonging to multiple bit stream groups; the multiple bit stream groups include a first bit stream group and a second bit stream group, the first bit stream group includes a bit stream obtained by slice encoding of the first tile, and the second bit stream group includes a bit stream obtained by slice encoding of the second tile; in the bit stream, at least one bit stream belonging to the second bit stream group is included between two adjacent bit streams belonging to the first bit stream group; and multiple bit stream groups are obtained according to the bit stream, the multiple bit stream groups being used to obtain image data after decoding.

[0020] In the above technical solution, the two adjacent bitstreams belonging to the first bitstream group include at least one bitstream belonging to the second bitstream group. This means that a bitstream in either the first or second bitstream group is transmitted in advance, allowing for earlier decoding. This breaks through the constraints of the encoding / decoding standard, enabling processing with smaller granular slices and reducing decoding latency. Furthermore, if the bitstream belongs to a bitstream group that should have been transmitted last, it may also allow the image frame to be output earlier, further reducing the processing latency for image frame display.

[0021] In one possible implementation of the second aspect, the multiple tiles also include a third tile, and the multiple bitstream groups also include a third bitstream group; the third bitstream group includes bitstreams obtained by slicing and encoding the third tiles; in the bitstream, between two adjacent bitstreams belonging to the first bitstream group, there are M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group; M and N are both integers greater than or equal to 1. In the above possible implementation, between two adjacent bitstreams belonging to the first bitstream group, there are M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group, that is, at least one bitstream in the first, second, and third bitstream groups is transmitted in advance. The receiving device can decode at least one bitstream in advance, thus reducing processing latency. Furthermore, if the at least one bitstream belongs to a bitstream group that should have been transmitted last, then the image frame may be started output earlier, further reducing processing latency.

[0022] In one possible implementation of the second aspect, M = N. In the above possible implementations, the order of the various code streams in the bitstream is relatively simple, and the process of recovering the order of the code stream groups based on the bitstream is also relatively simple.

[0023] In one possible implementation of the second aspect, the transmission order of each bitstream in the multiple bitstream groups is determined according to the encoding time order of each bitstream. In the above possible implementations, determining the transmission order of each bitstream according to its encoding time order can reduce the waiting time for already encoded bitstreams to be transmitted. Therefore, processing latency can be reduced.

[0024] In one possible implementation of the second aspect, the bitstream includes indication information used to indicate the tile where the slice corresponding to the bitstream is located. In the above possible implementations, the transmission order of each bitstream is determined according to its encoding time sequence. While this process is relatively simple on the transmitting device side, the receiving device cannot determine which bitstream corresponds to which tile during decoding. By setting indication information in the bitstream, the receiving device can reconstruct multiple bitstream groups based on the indication information, placing bitstreams belonging to the same bitstream group into their corresponding decoding buffers. The receiving device can then decode the bitstream to obtain slices and place the slices at the image positions of the tiles corresponding to those slices.

[0025] In one possible implementation of the second aspect, each slice in the multiple tiles includes multiple Maximum Coding Units (LCUs); the indication information includes the LCU start address carried in the bitstream; or, the bitstream includes a Network Abstraction Layer (NAL), and the indication information includes tile identification information in the NAL. In the above possible implementations, the form of the indication information is quite diverse. Existing LCU start addresses can be used as indication information. Indication information can also be added to the NAL of the bitstream. By setting indication information, the accuracy of image data transmission can be improved.

[0026] In one possible implementation of the second aspect, the bitstream is encapsulated in a NAL. In the above possible implementations, indicator information is added to the existing encapsulated bitstream's NAL, eliminating the need to create a new NAL, thus simplifying the bitstream implementation.

[0027] In one possible implementation of the second aspect, each slice in the plurality of tiles includes P maximum coding units (LCUs); P is an integer greater than or equal to 1. In the above possible implementations, each slice includes the same number of LCUs, and the size of the image region corresponding to each slice is the same (e.g., all with 4 LCUs, or all with 1 LCU row). This simplifies the complexity of slice division and encoding / decoding.

[0028] In one possible implementation of the second aspect, the number of P is determined based on the network transmission speed. In the above possible implementations, considering the size of the slice based on the network transmission speed can reduce transmission latency.

[0029] In one possible implementation of the second aspect, the bitstream corresponding to each slice in the multiple tiles has the same byte size. In the above possible implementations, the bitstream after encoding each slice has the same byte size, which makes the processing latency of slice encoding, transmission, and decoding more stable and improves network transmission efficiency.

[0030] In one possible implementation of the second aspect, the byte size of the bitstream is determined based on the network transmission unit size. In the above possible implementations, considering the size of the slice division based on the network transmission unit size can reduce transmission latency and simplify the transmission process.

[0031] Thirdly, an image data transmission apparatus is provided, the apparatus including a module for performing the method provided by the first aspect, or any possible implementation of the first aspect, or the second aspect, or any possible implementation of the second aspect.

[0032] Fourthly, an image data transmission apparatus is provided, the apparatus including a processing circuit and a transceiver circuit, the processing circuit and the transceiver circuit being used to implement the method provided by the first aspect, or any possible implementation of the first aspect, or the second aspect, or any possible implementation of the second aspect.

[0033] Fifthly, a computer-readable storage medium is provided, wherein program code is stored therein, and the program code can be invoked by a processor to execute the methods provided by the first aspect, or any possible implementation thereof, or the second aspect, or any possible implementation thereof.

[0034] Sixthly, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method provided by the first aspect, or any possible implementation of the first aspect, or the second aspect, or any possible implementation of the second aspect.

[0035] Understandably, the apparatus, computer storage medium, or computer program product of any of the image data transmission methods provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0036] Figure 1A schematic diagram illustrating an image transmission process provided in an embodiment of this application;

[0037] Figure 2 This application provides a schematic diagram of the architecture of a transmitting device.

[0038] Figure 3 This application provides a schematic diagram of the architecture of a receiving device.

[0039] Figure 4 A schematic diagram of image segmentation provided in this application embodiment. Figure 1 ;

[0040] Figure 5 A schematic diagram illustrating a process provided for an embodiment of this application;

[0041] Figure 6 A schematic diagram of image segmentation provided in this application embodiment. Figure 2 ;

[0042] Figure 7 A schematic diagram of image segmentation provided in this application embodiment. Figure 3 ;

[0043] Figure 8 A schematic diagram of a packaging process provided for an embodiment of this application;

[0044] Figure 9 A schematic diagram illustrating an encoding / decoding process provided in an embodiment of this application;

[0045] Figure 10 A schematic diagram of image segmentation provided in this application embodiment. Figure 4 ;

[0046] Figure 11 A schematic diagram illustrating an encoding process provided in an embodiment of this application;

[0047] Figure 12 A schematic diagram illustrating a decoding process provided in an embodiment of this application;

[0048] Figure 13 A schematic diagram illustrating the division of tiles and slices provided in an embodiment of this application;

[0049] Figure 14 This application provides an illustration of an image data transmission method. Figure 1 ;

[0050] Figure 15 This application provides an illustration of an image data transmission method. Figure 2 ;

[0051] Figure 16A schematic diagram of a bitstream group provided in an embodiment of this application;

[0052] Figure 17 A schematic diagram of a bitstream provided for an embodiment of this application;

[0053] Figure 18 This application provides an illustration of an image data transmission method. Figure 3 ;

[0054] Figure 19 This application provides an illustration of an image data transmission method. Figure 4 ;

[0055] Figure 20 This application provides an illustration of an image data transmission method. Figure 5 ;

[0056] Figure 21 This application provides an illustration of an image data transmission method. Figure 6 ;

[0057] Figure 22 This application provides an illustration of an image data transmission method. Figure 7 ;

[0058] Figure 23 This application provides an illustration of an image data transmission method. Figure 8 ;

[0059] Figure 24 This application provides an illustration of an image data transmission method. Figure 9 ;

[0060] Figure 25 This application provides an illustration of an image data transmission method. Figure 10 ;

[0061] Figure 26 A schematic diagram of tile division provided in this application embodiment. Figure 1 ;

[0062] Figure 2 A schematic diagram of tile division provided in this application embodiment. Figure 2 ;

[0063] Figure 3 A schematic diagram of tile division provided in this application embodiment. Figure 3 ;

[0064] Figure 4 A schematic diagram illustrating an encoding, transmission, and decoding process provided in an embodiment of this application;

[0065] Figure 5 This is a schematic diagram of a first image data transmission device provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of a second image data transmission device provided in an embodiment of this application;

[0067] Figure 4 A schematic diagram of a third image data transmission device provided in an embodiment of this application;

[0068] Figure 5 This is a schematic diagram of a fourth image data transmission device provided in an embodiment of this application. Detailed Implementation

[0069] It should be noted that the terms "first" and "second" used in the embodiments of this application are only used to distinguish features of the same type and should not be construed as indicating relative importance, quantity, order, etc.

[0070] The terms "exemplary" or "for example" used in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0071] The terms "coupling" and "connection" used in the embodiments of this application should be interpreted broadly. For example, they can refer to a physical direct connection or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0072] First, the application scenarios of the embodiments of this application will be introduced.

[0073] The embodiments of this application can be applied to scenarios such as video and image data processing. Figure 4 As shown, a possible image processing procedure includes: a transmitting device acquiring an image, encoding the image data into a bitstream, and sending the bitstream to a receiving device; the receiving device receiving the bitstream, decoding the bitstream to obtain image data, and displaying the image. Encoding the acquired image into a bitstream by the transmitting device reduces the amount of image data, facilitating transmission. The receiving device decodes the bitstream to obtain image data, thereby displaying the image.

[0074] As the requirements for video image resolution, frame rate (fps), and quality continue to increase (e.g., resolution from standard definition to high definition, then to ultra-high definition; frame rate from 30fps to 60fps, then to 120fps; bit width from 8 bits to 10 bits, then to 12 bits), the performance requirements for devices processing video images are also increasing. The processing power of a single chip or a single processor core is insufficient to meet the performance requirements. Currently, image data can be processed in parallel using solutions such as multiple die-on-die (DIE) chips, multiple processor core chips, or multiple chips.

[0075] This application provides a transmitting device. For example... Figure 5 As shown, the architecture of the transmitting device may include a data acquisition unit, a first processing circuit, multiple encoders (e.g., a first encoder, a second encoder, and a third encoder), multiple encoding buffers (e.g., a first encoding buffer, a second encoding buffer, and a third encoding buffer), and a first transceiver circuit. Please refer to [reference needed] for the coupling relationships between the various components. Figure 6 The coupling relationship in the circuit. The first processing circuit can also be coupled to multiple encoding buffers and the first transceiver circuit. Embodiments of this application also provide a receiving device. Figure 6 As shown, the architecture of the receiving device may include a second transceiver circuit, a second processing circuit, multiple decoding buffers (e.g., a first decoding buffer, a second decoding buffer, and a third decoding buffer), multiple decoders (e.g., a first decoder, a second decoder, and a third decoder), and a display device. Please refer to [reference needed] for the coupling relationships between the various components. Figure 6 The coupling relationship within the circuit. The second processing circuit can also be coupled with multiple decoding buffers and a second transceiver circuit.

[0076] For example, the acquisition device can be a device for acquiring video and images, such as a camera, radar, etc. The processing circuit can be used to perform control and information processing functions. The processing circuit can be a processing chip core, which is the core circuit of the processing chip excluding packaging and auxiliary circuits (such as pin interface circuits, power supply circuits, clock circuits, etc.). The processing chip core can execute one or more steps in the embodiments of this application under software scheduling. The processing circuit can also be a hardware circuit, executing one or more steps in the embodiments of this application through an internal hardware circuit structure. The encoder can be a device for converting input data into a form that can be stored, transmitted, or processed by the processing chip. Encoding is usually performed through fixed rules or encoding rules; for example, video encoding compresses continuous pixel signals into a bitstream. Encoders can improve data readability and efficiency, enabling different types of data to communicate with each other between different systems. The decoder can be used to receive encoded data and convert it back to its original form. The display device can be a device for displaying video and images, such as a display screen, etc. The transceiver circuit can include a receiving circuit and a transmitting circuit. The receiving circuit is used to implement the receiving function, and the transmitting circuit is used to implement the transmitting function. In the transmitting device, the number of acquisition devices can be one or more, and the number of first processing circuits can also be one or more. The first processing circuits in the transmitting device can be integrated into a single chip. For example, it can be a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips. The first processing circuit can be integrated with multiple encoders into a multi-core processing chip or a multi-DIE chip. Multiple encoder buffers can be integrated into this chip or set up separately from it. The first transceiver circuit can be integrated into this chip or set up separately from it. In the receiving device, the number of display devices can be one or more, and the number of second processing circuits can also be one or more. In the receiving device, the second processing circuit can be integrated into a single chip.The second processing circuit can be integrated with multiple decoders into a multi-core processing chip or a multi-DIE chip. Multiple decoding buffers can be integrated into this chip or set up separately. The second transceiver circuit can be integrated into this chip or set up separately.

[0077] For example, the acquisition device can output the acquired image to a first processing circuit. The first processing circuit can process the image data, such as dividing the image data into regions, so that the encoder can encode the partitioned image data. The encoder can encode the image data and store the encoded data in an encoding buffer. The encoded data in the encoding buffer can be transmitted to a receiving device through a first transceiver circuit. The receiving device receives the encoded data through a second transceiver circuit. The encoded data can be stored in a decoding buffer. The decoder can read the data in the decoding buffer and decode the encoded data. The decoded data is placed in the corresponding image region and output to a display device. The display device can display the image.

[0078] For example, such as Figure 7 As shown, the first processing circuit can divide the image horizontally into multiple vertical regions (e.g., region 1, region 2, and region 3). Alternatively, the first processing circuit can divide the image vertically into multiple horizontal regions (e.g., region 4, region 5, and region 6). Each region can be encoded by a different encoder or decoded by a different decoder. Because each region can be processed in parallel, the overall processing performance can be multiplied; for example, a single-core processing chip can achieve 4K@60fps, a dual-core processing chip can achieve 8K@30fps, and a quad-core processing chip can achieve 8K@60fps.

[0079] For example, such as Figure 8 As shown, an image frame can include multiple rectangular blocks, each containing multiple pixels. For example, in the H.265 protocol, a rectangular block is called a large coding unit (LCU). The maximum size of an LCU can be 64*64 pixels, meaning 64 pixels wide and 64 pixels high. Depending on the image encoding requirements, an LCU can be divided into multiple coding units (CUs). When encoding an image, the encoder can encode a row of rectangular blocks from left to right. After one row of rectangular blocks is encoded, the next row is encoded from left to right. This process continues until the last row of rectangular blocks is encoded. For details, please refer to [link / reference]. Figure 9 The arrows in the image. This encoding order can be called Z-scan, or raster scan. Figure 6Each region can comprise multiple rectangular blocks. After dividing the regions, for each region, it can be configured according to... Figure 10 The Z-scan method shown is used for encoding. In the undivided... Figure 11 When the area shown is selected, the entire image can be processed according to... Figure 12 The Z-scan method shown is used for encoding.

[0080] In one possible implementation, such as Figure 13 As shown, an image can be divided into multiple tiles vertically. Each tile can also be divided into multiple slices. Each slice contains multiple LCUs. Figure 14 Only tiles 10A and 20A are shown; in reality, many more tiles could be divided. Figure 15 This only shows tile 10A divided into slices 11A and 12A, and tile 20A divided into slices 21A and 22A; in reality, even more slices can be created. Multiple encoders can encode multiple tiles in parallel, with each tile's encoding independent of the others. Each encoder can encode one tile using a Z-scan method. Multiple decoders can decode multiple tiles in parallel, with each tile's decoding independent of the others. Parallel processing improves encoding and decoding performance.

[0081] In some examples, such as Figure 13 As shown, slices within the same tile can include dependent slices and independent slices. For example, tile 30A includes dependent slice 31A, dependent slice 32A, and dependent slice 33A. Tile 40A includes dependent slice 41A, dependent slice 42A, and dependent slice 43A. The syntax description information of a dependent slice is independent and does not depend on other slices. The syntax information of a dependent slice is not independent; some of its syntax information is contained within the dependent slice of the tile it belongs to. For example, some of the syntax information of dependent slices 32A and 33A is contained in dependent slice 31A, and some of the syntax information of dependent slices 42A and 43A is contained in dependent slice 41A. Both dependent slices 31A and 41A do not depend on other slices. When dividing a tile into slices, the first slice of each tile can be a dependent slice, and the other slices can be dependent slices. Since dependent slices and independent slices share some syntax information, the bitstream size is reduced, which can improve bitstream compression efficiency.

[0082] In other examples, such as Figure 16As shown, the bitstream encoded from each slice of the image is encapsulated into a Network Abstract Layer (NAL) unit according to the protocol. The transmitting device packages, stores, or transmits the NAL units. For example, slice 11A is encoded to obtain bitstream 11A, which is then encapsulated into NAL11A; slice 12A is encoded to obtain bitstream 12A, which is then encapsulated into NAL12A; slice 21A is encoded to obtain bitstream 21A, which is then encapsulated into NAL21A; and slice 22A is encoded to obtain bitstream 22A, which is then encapsulated into NAL22A. NAL11A, NAL12A, NAL21A, and NAL22A are then used as the bitstream of the image, encapsulated together with the audio or subtitle bitstream, and stored sequentially in a medium or transmitted to other devices.

[0083] In this implementation, video image encoding efficiency can be improved by dividing the video into tiles and slices. However, in end-to-end low-latency scenarios involving encoding, transmission, and decoding, the overall latency of this implementation exceeds the frame level, resulting in high processing delay. For example... Figure 17 As shown, tiles 10A and 20A start encoding simultaneously, and slices 11A and 21A have similar encoding times, so their encoding completion times are also similar. However, after the bitstream corresponding to slice 11A is sent, the bitstream corresponding to slice 21A is not sent; instead, it waits for slice 12A to be encoded and sent. This reduces the complexity of decoding by a single decoder but increases the latency of decoding by multiple decoders. When multiple decoders are decoding, the first encoder used to encode tile 10A finishes processing slice 11A, and the second encoder used to encode tile 20A finishes processing slice 21A. The transmitting device then sends slice 11A to the receiving device. The first decoder used to decode tile 10A receives slice 11A and begins decoding. Slice 21A in the transmitting device is in a waiting state, and the second decoder used to decode tile 20A is in an idle state. After all slices of tile 10A are encoded and transmitted, slice 21A in tile 20A is then transmitted (see reference). Figures 6 to 9 (The arrow between tile 10A and tile 20A). After transmission is complete, decoding of slice 21A begins. The encoding time for each tile is at the frame level. Adding the transmission time, the delay displayed in the decoded output exceeds the frame level.

[0084] In another possible implementation, such as Figure 14As shown, an image can be divided into multiple larger regions, such as larger region 10B and larger region 20B. Within each larger region, the encoder encodes it using a Z-scan method. Each larger region can also be divided into multiple smaller regions. For example, larger region 10B can be divided into smaller regions 11B–14B, and larger region 20B can be divided into smaller regions 21B–28B. In a single-encoder scenario, the encoder processes multiple larger regions sequentially. In a multi-encoder scenario, multiple encoders process multiple larger regions in parallel. Figure 15 As shown, in the transmitting device, encoding is performed by dividing a larger region into smaller regions. The bitstream is stored in the buffer corresponding to the larger region. The bitstream in the buffer is then split into bitstream segments of fixed size, and multiple bitstream segments are encapsulated. For example... Figure 16 As shown, in the receiving device, multiple bitstream segments are decapsulated and stored in their corresponding buffers. The bitstreams in each buffer are then decoded, and the smaller decoded regions are placed at their corresponding image locations.

[0085] In this implementation, the image is divided into larger and smaller regions for parallel processing, and the bitstream is then divided into bitstream segments for transmission, reducing buffer size and latency. This implementation is mainly used in light compression fields such as data transmission interfaces, including Mobile Industry Processor Interface (MIPI-DSI), Mobile Industry Processor Interface Xamera Serial Interface (MIPI-CSI), and Embedded Display Port (EDP). However, the compression ratio is relatively low in light compression fields. If this implementation is to be applied to heavy compression fields with higher compression ratios, such as video codec standards H.265 and H.266, the following problems exist:

[0086] First, the division of larger and smaller regions is not limited to LCUs (Local Code Units). The unit for dividing a smaller region may be smaller than an LCU, meaning smaller granular coding units are needed. This increases the implementation cost of encoding and decoding, and reduces processing performance. Second, the bitstream encoded from the smaller region is split into bitstream segments and then encapsulated into NALs. Since the bitstream segments are small and numerous, the number of encapsulation headers is also large, increasing processing and transmission latency. Third, while splitting the bitstream encoded from the smaller region into bitstream segments for transmission can save buffer size, the transmitted bitstream does not conform to the protocol, increasing the complexity of encoding and decoding implementation.

[0087] In yet another possible implementation, such as Figure 16 As shown, to meet the end-to-end low latency requirement for video image acquisition and display, the video image is divided into multiple tiles (e.g., tile 10C and tile 20C), each tile is divided into at least one slice, and each slice includes multiple rectangular blocks (e.g., LCUs). For example, multiple tiles are divided vertically, left and right, such as tile 10C on the left and tile 20C on the right. Left-right division is chosen instead of top-bottom division because image display is horizontally scanned, and left and right tiles can be processed and displayed simultaneously. If tiles are divided top-bottom, the upper tiles are displayed first, followed by the lower tiles. The first few rows of LCUs for the lower tiles have already been decoded, but because the upper tiles have not yet been fully displayed, the first few rows of LCUs for the lower tiles need to wait for display. This increases display latency. After encoding multiple slices within the multiple tiles, the transmitting device generates multiple bitstream groups, each bitstream group including at least one bitstream. Each tile corresponds to one bitstream group, and each slice corresponds to one bitstream. The code streams in different code stream groups are transmitted in a different order, instead of waiting for one code stream group to finish transmitting before transmitting the next. In other words, some code streams that were originally scheduled to be transmitted later are transmitted in advance, and the encoder of the receiving device can start decoding these code streams earlier, thereby reducing processing latency.

[0088] This application provides an image data transmission method. For example... Figure 16 and Figure 18 As shown, the method for dividing image data can be referenced. Figure 19 Image data may include multiple tiles. Each tile may include at least one slice. The multiple tiles may include a first tile and a second tile. The method may include:

[0089] S110: The transmitting device acquires multiple bitstream groups.

[0090] For example, each bitstream group includes at least one bitstream. One bitstream group corresponds to one tile, and one bitstream corresponds to one slice. Specifically, one bitstream is obtained by encoding one slice. A bitstream group includes bitstreams obtained by encoding all slices of one tile. Figure 20As shown, tile 10C includes slices 11C, 12C, and 13C. Slice 11C is encoded to obtain bitstream 11C, slice 12C is encoded to obtain bitstream 12C, and slice 13C is encoded to obtain bitstream 13C. Bitstream group 10C corresponds to tile 10C, and bitstream group 10C includes bitstreams 11C, 12C, and 13C. Tile 20C includes slices 21C, 22C, and 23C. Slice 21C is encoded to obtain bitstream 21C, slice 22C is encoded to obtain bitstream 22C, and slice 23C is encoded to obtain bitstream 23C. Bitstream group 20C corresponds to tile 20C, and bitstream group 20C includes bitstreams 21C, 22C, and 23C. Tile 30C includes slices 31C, 32C, and 33C. Slice 31C is encoded to obtain bitstream 31C, slice 32C is encoded to obtain bitstream 32C, and slice 33C is encoded to obtain bitstream 33C. Bitstream group 30C corresponds to tile 30C, and bitstream group 30C includes bitstream 31C, bitstream 32C, and bitstream 33C.

[0091] For example, the multiple bitstream groups include a first bitstream group and a second bitstream group. The first bitstream group includes bitstreams obtained by slicing and encoding a first tile, and the second bitstream group includes bitstreams obtained by slicing and encoding a second tile.

[0092] For example, the transmitting device can acquire image data. This image data can be acquired by the acquisition device within the transmitting device or received from other devices. The transmitting device divides the image data into multiple tiles, and each tile is divided into at least one slice. To ensure that multiple tiles can be encoded in parallel by multiple encoders and decoded in parallel by multiple decoders, dependency processing such as referencing and filtering between tiles can be disabled through syntax information. The transmitting device encodes multiple tiles one-to-one using multiple encoders; for example, one encoder is used to encode one tile. Each encoder can correspond to an encoding buffer. The bitstream group obtained from encoding each tile can be stored in the encoding buffer corresponding to the encoder that encoded that tile.

[0093] S120: The transmitting device transmits a bit stream to the receiving device. The bit stream includes multiple bit streams belonging to multiple bit stream groups, and in the bit stream, at least one bit stream belonging to a second bit stream group is included between two adjacent bit streams belonging to the first bit stream group.

[0094] For example, such as Figure 21 As shown, bitstream 1 is Figure 22 The bit streams transmitted in the illustrated implementation are bit stream 2 and bit stream 3. Figure 23 and Figure 24 The bit stream transmitted in the illustrated implementation.

[0095] In bitstream 1, the bitstreams within a bitstream group are continuous. For example, bitstreams 11C to 13C in bitstream group 10C are transmitted continuously, then bitstreams 21C to 23C in bitstream group 20C are transmitted continuously, and then bitstreams 31C to 33C in bitstream group 30C are transmitted continuously. When bitstream 21C is completed, the second decoder used to decode bitstream group 20C starts decoding. When bitstream 31C is completed, the third decoder used to decode bitstream group 30C starts decoding. Figure 25 As shown by the dashed circle, since the first line of LCU needs to be decoded before the output image frame can be started, the image frame is started to be output when the first line of LCU is decoded in the bitstream 31C (for example, during the transmission of the bitstream 32C).

[0096] In bitstream 2, the first bitstream group can be bitstream group 10C, and the second bitstream group can be bitstream group 20C. Two adjacent bitstreams in the first bitstream group can be bitstream 12C and bitstream 13C, and at least one bitstream in the second bitstream group can be bitstream 21C. Compared to bitstream 1, the transmission order of bitstream 21C is advanced, and the second decoder starts decoding earlier. Therefore, the processing latency of this implementation is reduced.

[0097] In bitstream 3, the first bitstream group can be bitstream group 20C, and the second bitstream group can be bitstream group 30C. Two adjacent bitstreams in the first bitstream group can be bitstream 22C and bitstream 23C, and at least one bitstream in the second bitstream group can be bitstream 31C. Compared to bitstream 1, the transmission order of bitstream 31C is advanced, the third decoder starts decoding earlier, and the image frames start output earlier. Therefore, the processing latency of this implementation is reduced.

[0098] S210: The receiving device receives the bit stream transmitted by the sending device.

[0099] S220: The receiving device acquires multiple bitstream groups based on the bitstream, and these multiple bitstream groups are used to obtain image data after decoding.

[0100] For example, the transmitting device sorts the multiple bitstreams of multiple bitstream groups according to a preset order to obtain a bitstream, and then transmits the bitstream. The receiving device can recover the order of each bitstream from the bitstream according to a preset order, thereby obtaining multiple bitstream groups. The receiving device can also decode multiple bitstream groups to obtain multiple slices, and place the slices at the corresponding image positions to obtain image data.

[0101] In this embodiment, at least one bitstream belonging to the second bitstream group is included between two adjacent bitstreams belonging to the first bitstream group. This means that a bitstream in either the first or second bitstream group is transmitted in advance, allowing the receiving device to decode it earlier. This breaks through the constraints of the encoding / decoding standard, allowing processing at a smaller granularity of slices, thus reducing decoding latency. Furthermore, if the bitstream belongs to a bitstream group that should have been transmitted last, it may also allow the image frame to be output earlier, further reducing the processing latency for image frame display. Additionally, slicing is done according to LCUs, eliminating the need to support even smaller granularity coding units. Encoding and decoding can be performed according to the protocol without increasing the implementation cost or reducing processing performance. Moreover, there is no need to split the bitstream into segments, resulting in fewer encapsulation headers and no increase in processing and transmission latency.

[0102] In one possible implementation, the multiple tiles further include a third tile. The multiple bitstream groups also include a third bitstream group. The third bitstream group includes bitstreams obtained by slicing and encoding the third tiles. In the bitstream, between two adjacent bitstreams belonging to the first bitstream group, there are M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group; M and N are both integers greater than or equal to 1. In this implementation, between two adjacent bitstreams belonging to the first bitstream group, there are M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group, meaning that at least one bitstream from the first, second, and third bitstream groups is transmitted in advance. The receiving device can decode at least one bitstream in advance, thus reducing processing latency. Furthermore, if the at least one bitstream belongs to a bitstream group that should have been transmitted last, then the image frame may be started output earlier, further reducing processing latency.

[0103] In some examples, M = N. For instance, after transmitting M streams from the current stream group, M streams from the next stream group are transmitted; after transmitting M streams from the last stream group, M streams from the first stream group are transmitted. For example, M = 1. Figure 26 The bitstream groups 10C, 20C, and 30C can be transmitted in the following order: bitstream 11C, bitstream 21C, bitstream 31C, bitstream 12C, bitstream 22C, bitstream 32C, bitstream 13C, bitstream 23C, and bitstream 33C. In this embodiment, M=N, the order of the bitstreams in the bitstream is relatively simple, and the process of the receiving device reconstructing the order of the bitstream groups based on the bitstream is also relatively simple.

[0104] In other examples, within a bitstream, the transmission order of each bitstream in multiple bitstream groups is determined according to the encoding time order of each bitstream. For instance, in a bitstream, the transmission order of each bitstream in multiple bitstream groups is determined from first to last according to the encoding time order of each bitstream. For example, forFigure 27 The encoding time order of each bitstream in bitstream group 10C, bitstream group 20C, and bitstream group 30C is: bitstream 11C, bitstream 21C, bitstream 22C, bitstream 31C, bitstream 23C, bitstream 12C, bitstream 13C, bitstream 32C, and bitstream 33C. Therefore, their transmission order is also: bitstream 11C, bitstream 21C, bitstream 22C, bitstream 31C, bitstream 23C, bitstream 12C, bitstream 13C, bitstream 32C, and bitstream 33C. In this embodiment, determining the transmission order of each bitstream based on its encoding time order reduces the waiting time for already encoded bitstreams to be transmitted. Therefore, processing latency can be reduced.

[0105] For example, the bitstream includes indication information that indicates the tile on which the slice corresponding to the bitstream is located. The receiving device can determine the tile on which the slice corresponding to the bitstream is located based on the indication information, and then place the slice at the image position corresponding to that tile. For example, each slice in multiple tiles includes multiple Maximum Coding Units (LCUs), and the indication information includes the LCU start address carried by the bitstream. Since the indication information typically carries the LCU start address, the LCU start address can be used as the indication information without adding any other information.

[0106] For example, the bitstream includes a NAL, and the indication information includes tile identification information in the NAL. For instance, the bitstream is encapsulated in a first NAL. In this case, the indication information can be information added to the existing NAL header of the first NAL. Another example is that the bitstream also includes a custom second NAL, and the indication information includes tile identification information in the custom second NAL. In this case, the indication information can be information added to the newly added custom NAL. In this embodiment, the transmission order of each bitstream is determined according to its encoding time sequence. Although the transmission process on the transmitting device side is relatively simple, the receiving device cannot determine which bitstream corresponds to which tile during decoding. By setting indication information in the bitstream, the receiving device can reconstruct the bitstream into multiple bitstream groups based on the indication information, placing bitstreams belonging to the same bitstream group into the corresponding decoding buffer. Then, the receiving device can decode the bitstream to obtain slices and place the slices at the image positions of the tiles corresponding to those slices. Furthermore, the form of the indication information is quite diverse. An existing LCU start address can be used as the indication information. Indication information can also be added to the NAL of the bitstream. By setting instruction information, the accuracy of image data transmission can be improved.

[0107] In some possible implementations, slices can be divided according to the size of the image region, or according to the number of LCUs.

[0108] In some examples, the size of each slice in a plurality of tiles is determined based on the number of LCUs in the image data. For example, each slice in a plurality of tiles includes P LCUs, where P is an integer greater than or equal to 1. In this implementation, the size of each slice is determined based on the number of LCUs, and may include similar or identical numbers of LCUs, with each slice corresponding to an image region of similar or identical size (e.g., all with 4 LCUs, or all with 1 LCU row). This simplifies the complexity of slice division and encoding / decoding.

[0109] For example, the number of LCUs (e.g., the number of Ps) included in each slice is determined based on the network transmission speed. The smaller the number of LCUs included in each slice, the smaller the granularity of the slice, the lower the transmission latency, and the lower the requirement for network transmission speed. In this embodiment, considering the size of the slice division based on the network transmission speed can reduce transmission latency.

[0110] When dividing the image into slices based on the region size, the transmission order of the bitstreams can be as follows: after transmitting one bitstream from the current bitstream group, transmit one bitstream from the next bitstream group; after transmitting one bitstream from the last bitstream group, transmit one bitstream from the first bitstream group. The processing procedure of the transmitting device will now be illustrated using this implementation as an example.

[0111] like Figure 28 As shown, tile 10C is divided into slices 11C, 12C, ..., 1nC. Tile 20C is divided into slices 21C, 22C, ..., 2nC. Each slice has the same area size, but the size of the encoded bitstream bytes may differ. The first encoder corresponding to tile 10C inputs slices 11C to 1nC and outputs the encoded bitstreams 11C to 1nC to the first encoding buffer. The second encoder corresponding to tile 20C inputs slices 21C to 2nC and outputs the encoded bitstreams 21C to 2nC to the second encoding buffer. The first processing circuit inputs bitstreams 11C to 1nC and bitstreams 21C to 2nC from the first and second encoding buffers. The first processing circuit sorts these bitstreams according to a preset order to obtain the bitstreams: bitstream 11C, bitstream 21C, bitstream 12C, bitstream 22C, ..., bitstream 1nC, bitstream 2nC.

[0112] like Figure 28As shown, the first processing circuit sorts the bitstreams according to a preset order. The process of obtaining the bitstream may include the following steps: The first processing circuit traverses the bitstreams in Y encoding buffers (e.g., the first encoding buffer and the second encoding buffer). When there is a bitstream in the first encoding buffer, it retrieves one bitstream from that encoding buffer (e.g., bitstream 11C). When there is a bitstream in the next encoding buffer, the first processing circuit retrieves one bitstream from that encoding buffer. After traversing the Y-th encoding buffer and retrieving one bitstream from the Y-th encoding buffer, if there are still unretrieved bitstreams in the first encoding buffer, the circuit traverses the bitstreams in the Y encoding buffers again. When there is a bitstream in the first encoding buffer, it retrieves one bitstream from that encoding buffer, and repeats the subsequent steps. This continues until the Y-th encoding buffer has been traversed and one bitstream from the Y-th encoding buffer has been retrieved, and there are no unretrieved bitstreams in the first encoding buffer. At this point, the traversal is complete, and the first processing circuit obtains the bitstream based on the multiple bitstreams retrieved according to the preset order.

[0113] The following example illustrates the processing procedure of the receiving device.

[0114] like Figure 28 As shown, the receiving device receives bitstreams: bitstream 11C, bitstream 21C, bitstream 12C, bitstream 22C, ..., bitstream 1nC, bitstream 2nC. The second processing circuit reconstructs the bitstreams into bitstream groups 10C and 20C according to a preset order. The second processing circuit outputs bitstream group 10C to the first decoding buffer corresponding to tile 10C and bitstream group 20C to the second decoding buffer corresponding to tile 20C. The first decoder inputs bitstream group 10C from the first decoding buffer and decodes it to obtain slices 11C to 1nC. The receiving device places slices 11C to 1nC into their corresponding image positions. The second decoder inputs bitstream group 20C from the second decoding buffer and decodes it to obtain slices 21C to 2nC. The receiving device places slices 21C to 2nC into their corresponding image positions.

[0115] like Figure 28 As shown, the process by which the second processing circuit restores the bitstream into bitstream groups according to a preset order can include the following steps: The second processing circuit traverses the bitstreams in the bitstream, writing the current first bitstream into the first decoding buffer. The next bitstream is written into the next decoding buffer. After traversing to the Y-th bitstream and writing it into the Y-th decoding buffer, if there are still unwritten bitstreams in the bitstream, the circuit traverses the bitstreams again. The current first bitstream (e.g., the (Y+1)-th bitstream) is written into the first decoding buffer, and the subsequent steps are repeated. This continues until there are no unwritten bitstreams in the bitstream. At this point, the traversal is complete, and each bitstream group is written into its corresponding decoding buffer.

[0116] In some other possible implementations, slices can be divided based on the size of the bitstream bytes after slice encoding.

[0117] In some examples, the bitstream size corresponding to each slice in multiple tiles is the same. Alternatively, if it is not possible to achieve the same bitstream size for each slice, the bitstream size for each slice is made as close as possible. In this implementation, having the same bitstream size after encoding each slice allows for more stable processing latency in slice encoding, transmission, and decoding, thus improving network transmission efficiency.

[0118] For example, the byte size of the bitstream is determined based on the network transmission unit (NTU) size. The NTU size refers to the size of data transmitted per transmission. If the bitstream size after slice encoding is small, and the NTU size is large, there will be more empty data in each transmission, wasting network transmission speed. If the bitstream size after slice encoding is large, and the NTU size is small, a bitstream needs to be transmitted multiple times, making the transmission process more complex. Taking an Ethernet NTU size of 1500 bytes per data packet as an example, the bitstream size after slice encoding can be slightly less than 1500 bytes. In this implementation, considering the slice size based on the NTU size can reduce transmission latency and simplify the transmission process.

[0119] When dividing the bitstream into slices based on the size of the encoded bitstream bytes, the transmission order of the bitstream can be determined according to the encoding time order of the bitstream. The following explanation uses this implementation as an example to illustrate the processing procedure of the transmitting device.

[0120] like Figure 28 As shown, tile 10C is divided into slices 11C, 12C, 13C, ..., 1mC. Tile 20C is divided into slices 21C, 22C, 23C, ..., 2mC. The bitstream size after encoding each slice is the same, but the area size of each slice may be different. The first encoder corresponding to tile 10C inputs slices 11C to 1mC and outputs the encoded bitstreams 11C to 1mC to the first encoding buffer. The second encoder corresponding to tile 20C inputs slices 21C to 2mC and outputs the encoded bitstreams 21C to 2mC to the second encoding buffer. The first processing circuit inputs bitstreams 11C to 1mC and bitstreams 21C to 2mC from the first encoding buffer and the second encoding buffer. The first processing circuit sorts these bit streams in a preset order to obtain the bit streams: bit stream 11C, bit stream 21C, bit stream 22C, bit stream 12C, bit stream 13C, bit stream 23C, ..., bit stream 2mC, bit stream 1mC.

[0121] like Figure 28As shown, the first processing circuit sorts the bitstreams according to a preset order to obtain the bitstream. The processing steps include the following: The first processing circuit continuously traverses the bitstreams in Y encoding buffers. The bitstreams are obtained according to their encoding times in the encoding buffers. The bitstreams are sorted according to their encoding times to obtain the bitstream.

[0122] The following example illustrates the processing procedure of the receiving device.

[0123] like Figure 14 As shown, the receiving device receives the following bitstreams: bitstream 11C, bitstream 21C, bitstream 22C, bitstream 12C, bitstream 13C, bitstream 23C, ..., bitstream 2mC, bitstream 1mC. The second processing circuit, based on the indication information in the bitstreams, reconstructs the bitstreams into bitstream groups 10C and 20C. The second processing circuit outputs bitstream group 10C to the first decoding buffer corresponding to tile 10C and bitstream group 20C to the second decoding buffer corresponding to tile 20C. The first decoder inputs bitstream group 10C from the first decoding buffer and decodes it to obtain slices 11C to 1mC. The receiving device places slices 11C to 1mC into their corresponding image positions. The second decoder inputs bitstream group 20C from the second decoding buffer and decodes it to obtain slices 21C to 2mC. The receiving device places slices 21C to 2mC into their corresponding image positions.

[0124] like Figure 15 As shown, the process by which the second processing circuit restores the bitstream into a bitstream group based on the indication information in the bitstream may include the following steps: The second processing circuit traverses the bitstreams in the bitstream. Based on the indication information, it determines the bitstream group corresponding to each bitstream and inputs the bitstream into the decoding buffer corresponding to its bitstream group. After traversal, each bitstream group is written into its corresponding decoding buffer.

[0125] In some possible implementations, tiles can be divided based on the number and performance of the encoders.

[0126] In some examples, the number of tiles is equal to the number of encoders. In this implementation, the number of tiles equals the number of encoders, allowing one encoder to encode a slice of one tile, thus enabling parallel processing of multiple tiles.

[0127] For example, the size of the tile is determined based on the encoding speed of the encoder that encodes the tile. The encoding speeds of multiple encoders can be the same or different, so the division of multiple tiles can be uniform or non-uniform.

[0128] like Figures 6 to 9As shown, the first encoder used to encode tile 10C and the second encoder used to encode tile 20C have the same encoding speed. Therefore, the division of tile 10C and tile 20C can be uniform. If the image can be equally divided, the sizes of tile 10C and tile 20C can be exactly the same. For example, the image may include 4*6 LCUs, tile 10C may include 4*3 LCUs, and tile 20C may include 4*3 LCUs. If the image cannot be equally divided, the sizes of tile 10C and tile 20C can not be exactly the same, but should be as close as possible. For example, the image may include 4*5 LCUs, tile 10C may include 4*2 LCUs, and tile 20C may include 4*3 LCUs.

[0129] like Figure 29 As shown, the encoding speed of the first encoder used to encode tile 10C is twice that of the second encoder used to encode tile 20C. Therefore, the size of tile 10C is ideally twice the size of tile 20C. If the image can be divided into three equal parts, then the size of tile 10C is twice the size of tile 20C. For example, the image may include 4*6 LCUs, tile 10C includes 4*4 LCUs, and tile 20C includes 4*2 LCUs. If the image cannot be divided into three equal parts, then the size of tile 10C is approximately twice the size of tile 20C. For example, the image may include 4*5 LCUs, tile 10C includes 4*3 LCUs, and tile 20C includes 4*2 LCUs.

[0130] In this implementation, the size of the tile is determined based on the encoding speed of the encoder that encodes the tile. This ensures that the performance and latency of each tile encoding are the same or similar, resulting in optimal overall image encoding performance and latency.

[0131] In some examples, the number of decoders in the receiving device may be the same as or different from the number of encoders in the transmitting device. The decoding speed of the decoders may be the same as or different from the encoding speed of the encoders.

[0132] like Figures 6 to 9 As shown in (a), the encoding speed of both the first encoder and the second encoder is X, and the sizes of tile 10C and tile 20C are the same. The receiving device includes a first decoder corresponding to tile 10C and a second decoder corresponding to tile 20C. The encoding speed of both the first decoder and the second decoder is X. The overall performance of image data transmission is optimized, with minimal latency.

[0133] like Figure 29As shown in (b), the encoding speed of both the first and second encoders is X, and the sizes of tile 10C and tile 20C are the same. The encoding speed of the first decoder is 1.5X, and the encoding speed of the second decoder is 0.5X. The overall performance of image data transmission is not optimal, and the latency is not minimum.

[0134] like Figure 14 As shown in (c), the encoding speed of both the first and second encoders is X, and the size of tile 10C and tile 20C are the same. The encoding speed of the first decoder is 2X, and the encoding speed of the second decoder is X. The second decoder works at full capacity, while the first decoder has some performance headroom. The overall performance of image data transmission is optimized, with minimal latency.

[0135] like Figure 15 As shown in (d), the encoding speed of both the first and second encoders is X, and the sizes of tile 10C and tile 20C are the same. The receiving device also includes a third decoder. The encoding speed of the first decoder is X, the encoding speed of the second decoder is X, and the encoding speed of the third decoder is X. The third decoder is idle and does not work, achieving optimal overall performance of image data transmission with minimal latency.

[0136] like Figure 29 As shown in (e), the encoding speed of the first encoder is 2X, the encoding speed of the second encoder is X, and the size of tile 10C is twice the size of tile 20C. The encoding speed of the first decoder is 2X, and the encoding speed of the second decoder is X. The overall performance of image data transmission is optimized, with minimal latency.

[0137] like Figure 30 As shown in (f), the encoding speed of the first encoder is 2X, the encoding speed of the second encoder is X, and the size of tile 10C is twice the size of tile 20C. The encoding speed of the first decoder is 1.5X, and the encoding speed of the second decoder is 1.5X. The overall performance of image data transmission is not optimal, and the latency is not minimum.

[0138] The following is a detailed analysis. Figure 31 and Figure 32 The implementation shown is compared to Figure 33 The illustrated implementation reduces the causes of processing latency. For example, the transmitting device includes two encoders, the receiving device includes two decoders, the maximum coding unit is 64, the image frame is divided into two tiles, each tile is divided into 16 slices, and each slice contains two rows of maximum coding units.

[0139] like Figures 31 to 33 As shown in (a), in ​In the illustrated implementation, slice 11C and slice 21C can be encoded simultaneously (or at approximately the same time), ..., slice 1nC and slice 2nC can be encoded simultaneously (or at approximately the same time). For example, each slice encoding takes 1 millisecond, and each tile encoding takes 16 milliseconds. After slices 11C to 1nC are transmitted, slices 21C to 2nC begin transmission. After slices 11C to 1nC are decoded, slice 21C begins decoding, and the image frame output begins. Therefore, the end-to-end delay for encoding, transmission, and decoding is greater than 16 milliseconds.

[0140] like ​ As shown in (b), in ​ and ​ In the illustrated implementation, slice 11C and slice 21C can be encoded simultaneously (or at approximately the same time), ..., slice 1nC and slice 2nC can be encoded simultaneously (or at approximately the same time). After slice 11C is transmitted, slice 21C is transmitted, ..., slice 1nC is transmitted, and then slice 2nC is transmitted. After slice 21C is transmitted, slice 21C begins decoding. After decoding one LCU line, the image frame starts output. It can be seen that the decoding time of slice 21C, from slice 11C to slice 1nC and the completion of slice 21C transmission, is advanced to the completion of slice 11C and slice 21C transmission. The time for image frame output, from slice 11C to slice 1nC and the completion of the first LCU line decoding of slice 21C, is advanced to the completion of slice 11C and the first LCU line decoding of slice 21C. Therefore, the encoding, transmission, and decoding delays are all less than 1 millisecond, and the end-to-end delay is less than 3 milliseconds, which is much smaller than ... ​ The 16 milliseconds shown in (a) indicates that increasing the number of slices and reducing the interaction granularity will result in lower end-to-end latency and a better actual business experience.

[0141] This application also provides an image data transmission device, wherein the image data includes multiple tiles; each tile includes at least one slice; the multiple tiles include a first tile and a second tile. ​ As shown, the first image data transmission device includes a coupled first acquisition unit and a transmission unit. The first acquisition unit is used to acquire multiple bitstream groups. The multiple bitstream groups include a first bitstream group and a second bitstream group. The first bitstream group includes a bitstream obtained by slicing and encoding a first tile, and the second bitstream group includes a bitstream obtained by slicing and encoding a second tile. The transmission unit is used to transmit a bitstream input from the first acquisition unit to a receiving device; the bitstream includes multiple bitstreams belonging to the multiple bitstream groups, and in the bitstream, at least one bitstream belonging to the second bitstream group is included between two adjacent bitstreams belonging to the first bitstream group.

[0142] This application also provides an image data transmission device, wherein the image data includes multiple tiles; each tile includes at least one slice; the multiple tiles include a first tile and a second tile. ​ As shown, the second image data transmission device includes a coupled receiving unit and a second acquiring unit. The receiving unit receives a bitstream transmitted by a transmitting device. The bitstream includes multiple bitstreams belonging to multiple bitstream groups; the multiple bitstream groups include a first bitstream group and a second bitstream group, the first bitstream group including bitstreams obtained by slicing first tiles, and the second bitstream group including bitstreams obtained by slicing second tiles; in the bitstream, at least one bitstream belonging to the second bitstream group is included between two adjacent bitstreams belonging to the first bitstream group. The second acquiring unit acquires the multiple bitstream groups based on the bitstream input from the receiving unit, and the multiple bitstream groups are used to obtain image data after decoding.

[0143] This application also provides an image data transmission device, wherein the image data includes multiple tiles; each tile includes at least one slice; the multiple tiles include a first tile and a second tile. ​ As shown, the third image data transmission device includes a coupled third processing circuit and a third transceiver circuit. The third processing circuit is used to acquire multiple bitstream groups; the multiple bitstream groups include a first bitstream group and a second bitstream group, the first bitstream group including bitstreams obtained by slicing and encoding first tiles, and the second bitstream group including bitstreams obtained by slicing and encoding second tiles. The third transceiver circuit is used to transmit a bitstream input from the third processing circuit to a receiving device. The bitstream includes multiple bitstreams belonging to the multiple bitstream groups, and in the bitstream, at least one bitstream belonging to the second bitstream group is included between two adjacent bitstreams belonging to the first bitstream group.

[0144] This application also provides an image data transmission device, wherein the image data includes multiple tiles; each tile includes at least one slice; the multiple tiles include a first tile and a second tile. ​ As shown, the fourth image data transmission device includes a coupled fourth processing circuit and a fourth transceiver circuit. The fourth transceiver circuit is used to receive a bit stream transmitted by a transmitting device. The bit stream includes multiple bit streams belonging to multiple bit stream groups; the multiple bit stream groups include a first bit stream group and a second bit stream group, the first bit stream group including bit streams obtained by slicing and encoding first tiles, and the second bit stream group including bit streams obtained by slicing and encoding second tiles; in the bit stream, at least one bit stream belonging to the second bit stream group is included between two adjacent bit streams belonging to the first bit stream group. The fourth processing circuit is used to acquire the multiple bit stream groups based on the bit stream, and the multiple bit stream groups are used to obtain image data after decoding.

[0145] Based on the above ​ The image data transmission device shown may also include the following embodiments.

[0146] In one possible implementation, the plurality of tiles further includes a third tile, and the plurality of bitstream groups further includes a third bitstream group; the third bitstream group includes bitstreams obtained by slicing the third tiles;

[0147] In a bitstream, between two adjacent bitstreams belonging to the first bitstream group, there are M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group; M and N are both integers greater than or equal to 1.

[0148] In one possible implementation, M = N.

[0149] In one possible implementation, in a bitstream, the transmission order of each bitstream in a plurality of bitstream groups is determined according to the encoding time order of each bitstream.

[0150] In one possible implementation, the bitstream includes indication information that indicates the tile on which the slice corresponding to the bitstream is located.

[0151] In one possible implementation, each slice in the plurality of tiles includes a plurality of maximum coding units (LCUs); the indication information includes the LCU start address carried in the bitstream; or,

[0152] The bitstream includes the Network Abstraction Layer (NAL), and the indication information includes tile identification information in the NAL.

[0153] In one possible implementation, the bitstream is encapsulated in NAL.

[0154] In one possible implementation, each slice in the plurality of tiles includes P maximum coding units (LCUs); P is an integer greater than or equal to 1.

[0155] In one possible implementation, the number of P is determined based on the network transmission speed.

[0156] In one possible implementation, the bitstream corresponding to each slice in the multiple tiles has the same byte size.

[0157] In one possible implementation, the byte size of the bitstream is determined based on the network transmission unit size.

[0158] In one possible implementation, the method is applied to a transmitting device that includes multiple encoders for encoding slices of tiles to obtain a bitstream; the number of tiles is equal to the number of encoders.

[0159] In one possible implementation, each encoder is used to encode a slice of a tile; the size of the tile is determined according to the encoding speed of the encoder that encodes the tile.

[0160] It is understood that each component of the above-mentioned image data transmission device can be used to implement the corresponding steps in the foregoing method embodiments. Since each step has been described in detail in the foregoing method embodiments, it will not be repeated here.

[0161] This application also provides a computer-readable storage medium storing program code. When the medium is run on a device (e.g., a microcontroller, chip, computer, or processor), the program code can be invoked by the processor to execute one or more steps in the above method embodiments.

[0162] Based on this understanding, this application also provides a computer program product containing instructions. The technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or its processor to execute all or part of the steps of the methods described in the various embodiments of this application.

[0163] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0164] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0167] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.

[0169] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0170] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image data transmission method, characterized by, The image data comprises a plurality of tiles; each of the tiles comprises at least one slice; the plurality of tiles comprises a first tile and a second tile; the method comprises: obtaining a plurality of bitstream groups; the plurality of bitstream groups comprises a first bitstream group and a second bitstream group; the first bitstream group comprises bitstreams obtained by encoding slices of the first tile; the second bitstream group comprises bitstreams obtained by encoding slices of the second tile; transmitting a bitstream to a receiving device; the bitstream comprises a plurality of bitstreams belonging to the plurality of bitstream groups; in the bitstream, between two adjacent bitstreams belonging to the first bitstream group, at least one bitstream belonging to the second bitstream group is included.

2. The method of claim 1, wherein, The plurality of tiles further comprises a third tile, and the plurality of bitstream groups further comprises a third bitstream group; the third bitstream group comprises bitstreams obtained by encoding slices of the third tile; In the bitstream, between two adjacent bitstreams belonging to the first bitstream group, M bitstreams belonging to the second bitstream group and N bitstreams belonging to the third bitstream group are included; M and N are both integers greater than or equal to 1.

3. The method of claim 2, wherein, M = N.

4. The method according to claim 1 or 2, characterized in that, In the bitstream, the transmission order of each bitstream of the plurality of bitstream groups in the bitstream is determined according to the encoding time order of the each bitstream.

5. The method according to any one of claims 1 to 4, characterized in that, The bitstream comprises indication information, and the indication information is used to indicate a tile in which a slice corresponding to the bitstream is located.

6. The method of claim 5, wherein Each slice in the plurality of tiles comprises a plurality of largest coding units (LCUs); the indication information comprises an LCU start address carried by the bitstream; or The bitstream comprises a network abstraction layer (NAL), and the indication information comprises tile identification information in the NAL.

7. The method of claim 6, wherein, The bitstream is encapsulated in the NAL.

8. The method according to any one of claims 1 to 7, characterized in that, Each slice in the plurality of tiles comprises P largest coding units (LCUs); P is an integer greater than or equal to 1.

9. The method of claim 8, wherein, The number of P is determined according to a network transmission speed.

10. The method according to any one of claims 1 to 7, characterized in that, The bitstream corresponding to each slice in the plurality of tiles has a same byte size.

11. The method of claim 10, wherein, The byte size of the bitstream is determined according to a network transmission unit size.

12. The method according to any one of claims 1 to 11, characterized in that, The method is applied to a sending device, and the sending device comprises a plurality of encoders configured to encode slices of the tiles to obtain the bitstreams; the number of the tiles is equal to the number of the encoders.

13. The method of claim 12, wherein, Each encoder is configured to encode slices of one tile; and the size of the tile is determined according to an encoding speed of the encoder encoding the tile.

14. An image data transmission method characterized by, The image data comprises a plurality of tiles; each of the tiles comprises at least one slice; the plurality of tiles comprises a first tile and a second tile; the method comprises: receiving a bitstream transmitted by a sending device; the bitstream comprises a plurality of bitstreams belonging to a plurality of bitstream groups; the plurality of bitstream groups comprises a first bitstream group and a second bitstream group; the first bitstream group comprises bitstreams obtained by encoding slices of the first tile; the second bitstream group comprises bitstreams obtained by encoding slices of the second tile; in the bitstream, between two adjacent bitstreams belonging to the first bitstream group, at least one bitstream belonging to the second bitstream group is included. The plurality of code stream groups are obtained from the bitstream, and the plurality of code stream groups are used to obtain the image data after decoding.

15. The method of claim 14, wherein, The plurality of tiles further include a third tile, and the plurality of code stream groups further include a third code stream group; the third code stream group includes code stream obtained by slice encoding of the third tile. In the bitstream, between two adjacent code streams belonging to the first code stream group, M code streams belonging to the second code stream group and N code streams belonging to the third code stream group are included; M and N are both integers greater than or equal to 1.

16. The method of claim 15, wherein, M = N.

17. The method of claim 14 or 15, wherein, In the bitstream, the transmission order of each code stream in the plurality of code stream groups is determined according to the encoding time order of the each code stream.

18. The method according to any one of claims 14-17, characterized by, The bitstream includes indication information, and the indication information is used to indicate a tile in which a slice corresponding to the code stream is located.

19. The method of claim 18, wherein, Each slice in the plurality of tiles includes a plurality of largest coding units (LCUs), and the indication information includes an LCU start address carried by the code stream; or The bitstream includes a network abstraction layer (NAL), and the indication information includes tile identification information in the NAL.

20. The method of claim 19, wherein, The code stream is encapsulated in the NAL.

21. The method according to any one of claims 14-20, characterized by, Each slice in the plurality of tiles includes P largest coding units (LCUs), and P is an integer greater than or equal to 1.

22. The method of claim 21, wherein, The number of P is determined according to a network transmission speed.

23. The method according to any one of claims 14-20, characterized by, The byte size of the code stream corresponding to each slice in the plurality of tiles is the same.

24. The method of claim 23, wherein, The byte size of the code stream is determined according to a network transmission unit size.

25. An image data transmission apparatus characterized by comprising: The apparatus includes a module for performing the method of any one of claims 1-24.

26. An image data transmission apparatus characterized by comprising: The apparatus includes processing circuitry and transceiver circuitry, and the processing circuitry and the transceiver circuitry are used to implement the method of any one of claims 1-24.

27. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code, and the program code can be invoked by a processor to execute the method of any one of claims 1-24.

28. A computer program product, characterised in that, When the computer program product runs on a computer, the computer is caused to execute the method of any one of claims 1-24.

Images