Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to suppress an increase in the period in which the image quality of a decoded image decreases due to an error that has occurred on the receiving end during transmission of encoded data of a moving image. Error information transmitted from a receiving device, which receives encoded data of a moving image transmitted via a first wireless communication path, via a second wireless communication path capable of transmission with less delay than in the first wireless communication path is acquired, and encoding of the moving image is controlled on the basis of the acquired error information. The present disclosure can be applied to, for example, an information processing device, an encoding device, a decoding device, an electronic apparatus, an information processing method, or a program.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress an increase in the period during which the image quality of a decoded image is reduced due to an error occurring on the receiving side when transmitting encoded data of a moving image. [Background technology]

[0002] In recent years, the Third Generation Partnership Project (3GPP) has been studying and creating specifications for a fifth-generation mobile communication system (hereinafter also referred to as 5G), which is a wireless communication system that satisfies the International Mobile Telecommunications (IMT)-2020 standard established by the International Telecommunication Union (see, for example, Non-Patent Document 1).

[0003] 5G defines use cases according to the intended use, such as eMBB (enhanced Mobile Broadband) which enables large-capacity data transmission, and URLLC (Ultra Reliable Low Latency Communication) which enables highly reliable and low-latency data transmission. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "TR 21.916 V1.0.0 (2020-12)", 3rd Generation Partnership Project;Technical Specification Group Services and System Aspects;Release 16 Description;Summary of Rel-16 Work Items (Release 16) Summary of the Invention [Problem to be solved by the invention]

[0005] However, the latency requirements for each use case are different. For example, in the case of the high-capacity use case (eMBB), the latency requirement for the wireless section is 4 ms. In contrast, in the case of the low-latency use case (URLLC), the latency requirement for the wireless section is 0.5 ms.

[0006] Therefore, when considering a high-capacity use case (eMBB) for wireless networks to transmit high-quality video, there is a risk that network delays will result in long error recovery times.

[0007] The present disclosure has been made in consideration of such circumstances, and is intended to suppress an increase in the period during which the image quality of a decoded image is reduced due to the occurrence of an error on the receiving side when transmitting encoded data of a moving image. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present technology includes an error information acquisition unit that acquires, from a receiving device that receives encoded data of a moving image transmitted via a first wireless communication channel, error information transmitted via a second wireless communication channel that allows transmission with lower delay than the first wireless communication channel, and an encoding control unit that controls encoding of the moving image based on the error information acquired by the error information acquisition unit, wherein, when encoding each frame of the moving image, a part of the frame is set as an intra region and the position of the intra region is moved in a predetermined direction for each frame so as to rotate every predetermined number of frames, and the encoding control unit, when the error information is acquired by the error information acquisition unit, In the above tour This is an information processing device that returns the device to its initial position.

[0009] According to one aspect of the present technology, there is provided an information processing method for encoding data of a moving image, the data being encoded such that, when encoding each frame of a moving image, a part of the frame is set as an intra region and intra-coded, and the position of the intra region is moved in a predetermined direction for each frame so as to rotate every predetermined number of frames, and the data being encoded such that, when error information transmitted via a second wireless communication channel that can transmit with lower delay than the first wireless communication channel is acquired from a receiving device that receives the encoded data transmitted via a first wireless communication channel, the position of the intra region is moved in a predetermined direction for each frame. In the above tour This is an information processing method for returning to the initial position.

[0012] In an information processing device and method according to one aspect of the present technology, when encoding each frame of a moving image, a part of the frame is set as an intra region and intra-coded, and the position of the intra region is coded so as to move in a predetermined direction for each frame so as to rotate every predetermined number of frames, and when error information transmitted via a second wireless communication channel that can transmit with lower delay than the first wireless communication channel is acquired from a receiving device that receives the coded data transmitted via a first wireless communication channel, the position of the intra region is During that tour It is returned to the initial position. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of an image transmission system. [Figure 2] FIG. 10 is a diagram illustrating an example of a delay when dealing with an error. [Figure 3] FIG. 1 is a diagram illustrating an example of a main configuration of an image transmission system. [Figure 4] FIG. 1 is a block diagram illustrating an example of the main configuration of an image encoding device. [Figure 5] FIG. 2 is a block diagram illustrating an example of the main configuration of an encoding unit. [Figure 6] FIG. 1 is a block diagram illustrating an example of the main configuration of an image decoding device. [Figure 7] FIG. 2 is a block diagram illustrating an example of the main configuration of a decoding unit. [Figure 8] 10 is a flowchart illustrating an example of the flow of an image encoding process. [Figure 9] 10 is a flowchart illustrating an example of the flow of an image decoding process. [Figure 10] FIG. 10 is a diagram illustrating an example of a delay when dealing with an error. [Figure 11] FIG. 1 is a diagram illustrating an example of video encoding. [Figure 12] FIG. 10 is a diagram illustrating an example of intrastripe. [Figure 13] FIG. 10 is a diagram illustrating an example of code amount. [Figure 14] FIG. 10 is a diagram illustrating an example of encoding control. [Figure 15] 10 is a flowchart illustrating an example of the flow of an encoding control process. [Figure 16] FIG. 10 is a diagram illustrating an example of encoding control. [Figure 17] 10 is a flowchart illustrating an example of the flow of an encoding control process. [Figure 18] FIG. 1 is a diagram illustrating an example of a main configuration of an image transmission system. [Figure 19] FIG. 1 is a diagram illustrating an example of a main configuration of an image transmission system. [Figure 20] FIG. 1 is a diagram illustrating an example of a main configuration of an image transmission system. [Figure 21] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Delays when responding to errors 2. First embodiment (image transmission system) 3. Second embodiment (encoding control 1) 4. Third embodiment (encoding control 2) 5. Fourth embodiment (another example of an image transmission system) 6. Supplementary Notes

[0016] <1. Delays when responding to errors> <References supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents and patent documents that were publicly known at the time of filing.

[0017] Non-patent document 1: (mentioned above) Non-patent document 2: Recommendation ITU-T H.264 (04 / 2017) "Advanced video coding for generic audiovisual services", April 2017 Non-patent document 3: Recommendation ITU-T H.265 (02 / 18) "High efficiency video coding", February 2018 Non-patent document 4: Benjamin Bross, Jianle Chen, Shan Liu, Ye-Kui Wang, "Versatile Video Coding (Draft 7)", JVET-P2001-vE, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 16th Meeting: Geneva, CH, 1-11 Oct 2019 Non-patent document 5: Satoshi Nagata, Kazuaki Takeda, Daisuke Umeda, Hideaki Takahashi, Kenichiro Aoyagi, "3GPP Release 15 Standardization Technical Overview", https: / / www.nttdocomo.co.jp / binary / pdf / corporate / technology / rd / technical_journal / bn / vol26_3 / vol26_3_007jp.pdf Patent Document 1: Japanese Patent Application Laid-Open No. 2010-062946

[0018] In other words, the contents of the above-mentioned non-patent documents and patent documents, as well as the contents of other documents referenced in the above-mentioned non-patent documents and patent documents, are also used as the basis for determining support requirements.

[0019] In other words, the contents of the above-mentioned non-patent documents and patent documents also serve as the basis for determining the support requirements. For example, even if the Quad-Tree Block Structure and QTBT (Quad Tree Plus Binary Tree) Block Structure described in the above-mentioned non-patent documents are not directly described in the examples, they are considered to be within the scope of the disclosure of the present technology and meet the support requirements of the claims. Similarly, even if technical terms such as parsing, syntax, and semantics are not directly described in the examples, they are considered to be within the scope of the disclosure of the present technology and meet the support requirements of the claims.

[0020] Furthermore, in this specification, a "block" (not a block indicating a processing unit) used in the description as a partial region or processing unit of an image (picture) refers to any partial region within a picture, and its size, shape, characteristics, etc. are not limited unless otherwise specified. For example, a "block" includes any partial region (processing unit) such as a TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU (Coding Unit), LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), sub-block, macroblock, tile, or slice, as described in the above-mentioned non-patent document.

[0021] Furthermore, when specifying such block sizes, the block sizes may be specified not only directly but also indirectly. For example, the block sizes may be specified using identification information for identifying the sizes. Furthermore, for example, the block sizes may be specified by the ratio or difference with respect to the size of a reference block (e.g., LCU, SCU, etc.). For example, when transmitting information specifying the block size as a syntax element, the information indirectly specifying the size as described above may be used as the information. This may reduce the amount of information and improve coding efficiency. Furthermore, the specification of the block sizes may also include specification of a range of block sizes (e.g., specification of a range of allowable block sizes, etc.).

[0022] <Delays in error handling in image transmission systems> Conventionally, various systems have been developed as image transmission systems for transmitting image data. For example, systems have been developed that transmit moving images using wireless communication. Generally, image data such as moving images has a large data size, so it has been considered to encode (compress) it before transmission.

[0023] For example, an image transmission system 10 shown in Fig. 1 has an encoder 11 on the transmitting side (i.e., the transmission source side) and a decoder 12 on the receiving side (i.e., the transmission destination side). Image data is encoded by the encoder 11. Then, the encoded data (bit stream) is transmitted to the decoder 12 via a wireless network 21. The bit stream is decoded by the decoder 12 and output as image data (decoded image).

[0024] In such an image transmission system 10, it is conceivable that an error may occur during reception or decoding of the bitstream. In such a case, the decoder 12 will be unable to obtain the decoded image. If the image data to be transmitted is a moving image and the frames following the frame in which the error occurred are inter-coded, the error may propagate to the following frames, and the situation may persist in which it will be impossible to obtain decoded images for the following frames.

[0025] Therefore, it has been considered to control the transmission of the bit stream (i.e., the encoding of the image data) in response to the occurrence of an error on the receiving side. For example, if the decoder 12 fails to receive or decode the data, it transmits error information indicating the error to the encoder 11 via the wireless network 21. Upon receiving the error information, the encoder 11 performs encoding so that the error does not propagate to subsequent frames.

[0026] By doing so, the decoder 12 can obtain the decoded image earlier.

[0027] In recent years, as shown in Non-Patent Document 1, for example, the Third Generation Partnership Project (3GPP) has been studying and creating specifications for a fifth-generation mobile communication system (hereinafter also referred to as 5G), which is a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union.

[0028] 5G defines use cases according to applications. For example, a use case enabling large-capacity data transmission (eMBB (enhance Mobile Broadband)) and a use case enabling highly reliable and low-latency data transmission (URLLC (Ultra Reliable Low Latency Communication)) have been defined. For example, assuming a high-capacity use case (eMBB) as a wireless network, high-quality video images can be transmitted. For example, in the case of an image transmission system 10 as shown in the example of FIG. 1, the high-capacity use case (eMBB) is applied to the wireless network 21. In this case, not only the transmission of a video bitstream from the encoder 11 to the decoder 12 but also the transmission of error information from the decoder 12 to the encoder 11 is performed via the wireless network 21 of this high-capacity use case (eMBB).

[0029] However, the latency requirements for each use case are different. For example, in the case of the high-capacity use case (eMBB), the latency requirement for the wireless section is 4 ms. In contrast, in the case of the low-latency use case (URLLC), the latency requirement for the wireless section is 0.5 ms.

[0030] Therefore, when the wireless network 21 of the image transmission system 10 in Fig. 1 is applied with the high-capacity use case (eMBB) for transmitting high-quality video as described above, the network delay for transmitting error information, which has a smaller data volume than the video bitstream, may be larger than that of the low-latency use case (URLLC). In other words, there is a risk of delaying the timing of encoding control based on the error information. If this delay in encoding control timing occurs, there is a risk of increasing the time until the decoder 12 can obtain a decoded image.

[0031] For example, as shown in FIG. 2, assume that each frame is encoded on the transmitting side, the encoded data is sequentially transmitted from the transmitting side to the receiving side, and decoded on the receiving side. For example, assume that an error is mixed into a packet at time t1, and that the error is detected on the receiving side at time t2. As in the example of FIG. 1, when the error is notified via the wireless network 21 of the high-capacity use case (eMBB), the error is notified to the transmitting side at time t3 (e.g., 10 ms later) due to network delays, etc. Therefore, encoding control based on the error is performed on the frame next to frame P2, which is the processing target at time t3. Therefore, in the example of FIG. 2, three frames of decoded images cannot be obtained (are lost). If the frames (decoded images) are not obtained, the image quality of the video (decoded video) will be reduced.

[0032] In this way, when transmitting a bitstream and error information via the wireless network 21 in the large capacity use case (eMBB), there is a risk that the period during which the quality of the decoded image is degraded due to the occurrence of an error on the receiving side may increase.

[0033] In addition, if a low latency use case (URLLC) is applied to the wireless network 21 of the image transmission system 10 in Figure 1 in order to reduce the delay time, there is a risk that transmission of the bit stream of moving images may become difficult due to insufficient transmission data rate.

[0034] <Building a network for transmitting error information> Therefore, the error information is transmitted via a wireless communication path that is different from the wireless communication path used for transmitting the bit stream and has a lower delay than the wireless communication path used for transmitting the bit stream.

[0035] For example, in an information processing method, error information transmitted via a second wireless communication channel that allows transmission with lower delay than the first wireless communication channel is acquired from a receiving device that receives encoded data of a moving image transmitted via a first wireless communication channel, and the encoding of the moving image is controlled based on the acquired error information.

[0036] For example, an information processing device may include an error information acquisition unit that acquires error information transmitted via a second wireless communication channel that allows transmission with lower latency than the first wireless communication channel from a receiving device that receives encoded data of a moving image transmitted via a first wireless communication channel, and an encoding control unit that controls the encoding of the moving image based on the error information acquired by the error information acquisition unit.

[0037] Also, for example, in an information processing method, encoded data of a moving image transmitted via a first wireless communication channel is received, and error information indicating an error in the encoded data is transmitted to the sender of the encoded data via a second wireless communication channel that allows transmission with less delay than the first wireless communication channel.

[0038] For example, an information processing device may include a data receiving unit that receives encoded data of a moving image transmitted via a first wireless communication channel, and an error information transmitting unit that transmits error information, which is information indicating an error in the encoded data received by the data receiving unit, to a sender of the encoded data via a second wireless communication channel that allows transmission with lower delay than the first wireless communication channel.

[0039] By doing so, it is possible to prevent an increase in the period during which the quality of the decoded image is reduced due to an error occurring on the receiving side when transmitting coded data of a moving image.

[0040] 2. First Embodiment <Image transmission system> Fig. 3 is a diagram showing an example of the main configuration of an image transmission system to which the present technology is applied. The image transmission system 100 shown in Fig. 3 is a system for transmitting moving images. As shown in Fig. 3, the image transmission system 100 includes an image encoding device 111 and an image decoding device 112. The image encoding device 111 and the image decoding device 112 are communicatively connected to each other via a wireless network 121. The image encoding device 111 and the image decoding device 112 are also communicatively connected to each other via a wireless network 122.

[0041] The image encoding device 111 acquires image data of a video to be transmitted, encodes the data, and generates encoded data (bit stream). The image encoding device 111 transmits the bit stream to the image decoding device 112 via a wireless network 121. The image decoding device 112 receives and decodes the bit stream. The image decoding device 112 outputs image data of a decoded image (decoded video) obtained by the decoding.

[0042] The wireless network 121 is a wireless communication channel capable of transmitting large amounts of data (having a high transmission data rate) compared to the wireless network 122. The specifications of the wireless network 121 are arbitrary, but a transmission data rate capable of transmitting a bit stream of image data is required.

[0043] Furthermore, if an error occurs during reception or decoding of the bitstream (i.e., a decoded image cannot be obtained), the image decoding device 112 transmits error information indicating the error to the image coding device 111 via the wireless network 122. The image coding device 111 receives the error information. The image coding device 111 controls the coding of moving images based on the received error information, etc. For example, the image coding device 111 performs coding so that the error does not propagate to subsequent frames.

[0044] The wireless network 122 is a wireless communication path that allows data transmission with higher reliability and lower delay than the wireless network 121. The specifications of the wireless network 122 are arbitrary, but the wireless network 122 must have a shorter delay time requirement than the wireless network 121.

[0045] The wireless networks 121 and 122 are wireless communication channels with different frequency bands (channels). For example, a 5G high capacity use case (eMBB) may be applied to the wireless network 121. For example, a 5G low latency use case (URLLC) may be applied to the wireless network 122. In the following description, it is assumed that the wireless network 121 is a wireless communication channel for the 5G high capacity use case (eMBB) and the wireless network 122 is a wireless communication channel for the 5G low latency use case (URLLC).

[0046] The image encoding device 111 can also monitor the state of the wireless network 121 and obtain QoE (Quality of Experience) information, which is a subjective evaluation of the wireless network 121. The image encoding device 111 can control the encoding of moving images based on the QoE information. This QoE information may be any information. For example, the QoE information may include information on wireless disconnection during communication, handover failure, etc., collected from a terminal using a MDT (Minimization of Drive Test) mechanism, as in the method described in Non-Patent Document 5.

[0047] 3 shows one image encoding device 111 and one image decoding device 112, the number of these devices is arbitrary, and may be, for example, multiple. That is, the image transmission system 100 may have any number of image encoding devices 111 and image decoding devices 112. The image transmission system 100 may also have devices other than the image encoding device 111 and the image decoding device 112. Furthermore, the image transmission system 100 may have wireless communication paths other than the wireless network 121 and the wireless network 122.

[0048] <Image encoding device> FIG. 4 is a block diagram showing an example of the main configuration of the image encoding device 111 in FIG.

[0049] Note that Fig. 4 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 4. In other words, in the image encoding device 111, there may be processing units that are not shown as blocks in Fig. 4, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 4.

[0050] 4, the image encoding device 111 includes an encoding unit 211, a communication unit 212, and an encoding control unit 213. The communication unit 212 includes a data transmission unit 221, a network status monitoring unit 222, and an error information monitoring unit 223.

[0051] The encoding unit 211 encodes image data (video images to be transmitted) input to the image encoding device 111 and generates the encoded data (bitstream). Any encoding method may be used. For example, AVC (Advanced Video Coding) described in Non-Patent Document 2, HEVC (High Efficiency Video Coding) described in Non-Patent Document 3, or VVC (Versatile Video Coding) described in Non-Patent Document 4 may be applied. Of course, encoding methods other than these may also be applied. The encoding unit 211 supplies the generated bitstream to the communication unit 212 (data transmission unit 221).

[0052] The communication unit 212 performs processing related to communication.

[0053] The data transmission unit 221 acquires the bitstream supplied from the encoding unit 211. The data transmission unit 221 transmits the acquired bitstream to the image decoding device 112 via the wireless network 121 (eMBB).

[0054] The network status monitoring unit 222 monitors the status of the wireless network 121 and obtains QoE information about the network. The network status monitoring unit 222 supplies the obtained QoE information to the encoding control unit 213.

[0055] The error information monitoring unit 223 monitors error information transmitted from the image decoding device 112 via the wireless network 122 (URLLC). When error information is transmitted from the image decoding device 112, the error information monitoring unit 223 receives the error information via the wireless network 122. That is, the error information monitoring unit 223 acquires error information transmitted via the wireless network 122, which allows transmission with shorter delay than the wireless network 121, from the image decoding device 112, which receives encoded data of a moving image transmitted via the wireless network 121. The error information monitoring unit 223 supplies the received error information to the encoding control unit 213.

[0056] The encoding control unit 213 controls the encoding process performed by the encoding unit 211. The encoding control unit 213 controls the encoding process performed by the encoding unit 211 by supplying the encoding unit 211 with encoding control information that specifies an encoding method, parameters, etc.

[0057] For example, the encoding control unit 213 acquires error information supplied from the error information monitoring unit 223, and controls the encoding unit 211 based on the error information. For example, when the encoding control unit 213 acquires error information, it causes the encoding unit 211 to perform encoding processing so that the error indicated by the error information does not propagate to subsequent frames.

[0058] Furthermore, the encoding control unit 213 acquires QoE information supplied from the network status monitoring unit 222, and controls the encoding unit 211 based on the QoE information. For example, the encoding control unit 213 causes the encoding unit 211 to perform encoding processing so as to improve the communication status of the wireless network 121.

[0059] <Encoding part> FIG. 5 is a block diagram showing an example of the main configuration of the encoding unit 211 in FIG.

[0060] Note that Fig. 5 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 5. In other words, in the encoding unit 211, there may be processing units that are not shown as blocks in Fig. 5, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 5.

[0061] 5, the encoding unit 211 includes a rearrangement buffer 251, a calculation unit 252, a coefficient conversion unit 253, a quantization unit 254, an encoding unit 255, and an accumulation buffer 256. The encoding unit 211 also includes an inverse quantization unit 257, an inverse coefficient conversion unit 258, a calculation unit 259, an in-loop filter unit 260, and a frame memory 261. The encoding unit 211 also includes a prediction unit 262 and a rate control unit 263. The prediction unit 262 includes an inter prediction unit 271 and an intra prediction unit 272.

[0062] Each frame (input image) of a video is input to the encoding unit 211 in its playback order (display order). The reordering buffer 251 acquires and holds (stores) each input image in its playback order (display order). The reordering buffer 251 reorders the input images in coding order (decoding order) and divides them into blocks, which are processing units. The reordering buffer 251 supplies each processed input image to the calculation unit 252.

[0063] The calculation unit 252 subtracts the predicted image supplied from the prediction unit 262 from the image corresponding to the block of processing units supplied from the sorting buffer 251 to derive residual data, and supplies this to the coefficient conversion unit 253.

[0064] The coefficient conversion unit 253 acquires the residual data supplied from the calculation unit 252. The coefficient conversion unit 253 then performs coefficient conversion on the residual data using a predetermined method to derive transform coefficient data. Any method may be used for this coefficient conversion process. For example, orthogonal transform may be used. The coefficient conversion unit 253 supplies the derived transform coefficient data to the quantization unit 254.

[0065] The quantization unit 254 acquires the transform coefficient data supplied from the coefficient conversion unit 253. The quantization unit 254 also quantizes the transform coefficient data to derive quantized coefficient data. At this time, the quantization unit 254 performs quantization at a rate specified by the rate control unit 263. The quantization unit 254 supplies the derived quantized coefficient data to the encoding unit 255 and the inverse quantization unit 257.

[0066] The encoding unit 255 acquires the quantization coefficient data supplied from the quantization unit 254. The encoding unit 255 also acquires information about filters, such as filter coefficients, supplied from the in-loop filter unit 260. The encoding unit 255 also acquires information about the optimal prediction mode supplied from the prediction unit 262.

[0067] The encoding unit 255 performs entropy encoding (lossless encoding) on ​​the information, generates a bit string (encoded data), and multiplexes it. Any entropy encoding method can be used. For example, the encoding unit 255 can apply CABAC (Context-based Adaptive Binary Arithmetic Code) as the entropy encoding. The encoding unit 255 can also apply CAVLC (Context-based Adaptive Variable Length Code) as the entropy encoding. Of course, encoding methods other than these examples can also be applied.

[0068] The encoding unit 255 supplies the encoded data derived in this manner to the accumulation buffer 256.

[0069] The accumulation buffer 256 temporarily stores the encoded data obtained by the encoding unit 255. The accumulation buffer 256 supplies the stored encoded data to the data transmission unit 221 as, for example, a bit stream at a predetermined timing.

[0070] The inverse quantization unit 257 obtains the quantized coefficient data supplied from the quantization unit 254. The inverse quantization unit 257 inversely quantizes the quantized coefficient data to derive transform coefficient data. This inverse quantization process is the inverse process of the quantization process executed in the quantization unit 254. The inverse quantization unit 257 supplies the derived transform coefficient data to the inverse coefficient conversion unit 258.

[0071] The inverse coefficient transform unit 258 obtains the transform coefficient data supplied from the inverse quantization unit 257. The inverse coefficient transform unit 258 performs inverse coefficient transform on the transform coefficient data using a predetermined method, and derives residual data. This inverse coefficient transform processing is the inverse processing of the coefficient transform processing executed in the coefficient transform unit 253. For example, when the coefficient transform unit 253 performs an orthogonal transform processing on the residual data, the inverse coefficient transform unit 258 performs an inverse orthogonal transform processing on the transform coefficient data, which is the inverse processing of the orthogonal transform processing. The inverse coefficient transform unit 258 supplies the derived residual data to the calculation unit 259.

[0072] The calculation unit 259 acquires the residual data supplied from the inverse coefficient conversion unit 258 and the predicted image supplied from the prediction unit 262. The calculation unit 259 adds the residual data to the predicted image corresponding to the residual data to derive a locally decoded image. The calculation unit 259 supplies the derived locally decoded image to the in-loop filter unit 260 and the frame memory 261.

[0073] The in-loop filter unit 260 acquires the locally decoded image supplied from the calculation unit 259. Furthermore, the in-loop filter unit 260 acquires the input image (original image) supplied from the rearrangement buffer 251. Note that any information is input to the in-loop filter unit 260, and information other than the above information may also be input. For example, information such as a prediction mode, motion information, a code amount target value, a quantization parameter qP, a picture type, and a block (CU, CTU, etc.) may be input to the in-loop filter unit 260 as necessary.

[0074] The in-loop filter unit 260 performs appropriate filtering on the locally decoded image. The in-loop filter unit 260 also uses the input image (original image) and other input information for the filtering, as necessary.

[0075] For example, the in-loop filter unit 260 may apply a bilateral filter as its filtering process. For example, the in-loop filter unit 260 may apply a deblocking filter (DBF (DeBlocking Filter)) as its filtering process. For example, the in-loop filter unit 260 may apply an adaptive offset filter (SAO (Sample Adaptive Offset)) as its filtering process. For example, the in-loop filter unit 260 may apply an adaptive loop filter (ALF (Adaptive Loop Filter)) as its filtering process. Furthermore, the in-loop filter unit 260 may apply a combination of multiple of these filters as its filtering process. Note that which filters to apply and in what order to apply them are arbitrary and can be selected as appropriate. For example, the in-loop filter unit 260 applies four in-loop filters, namely a bilateral filter, a deblocking filter, an adaptive offset filter, and an adaptive loop filter, in this order as its filtering process.

[0076] Of course, the filtering process performed by the in-loop filter unit 260 is arbitrary and is not limited to the above example. For example, the in-loop filter unit 260 may apply a Wiener filter or the like.

[0077] The in-loop filter unit 260 supplies the filtered locally decoded image to the frame memory 261. When transmitting information about the filter, such as a filter coefficient, to the decoding side, the in-loop filter unit 260 supplies the information about the filter to the encoding unit 255.

[0078] The frame memory 261 executes processing related to the storage of image-related data. For example, the frame memory 261 acquires and holds (stores) a locally decoded image supplied from the calculation unit 259 or a locally decoded image that has been subjected to filtering processing supplied from the in-loop filter unit 260. The frame memory 261 also reconstructs and holds a decoded image for each picture using the locally decoded image (storing the image in a buffer within the frame memory 261). The frame memory 261 supplies the decoded image (or a portion thereof) to the prediction unit 262 in response to a request from the prediction unit 262.

[0079] The prediction unit 262 executes processing related to generation of a predicted image. For example, the prediction unit 262 obtains an input image (original image) supplied from the rearrangement buffer 251. For example, the prediction unit 262 obtains a decoded image (or a part thereof) read from the frame memory 261.

[0080] The inter prediction section 271 of the prediction section 262 performs inter prediction and motion compensation by referring to a decoded image of another frame as a reference image to generate a predicted image. Also, the intra prediction section 272 of the prediction section 262 performs intra prediction by referring to a decoded image of the current frame as a reference image to generate a predicted image.

[0081] The prediction unit 262 evaluates the predicted image generated in each prediction mode and selects the optimal prediction mode based on the evaluation result. Then, the prediction unit 262 supplies the predicted image generated in the optimal prediction mode to the calculation unit 252 and the calculation unit 259. Furthermore, the prediction unit 262 supplies information on the optimal prediction mode selected by the above processing to the encoding unit 255 as necessary.

[0082] It should be noted that the prediction unit 262 (the inter prediction unit 271 and the intra prediction unit 272) can also perform prediction under the control of the encoding control unit 213. For example, the prediction unit 262 can obtain encoding control information supplied from the encoding control unit 213, and perform intra prediction or inter prediction according to the encoding control information.

[0083] The rate control unit 263 controls the rate of the quantization operation of the quantization unit 254 based on the code amount of the coded data stored in the storage buffer 256 so as to prevent overflow or underflow.

[0084] <Image decoding device> FIG. 6 is a block diagram showing an example of the main configuration of the image decoding device 112 in FIG.

[0085] Note that Fig. 6 shows the main processing units, data flows, etc., and does not necessarily include all of them. In other words, the image decoding device 112 may include processing units that are not shown as blocks in Fig. 6, or processes or data flows that are not shown as arrows, etc. in Fig. 6.

[0086] 6, the image decoding device 112 includes a communication unit 311, a decoding control unit 312, and a decoding unit 313. The communication unit 311 includes a data receiving unit 321, a reception error detection unit 322, and an error information transmitting unit 323.

[0087] The communication unit 311 performs processing related to communication.

[0088] The data receiving unit 321 receives a bitstream transmitted from the image encoding device 111 via the wireless network 121 (eMBB). The data receiving unit 321 supplies the received bitstream to the decoding unit 313.

[0089] The reception error detection unit 322 monitors the reception status of the data reception unit 321 and detects errors (reception errors) that occur in the data reception unit 321. When the reception error detection unit 322 detects a reception error, it supplies error information indicating the reception error to the error information transmission unit 323. In addition, the reception error detection unit 322 supplies the error detection result (information indicating whether or not a reception error has been detected, etc.) to the decoding control unit 312.

[0090] The error information sending unit 323 sends the error information to the image encoding device 111 via the wireless network 122 (URLLC). This error information is transmitted to the image encoding device 111 via the wireless network 122 (URLLC) and received by the error information monitoring unit 223.

[0091] In other words, the error information sending unit 323 sends error information, which is information indicating an error in the encoded data received by the data receiving unit 321, to the sender of the encoded data via the wireless network 122, which is capable of transmission with lower latency than the wireless network 121.

[0092] The error information transmitting unit 323 acquires error information indicating a reception error supplied from the reception error detecting unit 322. The error information transmitting unit 323 also acquires error information indicating a decoding error supplied from the decoding unit 313. The error information transmitting unit 323 transmits the acquired error information to the image encoding device 111.

[0093] That is, the error information transmitted by the error information transmission unit 323 may include information indicating an error that occurred when receiving the encoded data. Also, the error information transmitted by the error information transmission unit 323 may include information indicating an error that occurred when decoding the encoded data. Of course, the error information transmitted by the error information transmission unit 323 may include both of these types of information, or may include information indicating other errors.

[0094] The decoding control unit 312 controls the decoding process executed by the decoding unit 313. For example, the decoding control unit 312 controls the decoding process executed by the decoding unit 313 by supplying the decoding unit 313 with decoding control information that specifies a decoding method, parameters, etc.

[0095] For example, the decoding control unit 312 acquires the error detection result supplied from the reception error detection unit 322, and controls the encoding unit 211 based on the error detection result.

[0096] The decoding unit 313 obtains a bit stream supplied from the data receiving unit 321. The decoding unit 313 decodes the bit stream to generate image data of a decoded image (a decoded video image to be transmitted). The decoding unit 313 outputs the image data to the outside of the image decoding device 112. Note that the decoding unit 313 can perform this decoding process under the control of the decoding control unit 312. Furthermore, if an error (decoding error) occurs in the decoding process, the decoding unit 313 supplies error information indicating the decoding error to the error information transmitting unit 323.

[0097] <Decryption section> FIG. 7 is a block diagram showing an example of the main configuration of the decoding unit 313 in FIG.

[0098] Note that Fig. 7 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 7. In other words, in the decoding unit 313, there may be processing units that are not shown as blocks in Fig. 7, and there may be processing or data flows that are not shown as arrows, etc. in Fig. 7.

[0099] As shown in FIG. 7, the decoding unit 313 includes an accumulation buffer 351, a decoding unit 352, an inverse quantization unit 353, an inverse coefficient conversion unit 354, a calculation unit 355, an in-loop filter unit 356, a rearrangement buffer 357, a frame memory 358, and a prediction unit 359.

[0100] The accumulation buffer 351 acquires and holds (stores) the bit stream supplied from the data receiving unit 321. At a predetermined timing, or when a predetermined condition is met, the accumulation buffer 351 extracts coded data included in the accumulated bit stream and supplies the coded data to the decoding unit 352.

[0101] The decoding unit 352 acquires the coded data supplied from the accumulation buffer 351. The decoding unit 352 decodes the acquired coded data. In doing so, the decoding unit 352 applies entropy decoding (lossless decoding) such as CABAC or CAVLC. That is, the decoding unit 352 decodes the coded data using a decoding method corresponding to the coding method of the coding process executed by the coding unit 255. The decoding unit 352 decodes the coded data and derives quantization coefficient data. The decoding unit 352 supplies the derived quantization coefficient data to the inverse quantization unit 353.

[0102] Furthermore, if an error (decoding error) occurs in the decoding process, the decoding unit 352 generates error information indicating the decoding error and supplies it to the error information transmission unit 323.

[0103] The inverse quantization unit 353 performs inverse quantization processing on the quantized coefficient data to derive transform coefficient data. This inverse quantization processing is the inverse processing of the quantization processing performed in the quantization unit 254. The inverse quantization unit 353 supplies the derived transform coefficient data to the inverse coefficient conversion unit 354.

[0104] The inverse coefficient conversion unit 354 obtains the transform coefficient data supplied from the inverse quantization unit 353. The inverse coefficient conversion unit 354 performs inverse coefficient conversion processing on the transform coefficient data to derive residual data. This inverse coefficient conversion processing is the inverse processing of the coefficient conversion processing executed in the coefficient conversion unit 253. The inverse coefficient conversion unit 354 supplies the derived residual data to the calculation unit 355.

[0105] The calculation unit 355 obtains the residual data supplied from the inverse coefficient conversion unit 354 and the predicted image supplied from the prediction unit 359. The calculation unit 355 adds the residual data and the predicted image corresponding to the residual data to derive a locally decoded image. The calculation unit 355 supplies the derived locally decoded image to the in-loop filter unit 356 and the frame memory 358.

[0106] The in-loop filter unit 356 acquires the locally decoded image supplied from the calculation unit 355. The in-loop filter unit 356 performs appropriate filtering on the locally decoded image. For example, the in-loop filter unit 356 may apply a bilateral filter as the filtering process. For example, the in-loop filter unit 356 may apply a deblocking filter (DBF (DeBlocking Filter)) as the filtering process. For example, the in-loop filter unit 356 may apply an adaptive offset filter (SAO (Sample Adaptive Offset)) as the filtering process. For example, the in-loop filter unit 356 may apply an adaptive loop filter (ALF (Adaptive Loop Filter)) as the filtering process. Furthermore, the in-loop filter unit 356 may apply a combination of multiple of these filters as the filtering process. Note that which filters to apply and in what order they are applied are arbitrary and can be selected as appropriate. For example, the in-loop filter unit 356 applies four in-loop filters, namely, a bilateral filter, a deblocking filter, an adaptive offset filter, and an adaptive loop filter, in this order as filtering. Of course, the filtering performed by the in-loop filter unit 356 is arbitrary and is not limited to the above example. For example, the in-loop filter unit 356 may apply a Wiener filter or the like.

[0107] The in-loop filter unit 356 performs filtering corresponding to the filtering performed by the in-loop filter unit 260. The in-loop filter unit 356 supplies the filtered locally decoded image to a reordering buffer 357 and a frame memory 358.

[0108] The reordering buffer 357 receives the locally decoded images supplied from the in-loop filter unit 356 as input and holds (stores) them. The reordering buffer 357 uses the locally decoded images to reconstruct decoded images for each picture and holds them (stores them in the buffer). The reordering buffer 357 reorders the obtained decoded images from decoding order to playback order. The reordering buffer 357 outputs the group of decoded images reordered in playback order to the outside of the image decoding device 112 as video data.

[0109] The frame memory 358 acquires the locally decoded image supplied from the calculation unit 355, reconstructs the decoded image for each picture, and stores the reconstructed image in a buffer within the frame memory 358. The frame memory 358 also acquires the locally decoded image that has been in-loop filtered and supplied from the in-loop filter unit 356, reconstructs the decoded image for each picture, and stores the reconstructed image in a buffer within the frame memory 358. The frame memory 358 appropriately supplies the stored decoded image (or a part thereof) to the prediction unit 359 as a reference image.

[0110] The prediction unit 359 obtains the decoded image (or a part thereof) read from the frame memory 358. The prediction unit 359 executes a prediction process in the prediction mode adopted during encoding, and generates a predicted image by referring to the decoded image as a reference image. The prediction unit 359 supplies the generated predicted image to the calculation unit 355.

[0111] <Image encoding process flow> Next, a description will be given of the processing executed in the image transmission system 100. An example of the flow of image encoding processing executed by the image encoding device 111 will be described with reference to the flowchart in FIG.

[0112] When the image encoding process starts, in step S201, the encoding unit 211 acquires image data of a moving image to be transmitted.

[0113] In step S202, the encoding unit 211 encodes the image data acquired in step S201 in accordance with the encoding control of the encoding control unit 213, and generates a bit stream.

[0114] In step S203, the data transmission unit 221 transmits the bitstream generated in step S202 to the image decoding device 112 via the wireless network 121 (eMBB).

[0115] In step S204, the network status monitoring unit 222 monitors the status of the wireless network 121 and supplies QoE information to the encoding control unit 213 as appropriate.

[0116] In step S205, the error information monitoring unit 223 monitors the transmission of error information via the wireless network 122. When error information is transmitted from the image decoding device 112 via the wireless network 122, the error information monitoring unit 223 receives the error information and supplies it to the encoding control unit 213.

[0117] In step S206, the encoding control unit 213 controls the encoding process executed in step S202 based on the process results (monitoring results) of steps S204 and S205.

[0118] In step S207, the encoding control unit 213 determines whether or not to end the image encoding process. If the encoding of the moving image is continuing and it is determined not to end the image encoding process, the process returns to step S201, and the subsequent processes are repeated.

[0119] If it is determined in step S207 that the image encoding process is to be ended, the image encoding process ends.

[0120] <Flow of image decoding process> Next, an example of the flow of image decoding processing executed by the image decoding device 112 will be described with reference to the flowchart of FIG.

[0121] When the image decoding process starts, in step S301, the data receiving unit 321 receives a bitstream transmitted from the image encoding device 111 via the wireless network 121 (eMBB).

[0122] In step S302, the reception error detection unit 322 monitors the reception process in step S301, and if a reception error occurs, detects the reception error.

[0123] In step S303, the decoding control unit 312 controls the process (decoding process) of step S304, which will be described later, based on the reception error detection result of step S302.

[0124] In step S304, the decoding unit 313 decodes the bitstream received in step S301 in accordance with the decoding control in step S303 to generate image data of a decoded video. This image data is output to the outside of the image decoding device 112.

[0125] In step S305, if a decoding error occurs in the decoding process in step S304, the decoding unit 313 detects the decoding error.

[0126] In step S306, the error information transmission unit 323 determines whether or not an error has been detected. That is, the error information transmission unit 323 determines whether or not a reception error has been detected in step S302, and whether or not a decoding error has been detected in step S305. If an error has been detected, that is, if at least one of a reception error and a decoding error has been detected, the process proceeds to step S307.

[0127] In step S307, the error information transmission unit 323 transmits error information indicating the detected error to the image encoding device 111 via the wireless network 122 (URLLC). When the process of step S307 ends, the process proceeds to step S308.

[0128] Also, if it is determined in step S306 that no error has been detected, that is, that neither a reception error nor a decoding error has been detected, the process of step S307 is skipped and the process proceeds to step S308.

[0129] In step S308, the error information transmission unit 323 determines whether or not to terminate the image decoding process. If the transmission of the bitstream continues and it is determined not to terminate the image decoding process, the process returns to step S301, and the subsequent processes are repeated.

[0130] If it is determined in step S308 that the image decoding process is to be ended, the image decoding process ends.

[0131] As described above, by transmitting error information via wireless network 122 (URLLC), which allows communication with lower delay than wireless network 121 (eMBB) that transmits video bitstreams, the delay associated with transmitting the error information can be made shorter than in the example of Fig. 2, as shown in Fig. 10. For example, in the case of Fig. 10, the loss of decoded images can be reduced to two frames.

[0132] In other words, it is possible to prevent an increase in the period during which the quality of the decoded image is reduced due to an error occurring on the receiving side when transmitting coded data of a moving image.

[0133] 3. Second Embodiment <Intrastripe> Any coding control method can be used to prevent errors from propagating to subsequent frames. For example, when a technique called intrastripe is applied to image coding, this intrastripe may be used.

[0134] For example, as shown in A of FIG. 11, it is assumed that each frame of a moving image is made up of intraframes (I) that are frames subjected to intra-coding and interframes (P) that are frames subjected to inter-coding.

[0135] In this case, the amount of code for intraframes may become excessively large compared to the amount of code for interframes, as shown in B of Fig. 11. Since it is necessary to match the amount of code for intraframes with a large amount of code, the buffer capacity increases, which may increase delays.

[0136] Therefore, as shown in A of Fig. 12, all frames are set as interframes (P), and a partial area of ​​each frame is set as an intra area and intra-coded. This intra area is also called an intra stripe. As shown in B of Fig. 11, the position of the intra stripe (intra area) is moved for each frame, and rotated every predetermined number of frames. For example, a frame is divided into N parts, and one partial area is set as an intra area. Then, for each frame, one intra area is moved to the adjacent partial area, and returned to its original position after N frames.

[0137] By doing this, as shown in Fig. 13, the code amount of each frame can be smoothed compared to the example B of Fig. 11. This makes it possible to suppress an increase in buffer capacity and an increase in delay.

[0138] Even if an error occurs, the intra-area circulates around the frame, so that a decoded image for one frame can be obtained. For example, the propagation of errors can be suppressed by performing vector control as described in Patent Document 1.

[0139] However, with this method, there is a risk that vector control may degrade the quality of the decoded image in the intra-frame region. Therefore, even if a decoded image for one frame is obtained, the image quality may be degraded. Therefore, there is a risk that the period during which the quality of the decoded image is degraded due to an error occurring on the receiving side when transmitting coded image data may increase.

[0140] <Intrastripe position control> Therefore, when error information is acquired, coding control may be performed so as to return the position of the intra stripe to the initial position.

[0141] In other words, when encoding each frame of a moving image to be transmitted, a part of the frame is set as an intra-area and intra-encoded, and the position of the intra-area is moved in a predetermined direction for each frame so as to rotate every predetermined number of frames. In such a case, when error information is acquired by the error information monitoring unit 223, the encoding control unit 213 may return the position of the intra-area to its initial position.

[0142] For example, as shown in B of Fig. 12, the intra region is a partial region of a frame that is composed of multiple blocks aligned in the vertical direction of the frame, and the position of the intra region is initially set at the left edge of the frame and is moved rightward for each frame. In such a case, when error information is acquired by the error information monitoring unit 223, the encoding control unit 213 may return the position of the intra region to the left edge of the frame.

[0143] For example, as shown in FIG. 14, if an error occurs in Pic0, the encoding control unit 213 controls the encoding unit 211 to move the position of the intra stripe of Pic1 to the initial position (the left edge of the frame).

[0144] By doing this, it is possible to obtain an intra-stripe decoded image without reducing the image quality. Therefore, when one frame's worth of decoded image is obtained, a frame image with unreduced image quality can be obtained. Therefore, it is possible to prevent an increase in the period during which the image quality of the decoded image is reduced due to an error occurring on the receiving side when transmitting encoded image data.

[0145] <Encoding control process flow> An example of the flow of the encoding control process executed in step S206 in FIG. 8 in this case will be described with reference to the flowchart in FIG.

[0146] When the encoding control process starts, the encoding control unit 213 determines whether or not an error has been detected in step S401. If it is determined that an error has been detected, the process proceeds to step S402.

[0147] In step S402, the encoding control unit 213 controls the encoding unit 211 to return the intrastripe to the left end (initial position) of the frame. When the process of step S402 ends, the encoding control process ends, and the process proceeds to step S207 in FIG. 8.

[0148] 8. If it is determined in step S401 that no error has been detected, the process of step S402 is skipped, the encoding control process ends, and the process proceeds to step S207 in FIG.

[0149] By executing the encoding control process in this manner, it is possible to prevent an increase in the period during which the quality of the decoded image is reduced due to an error occurring on the receiving side when transmitting encoded image data.

[0150] Note that mode restrictions may be imposed on the boundaries of intra-stripe to prevent error data from propagating from the error area. For example, in the case of VVC described in Non-Patent Document 4, setting a virtual boundary at the boundary of intra-stripe and encoding it can prevent the introduction of error data.

[0151] 4. Third Embodiment <Intraframe insertion> For example, as shown in A of Fig. 11, it is assumed that each frame of a video is composed of an intra-frame (I) which is a frame that is intra-coded, and an inter-frame (P) which is a frame that is inter-coded. In this case, control may be performed to insert an intra-frame, as shown in Fig. 16.

[0152] In other words, the video to be transmitted includes intraframes, which are frames that are intra-coded. In this case, when the error information monitoring unit 223 acquires error information, the encoding control unit 213 may set the next frame to be encoded to an intraframe.

[0153] For example, as shown in FIG. 16, if an error occurs in Pic0, the encoding control unit 213 controls the encoding unit 211 to set Pic1 as an intra-frame.

[0154] This prevents errors from propagating to frames after Pic 2. This prevents the increase in the period during which the quality of the decoded image is reduced due to errors occurring on the receiving side when transmitting coded image data.

[0155] <Encoding control process flow> An example of the flow of the encoding control process executed in step S206 in FIG. 8 in this case will be described with reference to the flowchart in FIG.

[0156] When the encoding control process starts, the encoding control unit 213 determines whether or not an error has been detected in step S431. If it is determined that an error has been detected, the process proceeds to step S432.

[0157] In step S432, the encoding control unit 213 inserts an intra-frame by controlling the encoding unit 211. When the process of step S432 ends, the encoding control process ends, and the process proceeds to step S207 in FIG.

[0158] 8. If it is determined in step S431 that no error has been detected, the process of step S432 is skipped, the encoding control process ends, and the process proceeds to step S207 in FIG.

[0159] By executing the encoding control process in this manner, it is possible to prevent an increase in the period during which the quality of the decoded image is reduced due to an error occurring on the receiving side when transmitting encoded image data.

[0160] 5. Fourth Embodiment <Other configurations of image transmission system 1> The configuration of the image transmission system 100 is not limited to the example in Fig. 3. For example, as shown in Fig. 18, the transmission of the bit stream and the transmission of the error information may be performed in the same channel (same frequency band).

[0161] 18, the bit stream is transmitted in a downlink 511 of a wireless network 501, and the error information is transmitted in an uplink 512 of the same wireless network 501 (i.e., the same frequency band). In this way, the bit stream can be transmitted by communication of the large capacity use case (eMBB), and the error information can be transmitted by communication of the low latency use case (URLLC).

[0162] In other words, the first wireless communication channel for transmitting the bit stream may be a wireless communication channel that is a downlink in the same frequency band as the second wireless communication channel for transmitting the error information and that satisfies the requirements of eMBB (enhanced Mobile Broadband) of a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union, and the second wireless communication channel may be a wireless communication channel that is an uplink in the same frequency band as the first wireless communication channel and that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of the above wireless communication system.

[0163] By doing so, as in the example of FIG. 3, it is possible to suppress an increase in the period during which the quality of the decoded image is reduced due to the occurrence of an error on the receiving side when transmitting coded image data.

[0164] In addition, in order to prevent the quality of URLLC communication from deteriorating due to interference from downlink eMBB communication (i.e., to ensure the quality of uplink URLLC communication), control may be performed to stop eMBB communication during the period when an error occurs.

[0165] <Other configurations of image transmission system 2> Also, for example, as shown in Figure 19, the transmission of the bit stream and the transmission of the error information may be performed in different network slices. For example, in 5G, network slicing allows a network to be virtually divided into multiple network slices, and each of the network slices can be used. By using such a function, the transmission of the bit stream and the transmission of the error information may be performed.

[0166] In the example of Fig. 19, the bit stream is transmitted in a network slice 551 of a 5G network 541, and the error information is transmitted in another network slice 552 of the same 5G network 541. In this way, the bit stream can be transmitted by communication of the high-capacity use case (eMBB), and the error information can be transmitted by communication of the low-latency use case (URLLC).

[0167] In other words, the first wireless communication channel for transmitting the bit stream may be a network slice different from that of the second wireless communication channel for transmitting the error information, and may be a wireless communication channel that satisfies the eMBB (enhance Mobile Broadband) requirements of a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union, and the second wireless communication channel may be a network slice different from that of the first wireless communication channel, and may be a wireless communication channel that satisfies the URLLC (Ultra Reliable Low Latency Communication) requirements of the above wireless communication system.

[0168] By doing so, as in the example of FIG. 3, it is possible to suppress an increase in the period during which the quality of the decoded image is reduced due to the occurrence of an error on the receiving side when transmitting coded image data.

[0169] <Other configurations of image transmission system 3> Also, for example, as shown in FIG. 20, the transmission of the bit stream and the transmission of the error information may be performed over communication paths that comply with different wireless communication standards.

[0170] In the example of FIG. 20, the bit stream is transmitted over a wireless network 571, and the error information is transmitted over a wireless network 572 that has a different communication standard from that of the wireless network 571.

[0171] The wireless network 571 may be, for example, a wireless communication channel conforming to the International Mobile Telecommunications (IMT)-Advanced standard (hereinafter also referred to as 4G). The wireless network 571 may also be a wireless communication channel conforming to the Long Term Evolution (LTE) standard established by the Third Generation Partnership Project (3GPP). The wireless network 571 may also be a wireless communication channel using the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard (hereinafter also referred to as Wi-Fi (registered trademark)). Of course, the wireless network 571 may also be a communication channel conforming to a communication standard other than these. In contrast, the wireless network 572 may be, for example, a 5G wireless communication channel.

[0172] In this way, the bit stream can be transmitted by high-capacity communication, and the error information can be transmitted by low-delay use case (URLLC) communication.

[0173] That is, the first wireless communication channel for transmitting the bit stream may be a wireless communication channel conforming to the International Mobile Telecommunications (IMT)-Advanced standard defined by the International Telecommunications Union, a wireless communication channel conforming to the Long Term Evolution (LTE) standard established by the Third Generation Partnership Project (3GPP), or a wireless communication channel using the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Also, the second wireless communication channel for transmitting the error information may be a wireless communication channel that satisfies the requirements of Ultra Reliable Low Latency Communication (URLLC) for a wireless communication system that satisfies the IMT-2020 standard defined by the International Telecommunications Union.

[0174] By doing so, as in the example of FIG. 3, it is possible to suppress an increase in the period during which the quality of the decoded image is reduced due to the occurrence of an error on the receiving side when transmitting coded image data.

[0175] <6. Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0176] FIG. 21 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0177] In a computer 900 shown in FIG. 21, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0178] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0179] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0180] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0181] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0182] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0183] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0184] <Applicable targets of this technology> This technology can be applied to any image encoding / decoding method.

[0185] The present technology can be applied to any configuration, for example, transmitters and receivers (e.g., television receivers and mobile phones) used in satellite broadcasting, distribution over the Internet, and distribution to terminals via cellular communication, or various electronic devices such as devices that record images on media such as optical disks, magnetic disks, and flash memories, or that play images from these storage media (e.g., hard disk recorders and cameras).

[0186] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0187] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0188] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0189] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, their uses are also arbitrary.

[0190] For example, the present technology can be applied to systems and devices used to provide viewing content, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for transportation, such as monitoring traffic conditions and controlling automatic driving. Furthermore, for example, the present technology can also be applied to systems and devices used for security. Furthermore, for example, the present technology can also be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can also be applied to systems and devices used to monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife. Furthermore, for example, the present technology can also be applied to systems and devices used for sports.

[0191] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0192] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0193] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0194] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0195] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0196] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0197] The present technology can also be configured as follows. (1) an error information acquisition unit that acquires error information transmitted via a second wireless communication channel that can transmit encoded data of a moving image transmitted via a first wireless communication channel from a receiving device; and an encoding control unit that controls encoding of the video based on the error information acquired by the error information acquisition unit; An information processing device comprising: (2) When encoding each frame of the moving image, a part of the frame is set as an intra region and the moving image is intra-encoded; the position of the intra region is moved in a predetermined direction for each frame so as to cycle every predetermined number of frames; When the error information acquisition unit acquires the error information, the encoding control unit returns the position of the intra region to an initial position. The information processing device described in (1). (3) the intra region is a partial region of the frame that is composed of a plurality of blocks aligned in the vertical direction of the frame; The position of the intra region is moved rightward for each frame, with the left edge of the frame being the initial position; When the error information acquisition unit acquires the error information, the encoding control unit returns the position of the intra region to the left end of the frame. (2) An information processing device according to the present invention. (4) The moving image includes an intraframe, which is a frame that is intra-coded, When the error information acquisition unit acquires the error information, the encoding control unit sets the next frame to be encoded to the intraframe. The information processing device described in (1). (5) The error information includes information indicating an error that occurred during reception of the encoded data. An information processing device according to any one of (1) to (4). (6) The error information includes information indicating an error that occurs during decoding of the coded data. An information processing device according to any one of (1) to (5). (7) A coding unit is further provided to code the moving image and generate the coded data. The encoding control unit controls the encoding unit based on the error information acquired by the error information acquisition unit. An information processing device according to any one of (1) to (6). (8) further comprising a wireless communication path status monitoring unit that monitors the status of the first wireless communication path; The encoding control unit further controls encoding of the video based on the state of the first wireless communication channel monitored by the wireless communication channel state monitoring unit. An information processing device according to any one of (1) to (7). (9) The first wireless communication channel and the second wireless communication channel are communication channels of different frequency bands. An information processing device according to any one of (1) to (8). (10) The first wireless communication channel is a communication channel that satisfies the requirements of eMBB (enhance Mobile Broadband) in a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union, The second wireless communication path is a communication path that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of the wireless communication system. (9) An information processing device according to (9). (11) The first wireless communication path is a downlink of the same frequency band as the second wireless communication path, and is a wireless communication path that satisfies requirements for eMBB (enhance Mobile Broadband) of a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union; The second wireless communication path is an uplink in the same frequency band as the first wireless communication path and is a wireless communication path that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of the wireless communication system. An information processing device according to any one of (1) to (8). (12) The first wireless communication path is a network slice different from the second wireless communication path, and is a wireless communication path that satisfies requirements for eMBB (enhance Mobile Broadband) in a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard defined by the International Telecommunication Union; The second wireless communication path is the network slice different from the first wireless communication path and is a wireless communication path that satisfies requirements of URLLC (Ultra Reliable Low Latency Communication) of the wireless communication system. An information processing device according to any one of (1) to (8). (13) The first wireless communication channel and the second wireless communication channel are communication channels of different wireless communication standards. An information processing device according to any one of (1) to (8). (14) The first wireless communication channel is a wireless communication channel conforming to the IMT (International Mobile Telecommunications)-Advanced standard defined by the International Telecommunications Union, a wireless communication channel conforming to the LTE (Long Term Evolution) standard established by the 3GPP (Third Generation Partnership Project), or a wireless communication channel using the IEEE (Institute of Electrical and Electronics Engineers) 802.11 standard; The second wireless communication channel is a wireless communication channel that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of a wireless communication system that satisfies the IMT-2020 standard established by the International Telecommunication Union. (13) An information processing device according to (13). (15) Obtaining error information transmitted via a second wireless communication channel, which is capable of transmission with lower delay than the first wireless communication channel, from a receiving device that receives encoded data of a moving image transmitted via the first wireless communication channel; and controlling encoding of the video based on the acquired error information. Information processing methods.

[0198] (16) a data receiving unit that receives encoded data of a moving image transmitted via the first wireless communication channel; an error information transmitting unit that transmits error information, which is information indicating an error in the encoded data received by the data receiving unit, to a transmission source of the encoded data via a second wireless communication channel that is capable of transmission with lower delay than the first wireless communication channel; An information processing device comprising: (17) A reception error detection unit is further provided to detect a reception error of the encoded data by the data receiving unit, The error information includes information indicating the reception error detected by the reception error detection unit. (16) An information processing device according to (16). (18) The error information includes information indicating a decoding error related to the decoding of the encoded data received by the data receiving unit. The information processing device according to (16) or (17). (19) A decoding unit that decodes the encoded data received by the data receiving unit, The error information transmission unit acquires the information indicating the decoding error supplied from the decoding unit, and transmits the error information including the information indicating the decoding error. (18) An information processing device according to (18). (20) receiving encoded data of a moving image transmitted via the first wireless communication channel; Transmitting error information, which is information indicating an error in the encoded data, to a sender of the encoded data via a second wireless communication channel that allows transmission with lower delay than the first wireless communication channel. Information processing methods. [Explanation of symbols]

[0199] 100 Image transmission system, 111 Image encoding device, 112 Image decoding device, 121 Wireless network, 122 Wireless network, 211 Encoding unit, 212 Communication unit, 213 Encoding control unit, 221 Data transmission unit, 222 Network status monitoring unit, 223 Error information monitoring unit, 255 Encoding unit, 262 Prediction unit, 271 Inter prediction unit, 272 Intra prediction unit, 311 Communication unit, 312 Decoding control unit, 313 Decoding unit, 321 Data reception unit, 322 Reception error detection unit, 323 Error information transmission unit, 352 Decoding unit, 359 Prediction unit, 501 Wireless network, 511 Downlink, 512 Uplink, 541 5G network, 551 and 552 Network slice, 571 Wireless Networks, 572 Wireless Networks, 900 Computers

Claims

1. an error information acquisition unit that acquires error information transmitted via a second wireless communication channel that allows transmission with lower delay than the first wireless communication channel from a receiving device that receives encoded data of a moving image transmitted via a first wireless communication channel; an encoding control unit that controls encoding of the video based on the error information acquired by the error information acquisition unit; Equipped with When encoding each frame of the moving image, a part of the frame is set as an intra region and the moving image is intra-encoded; the position of the intra region is moved in a predetermined direction for each frame so as to cycle every predetermined number of frames; When the error information acquisition unit acquires the error information, the encoding control unit returns the position of the intra region to an initial position in the cycle. Information processing device.

2. the intra region is a partial region of the frame that is configured by a plurality of blocks aligned in the vertical direction of the frame, The position of the intra region is moved rightward for each frame, with the left edge of the frame being the initial position; When the error information acquisition unit acquires the error information, the encoding control unit returns the position of the intra region to the left end of the frame. The information processing device according to claim 1 .

3. The error information includes information indicating an error in receiving the encoded data. The information processing device according to claim 1 .

4. The error information includes information indicating an error that occurred during decoding of the encoded data. The information processing device according to claim 1 .

5. further comprising an encoding unit that encodes the video and generates the encoded data; The encoding control unit controls the encoding unit based on the error information acquired by the error information acquisition unit. The information processing device according to claim 1 .

6. further comprising a wireless communication path state monitoring unit that monitors a state of the first wireless communication path; The encoding control unit further controls encoding of the moving image based on the state of the first wireless communication channel monitored by the wireless communication channel state monitoring unit. The information processing device according to claim 1 .

7. The first wireless communication channel and the second wireless communication channel are communication channels of different frequency bands. The information processing device according to claim 1 .

8. The first wireless communication channel is a communication channel that satisfies the requirements of eMBB (enhance Mobile Broadband) in a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union, The second wireless communication path is a communication path that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of the wireless communication system. The information processing device according to claim 7 .

9. the first wireless communication path is a downlink in the same frequency band as the second wireless communication path, and is a wireless communication path that satisfies the requirements of eMBB (enhance Mobile Broadband) in a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard established by the International Telecommunication Union; The second wireless communication path is an uplink in the same frequency band as the first wireless communication path and is a wireless communication path that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of the wireless communication system. The information processing device according to claim 1 .

10. The first wireless communication path is a network slice different from the second wireless communication path, and is a wireless communication path that satisfies the requirements of eMBB (enhance Mobile Broadband) in a wireless communication system that satisfies the IMT (International Mobile Telecommunications)-2020 standard defined by the International Telecommunication Union, The second wireless communication path is the network slice different from the first wireless communication path and is a wireless communication path that satisfies requirements of Ultra Reliable Low Latency Communication (URLLC) of the wireless communication system. The information processing device according to claim 1 .

11. The first wireless communication channel and the second wireless communication channel are communication channels of different wireless communication standards. The information processing device according to claim 1 .

12. The first wireless communication channel is a wireless communication channel conforming to the IMT (International Mobile Telecommunications)-Advanced standard established by the International Telecommunications Union, a wireless communication channel conforming to LTE (Long Term Evolution) established by 3GPP (Third Generation Partnership Project), or a wireless communication channel conforming to IEEE (Institute of Electrical and Electronics Engineers) 80 It is a wireless communication channel using the 2.11 standard, The second wireless communication channel is a wireless communication channel that satisfies the requirements of URLLC (Ultra Reliable Low Latency Communication) of a wireless communication system that satisfies the IMT-2020 standard established by the International Telecommunication Union. The information processing device according to claim 11.

13. When encoding each frame of a moving image, a part of the frame is set as an intra area and intra-encoded, and the position of the intra area is encoded so as to rotate every predetermined number of frames and move in a predetermined direction for each frame, and when error information transmitted via a second wireless communication channel that allows transmission with lower delay than the first wireless communication channel is acquired from a receiving device that receives the encoded data transmitted via a first wireless communication channel, the position of the intra area is returned to an initial position in the rotation. Information processing methods.

Images