Data encoding apparatus, data decoding apparatus, and data processing system

Abstract

A data encoding apparatus (10) includes: a video encoding unit (11) that encodes a video to generate a video bitstream; a feature encoding unit (12) that encodes features of the video to generate a feature bitstream; and a synchronization unit (13) that synchronizes the random access point in the video bitstream with the random access point in the feature bitstream.

Description

Technical Field

[0001] The present invention relates to a data encoding device, a data decoding device, a data encoding method, a data decoding method and a data processing system for machine recognition task processing. Background Art

[0002] SC29, one of the SCs (subcommittees) of ISO / IEC JTC1, is responsible for the standardization of the coding of audio, video and multimedia information. WG11, one of the WGs (working groups), is responsible for video coding, media transmission, streaming, audio coding, etc.

[0003] WG11 is considering standardizing a video compression method specifically for machine learning called video coding for machines (VCM). VCM has the functions of video coding suitable for machine recognition task processing and feature coding for task processing. Non-patent document 1 discloses a use case of VCM.

[0004] Fig.17 is a block diagram illustrating a data processing system using a VCM. Fig.17 The data processing system shown includes a VCM encoder 300 and a VCM decoder 400. For example, the VCM encoder 300 and the VCM decoder 400 are communicatively connected via a transmission line. Take the example that the VCM encoder 300 generates a video bitstream and a feature bitstream related to feature A for some machine processing task A.

[0005] Fig.17 The arrows in the block diagram indicate the direction of signal (data) flow in a direct manner, but do not exclude bidirectionality. This is also true for the other block diagrams.

[0006] The VCM encoder 300 includes a video encoder 301 , a feature extractor 302 , and a feature encoder 303 .

[0007] The video encoder 301 encodes each video frame at a time in the digitized video signal (see Fig.17 ) to generate the video bitstream.

[0008] The feature extractor 302 extracts a feature A of the video signal from the video frame each time. The feature encoder 303 encodes the feature A to generate a feature bitstream.

[0009] The VCM decoder 400 includes a video decoder 401 and a feature decoder 402 .

[0010] The video decoder 401 decodes the video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing or visual confirmation. The decoded video signal can be used for task X, which is a machine processing task different from task A.

[0011] The feature decoder 402 decodes the feature bitstream to generate feature A (decoded feature A). The decoded feature A is used for task A processed by the machine.

[0012] Reference List

[0013] Non-patent literature

[0014] Non-patent document 1: “Use cases and requirements for Video Coding for Machines”, ISO / IEC JTC1 / SC29 / WG11w19365, April 2020

[0015] Non-Patent Document 2: Keiichi Chono et al., "Reduced-reference image quality assessment using distributed source coding", IEEE International Multimedia Conference and Exposition, June 2008 Summary of the invention

[0016] Technical issues

[0017] Background of the Invention There is no guarantee of simultaneous random access to the video bitstream and the feature bitstream at a given time.

[0018] It is therefore an object of the present invention to guarantee simultaneous random access to a video bitstream and a feature bitstream.

[0019] Solving the problem

[0020] According to the present invention, the data encoding device includes: a video encoding device, which is used to encode the video to generate a video bit stream; a feature encoding device, which is used to encode the features of the video to generate a feature bit stream; and a synchronization device, which is used to synchronize the random access points in the video bit stream with the random access points in the feature bit stream.

[0021] According to the present invention, the data decoding device includes: a video decoding device, which is used to receive a video bit stream generated by encoding a video and in which a random access point is set to decode the video bit stream; and a feature decoding device, which is used to receive a feature bit stream generated by encoding features of a video and in which a random access point that is temporally aligned with the random access point in the video bit stream is set to decode the feature bit stream.

[0022] The data encoding method according to the present invention includes encoding a video to generate a video bitstream, encoding a feature of the video to generate a feature bitstream, and synchronizing a random access point in the video bitstream with a random access point in the feature bitstream.

[0023] The data decoding method according to the present invention includes: receiving a video bit stream generated by encoding a video and in which a random access point is set to decode the video bit stream; and receiving a feature bit stream generated by encoding a feature of a video and in which a random access point that is temporally aligned with a random access point in the video bit stream is set to decode the feature bit stream.

[0024] The data encoding program according to the present invention enables a computer to execute the process of encoding a video to generate a video bitstream, the process of encoding a feature of the video to generate a feature bitstream, and the process of synchronizing a random access point in the video bitstream with a random access point in the feature bitstream.

[0025] Beneficial effects of the present invention

[0026] According to the present invention, simultaneous random access to the video bitstream and the feature bitstream is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] [ Figure 1 ]It depicts an illustrative diagram showing an example of the relationship between a video bitstream and a feature bitstream in the time direction.

[0028] [ Figure 2 ] which depicts a block diagram of a data processing system illustrating a first example embodiment.

[0029] [ Figure 3 ] which depicts a flowchart illustrating the operation of the VCM encoder in the first example embodiment.

[0030] [ Figure 4 ] which depicts a flowchart illustrating the operation of the VCM decoder in the first example embodiment.

[0031] [ Figure 5 ]It depicts an illustrative diagram showing an example of the relationship between a video bitstream and a feature bitstream in the time direction.

[0032] [ Figure 6 ] which depicts a block diagram of a data processing system illustrating a second example embodiment.

[0033] [ Figure 7 ] which depicts a flowchart illustrating the operation of the VCM encoder in the second example embodiment.

[0034] [ Figure 8] which depicts a flowchart illustrating the operation of the VCM decoder in the second example embodiment.

[0035] [ Fig. 9 ]It depicts an illustrative diagram showing an example of the relationship between a video bitstream and a feature bitstream in the time direction.

[0036] [ Fig.10 ] which depicts an illustrative diagram showing a first variation of a data processing system.

[0037] [ Fig.11 ]It depicts an illustrative diagram showing the encoding of feature frame data using error correction codes.

[0038] [ Fig.12 ]It depicts an illustrative diagram showing the decoding of feature frame data using error correction codes.

[0039] [ Fig.13 ] which depicts a block diagram illustrating a specific configuration example of a data processing system.

[0040] [ Fig.14 ]It depicts a block diagram showing an example configuration of an information processing system.

[0041] [ Fig.15 ]It depicts a block diagram showing the main parts of a data encoding device.

[0042] [ Fig.16 ]It depicts a block diagram showing the main parts of the data decoding device.

[0043] [ Fig.17 ] which depicts a block diagram of a data processing system showing the use of VCM. DETAILED DESCRIPTION

[0044] Hereinafter, exemplary embodiments of the present invention will be explained with reference to the accompanying drawings.

[0045] Figure 1 : is an explanatory diagram showing an example of the relationship between a video bit stream (video frame data) and a feature bit stream (feature frame data) in the time direction. Figure 1 In the figure, arrows indicate frame dependencies. Circles indicate frames that have no dependencies on other frames. In other words, the circled frames are frames where the encoding and decoding processes are closed within the frame. In the time direction, the position where such a frame is located is called a random access point. Figure 1 , the width of the rectangle indicating the frame corresponds to the amount of data. For example, the frame at a random access point in a video bitstream is an encoded I (Intra-coded) video frame.

[0046] exist Figure 1In the example shown in , the random access points in the video bitstream are not synchronized with the random access points in the feature bitstream. That is, the random access points in the video bitstream are not synchronized in time with the random access points in the feature bitstream. Synchronization of random access points means that the two random access points occur at the same time. Simultaneous random access means that the temporal positions of the random access points in the two bitstreams are the same.

[0047] When the random access points in the video bitstream are not aligned with the random access points in the feature bitstream, for example, when the video decoder 401 has regenerated the frame at the random access point (e.g., Figure 1 At time t+2 shown in ), the machine recognition task using feature A cannot be started.

[0048] There is a time when two random access points happen to coincide. In this case, the overall data volume (total data volume) at the random access point increases. In other words, the data transfer volume increases instantaneously.

[0049] Next, predictive coding will be explained. Predictive coding includes intra-frame prediction and inter-frame prediction. First, intra-frame prediction and inter-frame prediction are explained. First, intra-frame prediction and inter-frame prediction are explained for video bitstream.

[0050] Intra-frame prediction is the prediction of an image of a reconstructed video frame whose display time is different from the video frame when encoding. Hereinafter, a video block encoded based on intra-frame prediction is referred to as an intra-frame decoded image block. Inter-frame prediction is the prediction of an image of a reconstructed video frame whose display time is different from the video frame when encoding. Hereinafter, a video block encoded based on inter-frame prediction is referred to as an inter-frame decoded image block. Inter-frame prediction is also called motion compensated prediction.

[0051] A coded video frame consisting only of intra-frame decoded image blocks is called an I video frame. Another coded video frame is called a non-I video frame. As non-I video frames, there are P (predicted) video frames and B (bidirectionally predicted) video frames. B video frames can include inter-frame decoded image blocks that use one reconstructed video frame for inter-frame prediction and inter-frame decoded image blocks that use two reconstructed video frames for inter-frame prediction at the same time.

[0052] The above concepts of intra prediction and inter prediction can also be applied to feature bitstreams.

[0053] In the case of a feature bitstream, prediction of a feature without using a video frame whose display time is different from the time to be encoded is called intra-frame prediction of the feature. In the following, features encoded based on intra-frame prediction are called intra-frame decoded features. Prediction of a feature using a video frame whose display time is different from the time of encoding is called inter-frame prediction of the feature. In the following, features encoded based on inter-frame prediction are called inter-frame decoded features.

[0054] A frame consisting of only intra-frame decoding features is called an I-feature frame. Another I-feature frame is called a non-I-feature frame.

[0055] Example Embodiment 1.

[0056] Figure 2 is a block diagram illustrating a data processing system of the first exemplary embodiment. Figure 2 The data processing system shown includes a VCM encoder 100 and a VCM decoder 200. For example, the VCM encoder 100 and the VCM decoder 200 are communicatively connected via a transmission line. In this exemplary embodiment, it is assumed that the VCM encoder 100 generates a video bitstream and a feature bitstream associated with feature A for a certain machine processing task A.

[0057] The VCM encoder 100 includes a video encoder 101 , a feature extractor 102 , a feature encoder 103 , and a controller 104 .

[0058] The video encoder 101 encodes each video frame of the digitized video signal using the video frame type (I video frame type or non-I video frame type) supplied by the controller 104 to generate a video bitstream. For example, the video encoder 101 performs the encoding process based on the H.266 / VVC (Versatile Video Coding) standard. The video encoder 101 may perform the encoding process based on other standards such as the H.265 / HEVC (High Efficiency Video Coding) standard or the H.264 / AVC (Advanced Video Coding) standard.

[0059] The feature extractor 102 extracts a feature A of a video signal from a video frame at a time. The feature encoder 103 encodes the feature A using a feature frame type (I feature frame type or non-I feature frame type) supplied from the controller 104 to generate a feature bitstream.

[0060] The controller 104 determines the video frame type (VideoFrameType) and the feature frame type (FeatureFrameType) to be encoded by the video encoder 101 and the feature encoder 103, respectively. The controller 104 supplies the video frame type as a control signal to the video encoder 101. The controller 104 also outputs the feature frame type as a control signal to the feature encoder 103.

[0061] The controller 104 synchronizes the output timing of the control signal indicating the I video frame and the output timing of the control signal indicating the I feature frame, thereby ensuring simultaneous random access to the video bitstream and the feature bitstream. For example, the controller 104 synchronizes them when a predetermined period has passed. Ensuring simultaneous random access to the video bitstream and the feature bitstream means, for example, ensuring that a random access point in the feature bitstream and a random access point in the video bitstream coincide in time.

[0062] Synchronizing the random access point of the video frame with the random access point of the feature frame means that the random access points of the two are aligned in time. Specifically, for example, for a video frame input at a certain time, the video encoder 101 responds to an instruction from the controller 104 and encodes based on the I video frame type, and at the same time, the feature encoder 103 encodes based on the I feature frame type.

[0063] The VCM decoder 200 includes a video decoder 201 and a feature decoder 202 .

[0064] The video decoder 201 decodes the video bitstream to generate a decoded video signal. The decoded video signal is for human viewing or visual confirmation. The decoded video signal can be used for task X, which is a machine processing task different from task A (see Fig.17 ).

[0065] The feature decoder 202 decodes the feature bitstream to generate a decoded feature A. The decoded feature A is used for the task A processed by the machine (see Fig.17 ).

[0066] Next, the operation of the VCM encoder 100 in the first exemplary embodiment will be explained. Figure 3 is a flow chart showing the operation of the VCM encoder. Figure 3 The process shown.

[0067] The controller 104 in the VCM encoder 100 determines the video frame type and the feature frame type (step S100). As described above, for example, the controller 104 periodically determines that the video frame type is an I video frame type and the feature frame type is an I feature frame type. When a period is used to determine the frame type, the controller 104 determines that the video frame type is a non-I video frame type and the feature frame type is a non-I feature frame type when a predetermined period has not passed.

[0068] Then, the controller 104 outputs a control signal indicating the video frame type to the video encoder 101 , and outputs a control signal indicating the feature frame type to the feature encoder 103 .

[0069] The video encoder 101 encodes a video frame using a video frame type indicated by a control signal from the controller 104 to generate a video bitstream (step S101 ).

[0070] The feature extractor 102 extracts features from the video frame (step S102 ). The feature encoder 103 encodes the features using the feature frame type indicated by the control signal from the controller 104 to generate a feature bitstream (step S103 ).

[0071] For example, the video encoder 101 sends the video bitstream to the transmission line (step S104 ). For example, the feature encoder 103 sends the feature bitstream to the transmission line (step S105 ).

[0072] Next, the operation of VCM decoder 200 in the first exemplary embodiment will be explained. Figure 4 is a flow chart showing the operation of the VCM decoder 200.

[0073] The video decoder 201 in the VCM decoder 200 decodes the received video bitstream to generate a decoded video signal (step S201 ). The feature decoder 202 decodes the feature bitstream to generate a decoded feature (step S202 ).

[0074] The video decoder 201 outputs the decoded video signal (step S203 ). The feature decoder 202 outputs the decoded feature (step S204 ).

[0075] Figure 5 1 is an explanatory diagram showing an example of the relationship between the video bitstream and the feature bitstream in the time direction in the first exemplary embodiment. Figure 5 In FIG. 1 , an arrow indicates a frame at a random access point. A frame at a random access point is a frame that has no dependency on other frames. The width of a rectangle indicating a frame corresponds to the amount of data.

[0076] like Figure 5 As shown, in this exemplary embodiment, since simultaneous random access to the video bitstream and the feature bitstream is guaranteed, the random access points in the video bitstream and the random access points in the feature bitstream are aligned.

[0077] As a result, a system using a video bitstream and a feature bitstream output by the VCM decoder 200 can immediately initiate a machine recognition task based on features concurrent with a certain video frame.

[0078] Example 2.

[0079] Figure 6 is a block diagram illustrating a data processing system of a second exemplary embodiment. Figure 6 The data processing system shown includes a VCM encoder 110 and a VCM decoder 210. For example, the VCM encoder 110 and the VCM decoder 210 are communicatively connected via a transmission line. In this exemplary embodiment, the case where the VCM encoder 110 generates a video bitstream and a feature bitstream related to feature A for a task A processed by a certain machine is also taken as an example.

[0080] The VCM encoder 110 includes a video encoder 101, a feature extractor 102, a side information generator 105, a feature encoder 106, and a controller 104. The functions of the video encoder 101, the feature extractor 102, and the controller 104 are the same as those in the first exemplary embodiment.

[0081] The side information generator 105 generates information related to the feature A extracted by the feature extractor 102 from the reconstructed video signal generated inside the video encoder 101. Hereinafter, the information related to the feature A extracted by the feature extractor 102 is referred to as side information A'.

[0082] The reconstructed video signal in the video encoder 101 is a video signal (video frame) generated by a decoding function included in the video encoder 101. As an example, the auxiliary information generator 105 generates the auxiliary information A' through the same process as that performed by the feature extractor 102.

[0083] In addition to the functions of feature encoder 103 in the first exemplary embodiment, feature encoder 106 has the following functions: That is, when encoding an I feature frame type, feature encoder 106 reduces the data amount of a feature frame by encoding the feature frame based on the correlation between feature A and auxiliary information A'.

[0084] Take the case where the auxiliary information generator 105 generates auxiliary information A' from the reconstructed video signal in the video encoder 101 by the same process as that performed by the feature extractor 102 as an example. In this case, the auxiliary information generator 105 extracts features from the reconstructed video signal as auxiliary information A'. The feature encoder 106 can reduce the amount of data of the feature frame by predictively encoding the feature A using the features (auxiliary information A') extracted by the auxiliary information generator 105. This is because the feature A of the input video signal and the feature of the reconstructed video signal are similar. The higher the bit rate of the video bitstream, the higher the similarity between the feature A of the video signal and the reconstructed video signal.

[0085] The feature encoder 106 may also use the correlation between the feature A and the auxiliary information A' to encode the feature frame when encoding a non-I feature frame type.

[0086] The VCM decoder 210 includes a video decoder 201, an auxiliary information generator 203, and a feature decoder 204. The video decoder 201 is the same as in the first exemplary embodiment. That is, the video decoder 201 decodes the video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing and visual confirmation. The decoded video signal can be used for task X, which is a machine processing task different from task A (see Fig.17 ).

[0087] The auxiliary information generator 203 operates in the same manner as the auxiliary information generator 105 in the VCM encoder 110 to generate auxiliary information A' when a video frame of the I video frame type is decoded. However, the auxiliary information generator 105 generates the auxiliary information A' from the reconstructed video signal, while the auxiliary information generator 203 generates the auxiliary information A' from the decoded video signal.

[0088] In addition to the functions of feature decoder 202 in the first exemplary embodiment, feature decoder 204 has the following functions: That is, when decoding a feature frame of the I feature frame type, feature decoder 204 decodes the feature frame based on the correlation between feature A and auxiliary information A'.

[0089] The feature decoder 204 may decode the feature frame by the reverse process of the predictive encoding performed by the feature encoder 106. For example, the feature decoder 204 predictively decodes the features decoded from the feature bitstream using the features extracted by the auxiliary information generator 105.

[0090] Next, the operation of the VCM encoder 110 in the second exemplary embodiment will be explained. Figure 7 is a flow chart showing the operation of the VCM encoder 110. Figure 7 The process shown.

[0091] The process of steps S100 - S102 is the same as that in the first exemplary embodiment.

[0092] In this example embodiment, the side information generator 105 generates side information A′ from a reconstructed video signal generated internally by the video encoder 101 (step S301 ).

[0093] The feature encoder 106 predictively encodes the feature A using the feature (side information A′) extracted by the side information generator 105 to generate a feature bitstream (step S302 ).

[0094] As in the case of the first exemplary embodiment, the video encoder 101 sends the video bitstream to, for example, a transmission line (step S104). As in the case of the first exemplary embodiment, the feature encoder 103 sends the feature bitstream to a transmission line, for example (step S105).

[0095] Next, the operation of VCM decoder 210 in the second exemplary embodiment will be explained. Figure 8 is a flow chart illustrating the operation of the VCM decoder 210.

[0096] As in the first exemplary embodiment, the video decoder 201 decodes the received video bitstream to generate a decoded video signal (step S201). When a video frame of the I video frame type is decoded, the auxiliary information generator 203 generates auxiliary information A' from the decoded video signal (step S401).

[0097] The feature decoder 204 decodes the feature bitstream and generates decoded features, as in the first exemplary embodiment (step S402). However, as described above, in this exemplary embodiment, the feature decoder 204 decodes the feature frame using the correlation between the feature A and the auxiliary information A' when decoding a feature frame of the I feature frame type.

[0098] The video decoder 201 outputs the decoded video signal (step S203 ). The feature decoder 204 outputs the decoded feature (step S204 ).

[0099] Fig. 9 : is an explanatory diagram showing an example of the relationship between the video bitstream and the feature bitstream in the time direction in the second exemplary embodiment. Fig. 9 , arrows (except for arrows related to "reduced data amount") indicate frames at random access points. Frames at random access points are frames that have no dependencies with other frames. The width of a rectangle indicating a frame corresponds to the data amount.

[0100] like Fig. 9 As shown, also in this exemplary embodiment, since simultaneous random access to the video bitstream and the feature bitstream is guaranteed, the random access points in the video bitstream and the random access points in the feature bitstream are synchronized.

[0101] Furthermore, in this exemplary embodiment, since the feature frame is encoded using the correlation between the feature A and the auxiliary information A', the data amount of the feature frame at the random access point is reduced. As a result, an increase in the total data amount at the random access point is suppressed.

[0102] [Variant 1]

[0103] In the first and second exemplary embodiments, as Fig.10 As shown, the video encoder 101 and the feature encoders 103 and 106 may add identification headers to identify random access points at the beginning of the video frame data and the feature frame data.

[0104] The identification header of the video frame data includes information indicating whether the frame is an I video frame, a P video frame or a B video frame. The identification header of the feature frame data includes information indicating whether the frame is an I feature frame or a non-I feature frame.

[0105] In the case of the second exemplary embodiment, the identification header of the feature frame data further includes information indicating whether the data is encoded using auxiliary information.

[0106] By adding the above identification header, the decoding side can check the random accessibility without decoding the content of the video frame data and the feature frame data. Therefore, the interoperability between the encoding side and the decoding side is enhanced.

[0107] [Variant 2]

[0108] In the second exemplary embodiment, the feature extracted from the video frame of the reconstructed video signal by the same process performed by the feature extractor 102 is used as the auxiliary information A'. However, the encoded data itself can also be used as the auxiliary information. Hereinafter, the feature extracted by the same process as that performed by the feature extractor 102 is denoted as the feature A'.

[0109] For example, an error correction code can be applied as described in Non-Patent Document 2.

[0110] Let the encoded data of the feature A extracted from the video frame by the feature extractor 102 at a certain time t of the input video signal be Y(A, t). Additionally, let the encoded data of the feature A' be the auxiliary information Y(A', t).

[0111] The encoding of the feature A' is shown in Fig.11 That is, the feature encoder 106 sets a syndrome to the encoded data, which is obtained by multiplying the encoded data Y(A, t) to be transmitted by a parity-check matrix.

[0112] As shown in Fig.11 , the feature encoder 106 determines the number of columns M of the parity-check matrix based on the correlation between the encoded data Y(A, t) (N-bit data) to be transmitted and the auxiliary information Y(A', t). The feature encoder 106 generates a syndrome (M-bit data) by multiplying the encoded data Y(A, t) by a parity-check matrix of N rows and M columns. Then, the feature encoder 106 outputs the syndrome as the feature frame data. It should be noted that if M < N, the data amount is reduced.

[0113] The decoding of the feature A' is shown in Fig.12 That is, the feature decoder 204 can obtain the decoded feature by using, as the decoded value, Y(A, t) that is corrected based on the relationship between the auxiliary information Y(A', t), the parity-check matrix, and the syndrome (M-bit data).

[0114] The value of each element of the check matrix can be pre-shared between the encoding end and the decoding end to further reduce the data volume of each feature frame data. In this case, the feature encoder 106 only needs to include the ID of the check matrix for the number of columns M in the identification header.

[0115] In each of the above exemplary embodiments and each of the above modifications, one feature is used as an example to simplify the explanation. However, the present invention can be applied to a plurality of features.

[0116] When encoding and decoding feature data using an error correction code in the second variant is applied to multiple features, instead of calculating the syndrome independently for each feature frame data, the syndrome can be calculated for all feature frame data concatenated together. In this case, the effect is to suppress the overhead caused by applying a check matrix with a small number of columns when the feature frame data of each feature is short.

[0117] Fig.13 is a block diagram showing a specific configuration example of the data processing system. Fig.13 The data processing system shown is a system of a data encoding device 10 (corresponding to the VCM encoder 100 in the first exemplary embodiment or the VCM encoder 110 in the second exemplary embodiment) and a data decoding device 20 (corresponding to the VCM decoder 200 or the VCM decoder 210 in the first exemplary embodiment) connected via a transmission line (wireless or wired transmission line) 30.

[0118] In the data processing system, the data encoding device 10 can generate a video bitstream and a feature bitstream having the features described in each of the above exemplary embodiments. In addition, in the data processing system, the data decoding device 20 can decode the video bitstream and the feature bitstream using the features described in the above exemplary embodiments.

[0119] Although the above-described exemplary embodiments can be configured by hardware, they can be realized by computer programs.

[0120] Fig.14 The information processing system shown includes a processor 1001 such as a central processing unit (CPU), a program memory 1002, a storage medium 1003 for storing video data, and a storage medium 1004 for storing a bit stream. The storage medium 1003 and the storage medium 1004 may be separate storage media or storage areas in the same storage medium. A magnetic storage medium such as a hard disk may be used as the storage medium.

[0121] In the information processing system, the program memory 1002 stores a program (a data encoding program or a data decoding program) to realize the function of each block of each indication in the above-described exemplary embodiments.

[0122] The processor 1001 executes the functions of the data encoding device 10 , VCM encoders 100 , 110 , data decoding device 20 , and VCM decoders 200 , 210 shown in each exemplary embodiment by following a program stored in the program memory 1002 .

[0123] For example, the processor 1001 performs processing according to a data encoding program (specifically, a VCM encoding program) to implement Figure 2 The functions of each block in the VCM encoder 100 shown in FIG. 1 are implemented to realize the functions of the VCM encoder 100. For example, the processor 1001 performs processing according to a data decoding program (specifically, a VCM decoding program) to implement the following. Figure 2 The functions of each block in the VCM decoder 200 are shown, thereby realizing the functions of the VCM decoder 200.

[0124] At least program memory 1002 is a non-transitory computer readable medium. However, the program may be stored in various types of transient computer readable media. The program is supplied to a transient computer readable medium via, for example, a wired or wireless communication channel (ie, via electrical signals, optical signals, or electromagnetic waves).

[0125] Fig.15 is a block diagram showing the main parts of a data encoding device. Fig.15 The data encoding device 10 shown includes: a video encoding unit (video encoding device) 11 (in an exemplary embodiment, implemented by a video encoder 101), which encodes a video to generate a video bitstream; a feature encoding unit (feature encoding device) 12 (in an exemplary embodiment, implemented by a feature encoder 103 or a feature encoder 106), which encodes features of the video to generate a feature bitstream; and a synchronization unit (synchronization device) 13 (in an exemplary embodiment, implemented by a controller 104), which synchronizes a random access point in a video bitstream with a random access point in a feature bitstream.

[0126] The data encoding device 10 may include: a first feature extraction unit (first feature extraction device) (in an exemplary embodiment, implemented by the auxiliary information generator 105), which extracts features of the input video; and a second feature extraction unit (second feature extraction device) (in an exemplary embodiment, implemented by the feature encoder 106), which extracts features of a video generated by a video encoding device through a decoding process included in the encoding process, wherein the feature encoding unit 12 encodes the features based on the correlation between the features extracted from the input video and the features extracted from the video generated by the decoding process.

[0127] Fig.16 is a block diagram showing the main parts of the data decoding device. Fig.16The data decoding device 20 shown in the figure includes: a video decoding unit (video decoding device) 21, which receives a video bit stream generated by encoding a video and in which a random access point is set, so as to decode the video bit stream; and a feature decoding unit (feature decoding device) 22, which receives a feature bit stream generated by encoding a feature of a video and in which a random access point that is temporally aligned with a random access point in the video bit stream is set, so as to decode the feature bit stream.

[0128] The data decoding device 20 may include a feature extraction unit (feature extraction means) (in this example embodiment, implemented by the auxiliary information generator 203), which extracts features of a decoded video obtained by decoding a video bit stream, wherein the feature decoding unit 22 is used to perform decoding based on a correlation between features obtained by decoding the feature bit stream and features extracted from the decoded video.

[0129] Part or all of the above exemplary embodiments may also be described as, but not limited to, the following supplementary notes.

[0130] (Supplementary Note 1) A data encoding device comprising:

[0131] A video encoding device, which is used to encode a video to generate a video bit stream;

[0132] A feature encoding device, which is used to encode the features of the video to generate a feature bit stream, and

[0133] The synchronization device is used to synchronize the random access points in the video bit stream with the random access points in the feature bit stream.

[0134] (Supplementary Note 2) Data encoding device according to Supplementary Note 1, comprising:

[0135] A first feature extraction device, which is used to extract features of an input video; and

[0136] The second feature extraction device is used to extract the features of the video generated by the video encoding device through the decoding process included in the encoding process, wherein

[0137] The feature encoding means encodes the features based on the correlation between the features extracted from the input video and the features extracted from the video generated by the decoding process.

[0138] (Supplementary Note 3) A data encoding device according to Supplementary Note 2, wherein:

[0139] The feature encoding device predictively encodes the features extracted from the input video using the features extracted from the video generated by the decoding process.

[0140] (Supplementary Note 4) A data decoding device comprising:

[0141] a video decoding device for receiving a video bit stream generated by encoding a video and in which a random access point is set, to decode the video bit stream; and

[0142] A feature decoding device is used to receive a feature bit stream generated by encoding features of a video and in which random access points that are temporally aligned with random access points in a video bit stream are set, so as to decode the feature bit stream.

[0143] (Supplementary Note 5) Data decoding device according to Supplementary Note 4, including

[0144] A feature extraction device for extracting features of a decoded video obtained by decoding a video bit stream;

[0145] The feature decoding device performs decoding based on the correlation between the feature obtained by decoding the feature bit stream and the feature extracted from the decoded video.

[0146] (Supplementary Note 6) A data decoding device according to Supplementary Note 5, wherein:

[0147] The feature decoding means predictively decodes the features decoded from the feature bitstream using the features extracted from the feature extraction means.

[0148] (Supplementary Note 7) A data encoding method comprising:

[0149] Encode the video to generate a video bitstream;

[0150] encoding the features of the video to generate a feature bitstream; and

[0151] Synchronize random access points in the video bitstream with random access points in the feature bitstream.

[0152] (Supplementary Note 8) Data encoding method according to Supplementary Note 7, including

[0153] Extract features of the input video,

[0154] extracting features of a video generated by a decoding process included in a video encoding process; and

[0155] Features are encoded based on the correlation between features extracted from the input video and features extracted from the video generated by the decoding process.

[0156] (Supplementary Note 9) A data decoding method comprising:

[0157] receiving a video bitstream generated by encoding a video and in which a random access point is set, to decode the video bitstream;

[0158] A feature bitstream generated by encoding a feature of a video and in which random access points temporally aligned with random access points in a video bitstream are set is received to decode the feature bitstream.

[0159] (Supplementary Note 10) The data decoding method according to Supplementary Note 9, comprising:

[0160] extracting features of a decoded video obtained by decoding a video bitstream;

[0161] The decoding is performed based on the correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video.

[0162] (Supplementary Note 11) A data processing system comprising:

[0163] The data encoding device according to any one of Supplementary Note 1 to Supplementary Note 3; and

[0164] The data decoding device according to any one of Supplementary Note 4 to Supplementary Note 6.

[0165] (Supplementary Note 12) A data encoding program for causing a computer to execute:

[0166] The process of encoding a video to generate a video bitstream;

[0167] The process of encoding the features of a video to generate a feature bitstream; and

[0168] The process of synchronizing random access points in the video bitstream with random access points in the feature bitstream.

[0169] (Supplementary Note 13) A data decoding program for causing a computer to execute:

[0170] A process of receiving a video bit stream generated by encoding a video and in which a random access point is set, to decode the video bit stream;

[0171] A process of receiving a feature bitstream generated by encoding a feature of a video and in which a random access point is arranged to be time-aligned with a random access point in a video bitstream, to decode the feature bitstream.

[0172] Although the invention of the present application has been described above with reference to the exemplary embodiments, the present invention is not limited to the exemplary embodiments described above. It can be understood by those skilled in the art that various changes can be made to the configuration and details of the present invention within the scope of the present invention.

[0173] Reference numerals list

[0174] 10 Data encoding equipment

[0175] 11 Video Encoding Unit

[0176] 12 Feature Encoding Unit

[0177] 13 Synchronous unit

[0178] 20 Data decoding equipment

[0179] 21 Video decoding unit

[0180] 22 Feature decoding unit

[0181] 30 Transmission Line

[0182] 100, 110 VCM encoder

[0183] 101 Video Encoder

[0184] 102 Feature Extractor

[0185] 103, 106 Feature Encoder

[0186] 104 Controller

[0187] 105 Auxiliary Information Generator

[0188] 200, 210 VCM decoder

[0189] 201 Video Decoder

[0190] 202, 204 Feature decoder

[0191] 203 Auxiliary Information Generator

[0192] 1001 Processor

[0193] 1002 Program Memory

[0194] 1003, 1004 Storage media

Claims

1. A data encoding device, comprising: A video encoding device, the video encoding device is used to encode a video to generate a video bit stream; a feature encoding device, the feature encoding device being used to encode the features of the video to generate a feature bit stream, and A synchronization device is used to synchronize the random access points in the video bit stream with the random access points in the feature bit stream.

2. The data encoding device according to claim 1, comprising: A first feature extraction device, the first feature extraction device is used to extract features of an input video; as well as a second feature extraction device, the second feature extraction device being used to extract features of a video generated by a decoding process included in the encoding process performed by the video encoding device, wherein The feature encoding means encodes the features based on correlation between the features extracted from the input video and features extracted from the video generated by the decoding process.

3. A data decoding device, comprising: a video decoding device for receiving a video bit stream generated by encoding a video and in which a random access point is set, to decode the video bit stream, and A feature decoding device is used to receive a feature bit stream generated by encoding the features of the video and in which random access points that are temporally aligned with the random access points in the video bit stream are set, so as to decode the feature bit stream.

4. The data decoding device according to claim 3, comprising: A feature extraction device, the feature extraction device is used to extract features of a decoded video obtained by decoding the video bit stream, wherein The feature decoding means performs decoding based on a correlation between a feature obtained by decoding the feature bit stream and a feature extracted from the decoded video.

5. A data encoding method, comprising: Encode the video to generate a video bitstream; Encoding the features of the video to generate a feature bitstream; as well as Random access points in the video bitstream are synchronized with random access points in the feature bitstream.

6. The data encoding method according to claim 5, comprising: Extract features of the input video, extracting features of a video generated by a decoding process included in a video encoding process, and The features are encoded based on correlations between features extracted from the input video and features extracted from a video generated by the decoding process.

7. A data decoding method, comprising: receiving a video bitstream generated by encoding a video and having a random access point set therein, to decode the video bitstream; as well as A feature bitstream generated by encoding the feature of the video and having a random access point temporally aligned with the random access point in the video bitstream is received to decode the feature bitstream.

8. The data decoding method according to claim 7, comprising: extracting features of a decoded video obtained by decoding the video bitstream; as well as The decoding is performed based on a correlation between features obtained by decoding the feature bitstream and features extracted from the decoded video.

9. A data processing system comprising: The data encoding device according to claim 1 or claim 2; as well as A data decoding device according to claim 3 or claim 4.

10. A computer-readable storage medium for storing a data encoding program, wherein the data encoding program is used to cause a computer to execute: The process of encoding a video to generate a video bitstream; A process of encoding the features of the video to generate a feature bitstream, and The process of synchronizing random access points in the video bitstream with random access points in the feature bitstream.

Images