Information encoding device and method, information decoding device, program, and storage medium

Abstract

To provide an information encoding device and method for efficiently encoding image information and tactile information.SOLUTION: In an information encoding device, an information compression encoding part 103 includes an image information encoding part for encoding image information, an analysis part for analyzing a correlation between unencoded image information and unencoded tactile information, a pixel value conversion information generation part for generating pixel value conversion information being information for converting a pixel value in the unencoded image information into a conversion pixel value being a different pixel value on the basis of an analysis result by the analysis part, a subtracter for calculating a difference value between a tactile signal value in the tactile information and a conversion pixel value corresponding to the tactile signal value at the same spatial position, and a compression encoding part for compressing and encoding the difference value. It includes a tactile information encoding part for encoding the tactile information, and a multiplex processing part for multiplexing image encoding data encoded by the image information encoding part, the tactile encoding data encoded by the tactile information encoding part and the pixel value conversion information into one bit stream.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information encoding device that encodes image information and tactile information. [Background technology]

[0002] In recent years, there have been increasing opportunities for users to wear head-mounted displays and other devices to experience highly realistic and immersive video experiences even when they are not actually present in the actual location. Furthermore, in order to provide users with a higher sense of reality, functions that provide feedback of tactile information in addition to images and sounds are beginning to be used.

[0003] Therefore, it is necessary to transmit and store large amounts of data, including images, audio, and haptic information, more efficiently than ever before.In this regard, Patent Document 1 discloses a technology in which image information, audio information, and haptic information are each encoded using different compression methods, and then processed into a single stream in a multiplexing unit at a subsequent stage before being output. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-239430 [Non-patent literature]

[0005] [Non-Patent Document 1] Takahashi, Takashi, and 1 other person, "Deep visuo-tactile learning: Estimation of Tactile Properties from Images" [online], July 9, 2019, in IEEE International Conference on Robotics and Automation (ICRA), 2019, Internet<URL:https: / / arxiv.org / pdf / 1803.03435.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the configuration of the conventional technology, each piece of information is coded separately, so there remains a problem that it is difficult to achieve high compression performance depending on the content.

[0007] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an information encoding device that can efficiently encode image information and tactile information. [Means for solving the problem]

[0008] An information coding device according to the present invention is an information coding device that codes image information and tactile information, and is characterized by comprising: a first coding means for coding the image information; a second coding means for coding the tactile information, and the second coding means comprises an analysis means for analyzing the correlation between the uncoded image information and the uncoded tactile information; a generation means for generating pixel value conversion information, which is information for converting pixel values ​​in the uncoded image information into converted pixel values ​​that are different pixel values, based on the analysis results by the analysis means; a second coding means comprising: a calculation means for calculating a difference value between a tactile signal value in the tactile information and the converted pixel value corresponding to the same spatial position as the tactile signal value; and an encoding means for encoding the difference value; and a multiplexing means for multiplexing the image coded data coded by the first coding means, the tactile coded data coded by the second coding means, and the pixel value conversion information into a single bit stream. [Effects of the Invention]

[0009] According to the present invention, it is possible to efficiently encode image information and tactile information. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a system configuration diagram of an information compression device 100 according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the internal configuration of an information compression encoding unit 103 according to the first embodiment. [Figure 3] 4 is a flowchart showing the operation of an information compression encoding unit 103 in the first embodiment. [Figure 4] FIG. 4 is a diagram showing the correspondence between pixel information and tactile information in the first embodiment. [Figure 5] 5A and 5B are diagrams showing specific examples of pixel values ​​and haptic signal values ​​in the first embodiment. [Figure 6] 5A and 5B are diagrams showing specific examples of pixel values ​​and converted pixel values ​​according to the first embodiment. [Figure 7] 5A and 5B are diagrams showing specific examples of corresponding pixel values, haptic signal values, pixel value converted data, and difference values ​​in the first embodiment. [Figure 8] FIG. 3 is a data structure diagram of a multiplexed bitstream in the first embodiment. [Figure 9] 4 is a diagram showing an example of the format of pixel value conversion information transmitted as header data in the first embodiment. FIG. [Figure 10] FIG. 3 is a diagram showing an example of a format of haptic encoded data in the first embodiment. [Figure 11] 10A and 10B are diagrams showing specific examples of pixel values ​​and haptic signal values ​​in the second embodiment. [Figure 12] 10 is a flowchart showing the operation of an information compression encoding unit 103 in the second embodiment. [Figure 13] FIG. 11 is a conceptual diagram showing a change in an object to be coded in the third embodiment. [Figure 14] 10 is a flowchart showing the operation of an information compression encoding unit 103 in the third embodiment. [Figure 15] FIG. 11 is a diagram showing a stream structure according to the third embodiment. [Figure 16] System configuration diagram of a decoding / playback device 1600 according to the fourth embodiment [Figure 17] FIG. 16 is a functional block diagram of an information decoding unit 1603 according to the fourth embodiment. [Figure 18] 16 is a flowchart showing the operation of an information decoding unit 1603 in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (First embodiment) <System configuration> FIG. 1 is a block diagram showing an example of the system configuration of an information compression device (information encoding device) 100 according to a first embodiment of the present invention.

[0013] The system of this embodiment is configured with a camera unit 101, a tactile information acquisition unit 102, an information compression encoding unit 103, a recording unit 104, a network unit 105, a work memory 106, a CPU 107, a primary storage unit 108, a CPU bus 109, and a memory bus 110.

[0014] The camera unit 101 includes a camera unit and an optical unit equipped with a lens, an image sensor, etc. (not shown), and converts an optical signal captured through the lens into an electrical signal using the image sensor to generate, for example, a RAW image of one frame in Bayer format. After further processing such as optical correction, noise removal, shake correction, white balance correction, and color conversion on this RAW image, the image is output as image information in RGB format or YUV format to a work memory 106, which is made up of a large-capacity DRAM, etc., via a memory bus 110 and stored therein.

[0015] The tactile information acquisition unit 102 includes a piezoelectric element, such as a piezo sensor that utilizes the piezoelectric effect, and converts analog signals sensed from any or specific tactile target into electrical signals. The electrical signals are then assigned to a range that can be expressed using a predetermined number of bits, and the resulting digital signal values ​​are output as tactile information.

[0016] In this embodiment, the tactile information is the sensing of continuously changing physical quantities that humans can perceive as touch, such as hardness / softness and vibration due to pressure, as well as hot / cold, dry / humid, etc., and any one of these may be handled individually or in combination.

[0017] Alternatively, tactile information may be virtually generated from image information of a tactile object without actually performing physical tactile sensing. For example, the deep learning technology disclosed in Non-Patent Document 1 may be applied to infer tactile information from image information using a tactile model that has learned images and tactile information, and the tactile information may be acquired.

[0018] Furthermore, the tactile information handled in this embodiment is configured so that one piece of tactile information corresponds to one pixel of the image information acquired by the camera unit 101, and is handled so that the positions of the tactile information and the image information match in two-dimensional space. The tactile information generated in this manner is output to and stored in work memory 106.

[0019] The information compression encoding unit 103 reads the image information and haptic information from the work memory 106, performs compression encoding, and generates a bit stream in which the compressed data is multiplexed. The bit stream is written to the work memory 106.

[0020] The recording unit 104 reads the bit stream from the work memory 106 and writes it to a storage device such as a USB, an SD card, a hard disk drive, or a nonvolatile memory such as a flash memory.

[0021] The network unit 105 is an interface for connecting the information compression device 100 to an external device. In this embodiment, the network unit 105 mainly reads out a bit stream stored in the work memory 106, communicates with an external device via a network, and transmits the bit stream.

[0022] The above network may be the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a public line, or the like. In other words, any system that can establish transmission and reception of information with the information compression device 100 is acceptable, and is not particularly limited. The above network may be a wireless network or a wired network. Furthermore, it may include multiple different types of networks.

[0023] The CPU (Central Processing Unit) 107 controls the camera unit 101, tactile information acquisition unit 102, information compression encoding unit 103, recording unit 104, and network unit 105 that make up the system of this embodiment, such as starting, stopping, and sending interrupt notifications, via the CPU bus 109, and controls various operations of the entire information compression device 100.

[0024] The primary storage unit 108 is a storage area used as a work area or the like for the CPU 107. The primary storage unit 108 is realized by, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), or a non-volatile flash memory. For example, the CPU 107 expands and executes a control program stored in the primary storage unit 108 to realize various functions provided by the information compression device 100.

[0025] The CPU bus 109 is a control bus that connects the CPU 107 with the above-mentioned processing blocks, and may use a standardized bus format similar to that of the memory bus 110 described below, or may use a low-speed serial format such as I2C if there is sufficient processing margin. In this embodiment, the format is not particularly limited.

[0026] Memory bus 110 is a data bus that connects camera unit 101, tactile information acquisition unit 102, information compression / encoding unit 103, recording unit 104, network unit 105, and work memory 106, and is used for high-speed transfer of image data and various parameter data. The bus transfer method may be a standard bus standard such as ISA, PCI-Express, or AXI, or may be a unique bus method, and is not particularly limited in this embodiment.

[0027] <Internal configuration of the information compression coding unit> 2 is a diagram showing the internal configuration of the information compression encoding unit 103, which is a characteristic configuration of this embodiment. The internal processing of the information compression encoding unit 103 will be described below with reference to FIG.

[0028] The information compression coding unit 103 comprises an image information coding unit 201 , a haptic information coding unit 202 , and a multiplexing processing unit 209 .

[0029] The image information encoding unit 201 reads out the image data recorded by the camera unit 101 stored in the work memory 106 as the original image, performs image compression encoding based on standard specifications such as H.264 and HEVC, and writes the encoded data to the work memory 106.

[0030] In this embodiment, the image information encoding unit 201 is configured to output decoded image information temporarily generated for use as a reference image during the compression process, so-called locally decoded images, directly to the work memory 106 without retaining or discarding them within the encoding unit, and to reuse them in the haptic encoding unit 202, which will be described later. Furthermore, for so-called B pictures, which are non-reference pictures, outputting locally decoded images is not necessary for the original encoding purpose, but they are output in the same way to realize the operation of this embodiment, which will be described later.

[0031] The haptic information encoding unit 202 includes an analyzing unit 203 , a pixel value conversion information generating unit 204 , a memory 205 , a pixel value converting unit 206 , a subtractor 207 , a compression encoding unit 208 , a selector 211 , and a selector 212 .

[0032] The analysis unit 203 receives image information, which is either an original image acquired by the camera unit 101 from the work memory 106 or a decoded image processed by the image encoding unit 201, selected via the selector 211, and haptic information output from the haptic information acquisition unit 102. The analysis unit 203 then analyzes the correlation between the image information and the haptic information. The analysis result is then reported to the pixel value conversion information generation unit 204 and the selector 212, which is located downstream of the subtractor 207. The internal operation of the analysis unit 203 will be described in detail below.

[0033] Here, the selection control signal of the selector 211 is set by the CPU 107. The selection control signal can be switched mainly by software processing, for example, in units of a series of contents (hereinafter referred to as a sequence in this embodiment) from shooting and recording to stopping, triggered by a user operation (not shown).

[0034] For example, when a B picture is encoded by the image encoding unit 201 described above, the decoded image is not output to the work memory 106, and when the corresponding haptic information is encoded, the selector 211 selects the original image input. Alternatively, it is preferable to control the original image input so that lossless compression, low compression ratio mode, or the like is set as the operating mode and there is no quantization error between the original image and the decoded image or the quantization error is equal to or less than a predetermined value. This allows for flexible design and optimization of memory access in the information compression encoding unit 103.

[0035] Based on the results of analysis unit 203, pixel value conversion information generation unit 204 generates pixel value conversion information that converts the values ​​of the image information to values ​​close to the values ​​of the tactile information, and stores the information in memory 205. The detailed internal operation of pixel value conversion information generation unit 204 will be described later.

[0036] The memory 205 has a configuration similar to that of the primary storage unit 108 and the work memory 106 , and temporarily stores the information output by the pixel value conversion information generation unit 204 .

[0037] In this embodiment, the memory 205 is provided inside the haptic information encoding unit 202 for the purpose of simplifying the controllability of the haptic encoded data that is output in the subsequent compression encoding stage. However, if the conversion information exceeds the capacity of the memory 205 or if the system operation does not fail from the viewpoint of memory bandwidth, the conversion information may be stored in the work memory 106.

[0038] However, if the conversion information is stored in the work memory 106 without going through the memory 205 described above, it is read out again from the work memory 106 in the direction of a data path not shown and input to the pixel value conversion unit 206 described later.

[0039] Furthermore, the memory 205 may be configured to be provided inside the pixel value conversion information generating unit 204, and is not particularly limited in this embodiment as long as the same effect can be obtained.

[0040] The pixel value conversion unit 206 receives the image information output from the selector 211 and the pixel value conversion information read from the memory 205 as input, and performs pixel value conversion on the image information.

[0041] The subtractor 207 receives the pixel value converted data output from the pixel value conversion unit 206 and the tactile information corresponding to the same spatial position read from the work memory 106, and calculates the difference.

[0042] The selector 212 uses the signal indicating whether or not there is a correlation notified from the analysis unit 203 as a selection control signal, and if there is a correlation, it outputs the difference input from the subtractor 207, and if there is no correlation, it outputs the tactile information read out from the work memory 106.

[0043] The compression encoding unit 208 receives the differential value or haptic information value output from the selector 212 at the previous stage and performs compression encoding.

[0044] The compression encoding algorithm executed by the compression encoding unit 208 is a hybrid method that reduces spatial and temporal redundancy using frequency conversion and frame prediction, based on standardized specifications such as H.264 and HEVC, just like the image information encoding unit 201.

[0045] In this embodiment, the image information encoding unit 201 and the compression encoding unit 208 inside the haptic information encoding unit 202 are realized as different component configurations. However, if the individual encoding tools, such as prediction processing, frequency conversion, quantization, and entropy encoding processing inside the encoding unit (not shown), are configured the same, they may be shared as a single device configuration.

[0046] Furthermore, when the difference value from the subtractor 207 is input, the processing steps within the encoding unit may be switched appropriately so that the prediction process is omitted and only frequency conversion, quantization, and entropy encoding processes are performed.

[0047] On the other hand, when haptic information is input directly to the compression encoding unit 208, the prediction process is not omitted, and the compression encoding is performed by applying the same encoding process steps as for image information.

[0048] The multiplexing processor 209 reads the image coded data, haptic coded data, and pixel value conversion information from the work memory 106, adds header information (described later), multiplexes the data, and outputs a bit stream.

[0049] The header generation unit 210 is installed in the multiplexing processing unit 209, and embeds (describes) necessary information (encoding control information) in the header prior to the actual compressed encoded data so that the bit stream generated by the information compression device 100 can be correctly decoded by an external decoding / reproducing device. Details of the contents and generation method of the header information generated by the header generation unit 210 will be described later.

[0050] The functions of the information compression coding unit 103 explained so far are realized by dedicated hardware such as a DSP (Digital Signal Processor) or hard-wired logic, and are configured to perform high-speed real-time processing. However, there are no particular limitations as long as the same functions and performance can be achieved by software processing by the CPU 107.

[0051] The above is an outline of the internal configuration and functions of the information compression encoding unit 103 in this embodiment.

[0052] <Operation flow of the information compression coding unit> 3 is a flowchart showing the operation of the information compression encoding unit 103. Unless otherwise specified in this operation flow, the execution, judgment, and state transition of processing steps are performed by the CPU 107 controlling the start, stop, etc. of each of the above-mentioned functional blocks.

[0053] This operational flow is assumed to start when the camera unit 101 and tactile information acquisition unit 102 in the information compression device 100 have completed recording all of the image information and tactile information data that make up the above-mentioned "sequence" in the work memory 106 or recording unit 104.

[0054] First, in step S300 (hereinafter, "step" will be omitted), the image information encoding unit 201 reads out the original images from the work memory 106 one picture at a time, and compresses and encodes the image information of the entire sequence.

[0055] In S301, the analysis unit 203 reads image information and haptic information from the work memory 106 and derives the correlation between the image information and the haptic information. At this time, the selection decision of whether to input a decoded image or an original image as image information is as described above, and the processing content of this step is the same.

[0056] Here, the analysis process performed in S301 will be described in detail.

[0057] <Processing contents of the analysis unit 203> The analysis target in the analysis unit 203 is the image and tactile information of the entire sequence. For all pixels where the spatial positions of the tactile information and image information that make up the entire sequence coincide, the analysis unit 203 determines whether or not the signal values ​​of each have a relationship such that determining one determines the other, i.e., whether or not there is a correlation.

[0058] In this embodiment, the determination of whether or not there is a correlation between image information and tactile information is made by calculating the correlation coefficient r between two variables based on the following calculation formula (1), which is commonly used in the field of statistics.

[0059]

number

[0060] In the above formula (1), r is the correlation coefficient, x is the tactile value of the tactile information, y is the pixel value of the image information, and n is the total number of pixels and tactile information in the entire sequence.

[0061] The subscript i in x and y indicates the ith pixel information or tactile information in the sequence, and x - is the arithmetic mean of x, y - indicates the arithmetic mean of y. The value of r ranges from -1 to +1, and it can be determined that the closer the value of r is to 0, the less correlation there is. The above is the content of the analysis process executed in S301.

[0062] 3, in S302, it is determined whether or not there is a correlation based on the analysis results in S301. If it is determined that there is a correlation (YES in S302), the process proceeds to S303, and if it is determined that there is no correlation (NO in S302), the process branches to S309.

[0063] Here, although this embodiment does not limit the value to a specific one, the threshold for whether or not there is a correlation is, for example, when the absolute value of the above-mentioned correlation coefficient r is 0.8 to 0.9 or more, it is determined that there is a correlation between the pixel information and the tactile information.

[0064] In S303, when it is determined that there is a correlation between the image information and the tactile information, the pixel value conversion information generating unit 204 generates pixel value conversion information.

[0065] <Processing by the pixel value conversion information generating unit 204> Here, the processing contents of the pixel value conversion information generating unit 204 will be explained with reference to FIGS.

[0066] Figure 4 shows the correspondence between tactile information and image information in two-dimensional space. Figure 4(a) shows image information for one picture generated by the camera unit 101, with pixel values ​​for each pixel stored in raster order.

[0067] 4(b) shows haptic information for one picture generated by the haptic information acquisition unit 102, which has haptic signal values ​​at the same information granularity of area and resolution as the image information, equivalent to one pixel unit, and is stored in raster order. Hereinafter, in this embodiment, for convenience, the information unit of the haptic signal will be referred to as a "haptic sample."

[0068] In addition, the pixel information assigns a label to each pixel to identify its spatial position starting from P0, and similarly, the tactile information assigns a label to each tactile sample to identify its spatial position starting from H0. By identifying the label, it is possible to uniquely determine where a given pixel is located in space.

[0069] In this embodiment, for the sake of simplicity, identification is performed by assigning the above labels, but a method may also be used that can uniquely identify a pixel or tactile sample using so-called xy two-dimensional coordinates consisting of two-dimensional horizontal and vertical address values.

[0070] In this embodiment, pixel P0 corresponds to haptic sample H0. Similarly, pixel P1 corresponds to haptic sample H1, and so on, so that for the entire region up to Pn and Hn, the positions of pixels and haptic samples correspond to each other.

[0071] FIG. 5 shows a table of specific examples of pixel values ​​and tactile signal values ​​obtained based on the arrangement of the pixel information and tactile information described above.

[0072] In the example of Figure 5, for ease of explanation, seven spatially corresponding pairs of pixel values ​​and haptic signal values ​​are shown in table format, which corresponds to only a small portion of the pixel and haptic information for the entire actual sequence.

[0073] Specifically, this indicates that the haptic signal value corresponding to pixel value P0 of 218 is H0 of 200, and the haptic signal value corresponding to pixel value P1 of 200 is H1 of 217. Similarly, specific values ​​are shown for P6 and H6.

[0074] FIG. 6 shows, in table form, an example of image value conversion information generated in S303 after determining in S302 that there is a correlation with the signal values ​​in FIG.

[0075] The pixel value conversion information in FIG. 6 has a data structure in the form of a so-called lookup table, in which when the input pixel value in the first column is designated as an index, the output pixel value in the second column is obtained as pixel value converted data.

[0076] For this reason, the pixel values ​​of the original sequence, P0 to P6 in Figure 5, as well as the pixel values ​​of the entire sequence other than these, are sorted in ascending order and used as input values ​​to enable index search. As a result, the haptic signal values ​​of H0 to H6 corresponding to each pixel and the entire sequence thereafter are shaped so that they are output as pixel-value converted data.

[0077] In this embodiment, the above-described lookup table is called pixel value conversion information, and is output from the pixel value conversion information generation unit 204 .

[0078] Also, although a method of realizing this using a lookup table is shown, it is not necessarily limited to the same data structure, and other means may be used as long as the same effect can be expected.

[0079] 5 and 6, the pixel values ​​are assumed to be numerical values ​​ranging from 0 to 255, and the haptic signal values ​​are assumed to be information also ranging from 0 to 255. However, if the pixel values ​​and haptic signal values ​​have different bit depths, the bit depths are made uniform by known operations such as bit expansion.

[0080] 3, in S304, it is determined whether processing has been performed up to the final picture. If processing has not been performed up to the final picture (NO in S304), the process proceeds to S305, where the same processing from S304 onwards is performed on the next picture. On the other hand, if processing has been performed (YES in S304), it can be determined that encoding of haptic information has been completed for all pictures in the sequence, and this flow ends.

[0081] In S305, the image information is converted using the pixel value conversion information generated in S303, and the converted image information is output.

[0082] In the next step S306, the difference between the converted image information and the tactile information is calculated.

[0083] <Detailed explanation of difference values> Here, the difference value after passing through the subtractor 207 executed in S306 and a compression efficiency improvement mechanism that is a feature of this embodiment will be described with reference to FIG.

[0084] Fig. 7(a) is a graph of the pixel values ​​in Fig. 5, Fig. 7(b) is a graph of the haptic signal values ​​in Fig. 5, and Fig. 7(c) is a graph of the difference values ​​obtained by subtracting the haptic signal values ​​from the pixel values ​​in Fig. 5. Furthermore, Fig. 7(d) is a graph of the pixel value converted data in Fig. 6 generated in S305, and Fig. 7(e) is a graph of the difference values ​​obtained by subtracting the haptic signal values ​​in Fig. 5 from the pixel value converted data in Fig. 6.

[0085] The vertical axis of the graph represents pixel values ​​and haptic signal values, and the horizontal axis represents the labels described above with reference to FIG. 4, which in this embodiment represent positions in raster order in two-dimensional space.

[0086] As can be seen from the visualization in this manner, both Figures 7(c) and 7(e) are graphs of difference values, but Figure 7(e), which shows the difference value to which the pixel value conversion information of this embodiment has been applied, can produce a smaller difference value than Figure 7(c).

[0087] 5, when the pixel value is 201 and the haptic signal value is 36, the pixel value conversion information converts 201 as an input value to 36 as an output value. By utilizing the characteristic that pixel information and haptic information are highly correlated, the pixel value is converted to a value close to the haptic signal value (converted pixel value), and the input to the subsequent subtractor 108 becomes a pixel value converted to 36, with a haptic signal value of 36, and the result after subtraction is 0.

[0088] Here, simply calculating the difference based on the determination result that there is a correlation between the haptic information and the pixel information will result in a different data distribution characteristic, resulting in a result like that shown in Figure 7(c), and the difference will not necessarily be small. Therefore, in this embodiment, the pixel value conversion information generation unit 204 converts the pixel values ​​from Figure 7(a) to the converted pixel values ​​shown in Figure 7(d), and performs processing to generate conversion information with a unique correspondence that is consistent with the data distribution characteristic of the haptic signal value.

[0089] By using the above process, i.e., by encoding the difference between the converted pixel value corresponding to the pixel value in the same space, i.e., the predicted tactile value (converted pixel value), rather than encoding the tactile signal value as is, it is possible to improve compression efficiency.

[0090] Returning to the explanation of the flowchart in FIG. 3, in S307, the differential value calculated in S306 is input and compression-encoded in the compression-encoding unit 208, and haptic encoded data for one picture is output.

[0091] In S308, the image encoded data already encoded in S300 and the haptic encoded data compression-encoded in S307 are read out from the work memory 106 in units that are easy to synchronize with, for example, a decoder described later, such as one picture at a time, and multiplexed as a single bit stream.

[0092] In addition, the pixel value conversion information generated in S303 is also read from the work memory 106 and multiplexed as header data at a position in the bit stream corresponding to the beginning of the sequence, which will be described later.

[0093] After this step is executed, the process returns to S304 and repeats this process until the compression and encoding of haptic information for all pictures in the sequence is completed.

[0094] On the other hand, in S309, which is reached after determining in S302 that there is no correlation between image information and haptic information, a check is made to see if processing has been performed up to the final picture. If the answer is NO, the process proceeds to S310, where processing from S310 onwards is performed on the next picture. If the answer is YES, it can be determined that encoding of haptic information has been completed for all pictures in the sequence, and this flow ends.

[0095] In S310, since there is no correlation between pixel information and haptic information, the pixel value conversion information described above is not used, and the data path of selector 212 is switched to compress and encode (directly encode) the haptic information as is.

[0096] Then, in S311, similar to S308, the image encoded data already encoded in S300 and the haptic encoded data compression-encoded in S307 are read from the work memory 106 and multiplexed into one bit stream.

[0097] After this step is executed, the process returns to S309 and repeats this process until the compression and encoding of haptic information for all pictures in the sequence is completed.

[0098] The above is the processing flow executed by the information compression encoding unit 103 of this embodiment.

[0099] <Processing Contents of Multiplexing Processor 209> Next, the processing contents and data format required for decoding and decompressing the haptic encoded data multiplexed in the above-mentioned S308 and S311 back into the original haptic information in a decoder outside the device will be described.

[0100] This processing is mainly performed in the multiplexing processing unit 209 and the internal header generating unit 210.

[0101] As explained above as the functions of the image encoding unit 201 and the compression encoding unit 208, this embodiment assumes compression techniques standardized by international standards such as H.264 and HEVC. Therefore, the syntax structure and semantics of the bitstream also follow the above standards, with some extensions. The same applies to header information.

[0102] However, it is not limited to the data structure described in this embodiment, but rather it is sufficient that the data has synonymous identification information and data, and that the haptic information can be correctly decoded by a decoding device that receives the compressed data.

[0103] In this embodiment, a bitstream structure defined in the HEVC standard is adopted. Then, image coded data and haptic coded data are encapsulated in byte units using Network Abstraction Layer (hereinafter referred to as "NAL") units, and are packetized or converted into a byte stream depending on the application. In this case, the multiplexing unit 209 generates the image coded data, the haptic coded data, and the pixel value conversion information as different NAL units.

[0104] Image coding data is structured as a video coding layer (hereinafter referred to as VCL). Haptic coding data and pixel value conversion information are structured using user-definable supplemental enhancement information (SEI), a type of NonVCL.

[0105] FIG. 8 is a diagram showing a bitstream structure in this embodiment.

[0106] In Fig. 8, AUD (Access Unit Delimiter), VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SH (Slice Header), etc. are header parameter information required for decoding standardized image coded data, and are generated in a transmission order and format that conforms to the standard. Detailed description of the header parameters will be omitted.

[0107] Then, the SEI storing the pixel value conversion information described above is placed in the access unit at the beginning of the sequence, that is, preceding the first image coded data and haptic coded data.

[0108] Thereafter, in this embodiment, the VCL of image coded data coded in synchronized picture units and the SEI of NonVCL storing haptic coded data are multiplexed and output alternately in chronological order.

[0109] <Pixel Value Conversion Information SEI Format> Here, the data format of the SEI that stores pixel value conversion information will be described. This SEI is placed before slice data corresponding to the first encoded image data in the sequence as a P-SEI (Prefix SEI) 801 in Fig. 8.

[0110] To distinguish the P-SEI that stores this pixel value conversion information from other SEIs, it is created as an SEI_message of user_data_unregistered, with a user-definable payloadType value of 5. Furthermore, the user_data_payload_byte field in this message stores the pixel value conversion information and related information described above.

[0111] The data format will be explained below with reference to FIG. 9 based on the same pseudo-C program syntax as the HEVC standard.

[0112] The first column lists the line number, the second column lists the statement that defines the branch or loop operation or variable, and the third column lists the descriptor that indicates the unit and format of the variable symbol on that line.

[0113] The input argument pix_conversion_info_size_minus1 is the number of bytes of pixel value conversion information minus 1. It corresponds to the payload size of the upper layer sei_message() minus the 16-byte uuid_iso_iec_11578 field. This value is generated by the header generator 210.

[0114] The haptics_flag on the second line is a flag indicating whether or not haptic information is present. In this embodiment, this is set to 1. A value of 0 indicates that there is no haptic information, and that conventional image-only coding is being performed.

[0115] The pixel_conversion_info_valid on the fourth line indicates whether pixel value conversion information is available. In this embodiment, the setting value is determined based on whether there is a correlation between the pixel value and the haptic signal value, and is set to 1 if there is a correlation, and 0 if there is no correlation.

[0116] The pixel_conversion_info_minus1 on the 6th line indicates the number of types of pixel value conversion information minus 1. In this embodiment, 0 is set.

[0117] The pix_conversion_info on the 9th line indicates the pixel value conversion information itself in bytes.

[0118] The above is the data format of the user_data_payload_byte field of the P-SEI that stores pixel value conversion information in this embodiment.

[0119] <Haptic coding data SEI format> Here, the data format of the SEI that stores the encoded haptic data will be described. This SEI is placed after one picture's worth of encoded haptic data as an S-SEI (Suffix SEI) 802 in Fig. 8.

[0120] The format of haptic encoded data for one picture is shown in Figure 10. This haptic encoded data is packaged in the upper layer of the NAL unit as an SEI_message of user_data_unregistered, just like the P-SEI that stores the pixel value conversion information described above.

[0121] The input argument haptics_frame_code_size is the number of bytes of encoded haptic data minus 1. It corresponds to the payload size of the upper layer sei_message() minus the 16-byte uuid_iso_iec_11578 field. This value is generated by the header generator 210.

[0122] haptics_info_valid on the second line is a flag indicating whether pixel value conversion information is used. If the haptics encoded data to be transferred uses pixel value conversion information, set this to 1; if not, set this to 0.

[0123] The haptics_info_type on the third line is information for identifying the type of pixel value conversion information previously sent in the P-SEI. In this embodiment, since there is only one type of pixel value conversion information, this information is ignored.

[0124] As described above, the tactile encoded data of this embodiment basically uses the same compression encoding method as image information, and therefore can be stored as having been generated in the order of slice_segment_header(), slice_segment_data(), and rbsp_slice_segment_trailing_bits() as specified by the standard shown in the figure.

[0125] The above is the data format when haptic encoded data is stored in the user_data_payload_byte field of the S-SEI in this embodiment.

[0126] In this embodiment, the P-SEI storing pixel value transformation information is inserted only in the first picture, and the transferred pixel value transformation information is applied to the entire sequence, whereas the S-SEI storing haptic encoded data is inserted for every picture.

[0127] The above is the bit stream structure and data format when haptic encoded data and pixel value conversion information are multiplexed in this embodiment.

[0128] As described above, according to this embodiment, haptic information can be coded efficiently by coding the haptic information using correlation with image information.

[0129] (Second embodiment) In the first embodiment described above, a method for obtaining pixel value converted data was shown, assuming that the haptic signal values ​​corresponding to pixel values ​​have a one-to-one correspondence throughout the entire sequence.

[0130] In the second embodiment, a method for generating pixel value conversion information when there are multiple haptic signal values ​​paired with pixel values ​​will be described. Note that in the description of the second embodiment, parts that are common to the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted where appropriate.

[0131] FIG. 11 is a diagram showing an example of pixel values ​​and haptic signal values ​​acquired throughout the entire sequence in this embodiment, similar to FIG. 5 in the first embodiment.

[0132] As shown in FIG. 11, depending on the tactile target object, there may be multiple pixel values ​​218 in two-dimensional space, and the corresponding haptic signal values ​​may be different values ​​such as 31, 44, 20, 45, and 20.

[0133] Even in such cases, in order to achieve efficient compression of haptic information by utilizing the correlation with pixel information described above, in this embodiment, pixel value conversion information is generated using representative values ​​such as the average, median, and mode of haptic signal values ​​having the same pixel value.

[0134] <Operation flow> 12 is a flowchart showing the operation of the pixel value conversion information generation unit 204, which is a characteristic operation of this embodiment. As with the first embodiment, the operation of this flowchart starts when recording of the image information and haptic information of the entire sequence into the work memory 106 or the recording unit 104 is completed.

[0135] In S1200, it is determined whether pixel value conversion information has been created for all pixel value ranges. If there is pixel value conversion information that has not yet been created, proceed to S1201 (NO in S1200). On the other hand, if the creation of pixel value conversion information has been completed (YES in S1200), this operation flow ends.

[0136] In S1201, a pixel having the same pixel value as the index, which is the input value of the pixel value conversion information, is searched sequentially from the pixel information recorded as a sequence. The initial value of the index starts with zero as the minimum value and the maximum value is determined by the bit depth of the pixel.

[0137] In S1202, it is determined whether a pixel having the same pixel value as the index value was detected during the sequence. If a pixel was detected (YES in S1202), the haptic signal value corresponding to that pixel value is acquired and stored in S1203. If not found (NO in S1202), the process proceeds to the determination step of S1207.

[0138] After S1203 is executed, the process proceeds to the decision step S1204, where it is determined whether or not searching and checking for overlap has been completed for the current pixel value and the haptic signal value corresponding to that pixel value for all sequences.

[0139] If the sequence is in progress (NO in S1204), the process proceeds to S1205, where the haptic signal value corresponding to the currently detected pixel value is compared with the previously detected haptic signal value to determine whether they are different values.

[0140] On the other hand, if the search reaches the end of the sequence (YES in S1204), the process proceeds to S1208, where the process moves to registration processing for a series of pixel value conversion information.

[0141] Furthermore, if it is determined in the comparison and determination step of S1205 that different haptic signal values ​​exist for the same pixel value (YES in S1205), the haptic signal value is saved and the count value indicating the number of overlapping pixels is incremented in S1206.

[0142] After S1206 is executed, or if the haptic signal value is the same in S1205 (NO in S1205), the process returns to S1201 and the search for a pixel having the same pixel value as the index pixel value is repeated.

[0143] In S1207, to which the flow transitions from NO in S1202, it is determined whether the index pixel value currently being searched has never appeared in the sequence.

[0144] If it has been detected at least once (NO in S1207), proceed to the decision step of S1204. On the other hand, if the pixel value does not appear even once in the sequence (YES in S1207), there is no haptic signal value corresponding to that index value. Therefore, a predetermined invalid value (not shown) is registered in the pixel value conversion information, and proceed to setting the next pixel value for index in S1212.

[0145] S1208 is a decision step reached after checking the currently searched index value up to the end of the sequence, and here it is determined whether the pixel corresponding to that index value has multiple haptic signal values.

[0146] If multiple haptic values ​​are found for one pixel value throughout the sequence (YES in S1208), the process proceeds to S1209, where multiple different representative values ​​are calculated for the multiple stored haptic signal values. In this embodiment, three types of representative values ​​are derived: the arithmetic mean, the mode, and the median.

[0147] Furthermore, in the following step S1210, the representative value having the smallest sum of absolute differences from each haptic signal value is calculated, and this representative value is registered as pixel value conversion information.

[0148] On the other hand, if there are no overlapping haptic signal values ​​(NO in S1208), the haptic signal value corresponding to that pixel value is registered in the pixel value conversion information as pixel value converted data, as in the first embodiment described above.

[0149] After S1210 and S1211 are executed, the process proceeds to S1212, where the next index pixel value to be searched is set, and the series of operations starting from S1200 is repeated.

[0150] The above is the operation flow of this embodiment.

[0151] Here, the method for determining pixel value converted data performed in S1209 and S1210, which is a feature of this embodiment, will be described using an example in which the pixel value 218 shown by labels P0 to P4 in Figure 11 overlaps with haptic signal values ​​of 31, 44, 20, 45, and 20.

[0152] 11, for index pixel value 218, there are five pixels for which haptic signal values ​​have already been stored: 31, 44, 20, 45, and 20. The average value of the haptic signal values ​​for the five overlapping pixels is 32, the median is 31, and the mode is 20.

[0153] The cumulative sum of the absolute differences between these three representative values ​​and each haptic signal value is calculated. Although details of the calculation process are omitted, in this embodiment, the cumulative sum of the absolute differences between the mean value and the median value is 50, the cumulative sum of the absolute differences between the median value and the mode value is 49, and the cumulative sum of the absolute differences between the mode value and the mean value is 60.

[0154] From the above, the median value is the value that minimizes the cumulative sum of the absolute difference values, and the median value 31 is determined as the pixel value converted data.

[0155] In this way, according to the second embodiment, even when there are multiple haptic signal values ​​paired with pixel values, it is possible to determine optimal pixel value converted data and generate pixel value conversion information.

[0156] (Third embodiment) In the first and second embodiments described above, the compression encoding method is described assuming that there is only one subject object to be compressed and encoded during the sequence from the start to the end of image capture.

[0157] Next, as a third embodiment, a data generation method for cases where there are multiple subject objects or where objects with different tactile sensations appear in the same picture will be described. In the description of the third embodiment, parts common to the first and second embodiments will be assigned the same reference numerals and descriptions thereof will be omitted where appropriate.

[0158] 13 is a conceptual diagram showing a case where the encoding target, which is image and haptic information handled in this embodiment, changes over time during a sequence. The arrow at the bottom of the diagram, pointing from left to right, indicates the time direction.

[0159] As shown in Fig. 13, in this embodiment, the following case will be described. First, starting from time t0, the coding target changes from one coded object (one rabbit) to two coded objects (two rabbits) having the same texture at time t1. Then, at time t2, it changes to one coded object (one dog) having a different texture. Finally, it changes to three coded objects (dog, rabbit, cat) having different textures.

[0160] In the first embodiment described above, pixel value conversion information is generated and transmitted only once at the beginning of a sequence, and then image encoded data and haptic encoded data synchronized on a picture-by-picture basis are transmitted.

[0161] In this embodiment, when multiple objects to be coded exist within the same screen, such as at times t1 and t3, the concept of tile division defined in the HEVC standard is adopted. Specifically, one picture is divided into two-dimensional partial regions, each containing a rectangular region that contains an object to be haptic coded, and these are treated as a format that can be independently compressed and decoded. Then, whether or not to generate and output S-SEI, which is haptic coded data, is switched for each tile.

[0162] Furthermore, if the object to be coded changes to one with a different tactile sensation midway through the sequence at time t2, the change is detected, pixel value conversion information is generated again (regenerated), and P-SEI is retransmitted midway through the stream. A method for detecting changes in objects may be, for example, a well-known technique in which a model is created and used that has previously learned image and object labels for multiple objects using deep learning technology. The created model may be used to infer objects from target image information and detect changes. Furthermore, the tile division area may be determined based on the state of the change.

[0163] Furthermore, when multiple objects with different tactile sensations appear simultaneously, such as at time t3, a P-SEI is defined that transmits multiple pieces of pixel value conversion information, and the pixel value conversion information, which differs for each tile, is referenced and associated with the tactile encoding data (S-SEI).

[0164] The operation up to generating pixel value conversion information, which is a feature of the present embodiment, will be described below with reference to the flowchart in Fig. 14. Also, the operation of multiplexing the haptic encoded data will be described with reference to Fig. 15.

[0165] <Operation flow> 14 is a flowchart showing the operation up to generating pixel value conversion information. Similar to the start timing of the flowchart in FIG. 3, this flowchart starts when all of the image information and haptic information that make up the sequence have been stored in work memory 106.

[0166] First, in S1400, it is confirmed whether the generation process of pixel value conversion information for the entire sequence has been completed. If the process has been completed (YES in S1400), this flowchart ends. If the process has not been completed (NO in S1400), proceed to S1401.

[0167] In S1401, image information and haptic information for one picture are read, and it is determined whether the picture has been divided into tiles. If it has been divided into tiles (YES in S1401), proceed to S1402. If it has not been divided into tiles (NO in S1401), proceed to S1404.

[0168] In S1402, it is determined whether or not to perform information accumulation processing for generating pixel value conversion information.

[0169] In the tile division defined by the HEVC standard, tile identifiers are assigned in raster scan order starting from 1. For example, in the image at time t1 in Fig. 13, one picture is divided into four, with rabbit objects present in tiles 1 and 4, and no object present in tiles 2 and 3. For tiles with no object present in this way, information accumulation processing is skipped.

[0170] In addition, if there is an object with the same tactile sensation in the information accumulated so far, the information accumulation process will be skipped. For example, in the processing of the image at time t1 in Figure 13, when tile 4 is executed, it is an object with the same tactile sensation as tile 1, so this branch transitions to NO.

[0171] If it is determined in S1402 that the information is to be stored (YES in S1402), the process proceeds to S1403. If it is determined that the information is not to be stored (NO in S1402), the process proceeds to S1405.

[0172] In S1403, pixel information and tactile information values ​​for one tile are temporarily stored in memory (not shown) inside the analysis unit 203. At this time, a memory area is reserved for each tile area, and information for the same tile area is stored in the same memory area.

[0173] In S1404, image information and tactile information for one picture are stored in one memory area of ​​the internal memory of the analysis unit 203 (not shown).

[0174] In S1405, it is checked whether processing of one picture has been completed. If the pixel information and haptic information values ​​for all tile divisions have been stored in memory (YES in S1405), the process proceeds to S1406. If processing of one picture has not been completed (NO in S1405), the process returns to S1402.

[0175] In S1406, it is confirmed whether or not a switching timing has occurred.

[0176] As explained in Figure 13, a switch occurs when an object with a different tactile sensation appears. For example, in Figure 13, this occurs at times t2 and t3. At time t1, the object has the same tactile sensation as at time t0, so no switch occurs.

[0177] Here, it is a known technique that the image information encoding unit 201 can recognize the timing of switching the tactile object, and the analysis unit 203 is notified of this timing.

[0178] If a change occurs in S1406 (YES in S1406), the process proceeds to S1407. If a change does not occur (NO in S1406), the process returns to S1401 and continues to accumulate data for the next picture.

[0179] In S1407, it is determined whether or not analysis of all data accumulated up to that point has been completed. If analysis of all data has been completed (YES in S1406), the process returns to S1400, and data of pictures after the switching timing is accumulated from the beginning. If analysis has not been completed (NO in S1407), the process proceeds to S1408.

[0180] In S1408, the analysis unit 203 performs the analysis, and the process proceeds to S302. S302 and S303 are as described above.

[0181] The pixel value conversion information generated in S303 is stored in the memory 205.

[0182] The method for generating pixel value conversion information when there is a change over time in a haptic object has been described above with reference to FIG.

[0183] <Stream structure> Next, FIG. 15 is a diagram showing the structure of a bitstream output based on the above-described operation flow when there is a change over time in the haptic object as shown in FIG.

[0184] Note that the letters written within the rectangular areas corresponding to NAL units in the bitstream in the figure are abbreviations of NAL unit types: A stands for AccessUnitDelimiter, V for VideoParameterSet, S for SequenceParameterSet, P for PictureParameterSet, H for Slice Header, and D for SliceData.

[0185] First, the data structure and arrangement of various NAL units corresponding to the first picture of the bitstream shown in FIG. 15 correspond to the data generated and coded at time t0 in FIG.

[0186] The pixel value conversion information is for rabbits, and is embedded in the P-SEI of the bitstream before being output. From the second picture onwards, the pixel value conversion information sent for the first picture is inherited and decoded, so the P-SEI is not sent.

[0187] Picture P1 corresponds to time t1 in Figure 13. At time t1, the number of rabbits increases from one to two, but the pixel value conversion information itself does not change because they are objects with the same tactile sense. Therefore, P-SEI is not transmitted.

[0188] Since the P1 picture is divided into four tiles, S-SEI is transmitted for each tile. In the P1 picture, objects exist in tiles 1 and 4, but no objects exist in tiles 2 and 3, so the S-SEI for tiles 2 and 3 is not transmitted.

[0189] Picture P2 corresponds to time t2 in Figure 13. At time t2, the two rabbits have changed to one dog, and the objects have different textures. Therefore, the pixel value conversion information is conversion information for the dog, and the pixel value conversion information is embedded again in the P-SEI of the bitstream and output.

[0190] Picture P3 corresponds to time t3 in Figure 13. At time t3, the picture changes from a single dog to a picture containing multiple objects with different textures, such as a dog, a rabbit, and a cat. Therefore, three types of pixel value conversion information for dog, rabbit, and cat are transmitted to P-SEI.

[0191] For the dog, it is possible to implement the process without sending it because it is transferred at the P2 picture. However, in this embodiment, in order to simplify the process, after the switching timing, the P-SEI of the previous picture is not used, and the pixel value conversion information required to decode the relevant picture is sent again.

[0192] The value of the user_data_payload_byte field stored in the P-SEI shown in FIG. 9 when the P3 picture in FIG. 15 is executed will be described.

[0193] 9. The haptics_flag in the second line of FIG. 9 is a flag indicating the presence or absence of haptic information, and is set to 1.

[0194] The pixel_conversion_info_valid in the fourth line is set to 1 to indicate whether pixel value conversion information is present.

[0195] The pixel_conversion_info_minus1 on the sixth line indicates the number of types of pixel value conversion information minus 1, and is therefore 2 in this embodiment.

[0196] The pix_conversion_info on the 9th line indicates the pixel value conversion information itself in bytes.

[0197] The number of bytes of pixel value conversion information minus 1 is stored in pic_conversion_info_size_minus1. This value is an array, and in this embodiment, it corresponds to the following:

[0198] pic_conversion_info_size_minus1[0] = byte size of pixel value conversion information for dog - 1 pic_conversion_info_size_minus1[1] = rabbit pixel value conversion information byte size - 1 pic_conversion_info_size_minus1[2] = Cat pixel value conversion information byte size - 1 In this case, the array subscripts 0 to 2 correspond to the haptics_info_type in the user_data_payload_byte field of the S-SEI as identification signals of pixel value conversion information. 0 indicates pixel value conversion information for dogs, 1 indicates pixel value conversion information for rabbits, and 2 indicates pixel value conversion information for cats.

[0199] The operation of the multiplexing process of haptic encoded data in this embodiment has been described above with reference to FIG.

[0200] In this way, according to the third embodiment, even when a haptic object changes over time, pixel conversion information can be created for each tile or picture, and the corresponding pixel conversion information can be included in a bitstream and transferred to a decoding device.

[0201] (Fourth embodiment) In the above-described first to third embodiments, a method and device for compressing and encoding haptic information using image information has been described.

[0202] In the fourth embodiment, a method will be described in which compressed and encoded data generated by the information compression device of the first to third embodiments is received and decoded and expanded to the original image information and haptic information.

[0203] In the description of the fourth embodiment, parts common to the first to third embodiments will be denoted by the same reference numerals and description thereof will be omitted as appropriate.

[0204] <System configuration> FIG. 16 is a block diagram showing an example of the system configuration of an information decoding device 1600 according to this embodiment.

[0205] The system of this embodiment is configured to include an image display unit 1601, a haptic output unit 1602, an information decoding unit 1603, a recording unit 104, a network unit 105, a work memory 106, a CPU 107, a primary storage unit 108, a CPU bus 109, and a memory bus 110.

[0206] The recording unit 104, work memory 106, CPU 107, primary storage unit 108, CPU bus 109, and memory bus 110 have the same functions as those described in the first embodiment, so their description will be omitted.

[0207] The image display unit 1601 is a display device such as a monitor or a head-mounted display, and reads out the image information decoded by the information decoding unit 1603 from the work memory and displays the image.

[0208] The haptic output unit 1602 is a device that presents haptic information to the human body, such as a haptic suit or haptic gloves, and reads out the haptic information decoded by the information decoding unit 1603 from the work memory and reproduces the haptic information.

[0209] The network unit 105 is an interface for connecting the information decoding device 1600 to an external device, and in this embodiment, communicates with the external device via a network, receives a bitstream, and stores it in the work memory 106. The information decoding unit 1603 will be described in detail later.

[0210] <Internal configuration of the information decoding unit 1603> 17 is a block diagram showing the configuration of the information decoding unit 1603 in this embodiment. The information decoding unit 1603 is configured to include a separation processing unit 1701, an image information decoding unit 1702, and a haptic information decoding unit 1703.

[0211] The haptic information decoding unit 1703 is configured to include a memory 205 , a pixel value conversion unit 1705 , a decoding / decompression unit 1706 , an adder 1707 , and a selector 1708 .

[0212] The demultiplexing processor 1701 includes a header analyzer 1704, which analyzes the header of the bitstream. The demultiplexing processor 1701 reads the bitstream from the work memory 106 and obtains haptics_flag, pix_conversion_info_valid, pix_conversion_info_size_minus1, and pix_conversion_info_minus1, which are included in the P_SEI in the bitstream. The demultiplexing processor 1701 also obtains haptics_info_valid and haptics_info_type, which are included in the S_SEI in the bitstream. The meaning of each piece of data is as described above.

[0213] If pix_conversion_info_valid is 1, pixel value conversion information is obtained from pix_conversion_info and written to the memory 205. Also, haptics_info_valid is notified to the pixel value conversion unit 1705 and the selector 1708. Then, haptics_info_type is notified to the pixel value conversion unit 1705.

[0214] Furthermore, the image encoded data obtained from the slice data in the bitstream is transferred to the image information decoding unit 1702 , and the haptic encoded data obtained from the S-SEI is transferred to the haptic information decoding unit 1703 .

[0215] The image information decoding unit 1702 decodes the image coded data input from the separation processing unit 1701. The decoded image information is written to the work memory .

[0216] The pixel value conversion unit 1705 reads pixel value conversion information from the memory 205. It also reads a decoded image from the work memory 106, applies the pixel value conversion information to the pixel values ​​of the decoded image, generates pixel value converted data, and outputs the data to the adder 1707.

[0217] If there are multiple types of pixel value conversion information, the haptics_info_type notified by the separation processing unit is used to determine which pixel value conversion information to use, and the pixel conversion information is then read from the memory area in memory 205 where the corresponding information is stored.

[0218] The decoding and expansion unit 1706 receives the haptic encoded data and outputs the decoded result, which is the difference between the haptic information and the pixel value converted data when image conversion information is used, or the haptic information itself when image conversion information is not used.

[0219] The adder 1707 adds the difference value output from the decoding / decompression unit 1706 and the pixel value converted data output from the pixel value converter.

[0220] A selector 1708 uses the signal haptics_info_valid from the separation processor as a selection control signal, and if this signal is 1, it selects the output result of the adder as haptic information, and if it is 0, it selects the output of the decoding / expansion unit 1706 as haptic information. The haptics information is written to the work memory 106.

[0221] As a result, the haptic information decoding unit 1703 can decode the haptic information and write it to the work memory 106. The image information and haptic information written to the work memory 106 are read by the image display unit 1601 and the haptic output unit 1602 and played back.

[0222] <Operation flow of the information decoding unit> 18 is a flowchart showing the operation of the information decoding unit 1603. This operation flow is assumed to start when the bit stream arrives at the information decoding unit 1603.

[0223] In S1800, separation processing unit 1701 analyzes the bitstream and separates it into image coded data, haptic coded data, and pixel value conversion information. The haptic coded data and pixel value conversion information may not be included in the bitstream.

[0224] The presence or absence of haptic information in the entire sequence can be determined by the haptics_flag in the user_data_payload_byte of the P-SEI. Furthermore, the use or non-use of pixel value conversion information can be determined by the pix_conversion_info_valid. Furthermore, the presence or non-existence of haptic encoded data for each tile can be determined by the presence or non-existence of the S-SEI. The result of this determination is used in the subsequent determination processes of S1801, S1805, and S1807.

[0225] In S1801, it is determined whether pixel value conversion information is present. Here, if the value of pix_conversion_info_valid in P-SEI is 1, it is determined that pixel value conversion information is present (YES in S1801), and the process proceeds to S1802. If not (NO in S1801), the process proceeds to S1803.

[0226] In S1802, the pix_conversion_info stored in the P-SEI is extracted and stored in the memory 205.

[0227] In S1803, the image information decoding unit 1702 receives the image coded data from the separation processing unit 1701 and decodes it.

[0228] In S1804, the decoded image is written to the work memory 106.

[0229] In S1805, it is determined whether or not haptic data is present. This determination is made using the determination explained in S1800. If there is no haptic data (NO in S1805), the process transitions to S1811. If there is haptic data (YES in S1805), the process transitions to S1806.

[0230] In S1806, the haptic encoded data separated by the separation processing unit 1701 is decoded.

[0231] In S1807, it is determined whether pixel value conversion information is being used. This determination is made using haptics_info_valid stored in user_data_payload_byte of S-SEI acquired from the stream by the separation processing unit 1701. If haptics_info_valid is 1 (YES in S1807), the process proceeds to S1808, and if it is 0 (NO in S1807), the process proceeds to S1810.

[0232] In S1808, pixel value conversion unit 1705 reads pixel value conversion information from memory 205. If there are multiple pieces of pixel value conversion information in memory 205, the haptics_info_type stored in user_data_payload_byte of S-SEI uniquely determines which pixel value conversion information to read. Then, a decoded image corresponding to the haptic information currently being decoded is read from work memory 106. This decoded image and the pixel value conversion information are used to generate pixel value converted data.

[0233] In S1809, the pixel value converted data generated in S1808 and the decoded haptic data are added together.

[0234] In S1810, the tactile information is written to the work memory 106.

[0235] In S1811, it is determined whether decoding of one picture has been completed, and if decoding has been completed (YES in S1811), the process proceeds to S1812. If decoding has not been completed (NO in S1811), the process returns to S1800.

[0236] In S1812, it is determined whether decoding of the entire stream is complete, and if not (NO in S1812), the process returns to S1800 to process the next picture. If decoding is complete (YES in S1812), the process ends.

[0237] As described above, according to this embodiment, the bit stream generated by the information compression device 100 can be decoded by the information decoding device 1600.

[0238] The disclosure of this specification includes the following information encoding device and method, information decoding device, program, and storage medium.

[0239] (Item 1) An information encoding device that encodes image information and tactile information, a first encoding means for encoding the image information; a second encoding means for encoding the haptic information, an analysis means for analyzing the correlation between the unencoded image information and the unencoded haptic information; a generating means for generating pixel value conversion information, which is information for converting pixel values ​​in the unencoded image information into converted pixel values ​​that are different pixel values, based on the analysis result by the analyzing means; a calculation means for calculating a difference between a haptic signal value in the haptic information and the converted pixel value corresponding to the same spatial position as the haptic signal value; a second encoding means for encoding the differential value; a multiplexing means for multiplexing the image coded data coded by the first coding means, the haptic coded data coded by the second coding means, and the pixel value conversion information into one bit stream; An information encoding device comprising:

[0240] (Item 2) 2. The information encoding device according to item 1, wherein the analysis means receives an original image or a decoded image as the image information.

[0241] (Item 3) 3. The information encoding device according to item 1 or 2, wherein the generating means generates the pixel value conversion information so that the difference value becomes a smaller value.

[0242] (Item 4) The information encoding device described in any one of items 1 to 3, characterized in that the pixel value conversion information has a data structure of a lookup table that can obtain the converted pixel value as an output from a pixel value in the image information as an input.

[0243] (Item 5) The information encoding device described in any one of items 1 to 4 is characterized in that, when there are multiple haptic signal values ​​paired with one pixel value in the unencoded image information, the generation means generates the pixel value conversion information so as to convert the one pixel value into a pixel value corresponding to one of the average, median, or mode of the multiple paired haptic signal values.

[0244] (Item 6) The information encoding device described in any one of items 1 to 5, characterized in that the generation means does not generate the pixel value conversion information when the correlation between the unencoded image information and the unencoded haptic information is lower than a predetermined value.

[0245] (Item 7) 7. The information encoding device according to item 6, wherein the second encoding means directly encodes the input tactile information when the pixel value conversion information is not generated.

[0246] (Item 8) 8. The information encoding device according to any one of items 1 to 7, wherein the generating means regenerates the pixel value conversion information at a predetermined timing when the image information and the tactile information change over time.

[0247] (Item 9) The information encoding device described in item 8, characterized in that the image information and the tactile information change over time when at least one of the following occurs: the subject included in the image information or the tactile information changes, the number of subjects changes, or the brightness of the image information changes.

[0248] (Item 10) The information encoding device described in any one of items 1 to 9, characterized in that when the tactile information includes multiple objects, the second encoding means analyzes the image information and tactile information for each area of ​​the object and generates the pixel value conversion information for each area of ​​the object.

[0249] (Item 11) The information encoding device described in any one of items 1 to 9, characterized in that the multiplexing means includes a generation means for generating header data describing encoding control information necessary for decoding the image encoded data and the haptic encoded data, and the header data is output prior to the image encoded data and the haptic encoded data.

[0250] (Item 12) Item 12. The information encoding device according to item 11, wherein the multiplexing means writes, in the header data, information indicating the presence or absence of the pixel value conversion information, information on the number of pieces of pixel value conversion information, identification information of the plurality of pieces of pixel value conversion information, and at least one of the pixel value conversion information.

[0251] (Item 13) An information decoding device that decodes the bit stream generated by the information encoding device according to any one of items 1 to 12, a separating means for obtaining the image coded data, the haptic coded data, and the pixel value conversion information from the bitstream; a first decoding means for decoding the image encoded data; a second decoding means for decoding the haptic encoded data, means for decoding the encoded difference value; a second conversion means for converting the pixel values ​​decoded by the first decoding means using the pixel value conversion information; a second decoding means including an adding means for adding the pixel value converted by the second converting means and the decoded difference value; An information decoding device comprising:

[0252] (Item 14) Item 14. The information decoding device according to item 13, wherein the separating means comprises a header analyzing means for analyzing header data describing encoding control information necessary for decoding the image encoded data and the haptic encoded data.

[0253] (Item 15) Item 15. The information decoding device according to item 14, wherein the second decoding means decodes a signal obtained by directly encoding the haptic information using the second encoding means when the pixel value conversion information is not available.

[0254] (Item 16) 1. An information encoding method for encoding image information and tactile information, comprising: a first encoding step of encoding the image information; a second encoding step of encoding the haptic information, an analyzing step of analyzing a correlation between the uncoded image information and the uncoded haptic information; a generating step of generating pixel value conversion information, which is information for converting pixel values ​​in the unencoded image information into converted pixel values ​​that are different pixel values, based on the analysis result in the analyzing step; a calculation step of calculating a difference value between a haptic signal value in the haptic information and the converted pixel value corresponding to the same spatial position as the haptic signal value; a second encoding step comprising: encoding the difference value; a multiplexing step of multiplexing the image coded data coded in the first coding step, the haptic coded data coded in the second coding step, and the pixel value conversion information into one bit stream; 1. An information encoding method comprising:

[0255] (Item 17) A program for causing a computer to function as each means of the information encoding device according to any one of items 1 to 12.

[0256] (Item 18) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information encoding device according to any one of items 1 to 12.

[0257] (Item 19) A program for causing a computer to function as each means of the information decoding device according to any one of items 13 to 15.

[0258] (Item 20) A computer-readable storage medium storing a program for causing a computer to function as each means of the information decoding device according to any one of items 13 to 15.

[0259] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

[0260] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0261] 100: Information compression device, 101: Camera unit, 102: Tactile information acquisition unit, 103: Information compression encoding unit, 104: Recording unit, 105: Network unit, 106: Work memory, 107: CPU, 108: Primary storage unit, 109: CPU bus, 110: Memory bus

Claims

1. An information encoding device that encodes image information and tactile information, a first encoding means for encoding the image information; a second encoding means for encoding the haptic information, an analysis means for analyzing the correlation between the unencoded image information and the unencoded haptic information; a generating means for generating pixel value conversion information, which is information for converting pixel values ​​in the unencoded image information into converted pixel values ​​that are different pixel values, based on the analysis result by the analyzing means; a calculation means for calculating a difference between a haptic signal value in the haptic information and the converted pixel value corresponding to the same spatial position as the haptic signal value; a second encoding means for encoding the differential value; a multiplexing means for multiplexing the image coded data coded by the first coding means, the haptic coded data coded by the second coding means, and the pixel value conversion information into one bit stream; An information encoding device comprising:

2. 2. The information encoding device according to claim 1, wherein an original image or a decoded image is input to said analyzing means as said image information.

3. 2. The information encoding device according to claim 1, wherein said generating means generates said pixel value conversion information so that said difference value becomes a smaller value.

4. 2. The information encoding device according to claim 1, wherein the pixel value conversion information has a data structure of a lookup table that receives pixel values ​​in the image information as input and can obtain the converted pixel values ​​as output.

5. The information encoding device according to claim 1, characterized in that, when there are multiple haptic signal values ​​paired with one pixel value in the unencoded image information, the generating means generates the pixel value conversion information so as to convert the one pixel value into a pixel value corresponding to one of the average, median, or mode of the multiple paired haptic signal values.

6. The information encoding device according to claim 1, characterized in that the generating means does not generate the pixel value conversion information when the correlation between the unencoded image information and the unencoded tactile information is lower than a predetermined value.

7. 7. The information encoding device according to claim 6, wherein the second encoding means directly encodes the input tactile information when the pixel value conversion information is not generated.

8. 2. The information encoding device according to claim 1, wherein said generating means regenerates said pixel value conversion information at a predetermined timing when said image information and said tactile information change over time.

9. The information encoding device described in claim 8, characterized in that the image information and the tactile information change over time when at least one of the following occurs: the subject included in the image information or the tactile information changes, the number of subjects changes, or the brightness of the image information changes.

10. The information encoding device according to claim 1, characterized in that, when the tactile information includes multiple objects, the second encoding means analyzes the image information and tactile information for each area of ​​the object and generates the pixel value conversion information for each area of ​​the object.

11. The information encoding device according to claim 1, characterized in that the multiplexing means includes a generating means for generating header data describing encoding control information necessary for decoding the image encoded data and the haptic encoded data, and the header data is output prior to the image encoded data and the haptic encoded data.

12. The information encoding device according to claim 11, characterized in that the multiplexing means writes in the header data at least one of information indicating whether or not the pixel value conversion information is present, information on the number of pieces of pixel value conversion information, identification information for the plurality of pieces of pixel value conversion information, and the pixel value conversion information.

13. 13. An information decoding device that decodes the bit stream generated by the information encoding device according to any one of claims 1 to 12, a separating means for obtaining the image coded data, the haptic coded data, and the pixel value conversion information from the bitstream; a first decoding means for decoding the image encoded data; a second decoding means for decoding the haptic encoded data, means for decoding the encoded difference value; a second conversion means for converting pixel values ​​decoded by the first decoding means using the pixel value conversion information; a second decoding means including an adding means for adding the pixel value converted by the second converting means and the decoded difference value; An information decoding device comprising:

14. 14. The information decoding device according to claim 13, wherein the separating means comprises header analyzing means for analyzing header data describing encoding control information required for decoding the image encoded data and the haptic encoded data.

15. 15. The information decoding device according to claim 14, wherein the second decoding means decodes a signal obtained by directly encoding the haptic information using the second encoding means when the pixel value conversion information is not available.

16. 1. An information encoding method for encoding image information and tactile information, comprising: a first encoding step of encoding the image information; a second encoding step of encoding the haptic information, an analyzing step of analyzing a correlation between the uncoded image information and the uncoded haptic information; a generating step of generating pixel value conversion information, which is information for converting pixel values ​​in the unencoded image information into converted pixel values ​​that are different pixel values, based on the analysis result in the analyzing step; a calculation step of calculating a difference value between a haptic signal value in the haptic information and the converted pixel value corresponding to the same spatial position as the haptic signal value; a second encoding step comprising: encoding the difference value; a multiplexing step of multiplexing the image coded data coded in the first coding step, the haptic coded data coded in the second coding step, and the pixel value conversion information into one bit stream; 1. An information encoding method comprising:

17. A program for causing a computer to function as each of the means of the information encoding device according to any one of claims 1 to 12.

18. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information encoding device according to any one of claims 1 to 12.

19. A program for causing a computer to function as each of the means of the information decoding device according to claim 13.

20. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information decoding device according to claim 13.

Images