Signal processing device and control method thereof

Abstract

This invention provides a signal processing device and a control method for acquiring tactile data with higher accuracy, even when the subject itself moves or when the field of view changes due to camera pan, tilt, zoom, etc. [Solution] In a decoding device 100 having a tactile data decoding processing unit 101 and an image data decoding processing unit 102, the tactile data decoding processing unit interpolates the tactile data based on the decoded motion vector when it detects a loss in tactile data using a loss detection unit 123.

Description

Technical Field

[0001] The present invention relates to signal processing of tactile data associated with image data.

Background Art

[0002] Devices implementing tactile presentation technology called haptics are known.

[0003] Patent Document 1 discloses a technique for visually presenting the tactile sensation of an object displayed on a display device according to a user's request. Also, in the case of a moving image, when tactile data is missing due to the influence of noise or the like, tactile data can be acquired from past frames.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1, when the subject itself moves or when the angle of view changes due to panning, tilting, zooming, etc. of the camera, and the content of the video is different between the current frame and the past frame, there are cases where tactile data cannot be correctly acquired and interpolated.

[0006] Therefore, in view of the prior art, an object of the present invention is to provide a signal processing device and its control method capable of acquiring tactile data with higher accuracy even when the angle of view changes.

Means for Solving the Problems

[0007] One aspect of the present invention is characterized by comprising: an image data decoding means for decoding image data; a tactile data decoding unit for decoding tactile data corresponding to the image data; a storage unit for holding motion vectors decoded based on the image data; a detection unit for detecting missing tactile data decoded by the tactile data decoding unit; and, if the detection unit detects missing tactile data, a processing means for interpolating the tactile data based on the motion vectors.

[0008] The system is characterized by comprising: an image data decoding means for decoding image data; a tactile data decoding unit for decoding tactile data corresponding to the image data; a detection unit for detecting missing tactile data decoded by the tactile data decoding unit; and a processing means for interpolating the abnormal tactile data using tactile data from an image data region similar to the region of the image data in which the tactile data is missing. [Effects of the Invention]

[0009] According to the present invention, tactile data can be acquired with higher accuracy even when the field of view changes. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example configuration of a decoding device. [Figure 2] This diagram illustrates a method for interpolating missing tactile data using motion vectors. [Figure 3] This diagram shows the timing of image decoding and tactile decoding. [Figure 4] This flowchart shows the control of tactile data interpolation processing. [Figure 5] This figure shows the presence or absence of tactile data at each pixel when an image is enlarged. [Figure 6] This diagram shows the area to be searched using images. [Figure 7] This is a block diagram showing an example configuration of a decoding device. [Figure 8]This figure shows the range for calculating the key frequency components of the DB. [Figure 9] This diagram shows the database and keys. [Figure 10] This diagram shows the relationship between images and haptic data planes. [Modes for carrying out the invention]

[0011] <First Embodiment> A decoding device according to a first embodiment to which the present invention is applied will be described with reference to Figure 1. The decoding device is an example of a signal processing device. In this embodiment, the processing is performed, for example, by a CPU (not shown) executing a program stored in a memory (not shown). The CPU is an example of a processing means.

[0012] (Configuration of the decoding device 100 and its peripheral modules) Figure 1 is a block diagram showing an example configuration of a decoding device 100 and its peripheral modules according to a first embodiment of the present invention.

[0013] The decoding device 100 consists of a tactile data decoding processing unit 101 and an image data decoding processing unit 102. The tactile data decoding processing unit 101 and the image data decoding processing unit 102 receive an encoded stream from the memory control unit 103.

[0014] The memory control unit 103 reads the encoded stream recorded on a recording medium (not shown), outputs the image encoded stream to the image data decoding unit 102, and outputs the tactile data encoded stream to the tactile data decoding unit 101.

[0015] The image data decoding processing unit 102 includes an entropy decoding unit 104, an inverse quantization unit 105, an inverse orthogonal transformation unit 106, an addition unit 107, and an addition unit 108. The image data decoding processing unit 102 also includes a switch 109, an intra prediction unit 111, an intra prediction memory 110, a motion compensation unit 112, a loop filter 113, an image frame memory 114, and an image display unit 115.

[0016] The tactile data decoding processing unit 101 is composed of a tactile data decoding unit 120, a motion vector storage unit 121, a tactile compensation unit 122, a missing detection unit 123, a switch 124, a tactile frame memory 125, and a tactile display unit 126.

[0017] The tactile display unit 126 is a device that conveys the tactile sensation to the user.

[0018] (Relationship between tactile data and image) Here, the relationship between tactile data and image will be described using FIG. 10. The planes indicated by Y, Cb, and Cr represent the image. It is the image decoded by the image data decoding processing unit 102.

[0019] The Hp plane existing above the Y plane represents the tactile data plane. The Hp plane is the tactile data decoded by the tactile data decoding processing unit 101.

[0020] In this example, the explanation will proceed assuming that the resolutions of the image and the tactile data are the same, but they may also be different.

[0021] (Configuration of the image data decoding processing unit 102) The configuration of the image data decoding processing unit 102 will be described.

[0022] The entropy decoding unit 104 inputs the encoded stream from the memory control unit 103 and performs decoding processing according to the standard. The decoded motion vector is output to the motion compensation unit 112, the decoded intra prediction mode is output to the intra prediction unit 111, and the decoded quantized orthogonal transform coefficients are output to the inverse quantization unit 105 respectively.

[0023] [[ID=3A]]The inverse quantization unit 105 performs inverse quantization on the input quantized orthogonal transform coefficients and outputs the generated orthogonal transform coefficients to the inverse orthogonal transform unit 106.

[0024] The inverse orthogonal transformation unit 106 performs an inverse orthogonal transformation on the input orthogonal transformation coefficients and outputs the generated predicted difference data (orthogonal transformation data) to the adder units 107 and 108.

[0025] The addition unit 107 adds the predicted image generated by the intra-prediction unit 111 and the predicted difference data generated by the inverse orthogonal transform unit 106 to generate a decoded image.

[0026] The addition unit 108 adds the predicted image generated by the motion compensation unit 112 and the predicted difference data generated by the inverse orthogonal transform unit 106 to generate a decoded image.

[0027] Switch 109 selects either the decoded image output from the adder 107 or the adder 108. If the current block to be decoded is an intra-prediction, it selects the decoded image from adder 107; if it is an inter-prediction, it selects the decoded image from adder 108. The decoded images are output to the intra-prediction memory 110 and the loop filter 113, respectively.

[0028] The intra-prediction memory 110 is a memory that holds reference images used to generate prediction images for intra-prediction.

[0029] The intra prediction unit 111 reads a reference image from the intra prediction memory 110, generates a predicted image based on the prediction mode input from the entropy decoding unit 104, and outputs it to the summing unit 107.

[0030] The motion compensation unit 112 reads a reference image from the image frame memory 114, generates a predicted image based on information such as motion vectors input from the entropy decoding unit 104, and outputs it to the summing unit 108.

[0031] The loop filter 113 filters the decoded image to remove block noise. This improves the quality of the decoded image used as a reference image.

[0032] The image frame memory 114 is a memory that stores multiple images that have been processed by the loop filter and are used as reference images during interpretation.

[0033] The image display unit 115 is a display device that displays the decoded image. The image can be enlarged or reduced by performing operations such as pinch-in and pinch-out on the image display unit 115.

[0034] (Configuration of the haptic data decoding processing unit 101) Next, we will explain the configuration of the tactile data decoding processing unit 101.

[0035] The tactile data decoding unit 120 receives the encoded stream from the memory control unit 103 and performs the decoding process for the tactile data. Here, it is assumed that there is one data item per pixel. When the tactile data corresponding to one pixel is decoded, the position information of the tactile data is output to the motion vector storage unit 121. The decoded tactile data is output to the missing data detection unit 123 and the switch 124.

[0036] The motion vector storage unit 121 is a storage unit that stores the motion vector decoded by the entropy decoding unit 104 of the image data decoding processing unit 102 and the reference image number referenced by the motion vector. It outputs the motion vector for the pixel of the tactile data currently being decoded to the tactile compensation unit 122. It outputs the reference image number referenced by the motion vector to the tactile frame memory 125.

[0037] The tactile frame memory 125 stores tactile data from past images (frames). The tactile frame memory 125 receives the reference image number referenced by the motion vector from the motion vector storage unit 121. It outputs the tactile data of the image (frame) corresponding to the reference image number to the tactile compensation unit 122.

[0038] The tactile compensation unit 122 outputs the input motion vector and the tactile data of the coordinates pointed to by the motion vector from the tactile data to the read switch 124.

[0039] The data loss detection unit 123 receives the tactile data decoded from the tactile data decoding unit 120 and detects whether there are any missing data points in the tactile data. Detection may also be performed using CRC or checksums. If there are missing data points in the tactile data, a signal indicating the presence of missing data is output to the switch 124.

[0040] In the data loss detection 123, switch 124 selects the tactile data output from the tactile compensation unit 122 if there is a data loss, and selects the tactile data decoded by the tactile data decoding unit 120 if there is no data loss. The tactile data selected by switch 124 is output to the tactile frame memory 125 and the tactile display unit 126.

[0041] (Interpolation of tactile data) Next, we will explain how to interpolate tactile data in the event of data gaps, using Figure 2.

[0042] Figure 2 shows the current image being decoded on the left, and the reference image referenced by the motion vector obtained from decoding the current image on the right.

[0043] Block 201 of the current image represents a block in which the loss detection unit 123 detected a loss while decoding the tactile data from the tip of the dog's ear. Although the description of the loss detection unit 123 states that it operates on a pixel-by-pixel basis, here it means a block containing a missing pixel. The detection unit of the loss detection unit 123 may also be a block-by-block unit.

[0044] Motion vector 202 is the motion vector obtained when block 201 is image-encoded. Motion vector 202 points to block 203 in the reference image.

[0045] The tactile compensation unit 122 can determine the position of block 203 by receiving motion vectors from the motion vector storage unit 121. It can also acquire tactile data at the position of block 203 by receiving a reference image including block 203 from the tactile frame memory 125.

[0046] When decoding the current image, tactile data was missing from the ear tip block. However, by using motion vectors, it was possible to obtain tactile data from blocks with similar image patterns, thereby interpolating the missing tactile data.

[0047] Furthermore, because motion vectors are used, it is possible to obtain tactile data from a reference image even when the subject is moving.

[0048] (Operation timing of the tactile data decoding unit 101 and the image data decoding unit 102) Next, the operating timing of the tactile data decoding processing unit 101 and the image data decoding processing unit 102 will be explained using Figure 3.

[0049] "Image Decoding Processing" indicates the processing timing of the image decoded by the image data decoding processing unit 102. The letters IBP indicate the picture type during encoding; I is an I-picture, which is encoded using intra-prediction only. B and P are pictures that primarily use inter-prediction. The number next to IBP indicates the display order.

[0050] The "motion vector storage unit" indicates the timing at which the motion vector and reference image number are stored in the motion vector storage unit 121. For example, if B0 is written between time T1 and time T2, this indicates that the image decoding process starts at time T0, and the timing at which the motion vector and reference image number of B0 can be accessed from the motion vector storage unit 121 after all of the decoding of B0 is completed is from time T1.

[0051] The "tactile decoding process" starts one image after the image decoding process. This is to store the motion vectors and reference image numbers obtained from the image decoding process.

[0052] The tactile decoding process follows the same sequence as the image decoding process, decoding in the order of the images.

[0053] The "tactile frame memory" indicates the timing at which the tactile data decoded by the tactile decoding process is stored in the tactile frame memory 122. It shows that all the tactile data of I2, which was decoded starting at time T0, has been stored in memory by time T1.

[0054] As shown in Figure (1), during the "tactile decoding process," a missing block is detected by the missing block detection unit 123 while decoding the tactile information of B3. P5 is used as the reference image for the image of B3 during image encoding.

[0055] Since a loss was detected during decoding of B3 at time T4, the motion vector of B3 recorded at time T4 can be obtained from the motion vector storage unit 121, and the tactile data of the referenced P5 can also be obtained as it was recorded in the tactile frame memory 125 at time T4.

[0056] By using motion vectors in this way, it is possible to interpolate tactile data even when tactile data is missing by finding similar tactile data in the image, and it is also possible to interpolate tactile data even when the subject is moving.

[0057] <Second Embodiment> The decoding device of a second embodiment to which the present invention is applied will be described below with reference to Figure 4. The second embodiment is a decoding device applicable to cases such as when no motion vector exists. The decoding device is an example of a signal processing device. The following description will omit explanations of configurations similar to the first embodiment and will focus on the differences. The processing of this embodiment is performed, for example, by a CPU (not shown) executing a program stored in a memory (not shown). The CPU is an example of a processing means.

[0058] (Tactile data interpolation processing) Figure 4 is a flowchart showing the tactile data interpolation process. A control method using the tactile data decoded by the tactile data decoding unit 120 shown in Figure 1 will be described.

[0059] In step 400, the tactile data decoded by the tactile data decoding unit 120 is scanned (read out) from the area displayed on the image display unit 115 or the tactile display 126. The scanning unit may be one pixel or a predetermined block.

[0060] In step 401, the missing data detection unit 123 is used to detect whether tactile data exists in the scanned tactile data. This is confirmed using CRC or similar methods, and if tactile data exists, the tactile data interpolation process is terminated. If tactile data does not exist, the process proceeds to step 402.

[0061] In step 402, it is determined whether there are more than a threshold number of pixels or blocks containing tactile data in the image area displayed by the image display unit 115 or the tactile display unit 126. If there is more than a threshold amount of tactile data, the process proceeds to step 403; otherwise, the process proceeds to step 404.

[0062] Steps 403, 404, and 405 involve interpolating tactile data for pixels and blocks where tactile data detected in step 401 does not exist.

[0063] In step 403, the surrounding image is searched using an image with the same coordinates as pixels or blocks for which tactile data does not exist. This image is the image decoded by the image data decoding processing unit 102.

[0064] The areas to search are pixels and blocks that have tactile data, and pixels and blocks that do not have tactile data, surrounding the pixels and blocks that do. Since there is more than a threshold of tactile data within the displayed image area, searching the surrounding area makes it easier to find images similar to the image at the same coordinates as pixels and blocks that do not have tactile data.

[0065] Step 404 involves retrieving and interpolating tactile data for pixels or blocks where tactile data is missing from the database. Using the image features at the same coordinates as the pixels or blocks where tactile data is missing as keys, tactile data from images similar to the aforementioned image is read from the database. It is assumed that the database stores tactile data with pixel values ​​as keys. Although only pixel values ​​are described as keys here, the object of the subject could also be used as a key.

[0066] In step 404, since there are few pixels or blocks with tactile data around pixels or blocks that do not have tactile data, it is difficult to find tactile data that can be interpolated by searching, so interpolation is performed by referring to the database.

[0067] In step 405, the tactile data obtained from the surrounding area search in step 403 and the database lookup in step 404 is replaced with tactile data for pixels or blocks where tactile data does not exist.

[0068] (Example of missing tactile data) Next, Figure 5 illustrates an example of missing tactile data.

[0069] Image 501 is a picture of a dog, and every pixel in this image contains tactile data.

[0070] Image 502 is a magnified view of the dog's ear in image 501. When magnified, the pixels become coarser, but various interpolation algorithms exist for images, so these are used to interpolate between pixels and generate a high-resolution image.

[0071] When using a device that allows users to experience tactile sensations, there are times when users want to check the texture in more detail or examine the finer points, and there are use cases where the image is enlarged by operating the image display unit 115. However, in this case, tactile data may be lost due to the enlargement.

[0072] Figure 503 shows the pixels in Figure 502 that have tactile data and those that do not. Pixels enclosed in shaded rectangles are those with tactile data, while all other pixels do not.

[0073] Figure 6 illustrates the search area around step 403. Figure 6 is the same as 503 in Figure 5, showing pixels where tactile data exists and pixels where it does not.

[0074] Let's assume that pixel 601 is missing tactile data and we want to interpolate it. We search the pixels surrounding pixel 601 that have tactile data. This area is represented by 602. In the case of Figure 6, there are 12 target pixels within 602. We search for pixel values ​​similar to the pixel value of 601 among the 12 pixels in 602. We look for the pixel value that minimizes the difference between the pixel value of 601 and each of the other pixel values. This has been explained on a pixel-by-pixel basis, but the search can also be done on a pixel block basis.

[0075] Let's assume that the search results show that the pixel value of 603 has the smallest difference from the pixel value of 601. The tactile data of 601 is interpolated by copying the tactile data held by 603 as the tactile data of 601.

[0076] Next, we will move to step 404, where we will explain how to obtain haptic data from the database using Figure 7.

[0077] (Example of retrieving tactile data from a database using pixel values ​​as keys) Figure 7 is a block diagram showing an example configuration of the decoding device 700 and its peripheral modules.

[0078] It consists of a memory control unit 701, a tactile data decoding unit 702, an image decoding unit 703, an image frame memory 704, a memory controller 705, a tactile data acquisition unit 706, a tactile data DB 707, a data loss detection unit 708, a switch 709, and a tactile display unit 710.

[0079] The memory control unit 701, haptic data decoding unit 702, image frame memory 704, missing data detection unit 708, and haptic display unit 710 have the same configuration as shown in Figure 1, so their explanation is omitted.

[0080] The image decoding unit 703 is the same as the image data decoding processing unit 102. The image frame memory 704 and the image frame memory 114 are the same. The image frame memory 704 stores the decoded image.

[0081] The memory controller 705 reads the image of the tactile data that the tactile data decoding unit 702 is currently decoding from the image frame memory 704. It also receives the coordinates currently being decoded from the tactile data decoding unit 702 as input, extracts the pixels at the same coordinates from the read image, and outputs them to the tactile data acquisition unit 706.

[0082] The tactile data acquisition unit 706 acquires tactile data from the tactile data database using the input pixel as a key. The acquired tactile data is output to the switch 709.

[0083] The missing data detection unit 708 scans the tactile data and, if there is missing data, selects the tactile data output from the tactile data acquisition unit 706. If there is no missing data, it selects the tactile data output from the tactile decoding unit 702.

[0084] (Example of acquiring tactile data using keys other than pixel values) Next, Figure 7 shows how tactile data was retrieved from the database using pixel values ​​as the key. Figures 8 and 9 will explain an example of retrieving tactile data using keys other than pixel values.

[0085] Figure 9 shows the database keys and their corresponding haptic data. In this example, pixel value, object, and frequency component are listed as keys, but other keys that can identify haptic data may also be used.

[0086] In this example, the tactile data is 8 bits, and values ​​closer to 255 indicate a harder touch, while values ​​closer to 0 indicate a softer touch. Although hardness is used as an example for tactile data in this example, other values ​​representing different tactile sensations may also be used.

[0087] In addition to pixel values ​​and objects, high, mid, and low frequency components are defined. The presence of frequency components allows for the reading of haptic data that takes into account the state of surrounding pixels, even when the pixel values ​​and objects are the same.

[0088] Figure 8 will be used to explain the method for calculating frequency components. 801 is a diagram showing the region where the pixels to be interpolated from the tactile data and the orthogonal transformation coefficients are obtained.

[0089] Assume that pixel position 802 is a pixel position where no tactile data exists and that interpolation is to be performed. In this example, the region to be orthogonally transformed is defined as a range of ±3 pixels centered on pixel position 802. The orthogonal transformation can be performed using the orthogonal transformation processing unit within the image data decoding processing unit 102, or a new orthogonal transformation unit can be created.

[0090] A high value of the orthogonal transformation coefficient centered on pixel position 802 means that the surrounding pixels of 802 have a high concentration of high-frequency components. In the example of the dog's fur tip, this means that a hard tactile sensation is obtained because it is the tip of the fur.

[0091] In this way, even when tactile data is missing in pixels or blocks where motion vectors do not exist, it is possible to interpolate tactile data that is similar to the image.

[0092] <Other Embodiments> Furthermore, the operation described using the flowchart in the above embodiment can be modified in the order of the steps to achieve the same objective.

[0093] Furthermore, it is also possible to implement this by supplying a program that implements one or more of the functions of the above-described embodiment to a system or device via a network or storage medium, and having one or more processors in the computer of that system or device read and execute the program. It is also possible to implement this by a circuit (e.g., an ASIC) that implements one or more functions.

[0094] Furthermore, the operation described using the flowchart in the above embodiment can be modified in the order of the steps to be executed as appropriate, in order to achieve the same objective.

[0095] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0096] 100, 700 Decoders 101 Tactile data decoding processing unit 102, 703 Image Data Decoding Processing Unit 103, 701 Memory Control Unit 109, 124, 709 switches 114,704 image frame memory 120, 702 Tactile data decoding unit 121 Motion vector memory unit 122 Tactile compensation unit 123, 708 Missing parts detection unit 125 Tactile Frame Memory 705 Memory Controller 706 Tactile data acquisition unit 707 Tactile Data Database

Claims

1. Image data decoding means for decoding image data, A tactile data decoding means for decoding tactile data corresponding to the aforementioned image data, A storage means for holding motion vectors decoded based on the aforementioned image data, A detection means for detecting missing tactile data decoded by the tactile data decoding unit, A signal processing device characterized by having a processing means for interpolating tactile data based on the motion vector when the detection unit detects a loss of tactile data.

2. The signal processing device according to claim 1, characterized in that, if there is no motion vector corresponding to a region of image data in which the tactile data is missing, the processing means interpolates using tactile data from a region of image data similar to the region.

3. The signal processing device according to claim 1, characterized in that, if there is no motion vector corresponding to a region of image data in which the tactile data is missing, the processing means interpolates using tactile data identified from a database of tactile data based on the image data decoding means and orthogonal transformation data.

4. The signal processing apparatus according to claim 3, further comprising orthogonal transformation means, wherein a decoded image decoded by an image data decoding means is input, an orthogonal transformation is performed, and orthogonal transformation data is obtained.

5. Image data decoding means for decoding image data, A tactile data decoding means for decoding tactile data corresponding to the aforementioned image data, A detection means for detecting missing tactile data decoded by the tactile data decoding unit, The signal processing apparatus is characterized by having a tactile data decoding unit that performs interpolation of abnormal tactile data using tactile data from an image data region similar to the region of image data in which the tactile data is missing.

6. The signal processing device according to claim 5, wherein the processing means performs interpolation when the number of regions having tactile data is less than a predetermined threshold, and when the number of regions having tactile data is equal to or greater than a predetermined threshold, it performs interpolation using tactile data identified from a database of tactile data based on the image data decoding means and orthogonal transformation data.

7. The signal processing apparatus according to claim 6, further comprising orthogonal transformation means, inputting a decoded image decoded by an image data decoding means, performing an orthogonal transformation, and acquiring orthogonal transformation data.

8. The signal processing device according to claim 5, characterized in that the detection unit detects the loss of tactile data when the image is enlarged.

9. The signal processing device according to claim 5, characterized in that, if there is no motion vector corresponding to a region of image data in which the tactile data is missing, the processing means interpolates using tactile data from a region of image data similar to the region.

10. The signal processing apparatus according to claim 5, characterized in that, if there is no motion vector corresponding to a region of image data in which the tactile data is missing, the processing means interpolates using tactile data identified from a database of tactile data based on the image data decoding means and orthogonal transformation data.

11. A method for controlling a signal processing device, Image data decoding step, A tactile data decoding step for decoding tactile data corresponding to the aforementioned image data, A storage step that holds a motion vector decoded based on the aforementioned image data, A detection step for detecting a loss in the tactile data decoding unit, A control method for a signal processing device, characterized by comprising: a processing step of interpolating tactile data based on the motion vector when the detection unit detects a loss of tactile data.

12. A method for controlling a signal processing device, Image data decoding step, A tactile data decoding step for decoding tactile data corresponding to the aforementioned image data, A detection step to detect missing tactile data decoded by the tactile data decoding unit, A control method for a signal processing device, characterized by comprising a processing step of interpolating abnormal tactile data using tactile data from an image data region similar to an image data region in which tactile data is missing.

Images