Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus and an image processing method that can appropriately handle image data with which information for reproducing a tactile sensation of a subject is associated.SOLUTION: An image processing apparatus applies image processing on image data with which haptics data used for presenting a haptic force sense is associated. When the image processing has been applied to the image data, the image processing apparatus changes the haptics data associated with image data of an area to which the image processing has been applied, of the image data, to present senses of touch and force corresponding to an image shown by the image data in the area after the image processing is applied.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method, and particularly to an image processing apparatus and an image processing method capable of handling image data with information for reproducing tactile and force sensations attached thereto.

Background Art

[0002] In recent years, haptics technology for presenting tactile and force sensations to users has attracted attention (Patent Document 1). By using haptics technology, it is possible to provide users with information such as the tactile sensation and hardness of an object, which cannot be obtained from vision or hearing.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, it is conceivable to record information (haptics data) for presenting the tactile force sensation of a subject in association with video data. By providing a tactile sensation to the user through a known tactile force sensation interface device using an actuator or the like during playback of the video data, a richer user experience can be realized.

[0005] Therefore, in one aspect of the present invention, there is provided an image processing apparatus and an image processing method capable of appropriately handling image data associated with information for reproducing the tactile sensation of a subject.

Means for Solving the Problems

[0006] According to one aspect of the present invention, there is provided an image processing apparatus that applies image processing to image data associated with haptic data used to present tactile sensation. When image processing is applied to the image data, a first haptic data, which is haptic data associated with the image data of the area to which the image processing is applied among the image data, is changed so as to present a tactile sensation corresponding to the image represented by the image data of the area after the image processing is applied. The image processing apparatus is characterized by having a changing means.

Advantages of the Invention

[0007] According to one aspect of the present invention, it is possible to provide an image processing apparatus and an image processing method capable of appropriately handling image data associated with information for reproducing the tactile sensation of a subject.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and a plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0010] ●(First Embodiment) FIG. 1 is a block diagram showing a functional configuration example of an image processing apparatus 100 according to an embodiment of the present invention. The image processing apparatus 100 may be a computer device such as a desktop or laptop personal computer, a tablet computer. The image processing apparatus 100 may also be an electronic device having one or more processors capable of executing an application program, such as a digital camera, a media player, a personal digital assistant (PDA), a smartphone, a game machine, or the like.

[0011] The image processing apparatus 100 handles information (haptic data) representing a tactile sensation (tactile sense, force sense, temperature sense, etc.), but does not necessarily have a sensor for acquiring haptic data and a tactile interface device for reproducing the haptic data to present a tactile sensation to the user.

[0012] Before explaining the functional configuration of the image processing apparatus 100, the image data handled by the image processing apparatus 100 will be explained. The image data is data representing a still image or a moving image, and haptic data is associated therewith. There is no limitation on the method of association, but here, for example, it is assumed that the image data and the haptic data are stored in one data file using a known container format.

[0013] The haptic data may be stored in a data file separate from the data file storing the image data. In this case, the association of the data files can be realized by any known method, such as giving a commonality to the file names of the two data files or including information for specifying the associated data file as metadata or the like.

[0014] Haptic data is data for reproducing a specific haptic sensation by controlling devices such as motors, actuators, vibrators, and heaters that a haptic interface device having a tactile sense has. Haptic data can be generated by known techniques based on values detected using various sensors. Further, the present invention does not depend on the method of generating haptic data or the configuration of the data. Therefore, the description of the content of the haptic data will be limited to the minimum necessary for carrying out the invention.

[0015] FIG. 2 is a diagram schematically showing a configuration example of image data associated with haptic data handled in the present embodiment. Here, it is assumed that the image data is one frame of a moving image or data of a still image. Further, it is assumed that the haptic data and the image data are stored in the same data file.

[0016] The image data file 200 includes image data 201 and haptic data 202 associated with the image data 201. The haptic data 202 is associated with each region obtained by dividing the image represented by the image data 201. In FIG. 2, for easy understanding of the correspondence with the image, the haptic data 202 is represented as being two-dimensionally arranged, but the actual data structure may be different.

[0017] The haptic data 203 (hereinafter referred to as region haptic data 203) associated with the image region includes one or more combinations of perception type information 204 and intensity information 205 (hereinafter referred to as unit haptic data). Note that the shape and size of the region to which the region haptic data 203 is associated do not have to be constant. For example, the region haptic data 203 may be associated with each pixel, or one region haptic data 203 may be associated with the entire image. Further, regions having different sizes may exist. Furthermore, the region haptic data 203 may be associated with a specific subject region included in the image. Also, there may be a region in the image where the region haptic data 203 is not associated.

[0018] The perception type information 204 is information indicating the type of perception. The type of perception is not limited to the type of perception itself, such as temperature, stiffness, and friction, but may also be the type of a specific object that the perception evokes, such as the feeling of wind, water, or thorns. The strength information 205 indicates the strength of the perception in stages such as strong / medium / weak. For perceptions that cannot be expressed in stages, such as temperature, other expressions can be used, such as using the temperature itself or another numerical value corresponding to the temperature. Note that the configuration of the haptics data is not particularly limited, and it may have a structure and elements different from the above-described example. It may conform to future standard specifications or use a proprietary format.

[0019] FIG. 3 is a diagram conceptually showing the relationship between the image data and the haptics data. Here, it is assumed that the region haptics data 203 having unit haptics data whose perception type information 204 is temperature is associated with the image.

[0020] FIG. 3(A) shows the image 300 represented by the image data. The image 300 is composed of one human subject 301, two tree subjects 302, and a background 303. FIG. 3(B) visually shows the region haptics data for each rectangular region obtained by dividing the image 300 shown in FIG. 3(A) into a plurality in both the horizontal and vertical directions.

[0021] Region haptics data 311 is associated with the region of the human subject 301, region haptics data 312 is associated with the regions of the two tree subjects 302, and region haptics data 313 is associated with the background 303. All of the region haptics data 311 to 313 include unit haptics data in which the perception type information 204 is temperature and the strength information 205 indicates different temperatures.

[0022] Note that in practice, assume that the area haptic data 311 or 312 is associated with a rectangular area that includes at least part of the subject area, and the area haptic data 313 is associated with the other areas. As described above, the area haptic data may be associated with the subject area as a unit.

[0023] Here, when the perception type information 204 is "temperature", assume that the strength information 205 is described by a numerical value corresponding to a specific temperature. Fig. 3(C) shows an example of the relationship between the numerical value and the temperature. In Fig. 3(C), assume that the strength information is an 8-bit value (0 to 255) and represents the range of 0 to 50 °C. Also assume that the haptic data 311 to 313 have strength information 192, 128, and 64 respectively. The correspondence between the range of the strength information and the temperature range may be included in the area haptic data 203 as, for example, unit haptic data having a specific perception type information 204, or as information separate from the perception type information 204 and the strength information 205.

[0024] A tactile interface device or its control device capable of presenting a temperature sensation can use the strength information included in the unit haptic data whose perception type information is temperature for presenting the temperature sensation. For example, when a touch operation is detected while the image 300 is displayed on the touch display, the tactile interface device is controlled based on the haptic data associated with the area including the coordinates where the touch operation is detected, and the temperature sensation can be presented to the user of the device.

[0025] Returning to Fig. 1, the configuration of the image processing apparatus 100 will be described. The CPU 102 is one or more processors capable of executing programs. The CPU 102 reads a program stored in, for example, the ROM 103 or the storage medium 106 into the RAM 104 and executes it. The CPU 102 realizes the functions of the image processing apparatus 100, including the image editing operation described later, by executing the program and controlling the operations of each functional block.

[0026] The ROM 103 is, for example, an electrically rewritable non-volatile memory that stores programs executable by the CPU 102, set values, GUI data, and the like.

[0027] The RAM 104 is used to load programs executed by the CPU 102 and store values necessary during program execution. Also, a part of the RAM 104 may be used as a video memory for the display unit 110.

[0028] The storage medium 106 has a larger storage capacity than the ROM 103 and stores basic software (OS), application programs, user data, and the like. Image data handled by the image processing apparatus 100 and application programs for handling the image data are stored in the storage medium 106. The storage medium 106 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a removable memory card, or the like.

[0029] The medium control unit 105 controls data writing to and data reading from the storage medium 106 according to instructions from the CPU 102. The medium control unit 105 reads data from the storage medium 106 in fixed-size blocks according to a read instruction from the CPU 102 and stores it in the RAM 104. Also, the medium control unit 105 writes the write data stored in the RAM 104 to the storage medium 106 in fixed-size blocks according to a write instruction from the CPU 102. When the reading or writing of data according to the instruction is completed, the medium control unit 105 notifies the CPU 102 of the completion. Note that the CPU 102 may execute the functions of the medium control unit 105.

[0030] The operation unit 109 is an input device operable by the user and may be a keyboard, a mouse, a touch pad, or the like. Also, the operation unit 109 may include an external input device. When the display unit 110 is a touch display, the touch panel included in the display unit 110 is included in the operation unit 109. When the CPU 102 detects an operation on the operation unit 109, it executes an operation corresponding to the detected operation.

[0031] The display unit 110 displays the GUI (menus, windows, etc.) provided by the OS or application programs. The image data stored in the video memory area of the RAM 104 is displayed on the display unit 110. When the display unit 110 is a touch display, software keys can be realized by the combination of the image displayed on the display unit 110 and the touch panel. Note that the display unit 110 may be an external device instead of a component of the image processing apparatus 100.

[0032] The image data processing unit 107 provides the user with functions related to the editing and processing of image data associated with haptics data. The image data processing unit 107 enables editing and processing of the image data selected by the user for editing among the image data stored in the storage medium 106. The functions provided by the image data processing unit 107 may be the same as those provided by known retouch applications or video editing applications. To exemplify some of the functions, there are tone correction, color correction, brightness correction, white balance adjustment, area cropping (trimming), synthesis, scaling, file (clip) combination / splitting / deletion, addition of various effects, etc., but are not limited thereto.

[0033] Each time the image data processing unit 107 applies editing or processing (hereinafter referred to as image processing) to the image data, it notifies the haptics data processing unit 108 of the image processing information. The image processing information includes information indicating the content of the applied image processing (type of processing and, for example, amount of processing) and information indicating the area to which the editing or processing has been applied. The information indicating the content of the image processing may vary depending on the type of processing. Note that the image data processing unit 107 also notifies the haptics data processing unit 108 of the image processing information regarding the cancellation (undo) of the application.

[0034] The editing of the image data is interactively performed by the user operating, through the operation unit 109, the GUI provided by the image processing application executed by the CPU 102 on the display unit 110. The CPU 102 edits the video data according to the GUI and the operation content operated through the operation unit 109, and displays the editing result. Since such editing of the image data is well-known, further details thereof will be omitted.

[0035] The haptics data processing unit 108 changes the haptics data associated with the image data according to the image processing applied by the image data processing unit 107 to the image data. Specifically, the haptics data processing unit 108 changes the haptics data associated with the image so as to match the image represented by the image data to which the image processing is applied, based on the image processing information notified from the image data processing unit 107. Details of the operation of the haptics data processing unit 108 will be described later.

[0036] Note that both the image data processing unit 107 and the haptics data processing unit 108 can be implemented using hardware circuits such as ASIC, FPGA, and GPU. Alternatively, the processes described below as the operations of the image data processing unit 107 and the haptics data processing unit 108 may actually be realized by the CPU 102 executing an image processing application program.

[0037] The video data (including haptics data) being edited is temporarily stored in the RAM 104. When a save instruction or an editing completion instruction from the user is detected, the CPU 102 causes the data stored in the RAM 104 to be stored in the storage medium 106 through the medium control unit 105.

[0038] The haptics device connection part 111 is an interface for connecting a tactile sensation interface device that presents a tactile sensation to the user based on haptics data. The haptics device connection part 111 may be a communication interface generally used for communicably connecting a computer device and an external device, or may be a dedicated interface. The haptics device connection part 111 may include one or more of, for example, a USB interface, a Bluetooth (registered trademark) interface, an NFC interface, etc.

[0039] The tactile sensation interface device may be, for example, a glove-shaped device having devices that generate vibration or pressure (motors, actuators, vibrators, etc.), devices with variable temperature (heaters, Peltier elements, etc.). When the user wears the tactile sensation interface device on the hand and touches the display part 110 which is a touch display, a tactile sensation output corresponding to the image at the touched position is presented. The user can experience, for example, the feel and temperature of the subject shown at the touched position.

[0040] Note that the image processing device 100 may have components that can be used as a tactile sensation interface device that presents a tactile sensation to the user through the housing of the image processing device 100 and / or the display part 110, such as a vibrator or an actuator. In this case, the components that can be used as a tactile sensation interface device may be regarded as being connected to the haptics device connection part 111.

[0041] Function blocks other than the storage medium 106 are communicably connected to each other by the bus 101. The bus 101 has an address bus, a data bus, and a control bus.

[0042] <Haptics Data Processing> FIG. 4 is a flowchart related to the operation of the haptics data processing unit 108. For example, assume that an image editing application program is being executed by the CPU 102 in the image processing apparatus 100. When the CPU 102 detects that an editing or processing instruction for the image data to be edited has been input through the operation unit 109, it instructs the image data processing unit 107 to apply the processing corresponding to the input instruction to the image data.

[0043] The image data processing unit 107 applies the processing corresponding to the instruction to the image data, and stores the image data to which the processing has been applied in the video memory area in the RAM 104. As a result, the processing is reflected in the image data being edited that is displayed on the display unit 110. Further, the image data processing unit 107 notifies the haptics data processing unit 108 of the image processing information.

[0044] Here, assume that the haptics data is composed of the area haptics data 203 associated with each area of the image as shown in FIG. 2. Each time the haptics data processing unit 108 is notified of the image processing information from the image data processing unit 107, it specifies the area to which the image processing has been applied based on the image processing information. Then, the haptics data processing unit 108 executes the operations shown in the flowchart of FIG. 4 on the area haptics data associated with the area to which the image processing has been applied.

[0045] In S401, the haptics data processing unit 108 acquires the area haptics data associated with the image area. The haptics data processing unit 108 can acquire the target area haptics data from, for example, the haptics data stored in the RAM 104 together with the image data being edited. Note that if the target area haptics data can be specified, it is not necessary to read the area haptics data into the haptics data processing unit 108.

[0046] In S402, the haptics data processing unit 108 determines whether the target area haptics data includes unit haptics data whose perception type information is temperature. If the haptics data processing unit 108 determines that it includes, it executes S403; if not, it executes S404.

[0047] In S403, the haptics data processing unit 108 calculates temperature information based on the content of the processing applied to the image area. For example, the haptics data processing unit 108 determines whether the type of processing corresponds to a processing that affects a predetermined temperature. For example, processing such as color temperature change processing or brightness change processing can be regarded as processing that affects temperature, but it is not limited to these.

[0048] When the haptics data processing unit 108 determines that the type of processing corresponds to processing that affects temperature, it calculates the temperature (strength information) according to the processing amount. For example, the haptics data processing unit 108 can calculate the temperature by applying a gain corresponding to the processing amount to the temperature described in the strength information of the current unit haptics data. The gain can be determined based on, for example, a table or calculation formula that describes the relationship between the predetermined processing amount and the gain for each type. The relationship between the processing amount and the gain is determined to match the temperature change generally associated with the change in the image before and after processing, such as the temperature rising in processing to lower the color temperature (increase redness) and the temperature dropping in processing to raise the color temperature (increase blueness). Therefore, in the case of brightness, the relationship between the processing amount and the gain is determined so that the temperature rises as it gets brighter and drops as it gets darker. In the image processing information, the direction of processing (increase / decrease, + / -, etc.) is indicated by the sign of the processing amount or other information. When the haptics data processing unit 108 calculates the temperature, it executes in S407 the process of updating the strength information with the calculated temperature.

[0049] Note that the strength information (temperature) of the haptics data of the perception type information "temperature" associated with the image to which image processing is applied can also be calculated by adding (or subtracting) an offset value to the strength information.

[0050] Using FIGS. 5 and 6, a method for calculating the strength information of haptic data of the perception type information "temperature" using an offset value when the image data processing unit 107 applies color temperature adjustment to image data will be described. The calculation of temperature is to process by adding or subtracting a predetermined value according to image processing to the strength information of temperature. Hereinafter, the value added or subtracted to the strength information is called an offset value.

[0051] Here, as an example, it is assumed that the color temperature of the ambient light is adjusted to 3200K by the image processing by the image data processing unit 107 for the image data taken with a color temperature setting of 4800K. When applying the color temperature adjustment, the image data processing unit 107 includes the color temperatures before and after the adjustment in the image processing information as the processing amounts.

[0052] FIG. 5(B) shows an example of the relationship between the processing amounts (color temperatures before and after the change) in the image processing information and the offset value. When the color temperature is adjusted to a value smaller than that at the time of shooting, the blue color of the image increases as compared with before the adjustment. Generally, since blue reminds of coldness, the offset value applied to the temperature is a negative value. On the other hand, when the color temperature is adjusted to a value larger than that at the time of shooting, the red color of the image increases as compared with before the adjustment. Generally, since red reminds of warmth, the offset value applied to the temperature is a positive value. However, depending on the value of the color temperature before the change, the relationship between the amount of change in the color temperature and the magnitude of the offset value may be different. Therefore, a table or the like showing the relationship between the amount of change and the offset value can be prepared in advance according to the color temperature before the change.

[0053] Here, since it is assumed that the adjustment from 4800K to 3200K is applied, the offset value is -32. When the color temperature adjustment is applied to the entire image, as schematically shown by 701 in FIG. 5(A), a uniform offset value (-32) is distributed over the entire image.

[0054] In FIG. 6(A), 801 schematically shows the distribution of the strength information of the haptics data of the perceptual type information "temperature" associated with the image 300 before the image processing is applied to the image 300.

[0055] 802 schematically shows the offset value determined by the haptics data processing unit 108 based on the image processing information and its distribution. It is the same as FIG. 5(A). 803 schematically shows the value and its distribution of the strength information after processing, where the offset value shown by 802 is applied (added) to the strength information shown by 801.

[0056] Before the image processing is applied, the haptics data of the perceptual type information "temperature" with strength information of 192, 128, and 64 are associated with the human subject 806, the two tree subjects 804 and 805, and the background 810 included in the image 300, respectively. Due to the application of the image processing that changes the color temperature from 4800K to 3200K, the offset value -32 is uniformly applied to the strength information. As a result, the strength information of the human subject 807, the two tree subjects 808 and 809, and the background 811 in the image to which the image processing is applied is changed to 160, 96, and 32, respectively.

[0057] In S403, the process up to the calculation (determination) of the offset value is executed, and the process of changing (processing) the strength information using the offset value is to be executed in S407.

[0058] The haptics data processing unit 108 processes (modifies) the haptics data so as to suppress the sense of incongruity between the image after the image processing is applied and the tactile output based on the image processing information of the image data processing unit 107. For example, consider a case where the redness in a region associated with haptics data presenting warmth decreases due to image processing that changes the color temperature. If the haptics data is not changed, the user will feel a sense of incongruity because warmth is presented despite the decrease in redness. On the other hand, according to the present embodiment, since the haptics data processing unit 108 changes the haptics data so as to lower the temperature presented, the possibility that the user feels a sense of incongruity can be reduced.

[0059] Also, the modification of the haptics data is automatically performed in accordance with the application of the image processing. Therefore, the user only needs to interactively apply the image processing without being aware of whether haptics data is associated with the image data being processed.

[0060] Note that if the type of processing does not correspond to processing that affects the temperature, the haptics data processing unit 108 does not calculate the temperature. In this case, the haptics data processing unit 108 may skip S407 and execute S408.

[0061] In S404, the haptics data processing unit 108 determines whether to change the perception type information. If the haptics data processing unit 108 determines to change, it determines the perception type information after the change. The details of the operation of S404 will be described later.

[0062] In S405, the haptics data processing unit 108 branches the processing according to the determination result in S404. Specifically, if the haptics data processing unit 108 determines to change the perception type information, it executes S406, and if it determines not to change, it executes S407.

[0063] In S406, the haptics data processing unit 108 changes the type of perception information of the unit haptics data to be processed to the type determined in S404. After that, the haptics data processing unit 108 executes S407.

[0064] In S407, the haptics data processing unit 108 processes (changes) the strength information of the unit haptics data. Details will be described later.

[0065] In S408, the haptics data processing unit 108 determines whether all of the unit haptics data included in the region haptics data has been processed. If the haptics data processing unit 108 determines that all of the unit haptics data has been processed, it executes S409. If not, it executes S402 for the unprocessed unit haptics data.

[0066] In S409, the haptics data processing unit 108 determines whether there is an image region for which the processing of the region haptics data has not been completed. If it determines that there is, it executes S410. If not, it ends the operation of the flowchart in FIG. 4.

[0067] In S410, the haptics data processing unit 108 updates the image region to be processed and executes the processing after S401.

[0068] Next, the operation of the haptics data processing unit 108 in S404 will be further described using the flowchart shown in FIG. 7.

[0069] In S901, the haptics data processing unit 108 refers to the perception type information 204 of the unit haptics data to be processed and determines whether it corresponds to a predetermined type of change target. Here, it is assumed that types other than temperature are change targets. If the haptics data processing unit 108 determines that the unit haptics data to be processed is a predetermined type of change target, it executes S902. If not, since the change of the perception type information is not necessary, it ends the processing in FIG. 7.

[0070] In S902, the haptics data processing unit 108 determines whether the image processing applied to the area corresponds to a processing to be changed as predetermined. The haptics data processing unit 108 can perform the determination, for example, by referring to a list of processes to be changed as predetermined using information indicating the content of the image processing extracted from the image processing information. Note that the list of processes to be changed may differ for each type of perception. Therefore, the haptics data processing unit 108 may determine whether the image processing included in the list of processes to be changed corresponding to the perception type information of the unit haptics data to be processed is applied.

[0071] If it is determined that the image processing applied to the area corresponds to a processing to be changed as predetermined, the haptics data processing unit 108 executes S903. If it is not determined, since the change of the perception type information is unnecessary, the process of FIG. 7 is terminated.

[0072] In S903, the haptics data processing unit 108 determines whether the processing amount of the image processing applied to the area is equal to or greater than a predetermined threshold value. If it is determined that the value is equal to or greater than the threshold value, S904 is executed. If it is not determined, since the change of the perception type information is unnecessary, the process of FIG. 7 is terminated. The haptics data processing unit 108 can perform the determination, for example, by comparing a predetermined threshold value for at least one of the type of image processing and the type of perception with the processing amount included in the image processing information. Note that the processing amount does not necessarily have to be a numerical value. When the processing amount is not a numerical value, in S903, the haptics data processing unit 108 determines whether the processing amount corresponds to predetermined content. If it is determined that it corresponds, S904 may be executed.

[0073] As described above, when the perception type of the unit haptics data, the content of the image processing, and the processing amount satisfy predetermined conditions, the haptics data processing unit 108 determines that it is necessary to change the perception type. The conditions used for the determination in S901 to S903 are set in advance so that the perception type is changed when an appropriate tactile sensation cannot be presented for the image after the change with the type before the change remaining.

[0074] In S904, the haptics data processing unit 108 determines the changed perception type information. Here, with reference to FIG. 8, an example of the operation of changing the perception type information in the haptics data processing unit 108 will be described.

[0075] FIG. 8 schematically shows an example of the change of haptics data performed by the haptics data processing unit 108 when image processing is applied to the image 1001 and it changes to the image 1011.

[0076] The image 1001 before the application of image processing is an image focused on a cactus. Assume that strong filter processing that greatly reduces the sharpness is applied to at least the cactus area of this image 1001 by the image data processing unit 107. As a result, in the image 1011 to which the filter processing is applied, the spines of the cactus are blurred and in a state where they can hardly be distinguished.

[0077] Among the haptics data 1002 associated with the image data 1001, the area haptics data associated with the area 1004 includes the unit haptics data 1003. Since the area 1004 corresponds to the cactus, the unit haptics data 1003 has the perception type information "spine" to present the feeling of spines.

[0078] If the unit haptics data 1003 is maintained also for the image 1011, the appearance of the area 1004 of the image 1011 and the tactile output (interval of spines) presented for the area 1004 do not match. However, it is difficult to present an appropriate tactile sensation for the image area that appears smooth and corresponds to the area 1004 with the unit haptics data whose perception type is "spine".

[0079] Therefore, the haptics data processing unit 108 · the perception type of the target unit haptics data is "spine" (S901, Y), ·A filter process that reduces sharpness is executed based on the image processing information (S902, Y), and the sharpness reduction by the filter process is large (the processing amount is large) (S903, Y). Based on this, it is determined to change the perception type information of the unit haptic data associated with the region 1004 from "thorn" to "friction" that can present a tactile sense of force corresponding to a smooth surface with less tactile stimulation.

[0080] Note that what is described here is an example of the change in perception type information. In reality, the haptic data processing unit 108 can change the perception type information according to more patterns.

[0081] Next, a specific example of the processing operation of the strength information in S407 will be described with reference to FIGS. 9 and 10. FIG. 9 schematically shows an example of the processing of the strength information when a filter process for blurring areas other than the target area is applied in the image data processing unit 107. For the haptic data of the perception type information "temperature", the strength information may be replaced with the temperature calculated in S403, or the offset value determined in S403 may be applied (added) to the strength information. Therefore, here, the processing operations of other types of haptic data will be described.

[0082] In FIG. 9(A), 500 schematically represents the content represented by the image processing information notified from the image data processing unit 107. The content of the image processing represented by the image processing information is a filter process that reduces sharpness, and the processing amount is the intensity setting value of the filter process. For example, in the case of Gaussian filter processing, the processing amount is the setting value of the blur radius, and the unit is pixels. Here, it is assumed that the image data of the image 300 in FIG. 3 is processed by the image data processing unit 107. Also, it is assumed that the target area is the area of the central human subject 301.

[0083] As shown in FIG. 9(A), filter processing with different processing amounts 501 to 504 is applied to the regions of individual subjects and the background, respectively. Also, an example of specific processing amounts is shown in FIG. 9(B). The processing amount is the set value of the blur radius, and the larger the value, the stronger the filter intensity (the more the image is blurred). Therefore, the filter applied to the background is the strongest, and the filter intensity weakens in the order of the foreground tree subject and the background tree subject. Since the processing amount for the target subject is 0, no filter processing is applied.

[0084] When the haptics data processing unit 108 processes the strength information of the unit haptics data included in the region haptics data associated with the image 300, it shall multiply the strength information by a gain (coefficient).

[0085] In FIG. 9(C), 510 is a diagram schematically showing the gain values and their distributions applied by the haptics data processing unit 108 to the strength information. As shown in the figure, different gains 511 to 514 are used for processing the strength information for the regions of individual subjects and the background, respectively. Also, FIG. 9(D) shows specific examples of the gains 511 to 514.

[0086] The magnitudes of the gains 511 to 514 have the same relationship as the intensity of the filter processing applied to the image data. Specifically, the lower the intensity of the filter processing, the closer the gain value is to 1. A gain value of 1 corresponds to not changing the strength information. Also, the higher the intensity of the filter processing, the closer the gain value is to 0. A gain value of 0 corresponds to not presenting the tactile output.

[0087] Therefore, in the example shown in FIG. 9, the haptics data processing unit 108 does not process the strength information of the haptics data associated with the area of the subject of interest. On the other hand, for the haptics data associated with either of the two tree subjects and the background 303, the haptics data processing unit 108 applies a gain such that the strength of the haptics data decreases as the intensity of the filtering process increases. As a result, the lower the sharpness of the area, the weaker the tactile output becomes, and a tactile output with an intensity consistent with the impression received from the appearance of the image is presented.

[0088] Here, the case where image processing for reducing sharpness is applied has been described. However, even when image processing for enhancing sharpness, such as edge enhancement processing or unsharp masking processing, is applied, the strength information can be processed in the same manner. Specifically, the haptics data processing unit 108 can process the strength information by applying a gain greater than 1 so that the tactile output also becomes stronger in the area where the intensity of the applied image processing is stronger.

[0089] FIG. 10 shows an example of the processing operation of the strength information using the gain calculated as described in FIG. 9. FIG. 10(A) is a diagram schematically showing the content of the processing by the haptics data processing unit 108. FIGS. 10(B) and 10(C) show specific examples of the strength information and the gain value.

[0090] In FIG. 10(A), 601 schematically shows the distribution of the strength information of the haptics data of the perception type information "friction" associated with the image 300 before the image processing is applied to the image 300. Note that other perception types may also be used.

[0091] 602 schematically shows the gain value calculated by the haptics data processing unit 108 based on the image processing information and its distribution. This is the same as FIG. 8(C). 603 schematically shows the values and the distribution of the strength information after processing, in which the gain shown in 602 is applied (multiplied) to the strength information shown in 601.

[0092] Before the image processing is applied, the strength information 604 and 605 of the haptics data associated with the two tree subjects are the same at 128. However, as shown in FIGS. 9(A) and 9(B), due to the application of different intensity filtering processes to the two tree subjects, the gain values 606 and 607 applied to the strength information of each region become different values (1 / 2 and 3 / 4). Therefore, the processed strength information 608 and 609 also have different values (96 and 64). A gain of 1 / 4 is applied to the strength information 610 regarding the background, and the processed strength information 611 becomes 16.

[0093] As described above, the haptics data processing unit 108 processes (modifies) the haptics data based on the image processing information of the image data processing unit 107 so as to suppress the discomfort between the image after the image processing is applied and the tactile output. For example, consider the case where a filtering process that reduces sharpness is applied to a region associated with haptics data presenting a rough tactile sensation. If the haptics data is not changed, although the image has low sharpness, a rough tactile sensation is presented, so the user feels discomfort. On the other hand, according to the present embodiment, since the haptics data processing unit 108 changes the haptics data so as to reduce the roughness of the tactile sensation, the possibility that the user feels discomfort can be reduced.

[0094] Also, the modification of the haptics data is automatically performed along with the application of the image processing. Therefore, the user only needs to interactively apply the image processing without being aware of whether haptics data is associated with the image data being processed.

[0095] In the present embodiment, the haptics data processing unit 108 changes the strength information of the haptics data more greatly as the change in the image due to the applied image processing is larger. By setting the upper limit value and the lower limit value for the changed strength information, even when the image changes significantly due to the application of the image processing, the changed strength information can be kept within an appropriate range.

[0096] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. Since this embodiment can also be implemented by the image processing apparatus 100, the description of the configuration of the image processing apparatus 100 will be omitted.

[0097] In this embodiment, the haptics data processing unit 108 determines whether to process the strength information of the haptics data according to the image processing of the image data processing unit 107. If it is determined to process, the haptics data is processed in the same manner as in the first embodiment. If it is determined not to process, the haptics data processing is not performed.

[0098] FIG. 11 is a flowchart related to the operation of the haptics data processing unit 108 in this embodiment. This operation can be executed, for example, when the image data processing unit 107 notifies image processing information.

[0099] In S1101, the haptics data processing unit 108 determines whether image processing for processing haptics data is applied based on the image processing information. If it is determined that the image processing is applied, S1102 is executed; otherwise, S1103 is executed. For example, when the type of image processing described in the image processing information corresponds to the type of image processing for processing haptics data stored in advance, the haptics data processing unit 108 determines that the image processing for processing haptics data is applied.

[0100] Note that depending on the perception type of the haptics data, the type of image processing for processing the haptics data may be different. In this case, for all unit haptics data associated with the image data to which the image processing is applied, if it is determined that the type of image processing for processing the haptics data does not correspond, S1103 is executed.

[0101] For example, when the image processing is a conversion process from a color image to a black-and-white image, although the color of the image is lost, the user will not feel uncomfortable even if the strength information of the haptic data remains unchanged. Thus, since it is not necessary to change the strength information of the haptic data before and after the application of the image processing, it is determined that the processing of the strength information is invalid, and the processing of the strength information is not performed. Also, when the perception type information is temperature information, since there is no need to process the strength information for an image subjected to image processing such as texture adjustment or filter processing, it is determined that the processing of the strength information is invalid.

[0102] In S1102, the haptic data processing unit 108 executes the processing from S401. On the other hand, in S1103, the haptic data processing unit 108 waits for the notification of the next image processing information without executing the processing of the haptic data.

[0103] According to the present embodiment, when image processing that does not require processing of haptic data is applied, unnecessary processing does not need to be executed, so the power of the image processing apparatus 100 can be saved. Also, when unnecessary processing is executed, there is a risk that incorrect processing of haptic data may be executed, but such a situation can be avoided.

[0104] ●(Third Embodiment) Next, a third embodiment of the present invention will be described. Since this embodiment can also be implemented by the image processing apparatus 100, the description of the configuration of the image processing apparatus 100 will be omitted.

[0105] In the present embodiment, the operation of the haptic data processing unit 108 when the image processing applied by the image data processing unit 107 is a process of pasting another image or an area of the same image will be described. Hereinafter, it is assumed that an area of one image is pasted onto the other image, but an area within the same image may be copied and pasted.

[0106] First, the image data before processing will be described with reference to FIG. 12. FIGS. 12(A) and (C) show the images represented by the image data before the application of image processing. Also, the perceptual type information of the haptics data associated with each image is schematically shown in FIGS. 12(C) and (D). Haptics data is associated with each region of the image.

[0107] The image 1201 in FIG. 12(A) is an image that includes an area to be pasted onto another image. In this example, the image 1201 is an image of a dog 1202 in a park. 1203 in FIG. 12(B) schematically shows the distribution of the perceptual type information of the haptics data associated with the image 1201. For example, haptics data representing the texture of the dog's fur is associated with the area 1204 of the dog 1202.

[0108] FIG. 12(C) shows an image (background image) 1205 onto which an area image of another image is to be pasted. Here, it is an image of a lawn square. 1206 in FIG. 12(D) schematically shows the distribution of the perceptual type information of the haptics data associated with the image 1205. Specifically, haptics data that realizes the corresponding tactile sensations for a building in the area 1207, trees in the area 1208, and grass in the area 1209 are associated respectively.

[0109] It is assumed that the data of the images 1201 and 1205 and the associated haptics data are stored in the storage medium 106 in advance.

[0110] Next, the image processing applied by the image data processing unit 107 will be described. It is assumed that an image editing application program is being executed by the CPU 102 in the image processing apparatus 100, and the images 1201 and 1205 are arranged and displayed as editing targets.

[0111] When the image data processing unit 107 detects an instruction to select an area of the image 1201, specifically the area of the dog 1202, through the operation unit 109, it executes a known contour detection process or the like to set the area of the dog 1202 in a selected state. When a copy (or cut) instruction is detected through the operation unit 109 in this state, the image data processing unit 107 stores the data of the area of the dog 1202 in, for example, the clipboard provided by the OS. Then, when a paste instruction is detected through the operation unit 109 with the image 1205 in focus, the image data processing unit 107 superimposes and displays the image of the dog 1202 stored in the clipboard on the image 1205 in a selected state. When an operation to cancel the selected state is detected, the image data processing unit 107 composes the image of the dog 1202 with the image 1205. The composition may be a replacement, or logical operations may be performed pixel by pixel.

[0112] FIG. 13(A) shows an image 1301 in which the image of the dog 1302 is composed with the image 1205. When the image data processing unit 107 completes the composition of the images, it notifies the haptics data processing unit 108 of the image processing information. The image processing information regarding pasting (composing) one image area onto another image includes information about the pasted area (information specifying the position and contour in one image) and information specifying the composition position in the background image.

[0113] That is, in the case of this example, the image processing information includes the coordinate position information and contour information (area information of the original area to be processed) of the area of the dog 1202 in the image 1201, and the coordinate position information (area information of the destination area to be processed) of the area of the dog 1302 in the image 1301.

[0114] Next, the operation of the haptics data processing unit 108 will be described. The haptics data processing unit 108 acquires haptics data 1203 and 1206 from the RAM 104 based on the image processing information notified from the image data processing unit 107. Then, the haptics data processing unit 108 reflects the pasting (composition) process applied by the image data processing unit 107 on the haptics data 1206. FIG. 13(B) schematically shows the haptics data 1303 in which the haptics data processing unit 108 has reflected the pasting (composition) process. After the image 1301 and the haptics data 1303 are associated with each other, they are stored in, for example, the storage medium 106.

[0115] The reflection of the pasting (composition) process on the haptics data can be realized by performing the same operation as the pasting (composition) process of the image on the haptics data 1203 and 1206.

[0116] Specifically, the haptics data processing unit 108 copies or cuts the haptics data of the area indicated by the source area information included in the image processing information from the haptics data 1203. Then, the haptics data processing unit 108 replaces the haptics data of the area indicated by the destination area information included in the image processing information in the haptics data 1206 with the copied or cut haptics data.

[0117] Note that when the haptics data is associated with each rectangular area of the image as described in the first embodiment, the haptics data processing unit 108 also performs the copy (cut) and replacement of the haptics data in units of rectangular areas.

[0118] In this way, when an image process of pasting an area of another image is applied, the haptics data processing unit 108 reflects the haptics data associated with that area on the haptics data associated with the destination image.

[0119] As a result, the haptics data of the lawn area with the dog image pasted on it is replaced with the haptics data representing the dog's coat texture from the lawn haptics data, and the consistency between the image and the haptics data can be achieved.

[0120] Note that the same processing can also be executed when pasting regions within the same image. For example, in the image shown in FIG. 12(A), when the region of dog 1202 is duplicated to increase the number of dogs. In this case, the haptics data associated with the pasted region where the duplication of the region of dog 1202 is pasted may be replaced with the haptics data associated with the region of dog 1202.

[0121] Note that the image data processing unit 107 may perform processing such as enlargement, reduction, inversion, rotation, etc. on the copied or cut image region and then paste it. In this case, the image data processing unit 107 includes information on the processing method and processing content in the original region information included in the image processing information. The information on the processing method is "enlargement", "reduction", "inversion", "rotation", etc. Also, the information on the processing content is parameters corresponding to the processing method (for example, enlargement ratio, reduction ratio, rotation angle, etc.).

[0122] When the haptics data processing unit 108 includes information on processing in the original region information, it reflects the processing on the haptics data associated with the copied or cut image region and then uses it for replacement.

[0123] FIG. 14 shows an example where image processing of enlarging the region of dog 1202 in FIG. 12 and then pasting it onto image 1205 is applied. FIG. 14(A) shows the image 1401 after image processing. FIG. 14(B) schematically shows the haptics data 1403 associated with the image 1401. Comparing with the haptics data 1304 in FIG. 13(B), it can be seen that the range of the haptics data 1404 is enlarged to correspond to the enlarged dog region 1402.

[0124] FIG. 15 shows an example in which image processing is applied to reduce the area of the dog 1202 in FIG. 12 and then paste it onto the image 1205. FIG. 15(A) shows the image 1501 after the image processing. FIG. 15(B) schematically shows the haptics data 1503 associated with the image 1501. Comparing with the haptics data 1304 in FIG. 13(B), it can be seen that the range of the haptics data 1504 is reduced so as to correspond to the reduced area 1502 of the dog.

[0125] FIG. 16 shows an example in which image processing is applied to flip the area of the dog 1202 in FIG. 12 horizontally and then paste it onto the image 1205. FIG. 16(A) shows the image 1601 after the image processing. FIG. 16(B) schematically shows the haptics data 1603 associated with the image 1601. Comparing with the haptics data 1304 in FIG. 13(B), it can be seen that the area of the haptics data 1604 is flipped so as to correspond to the flipped area 1602 of the dog.

[0126] FIG. 17 shows an example in which image processing is applied to rotate the area of the dog 1202 in FIG. 12 counterclockwise and then paste it onto the image 1205. FIG. 17(A) shows the image 1701 after the image processing. FIG. 17(B) schematically shows the haptics data 1703 associated with the image 1701. Comparing with the haptics data 1304 in FIG. 13(B), it can be seen that the area of the haptics data 1704 is rotated so as to correspond to the rotated area 1702 of the dog.

[0127] Haptics data is data associated with each area of the image data. In particular, when haptics data is associated with each image area divided at equal intervals in the horizontal and vertical directions as shown in FIG. 2, the haptics data can be regarded as data obtained by discretely sampling the space in the same way as the image data. Therefore, methods for enlarging, reducing, flipping, and rotating the image data can also be applied to the haptics data. For example, it is possible to enlarge and reduce by linearly interpolating or decimating the haptics data.

[0128] The processing applied before pasting is not limited to enlargement, reduction, inversion, and rotation. Other processing such as deformation can be applied, or multiple processing methods can be combined and applied.

[0129] According to this embodiment, when image processing for pasting an image is applied, the haptics data associated with the original image to be pasted is reflected in the haptics data associated with the image at the pasting destination. Therefore, the consistency between the image and the haptics data after the application of the image processing can be maintained. Here, the description has been made on the premise that haptics data is associated with the image (region image) to be pasted. However, if no haptics data is associated with the region image, the haptics data associated with the region at the pasting destination may be deleted.

[0130] ●(Fourth Embodiment) Next, a fourth embodiment of the present invention will be described. Since this embodiment can also be implemented by the image processing apparatus 100, the description of the configuration of the image processing apparatus 100 will be omitted.

[0131] This embodiment relates to the operation of the haptics data processing unit 108 when the image data processing unit 107 applies image processing to erase a subject in an image. Erasing a subject is an operation of replacing the subject area with the background.

[0132] FIG. 18(A) shows an image 1801 before the application of image processing. FIG. 18(B) schematically shows the haptics data 1803 associated with the image 1801. The image 1801 shows a dog 1802 sitting in a park. Also, haptics data 1804 representing the dog's fur pattern is associated with the area of the dog 1802.

[0133] Next, the image processing applied by the image data processing unit 107 will be described. Assume that an instruction to erase the dog 1802 in the image 1801 is detected through the operation unit 109. The image data processing unit 107 replaces the area of the dog 1802 with the background image estimated from the surrounding images in the image data stored in the RAM 104 according to the instruction. FIG. 19(A) shows the image 1901 after the image processing is applied.

[0134] In the image 1901, the area 1902 indicated by the dashed line has been replaced from the dog image to the background image. This background image is generated by estimation from the surrounding background. Since the estimation of the background image is a well-known technique implemented in commercially available image editing applications and the like, the detailed description thereof will be omitted.

[0135] When the image data processing unit 107 applies image processing to the image 1801, it notifies the haptics data processing unit 108 of the image processing information describing the applied image processing. When the subject erasure process is applied, the image processing information includes information indicating the type of image processing and information specifying the area to which the process is applied (for example, information indicating the contour or position). In the example of FIG. 18, subject deletion is included in the image processing information as the type of image processing, and information specifying the area of the dog 1802 is included as the information specifying the area.

[0136] Next, the operation of the haptics data processing unit 108 will be described. The haptics data processing unit 108 recognizes that the subject deletion process has been applied based on the image processing information notified from the image data processing unit 107. The haptics data processing unit 108 acquires, for example, the haptics data associated with the peripheral area of the target area based on the image processing information from the haptics data 1803 stored in the RAM 104. Here, the reason for acquiring the haptics data associated with the peripheral area of the target area is to generate the haptics data for replacing the haptics data associated with the target area.

[0137] As described above, haptics data can be handled in the same way as image data. Therefore, the haptics data processing unit 108 applies the same method to the haptics data as the method by which the image data processing unit 107 generates a background image for replacing the area of the dog 1802. As a result, the haptics data processing unit 108 can generate haptics data 1904 for replacing the haptics data 1804 associated with the area of the dog 1802.

[0138] The haptics data processing unit 108 replaces the haptics data 1804 stored in the RAM 104 with the haptics data 1904 generated by inferring from the surrounding haptics data.

[0139] As a result, it is possible to suppress a discrepancy between the image and the tactile output, such as presenting the texture of a dog's fur for the area 1902 of the image 1901.

[0140] According to the present embodiment, when the deletion process of an object is applied, the haptics data associated with the area of the deleted object is replaced with the background haptics data inferred from the haptics data associated with the surrounding area. Therefore, it is possible to maintain the consistency between the image after the application of the image processing and the haptics data.

[0141] FIG. 20 is a flowchart relating to the operation of the haptics data processing unit 108 in the third and fourth embodiments.

[0142] In S2001, the haptics data processing unit 108 determines, based on the image processing information, whether the applied image processing is an image pasting process. Here, it is assumed that the image to be pasted has haptics data associated therewith. If the haptics data processing unit 108 determines that the applied image processing is an image pasting process, it executes S2003; otherwise, it executes S2011.

[0143] In S2003, the haptics data processing unit 108 obtains haptics data associated with the pasted image (region image) from, for example, the RAM 104.

[0144] In S2005, the haptics data processing unit 108 determines whether the region image has been processed before pasting. If the haptics data processing unit 108 determines that it has been processed, it executes S2007; if not, it executes S2009.

[0145] In S2007, the haptics data processing unit 108 reflects the processing applied to the region image in the haptics data obtained in S2003.

[0146] In S2009, the haptics data processing unit 108 replaces the haptics data associated with the pasting destination region with the haptics data obtained in S2003 or obtained in S2007, and ends the processing based on the notified image processing information.

[0147] In S2011, the haptics data processing unit 108 determines whether the applied image processing is an object deletion process based on the image processing information. If the haptics data processing unit 108 determines that the applied image processing is an object deletion process, it executes S2013; if not, it ends the processing of FIG. 20. Note that when the haptics data processing unit 108 determines that the applied image processing is not an object deletion process, it may execute the processing from S401 in FIG. 4.

[0148] In S2013, the haptics data processing unit 108 obtains, for example, from the RAM 104 the haptics data associated with the peripheral region of the region of the deleted object in the image data.

[0149] In S2015, the haptics data processing unit 108 generates haptics data for replacing the haptics data associated with the region of the deleted object from the obtained haptics data.

[0150] In S2017, the haptics data processing unit 108 replaces the haptics data associated with the area of the deleted object with the haptics data generated in S2015, and ends the processing based on the notified image processing information.

[0151] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and having one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0152] The disclosure of this embodiment includes the following image processing apparatus, image processing method, and program. (Item 1) An image processing apparatus that applies image processing to image data associated with haptics data used to present a tactile sensation, When image processing is applied to the image data, a changing means for changing first haptics data, which is haptics data associated with the image data of the area to which the image processing is applied among the image data, so as to present a tactile sensation corresponding to the image represented by the image data of the area after the image processing is applied. The image processing apparatus is characterized by having this. (Item 2) The haptics data is associated with an area of the image represented by the image data, The area haptics data associated with one area includes one or more unit haptics data, and the unit haptics data includes a combination of tactile sensation type information and intensity information. The image processing apparatus according to Item 1, characterized by this. (Item 3) The image processing apparatus according to Item 2, characterized in that the changing means changes the haptics data for each unit haptics data. (Item 4) When the type of the image processing is a predetermined type and the processing amount of the image processing is equal to or greater than a predetermined threshold value, the changing means changes the type information and the strength information of the unit haptic data included in the first haptic data. The image processing apparatus according to item 2 or 3, characterized in that. (Item 5) When the type of the image processing is not a predetermined type or the type of the image processing is the predetermined type but the processing amount is less than the predetermined threshold value, the changing means does not change the type information of the unit haptic data included in the first haptic data and changes the strength information. The image processing apparatus according to item 4, characterized in that. (Item 6) The changing means does not change the type information for the unit haptic data whose type information is temperature. The image processing apparatus according to any one of items 3 to 5, characterized in that. (Item 7) When the image processing is a color temperature change, for the unit haptic data whose type information is temperature included in the first haptic data, the changing means changes the strength information so that the presented temperature decreases as the color temperature increases and the presented temperature increases as the color temperature decreases. The image processing apparatus according to any one of items 2 to 6, characterized in that. (Item 8) When the image processing is a filter process for reducing sharpness, the changing means changes the unit haptic data included in the first haptic data so that the tactile stimulus presented is reduced. The image processing apparatus according to any one of items 2 to 7, characterized in that. (Item 9) When the image processing is a filter process for changing sharpness, the changing means changes the strength information of the unit haptic data included in the first haptic data by applying a gain having a value corresponding to the magnitude of the processing amount of the filter process. The image processing apparatus according to any one of items 2 to 7, characterized in that. (Item 10) When the image processing is a filter process that changes sharpness, the changing means changes the strength information of the unit haptic data included in the first haptic data by applying a gain having a value corresponding to the amount of processing of the filter process. The image processing apparatus according to any one of items 2 to 7. (Item 11) When the image processing is pasting of an image associated with haptic data, the changing means replaces the first haptic data with the haptic data associated with the image to be pasted. The image processing apparatus according to any one of items 1 to 10. (Item 12) When the image to be pasted has been processed before being pasted, the changing means reflects the processing in the haptic data associated with the image to be pasted and then replaces the first haptic data. The image processing apparatus according to item 11. (Item 13) The processing includes one or more of enlargement, reduction, rotation, inversion, and deformation. The image processing apparatus according to item 12. (Item 14) When the image processing is deletion of an object, the changing means replaces the first haptic data with haptic data inferred from the haptic data associated with the peripheral region of the object in the image represented by the image data. The image processing apparatus according to any one of items 1 to 13. (Item 15) When the image processing is image processing that does not require changing the haptic data, the changing means does not change the first haptic data. The image processing apparatus according to any one of items 1 to 14. (Item 16) An image processing method executed by an image processing apparatus that applies image processing to image data associated with haptic data used to present a tactile sensation. When image processing is applied to the image data, changing first haptic data, which is haptic data associated with the image data of the area to which the image processing is applied among the image data, so as to present a tactile sensation corresponding to the image represented by the image data of the area after the image processing is applied. An image processing method characterized by comprising this. (Item 17) A program for causing a computer to function as each means included in the image processing apparatus according to any one of Items 1 to 15.

[0153] The present invention is not limited to the contents of the above-described embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Explanation of Reference Numerals

[0154] 100... Editing device, 102... CPU, 103... ROM, 104... RAM, 107... Image data processing unit, 108... Haptic data processing unit, 109... Operation unit, 110... Display unit

Claims

1. An image processing apparatus that applies image processing to image data associated with haptic data used to present tactile force sensation, wherein when image processing is applied to the image data, first haptic data, which is haptic data associated with the image data of the area to which the image processing is applied among the image data, is changed so as to present a tactile force sensation corresponding to the image represented by the image data of the area after the image processing is applied. The image processing apparatus is characterized by having changing means.

2. The haptic data is associated with an area of the image represented by the image data, the area haptic data associated with one area includes one or more unit haptic data, and the unit haptic data includes a combination of tactile force sensation type information and intensity information. The image processing apparatus according to claim 1, characterized in that.

3. The image processing apparatus according to claim 2, characterized in that the changing means changes the haptic data for each unit haptic data.

4. The image processing apparatus according to claim 2, characterized in that when the type of the image processing is a predetermined type and the processing amount of the image processing is equal to or more than a predetermined threshold value, the type information and the intensity information of the unit haptic data included in the first haptic data are changed.

5. The image processing apparatus according to claim 4, characterized in that when the type of the image processing is not a predetermined type or the type of the image processing is a predetermined type but the processing amount is less than the predetermined threshold value, the type information of the unit haptic data included in the first haptic data is not changed and the intensity information is changed.

6. The image processing apparatus according to claim 3, characterized in that the changing means does not change the type information for the unit haptic data whose type information is temperature.

7. The image processing apparatus according to claim 2, characterized in that when the image processing is a change in color temperature, for the unit haptic data whose type information is temperature included in the first haptic data, the intensity information is changed so that the presented temperature decreases as the color temperature increases and the presented temperature increases as the color temperature decreases.

8. The changing means changes the unit haptic data included in the first haptic data so that the tactile stimulus to be presented is reduced when the image processing is a filter process that reduces sharpness. The image processing apparatus according to claim 2.

9. The changing means changes the strength information of the unit haptic data included in the first haptic data by applying a gain having a value corresponding to the magnitude of the processing amount of the filter process when the image processing is a filter process that changes sharpness. The image processing apparatus according to claim 2.

10. The changing means changes the strength information of the unit haptic data included in the first haptic data by applying a gain having a value corresponding to the magnitude of the processing amount of the filter process when the image processing is a filter process that changes sharpness. The image processing apparatus according to claim 2.

11. The changing means replaces the first haptic data with the haptic data associated with the image to be pasted when the image processing is pasting of an image associated with haptic data. The image processing apparatus according to claim 1.

12. The changing means replaces the first haptic data after reflecting the processing in the haptic data associated with the image to be pasted when the image to be pasted has been processed before being pasted. The image processing apparatus according to claim 11.

13. The image processing apparatus according to claim 12, wherein the processing includes one or more of enlargement, reduction, rotation, inversion, and deformation.

14. The changing means replaces the first haptic data with haptic data inferred from the haptic data associated with the peripheral region of the object in the image represented by the image data when the image processing is deletion of an object. The image processing apparatus according to claim 1.

15. The changing means does not change the first haptic data when the image processing is image processing that does not require changing the haptic data. The image processing apparatus according to claim 1.

16. An image processing method executed by an image processing apparatus that applies image processing to image data associated with haptic data used to present tactile sensation, when image processing is applied to the image data, changing first haptic data, which is haptic data associated with the image data of the area to which the image processing is applied among the image data, so as to present a tactile sensation corresponding to the image represented by the image data of the area after the image processing is applied. The image processing method is characterized by including this step.

17. A program for causing a computer to function as each means included in the image processing apparatus according to any one of Claims 1 to 15.

Images