Image processing apparatus, imaging apparatus, image processing system, method for controlling image processing apparatus, and program

Abstract

To provide a technique to reduce information gap between visual information and haptics information.SOLUTION: An image processing apparatus comprises: acquisition means that acquires information obtained on the basis of photographing information when image data of an object to be processed is acquired through photographing, the information on a focusing level indicating the degree of focusing on an object to be photographed included in the image data of the object to be processed; haptics changing means that makes a first change to haptics data acquired in association with the image data of the object to be processed, on the basis of the information on the focusing level; and generation means that generates an image file by associating the haptics data after the first change is made with the image data of the object to be processed. The first change includes changing the value of the haptics data of at least part of the object to be photographed.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, an image processing system, a control method for an image processing apparatus, and a program.

Background Art

[0002] In recent years, the development of haptics technology for feedback of tactile information and temperature information (thermal sensation · cold sensation information) and haptics devices for realizing the same has been progressing. Imaging devices such as digital cameras may be linked in the future with a haptics acquisition sensor to record the image data acquired by the image sensor and the haptics data acquired by the haptics acquisition sensor in association with each other. In the content recorded in this way, it becomes possible to feedback haptics data such as tactile sensation, thermal sensation, and cold sensation to the user simultaneously with the reproduction of the image data.

[0003] On the other hand, since the use of the haptics acquisition sensor is different from the image acquisition use, it is considered to be composed of a sensor different from the image sensor. In such a configuration, there is a possibility that the visual information and the haptics information may deviate. Therefore, it is desirable to eliminate the discomfort caused by the information deviation between the visual information and the haptics information.

[0004] Patent Document 1 discloses a method of recording visual information, tactile information, trajectory information of a tactile sensor, and position information in association with each other, and performing tactile presentation synchronized with the visual position based on information such as the position, moving direction, and speed at which the reproduction device touches during reproduction.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the above-described conventional technology, it is impossible to cope with information divergence between visual information and haptics information caused by shooting conditions, such as distinguishing between a focused area and an out-of-focus area.

[0007] Therefore, the present invention provides a technique for suppressing information divergence between visual information and haptics information.

Means for Solving the Problems

[0008] The invention for solving the above problems is an image processing apparatus, comprising: acquisition means for acquiring information on a focus level indicating the degree of focus of a shooting object included in the image data of the processing object, the information being obtained based on shooting information when acquiring the image data of the processing object by shooting; haptics change means for performing a first change on haptics data acquired in association with the image data of the processing object based on the information on the focus level; and generation means for generating an image file in association with the image data of the processing object for the haptics data after the first change is performed, wherein the first change includes changing the value of the haptics data of at least a part of the shooting objects.

Effects of the Invention

[0009] According to the present invention, it is possible to provide a technique for suppressing information divergence between visual information and haptics information.

Brief Description of the Drawings

[0010]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0012] [Embodiment 1] First, a configuration example of the image processing system corresponding to this embodiment will be described. FIG. 1A is a block diagram showing an example of the hardware configuration of an image processing system 10 corresponding to an example of this embodiment. The image processing system 10 can be configured by connecting an imaging device 100, a haptics acquisition device 110, and an external output device 120.

[0013] In FIG. 1A, the imaging device 100 can include a lens optical system 101, an imaging element 102, an image processing unit 103, a control unit 104, a storage unit 105, a display unit 106, an operation unit 107, and a communication unit 108. These components are merely examples, and the imaging device may include components other than those shown in FIG. 1A.

[0014] The lens optical system 101 condenses light from a subject onto the imaging element 102 through an optical lens, a diaphragm, and focus control. The imaging element 102 is, for example, a CMOS imaging sensor, and converts the light incident through the lens optical system 101 into an electrical signal and outputs it as an image signal. The image processing unit 103 performs various corrections such as filter processing and digital image processing such as compression on the image signal output from the imaging element 102. The control unit 104 controls the driving timing of the imaging element 102 and performs overall driving and control of the entire imaging device, such as the image processing unit 103, the display unit 106, and the communication unit 108. The control unit 104 has, for example, a CPU, a ROM, and a RAM, and the CPU can expand the program stored in the ROM into the working area of the RAM and execute it to perform overall operation control of the device.

[0015] The image processing unit 103 acquires shooting information when shooting is performed in the imaging device 100. The shooting information includes information unique to the imaging device and information unique to the captured image. Examples of information unique to the imaging device include the size of the sensor or the allowable confusion circle diameter, and the brightness or focal length of the optical system. Examples of information unique to the captured image include the aperture value, the focusing distance, the Bv value, the RAW image, the exposure time, the gain (ISO sensitivity), the white balance coefficient, the distance information, the position information by GPS, etc., the time information such as the date and time, and so on. In addition, examples of information unique to the captured image include the gravity sensor value, the acceleration, the geomagnetic direction, the temperature, the humidity, the atmospheric pressure, or the altitude, etc. at the time of shooting.

[0016] The storage unit 105 is a storage medium such as a non-volatile memory or a memory card that stores and holds the image signal output from the image processing unit 103. The storage unit 105 may be configured to be able to mount an external storage medium (such as a memory card), and can take in the haptics data stored in the external storage medium. The display unit 106 is a display that displays the captured image, various setting screens, and so on. The operation unit 107 is composed of buttons, a touch panel, switches, etc., accepts operation inputs from the user, and reflects the user's commands to the control unit 104.

[0017] The communication unit 108 communicates with the haptics acquisition device 110 and the external output device 120 according to the control of the control unit 104. The communication unit 108 receives haptics data from the haptics acquisition device 110 and provides it to the control unit 104. The devices connected to the imaging device 100 can be connected by wire, for example, via a USB connection method. Also, the wired connection method is not limited to USB and can be any connection method. Also, not limited to the wired communication method, it may be connected by a wireless communication method (for example, IEEE802.11x, NFC, etc.).

[0018] The haptics acquisition device 110 can include a control unit 111, a communication unit 112, a haptics sensor 113, and a storage unit 114. These components are merely examples, and it may include components other than those shown in FIG. 1A.

[0019] The control unit 111 controls the driving timing of the haptics sensor 113 and performs overall driving and control of the entire device such as the communication unit 112 and the storage unit 114. For example, when a signal notifying the shutter timing of still image shooting in the imaging device 100 is received from the imaging device 100, haptics data can be acquired according to the notification signal. In the case of video shooting, a haptics acquisition command is periodically transmitted from the imaging device 100, and haptics data can be acquired in response to receiving the command. Since the acquired haptics data can include at least information on the acquisition time as attribute information, it is possible to specify at which timing of the moving image the haptics data was acquired based on the time information.

[0020] The haptics sensor 113 can be configured by, for example, a thermosensor or a tactile sensor, and can acquire an electrical signal representing haptics information such as the temperature or tactile information of an object. The electrical signal obtained from the haptics sensor 113 is converted into a digital signal and provided to the control unit 111 as haptics data. The acquired haptics data can include attribute information such as information regarding the acquisition time and position information. Also, in FIG. 1A, only one haptics sensor 113 is shown, but it may be configured to include a plurality of types of haptics sensors. Even in that case, the acquisition of haptics data in the haptics sensor 113 is performed in synchronization with the shooting of an image.

[0021] The communication unit 112 communicates with the imaging device 100 according to the control of the control unit 111 and transmits the haptics data acquired by the haptics sensor 113 to the imaging device 100. The communication unit 112 communicates, for example, via USB in the case of a wired connection method, and communicates, for example, via IEEE802.11x or NFC in the case of a wireless connection method.

[0022] The storage unit 114 can store the haptics data acquired by the haptics sensor 113. The stored haptics data may be read out from the control unit 111 and provided to the imaging device 100 via the communication unit 112. Further, the storage unit 114 may be configured to be able to mount an external storage medium (such as a memory card), and the haptics data stored in the external storage medium may be removed from the haptics acquisition device 110 and provided to the imaging device 100 by being mounted on the imaging device 100.

[0023] The external output device 120 may include a control unit 121, a display unit 122, a haptics output unit 123, and a communication unit 124. These components are merely examples, and the external output device 120 may include components other than those shown in FIG. 1A.

[0024] The control unit 111 controls the operation of the external output device 120. Further, the control unit 111 separates the image file received via the communication unit 124 into image data for display and haptics data for haptics output, supplies them to the display unit 122 and the haptics output unit 123, and controls their respective operations. When a touch input is performed via the display unit 122, the control unit 111 can detect the touch input and receive a request from the user. The request can be transmitted to the imaging device 100 side via the communication unit 124.

[0025] The display unit 122 is configured to be able to display image data and to be able to perform touch sensing, and can receive a touch input instruction from the user. The haptics output unit 123 can perform a tactile output on the display surface. A part of the functions of the haptics output unit 123 and the display unit 122 may be integrally configured as a display.

[0026] Further, the haptics output unit 123 may be configured to include a wearable device that can be worn by the user. The wearable device may be, for example, a glove that can be worn on the user's hand. The user can wear the glove and perform operation inputs on the display. The glove is connected to the external output device 120 by short-range wireless communication. Temperature data and tactile data are transmitted to the glove according to the operation input content, and the output according to the position where the user touches can be configured to be perceivable by the user via the glove.

[0027] The communication unit 124 communicates with the imaging device 100 under the control of the control unit 121. The imaging device 100 can receive an image file, and the user input information received by the display unit 122 can be transmitted to the imaging device 100. When using a wired connection method, the communication unit 124 communicates, for example, via USB. When using a wireless connection method, the communication unit 124 communicates, for example, via IEEE802.11x or NFC.

[0028] Next, with reference to FIG. 1B, the functional configuration of the image processing system 10 corresponding to Embodiment 1 will be described. In the image processing system 10, image data and haptics data can be associated to generate and record an image file. The image data includes still images and moving images, and the haptics data includes, for example, tactile data, temperature data, etc. In this embodiment, as an example, temperature data acquired using a thermosensor (temperature sensor) as haptics data will be described, and the case of generating and recording an image file by associating haptics data when recording a single still image will be described. Note that even in the case of a moving image, in the relationship between the video frame corresponding to the haptics data acquisition timing and the haptics data, the same relationship as that between the still image and the haptics data holds, and thus the processing can be performed in the same manner as the following description.

[0029] As shown in FIG. 1B, the image processing system 10 has, for example, the functional configurations of an image acquisition unit 151, a focus level information calculation unit 152, a haptics acquisition unit 153, a haptics change unit 154, a recording processing unit 155, and a storage medium 156.

[0030] The image acquisition unit 151 is mainly composed of a lens optical system 101, an image sensor 102, an image processing unit 103, etc., and converts the optical information from the lens optical system 101 into an electrical signal in the image sensor 102 and outputs it. The image acquisition unit 151 outputs the image data obtained by converting the electrical signal obtained from the sensor into a digital signal to the focus level information calculation unit 152 and the recording processing unit 155.

[0031] Also, when the image acquisition unit 151 acquires image data, it outputs the shooting information at that time to the focus level information calculation unit 152. The shooting information includes, for example, sensor diameter information, aperture value (F value) information, focal length (f) information, etc. The focus level information calculation unit 152 calculates a focus level indicating the degree of focus for each shooting target included in the image data based on the image data and the shooting information input from the image acquisition unit 151, and provides it to the haptics change unit 154. The focus level information and its calculation method will be described later. The focus level information calculation unit 152 is realized by the control unit 104.

[0032] Next, the haptics acquisition unit 153 is realized by the haptics acquisition device 110, and the haptics change unit 154 is realized by the control unit 104. The haptics acquisition unit 153 outputs the temperature data obtained from the haptics sensor to the haptics change unit 154.

[0033] Here, a configuration example of haptics data will be described with reference to FIG. 2. FIG. 2(A) shows an example of the image data 200 acquired by the image acquisition unit 151. The image data 200 to be processed in the present embodiment includes a plurality of shooting targets. The shooting targets include any of a person, an animal, a car, a flower, a tree, a background, etc., and these can be roughly classified into a subject and a background. FIG. 2(A) shows a case including a subject 201, a subject 202, a subject 203, and a background 204. FIG. 2(B) shows an example of the haptics data 210 acquired in association with the image data 200 shown in FIG. 2(A). The haptics data 210 has a correspondence relationship with the image data 200 in a predetermined unit (for example, pixel unit, or block or region unit). In the present embodiment, the haptics data 210 is two-dimensionally configured to match or correspond to the number of pixels in the horizontal and vertical directions of the image data 200, and it will be described that one value is associated with each pixel.

[0034] In FIG. 2(B), the values of the haptics data 210 are associated with the subject 201, the subject 202, the subject 203, and the background 204 included in the image data 200 of FIG. 2(A). The temperature of each shooting target corresponds to the color or pattern indicated by the reference numerals 204 to 206, respectively. Specifically, the temperature information of the subject 201 is recorded as a white area indicated by the reference numeral 204 at the coordinate position corresponding to the subject 201 in the image data 200 of FIG. 2(A). Similarly, the temperature information of the subject 202 and the subject 203 is indicated by the hatching of the reference numeral 205, and the temperature information of the background 204 is indicated by the black color of the reference numeral 206. Each temperature information is recorded at the same coordinate position as the subjects 202 and 203 and the background 204 in the image data 200.

[0035] In this embodiment, the haptics data 210 can be digital data expressed in a predetermined number of gradations in the same manner as the image data 200. An example of the data format corresponding to this embodiment is shown in FIG. 2(C). FIG. 2(C) shows an example in which the haptics data 210 is expressed in 8-bit 256 gradations for temperature. Temperatures are assigned for each haptics data value from 0 to 255. Thereby, for example, when any of the display areas is selected by the user while the image data 200 is being displayed on the display unit 106 of the imaging device 100 or the display unit 122 of the external output device 120, the temperature for feedback can be determined according to the value of the haptics data 210 in the selected area. In FIG. 2(C), an example is shown in which, for the haptics data 210, a value of 0 is assigned 0°C and a value of 255 is assigned 50°C. In FIG. 2(C), since the temperature of the subject 201 corresponds to gradation number 192, it is 37.5°C. Also, since the temperatures of the subjects 202 and 203 correspond to gradation number 128, they are 25°C. The temperature of the background 204 corresponds to gradation number 64, so it is 12.5°C.

[0036] In this embodiment, the haptics data 210 is expressed in 256 gradations, but it is not limited to this number of gradations and can be expressed in any number of gradations. Also, the configuration of the haptics data is not limited to the above, and it may conform to the specifications designated by the manufacturers of the imaging device 100 and the haptics acquisition device 110, or common specifications established by a standards organization such as MPEG may be used.

[0037] The haptics change unit 154 is realized by the control unit 104, and changes the haptics data input from the haptics acquisition unit 153 using the focus level information input from the focus level information calculation unit 152. Then, the changed haptics data is output to the recording processing unit 155. Details of the change of the haptics data will be described later.

[0038] The recording processing unit 155 is realized by the control unit 104, multiplexes the image data input from the image acquisition unit 151 and the haptics data input from the haptics change unit 154 into one file, generates an image file including the haptics data, and stores it in the storage medium 156. The storage medium 156 is realized by the storage unit 105 and holds the image file including the haptics data output by the recording processing unit 155.

[0039] Next, a configuration example of the focus level information will be described with reference to FIG. 3. FIG. 3(A) shows an example of the image data 300 acquired by the image acquisition unit 151, which includes the subject 301, the subject 302, the subject 303, and the background 304. FIG. 3(B) shows an example of the focus level information 310 in which the degree of focus of the image data 300 shown in FIG. 3(A) is color-coded. The focus level information 310 has a correspondence with the image in units of predetermined regions. In the present embodiment, the focus level information 310 is two-dimensionally configured to match the number of pixels in the horizontal and vertical directions of the image data 300, and it will be described that one focus level value is associated with each pixel.

[0040] In FIG. 3(B), the values of the focus levels of the subject 301, the subject 302, the subject 303, and the background 304 correspond to the colors or patterns indicated by the reference numerals 305 to 308, respectively. Specifically, the focus level information of the subject 301 is recorded as a white region indicated by the reference numeral 305 at the coordinate position corresponding to the subject 301 in the image data 300. Similarly, the focus level information of the subject 302 is indicated by the hatching of the reference numeral 306, the focus level information of the subject 303 is indicated by the hatching of the reference numeral 307, and the focus level information of the background 304 is indicated by the black color of the reference numeral 308, respectively. Each focus level information is recorded at the coordinate position corresponding to the subject 302, the subject 303, and the background 304 in the image data 300.

[0041] An example of the format of the focus level information in this embodiment is shown in FIG. 3(C). The focus level information is expressed as a numerical value between 0.0 and 1.0 as a gain value used for the conversion process of haptics data described later. The higher the focus level and the more in-focus the area is, the larger the gain value becomes. The lower the focus level and the less in-focus the area is, the smaller the gain value becomes.

[0042] In FIG. 3(B), an example of focus level information having four gradations indicated by reference numerals 305 to 308 is described, but the number of gradations is not limited to this, and it can be expressed with any number of gradations, and may be a number of gradations more than four gradations. On the other hand, for example, if it is only necessary to distinguish between the in-focus area and the out-of-focus area, it can also be expressed with two gradations as shown in FIG. 4. In the focus information 400 shown in FIG. 4(A), the white area represents the in-focus area and the black area represents the out-of-focus area. As shown in FIG. 4(B), a gain of 1.0 is assigned to the white in-focus area and a gain of 0.0 is assigned to the black out-of-focus area.

[0043] Next, an example of the processing in this embodiment will be described using the flowchart of FIG. 5. The processing corresponding to the flowchart can be realized, for example, by the CPU of the control unit 104 functioning as the focus level information calculation unit 152 executing a corresponding program (stored in a ROM or the like). In the following description, the case where the focus level is assigned to two values as shown in FIG. 4 will be described.

[0044] First, in S501, the focus level information calculation unit 152 acquires shooting information from the image acquisition unit 151. The shooting information includes, for example, the sensor diameter, the F value, and the focal length f. The focus level information calculation unit 152 also determines the allowable confusion circle based on the information of the sensor diameter. The allowable confusion circle may be determined as a fixed statistical value according to the sensor diameter, or may be determined by the cell pitch of the sensor, or the information of the allowable confusion circle included in the shooting information may be used.

[0045] In subsequent S502, the focus level information calculation unit 152 acquires the distance L to the imaging target obtained by using a distance measurement technique such as an autofocus function in the image acquisition unit 151. The imaging target for calculating the distance L may be a subject located in the in-focus area in the autofocus function, or may be selected by the user when a plurality of subjects are included. In subsequent S503, the focus level information calculation unit 152 calculates the depth of field based on the information obtained in S501 and S502. Specifically, the front depth of field d1 and the rear depth of field d2 are calculated according to the following formulas (1) and (2).

[0046] d1 = δFL^2 / (f^2 + δFL) ··· Formula (1) d2 = δFL^2 / (f^2 - δFL) ··· Formula (2) δ: Allowable confusion circle, F: F-number, L: Distance to the subject, f: Focal length In the processes after subsequent S504, the focus level information calculation unit 152 determines the focus level of each imaging target based on whether each imaging target included in the image data is included in the in-focus area set based on the subject of the imaging target located at the distance L (or whether it is included in either the in-focus area or the out-of-focus area).

[0047] Specifically, in S504, the positional relationship among the distance P from the imaging device 100 to an arbitrary area (in-focus determination target area) that is the target of in-focus determination, the distance L from the imaging device 100 to the subject, and the front depth of field d1 is determined. Here, the subject at the distance L is the subject 201 in the case of the image data 200 in FIG. 2(A). Also, the in-focus determination target area is set as an area of a predetermined size based on the coordinate position of the image data. The size of the area is arbitrary. The distance P can be measured for each in-focus determination target area by using the autofocus function.

[0048] In this process, based on the subject 201, the focus level in the focus determination target area is determined. If the distance P to the selected focus determination target area satisfies P < L - d1, the process proceeds to S505. On the other hand, if the distance P does not satisfy P < L - d1, the process proceeds to S506. In S505, the focus level information calculation unit 152 sets the focus determination target area to the second focus level. The focus determination target area is out of the focus area because it is too close to the imaging device 100 side. The second focus level is a focus level set as an out-of-focus area, and a value less than 1.0 is set as the gain value. In this embodiment, the gain value for the second focus level is set to 0.

[0049] Next, in S506, the focus level information calculation unit 152 further determines the positional relationship of the focus determination target area based on the distance P, the distance L to the subject, and the rear depth of field d2. If the distance P satisfies L + d2 < P, the process proceeds to S507. If L + d2 < P is not satisfied, the process proceeds to S508.

[0050] In S507, the focus level information calculation unit 152 sets the focus level of the selected focus determination target area to the second focus level. The focus determination target area is out of the focus area because it is too far from the imaging device 100. The process of S507 is the same as the process in S505. In S508, the focus level information calculation unit 152 sets the focus level of the selected focus determination target area to the first focus level. Since the first focus level is a focus level set as a focus area, a gain value of 1.0 is set.

[0051] In the subsequent S509, the focus level information calculation unit 152 determines whether the focus level has been set for all the focus determination target areas included in the imaging plane. If it is determined that the focus level setting for all the focus determination target areas is completed, this process ends. On the other hand, if there is an unprocessed focus determination target area, the process proceeds to S510. In S510, the focus level information calculation unit 152 selects the next focus determination target area and returns to S504 to repeat the above process.

[0052] In the flowchart of FIG. 5, the gain value is set to 0 when the second focusing level is set. However, the example of setting the gain value is not limited to this. For example, for a subject that belongs to the out-of-focus area on the side closer to the imaging device 100 (front side) than the subject at the distance L, the gain value may be set to 0.5 or the like instead of 0. Conversely, for a subject that belongs to the out-of-focus area on the side farther from the imaging device 100 (rear side) than the subject at the distance L, the gain value may be set to 0.5 or the like instead of 0. Thus, the relative positional relationship of the subjects can be notified by haptics data other than the visual information of the image.

[0053] Here, with reference to FIG. 6, an example of a method for setting the in-focus area based on the subject 201 will be described. FIG. 6 shows the positional relationship between the imaging device 100 and the subject 201 etc. when the image data 200 is captured. In FIG. 6, the imaging device 100 captures a shooting space in which the subjects 202, 201, and 203 are arranged in this order from the side closer to the imaging device 100. At this time, the distance between the imaging device 100 and the subject 201 is defined as L. In the present embodiment, the focus determination is performed for each area including the subjects 202 and 203. Among them, a range from L - d1 to L + d2 is recognized as the in-focus area with reference to the position of the subject 201, and the other areas are recognized as out-of-focus areas. In the case of the image data 200, since the subjects 202 and 203 are not located within the in-focus area, they are determined to be out-of-focus areas.

[0054] In the flowchart of FIG. 5, the case where the focus level is represented by two values was described as an example. The focus level may be represented by more values than two values. In that case, the determination conditions for the distance P of the focus determination target area may be further increased. For example, in FIGS. 5 and 6, there were only the front depth of field d1 and the rear depth of field d2, but the front depth of field d1 and the rear depth of field d2 may be subdivided. In that case, the determination conditions can be increased by bisecting or trisecting d1 and d2, respectively. When the determination conditions are increased in this way, a larger value can be assigned as the gain value to the focus area closer to the in-focus subject 201. For example, when d1 is divided into d11 and d12, and d2 is divided into d21 and d22, the gain value is set to 1.0 when belonging to the ranges of d11 and d21 closer to the subject at distance L, and a value smaller than 1.0 (for example, 0.5) is assigned as the gain value when belonging to the ranges of d12 and d22 farther from the subject at distance L.

[0055] Next, with reference to FIG. 7, an example of a method for changing haptics data performed by the haptics change unit 154 will be described. The change process of the haptics data in the present embodiment will hereinafter be referred to as the "first change process". FIG. 7 shows an example of multiplying the haptics data 701 and the focus level information 702. The haptics data 701 is the haptics data before the change. The haptics data 701 and the focus level information 702 have the same number of data in the horizontal and vertical directions, and by multiplying the data at the corresponding positions, the gain value of the focus level can be applied to each value of the haptics data 701. The focus level information 702 uses the example represented by the two values shown in FIG. 4. In the change process of the haptics data, the haptics data 701 and the focus level information 702 are multiplied in units of data or arbitrary data blocks.

[0056] In the first modification process of haptic data, since the gain value is large in the region with a high focusing level, the value (amplitude value) of the haptic data remains as it is. On the other hand, since the gain value is small in the region with a low focusing level, the value of the haptic data decays or becomes zero. Thereby, it is possible to suppress the information divergence between visual information and haptic information such as the presence of haptics in the out-of-focus region. The above-described process is repeatedly executed for the haptic data to be processed.

[0057] Since the gain value of the above-described focusing level information had a lower limit of 0.0 and an upper limit of 1.0, the haptic data after the change was either left as it was or the amplitude decayed. The upper and lower limits of the gain value are not limited to the numerical values shown in this embodiment and may be other values. For example, if the lower limit of the gain value is set to 1.0 and the upper limit is set to 2.0, such a haptic change is also possible in which the amplitude of the haptics is amplified more in the region with a higher focusing level.

[0058] In addition, regarding the setting of the gain value, it can be set to attenuate the value of the haptic data of the subject at the second focusing level, or to make the value of the haptic data of the subject at the first focusing level larger than the value of the haptic data of the subject at the second focusing level, or to make the degree of increase in the value of the haptic data of the subject at the first focusing level larger than the degree of increase in the haptic data of the subject at the second focusing level.

[0059] According to this embodiment, haptic data can be changed according to the focus level of image data. Specifically, a gain value corresponding to the focus level for each shooting target included in the image data is set, and by applying the gain value to the haptic data corresponding to the shooting target, individual values of the haptic data can be changed to values corresponding to the focus level. As a result, for shooting targets with a low focus level (out-of-focus subjects or backgrounds), the haptic data is attenuated or erased. Therefore, when the image data is reproduced, haptic data is not provided to shooting targets other than the in-focus subject, so the sense of discomfort given to the user during image reproduction can be reduced.

[0060] In the above-described embodiment, the gain value is set depending on whether the subject is located in the in-focus area. However, for example, the gain value may be set by referring to the type of the subject. For example, assume a case where the type of a subject such as a person, an animal (dog, cat, bird), a car, a motorcycle, a bicycle, a train, an airplane, etc. can be detected by an autofocus function. In this case, when any subject is detected, the type information of the subject is included in the imaging information as attribute information together with the coordinate information in the image data. Thereby, even if the subject is photographed outside the in-focus area, it is possible to adjust so that a value remains in the haptic data for a subject of a predetermined type. For example, in the above-described embodiment, the gain value is set to 0 when located in the out-of-focus area. However, when the subject is a subject of a predetermined type, by changing the gain value to 0.5 instead of 0, the gain value is lower than that of the subject located in the in-focus area, but a value can remain in the haptic data without being completely attenuated.

[0061] [Embodiment 2] In Embodiment 1, a method of changing haptics data according to the focus level of image data was described. However, when there is an obstacle in the foreground or in front of the subject, the haptics data desired by the user may not be obtained. For example, even in an image area where the main subject is in focus as visual information obtained from the image data, there may be a case where the haptics data of the main subject does not exist, or the haptics data of another subject exists. In such a case, a divergence occurs between the visual information obtained from the image data and the output haptics information, which gives the user a sense of discomfort. First, cases assumed as problems in this embodiment will be described below with reference to FIG. 8. In this embodiment, the same content as in Embodiment 1 will be omitted from the description.

[0062] FIG. 8(A) shows image data 800 of a processing target corresponding to this embodiment. The image data 800 includes a main subject 801, a subject 802, and a background 803 as shooting targets. In the image data 800, the contour of the subject 801 indicated by the solid line indicates that the subject 801 is in focus. On the other hand, the contour of the other subject 802 is indicated by a dotted line, and in this case, it represents that the subject 802 is out of focus.

[0063] Regarding the arrangement relationship of each subject included in this image data 800 with respect to the imaging device 100, it is as shown in FIG. 8(B). In this shooting scene, the subject 802 is arranged as an obstacle in front of the main subject 801. However, since the subject 802 is intentionally blurred using front blur for shooting, it is assumed that the subject 802 in the image data 800 is in a state where it cannot be visually recognized by the user.

[0064] Regarding this shooting scene, when haptics data is acquired in the same manner as in Embodiment 1 and the value is changed according to the focus level of the subject, the haptics data of the portion of the subject 802 with a low focus level is attenuated or erased. An example of the haptics data assumed to be acquired at this time is shown in FIG. 8(C).

[0065] In FIG. 8(C), the haptics data 810 is composed of a portion shown as a white 804 region associated with the main subject 801 and a region shown in black 805 whose value is attenuated or erased by the first change process described in Embodiment 1. Since the haptics region associated with the subject 802 is an out-of-focus region, it is masked to zero by the first change process. As a result, the haptics data corresponding to the main subject 801 shown in white 804 has missing data in the region where the main subject 801 and the subject 802 overlap.

[0066] Thus, in shooting using front blur, it is not possible to visually recognize that there is an obstacle in front of the main subject from the image data obtained by shooting. Therefore, when the haptics data is erased, an information divergence occurs between the image data and the haptics data. Therefore, in the present embodiment, a method for recovering haptics data when a part of the haptics data of the main subject is missing due to the foreground or an obstacle will be described.

[0067] An example of the functional configuration of the image processing system 900 is shown in FIG. 9. FIG. 9 shows an example of a functional configuration realized based on the configuration of the image processing system 10 shown in FIG. 1A. In the functional configuration of FIG. 9, an edge detection unit 901 is added to the image processing system 10 shown in FIG. 1B. In the following description, the content already described in relation to FIG. 1B will be omitted, and the processing specific to the present embodiment will be mainly described.

[0068] The image acquisition unit 151 outputs the image data to the edge detection unit 901 in addition to the focus level information calculation unit 152 and the recording processing unit 155. The haptics data is also input to the edge detection unit 901 from the haptics acquisition unit 153.

[0069] The edge detection unit 901 performs edge detection processing on the image data input from the image acquisition unit 151, and extracts edge information with abrupt changes in the data regarded as edges. Similarly, edge detection processing is also performed on the haptic data input from the haptics acquisition unit 153, and edge information with abrupt changes in the data regarded as edges is extracted. The edge detection unit 901 outputs the image edge information of the acquired image data and the haptic edge information of the haptic data to the haptics modification unit 905. The edge detection processing in the edge detection unit 901 can be implemented by using known techniques such as differential filters in the horizontal and vertical directions or Sobel filters. However, the edge detection method is not limited to the above method, and may be realized by other methods.

[0070] Here, an example of visualizing the acquired edge information is shown in FIG. 10. Edge information 1001 indicates the edge detection result (image edge information) for the image data. Edge information 1002 indicates the edge detection result (haptic edge information) for the modified haptic data. Edge difference information 1003 is the difference between edge information 1001 and edge information 1002, and indicates a missing area where data is missing in the haptic data. As can be seen by comparing edge information 1001 and 1002, when there is a defect in the haptic data, the edge detection results of the image data and the haptic data do not match. That is, there is an information divergence between the visual information and the haptic information. By performing edge detection in this way, it becomes possible to determine whether there is a defect in the haptic data.

[0071] Returning to the description of FIG. 9, the haptics modification unit 154 performs a first modification process using the focus level information input from the focus level information calculation unit 152 and the haptic data input from the haptics acquisition unit 153. Then, for the haptic data after the first modification process, a second modification process is performed using the edge information input from the edge detection unit 901. The haptics modification unit 154 outputs the modified haptics to the recording process unit 155.

[0072] Next, the modification process of haptics data in this embodiment will be described. FIG. 11 is a flowchart showing an example of the process corresponding to this embodiment. The process corresponding to this flowchart can be realized, for example, by the CPU of the control unit 104 functioning as the haptics modification unit 154, the edge detection unit 901, etc. executing a program (stored in a ROM or the like).

[0073] First, in S1101, a modification process of haptics data (first modification process) is performed. This process is the same as the process described with reference to FIGS. 5 to 7 and the like in Embodiment 1. The first modification process in S1101 may be performed in parallel with the processes after S1102, or may be performed prior to the process of S1102.

[0074] In S1102, the edge detection unit 901 acquires the image data to be processed from the image acquisition unit 151 and detects the edge information of the image data. Also, the corresponding haptics data is acquired from the haptics acquisition unit 153, and the edge information of the haptics data is detected. In the subsequent S1103, the haptics modification unit 154 acquires the image edge information and the haptics edge information from the edge detection unit 901 and compares the two edge informations. Then, in S1104, it is determined whether they match. If the degree of coincidence of the two edge informations is less than a predetermined threshold, it is considered that they do not match, and the process proceeds to S1105. On the other hand, if the degree of coincidence is equal to or greater than the predetermined threshold, it is considered that they match and this process ends. If there is unprocessed haptics data, the processes from S1101 to S1105 are repeatedly executed.

[0075] In the comparison process in S1103, as described with reference to FIG. 10, the difference between the edge informations is taken, and it can be determined whether they match by determining the amount of information remaining as the difference information. Specifically, if the amount of the difference information is less than the threshold, it can be determined that they match, and if the amount of the difference information is equal to or greater than the threshold, it can be determined that they do not match.

[0076] In S1105, the haptics change unit 154 identifies and interpolates (second change process) an area where haptics data is missing from the haptics data for which the first change process was performed in S1101, based on the difference information obtained in S1103. As a result, it is possible to supplement the haptics data for a portion of the main subject 801 visually recognized in the image data that is blocked by a subject 802 not visually recognized in the image data. In the second change process, the missing portion of the haptics data based on the difference information can be realized by replacing or interpolating with the maximum value, minimum value, average value, median, etc. of the neighboring data of the haptics data. However, the method for interpolating the putics data is not limited to the method described above, and may be realized by other methods. Even if a deviation in the contour portion of the main subject 801 remains as a difference, that portion can be regarded as subject information, so there is no problem in leaving it in the haptics data.

[0077] According to this embodiment, it is possible to determine whether there are any deficiencies in the haptics data due to foreground or front obstacles using feature quantities such as edge information, and perform interpolation processing according to the determination result. Therefore, it is possible to solve problems such as the presence of a foreground or front obstacle not shown in the image data remaining in the haptics data and giving the user a sense of discomfort. As a result, in this embodiment as well, it is possible to change the haptics data while suppressing the information divergence from the visual information.

[0078] [Embodiment 3] In Embodiments 1 and 2, a method of changing and recording haptics data using the calculated focus level information was described. In contrast, it is also possible to hold the focus level information as attribute information or meta information without changing the haptics data during recording, and read out the focus level information during playback to change the haptics data. In this embodiment, a method of changing the haptics data during playback using the previously held focus level information will be described. Note that descriptions of the same content as in Embodiments 1 and 2 will be omitted.

[0079] The configuration of the image processing system according to Embodiment 3 is shown in FIG. 12. FIG. 12 shows a functional configuration realized based on the configuration of the image processing system 10 shown in FIG. 1A. In the functional configuration of FIG. 12, the image processing system 1200 includes a storage medium 1201, a separation unit 1202, an edge detection unit 901, a haptics change unit 154, a combination unit 1205, and a playback device 1206. For the haptics change unit 154 and the edge detection unit 901, functions corresponding to this embodiment are further added to the configurations described in Embodiments 1 and 2.

[0080] The storage medium 1201 holds an image file generated by associating image data and haptics data. The storage medium 1201 may be the storage unit 105, or may be an external storage medium (such as a memory card) attached to the storage unit 105. The image file is also provided with focus level information. In response to a playback instruction from the user, the storage medium 1201 outputs the image file with haptics to the separation unit 1202. The separation unit 1202 is realized by the control unit 104, and separates the image file with haptics input from the storage medium 1201 into image data, haptics data, and focus level information, respectively. The separated haptics data is output to the edge detection unit 901 and the haptics change unit 154. Also, the separated focus level information is output to the haptics change unit 154. Further, the separated image data is output to the edge detection unit 901 and the combination unit 1205.

[0081] The edge detection unit 901 is realized by the control unit 104, and detects edge information of the image data and haptics data input from the separation unit 1202. The detected edge information is output to the haptics change unit 154. The haptics change unit 154 performs change processes (a first change process and a second change process) for changing the haptics data using the focus level information input from the separation unit 1202 and the edge information input from the edge detection unit 901. The content and method of changing the haptics data are the same as those described in Embodiments 1 and 2. Then, the changed haptics data is output to the combination unit 1205.

[0082] The combining unit 1205 is realized by the control unit 104, combines the image data input from the separating unit 1202 and the haptics data input from the haptics changing unit 154, and generates an image file with haptics again. The playback device 1206 is realized as an external output device 120, displays the image data on the display, touches the display, and can output the feedback of the haptics data in the touched area of the display with a haptics device. In this way, while playing back the image data of the image file with haptics provided from the combining unit 1205, it is possible to perform the feedback of the haptics data according to the content of the display operation from the user.

[0083] As described above, by attaching and pre-recording the focus level information to the image file with haptics data, it is possible to perform haptics changes that reduce the discomfort given to the user even during playback.

[0084] [Embodiment 4] In Embodiments 1 to 3, a method of changing haptics data using the focus level information has been described. However, it is also assumed that the focus level information cannot be obtained in order to change the haptics data. In such a case, depth information such as a depth map can be used.

[0085] Recently, the utilization of depth maps, which are depth information of images, in editing software has been progressing. A depth map is metadata that represents depth as black and white shading information. For example, it is expressed as darker towards the front and lighter towards the back. When the depth map is utilized for editing, it becomes possible to apply blurring to an arbitrary depth. Devices capable of recording such depth maps are increasing.

[0086] In a depth map, the subject in the shooting space is represented by grayscale information, and the density represented by the grayscale information represents the position of the subject. Although the information represented by the depth map has a different meaning from the information represented by the focus level information, they are common in that they are information reflecting the position of the subject in the shooting space. Therefore, it is possible to set the focus level for each subject, such as selecting a subject with a high degree of focus and a subject with a low degree of focus using depth information such as a depth map to increase the degree of focus.

[0087] Therefore, in this embodiment, when the focus level information is not recorded, a method of converting information having subject unit information such as a depth map into focus level information and changing the haptics data using this focus level information will be described. Specifically, a focused area and an unfocused area are set using the depth map. This can be done by specifying an area for blurring during editing using the depth map. The depth blur function in general editing software blurs a subject or area having the depth information specified by the user. That is, the haptics information is changed for the area where the depth blur is performed by the user operation.

[0088] A configuration example of the image processing system according to Embodiment 4 is shown in FIG. 13. FIG. 13 shows an example of a functional configuration realized based on the configuration of the image processing system 10 shown in FIG. 1A. In the functional configuration of FIG. 13, the image processing system 1300 includes a storage medium 1201, a separation unit 1202, a depth blur area specifying unit 1301, an image changing unit 1302, an edge detection unit 901, a haptics changing unit 154, and a combining unit 1205. The functional blocks described in Embodiments 1 to 3 are given common reference numerals, and functions corresponding to this embodiment are further added to the configurations described in Embodiments 1 to 3.

[0089] The storage medium 1201 holds an image file with haptics data. In this embodiment, unlike Embodiment 3, the image file further includes a depth map. The storage medium 1201 outputs the haptic image file to the separation unit 1202 in response to a playback instruction or an editing instruction received from the user via, for example, the operation unit 107 of the imaging device 100. The description of the depth map will be given later.

[0090] Hereinafter, with reference to the flowchart of FIG. 14, the processing flow in the image processing system 1300 will be described. The processing corresponding to the flowchart can be realized, for example, by the CPU of the control unit 104 functioning as the separation unit 1202, the depth blur region designating unit 1301, the image changing unit 1302, the edge detection unit 901, the haptics changing unit 154, and the combining unit 1205 executing a program (stored in a ROM or the like).

[0091] First, in S1401, the separation unit 1202 separates the haptic image file input from the storage medium 1201 into image data, haptics data, and a depth map. The separated depth map is output to the depth blur region designating unit 1301, the image data is output to the image changing unit 1302, and the haptics data is output to the edge detection unit 901 and the haptics changing unit 154, respectively.

[0092] In subsequent S1402, the depth blur region designating unit 1301 realized by the control unit 104 receives a designation of a region to which blur is to be applied from the user, and outputs information (designation information) of the designated region, such as coordinate information or mask data, to the image changing unit 1302. In subsequent S1403, the depth map input from the separation unit 1202 is converted into in-focus level information. When converting the depth map into in-focus level information, the depth blur region designating unit 1301 generates in-focus level information by setting a region not given blur as the in-focus region and a region given blur as the out-of-focus region based on the designation information in S1402. The generated in-focus level information is output to the haptics changing unit 154. A method for converting into in-focus level information will be described later. Also, in order to reconstruct an image file with haptics, the depth map is output to the combining unit 1205.

[0093] In subsequent S1404, the image changing unit 1302 realized by the control unit 104 applies predetermined image processing to the image input from the separation unit 1202 to give blur based on the designation information input from the depth blur region designating unit 1301. Then, the image data subjected to the image processing is output to the edge detection unit 901 and the combining unit 1205. In S1405, the edge detection unit 901 detects edge information from the changed image data input from the image changing unit 1302 and the haptics data. The detected edge information is output to the haptics changing unit 154.

[0094] The haptics changing unit 154 changes the haptics data input from the separation unit 1202 using the in-focus level information input from the depth blur region designating unit 1301 (first change process). A method for changing the haptics data will be described later. Then, the changed haptics data and the depth map are output to the combining unit 1205. Also, here, using the difference in the edge information detected from the image data and the haptics data, determination of the presence or absence of loss of the haptics data and interpolation processing (second change process) are performed. Since the second change process of the haptics data using the edge information has been described in Embodiment 2, the description in this embodiment is omitted.

[0095] In S1406, the combining unit 1205 combines the depth map input from the depth blurring region specifying unit 1301, the image data input from the image modifying unit 1302, and the haptics data input from the haptics modifying unit 154 to form an image file with haptics again. Then, the image file with haptics is written to the storage medium 1201.

[0096] A configuration example of the depth map according to this embodiment will be described with reference to FIG. 15. In FIG. 15, it is assumed that the acquired image data is the same as that in FIG. 3(A), and the acquired haptics data is the same as that in FIG. 2(B). An example of the acquired depth map is shown in FIG. 15(A). In the depth map 1500, shading is performed according to the depth level corresponding to the distance from the subject in the image data. The depth level associated with the subject 301 is 1502, the depth level associated with the subject 302 is 1501, the depth level associated with the subject 303 is 1503, and the depth level associated with the background 304 is 1504. The user can select the area to be blurred for each of these shaded areas.

[0097] Here, it is assumed that the user designates an area other than the depth level 1502 to be blurred. The focus level information generated by the depth blurring region specifying unit 1301 is shown in FIG. 15(B). Also, the data format is shown in FIG. 15(C). FIG. 15(C) represents the focus level in 1-bit binary. The area with a focus level of 0 is the area to be blurred, and the area with a focus level of 1 is the area not to be blurred. An example of assigning the binary focus level information as a gain value used in the conversion process of the haptics data described later with numerical values from 0.0 to 1.0 is shown. In this embodiment, an example of expressing the focus level in binary is described, but it is not limited to this number of tones, and any number of tones may be used.

[0098] Next, a method for changing haptics data performed by the haptics change unit 154 will be described. This change method can be performed in the same manner as the method described with reference to FIG. 7. In FIG. 7, haptics data 701 before change, focus level information 702, and haptics data 703 after change are shown respectively. In the present embodiment, the focus level information 702 can be the focus level information obtained from the depth map as described above. Also in the present embodiment, haptics data 703 can be obtained by multiplying haptics data 701 and focus level information 702. Here, the haptics data in the blurring area where the gain becomes 0.0 is changed to 0. Thereby, it is possible to suppress the information divergence between the visual information and the haptics information such as the presence of haptics in the blurred area.

[0099] In the present embodiment, the image is changed so as to add blurring to the specified area. However, the embodiment is not necessarily limited to this form. It is possible to receive the designation of the area using the depth map and perform only the process of changing the haptics data accordingly, without adding blurring to the image itself.

[0100] According to the present embodiment, even if the focus level information is not retained, it is possible to change the haptics data by converting general-purpose meta information such as a depth map into the focus level information. In the present embodiment, a method for changing haptics data has been described using the depth map as an example. However, other meta information capable of setting the focus level in the same manner as the depth map may be used, and the present invention is not limited thereto.

[0101] [Technical ideas derived from Embodiments 1 to 4] The disclosure of this specification includes the following image processing apparatuses, imaging apparatuses, image processing systems, control methods for image processing apparatuses, and programs according to the technical ideas of the above-described Embodiments 1 to 4. (1) An image processing apparatus, An acquisition means for acquiring information on a focus level indicating the degree of focus of a subject included in the image data to be processed, based on shooting information obtained when acquiring the image data to be processed by shooting; A haptics change means for making a first change to the haptics data acquired in association with the image data to be processed, based on the information on the focus level; A generation means for generating an image file in association with the image data to be processed, using the haptics data after the first change has been made; comprising; The first change includes changing the value of the haptics data of at least some of the subjects, an image processing apparatus. (2) The image processing apparatus according to (1), wherein the first change includes a change made so as to attenuate the haptics data for subjects other than a predetermined subject among the subjects. (3) The focus level includes at least a first focus level and a second focus level with a lower degree of focus than the first focus level, The first change is a change made so as to attenuate the value of the haptics data of the subject at the second focus level; a change made so that the value of the haptics data of the subject at the first focus level becomes larger than the value of the haptics data of the subject at the second focus level; a change made so that the degree of increase in the value of the haptics data of the subject at the first focus level is larger than the degree of increase in the haptics data of the subject at the second focus level The image processing apparatus according to (1) or (2), characterized by including at least any one of them. (4) The shooting information includes information indicating the type of the subject, When the imaging object at the second focusing level has a predetermined type, the haptics changing means adjusts the value of the haptics data after the first change for the imaging object so as to increase compared to the value of the haptics data after the first change when the imaging object does not have the predetermined type. The image processing apparatus according to (3). (5) The image processing apparatus further includes edge detection means for extracting edge information from the image data to be processed and the haptics data. The haptics changing means makes a second change to the haptics data on which the first change has been made based on the difference information between the first edge information extracted from the image data to be processed and the second edge information extracted from the haptics data. The second change includes changing so as to interpolate a missing area of a value in the haptics data corresponding to the difference information. The image processing apparatus according to any one of (1) to (4). (6) The missing area corresponds to a portion of the imaging object visually recognized in the image data to be processed that is shielded by another imaging object not visually recognized. The image processing apparatus according to (5). (7) The acquisition means includes calculation means for calculating information on the focusing level based on the imaging information. The image processing apparatus according to any one of (1) to (6). (8) The calculation means calculates the depth of field based on the imaging information, and calculates the focusing level such that, among the imaging objects included in the image data, the imaging objects included in the in-focus area are set as the first focusing level and the imaging objects not included in the in-focus area are set as the second focusing level. The in-focus area is set based on the distance to a predetermined imaging object and the depth of field. The image processing apparatus according to (7). (9) The depth of field includes a front depth of field and a rear depth of field. The image processing apparatus according to (8). (10) The imaging information includes the sensor size, the F-number, and the focal length f. The image processing apparatus according to any one of (7) to (9). (11) The acquisition means acquires the information on the focus level from a storage medium in which the image data to be processed and the information on the focus level are stored in association with each other, and the image processing apparatus according to any one of (1) to (10). (12) The acquisition means reads out the depth information from a storage medium in which the image data to be processed and the depth information of the object to be photographed are stored in association with each other, and generates the information on the focus level from the depth information, and the image processing apparatus according to any one of (1) to (11). (13) The acquisition means generates the information on the focus level from the depth information based on the information on a designated area set for the depth information, and the image processing apparatus according to (12). (14) The acquisition means generates the information on the focus level such that an area not given blurring has a higher degree of focus than an area given blurring based on the information on the designated area, and the image processing apparatus according to (13). (15) The image processing apparatus according to (13) or (14), further comprising an image changing unit that changes the image data to be processed by applying blurring to the designated area. (16) The haptics data includes at least one of temperature data and tactile data, and the image processing apparatus according to any one of (1) to (15). (17) An imaging apparatus including the image processing apparatus according to any one of (1) to (16). (18) The imaging apparatus according to (17), An external output device including display means for displaying the image data included in the image file and output means for outputting the haptics data and the image processing system including the same. (19) The image processing system according to (18), further comprising a haptics data acquisition device for acquiring the haptics data. (20) A control method for an image processing apparatus, An acquisition step of acquiring information on a focus level indicating the degree of focus of a subject included in the image data to be processed, based on shooting information obtained when acquiring the image data to be processed by shooting; A haptics change step of performing a first change on the haptics data acquired in association with the image data to be processed, based on the information on the focus level; A generation step of generating an image file in association with the image data to be processed, using the haptics data after the first change has been made; comprising: The first change includes changing the values of the haptics data for at least some of the subjects, a control method for an image processing apparatus. (21) A program for causing a computer to function as the image processing apparatus according to any one of (1) to (16).

[0102] Further, the present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0103] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Explanation of Signs

[0104] 10 Image processing system, 100 Imaging device, 110 Haptics data acquisition device, 120 External output device

Claims

1. An image processing apparatus, an acquisition means for acquiring information on a focus level indicating a degree of focus of a subject included in the image data of the processing target, the information being obtained based on shooting information at the time of acquiring the image data of the processing target by shooting; a haptics changing means for making a first change to haptics data acquired in association with the image data of the processing target based on the information on the focus level; a generation means for generating an image file in association with the image data of the processing target using the haptics data after the first change has been made; comprising: wherein the first change includes changing values of the haptics data of at least some of the subjects, an image processing apparatus.

2. The image processing apparatus according to claim 1, wherein the first change includes a change made so as to attenuate the haptics data for subjects other than a predetermined subject among the subjects.

3. The focus level includes at least a first focus level and a second focus level with a lower degree of focus than the first focus level, the first change is a change made so as to attenuate the value of the haptics data of the subject at the second focus level, a change made so that the value of the haptics data of the subject at the first focus level becomes larger than the value of the haptics data of the subject at the second focus level, a change made so that the degree of increase in the value of the haptics data of the subject at the first focus level is larger than the degree of increase in the haptics data of the subject at the second focus level The image processing apparatus according to claim 1, characterized by including at least any one of.

4. The shooting information includes information indicating the type of the subject, When the subject at the second focus level has a predetermined type, the haptics changing means makes an adjustment so that the value of the haptics data after the first change for the subject is increased compared to the value of the haptics data after the first change when the subject does not have the predetermined type. The image processing apparatus according to claim 3.

5. further comprising edge detection means for extracting edge information from the image data of the processing target and the haptics data, The haptics changing means makes a second change to the haptics data in which the first change has been made, based on difference information between first edge information extracted from the image data to be processed and second edge information extracted from the haptics data. The image processing apparatus according to claim 1, wherein the second change includes changing so as to interpolate a missing value area in the haptics data corresponding to the difference information.

6. The image processing apparatus according to claim 5, wherein the missing area corresponds to a portion of the image data to be processed in which a photographed object visually recognized is shielded by another photographed object that is not visually recognized.

7. The image processing apparatus according to claim 1, wherein the acquisition means includes a calculation means for calculating information on the in-focus level based on the photographing information.

8. The calculation means calculates a depth of field based on the photographing information, and calculates the in-focus level such that, among the photographed objects included in the image data, the photographed objects included in the in-focus area are set as a first in-focus level, and the photographed objects not included in the in-focus area are set as a second in-focus level. The image processing apparatus according to claim 7, wherein the in-focus area is set based on a distance to a predetermined photographed object and the depth of field.

9. The image processing apparatus according to claim 8, wherein the depth of field includes a front depth of field and a rear depth of field.

10. The image processing apparatus according to claim 7, wherein the photographing information includes a sensor size, an F value, and a focal length f.

11. The image processing apparatus according to claim 1, wherein the acquisition means acquires the information on the in-focus level from a storage medium in which the image data to be processed and the information on the in-focus level are stored in association with each other.

12. The image processing apparatus according to claim 1, wherein the acquisition means reads out the depth information from a storage medium in which the image data to be processed and the depth information of the photographed object are stored in association with each other, and generates the information on the in-focus level from the depth information.

13. The image processing apparatus according to claim 12, wherein the acquisition means generates the information on the in-focus level from the depth information based on information on a designated area set for the depth information.

14. The image processing apparatus according to claim 13, wherein the acquisition means generates information on the focus level so that an area without blurring has a higher degree of focus than an area with blurring, based on the information on the specified area.

15. The image processing apparatus according to claim 13, further comprising an image modification unit that modifies the image data to be processed by applying blurring to the specified area.

16. The image processing apparatus according to claim 1, wherein the haptic data includes at least one of temperature data and tactile data.

17. An imaging apparatus including the image processing apparatus according to any one of claims 1 to 16.

18. The imaging apparatus according to claim 17, and an external output device including display means for displaying the image data included in the image file and output means for outputting the haptic data. An image processing system comprising:

19. The image processing system according to claim 18, further comprising a haptic data acquisition device that acquires the haptic data.

20. A control method for an image processing apparatus, comprising: an acquisition step of acquiring information on a focus level indicating a degree of focus of a shooting target included in the image data to be processed, the information being obtained based on shooting information when the image data to be processed is acquired by shooting; a haptic modification step of performing a first modification on the haptic data acquired in association with the image data to be processed, based on the information on the focus level; a generation step of generating an image file in association with the image data to be processed, using the haptic data after the first modification is performed; including The control method for an image processing apparatus, wherein the first modification includes changing values of the haptic data of at least some of the shooting targets.

21. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 16.

Images