Image capturing apparatus, image capturing system, control method, and storage medium

Abstract

To provide an imaging apparatus or the like capable of reproducing tactile information so that a user can effectively obtain presence even when there is tactile information in each of a plurality of subjects, and SOLUTION: An information processing apparatus includes an acquisition unit configured to acquire tactile information of each of a plurality of subjects, a linking unit configured to link the tactile information acquired by the acquisition unit to each of the plurality of subjects, a determination unit configured to determine a main subject among the plurality of subjects, and a reproduction unit configured to reproduce a captured image of each of the subjects and to reproduce the captured image of the main subject. An information processing apparatus comprising: a reproduction unit configured to reproduce tactile information obtained by combining tactile information of each of a plurality of objects recognized with reference to a result of association by an association unit, or selectively reproduce tactile information of a main object determined by a determination unit; and an output unit configured to output the tactile information reproduced by the reproduction unit to another apparatus driven based on input tactile information, wherein SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an imaging device, an imaging system, a control method, and a program that have a function for reproducing tactile information such as vibration information of a plurality of subjects. [Background technology]

[0002] By capturing tactile information in addition to images of subjects using an imaging device and reproducing the tactile information along with the images, users can have an immersive and realistic experience. Here, "tactile information" refers to, for example, information such as vibration, acceleration, surface friction coefficient, pressure, area and distribution of pressed points, and temperature. Vibration information is an example of tactile information. For example, in a sports event, multiple subjects are present, and vibration information for the multiple subjects is acquired, allowing the user to reproduce and view the vibration information along with the images. In this case, the challenge is how to select and reproduce vibration information from the vibration information for each of the multiple subjects so that the user can effectively experience a sense of realism. Patent Documents 1 to 3 disclose devices and the like that solve this problem. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6056853 [Patent Document 2] Patent No. 6771548 [Patent Document 3] Patent No. 6664137 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology disclosed in Patent Document 1 simply reproduces predetermined vibration information stored before capturing an image in correspondence with the behavior of the subject in the captured image. In other words, because this predetermined vibration information does not measure the vibration of the subject, it is difficult for the user to experience a sense of realism even when the vibration information is reproduced. Furthermore, the conventional technology disclosed in Patent Document 2 acquires distance information of surrounding objects using two cameras to obtain more information, and notifies the user of the acquired distance information by corresponding it to vibration information. However, because this vibration information is simply generated in correspondence with the distance information, this system also has the problem of not providing a sense of realism to the user.

[0005] Furthermore, the imaging device disclosed in Patent Document 3, which is a conventional technology, acquires multiple pieces of tactile information experienced by a user and reproduces the tactile information when reproducing an image, but does not mention how to select the multiple pieces of tactile information.

[0006] The present invention has been made in consideration of these conventional problems, and its purpose is to provide an imaging device, imaging system, control method, and program that can reproduce tactile information from the tactile information for each of a plurality of subjects so that the user can effectively obtain a sense of realism. [Means for solving the problem]

[0007] In order to achieve the above object, one aspect of the present invention is an imaging device having a reading unit capable of reading captured images from a recording unit that records captured images of multiple subjects captured by an imaging unit, and is characterized by comprising: an acquisition unit that acquires tactile information for each of the multiple subjects; a linking unit that links each of the multiple subjects with the tactile information acquired by the acquisition unit; a determination unit that determines a main subject from the multiple subjects; a playback unit that plays back the captured images of each subject and either plays back tactile information that is the sum of the tactile information for each of the multiple subjects grasped by referring to the linking results by the linking unit, or selects and plays back the tactile information of the main subject determined by the determination unit; and an output unit that outputs the tactile information played back by the playback unit to another device that operates based on the input tactile information. [Effects of the Invention]

[0008] According to the present invention, it is possible to obtain an effect of being able to reproduce tactile information from among the tactile information for each of a plurality of subjects in such a way that the user can effectively obtain a sense of realism. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram of a system according to an embodiment of the present invention and an explanatory diagram of the data structure of an image file. [Figure 2] 1 is an explanatory diagram illustrating an outline of the configuration and operation of a system according to an embodiment of the present invention. [Figure 3] 4 is a flowchart showing a process according to the first embodiment. [Figure 4] 4 is an explanatory diagram showing the reproduction characteristics (frequency characteristics) of the vibration reproduction device. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. Furthermore, the "tactile sensation" described below refers to the sensation of touching an object, and the information obtained is "tactile information." Vibration information indicating vibration is an example of tactile information.

[0011] First Embodiment (FIG. 1(a): Configuration diagram of the imaging device 100) 1(a) is a block diagram of an imaging device 100 according to an embodiment of the present invention. The imaging device 100 includes a control unit 101, which is controllably connected to a power supply unit 102, an operation unit 103, an imaging unit 104, a distance measurement unit 105, and a memory unit 106. The control unit 101 is also controllably connected to an object recognition unit 107, a display unit 108, a receiving unit 109, a transmitting unit 110, an external recording unit 111, and a position acquisition unit 114. The imaging device 100 also includes a lens unit 112, which can communicate required information with the control unit 101 via a lens connection unit 113.

[0012] (Control unit 101: Power supply unit 102) The control unit 101 controls the overall operation of the imaging device 100 and has various functions. The control unit is realized by, for example, one or more processors (CPU, DSP, etc.). The power supply unit 102 is composed of a power detection circuit, a protection circuit, a DC-DC converter, an LDO regulator, and secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries. The power supply unit 102 supplies a predetermined voltage to various electronic devices used in the imaging device 100. The power supply unit 102 also has functions to detect whether a battery is installed, the battery type, and the remaining power level, and to protect load circuits connected to the power supply circuit by cutting off the power supply when an overcurrent is detected. The power supply unit 102 also has a power detection circuit that can detect the amount of power supplied to each unit in the imaging device 100.

[0013] (Operation unit 103) The operation unit 103 is an operating means for giving various operational instructions to the control unit 101. The operation unit 103 is configured, for example, by any one of a switch, a dial, a touch panel, a voice recognition device, etc., or a combination of these. A release button, which is one example of the operation unit 103, is configured by a two-stage switch consisting of a first release switch (not shown) and a second release switch (not shown). A "first operation instruction" is given to the control unit 101 by pressing the first release switch in the first stage, and a "second operation instruction" is given to the control unit 101 by pressing the second release switch in the second stage. For example, the first operation instruction is an instruction to perform an image capture preparation operation, and the control unit 101 controls the distance measurement unit 105 (described later) to perform an image capture operation of the subject as the image capture preparation operation. Furthermore, the second operation instruction is an instruction to perform an image capture operation, and the control unit 101 controls the image capture unit 104 (described later) to perform an image capture operation of the subject.

[0014] (Image capture unit 104) The imaging unit 104 has an imaging device such as a CMOS, and captures an image of a subject formed via the lens unit 112. In an imaging preparation stage, the distance measuring unit 105, which will be described later, calculates the distance to the subject using image information captured by the imaging unit 104. The control unit 101 obtains luminance information of the subject using the image information captured by the imaging unit 104, and calculates exposure information and sensitivity information of the imaging device that are optimal for imaging. Furthermore, the control unit 101 performs a focusing operation by driving the lens unit 112 to focus on the subject based on the distance information of the subject calculated by the distance measuring unit 105. In an imaging stage, the imaging unit 104 controls the exposure and sensitivity of the imaging device using the calculated exposure information and sensitivity information, and captures an image of the subject.

[0015] (Distance measurement unit 105: Memory unit 106) The distance measurement unit 105 calculates the distance between the subject and the camera. The distance measurement unit 105 calculates the distance between the subject and the camera from image information obtained from the imaging device of the imaging unit 104. When calculating the distance, the subject recognition unit 107 (described later) recognizes people who are subjects in the captured image, and can calculate the distance by prioritizing them based on this. The memory unit 106 has a dynamic memory and a non-volatile memory. The dynamic memory stores variables indicating the device status, captured image data, various calculation data, etc. The non-volatile memory can store and retain information even without power supply. The non-volatile memory stores programs executed by the control unit 101, operating constants, etc. These programs are programs for executing various operations (described later).

[0016] (Subject recognition unit 107) The object recognition unit 107 detects the presence of an object, such as a person or a person's face, in an image acquired by the imaging unit 104, and calculates the position, size, etc. of the object in the captured image. The captured object as the object may be a vehicle, an animal, etc., in addition to a person. The object recognition unit 107 also calculates the characteristics of the detected object. Examples of the object characteristics include multiple items such as the type, size, shape, brightness distribution, and color histogram distribution of the detected object. By calculating and understanding the characteristics of the object, it is possible to continue to recognize the same object as the object in the previous frame when performing object recognition in the next frame of the captured image, even if the position, posture, etc. of the object changes. This reduces the possibility of losing sight of the object.

[0017] (Display unit 108: Receiving unit 109: Transmitting unit 110: External recording unit 111) The display unit 108 displays the subject image, the captured image, and the status of the device, and can be realized by, for example, a liquid crystal display. The receiving unit 109 receives signals, information, etc. from a device other than the imaging device 100. The transmitting unit 110 transmits signals, information, etc. to other devices. The external recording unit 111 is realized by, for example, a removable recording medium (recording means) such as a memory card, and records captured image data, its accompanying information, vibration information, etc. in a non-volatile manner.

[0018] (Position acquisition unit 114) The position acquisition unit 114 acquires position information by, for example, a geographical position information acquisition method using radio waves from known satellites, and can be realized by, for example, a GPS device. Other methods include sequentially accumulating results obtained using an acceleration sensor to acquire the amount of movement, and multiple methods may be used in combination to acquire position information. The lens unit 112 is replaceable and is realized by an imaging lens (not shown), a lens driver for focusing control, an aperture driver for driving the aperture, and the like. The lens connection unit 113 is composed of a mechanism for attaching and detaching the lens unit 112, and a communication terminal for controlling focusing and aperture drive. In other words, the lens connection unit 113 has the function of communicating required information with the control unit 101.

[0019] (FIG. 1(b): Configuration diagram of vibration information acquisition device 200) 1(b) is a block diagram of a vibration information acquisition device 200 (acquisition means) that acquires vibration information, which is an example of tactile information according to an embodiment of the present invention. "Tactile information" includes, but is not limited to, vibration, surface friction coefficient, pressure, area and distribution of pressed points, temperature, etc. In other words, tactile information is a variety of information that can be sensed by humans through touch.

[0020] The vibration information acquisition device 200 has a control unit 201 to which a power supply unit 202, a vibration detection unit 203, a position acquisition unit 204, a memory unit 205, a transmission unit 206 and a reception unit 207 are controllably connected.

[0021] (Control unit 201: Power supply unit 202: Vibration detection unit 203) The control unit 201 controls the overall operation of the vibration information acquisition device 200 and is realized by, for example, one or more processors (CPU, DSP, etc.). The power supply unit 202 is composed of a power detection circuit, a protection circuit, a DC-DC converter, an LDO regulator, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, etc. The power supply unit 202 supplies a predetermined voltage to various electronic devices used in the vibration information acquisition device 200. The vibration detection unit 203 detects vibrations transmitted to the vibration information acquisition device 200 and can be realized by, for example, an acceleration sensor equipped with a piezoresistance element.

[0022] (Position acquisition unit 204: Memory unit 205) The position acquisition unit 204 has a configuration basically similar to that of the position acquisition unit 114. As described above, the position acquisition unit 204 acquires position information by a geographical position information acquisition method using radio waves from known satellites, and can be realized by, for example, a GPS device. Other methods include sequentially accumulating results obtained using an acceleration sensor to acquire the amount of movement, and position information may be acquired using a combination of multiple methods. The memory unit 205 has dynamic memory and non-volatile memory. The dynamic memory temporarily stores constants, variables, measured tactile information, etc. for operation. The non-volatile memory stores constants, programs, etc. for operation of the vibration information acquisition device 200.

[0023] (Transmitter 206: Receiver 207) The transmitting unit 206 transmits signals, information, etc. to the image capturing device 100. On the other hand, the receiving unit 207 receives signals, information, etc. from the image capturing device 100.

[0024] (FIG. 1(c): Configuration diagram of the playback output device 300) 1(c) is a block diagram of a video playback output device 300 (playback means) according to an embodiment of the present invention. The playback output device 300 may be built into the imaging device 100 or may be a separate device. The playback output device 300 has a control unit 301, to which a power supply unit 302, an operation unit 303, an external record reading unit 304, a display unit 305, a communication unit 307, and a vibration signal output unit 308 are controllably connected.

[0025] (Control unit 301) The control unit 301 controls the overall operation of the playback output device 300 and is realized by, for example, one or more processors (CPU, DSP, etc.). The control unit 301 reads recorded image information from an external recording reading unit 304 (described later) and plays back the image.

[0026] (Power supply unit 302: Operation unit 303: External record reading unit 304) The power supply unit 302 is realized by a protection circuit, a DC-DC converter, an LDO regulator, etc. The power supply unit 302 supplies predetermined voltages to various electronic devices used in the playback output device 300. The operation unit 303 is an operation means for giving various operation instructions to the control unit 301. The external record reading unit 304 can be equipped with a removable recording medium (external recording unit 111) and can read information recorded on the recording medium. The external record reading unit 304 reads, for example, captured image data and its accompanying information.

[0027] (Display unit 305: Communication unit 307: Vibration signal output unit 308) The display unit 305 reads image data and the like read from the external record reading unit 304 and displays the reproduced image. The display unit 305 is a large display device, for example, 40 to 60 inches or more, to effectively give the user a sense of realism. The communication unit 307 communicates required information with the vibration reproduction device 400. The vibration signal output unit 308 (output means) outputs the reproduced tactile signal to the vibration reproduction device 400.

[0028] (Fig. 1(d): Configuration diagram of vibration reproduction device 400 (device)) 1(d) is a block diagram of a vibration reproduction device 400 that reproduces vibration information, which is an example of tactile information according to an embodiment of the present invention. Examples of tactile information include, but are not limited to, vibration information, surface friction coefficient, pressure, area and distribution of pressed points, and temperature. The vibration reproduction device 400 has a control unit 401, to which a power supply unit 402, a communication unit 403, a vibration signal input unit 404, a memory unit 405, and a drive unit 407 are controllably connected. Furthermore, a vibration unit 408 is drivably connected to the drive unit 407.

[0029] (Control unit 401: Power supply unit 402) The control unit 401 controls the overall operation of the vibration reproduction device 400 and is realized by, for example, one or more processors (CPU, DSP, etc.). The power supply unit 402 is realized by a protection circuit, a DC-DC converter, an LDO regulator, etc. The power supply unit 402 supplies a predetermined voltage to various electronic devices used in the vibration reproduction device 400.

[0030] (Communication unit 403: vibration signal input unit 404: memory unit 405: vibration reproduction device information 406) The communication unit 403 communicates required information with the playback output device 300. The vibration signal input unit 404 receives tactile signals transmitted from the playback output device 300. The memory unit 405 has a dynamic memory and a non-volatile memory. The dynamic memory temporarily stores operational constants, variables, measured tactile information, etc. The non-volatile memory stores constants, programs, etc. required for the vibration playback device 400 to operate in a non-volatile manner. The non-volatile memory particularly holds tactile information. The memory unit 405 has an area for storing vibration playback device information 406. The vibration playback device information 406 is information related to the vibration playback device 400. The vibration unit 408 (described below) that generates vibrations has different output frequency characteristics depending on the vibration generation principle, and information related to this is stored as the vibration playback device information 406. Furthermore, if there are multiple vibration playback devices 400, different vibration playback device information 406 is stored for each device.

[0031] (Driver 407: Vibrator 408) The driving unit 407 drives the vibrating unit 408. The vibration signal input by the vibration signal input unit 404 is converted into a voltage or the like for driving the vibrating unit 408, which will be described later. The vibrating unit 408 generates a tactile effect. The tactile effect is, for example, vibration, and there are various methods for generating vibration, such as rotating an eccentric mass with a motor, driving an ultrasonic tactile element, using acoustic radiation pressure, and spraying compressed air, and there is no particular limitation as long as it generates vibration. Furthermore, each of these methods has different frequency characteristics, for example, and characteristic information when outputting the tactile effect is stored in the memory unit 405 as vibration reproduction device information 406.

[0032] (Figure 1(e): Structure of image file 500) FIG. 1(e) is a diagram illustrating the structure of an image file 500 (linking means) containing tactile information stored in the external recording unit 111 or the like of the imaging device 100 according to an embodiment of the present invention. The image file 500 includes a header section 501, subject information 502, image data 503, and vibration data (vibration information) 504. The header section 501 is the header of the imaging data and includes information such as the date and time of imaging, the aperture, the lens used, the imaging sensitivity, and the imaging exposure time. The subject information 502 is subject information captured by the imaging device 100 and includes information used to determine main subject information. The "main subject information" is information indicated by the main subject, such as the distance from the camera to the subject, the size of the face in the image, the brightness information of the facial area, the facial feature information of the face where the AF frame was located when the photographer locked AF, the color distribution information of the person image on which the photographer locked AF, and the facial features of a person designated in advance by the photographer.

[0033] The image data 503 is data of a captured image. The vibration data 504 is, for example, vibration information acquired by the vibration information acquisition device 200. The vibration data 504 (vibration information) acquired by the vibration information acquisition device 200 is recorded in the detachable external recording unit 111. When playing back using the playback output device 300, the control unit 301 causes the external record reading unit 304 to read the vibration data 504 (vibration information) recorded in the attached external recording unit 111 and passes it to the vibration signal output unit 308. The vibration signal output unit 308 then outputs the played back tactile signal to the vibration signal input unit 404 of the vibration playback device 400, and the vibration signal input unit 404 inputs the tactile signal transmitted from the playback output device 300.

[0034] Here, the association of the subject, its image data 503, and the vibration information (vibration data 504) of the vibration information acquisition device 200 can be realized, for example, as follows. The vibration information acquisition device 200 carried by each subject stores a device-specific ID, and when transmitting vibration information (vibration data 504) to the image capture device 100, the device-specific ID is added to the vibration information and transmitted. At the same time, the user associates the subject, its image data 503, and the device-specific ID in advance using a UI or the like displayed on the display unit 108. This allows the control unit 101 to refer to the ID of the transmitted vibration information and to ascertain which vibration information acquisition device 200 transmitted the vibration information, and which subject and its image data 503 correspond to the vibration information acquisition device 200 that transmitted the vibration information. Alternatively, instead of the device-specific ID, the vibration information acquisition device 200 may transmit vibration information of a device-specific signal waveform to the image capture device 100. For example, suppose there are two vibration information acquisition devices 200A and 200B, and the vibration information acquisition device 200A transmits sine wave vibration information, while the vibration information acquisition device 200B transmits square wave vibration information. In this case, the control unit 101 can distinguish between the two based on the signal waveform of the vibration information. The user can then associate the corresponding subject and its image data 503 with each sine wave and square wave in advance. There are also various other signal waveforms, such as triangular waves and square waves. Although the linking modes have been described above, the linking modes are not limited to these.

[0035] (Figure 2: System configuration overview) 2 is an explanatory diagram outlining the configuration and operation of a system according to an embodiment of the present invention. Each of a plurality of subjects imaged by imaging device 100 carries a respective vibration information acquisition device 200. Subject 11 carries vibration information acquisition device 200a, and tactile information acquired by vibration information acquisition device 200a is transmitted to imaging device 100. Similarly, subject 12 carries vibration information acquisition device 200b, and tactile information acquired by vibration information acquisition device 200b is transmitted to imaging device 100. Similarly, subject 13 carries vibration information acquisition device 200c, and tactile information acquired by vibration information acquisition device 200c is transmitted to imaging device 100. Reference numeral 14 denotes a viewer seated in front of display unit 305.

[0036] The playback output device 300 reads image data 503 using the external recording reading unit 304 and displays the played-back image on the display unit 305. In order to effectively give the user a sense of realism, the display unit 305 is a large display device, for example, of about 40 to 60 inches or more. The playback output device 300 may be built into the imaging device 100 or may be a separate device.

[0037] The playback output device 300 is connected to a plurality of vibration playback devices 400a to 400c, and provides images and vibrations (tactile sensations) to the viewer 14. In order to effectively provide a sense of realism to the user, the plurality of vibration playback devices 400a to 400c are configured with different vibration modes, sizes, vibration effects, etc. In the example shown in FIG. 2, 400a and 400b are vibration playback devices 400 that use acoustic radiation pressure. Furthermore, 400c is a chair-type vibration playback device 400 that uses seat tactile sensations, and 400d is a floor-type vibration playback device 400 that uses floor tactile sensations.

[0038] When attempting to acquire tactile information from multiple subjects using the imaging device 100, the tactile information from each of the multiple subjects may differ. For example, if the subjects are far apart, the tactile information cannot be considered to be the same, and it is preferable to acquire tactile information from each subject.

[0039] An identification number is assigned to the athlete, who is the subject of the image, and for example, the athlete's uniform number can be used as the identification number. Vibration information is an example of tactile information, and the athlete carries a tactile measuring device, such as a vibration information acquisition device 200, linked to the athlete's uniform number. The vibration information acquisition device 200 transmits vibration information linked to the identification number (for example, uniform number) that identifies the subject to the image capture device 100. In response, the image capture device 100 receives the vibration information linked to the uniform number. The uniform number (identification number) of the subject image is read, the subject image is linked to the uniform number (identification number), and the vibration information linked to the uniform number (identification number) is linked to the subject and saved in the image file 500 (see FIG. 1(e)). There may be multiple pairs of vibration information and subjects linked together.

[0040] Tactile information, such as vibration information, has a long cycle and takes time for the viewer to recognize it when played back, so it is desirable to store and play back vibration information for a period longer than the exposure period of the image capture. For example, it is desirable to store tactile information for 10 seconds before the start of image capture exposure and 10 seconds after the end of image capture. The object recognition unit 107 calculates the characteristics of the object detected within the image capture field of view. The object characteristics can include multiple items such as the type, size, shape, brightness distribution, and color histogram distribution of the detected object.

[0041] Furthermore, when there are multiple subjects within the imaging angle of view, imaging device 100 can recognize the main subject, for example, by receiving instructions from the photographer. When the photographer presses the first release switch of the release button, which is one of the operation units 103, imaging device 100 performs imaging preparation operations. Imaging device 100 recognizes the subject that overlaps with the focus point selected by the photographer in the imaging preparation operations as the main subject.

[0042] The imaging device 100 saves, in an image file 500, information on the focusing point specified by the photographer, information on each subject, such as the position, size, and main subject information of a person's face in a captured image. The image file 500 is recorded on a removable recording medium. The playback output device 300 reads the image file 500. If the image file contains tactile information associated with multiple subjects, the playback output device 300 also reads the information on each subject and the main subject information saved in the image file.

[0043] Then, when reproducing an image and a tactile sensation, the tactile information associated with the main subject is selected and reproduced in the reproduction output device 300. Furthermore, if multiple vibration reproduction devices 400 are connected to the reproduction output device 300, tactile signals are output according to the characteristics of each vibration reproduction device 400.

[0044] (FIG. 3: Flowchart showing the processing of the first embodiment) (Step S101: Step S102: Step S103) FIG. 3 is a flowchart showing the processing of the first embodiment. The processing shown in FIG. 3 is basically executed by the control unit 101. The flowchart in FIG. 3 explains processing that uses tactile information, which is a superordinate concept of vibration information. The processing shown in FIG. 3 is repeatedly executed at approximately the same cycle until it is terminated in S115. Tactile information such as vibration information is transmitted from the transmission unit 206 of the vibration information acquisition device 200 to the reception unit 109 of the imaging device 100 and recorded in the external recording unit 111 or the like. First, in step S101, the control unit 101 starts this sequence. Next, in step S102, the control unit 101 determines whether or not the tactile information has been received. If it is determined that the information has been received (Yes), the process proceeds to step S103. On the other hand, if it is determined that the information has not been received (No), the process returns to step S102 and enters a wait state. In step S103, the control unit 101 starts storing and retaining the tactile information in the external recording unit 111, and then proceeds to step S114.

[0045] (Step S114) In step S114, the control unit 101 determines whether or not an instruction to acquire tactile information has been detected by the operation unit 103. If it is determined that an instruction to acquire tactile information has been detected (Yes), the process proceeds to step S104. On the other hand, if it is determined that an instruction to acquire tactile information has not been detected (No), the process proceeds to step S115, where the sequence ends.

[0046] (Step S104: Step S105: Step S106) In step S104, the control unit 101 determines whether or not an image capture preparation operation has been detected by pressing the first release switch, which is one of the operation units 103 in FIG. 1(a). If it is determined that it has been detected (Yes), the process proceeds to step S105. On the other hand, if it is not determined that it has been detected (No), the process returns to step S114. Next, in step S105, the control unit 101 (storage means) determines whether or not the retention period of the retained tactile information has exceeded a predetermined time. The predetermined time is, for example, 10 seconds. If it is determined that it has exceeded the predetermined time (Yes), the process proceeds to step S106.

[0047] On the other hand, if it is not determined that the time has passed (No), the process proceeds to step S107. In S106, the control unit 101 (discarding means) discards tactile information from before the predetermined time. This prevents an excessive increase in memory capacity due to the storage of tactile information, and memory capacity is sufficient to store, for example, 10 seconds of tactile information for each subject.

[0048] (Step S107: Step S108) In step S107, the control unit 101 determines whether or not a person's face is present in the selected focus point frame (AF frame). If it is determined that there is a person's face (Yes), the process proceeds to step S108. On the other hand, if it is not determined that there is a person's face (No), the process proceeds to step S109. In step S108, the control unit 101 determines that the person's face in the focus point frame is the "main subject" (determination means), and the process proceeds to step S109.

[0049] (Step S109) In step S109, the control unit 101 determines whether or not an image capture instruction has been detected by pressing the second release switch, which is one of the operation units 103 in Fig. 1(a). If it is determined that an image capture instruction has been detected (Yes), the process proceeds to step S110. On the other hand, if it is not determined that an image capture instruction has been detected (No), the process proceeds to step S105.

[0050] (Step S110: Step S111) In step S110, the control unit 101 controls the image capturing unit 104 to capture an image of the subject, and the process proceeds to step S111. In step S111, the control unit 101 stores the captured image in the external recording unit 111, and the process proceeds to step S112.

[0051] (Step S112: Step S113) Next, in step S112, the control unit 1101 determines whether or not the uniform number of the subject can be read. If it is determined that it is possible (Yes), the process proceeds to step S113. On the other hand, if it is not determined that it is possible (No), the process proceeds to step S114. Then, in step S113, the control unit 101 associates the tactile information associated with the subject's uniform number with the subject, saves the image, and proceeds to step S114. Specifically, the control unit 191 associates tactile information (tactile data: vibration data 504) for a predetermined time before and after image capture with the subject of the image for all subjects carrying the vibration information acquisition device 200.

[0052] In the first embodiment, a uniform number is used as an example of an identification number, but the identification number is not limited to a uniform number as long as it is information that can identify the subject. In this way, the identification number, tactile information, and image are linked to all subjects carrying the vibration information acquisition device 200. As a result, it becomes possible to know which vibration information is from which subject. For example, it becomes possible to effectively obtain a sense of realism by performing weighting or other means to add up the playback.

[0053] Second Embodiment The summation of tactile signals in a second embodiment of the present invention will be described below. In the imaging device 100 shown in FIG. 1(a), the object recognition unit 107 calculates the characteristics of the object detected within the imaging field of view. The object characteristics include a number of items such as the type, size, shape, luminance distribution, and color histogram distribution of the detected object. In this embodiment of the present invention, the characteristic information for each object includes the position and size of the person's face in the captured image, main object information, etc., and these are stored in the image file 500.

[0054] 1(c) reads an image file using the external record reading unit 304. If the image file contains tactile information associated with multiple subjects, the playback output device 300 also reads the information about each subject, i.e., the feature information, stored in the image file. When the playback output device 300 plays back the "image" and "tactile sensation," it adds up and plays back the tactile information associated with each subject.

[0055] At this time, the subject information stored in the image file can be used to calculate weighting coefficients based on characteristic information such as the size of a person's face, which is a typical subject of imaging, and the weighting and summing can be performed. For example, if the weights based on the face sizes of three types of subjects "A," "B," and "C" are Fa, Fb, and Fc, and the vibration information is VA, VB, and VC, then the weighting and summing of "Fa x VA + Fb x VB + Fc x Vc" is performed. Furthermore, the weighting mode for each subject is not limited to being performed according to the characteristic information. For example, a configuration in which the user performs weighting operations for each subject using the operation unit 303 during playback using a UI or the like may be used.

[0056] As described above, according to the second embodiment of the present invention, when there are multiple pieces of tactile information, the tactile information can be reproduced so that the user can effectively obtain a sense of realism. In this way, when reproducing multiple pieces of stored vibration information, in addition to selecting and reproducing vibration information for the main subject determined by the subject recognition unit 107, it is also possible to reproduce vibration information that has been weighted according to the feature information of the subject and added together.

[0057] Third Embodiment Hereinafter, in the third embodiment of the present invention, weighting of vibration information (tactile information) in a vibration reproduction device 400 that reproduces vibration will be described. In tactile reproduction, the principles of vibration generation, which is an example of tactile sensation, and the mass of the vibration generator vary, so the time response speed, frequency characteristics, etc. of the vibration intensity that can be reproduced vary. As shown in FIG. 1(d), vibration reproduction device information 406 holds the reproducible frequency characteristics of the vibration reproduction device 400, and this is transmitted from the communication unit 403 included in the vibration reproduction device 400 to the communication unit 307 of the reproduction output device 300. In response to this, the communication unit 307 receives the vibration reproduction device information 406. In this way, before the reproduction output device 300 transmits the vibration information, it is possible to weight the vibration information according to the vibration reproduction device information 406.

[0058] (FIG. 4: An explanatory diagram showing the frequency characteristics of the vibration reproduction device 400) Fig. 4 is a schematic explanatory diagram showing the vibration reproduction characteristics (frequency characteristics) of the vibration reproduction device 400. Fig. 4 shows the characteristics of the "kth" vibration reproduction device when there are "n (plural)" vibration reproduction devices 400. 601 indicates the peak intensity "P k " and 602 is the lower limit frequency of the vibration reproduction band "f ek 603 is the upper limit frequency of the vibration reproduction band "f sk " Equation (101) represents the sum of the product of all frequency widths and peak intensities of "n" vibration reproduction devices 400. For example, when correcting differences in vibration intensity generated by the vibration reproduction devices 400, equation (101) is taken into consideration.

[0059] Furthermore, equation (102) uses the coefficient "G" which represents the "kth" weighting. k ". Equation (103) is the signal output to the "kth" vibration reproduction device 400. When the reproduction output device 300 transmits a tactile signal (vibration signal), it can perform weighting according to the vibration reproduction device information 406.

[0060]

number

[0061] As described above, according to the third embodiment, when performing combined playback, playback can be performed by multiplying by a weight (coefficient) based on the frequency characteristics of the vibration playback device 400. As a result, when there are multiple pieces of tactile information (vibration information), it is possible to play back tactile information that allows the user to effectively obtain a sense of realism during playback. Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist thereof.

[0062] <Other> (1) The vibration information to be reproduced may be selected by the user from among stored vibration information. (2) In the present invention, in addition to selecting and reproducing vibration information associated with a main subject detected by the imaging device 100 from among multiple subjects, the vibration information associated with each of the multiple subjects may be weighted and reproduced in combination, and this may be selectable by the user. (3) In FIG. 2, if a person's face is located within the selected focus point frame (AF frame), the subject having the person's face is determined as the main subject. However, the main subject may be determined from each of multiple subjects based on "main subject information." (4) Furthermore, the vibration information acquisition device 200 is not particularly limited as long as it is an information terminal such as a smartphone or smartwatch equipped with an acceleration sensor, and may be realized by having the acceleration sensor itself carried by the subject.

[0063] <Additional Note> The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An imaging device having a reading unit capable of reading captured images from a recording unit that records captured images of a plurality of subjects captured by an imaging unit, an acquisition means for acquiring tactile information for each of the plurality of subjects; a linking means for linking each of the plurality of subjects with the tactile information acquired by the acquisition means; a determining means for determining a main subject from among the plurality of subjects; a playback means for playing back the captured images of each subject, and playing back tactile information that is the sum of the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking means, or for selecting and playing back tactile information of the main subject determined by the determination means; and output means for outputting the tactile information reproduced by the reproduction means to another device that is driven based on the input tactile information. (Configuration 2) The reproducing means is The imaging device according to configuration 1 is characterized in that it plays back the captured images of each subject, and also plays back tactile information obtained by weighting and summing the tactile information of each of the multiple subjects that is grasped by referring to the linking results by the linking means. (Configuration 3) The reproducing means further comprises: When the output means outputs the tactile information that has been added together and reproduced by the reproduction means to a plurality of reproduction devices that are the other devices, the output means generates tactile information for each reproduction device by multiplying each reproduction device by a coefficient based on its frequency characteristics, The output means further 3. The imaging device according to configuration 1 or 2, wherein tactile information for each corresponding playback device is output to each playback device. (Configuration 4) The determining means is 3. The imaging device according to claim 1, wherein a main subject is determined from each of the plurality of subjects based on main subject information that is information indicated by the main subject. (Configuration 5) The determining means 3. The imaging device according to configuration 1 or 2, wherein if it is determined that a person's face is within the selected frame, the person is determined to be the main subject. (Configuration 6) The imaging device according to configuration 2, wherein the weighting is determined based on feature information of the subject. (Configuration 7) The imaging device according to claim 1 or 2, characterized in that the tactile information is vibration information indicating vibration of the subject. (Configuration 8) A storage means for storing the tactile information acquired by the acquisition means; 3. The imaging device according to claim 1, further comprising a discarding means for discarding tactile information that has been stored by the holding means for a predetermined period of time. (Configuration 9) An imaging system having an imaging device having a reading unit capable of reading captured images from a recording means that records captured images of a plurality of subjects captured by an imaging unit, a plurality of tactile information acquisition devices that acquire tactile information for each of the plurality of subjects, and a device that operates based on the input tactile information, The imaging device is An imaging system comprising: a linking means for linking each of the plurality of subjects with tactile information acquired by the plurality of tactile information acquisition devices; a determination means for determining a main subject from the plurality of subjects; a playback means for playing back an image of each subject and either playing back tactile information that is the sum of the tactile information of each of the plurality of subjects grasped by referring to the linking results by the linking means, or selecting and playing back the tactile information of the main subject determined by the determination means; and an output means for outputting the tactile information played back by the playback means to the device. (Method) A control method for an imaging device having a reading unit capable of reading captured images from a recording means that records captured images of a plurality of subjects captured by an imaging unit, comprising: an acquisition step of acquiring tactile information for each of the plurality of subjects; a linking step of linking each of the plurality of subjects with the tactile information acquired in the acquisition step; a determining step of determining a main subject from among the plurality of subjects; a reproduction step of reproducing the captured images of each subject and reproducing tactile information obtained by adding together the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking step, or selecting and reproducing the tactile information of the main subject determined in the determination step; an output step of outputting the tactile information reproduced in the reproduction step to another device that is driven based on the input tactile information. (Program) A program that causes a computer to execute a control method for an imaging device having a reading unit that can read captured images from a recording unit that records captured images of a plurality of subjects captured by an imaging unit, The control method includes: an acquisition step of acquiring tactile information for each of the plurality of subjects; a linking step of linking each of the plurality of subjects with the tactile information acquired in the acquisition step; a determining step of determining a main subject from among the plurality of subjects; a reproduction step of reproducing the captured images of each subject and reproducing tactile information obtained by adding together the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking step, or selecting and reproducing the tactile information of the main subject determined in the determination step; and an output step of outputting the tactile information reproduced in the reproduction step to another device that is driven based on the input tactile information.

[0064] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and various modifications and variations are possible within the scope of the present invention. For example, the present invention can be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or recording medium, and having a computer processor in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. [Explanation of symbols]

[0065] 100 Imaging device 101 Control section 104 Imaging unit 106 Memory section 107 Subject recognition section 101 Control section 111 External Recording Unit 200 Vibration information acquisition device 201 Control Unit 203 Vibration detection unit 300 Reproduction output device 301 Control Unit 304 External Record Reader 308 Vibration signal output unit 400 vibration regenerator 404 Vibration signal input section 408 Vibration part

Claims

1. An imaging device having a reading unit capable of reading captured images from a recording unit that records captured images of a plurality of subjects captured by an imaging unit, an acquisition means for acquiring tactile information for each of the plurality of subjects; a linking means for linking each of the plurality of subjects with the tactile information acquired by the acquisition means; a determining means for determining a main subject from among the plurality of subjects; a playback means for playing back the captured images of each subject, and playing back tactile information that is the sum of the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking means, or for selecting and playing back tactile information of the main subject determined by the determination means; and output means for outputting the tactile information reproduced by the reproduction means to another device that is driven based on the input tactile information.

2. The regeneration means 2. The imaging device according to claim 1, wherein the imaging device plays back the captured images of each subject, and plays back the sum of tactile information obtained by weighting the tactile information of each of the plurality of subjects grasped by referring to the linking results obtained by the linking means.

3. The regenerating means further comprises: When the output means outputs the tactile information that has been added together and reproduced by the reproduction means to a plurality of reproduction devices that are the other devices, the output means generates tactile information for each reproduction device by multiplying each reproduction device by a coefficient based on its frequency characteristics, The output means further 3. The imaging device according to claim 1, wherein tactile information is output to each of the playback devices.

4. The determining means 3. The imaging apparatus according to claim 1, wherein a main subject is determined from each of the plurality of subjects based on main subject information that is information indicated by the main subject.

5. The determining means 3. The imaging apparatus according to claim 1, wherein if it is determined that a human face is present within the selected frame, the human face is determined to be the main subject.

6. The imaging apparatus according to claim 2 , wherein the weighting is determined based on feature information of the subject.

7. 3. The imaging device according to claim 1, wherein the tactile information is vibration information indicating vibration of a subject.

8. a storage means for storing the tactile information acquired by the acquisition means; 3. The imaging device according to claim 1, further comprising a discarding means for discarding tactile information that has been stored in said holding means for a predetermined period of time.

9. An imaging system having an imaging device having a reading unit capable of reading captured images from a recording means that records captured images of a plurality of subjects captured by an imaging unit, a plurality of tactile information acquisition devices that acquire tactile information for each of the plurality of subjects, and a device that operates based on the input tactile information, The imaging device is a linking means for linking each of the plurality of subjects with the tactile information acquired by the plurality of tactile information acquisition devices; a determining means for determining a main subject from among the plurality of subjects; a playback means for playing back the captured images of each subject, and playing back tactile information that is the sum of the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking means, or for selecting and playing back tactile information of the main subject determined by the determination means; and an output unit that outputs the tactile information reproduced by the reproduction unit to the device.

10. A control method for an imaging device having a reading unit capable of reading captured images from a recording unit that records captured images of a plurality of subjects captured by an imaging unit, comprising: an acquisition step of acquiring tactile information for each of the plurality of subjects; a linking step of linking each of the plurality of subjects with the tactile information acquired in the acquisition step; a determining step of determining a main subject from among the plurality of subjects; a reproduction step of reproducing the captured images of each subject and reproducing tactile information obtained by adding together the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking step, or selecting and reproducing the tactile information of the main subject determined in the determination step; an output step of outputting the tactile information reproduced in the reproduction step to another device that is driven based on the input tactile information.

11. 1. A program for causing a computer to execute a control method for an imaging device having a reading unit capable of reading captured images from a recording unit that records captured images of a plurality of subjects captured by an imaging unit, The control method includes: an acquisition step of acquiring tactile information for each of the plurality of subjects; a linking step of linking each of the plurality of subjects with the tactile information acquired in the acquisition step; a determining step of determining a main subject from among the plurality of subjects; a reproduction step of reproducing the captured images of each subject and reproducing tactile information obtained by adding together the tactile information of each of the plurality of subjects grasped by referring to the linking results of the linking step, or selecting and reproducing the tactile information of the main subject determined in the determination step; and an output step of outputting the tactile information reproduced in the reproduction step to another device that is driven based on the input tactile information.

Images