Trigger signal generator and mobile terminal

Abstract

A trigger signal generation device according to an aspect of the present invention generates a trigger signal indicating a timing at which a gesture has been performed on a mobile terminal. The trigger signal generation device is provided with: a motion information detection unit that detects motion information for each of three axial directions of a coordinate system the origin of which is the mobile terminal; and a trigger signal generation unit that generates the trigger signal at a timing at which the positive / negative sign of one of the three items of motion information detected by the motion information detection unit becomes different from the other two positive / negative signs.

Description

[Technical field]

[0001] The present invention relates to a trigger signal generating device and a mobile terminal. [Background technology]

[0002] A technique has been disclosed in the past for operating an imaging device by moving a smartphone in a predetermined pattern (see Patent Document 1). Specifically, the technique of Patent Document 1 operates an imaging device mounted on the smartphone when it detects a series of movements in which the user swings the smartphone up counterclockwise, down clockwise, and then stops the smartphone.

[0003] Also, a technology has been disclosed that plays, stops, fast-forwards, rewinds, etc., a video by moving a smartphone in a predetermined pattern (see Patent Document 2). Specifically, the technology of Patent Document 2 detects an operation of a user tilting a smartphone around the x-axis or y-axis, and plays, stops, etc. a video based on the detected operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 61420609 [Patent Document 2] WO 2018 / 66705 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention solves the problems of the prior art. [Means for solving the problem]

[0006] The first aspect of the present invention is an image capturing unit that captures an image of a subject and generates frame images; an image storage unit that sequentially stores the frame images generated by the image capturing unit; a motion information detection unit that detects motion information indicating acceleration in each of three axial directions of a coordinate system; a stationary period detection unit that detects a stationary period in which the mobile terminal is in a stationary state based on the three pieces of motion information detected by the motion information detection unit; a trigger signal generation unit that generates a trigger signal indicating the timing at which a gesture indicating a flip or a tap was performed at a timing at which at least one of the three pieces of motion information detected by the motion information detection unit exceeds a threshold value and the positive / negative sign of one of the three pieces of motion information detected by the motion information detection unit becomes different from the positive / negative signs of the remaining two pieces of motion information; and a still image selection unit that selects, from a plurality of frame images stored in the image storage unit, a frame image corresponding to the stationary period detected by the stationary period detection unit before or after the timing at which the trigger signal was generated as a still image. Equipped with.

[0007] A second aspect of the present invention is a method for producing a frame image by imaging an object, comprising: an imaging unit for imaging an object and generating a frame image; an image storage unit for sequentially storing the frame images produced by the imaging unit; and a coordinate system for storing the frame images in each of three axial directions of a coordinate system. Shows acceleration A motion information detection unit that detects motion information, and three motion information detected by the motion information detection unit The above Based on movement information , mobile a stationary period detection unit that detects a stationary period during which the mobile terminal is in a stationary state; At least one of the absolute values ​​of the amount of change per unit time of each of the three pieces of motion information detected by the motion information detection unit exceeds a threshold value, and The three motion information detected by the motion information detection unit The above When the sign of one of the motion information becomes different from the signs of the remaining two, Indicates a flip or tap The device includes a trigger signal generating unit that generates a trigger signal indicating the timing at which a gesture is made, and a still image selecting unit that selects, as a still image, a frame image corresponding to the still period detected by the still period detecting unit before or after the timing at which the trigger signal is generated from a plurality of frame images stored in the image memory unit. Effect of the Invention

[0008] The present invention allows a user to operate a mobile terminal by simply performing simple gestures on the mobile terminal. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of a smartphone according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a state in which a user holds the smartphone. [Figure 3A] FIG. 3A is a diagram illustrating a tap operation by a user. [Figure 3B] FIG. 3B is a diagram illustrating a tap operation by a user. [Figure 4] FIG. 4 is a diagram showing changes in acceleration in each of the three axial directions when a tap operation is performed. [Diagram 5] FIG. 5 is a diagram illustrating a flip operation by a user. [Figure 6]FIG. 6 is a diagram showing changes in acceleration in each of the three axial directions when a flip operation is performed. [Figure 7] FIG. 7 is a diagram showing a stillness start point and a stillness end point that are set in determining the stillness state. [Figure 8] FIG. 8 is a diagram for explaining parameters of an equation for gesture detection. [Figure 9] FIG. 9 is a diagram illustrating a still image selected from a plurality of frame images. [Figure 10] FIG. 10 is a diagram showing an object displayed on a touch panel to assist with operation timing. [Figure 11] FIG. 11 is a diagram showing video playback by gesture operations. [Figure 12] FIG. 12 is a diagram showing acceleration in three axial directions when a gesture corresponding to playback is performed. [Figure 13] FIG. 13 is a diagram showing acceleration in three axial directions when a gesture corresponding to a stop is made. [Figure 14] FIG. 14 is a diagram showing acceleration in three axial directions when a gesture corresponding to fast forward is performed. [Figure 15] FIG. 15 is a diagram showing acceleration in three axial directions when a gesture corresponding to rewinding is performed. [Figure 16] FIG. 16 is a diagram showing a reference cell composed of 5×5 pixels. [Figure 17] FIG. 17 is a diagram for explaining a method for determining the similarity between two reference cells. [Figure 18] FIG. 18 is a diagram showing a virtual layer file associated with a frame image. [Figure 19] FIG. 19 is a diagram for explaining a situation in which an object in a frame image is tracked. [Figure 20] FIG. 20 is a diagram showing a state in which a high-resolution image is displayed as a preview image. [Figure 21A] FIG. 21A is a diagram showing a state in which a local image is tapped. [Figure 21B]FIG. 21B is a diagram showing the results of a search for similar images for sunglasses. [Figure 21C] FIG. 21C is a diagram showing the user editing screen. [Figure 22] FIG. 22 is a diagram showing a local image displayed on a background image. [Figure 23] FIG. 23 is a diagram illustrating a communication network of a smartphone. [Figure 24] FIG. 24 is a flowchart showing a photographing process routine of the control unit. [Diagram 25] FIG. 25 is a flowchart showing the local image extraction routine. [Figure 26] FIG. 26 is a flowchart showing a local image similarity search routine. [Figure 27] FIG. 27 is a diagram showing changes in angular velocity in each of the three axial directions when a tap operation is performed on the smartphone. [Figure 28] FIG. 28 is a diagram showing changes in angular velocity in each of the three axial directions when a flip operation is performed on the smartphone. [Figure 29] FIG. 29 is a diagram showing a stillness start point and a stillness end point that are set in determining a stillness state using an angular velocity. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. (Configuration of Smartphone 1) FIG. 1 is a diagram showing the configuration of a smartphone 1 according to this embodiment. The smartphone 1 includes an imaging unit 11 that captures an image of a subject, an image processing unit 12 that performs predetermined image processing on the image generated by the imaging unit 11, and a data storage unit 13 that stores the images and other data.

[0011] The smartphone 1 further includes a sensor 14 for detecting physical quantities relating to movements in three axial directions, a touch panel 15 for displaying images, an operation screen, and the like, and a communication unit 16 for communicating data with external devices.

[0012] The smartphone 1 operates not only in response to an operation of the touch panel 15 but also in response to a user's gesture (a flip operation or a tap operation, which will be described later). In this embodiment, the smartphone 1 mainly operates in response to a user's gesture or a stationary state.

[0013] 2 is a diagram showing a state in which a user holds the smartphone 1 in the xyz space of a right-handed coordinate system. The origin of the xyz space is the center of the smartphone 1.

[0014] The smartphone 1 is covered with a housing having a rectangular surface with one axial direction (the y-axis in FIG. 2) as a longitudinal direction. The touch panel 15 is provided on an xy plane, specifically on the above-mentioned rectangular surface. The z-axis is a direction perpendicular to the touch panel 15. In this embodiment, when taking a picture, the user holds the smartphone 1 in an xyz space as shown in FIG. 2. In this embodiment, the directions of the x-axis, y-axis, and z-axis when the user holds the smartphone 1 are defined as follows:

[0015] The upper side in the longitudinal direction of touch panel 15 is the positive y-axis direction, and the lower side is the negative y-axis direction. The right side in the direction perpendicular to the longitudinal direction of touch panel 15 is the positive x-axis direction, and the left side is the negative x-axis direction. The front side of touch panel 15 is the positive z-axis direction, and the rear side of touch panel 15 is the negative z-axis direction.

[0016] 1, the imaging unit 11 is an image sensor that generates an image in response to light from a subject. The imaging unit 11 is, for example, a CCD image sensor or a CMOS image sensor. In this embodiment, the smartphone 1 is provided with two imaging units 11. One of the two imaging units 11 is provided on the same rectangular surface as the touch panel 15, and the other is provided on a rectangular surface on the rear side of the touch panel 15.

[0017] The imaging unit 11 has an electronic shutter function and generates frame images for still images or moving images. A release operation for still image shooting is performed by tapping on a display button on the touch panel 15 or by making a gesture on the smartphone 1 body.

[0018] However, in this embodiment, the release operation is performed by a gesture on the main body of the smartphone 1. The frame images generated by the imaging unit 11 are displayed on the touch panel 15 as live moving images and stored in the data storage unit 13.

[0019] The image processing unit 12 performs predetermined image processing such as white balance adjustment and gamma correction on the frame images generated by the imaging unit 11. The images that have been subjected to predetermined processing by the image processing unit 12 are stored in the data storage unit 13. In the case of live moving images, the images are temporarily stored in the data storage unit 13 and then supplied to the touch panel 15 for displaying the live moving images.

[0020] The data storage unit 13 corresponds to, for example, a DRAM, a flash memory, etc., and stores data temporarily or for a long period of time. Various application programs are stored in the data storage unit 13. The control unit 17 executes a predetermined data processing in accordance with the above-mentioned application programs.

[0021] Furthermore, the data storage unit 13 stores images generated by the imaging unit 11, data received from an external network by the communication unit 16, etc. Writing of data to the data storage unit 13 and reading of data from the data storage unit 13 are controlled by the control unit 17.

[0022] The sensor 14 is a motion sensor that detects information related to the movement of the smartphone 1. In this embodiment, the sensor 14 is an acceleration sensor that detects the acceleration in each of the three axial directions of the smartphone 1. The sensor 14 detects the acceleration in each of the three axial directions of the smartphone 1 and supplies the detected values ​​to the control unit 17.

[0023] In this embodiment, the sensor 14 is an acceleration sensor, but is not limited to an acceleration sensor as long as it can detect information related to the movement of the smartphone 1. For example, the sensor 14 may be a speed sensor, an angular velocity sensor (gyro sensor), or a geomagnetic sensor.

[0024] The touch panel 15 is provided on the front surface (the wider rectangular surface) of the smartphone 1. Frame images generated by the imaging unit 11 or read from the data storage unit 13 are displayed on the touch panel 15. In addition, various operation buttons for operating the smartphone 1 are displayed on the touch panel 15.

[0025] (Operation Mode) The smartphone 1 has a photo shooting mode, a video shooting mode, and a live video (preview video) mode.

[0026] In the photography mode, the user determines the shooting position of the smartphone 1 while checking the subject displayed on the touch panel 15. When a release operation for a still image is performed, a single frame image (photo image) or multiple frame images (continuously shot images) specified by the release operation is stored in the data storage unit 13 and displayed on the touch panel 15.

[0027] In the video shooting mode, the user determines the shooting position of the smartphone 1 while checking the subject displayed on the touch panel 15. When a release operation for a video is performed, a plurality of frame images (video) after the release operation are stored in the data storage unit 13 and displayed on the touch panel 15.

[0028] In the live video mode, the user checks the subject displayed on the touch panel 15. At this time, a plurality of frame images (video) generated by the imaging unit 11 are stored in the data storage unit 13 and displayed on the touch panel 15. In other words, even in the live video mode, the frame images are stored in the data storage unit 13.

[0029] In any of the three modes described above, the frame images stored in the data storage unit 13 are displayed on the touch panel 15 when a playback operation is performed. The control unit 17 can also download still or video frame images from an external server and store them in the data storage unit 13. Thereafter, the control unit 17 can read out the downloaded frame images from the data storage unit 13 and display them on the touch panel 15.

[0030] The communication unit 16 can transmit and receive frame images and other data to and from external devices such as external servers via a communication network. The control unit 17 has, for example, a central processing unit (CPU), and controls the entire smartphone 1 by executing a shooting application and various other applications stored in the data storage unit 13.

[0031] The control unit 17 periodically receives detection values ​​in the three axial directions from the sensor 14, and controls the smartphone 1 based on the time-varying pattern of the detection values. Specifically, the control unit 17 determines whether the smartphone 1 is stationary or a gesture has been made on the smartphone 1, and based on the determination result, determines the release timing for still image capture, and obtains a frame image of the still image.

[0032] (Gesture content) A user can perform a release operation for still image shooting by making a gesture on the smartphone 1. In this embodiment, gestures for instructing a release operation for still image shooting include a tap operation and a flip operation.

[0033] 3A and 3B are diagrams illustrating a user's tap operation on the smartphone 1. As shown in FIG. 3, the user holds the lower part of the smartphone 1 with his / her right hand, and determines the position and shooting direction of the smartphone 1 so that a subject (for example, the user himself / herself in the case of taking a selfie) is appropriately displayed on the touch panel 15.

[0034] Next, the user holds the smartphone 1 still. Then, while the smartphone 1 is still, the user taps the back surface of the smartphone 1 (the side opposite the touch panel 15) with the index finger to vibrate the smartphone 1. In this manner, the tap operation refers to tapping the back surface of the smartphone 1 with a finger.

[0035] In the xyz space shown in FIG. 2, the user taps on the smartphone 1 in the z-axis direction. The x and y coordinates of the tapped location may be arbitrary. At this time, the smartphone 1 generates vibrations not only in the z-axis direction, but also in the x-axis and y-axis directions. The vibrations in each axis direction are detected by the sensor 14 in FIG. 1.

[0036] The hand holding the smartphone 1 may be the right hand or the left hand. The user may hold the smartphone 1 vertically or horizontally. In this embodiment, the user slaps the back of the smartphone 1, but the user may slap the front or side of the smartphone 1. The user may slap the upper or lower side of the smartphone 1 body.

[0037] 4 is a diagram showing changes in acceleration in each of the three axial directions when a tap operation is performed on the smartphone 1. When a flip operation is performed while the smartphone 1 is stationary, the acceleration in each of the three axial directions has a characteristic waveform.

[0038] Specifically, when a flip operation is performed, the acceleration d3 in the z-axis direction exceeds the threshold th and reaches a peak. i), the acceleration d1 in the x-axis direction and the acceleration d2 in the y-axis direction also reach their peaks. Thus, when the peak values ​​first reach their peaks, d1 is a negative value, but d2 and d3 are positive values.

[0039] 5 is a diagram illustrating a user's flip operation on the smartphone 1. As shown in FIG. 5, the user holds the lower part of the smartphone with his / her right hand, and determines the position and shooting direction of the smartphone 1 so that a subject (for example, the user himself / herself in the case of a selfie) is appropriately displayed on the touch panel 15.

[0040] Next, the user holds the smartphone 1 still. Then, the user tilts the smartphone 1 backward, quickly returns it to its original stationary position, and vibrates the smartphone 1. In this way, the flip operation refers to tilting the smartphone 1 from a stationary state in a predetermined direction, and then quickly returning it to its original stationary position.

[0041] 2, the user tilts the smartphone 1 around the z-axis. Note that the direction in which the smartphone 1 is tilted is not limited to around the z-axis, and it may be around the x-axis or y-axis. In addition, the hand holding the smartphone 1 may be the right hand or the left hand.

[0042] 6 is a diagram showing changes in acceleration in each of the three axial directions when a flip operation is performed on the smartphone 1. When a flip operation is performed while the smartphone 1 is in a stationary state, the acceleration in each of the three axial directions has a characteristic waveform.

[0043] Specifically, when a flip operation is performed, the acceleration d3 in the z-axis direction exceeds the threshold th and reaches a peak. i), the acceleration d1 in the x-axis direction and the acceleration d2 in the y-axis direction also reach their peaks. Thus, when the peak values ​​first synchronize, d1 is a negative value, but d2 and d3 are positive values. In other words, when a flip operation is performed, the accelerations in the three axes have the same characteristic waveforms as when a tap operation is performed.

[0044] (Determining stationary state) In the case of the photographing mode, the control unit 17 of the smartphone 1 determines the stationary state as follows.

[0045] In the photo shooting mode, the control unit 17 constantly monitors the detection values ​​in the three axial directions supplied from the sensor 14. When the state in which the absolute values ​​of all the detection values ​​of the sensor 14 are equal to or less than a predetermined value continues for a predetermined period of time or more, the control unit 17 determines that the smartphone 1 is in a stationary state for the continuous period of time.

[0046] Here, the detection values ​​(accelerations) of the sensor 14 in the x-axis, y-axis, and z-axis directions are denoted by x, y, and z, respectively. The infinitesimal time is denoted by Δt. The sensor output threshold for determining the stationary state is denoted by α, where α is a value close to zero. At this time, the control unit 17 determines whether or not the formula (1) is satisfied. (|d1|+|d2|+|d3|) / 3<α ···(1)

[0047] 7 is a diagram showing a rest start point ST and a rest end point ED that are set in determining the rest state. When a state that satisfies formula (1) exceeds a predetermined time, the control unit 17 determines that period as the rest state of the smartphone 1. Then, the control unit 17 sets the time when formula (1) is first satisfied as the rest start point ST, and stores information on the rest start point ST (rest start point information) in the data storage unit 13. The rest start point information is the time (hour) of the rest start point ST, and the accelerations d1, d2, and d3 of the rest start point ST in the x, y, and z axial directions.

[0048] When formula (1) is no longer satisfied after the above-mentioned predetermined time has elapsed, the control unit 17 sets the time when formula (1) is no longer satisfied as the rest end point ED, and stores information on the rest end point ED (rest end point information) in the data storage unit 13. The rest end point information is the time (time) of the rest start point ST, and the acceleration in each of the x, y and z axis directions.

[0049] The method by which the control unit 17 determines the stationary state is not limited to the above-mentioned method, and other methods can be applied as long as they can determine the start point information of the stationary state based on the detection value of the sensor 14.

[0050] For example, the condition for determining whether or not the vehicle is in a stationary state is not limited to formula (1), and may be, for example, the following. For example, the control unit 17 may determine whether or not at least one of the detection values ​​in the three axial directions output from the sensor 14 is less than a threshold value th1 (see FIG. 7). That is, the control unit 17 may determine whether or not at least one of the accelerations d1, d2, and d3 is less than a threshold value th1. Alternatively, the control unit 17 may use the three accelerations d1, d2, and d3 supplied from the sensor 14 every minute time Δt to determine whether or not at least one of the absolute values ​​of the amounts of change Δd1, Δd2, and Δd3 per unit time is less than a predetermined threshold value.

[0051] The control unit 17 determines that the motionless state has been reached when the state resulting in a positive determination as a result of the motionless determination process described above continues for a predetermined time (for example, 0.3 seconds). The time is not limited to 0.3 seconds, and may be any time between 0.1 and 1.0 seconds. In this case, the control unit 17 sets the time when the above conditions are first satisfied as the motionless start point ST, and sets the time when the above conditions are no longer satisfied as the motionless end point ED. The motionless state determination is performed before or after a gesture detection, which will be described later.

[0052] (gesture detection) As described with reference to FIGS. 4 and 6, when the user performs a gesture (tap operation or flip operation) on the smartphone 1, the time change pattern of the three detection values ​​(acceleration) of the sensor 14 has the following two characteristics.

[0053] The first feature is that when one detection value exceeds the threshold and becomes the first peak, the other two detection values ​​also become the first peak almost synchronously. In other words, the first peaks of the three detection values ​​occur almost synchronously. The second feature is that when one detection value exceeds the threshold and becomes the first peak, the positive / negative sign of one of the three detection values ​​is different from the positive / negative signs of the remaining two detection values.

[0054] Taking the above-mentioned two characteristics into consideration, the control unit 17 detects whether or not a gesture (tap operation or flip operation) has been made using three detection values ​​in each of the three axial directions of the sensor 14 at each predetermined period of time, according to the following procedure, and generates a trigger signal at the timing when the gesture has been made.

[0055] (Step 1) The control unit 17 determines whether or not the amount of change per given time period exceeds a predetermined threshold for at least one of the three detection values ​​in each of the three axial directions of the sensor 14. If the amount of change in at least one of the three detection values ​​exceeds a threshold during a given period, the control unit 17 proceeds to the next process.

[0056] (Step 2) The control unit 17 determines whether, during the above-mentioned specified period, one of the three detected values ​​will have one positive or negative sign (e.g., a positive value) and the remaining two will have the other positive or negative sign (e.g., negative values).

[0057] In the case of a positive determination result, the control unit 17 determines that the user is performing a gesture operation, and sets the time point when the determination result is obtained as the gesture start time point. In this way, at the gesture start time point, there is a characteristic that the three detection values ​​of the sensor 14 change instantaneously (almost synchronously), and the positive / negative sign of one of the three detection values ​​is different from the positive / negative signs of the remaining two detection values.

[0058] (Step 3) If there is a period of stillness after detection of a gesture, control unit 17 sets the end point of the gesture to one frame before the start point of stillness.

[0059] (Gesture detection method 1) The control unit 17 detects a gesture as follows: The control unit 17 receives three detection values ​​d1, d2, and d3 from the sensor 14 periodically at every minute time Δt.

[0060] The control unit 17 judges whether or not at least one of the three detection values ​​d1, d2, and d3 exceeds the threshold value for each minute time Δt. If the judgment is positive, the control unit 17 determines the time t at which at least one of the three detection values ​​d1, d2, and d3 exceeds the threshold value. i is the start point of the time change pattern. In FIG. 4, d2 and d3 exceed the threshold, but d1 does not exceed the threshold.

[0061] Next, the control unit 17 judges the signs of the three detection values ​​d1, d2, and d3 at the start of the time change pattern. Specifically, the control unit 17 judges whether the sign of one of the three detection values ​​is different from the signs of the remaining two. If the judgment is positive, the control unit 17 sets the time when the positive judgment is obtained as the gesture start time and generates a trigger signal indicating gesture detection. After that, if there is a period of stillness after gesture detection, the control unit 17 sets the time one frame before the stillness start point as the gesture end time.

[0062] (Gesture detection method 2) In addition to the above-described gesture detection method 1, the control unit 17 may detect a gesture in the following manner. 8 is a diagram for explaining parameters of the equations for gesture detection. The control unit 17 calculates the following equations (2) to (4) for each infinitesimal time Δt. Note that Δd1, Δd2, and Δd3 are the amounts of change of d1, d2, and d3 in the infinitesimal time Δt.

[0063] ax = Δd1 / Δt (2) ay = Δd2 / Δt (3) az = Δd3 / Δt (4)

[0064] ax, ay, and az indicate the gradients per minute time Δt (sampling period) of the detection values ​​in the x-axis direction, y-axis direction, and z-axis direction, respectively, of the sensor 14. Next, the control unit 17 determines whether or not at least one of the following expressions (5) to (7) is satisfied. |ax|≧2 (5) |ay|≧2 (6) |a-z|≧2 (7)

[0065] If at least one of the formulas (5) to (7) is satisfied and two of the values ​​ax, ay, and az have the same sign and the remaining value has an opposite sign, the control unit 17 sets the point in time when these conditions are satisfied as the gesture start point and generates a trigger signal indicating gesture detection.

[0066] The control unit 17 may store the amount of change in each of the three detection values ​​of the sensor 14 in the data storage unit 13 together with the frame image. The control unit 17 may perform calculations and judgments for gesture detection in real time while the subject is being photographed, or may perform the calculations and judgments independently after the subject is photographed. The "2" in the formulas (5) to (7) is a threshold value for gesture detection. This threshold value is not limited to "2" and may be another value.

[0067] In this embodiment, the control unit 17 uses the formulas (2) to (7) to detect the user's gesture, but is not limited to the formulas (2) to (7). That is, as long as the control unit 17 can execute the above-mentioned steps 1 to 3, the determination formula is not particularly limited.

[0068] (Still image capture in photo capture mode) In the photography mode, frame images generated by the imaging unit 11 are sequentially stored in the data storage unit 13. The user holds the smartphone 1 still and performs a shutter operation (gesture) while checking the subject displayed on the smartphone 15. At this time, the control unit 17 independently executes the still state determination process and the gesture detection process described above to obtain a still image.

[0069] Specifically, when the control unit 17 detects a gesture, it selects an optimal frame image as a still image from among the multiple frame images stored in the data storage unit 13 using the determination result of the stationary state.

[0070] 9 is a diagram for explaining a still image selected from a plurality of frame images. When a gesture is detected, there is usually a period of stillness (stillness period SD from stillness start point ST to stillness end point ED) before or after gesture detection time TJ. Note that the peak time TP of the detection value occurs after gesture detection time TJ. Therefore, control unit 17 selects a frame image optimal for a still image from a plurality of frame images of the stillness period SD before or after gesture detection JD. There are the following methods for selecting a frame image.

[0071] (How to select a frame image) For example, when there are still periods both before and after the gesture detection time, the control unit 17 may select the frame image closest in time to the gesture detection time as the still image from among all frame images within the still period.

[0072] If the time from the stillness end point to the detection of the next gesture exceeds a predetermined time (for example, any time from 0 to 5 seconds), control unit 17 can exclude the frame image of the stillness period including the stillness end point from the still image targets. Also, if the time from the gesture detection to the next stillness start point exceeds the above-mentioned predetermined time, control unit 17 can exclude the frame image of the stillness period including the stillness start point from the still image targets.

[0073] When the time between the gesture and the still period exceeds the above-mentioned predetermined time, the frame image during the still period is often different from the still image desired by the user. Therefore, when the above-mentioned condition is satisfied, the control unit 17 can prevent malfunction of the photographing operation by excluding the frame image during the still period from the still image targets. In addition, a proximity sensor mounted on the smartphone 1 may be used to prevent malfunction.

[0074] That is, the control unit 17 determines whether the time between the gesture and the still period exceeds the above-mentioned predetermined time, selects a still period in which the time between the gesture and the still period does not exceed the above-mentioned predetermined time, and selects a frame image from the selected still period.

[0075] When selecting a still image from among the frame images in the still period, the control unit 17 may select, for example, the frame image at the stillness start point. Also, the control unit 17 may select a frame image a predetermined time after the stillness start point, or may select a frame image a predetermined number of frames after the stillness start point.

[0076] Furthermore, the control unit 17 can also select an optimal still image from among the frame images during the stationary period, depending on the movement of the smartphone 1. Specifically, the control unit 17 detects the magnitude, tilt direction, and initial speed of the vibration of the smartphone 1 based on three detection values ​​from the sensor 14. Then, the control unit 17 may use the detection results to determine how many frames before or after the gesture detection time a frame image is optimal as a still image, and select the optimal frame image.

[0077] Furthermore, when the continuous shooting function is set, the control unit 17 may select a predetermined number of frame images from among the multiple frame images within the still period, or may select all frame images within the still period.

[0078] (Still image capture using stillness determination only) The control unit 17 can also select a still image by only performing stillness determination without detecting a gesture. Specifically, the control unit 17 performs the above-mentioned stillness determination process to set the stillness start point ST. Next, the control unit 17 determines whether or not a predetermined time has elapsed based on the stillness start point ST within the stillness period. The control unit 17 generates a trigger signal at the point when the predetermined time has elapsed based on the stillness start point ST.

[0079] Then, the control unit 17 selects, as a still image, the frame image at the time when the trigger signal is generated from among the multiple frame images stored in the data storage unit 13. As a result, the user can capture a still image by simply holding the smartphone 1 still.

[0080] (Frame image selection method using histogram) Using the histogram, the control unit 17 can select the most suitable frame image for the still image. Specifically, the control unit 17 calculates the RGB color histogram of the frame image when the gesture is detected and the RGB color histogram of the frame image during the still period, respectively, to obtain the difference between the RGB color histograms of each frame image.

[0081] If the difference in the RGB color histograms of each frame image is smaller than a threshold value, the control unit 17 selects the frame image within the still period as a still image, and if the difference in the RGB color histograms of each frame image is greater than or equal to the threshold value, the control unit 17 excludes the frame image within the still period from the still image list.

[0082] Usually, the user determines the position of the photographing frame (photographing position and photographing direction) before performing a gesture operation. If the difference in the RGB color histograms described above is equal to or greater than a threshold, the frame image at the time of gesture detection and the frame image during the still period are significantly different, and there is a possibility that the positional deviation of the photographing frame has occurred. Therefore, by performing the above-mentioned processing, the positional deviation of the photographing frame can be avoided.

[0083] Note that the user may set in advance a time change pattern, a threshold, and a judgment / detection method that is suitable for the user, thereby allowing the control unit 17 to use the time change pattern, threshold, and judgment / detection method that is optimal for each user.

[0084] (Visualization of detected values ​​to assist with operation timing) 10 is a diagram showing a display object D displayed on the touch panel 15 to assist with the operation timing. The control unit 17 can visualize the detection values ​​of the sensor 14 in the three axial directions by using a transparent virtual layer file VLF that is overlaid on the preview image PV of the touch panel 15, thereby assisting the user with the operation timing.

[0085] Specifically, the control unit 17 calculates the detection value of the sensor 14 in real time. Then, the control unit 17 reads out a display object D of a three-dimensional shape (for example, a horizontal standard, three axes in an xyz space, etc.) described in the virtual layer file VLF, and displays it on the touch panel 15. Then, the control unit 17 converts each detection value of the sensor 14 in the three axial directions into three-dimensional coordinates, and dynamically moves the display object D of a three-dimensional shape. As a result, the display object D that visualizes the state of the detection value of the sensor 14 can be displayed on the touch panel 15.

[0086] This makes it possible to visualize the resting state and gesture movements of the smartphone 1 on the touch panel 15, thereby assisting gesture operations on the smartphone 1. Note that the display object D and other information described in the virtual layer file VLF can be set for each user. The display object D may be a three-dimensional logo or mark.

[0087] (Related information for frame images) In the data storage unit 13, not only the frame images but also related information of the frame images are stored by the control unit 17. The related information corresponds to, for example, the respective detection values ​​in the three axial directions detected by the sensor 14 when the corresponding frame image was generated and the amount of change over time of each detection value.

[0088] Other related information includes, for example, a frame number indicating the number of frames from when the smartphone 1 vibrates or tilts (when a gesture is detected) that the corresponding frame image is, a frame number indicating the number of frames from when a stationary state is detected, and features such as an angle and tilt direction. Each of the above-mentioned frame numbers is automatically assigned in synchronization with T'. Also, when each of the above-mentioned frame numbers is specified, a frame image corresponding to each specified frame number is read out.

[0089] The frame image and related information may be transmitted to an external server by the communication unit 16 and stored in the external server. The frame image and related information stored in the data storage unit 13 or the external server are read out as appropriate when playing back still images or videos.

[0090] For example, when selecting a still image, the control unit 17 detects a still period during which all detection values ​​in the three axial directions or the amount of change over time of the detection values ​​are within a predetermined threshold range and this state (still state) continues for a predetermined period or longer. Next, the control unit 17 selects, as the still image, a frame image that is closest to a completely still state during the still period.

[0091] The frame image in the state closest to a completely still state corresponds to, for example, the frame image when the sum of the absolute values ​​of all the detection values ​​in the three axial directions or the sum of the absolute values ​​of the time change amounts of each detection value becomes the smallest value. Then, the control unit 17 reads out the still image from the data storage unit 13 or the external server, and displays the read out still image on the touch panel 15.

[0092] The control unit 17 can also perform the following processes when selecting a still image. For example, the control unit 17 may specify a still reference frame number or a gesture reference frame number, and read out a frame image corresponding to the specified number from the data storage unit 13 or an external server. The control unit 17 may also specify a feature amount such as an angle or tilt, and read out a frame image corresponding to the specified feature amount from the data storage unit 13 or an external server.

[0093] (Playback function) 11 is a diagram showing video playback by gesture operation. The user can watch a video or scroll information displayed on the touch panel 15 by performing a gesture operation (flip operation) of tilting the smartphone 1 in a predetermined direction.

[0094] When the control unit 17 detects a gesture by the above-described gesture detection process, the control unit 17 determines the gesture content (the direction in which the smartphone 1 is tilted) using the detection values ​​in the three axial directions of the sensor 14 at the time of gesture detection. A table showing the correspondence between the gesture content and the playback function is stored in advance in the data storage unit 13.

[0095] The contents of the table are as follows. Of the four sides that form the outer edge of the smartphone 1, the short side in the positive direction of the x-axis with respect to the smartphone 1 is defined as the top side. Furthermore, the short side in the negative direction of the x-axis is defined as the bottom side, the side in the positive direction of the y-axis is defined as the left side, and the side in the negative direction of the y-axis is defined as the right side. Gesture: Tilt the right side once in the negative z-axis direction → Play Gesture: Stop: Tilt the right side once in the negative z-axis direction → Stop Gesture: Tilt the top edge once in the negative direction of the z-axis → fast forward Gesture: Tilt the bottom edge once in the negative direction of the z-axis → rewind

[0096] Fig. 12 is a diagram showing acceleration in three axial directions when a gesture corresponding to play is performed. Fig. 13 is a diagram showing acceleration in three axial directions when a gesture corresponding to stop is performed. Fig. 14 is a diagram showing acceleration in three axial directions when a gesture corresponding to fast forward is performed. Fig. 15 is a diagram showing acceleration in three axial directions when a gesture corresponding to rewind is performed.

[0097] The three accelerations shown in Figures 12 to 15 have the two features described in Figures 4 and 6. Therefore, the control unit 17 can detect the gestures corresponding to play, stop, fast forward, and rewind by performing the gesture detection process described above.

[0098] Therefore, when control unit 17 detects that the right side has tilted once in the negative direction of the z axis by referring to the table during gesture detection, control unit 17 reads out frame images from data storage unit 13 and plays the video on touch panel 15. When control unit 17 detects a gesture during video playback, control unit 17 executes an operation corresponding to the tilt direction of smartphone 1 according to the table.

[0099] The smartphone 1 can perform multi-function playback or scrolling by increasing the number of gesture detections per predetermined time. In this case, the operation content table describes various playback functions (pause, frame-by-frame playback, frame-by-frame rewind playback, etc.) that are specified by the number of gesture detections per predetermined time and the tilt direction of the smartphone 1. Therefore, the control unit 17 detects the number of gesture detections per predetermined time and the tilt direction of the smartphone 1, and executes the playback function or scrolling corresponding to the detected contents according to the operation content table.

[0100] (Similar search) A large number of frame images are stored in the data storage unit 13. However, it is very difficult for a user to search for information about the subjects (objects) captured in the frame images one by one and prepare such information so that it can be used at any time.

[0101] In response to this, the smartphone 1 can access an external server and perform a similar image search for a frame image stored in the data storage unit 13, thereby obtaining information about a subject (object) appearing in the frame image. As a preprocessing step for performing the similar image search, the control unit 17 of the smartphone 1 performs a local image extraction process for extracting an object appearing in the frame image.

[0102] (Extraction of local images) The control unit 17 extracts a local image from a target frame image in order to identify an object captured in the frame image. The target image is a photograph (still image) obtained by the above-mentioned process, a frame image of a video, a frame image of a preview video, a frame image of a video during a user gesture operation, etc. The control unit 17 extracts a local image from the target image by using a reference cell.

[0103] 16 is a diagram showing a reference cell composed of 5x5 pixels on the xy plane. The x-axis is set in the horizontal direction of the target image, and the y-axis is set in the vertical direction. The coordinates of the reference center pixel, which is the center pixel of the reference cell, are placed at the intersection of the x-axis and y-axis, and the coordinates of the reference center pixel are set to (x0, y0).

[0104] In this case, the two pixels adjacent to the reference center pixel in the x direction and the two pixels adjacent to the reference center pixel in the y direction are the adjacent pixels. For example, the four pixels (x-1, y0), (x+1, y0), (x0, y+1), and (x0, y-1) are the adjacent pixels to the reference center pixel (x0, y0).

[0105] 17 is a diagram for explaining a method for determining the similarity between two reference cells. The control unit 17 creates color histograms for red (R), green (G), and blue (B) in each region of two adjacent reference cells (HSV color space conversion), and determines the similarity between the two reference cells. The control unit 17 detects the feature amounts of each element of hue, saturation, and brightness in the HSV color space for each of the two adjacent reference cells, and calculates a similarity value that indicates the degree of similarity between the two reference cells.

[0106] The similarity value is zero when the two comparison objects are completely identical, and increases as the difference between the comparison objects increases. If the similarity value is less than a threshold, the control unit 17 determines that the two adjacent reference cells are similar, and connects the two reference cells.

[0107] In this manner, when two adjacent reference cells are similar, the control unit 17 successively connects the two reference cells to group the reference cells, thereby extracting a local image.

[0108] (Procedure for extracting local images) The control unit 17 extracts local images from the frame images according to the following procedure. (Step 1) The control unit 17 creates a color histogram Chist = {C(H,1), C(H,2) ... C(H,N), C(S,1), C(S,2) ... C(S,N), C(V,1), C(V,2) ... C(V,N)} for each reference cell in the frame image. Here, H, S, and V are the elements of the HSV color space represented by hue (H), saturation (S), and brightness (V). N is the quantization number.

[0109] (Step 2) The control unit 17 calculates the following color histogram formula to obtain C(r∈{H, S, V}, n).

[0110]

number

[0111] Here, M is the size of the reference cell (pixels). n is the quantization number n = (1, 2, , N). f(Pc(x, y, r)) is the quantized value of Pc(x, y, r). Thus, C(r, n) is the cumulative value of the colors Pc(x, y, H), Pc(x, y, S), and Pc(x, y, V) of each pixel in the reference cell.

[0112] (Step 3) The control unit 17 uses the color histograms calculated in steps (1) and (2) to calculate the following histogram intersection equation for cells i and j, thereby finding the similarity S(i,j).

[0113]

number

[0114] Here, B is the number of bins in the color histogram. When the similarity (approximation value) of two adjacent reference cells is within a predetermined range, the control unit 17 connects the adjacent reference cells as approximate pixels. The control unit 17 groups the reference cells by connecting the adjacent reference cells one after another, and determines the grouped reference cells as local images.

[0115] The method of extracting the local image is not limited to the above-described embodiment, and various methods can be used. For example, the local image may be extracted by extracting a local feature amount for capturing the shape of an object or by detecting the edge of the object.

[0116] (Virtual layer file configuration) 18 is a diagram showing a virtual layer file VLF associated with a frame image (preview image PV). The control unit 17 generates a virtual layer file VLF for each frame image. A virtual layer file VLF is a file in which information related to the corresponding frame image is described. One virtual layer file VLF is associated with a frame image (T' synchronization).

[0117] A layer file number is assigned to the virtual layer file. The virtual layer file describes the area of ​​the local image extracted from the corresponding frame image, the xy coordinates indicating its center position, and the xy coordinates (x0, y0) of the focus point FP of the local image.

[0118] The xy coordinates indicating the area of ​​the local image and its center position are calculated by the control unit 17 during the local image extraction process. Information generated when the subject is photographed is used as the focus point FP.

[0119] The virtual layer file VLF also describes the image features (histogram, etc.) of the corresponding frame image, the detection values ​​in each of the three axial directions of the sensor 14, and the synchronization time T' with the corresponding frame image (the time with the corresponding frame).

[0120] Furthermore, the virtual layer file describes GPS location information indicating the location where the corresponding frame image was taken, and shooting information related to the frame image. The above-mentioned various information is automatically written in the virtual layer file in synchronization with the corresponding frame image. The user may manually write the above-mentioned information in the virtual layer file.

[0121] The control unit 17 can recognize and track the center position of the local image by continuously reading the virtual layer file as necessary. Note that, in this embodiment, all information about the frame image is described in the virtual layer file, but this is not limited to this. In other words, part of the information about the frame image may be described in a file different from the virtual layer file, or may be described and stored in a server on a communication network.

[0122] 19 is a diagram for explaining a situation in which an object in a frame image is tracked. When tracking an object (local image), the control unit 17 uses the focus point of a virtual layer file synchronized with the local image as position information.

[0123] First, the control unit 17 reads out a frame image including a local image from the data storage unit 13, and also reads out a virtual layer file synchronized with the frame image. The control unit 17 acquires the basic coordinates BC (x0, y0) which are the focus point (position information) FP of the object to be tracked from the virtual layer file.

[0124] Then, every time the control unit 17 reads out a frame image from the data storage unit 13, it also reads out a synchronized virtual layer file and sequentially reads out focus points from the virtual layer file to obtain tracking coordinates TC (x0, y+7), (x+1, y+4), (x0, y+2), (x, 0). Note that the tracking information may be written in the virtual layer file, or may be stored in the data storage unit 13 in the smartphone 1 or an external server.

[0125] (Information storage and management) The control unit 17 stores feature quantities such as the detection values ​​in the three axial directions of the sensor 14, the amount of change, the time change pattern, and the angle and tilt direction in the data storage unit 13. The control unit 17 also stores, for all frame images, RGB color histogram information, grayscale conversion information, time-wise position information of objects appearing in the local image, two-dimensional xy-axis coordinates representing the object region in the local image, and three-dimensional xyz-axis coordinates that are focus point data for the above objects in the data storage unit 13. Furthermore, the control unit 17 stores, in the data storage unit 13, shooting information when the object was shot, GPS position information, object tracking information of the local image, etc.

[0126] The control unit 17 may store the above-mentioned various information inside the virtual layer file, or inside the frame image. The control unit 17 may store the above-mentioned various information in an external server via a communication network, instead of storing the information in the data storage unit 13 inside the smartphone 1.

[0127] (Display of local image) When the control unit 17 detects a tap on an object (local image) displayed on the touch panel 15, it reads out information related to the object from the data storage unit 13 and displays the read out information on the touch panel 15.

[0128] Note that high-quality data of the local image may be stored in advance in the data storage unit 13. In this case, when the control unit 17 detects a tap on a local image displayed on the touch panel 15, the control unit 17 may read out the high-quality data from the data storage unit 13 and display a high-resolution image (e.g., 4K / 8K) of the tapped local image on the touch panel 15.

[0129] 20 is a diagram showing a situation in which a high-resolution image is displayed as the preview image PV. When the preview image PV is a high-resolution image, the control unit 17 can divide the preview image PV into, for example, eight parts and extract a local image for each divided high-resolution image. Note that the preview image PV is not limited to being divided into eight parts, and may be divided into four parts.

[0130] 21A is a diagram showing a state in which a local image is tapped. The control unit 17 displays an enlarged image of the tapped local image and a similar search result for the local image on the touch panel 15. For example, when sunglasses of a person displayed on the touch panel 15 are tapped, the control unit 17 displays an enlarged image of the tapped sunglasses on the touch panel 15.

[0131] The control unit 17 further instructs the external server to search for images similar to the tapped sunglasses. When the external server searches for images similar to the sunglasses in the database, the external server transmits images of the similar sunglasses and related information to the smartphone 1.

[0132] 21B is a diagram showing the results of a similar image search for sunglasses. The control unit 17 can display not only an enlarged image of the sunglasses, but also images and related information of multiple sunglasses similar to the sunglasses on the touch panel 15. Here, the user can input and edit information about the sunglasses.

[0133] 21C is a diagram showing a user editing screen. The user can input any information related to the sunglasses by operating the editing screen displayed on the touch panel 15. The control unit 17 can store the information input by the user in the data storage unit 13 or an external server and read it out when necessary.

[0134] (Information synchronized with the frame image) The control unit 17 not only extracts local images from the frame images, but also synchronizes various information for all the frame images. Specifically, the control unit 17 assigns frame numbers and time information as image features to each of a single frame image obtained by photographing, consecutive frame images obtained by continuous photographing, and consecutive frame images obtained by video shooting or preview video.

[0135] That is, a frame number and time information are assigned to every frame image. The frame images generated by photographing or videotaping are stored in the data storage unit together with the frame number and time information. The frame number and time information stored in the data storage unit 13 are used to extract and manage the frame images.

[0136] (Local image and background image) 22 is a diagram showing a local image LI displayed on a background image BI. After extracting the local image LI for each frame image, the control unit 17 can distinguish between the local image LI and the background image BI (frame image other than the local image LI) and store them in the data storage unit 13 or an external server. Furthermore, the control unit 17 may store image information related to the local image LI or the background image BI, feature amounts of the local image LI, and the like in the data storage unit 13 or an external server.

[0137] Furthermore, for a moving image (a plurality of consecutive frame images), the control unit 17 may distinguish between a moving local image LI and a motionless background image BI. During playback, the control unit 17 reads out the local image LI and the background image BI from the data storage unit 13 or the like, for example, based on the feature amount, and overwrites the local image LI on the background image BI. The control unit 17 displays the local image LI or an enlarged image thereof on the touch panel 15.

[0138] (Preview image display and local image extraction) Control unit 17 may extract a local image from a frame image of a preview image while displaying the preview image on touch panel 15. Control unit 17 may also temporarily store the preview image in data storage unit 13, and then read out the frame image of the preview image from data storage unit 13 and extract a local image from the frame image.

[0139] (Display correction according to tilt) When displaying a preview image or a reproduced image on the touch panel 15, the control unit 17 can adjust the displayed image in accordance with the vibration or tilt of the smartphone 1. Specifically, the control unit 17 determines in which direction and to what extent the smartphone 1 is tilted, based on the detection values ​​in each of the three axial directions from the sensor 14.

[0140] Then, based on the above-mentioned determination result, the control unit 17 configures a virtual display device in which the display screen is horizontal within the touch panel 15. This allows the user to view images without being affected by the vibration or tilt of the smartphone 1.

[0141] (Reduced or enlarged image depending on the distance to the subject) It is also possible to display a reduced or enlarged image according to the distance from the smartphone 1 to the subject. Specifically, the control unit 17 uses the focal length f and the aperture value x in the imaging unit 11 to obtain the distance a from the smartphone 1 (imaging lens) to the subject (the user himself in FIG. 24). The method of calculating the distance a is not particularly limited, and known techniques can be used. The control unit 17 can display a preview image and a reproduced image (e.g., a still image or a moving image) that is an image after shooting on the touch panel 15 in a reduced or enlarged manner according to the obtained distance a.

[0142] (Example of communication network configuration) 23 is a diagram illustrating a communication network of the smartphone 1. The smartphone 1 is connected to various external servers via the network NW. The external servers include, for example, a communication carrier server 101, a cloud server 102, an image database server 103, and a carrier image database server 104. These servers store data used by the smartphone 1 and various data related to the data.

[0143] 24 is a flowchart showing a photographing process routine of the control unit 17. The control unit 17 selects an optimal frame image as a still image from among a plurality of frame images generated by the imaging unit 11 according to the following photographing process routine. Note that in each of the following steps, the processing already described is performed, so the details are omitted.

[0144] When the camera function is activated, the control unit 17 receives detection values ​​in the three axial directions from the sensor 14 (step S1). The control unit 17 transitions to a photography mode in response to a user's operation (step S2) and displays a frame image generated by the imaging unit 11 on the touch panel 15 as a preview image.

[0145] The control unit 17 performs the above-described stationary state determination using the three detection values ​​of the sensor 14 to determine whether the smartphone 1 has come to a standstill (step S3). The control unit 17 waits in step S3 until it is determined that the smartphone 1 has come to a standstill, and when it is determined that the smartphone 1 has come to a standstill, the control unit 17 proceeds to the next step S4.

[0146] The control unit 17 performs the above-mentioned gesture detection process (step S4) using the three detection values ​​of the sensor 14. The control unit 17 waits in step S4 until it detects a gesture, and when it detects a gesture, it proceeds to the next step S5.

[0147] The control unit 17 again determines whether the smartphone 1 has stopped (step S5). If the control unit 17 determines that the smartphone 1 has stopped, the process proceeds to the next step S6. Note that the control unit 17 waits in step S5 until it is determined that the smartphone 1 has stopped, but if a predetermined time (e.g., 5 seconds) has elapsed since the gesture was detected, the process automatically proceeds to the next step S6.

[0148] The control unit 17 selects the most suitable frame image as a still image from among the frame images corresponding to the still period among the multiple frame images stored in the data storage unit 13 for still image shooting (step S6). The method of selecting the most suitable frame image as a still image is as described above and is not particularly limited. The control unit 17 saves the frame image selected in step S6 in the data storage unit 13 or an external server (step S7).

[0149] 24, control unit 17 executes all of steps S3 to S5, but it is not necessary to execute all of steps S3 to S5, and step S3 or step S5 may be omitted. For example, control unit 17 may omit step S5 and select a still image from among a plurality of frame images in a still period before the gesture is detected. Control unit 17 may also omit step S3 and select a still image from among a plurality of frame images in a still period after the gesture is detected.

[0150] 25 is a flowchart showing a local image extraction routine. The control unit 17 extracts local images from a frame image according to the following local image extraction routine. Since the processes already described are performed in the following steps, details thereof will be omitted.

[0151] When the camera function is activated, the control unit 17 receives detection values ​​in each of the three axial directions from the sensor 14 (step S11). The control unit 17 transitions to a live video mode in response to a user operation (step S12), and displays a frame image generated by the imaging unit 11 as a preview image on the touch panel 15 (step S13). Note that the image displayed on the touch panel 15 is not limited to a preview image in the live video mode, and may be a frame image read out from the data storage unit 13.

[0152] The control unit 17 extracts image features from the frame images of the display image (step S14), and further extracts local images from the frame images (step S15). The control unit 17 creates a virtual layer file that is synchronized with the frame images including the local images, and writes various information to the virtual layer file (step S16).

[0153] For example, the control unit 17 sets x and y coordinates at the center positions of the pixels constituting the local image, sets x and y coordinates of the focus point in the local image, and describes these setting information in the virtual layer file. Furthermore, the control unit 17 tracks the local image using the focus point of the virtual layer file as position information (step S17). Then, the control unit 17 manages and saves the above-mentioned processing for each frame rate (step S18).

[0154] 26 is a flowchart showing a local image similarity search routine. The control unit 17 searches for information about the local image according to the next local image similarity search routine, and displays it on the touch panel 15. In each of the following steps, the processing already described is performed, so the details are omitted.

[0155] The control unit 17 displays an image on the touch panel 15 (step S21). The image displayed on the touch panel 15 may be a preview image or a frame image read out from the data storage unit 13.

[0156] The control unit 17 instructs a similarity search for the local images of all images displayed on the touch panel 15, for example, in an image database shown in Fig. 23 (step S22). At this time, the image database searches for images similar to the local image or information related to the local image, and transmits the images and information found by the search to the smartphone 1.

[0157] When the control unit 17 of the smartphone 1 receives the images and information found by the search, the control unit 17 displays the images and information on the touch panel 15. This enables the control unit 17 to quickly search for images similar to the local image (object) of the frame image.

[0158] The control unit 17 determines whether or not a tap on the local image displayed on the touch panel 15 has been detected (step S23). The control unit 17 waits in step S23 until a tap is detected, and when a tap is detected, the control unit 17 proceeds to the next step S24.

[0159] The control unit 17 dynamically displays the information received from the image database for the tapped local image on the touch panel 15 (step S24). For example, if the local image shows sunglasses, sunglasses similar to the sunglasses are displayed on the touch panel 15. At this time, the user can manually write the image information in a virtual layer file. The control unit 17 saves this virtual layer file in the data storage unit 13, an external server, or the like (step S25).

[0160] (Other examples of information usage) By using the focus point as position information of the object region of the local image (object), the control unit 17 can acquire not only the object but also the position information from the frame images, and display the object and the position information on the touch panel 15. Furthermore, it is possible to track a specific object or the like from the reproduced video or live video images (all frame images) displayed on the touch panel 15.

[0161] The control unit 17 can store in the data storage unit 13 the three detection values ​​in the three axial directions of the sensor 14, the amount of change over time of each detection value, the time change pattern, feature quantities such as the angle and tilt direction, RGB color histogram information or grayscale conversion information of all frame images, and position information on the time axis of the local image (object).

[0162] Furthermore, the control unit 17 can store in the data storage unit 13 data sets such as two-dimensional xy coordinates representing the object area of ​​the local image, focus point data in three-dimensional xyz coordinates, shooting information, GPS position information, object tracking information of the local image, etc.

[0163] The control unit 17 may cache the above-mentioned data in a memory array, store it inside an image file or an overlaid virtual layer file, or store it in an external server as shown in Fig. 27. The control unit 17 may also read various data stored as described above as necessary, perform predetermined arithmetic processing, and display an image or other method on the touch panel 15.

[0164] As described above in detail, the smartphone 1 according to this embodiment includes the imaging unit 11 that captures an image of a subject, the sensor 14 that detects a position in three axial directions or a quantity related to the position, and the control unit 17 that controls the operation and playback function of the imaging unit 11 using at least one of the three detection values ​​output from the sensor 14. This enables the smartphone 1 to control the operation and playback function of the imaging unit 11 by a user performing a gesture operation on the smartphone.

[0165] The present invention is not limited to the above-described embodiment, and can be applied to designs modified within the scope of the claims. The present invention can be applied to devices other than the smartphone 1, for example, to mobile terminals without a telephone function.

[0166] In the above-described embodiment, the sensor 14 is an acceleration sensor, but the sensor 14 may be an angular velocity sensor (gyro sensor). Fig. 27 is a diagram showing changes in angular velocity in each of the three axial directions when a tap operation is performed on the smartphone 1. Fig. 28 is a diagram showing changes in angular velocity in each of the three axial directions when a flip operation is performed on the smartphone 1. When a tap operation or flip operation is performed while the smartphone 1 is stationary, the angular velocity in each of the three axial directions has two characteristics similar to those of the acceleration.

[0167] 29 is a diagram showing the stillness start point and the stillness end point set in the determination of the still state using the angular velocity. In this case, the control unit 17 can set the stillness start point ST and the stillness end point ED by using, for example, the above-mentioned formula (1). Note that α in formula (1) is set to an optimal value when a gyro sensor is used as the sensor 14. [Explanation of symbols]

[0168] 1. Smartphone 11 Imaging unit 12 Image processing section 13 Data storage unit 14 Sensors 15 Touch Panel 16 Communications Department 17 Control section

Claims

1. an imaging unit that captures an image of a subject and generates a frame image; an image storage unit that sequentially stores frame images generated by the imaging unit; a motion information detection unit that detects motion information indicating acceleration in each of three axial directions of a coordinate system; a rest period detection unit that detects a rest period during which the mobile terminal is in a rest state based on the three pieces of motion information detected by the motion information detection unit; a trigger signal generating unit that generates a trigger signal indicating a timing at which a gesture indicating a flip or a tap is performed at a timing when at least one of the three pieces of motion information detected by the motion information detecting unit exceeds a threshold value and a positive / negative sign of one of the three pieces of motion information detected by the motion information detecting unit becomes different from a positive / negative sign of the remaining two pieces of motion information; a still image selection unit that selects, as a still image, a frame image corresponding to the still period detected by the still period detection unit before or after the generation timing of the trigger signal from among a plurality of frame images stored in the image storage unit; A mobile terminal equipped with

2. an imaging unit that captures an image of a subject and generates a frame image; an image storage unit that sequentially stores frame images generated by the imaging unit; a motion information detection unit that detects motion information indicating acceleration in each of three axial directions of a coordinate system; a rest period detection unit that detects a rest period during which the mobile terminal is in a rest state based on the three pieces of motion information detected by the motion information detection unit; a trigger signal generating unit that generates a trigger signal indicating the timing at which a gesture indicating a flip or a tap is performed when at least one of the absolute values ​​of the amount of change per time of each of the three pieces of the movement information detected by the movement information detecting unit exceeds a threshold value and the positive / negative sign of one of the three pieces of the movement information detected by the movement information detecting unit becomes different from the positive / negative signs of the remaining two pieces of the movement information; a still image selection unit that selects, as a still image, a frame image corresponding to the still period detected by the still period detection unit before or after the generation timing of the trigger signal from among a plurality of frame images stored in the image storage unit; A mobile terminal equipped with

3. The still image selection unit excludes a still period that is a predetermined time or more away from the generation timing of the trigger signal from the still period before or after the generation timing of the trigger signal, and selects a frame image corresponding to the remaining still period as a still image. The mobile terminal according to claim 1 or 2.

4. a display unit that displays a frame image generated by the imaging unit and a three-dimensional object; a display object control unit that controls a movement of a display object displayed on the display unit in accordance with the motion information in each of the three axial directions detected by the motion information detection unit; The mobile terminal according to claim 1 or 2, further comprising:

5. a still image storage unit that stores a frame image selected as a still image by the still image selection unit; a local image extracting unit that extracts a local image corresponding to the subject from the frame image stored in the still image storage unit; a search request unit that requests an external server to search for information related to the local image extracted by the local image extraction unit; A search result display unit that displays the search results returned from the external server. The mobile terminal according to claim 1 or 2.

6. The search result display unit displays an image similar to the local image as information related to the local image. The mobile terminal according to claim 5.

7. a second trigger signal generating unit that generates a second trigger signal indicating a timing at which the gesture in the stationary state is performed at a timing when a predetermined time has elapsed based on a timing at which the stationary state is attained, based on at least one of the three pieces of movement information detected by the movement information detecting unit; and and a second still image selection unit that selects, as a still image, a frame image corresponding to the generation timing of the second trigger signal from among the plurality of frame images stored in the image storage unit. The mobile terminal according to claim 1 or 2.

Images