Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus and an image processing method with which it is possible to determine more detailed camera work by using a plurality of pieces of information.SOLUTION: An image processing apparatus determines the direction of movement of an imaging apparatus on the basis of a motion vector detected from moving images. The image processing apparatus discriminates the camera work of the imaging apparatus during photographing of the moving images on the basis of the determined direction of movement, information on a subject area detected in the moving images, and information on the movement of the imaging apparatus during photographing of the moving images.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Identifying the camerawork (camera movement method and shooting technique) used when shooting a video is useful, for example, for appropriately processing the video. Conventionally, a method for identifying specific camerawork based on the distribution of motion vectors within an image has been known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-344131 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional technology disclosed in Patent Document 1 distinguishes camerawork based on the frequency and shape of motion vector distribution, which means that it can only distinguish between rough camerawork such as panning, zooming, and tracking.

[0005] In some embodiments, the present invention provides an image processing device and an image processing method that are capable of determining camerawork in more detail using multiple pieces of information. [Means for solving the problem]

[0006] In one aspect, the present invention provides an image processing device comprising: a detection means for detecting a motion vector from a video; a first acquisition means for acquiring information on a subject area detected in the video; a second acquisition means for acquiring information on the movement of the imaging device when shooting the video; a determination means for determining the movement direction of the imaging device based on the motion vector; and a determination means for determining the camera work of the imaging device when shooting the video based on the movement direction, the information on the subject area, and the information on the movement. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device and an image processing method that are capable of determining camera work in more detail using a plurality of pieces of information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional arrangement of a digital camera according to a first embodiment; [Figure 2] FIG. 1 is a diagram showing an example of the configuration of a video file generated by a digital camera according to a first embodiment; [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of a shooting information generating unit according to a first embodiment; [Figure 4] 1 is a flowchart illustrating the operation of the camerawork determination unit in the first embodiment. [Figure 5] Schematic diagram of horizontal camera movement and detected motion vectors [Figure 6] Schematic diagram of camera forward movement and detected motion vectors [Figure 7] Flowchart showing details of S402 in FIG. 4 [Figure 8] Flowchart showing details of S702 in Figure 7 [Figure 9] FIG. 8 is a diagram for explaining the process performed in S702 of FIG. 7; [Figure 10] FIG. 8 is a diagram for explaining the process performed in S702 of FIG. 7; [Figure 11]FIG. 8 is a diagram for explaining the process performed in S702 of FIG. 7; [Figure 12] FIG. 10 is a diagram showing an example of estimating a moving direction in the first embodiment; [Figure 13] Flowchart showing details of S405 in Figure 4 [Figure 14] Flowchart showing details of S1302 in Figure 13 [Figure 15] Flowchart showing details of S1304 in Figure 13 [Figure 16] Flowchart showing details of S1306 in Figure 13 [Figure 17] FIG. 10 is a diagram showing a list of cameraworks that can be distinguished in the first embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of the appearance of a gimbal camera according to a second embodiment. [Figure 19] FIG. 10 is a block diagram showing an example of the functional configuration of a gimbal camera according to a second embodiment. [Figure 20] A diagram to explain misclassification when taking photos while walking [Figure 21] (A) is a diagram showing the time change in the gimbal control amount when shooting while walking, and (B) is a schematic diagram of the gimbal camera when shooting while walking. [Figure 22] Flowchart for camera work determination processing in the second embodiment [Figure 23] FIG. 10 is a diagram for explaining a third embodiment. [Figure 24] Flowchart for camera movement direction determination processing in the third embodiment [Figure 25] FIG. 11 is a diagram showing angles of motion vectors in the third embodiment. [Figure 26] A diagram showing an example of an angle histogram of motion vectors. [Figure 27] Flowchart showing details of S2404 in FIG. 24 [Figure 28] Flowchart showing details of S2405 in Figure 24 [Figure 29] Flowchart for details of S2801 in Figure 28 [Figure 30]Flowchart showing details of S2802 in Figure 28 [Figure 31] 13 is a schematic diagram of a partial region of a motion vector in the third embodiment; [Figure 32] Schematic diagram of motion vectors detected by a moving object DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following, the present invention will be described in terms of an embodiment using a digital camera as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] ●(First embodiment) FIG. 1 is a block diagram showing an example of the functional configuration related to video recording of a digital camera 1 as an example of an image processing device according to an embodiment, along with a processing flow. Each functional block of the digital camera 1 can be implemented by software or a combination of software and hardware, except for parts that can clearly only be realized by hardware (e.g., optical lenses, image sensors, etc.). For example, a functional block may be realized by dedicated hardware such as an ASIC. A functional block may also be realized by a processor such as a CPU executing a program stored in memory. Note that multiple functional blocks may be realized by a common configuration (e.g., a single ASIC). Furthermore, hardware that realizes part of the functions of one functional block may be included in hardware that realizes another functional block.

[0012] One or more processors (hereinafter simply referred to as CPUs) 100 are the control units of the digital camera 1. The CPU 100 controls the operation of each functional block by, for example, loading a program stored in a ROM 102 into a RAM 101 and executing it, thereby realizing the functions of the digital camera 1.

[0013] The ROM 102 is, for example, a rewritable nonvolatile memory, and stores programs executable by the CPU 100, setting values, GUI data, etc. The RAM 101 is used to load programs executed by the CPU 100 and to save values ​​required during program execution. Furthermore, part of the RAM 101 may be used as a video memory for storing display image data.

[0014] Operation unit 103 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to digital camera 1. The input devices that make up operation unit 103 have names according to the functions assigned to them. For example, operation unit 103 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, an enter key, etc.

[0015] The release switch is a switch for recording still images, and the CPU 100 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. The CPU 100 also recognizes a video recording switch pressed in shooting standby mode as an instruction to start video recording, and a switch pressed during video recording as an instruction to stop recording. The functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display.

[0016] The imaging unit 11 includes an optical lens that generates an optical image of the subject and an imaging element that converts the optical image into an image signal. The imaging unit 11 may further include a mechanical shutter, an aperture, and the like. The imaging element may be, for example, a known CCD or CMOS color image sensor with a primary-color Bayer array color filter. The imaging element includes a pixel array in which multiple pixels are arranged two-dimensionally and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. A group of pixel signals (analog image signals) representing the optical image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of charge accumulated during the exposure period. The operation of the imaging unit 11 is controlled by the CPU 100.

[0017] In the following description, the imaging unit 11 outputs a moving image signal having a predetermined frame rate, but it is also possible for the imaging unit 11 to output a still image signal. If the imaging element has an A / D conversion function, the imaging unit 11 outputs a digital moving image signal (moving image data).

[0018] The image signal output by the imaging unit 11 is supplied to the image processing unit 14. The image processing unit 14 applies predetermined image processing such as A / D conversion to the image signal output by the imaging unit 11 to generate signals and image data according to the application, and acquire and / or generate various types of information. The operation of the image processing unit 14 is controlled by the CPU 100.

[0019] The image processing unit 14 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 14 may be configured so that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing unit 14 outputs acquired or generated information and data to the CPU 100, RAM 101, etc. depending on the application.

[0020] The image processing applied by the image processing unit 14 includes, for example, preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, etc. In Fig. 1, of the various image processing applied by the image processing unit 14, particularly the detection of subject information and the detection of motion vectors, which are part of the detection processing, are described for convenience as being executed by individual function blocks 141 and 142.

[0021] The subject information detection unit 141 detects, as a subject area, an image area that is thought to contain a predetermined type of subject. For each detected subject area, the subject information detection unit 141 outputs the position and size within the image, the reliability of detection, and other detection results. The subject information detection unit 141 can detect a subject area using any known method, such as pattern matching or a machine learning model. There are no particular restrictions on the type of subject that the subject information detection unit 141 detects, and various types of subjects can be detected, such as the face or entire body of a person or animal, or a vehicle.

[0022] The motion vector detection unit 142 detects a motion vector between frames based on the current frame and a previous frame (e.g., the most recent frame). The motion vector detection unit 142 detects a motion vector for each region obtained by dividing the entire previous frame into multiple regions in the horizontal and vertical directions. The motion vector detection unit 142 also detects a motion vector for each object region detected in the previous frame. The motion vector detection unit 142 can detect a motion vector using any known method. For example, the motion vector of a template can be detected by performing template matching on the current frame using a region of the previous frame as a template. Specifically, the motion vector detection unit 142 can detect a vector whose starting point is the coordinate of the template and whose ending point is the coordinate of the region in the current frame that has the highest correlation with the template. The coordinates may be, for example, the image coordinates of the center or center of gravity of the region. The motion vector detection unit 142 stores the motion vector detection results in RAM 101.

[0023] The image processing unit 14 outputs the image data of the current frame to which the image processing has been applied to the moving image file generation unit 12. For convenience of explanation, the moving image file generation unit 12 is depicted in FIG. 1 as a functional block separate from the image processing unit 14, but the image processing unit 14 may include the moving image file generation unit 12.

[0024] The video file generator 12 generates a video file 2 having a data structure such as that shown in Fig. 2. The video file 2 stores at least video data 21 and shooting information data 22. The video data 21 and shooting information data 22 may be stored in the video file 2 in an arrangement opposite to that shown in Fig. 2. In practice, the video data 21 and shooting information data 22 are stored in the video file 2 in an arrangement that corresponds to the video file format adopted by the digital camera 1.

[0025] Like the image processing unit 14, the image file generation unit 12 may be a dedicated hardware circuit, or may be configured to realize a specific function by a processor executing a program stored in memory. The video data generation unit 121 and the shooting information data generation unit 122 are functional blocks that describe some of the functions of the image file generation unit 12. Therefore, the operations performed by the video data generation unit 121 and the shooting information data generation unit 122 are actually performed by the image file generation unit 12.

[0026] The video data generation unit 121 generates video data based on image data supplied from the image processing unit 14. The video data generation unit 121 applies necessary processing, such as encoding processing, to the image data to generate video data in a format set in the digital camera 1, for example. The video data generation unit 121 stores the generated video data in the RAM 101.

[0027] The shooting information data generation unit 122 generates shooting information data 22. In the example shown in Fig. 2, the shooting information data 22 includes date information 221, video length 222, subject recognition information 223, camerawork information 224, and frame rate 225. However, the number and types of information included in the shooting information data 22 are not limited to these. Details of the operation of the shooting information data generation unit 122 will be described later.

[0028] The moving image file generating unit 12 generates moving image file data that stores the moving image data generated by the moving image data generating unit 121 and the shooting information data generated by the shooting information data generating unit 122, and stores the data in the RAM 101.

[0029] The video file recording unit 13 records the data of the video file generated by the video file generating unit 12 in a predetermined recording destination. The recording destination may be, for example, a memory card attached to the digital camera 1 or an external recording device such as cloud storage.

[0030] The device vibration detection unit 123 outputs a signal corresponding to the movement of the digital camera 1 to the shooting information data generation unit 122. The device vibration detection unit 123 may be, for example, an acceleration sensor or an angular velocity sensor. Here, as an example, the device vibration detection unit 123 is assumed to be an angular velocity sensor that detects angular velocities around each axis of a Cartesian coordinate system consisting of the optical axis direction (x-axis) of the digital camera 1 and three axes that are orthogonal to the optical axis direction and extend in the horizontal direction (y-axis) and vertical direction (z-axis) of the image sensor. Note that the horizontal direction of the image sensor may be a direction parallel to the bottom surface of the digital camera 1 or a direction parallel to the long side of the image sensor.

[0031] 3 is a diagram showing, as functional blocks, each of the representative functions of the shooting information data generation unit 122. Each functional block of the shooting information data generation unit 122 can be implemented by software, hardware, or a combination thereof, depending on the implementation form of the video file generation unit 12. Note that the shooting information data generation unit 122 may have functions other than those shown in the figure.

[0032] The camerawork determination unit 122b determines the camerawork used during video shooting based on two or more of the detection results of the subject information detection unit 141, the detection results of the motion vector detection unit 142, and the movement of the digital camera 1 detected by the device shake detection unit 123. The camerawork is the method of camera movement and the type of shooting technique used to express the video.

[0033] Fig. 17 shows a list of cameraworks that can be distinguished in this embodiment. In Fig. 17, the camera being in a fixed position means that the viewpoint position of the camera is not moving, and does not limit the position of the camera. Similarly, the subject being in a fixed position means that the subject is not moving, and does not limit the position of the subject. The processing of the camerawork distinguishing unit 122b will be described in detail later.

[0034] The frame rate detection unit 122c detects the frame rate of the moving image data generated by the moving image data generation unit 121. The frame rate detection may be performed continuously or periodically while the moving image data is being generated. Note that if the frame rate of the moving image data generated by the moving image data generation unit 121 is constant, the frame rate detection unit 122c may detect the frame rate by referring to a setting value stored in the ROM 102 or RAM 101.

[0035] (Operation of camerawork determination unit 122b) 4 is a flowchart showing the overall operation of the camerawork determination unit 122b. As described above, the camerawork determination unit 122b is a function of the shooting information data generation unit 122. Therefore, the operation of the camerawork determination unit 122b is actually performed by the video file generation unit 12.

[0036] In S401, the camerawork determination unit 122b acquires the detection result (motion vector information) of the motion vector detection unit 142, which is stored in the RAM 101, for example.

[0037] Here, we will explain motion vectors. In Fig. 5, 500 shows a case where both subject 501 and digital camera 1 remain stationary from the time a previous frame (for example, the most recent frame) is captured until the current frame is captured. Also, 503 shows a case where subject 501 remains stationary from the time a previous frame (for example, the most recent frame) is captured until the current frame is captured, but digital camera 1 moves horizontally to the left. Note that although Fig. 5 shows only subject 501 in the image, in reality the background is also captured.

[0038] Reference numeral 510 schematically illustrates an example of regions 511 for which the motion vector detection unit 142 detects motion vectors for frame 500, and the motion vectors detected for each region. Here, an example is shown in which the motion vector detection unit 142 detects a motion vector for each of 64 regions 511 obtained by dividing the entire frame into eight equal parts horizontally and vertically. For convenience, FIG. 5 depicts each region 511 as if there is space around it, but in reality, no space exists around the regions 511. Since neither the subject 501 nor the digital camera 1 has moved in frame 500 since the previous frame, the magnitude of the motion vector detected for each region 511 is zero.

[0039] On the other hand, in frame 503, subject 501 remains stationary since the previous frame, but digital camera 1 has moved horizontally to the left. Therefore, as shown in 504, a motion vector 512 pointing horizontally to the right is detected for each region 511.

[0040] 6, 600 indicates a case where both the subject 601 and the digital camera 1 remain stationary from the time a previous frame (for example, the most recent frame) is captured until the time the current frame is captured. 603 indicates a case where the subject 601 remains stationary from the time a previous frame (for example, the most recent frame) is captured until the time the current frame is captured, but the digital camera 1 moves forward in the optical axis direction. Note that while only the subject 601 is shown in the image in FIG. 6, the background is also captured in reality.

[0041] Reference numeral 610 schematically shows an example of an area 611 in which the motion vector detection unit 142 detects motion vectors for the frame 600, and the motion vectors detected for each area. The area 611 is the same as the area 511 in Fig. 5. Since neither the subject 601 nor the digital camera 1 has moved in the frame 600 since the previous frame, the magnitude of the motion vector detected for each area 611 is 0.

[0042] On the other hand, in frame 603, the subject 601 remains stationary since the previous frame, but the digital camera 1 has moved forward in the optical axis direction. Therefore, as shown in 604, a motion vector 612 extending from the center of the image outward is detected for each region 611.

[0043] The camerawork determination unit 122b acquires motion vector information (for example, image coordinates of the start and end points of each motion vector) that is the detection result of the motion vector detection unit 142. The vector information acquired here includes motion vector information for the entire frame and motion vector information for the subject area.

[0044] In S402, the camerawork determination unit 122b executes a process for determining the movement direction of the digital camera 1 based on the motion vector information. There are six types of movement directions to be determined: up, down, left, right, forward, and backward. The camerawork determination unit 122b stores information indicating the determined movement direction in the RAM 101. Details of the operation of S402 will be described later with reference to FIG. 7.

[0045] In S403, the camerawork determination unit 122b (first acquisition means) acquires the detection results of the subject information detection unit 141 from, for example, the RAM 101. It is assumed that the RAM 101 stores the detection results of the subject information detection unit 141 for at least the most recent multiple frames (for example, 30 frames).

[0046] In S404, the camerawork determination unit 122b (second acquisition means) acquires motion information of the digital camera 1. The motion information may be a signal output by the device shake detection unit 123, for example.

[0047] In S405, the camerawork determination unit 122b executes camerawork determination processing using the movement direction of the digital camera 1 determined in S402, the subject detection result acquired in S403, and the movement information acquired in S404. The camerawork determination processing will be described in detail later. The camerawork determination unit 122b stores information indicating the type of camerawork determined in, for example, RAM 101.

[0048] The camerawork determination unit 122b performs the above operation for every predetermined number of frames (for example, every frame). Then, the shooting information data generation unit 122 includes information indicating the camerawork in the shooting information data as camerawork information 224. The shooting information data 22 is associated with data on the frames that make up the video data 21. Therefore, by referencing the video file, it is possible to know the camerawork used when shooting a video. For example, by presenting the total number of times or total time for each type of camerawork used when shooting the video data 21, it becomes possible to understand the frequency and habits of the camerawork used by the photographer when shooting.

[0049] Next, the camera movement direction determination process executed by the camera work determination unit 122b in S402 of FIG. 4 will be described in detail with reference to the flowchart shown in FIG.

[0050] In S701, the camerawork determination unit 122b initializes the reference motion vector detection position N to 0. The reference motion vector detection position N is information that identifies one of the 64 areas 511 for detecting motion vectors, as described in FIG. 5. The motion vector detected for the area identified by the reference motion vector detection position N is set as the reference motion vector. The reference motion vector detection position N is an integer from 0 to 63, for example, where the area in the upper left corner corresponds to 0, the area one to the right corresponds to 1, the area in the upper right corner corresponds to 7, ..., the area in the lower left corner corresponds to 56, and the area in the lower right corner corresponds to 63.

[0051] In S702, the camerawork determination unit 122b determines the intersection point between the direction of the reference motion vector and the direction of each of the other motion vectors. Then, based on the position of the intersection point and the direction of the reference motion vector, it controls the value of a counter provided for each movement direction to be determined. Details of the operation will be described later. The counter may be, for example, a variable stored in the RAM 101.

[0052] In S703, the camerawork determination unit 122b associates the direction corresponding to the counter with the largest value among the counters for each direction stored in RAM 101 with the value of N as a candidate movement direction and stores it in, for example, RAM 101. For example, if the counter value for the movement direction "left" is the largest for N=1, the camerawork determination unit 122b associates "left" with N=1 as a candidate movement direction and stores it in RAM 101. Details of the operation will be described later.

[0053] In S704, the camerawork determination unit 122b increments the value of N by +1.

[0054] In S705, the camerawork determination unit 122b determines whether the value of N is equal to or greater than the total number of motion vectors (=64). If the camerawork determination unit 122b determines that the value of N is equal to or greater than the total number of motion vectors, it executes S706, and if not, it executes S702.

[0055] In S706, the camerawork determination unit 122b determines the candidate movement direction associated with each value (0 to 63) of N to be the movement direction of the digital camera 1, which has the highest frequency. The camerawork determination unit 122b stores the value representing the determined movement direction in, for example, the RAM 101.

[0056] Next, details of S702 will be described with reference to the flowchart shown in FIG. In S801, the camerawork determination unit 122b initializes the motion vector detection position n to be processed to 0. The motion vector detection position n is information that identifies one of the 64 motion vector detection positions other than the detection position of the reference motion vector. Therefore, like the reference motion vector detection position N, the motion vector detection position n can be an integer between 0 and 63, but values ​​equal to N are excluded.

[0057] In S802, the camerawork determination unit 122b initializes all camera movement direction counters provided for each movement direction to be determined to 0. The camera movement direction counters can be realized by any configuration that can be used as a counter, such as a variable stored in the RAM 101. Here, as an example, the movement directions to be determined are assumed to be six directions: "leftward movement," "rightward movement," "backward movement," "forward movement," "upward movement," and "downward movement."

[0058] In S803, the camera work determination unit 122b calculates the intersection point between the reference motion vector N and the motion vector n. The method for calculating the intersection point will be explained using Figures 9 and 10. Figure 9 shows 64 regions into which the entire frame is divided, and some of the motion vectors detected for each region.

[0059] Here, it is assumed that the reference motion vector detection position N is 0, i.e., the motion vector detected in the upper left corner area is the reference motion vector. In this case, the processing from S803 onwards is repeatedly executed for the motion vector detection position n in the range of 1 to 63.

[0060] In FIG. 9, the direction of the base motion vector is indicated by a solid line, and the directions of the other motion vectors are indicated by dotted lines. In S803, the intersection point between the direction of the base motion vector (solid line) and the directions of the other motion vectors (dotted line) is calculated. In FIG. 9, the calculated intersection points are indicated by black circles (●). To avoid complexity, FIG. 9 only shows the intersection points calculated for the base motion vector and motion vectors where n=1, 2, 7, 8, 56, and 63, but in reality, intersection points are calculated sequentially for motion vectors where n=1 to 63. Hereinafter, the straight line indicating the direction of the vector will be referred to as the extension line of the vector. Therefore, the processing of S803 can also be said to be processing for calculating the intersection point between the extension line of the base motion vector and the extension lines of the other motion vectors.

[0061] A method for calculating the intersection of the extensions of vectors will be described using Figure 10. The intersection of the extensions of two vectors can be found by solving simultaneous equations consisting of linear functions that represent the extensions of each vector.

[0062] For example, in Figure 10, let y = ax + b be the extension of the base motion vector, and y = Ax + B be the extension of another motion vector. If the coordinates of the start and end points of the base motion vector are (p, q) and (r, s), then a = (sq) / (rp) and b = q - ap. Also, if the coordinates of the start and end points of another motion vector are (P, Q) and (R, S), then A = (SQ) / (RP) and B = Q - AP. In this case, the intersection C(t, u) of the straight lines of the two extensions can be calculated as (t, u) = ((Bb) / (aA), (aB - Ab) / (aA)). Because a, b, A, and B can all be found from coordinate values, the intersection can be calculated directly from the coordinate values.

[0063] The motion vector information is the image coordinates of the start and end points of the motion vector detection area, with the reference coordinates (e.g., center coordinates) of the motion vector detection area being the start point. Therefore, in S803, the camerawork determination unit 122b calculates the intersection points of the extension lines of the reference motion vector and another motion vector based on the coordinate values ​​of the reference motion vector and another motion vector.

[0064] From S804 onwards, a process is carried out to increase the counter value for each direction according to the combination of the position of the intersection point in the frame and the direction (sign) of the reference motion vector. In this embodiment, a plurality of predetermined determination areas are set in the image, and one counter value is increased according to the combination of the determination area including the position of the intersection point and the direction (sign) of the reference motion vector. FIG. 11 shows an example of the determination area settings. Here, it is assumed that a total of five determination areas (1) to (5) are set, one each in the peripheral areas above, below, left, and right, and one in the center of the image.

[0065] No judgment area is set in the shaded area. In addition, the boundary areas of the judgment area set in the peripheral area (the rectangular areas indicated by lines for judgment areas (1) and (4)) may be excluded from the judgment area or may be set as separate judgment areas. Judgment areas (1) and (2) are set to correspond to the left-right direction, (3) to the front-back direction, and (4) and (5) to the up-down direction. The setting information for the judgment area is assumed to be stored in, for example, ROM 102.

[0066] 8, in S804, the camerawork determination unit 122b determines whether the intersection calculated in S803 is included in the determination area (1) or (2). If the camerawork determination unit 122b determines that the intersection is included in the determination area (1) or (2), it executes S805, and if not, it executes S806.

[0067] In S805, the camerawork determination unit 122b determines whether the sign of the horizontal component of the reference motion vector is "positive." Here, it is assumed that movement to the right and upward in the image is "positive," and movement to the left and downward is "negative." Therefore, the camerawork determination unit 122b determines the sign of the horizontal component to be "positive" if the horizontal (x) coordinates of the start and end points of the reference motion vector increase, and determines the sign of the horizontal component to be "negative" if they decrease. If the camerawork determination unit 122b determines that the sign of the horizontal component of the reference motion vector is "positive," it executes S807; if not, it executes S808.

[0068] In S807, the camerawork determination unit 122b increments the value of the leftward movement counter by 1. Thereafter, the camerawork determination unit 122b executes S818. In S808, the camerawork determination unit 122b increments the value of the rightward movement counter by 1. Thereafter, the camerawork determination unit 122b executes S818.

[0069] In S806, the camerawork determination unit 122b determines whether the intersection calculated in S803 is included in the determination area (3). If the camerawork determination unit 122b determines that the intersection is included in the determination area (3), it executes S809, and if not, it executes S810.

[0070] In S809, the camerawork determination unit 122b determines whether the remainder when the reference motion vector detection position N is divided by 8 is 3 or less (N mod 8≦3). If it is determined that N mod 8≦3, the camerawork determination unit 122b executes S811, and if not, executes S813. Because S809 is a determination as to whether the starting point of the reference motion vector is in the left half or the right half of the image, the determination may be made by other methods. If it is determined that the starting point of the reference motion vector is in the left half of the image, the camerawork determination unit 122b executes S811, and if not, executes S813.

[0071] In S811, the camerawork determination unit 122b inverts the horizontal component sign of the reference motion vector. Then, the camerawork determination unit 122b executes S813. If N mod 8≦3, the starting point of the reference motion vector is located in the left half of the image. Because the determination area (3) corresponds to the front-to-back direction, the motion vector moves outward from the center of the image, or from the outside to the center of the image. In this case, the sign of the horizontal component of motion vectors corresponding to the same movement direction is reversed between motion vectors whose starting points are located in the left half of the image and motion vectors whose starting points are located in the left half of the image. Therefore, in S811, the horizontal component sign of the reference motion vector whose starting point is located in the left half of the image is inverted.

[0072] In S813, the camerawork determination unit 122b determines whether the sign of the horizontal component of the reference motion vector is "positive." If the camerawork determination unit 122b determines that the sign of the horizontal component of the reference motion vector is "positive," it executes S814, and if not, it executes S815.

[0073] In S814, the camera work determination unit 122b increments the value of the backward movement counter by 1. Thereafter, the camera work determination unit 122b executes S818. In S815, the camera work determination unit 122b increments the value of the forward movement counter by 1. Thereafter, the camera work determination unit 122b executes S818.

[0074] While the horizontal component of the reference motion vector is used to determine the forward / backward direction here, the vertical component may also be used. In this case, in S809, it is determined whether the starting point of the reference motion vector is in the upper half or the lower half of the image. This determination can be realized, for example, by determining whether the reference motion vector detection position N is greater than the total number of motion vector detection areas (64) / 2 (N>32). If the camerawork determination unit 122b determines that N>32, it executes S811; otherwise, it executes S813. Then, in S811, the camerawork determination unit 122b inverts the sign of the vertical component of the reference motion vector. In S813, it determines whether the sign of the vertical component of the reference motion vector is "positive," and if it is determined that the sign is "positive," it executes S814; otherwise, it executes S815.

[0075] In S810, the camerawork determination unit 122b determines whether the intersection calculated in S803 is included in the determination area (4) or (5). If the camerawork determination unit 122b determines that the intersection is included in the determination area (4) or (5), it executes S812, and if not, it executes S818.

[0076] In S812, the camerawork determination unit 122b determines whether the sign of the vertical component of the reference motion vector is "positive." If the camerawork determination unit 122b determines that the sign of the vertical component of the reference motion vector is "positive," it executes S816, and if not, it executes S817.

[0077] In S816, the camerawork determination unit 122b increments the value of the upward movement counter by 1. Thereafter, the camerawork determination unit 122b executes S818. In S817, the camerawork determination unit 122b increments the value of the downward movement counter by 1. Thereafter, the camerawork determination unit 122b executes S818.

[0078] In S818, the camerawork determination unit 122b increments n by 1. If the value of n after incrementing by 1 is the same as N, the camerawork determination unit 122b further increments n by 1. Thereafter, the camerawork determination unit 122b executes S819.

[0079] In S819, the camerawork determination unit 122b determines whether the value of n is equal to or greater than the total number of motion vector detection areas (here, 64). If the camerawork determination unit 122b determines that the value of n is equal to or greater than the total number of motion vector detection areas, it ends the operation shown in Fig. 8 and executes S703, and if not, it executes S803.

[0080] The processing described using Figure 8 is based on the relationship between the position of the intersection, the starting position of the reference motion vector, the sign of the horizontal or vertical component of the reference motion vector, and the estimated movement direction of the digital camera 1, as shown in Figure 12.

[0081] (Camera work determination processing) FIG. 13 is a flowchart showing the details of the camera work determination process performed by the camera work determination unit 122b in S405 of FIG. In S1301, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process (S402) is facing left or right. If the camera work determination unit 122b determines that the camera movement direction is facing left or right, it executes S1302, and if not, it executes S1303.

[0082] In S1302, the camerawork determination unit 122b executes pan / circle / dolly determination processing. The camerawork determination unit 122b stores a value indicating the determination result in, for example, the RAM 101, and ends the camerawork determination processing.

[0083] In S1303, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process (S402) is upward or downward. If the camera work determination unit 122b determines that the camera movement direction is upward or downward, it executes S1304, and if not, it executes S1305.

[0084] In S1304, the camerawork determination unit 122b executes tilt / elevator determination processing. The camerawork determination unit 122b stores a value indicating the determination result in, for example, the RAM 101, and ends the camerawork determination processing.

[0085] In S1305, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process of S402 is forward or backward. If the camera work determination unit 122b determines that the camera movement direction is forward or backward, it executes S1306, and if not, it ends the camera work determination process without determining the camera work.

[0086] In S1306, the camerawork determination unit 122b executes follow / lead / push-in / pull-out determination processing. The camerawork determination unit 122b stores a value indicating the determination result in, for example, the RAM 101, and ends the camerawork determination processing.

[0087] FIG. 14 is a flowchart showing the details of the pan / circle / dolly discrimination process in S1302. In S1401, the camerawork determination unit 122b determines whether the absolute value of the angular velocity (rad / s) of rotation around the yaw axis (around the z axis) from the movement information of the digital camera 1 acquired from the device vibration detection unit 123 in S404 is equal to or greater than a predetermined value. The predetermined value may be a fixed value or a changeable value. It is assumed that the predetermined value is stored in advance in the ROM 102. If the camerawork determination unit 122b determines that the absolute value of the yaw angular velocity is equal to or greater than the predetermined value, it executes S1402, and if not, it executes S1403.

[0088] In S1402, the camera work determination unit 122b determines whether the sign of the yaw angular velocity is negative. If the camera work determination unit 122b determines that the sign of the yaw angular velocity is negative, it executes S1404, and if not, it executes S1405.

[0089] In S1404, the camerawork determination unit 122b determines whether or not a subject has been detected near the center of the image based on the subject information acquired in S403. For example, if the distance between the position of the subject area included in the subject information and the center coordinates of the image is within a predetermined value, the camerawork determination unit 122b determines that a subject has been detected near the center of the image. It is assumed that the predetermined value is stored in ROM 102 in advance. If the camerawork determination unit 122b determines that a subject has been detected near the center of the image, it executes S1406; otherwise, it executes S1409. Note that the camerawork determination unit 122b may determine that a subject has been detected near the center of the image if the distance between the position of the subject area and the center coordinates of the image is within a predetermined value and the subject area includes the center coordinates of the image.

[0090] In S1406, the camerawork determination unit 122b determines whether the magnitude of the motion vector detected for the subject area is less than a predetermined value based on the motion vector information acquired in S401. The predetermined value is assumed to be stored in advance in ROM 102. For example, if the sum of squares (or the square root of the sum of squares) of the horizontal and vertical components of the motion vector detected for the subject information is less than a predetermined value, the camerawork determination unit 122b determines that the magnitude of the motion vector is less than the predetermined value. If the camerawork determination unit 122b determines that the magnitude of the motion vector is less than the predetermined value, it executes S1408; if not, it executes S1409.

[0091] In S1408, the camerawork determination unit 122b determines the camerawork to be “circle (clockwise).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the pan / circle / dolly determination process.

[0092] In S1409, the camerawork determination unit 122b determines the camerawork to be “pan (right to left).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the pan / circle / dolly determination process.

[0093] In S1405, the camerawork determination unit 122b determines whether or not a subject is detected near the center of the image, similar to S1404. If it is determined that a subject is detected near the center of the image, the camerawork determination unit 122b executes S1407, and if not, it executes S1411.

[0094] In S1407, the camerawork determination unit 122b determines whether the magnitude of the motion vector detected for the subject area is less than a predetermined value, as in S1406. If the camerawork determination unit 122b determines that the magnitude of the motion vector is less than the predetermined value, it executes S1410, and if not, it executes S1411.

[0095] In S1410, the camerawork determination unit 122b determines the camerawork to be “circle (counterclockwise).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the pan / circle / dolly determination process.

[0096] In S1411, the camerawork determination unit 122b determines the camerawork to be “pan (left to right).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the pan / circle / dolly determination process.

[0097] In S1403, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process of S402 is facing right. If the camera work determination unit 122b determines that the camera movement direction is facing right, it executes S1412, and if not, it executes S1413.

[0098] In S1412, the camerawork determination unit 122b determines the camerawork to be “dolly (left to right).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the pan / circle / dolly determination process.

[0099] In S1413, the camerawork determination unit 122b determines the camerawork to be “dolly (right to left).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the pan / circle / dolly determination process.

[0100] FIG. 15 is a flowchart showing the details of the tilt / elevator discrimination process in S1304. In S1501, the camerawork determination unit 122b determines whether the absolute value of the angular velocity (rad / s) of rotation around the pitch axis (around the y-axis) from the movement information of the digital camera 1 acquired from the device vibration detection unit 123 in S404 is equal to or greater than a predetermined value. The predetermined value may be a fixed value or a changeable value. It is assumed that the predetermined value is stored in advance in the ROM 102. If the camerawork determination unit 122b determines that the absolute value of the angular velocity of the pitch is equal to or greater than the predetermined value, it executes S1502, and if not, it executes S1503.

[0101] In S1502, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process of S402 is upward. If the camera work determination unit 122b determines that the camera movement direction is upward, it executes S1504, and if not, it executes S1505.

[0102] In S1504, the camerawork determination unit 122b determines the camerawork as “tilt (bottom to top).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the tilt / elevator determination process.

[0103] In S1505, the camerawork determination unit 122b determines the camerawork as “tilt (top to bottom).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the tilt / elevator determination process.

[0104] In S1503, the camera work determination unit 122b determines whether the camera movement direction is upward, as in S1502. If the camera work determination unit 122b determines that the camera movement direction is upward, it executes S1506, and if not, it executes S1507.

[0105] In S1506, the camerawork determination unit 122b determines the camerawork as “elevator (bottom to top).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the tilt / elevator determination process.

[0106] In S1507, the camerawork determination unit 122b determines the camerawork to be “elevator (from top to bottom).” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the tilt / elevator determination process.

[0107] FIG. 16 is a flowchart showing details of the follow / lead / push-in / pull-out determination process in S1306. In S1601, the camerawork determination unit 122b acquires subject information from m frames ago, for example, by referring to the RAM 101. m is a plural number, for example, 30. m may differ depending on the frame rate, but may be a value corresponding to, for example, one to several seconds ago. For example, it is assumed that the value of m for 30 frames per second is stored in the ROM 102.

[0108] In S1602, the camera work determination unit 122b determines whether or not a subject is detected near the center of the image, similar to S1404. If it is determined that a subject is detected near the center of the image, the camera work determination unit 122b executes S1604, and if not, it executes S1603.

[0109] In S1603, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process of S402 is forward. If the camera work determination unit 122b determines that the camera movement direction is forward, it executes S1605, and if not, it executes S1606.

[0110] In S1605, the camerawork determination unit 122b determines the camerawork to be “push-in.” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the follow / lead / push-in / pull-out determination process.

[0111] In S1606, the camerawork determination unit 122b determines the camerawork to be “pull-out.” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the follow / lead / push-in / pull-out determination process.

[0112] In S1604, the camerawork determination unit 122b determines whether the change in size of the subject area detected near the center of the image is within a predetermined magnification range, based on the subject information m frames prior acquired in S1601 and the latest subject information acquired in S403. The predetermined magnification range here is a range of approximately 1x to determine that the change is small, and may be, for example, a range of 0.85x to 1.15x. If the camerawork determination unit 122b determines that the change in size of the subject area detected near the center of the image is within the predetermined magnification range, it executes S1607; if not, it executes S1608.

[0113] In S1607, the camera work determination unit 122b determines whether the camera movement direction obtained by the camera movement direction determination process is forward, as in S1603. If the camera work determination unit 122b determines that the camera movement direction is forward, it executes S1609, and if not, it executes S1610.

[0114] In S1609, the camerawork determination unit 122b determines the camerawork to be “following.” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the follow / lead / push-in / pull-out determination process.

[0115] In S1610, the camerawork determination unit 122b determines the camerawork to be “lead.” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the follow / lead / push-in / pull-out determination process.

[0116] In S1608, the camerawork determination unit 122b determines whether the size of the subject area detected near the center of the image has increased, based on the subject information m frames ago acquired in S1601 and the latest subject information acquired in S403. If the camerawork determination unit 122b determines that the size of the subject area detected near the center of the image has increased, it executes S1611; if not, it executes S1612.

[0117] In S1611, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process is forward, as in S1607. If the camera work determination unit 122b determines that the camera movement direction is forward, it executes S1613, and if not, it ends the follow / lead / push-in / pull-out determination process without determining the camera work.

[0118] In S1613, the camerawork determination unit 122b determines the camerawork to be “push-in.” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the follow / lead / push-in / pull-out determination process.

[0119] In S1612, the camera work determination unit 122b determines whether the camera movement direction obtained in the camera movement direction determination process of S402 is backward. If the camera work determination unit 122b determines that the camera movement direction is backward, it executes S1614, and if not, it ends the follow / lead / push-in / pull-out determination process without determining the camera work.

[0120] In S1614, the camerawork determination unit 122b determines the camerawork to be “pull-out.” The camerawork determination unit 122b stores information specifying the determined camerawork in, for example, the RAM 101, and ends the follow / lead / push-in / pull-out determination process.

[0121] As described above, according to this embodiment, the camerawork used to capture a video is determined by taking into consideration not only the motion vectors obtained from the video but also the camera movement and subject area information used to capture the video, making it possible to determine a wider variety of camerawork types and in greater detail than ever before.

[0122] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In this embodiment, in a camera system having a gimbal mechanism, camera work is determined based on the amount of control of the gimbal and the detected amount of rotation angle.

[0123] FIG. 18 is a diagram showing an example of the appearance of a camera integrated with a gimbal mechanism (gimbal camera) according to this embodiment. FIG. 18(A) is a rear view, and FIG. 18(B) is a side view. The gimbal camera has an imaging unit 180, a grip unit 182, and a gimbal unit 183. An angular velocity meter 181 similar to device vibration detection unit 123 is provided inside imaging unit 180. Note that instead of the gimbal camera, a digital camera separate from the gimbal mechanism (including the grip unit) may be attached to the gimbal mechanism.

[0124] Grip unit 182 is the main body of the gimbal camera that is held by the photographer or attached to a tripod, and is equipped with operation members, display members, etc. Gimbal unit 183 is composed of a three-axis rotation mechanism (roll / pitch / yaw) and an arm that connects it, and prevents shaking that occurs in grip unit 182 from being transmitted to imaging unit 180 by inertial forces generated in the mechanical mechanism. Furthermore, shaking that could not be suppressed by the mechanical mechanism is detected by angular velocity meter 181, and is suppressed by driving a motor arranged in the rotation mechanism in accordance with the detected shaking, enabling shooting with even more suppressed shaking.

[0125] When vibration suppression control is performed by the gimbal unit 183, the output of the angular velocity meter 181 is fed back at a high-speed sampling period to drive the motor of the rotation mechanism. Therefore, when vibration suppression is stable, the output of the angular velocity meter 181 arranged in the imaging unit 180 is close to 0. Therefore, it is difficult to determine from the output of the angular velocity meter 181 whether the grip unit 182 is shaking.

[0126] On the other hand, since the gimbal section 183 drives the rotation mechanism so as to cancel out shaking of the grip section 182, it is possible to determine whether or not the grip 182 is shaking by observing the control amount of the rotation mechanism of the gimbal section 183. It is also possible to determine whether or not the grip section 182 is shaking by observing the rotation angle of the motor driven by the control of the gimbal section 183 with a sensor such as an encoder.

[0127] In this embodiment, camerawork during video shooting with a gimbal camera is determined based on the control amount for each axis of the rotation mechanism of the gimbal unit 183 or the rotation angle of the motor. In this embodiment, too, it is not necessary to determine camerawork during shooting. If the control amount for each axis of the rotation mechanism of the gimbal unit 183 or the rotation angle of the motor during shooting is recorded in association with the video data, it is possible to determine camerawork using the video file after recording.

[0128] 19 is a block diagram showing an example of the functional configuration related to video recording, along with the processing flow, of a gimbal camera 200 as an example of an image processing device according to this embodiment. In FIG. 19, functional blocks similar to those in the digital camera 1 are given the same reference numerals as in FIG. 1. Furthermore, the configuration shown in FIG. 18 is given the same reference numerals as in FIG. 18.

[0129] The gimbal control unit 191 of the gimbal unit 19 acquires an angular velocity, which is an example of a signal representing the movement of the grip unit 182, from the device shake detection unit 123. Then, the gimbal control unit 191 calculates a gimbal control amount that cancels out the movement of the grip unit 182 based on the angular velocity. The gimbal control unit 191 drives the motor of the rotation mechanism 193 based on the calculated gimbal control amount. This reduces the effect that the movement of the grip unit 182 has on the attitude of the imaging unit 11, making it possible to achieve vibration reduction of videos captured by the imaging unit 11.

[0130] The gimbal angle detection unit 192 detects the rotation angle of a motor disposed on each axis of the rotation mechanism 193. The gimbal angle detection unit 192 detects the rotation angle of the motor using, for example, a magnetic encoder that detects the rotation angle of a magnet attached to the motor. The rotation angle of each axis detected by the gimbal angle detection unit 192 can be used to calculate the relative attitude of the imaging unit 180 with respect to the grip unit 182. The relative attitude of the imaging unit 180 can also be used to control the gimbal mode. Examples of gimbal modes include a lock mode that maintains the imaging unit 180 in a horizontal direction regardless of the direction in which the grip unit 182 is moved, and a follow mode that follows the movement of the grip unit 182. By changing the gimbal mode depending on the scene and subject, the photographer can capture images with appropriate camerawork while suppressing camera shake.

[0131] 20(A) shows a schematic diagram of a situation in which a photographer holding a gimbal camera 200 follows a subject 2000 from behind as the subject 2000 moves while turning right. In such a situation, the direction of the camerawork intended by the photographer may not be correctly determined based on the direction of the motion vector.

[0132] FIG. 20(B) shows an example of motion vectors detected for one frame 2010 of a video shot under the circumstances shown in FIG. 20(A). A leftward motion vector is detected for each motion vector detection area 2011. If a leftward motion vector is detected, the camera movement direction determination process described with reference to FIG. 7 determines that the movement is "rightward movement." As a result, the camerawork determination process (FIG. 13) determines that S1301 is YES, and determines that the camerawork is pan, circle, or dolly, regardless of whether follow shooting is being performed.

[0133] Similarly, if you perform follow photography while climbing stairs, a downward motion vector will be detected. As a result, the camera movement direction determination process will determine this as "upward movement," and the camerawork determination process will determine this as either tilt or elevator camerawork.

[0134] In this embodiment, in addition to the motion vector information, the control amount of the rotation mechanism calculated by the gimbal control unit 191 or the rotation angle of the motor of the rotation mechanism detected by the gimbal angle detection unit 192 is also taken into consideration. This makes it possible to correctly determine the camerawork even when the photographer is taking follow shots while moving. The specific operation will be described later.

[0135] Fig. 21(B) shows a schematic diagram of a situation in which a photographer is performing follow shooting or push-in shooting while chasing a subject in front of him / her using gimbal camera 200. Fig. 21(A) shows the change over time in the amount of gimbal control around the yaw, pitch, and roll axes in the shooting situation shown in Fig. 21(B).

[0136] When capturing an image while chasing a subject in front (moving forward), the grip portion 182 of the gimbal camera 200 mainly vibrates in the up and down direction. Therefore, the amplitude of the gimbal control amount in the pitch direction is larger than that of the roll and yaw directions. Therefore, when the camera movement direction is determined to be leftward or rightward, if the amplitude of the pitch control amount is larger than that of the roll and yaw control amounts, it can be determined that the camera is following / leading / push-in / pull-out.

[0137] Furthermore, by using a common technique such as FFT to analyze the frequency of gimbal control to suppress shaking when a cameraman is walking forward or backward while filming, it is possible to detect waveforms with frequencies similar to those of walking movements. The frequencies detected for walking movements are relatively higher than those detected for camerawork that involves large upward or downward camera movements, such as tilting or elevator movements. Therefore, a frequency threshold can be set in advance to distinguish between the frequencies of walking movements and those of camerawork such as tilting or elevator movements. Therefore, if the camera movement direction is determined to be upward or downward and a frequency component higher than the frequency threshold is detected as the dominant frequency component of the gimbal control, the camerawork can be determined to be following / leading / push-in / pull-out. The dominant frequency component may be the frequency component with the largest amplitude in the frequency spectrum or above a threshold.

[0138] Figure 22 is a flowchart showing details of the camerawork determination process performed by the camerawork determination unit 122b in this embodiment. In Figure 22, the steps that perform the operations described in Figure 13 are given the same reference numerals as in Figure 13, and their description will be omitted. In this embodiment, S2201 is added between S1301 and S1302, and S2202 is added between S1303 and S1304.

[0139] In S1301, if the camera work determination unit 122b determines that the camera movement direction is leftward or rightward, it executes S2201, and if not, it executes S1303.

[0140] In S2201, the camerawork determination unit 122b compares the amplitudes of the gimbal control amount for each of the roll, pitch, and yaw axes calculated by the gimbal control unit 191, and determines whether the amplitude of the gimbal control amount for the pitch axis is the maximum. If the camerawork determination unit 122b determines that the amplitude of the gimbal control amount for the pitch axis is the maximum, it executes S1306, and if not, it executes S1302.

[0141] Furthermore, in S1303, if the camera work determination unit 122b determines that the camera movement direction is upward or downward, it executes S2202, and if not, it executes S1305.

[0142] In S2202, the camerawork determination unit 122b performs frequency analysis on the gimbal control amount for the pitch axis calculated by the gimbal control unit 191 over the most recent predetermined period. Then, the camerawork determination unit 122b determines whether the frequency of the gimbal control amount for the pitch axis is equal to or greater than a predetermined frequency threshold. If the camerawork determination unit 122b determines that the frequency of the gimbal control amount for the pitch axis is equal to or greater than the predetermined frequency threshold, it executes S1306; if not, it executes S1304.

[0143] In this way, the camerawork determination unit 122b of this embodiment determines whether the photographer is walking while shooting based on the gimbal control amount. Then, in addition to the determination result of the camera movement direction based on the motion vector, the camerawork is determined based on whether it is determined that the photographer is shooting while walking. Therefore, it is possible to appropriately determine the camerawork even for videos shot while walking.

[0144] 22 has described a case where it is determined whether or not shooting is being performed while walking based on the gimbal control amount. However, as described above, it is also possible to use the rotation angle of the rotation mechanism 183 detected by the gimbal angle detection unit 192. In this case, it is sufficient to determine in S2201 whether or not the rotation angle of the pitch axis is at its maximum, and to determine the frequency of the rotation angle over the most recent predetermined period in S2202.

[0145] Furthermore, whether the gimbal control amount or the rotation angle of the rotation mechanism is used, it may also be determined whether the frequency in the roll direction or yaw direction is equal to or greater than the frequency threshold. For example, if it is determined in S2202 that the frequency in the roll direction and yaw direction is less than the frequency threshold and that the frequency in the pitch direction is equal to or greater than the frequency threshold, S1306 is executed. This can increase the reliability of the determination.

[0146] Furthermore, it is possible to dynamically change the frequency threshold or the axis for determining the frequency, or to set multiple frequency thresholds, depending on the inclination of the grip portion 182. Furthermore, the frequency analysis method is not limited to a conversion into the frequency domain such as FFT, and simple analysis methods such as measuring the time of zero crossing or the time when the amount of change is reversed may also be used.

[0147] ●(Third embodiment) Next, a third embodiment of the present invention will be described. In this embodiment, the accuracy of the camerawork determination process described in the first embodiment is improved. This embodiment can be implemented by the digital camera 1 described in the first embodiment. Therefore, the details of this embodiment will be described below using the components shown in the block diagrams of FIGS. 1 to 3.

[0148] FIG. 23(A) is a diagram showing an example of motion vectors detected when a subject moving diagonally upward while climbing stairs or the like is photographed by following the camera. When such a motion vector is detected, the intersection of the motion vectors falls within determination area (4) in FIG. 11, and the vector sign is negative. Therefore, the camera movement direction determination process described in FIG. 7 determines the movement as "upward movement." As a result, the camerawork determination process in FIG. 13 determines YES in S1303, and determines that the camerawork is either tilt or elevator, even though follow photography is being performed.

[0149] 23(B) is a diagram showing an example of a motion vector detected when a subject is shot diagonally in front and moving forward, and when such a motion vector is detected, it is determined to be a pan / circle / dolly camerawork, even though follow shooting is being performed.

[0150] In this embodiment, the camerawork for such images captured while moving in a diagonal direction relative to the optical axis direction of the camera can be correctly determined by taking into account the angular distribution of the motion vectors.

[0151] 24 is a flowchart showing details of the camera movement direction determination process performed by the camerawork determination unit 122b in this embodiment. In S402 of FIG. 4, the camerawork determination unit 122b can perform S2401 to S2405, which will be described below, instead of S701 to S706 shown in FIG.

[0152] In S2401, the camerawork determination unit 122b excludes vectors that are smaller than a predetermined magnitude from the multiple motion vectors included in the motion vector information for the entire frame acquired in S401 of Fig. 4. By not taking into account motion vectors that are smaller than a predetermined magnitude when determining the camera movement direction, the influence of small movements such as camera shake on the determination result of the camera movement direction is suppressed.

[0153] In S2402, the camerawork determination unit 122b calculates a histogram of angle distribution for the motion vectors that were not excluded in S2401. The width of each bin in the histogram is assumed to be predetermined as a divisor of 360 (3 or greater).

[0154] 25 is a diagram showing the angles of motion vectors in this embodiment. In this embodiment, the angle of a motion vector has a range of ±180°, with the horizontal rightward angle in the image being the reference (0°), counterclockwise angles being positive, and clockwise angles being negative.

[0155] Fig. 26(A) shows a histogram of the angular distribution of the motion vectors shown in 504 of Fig. 5. The angular histogram of the motion vectors detected from images captured with camerawork in which the camera moves in up, down, left, and right directions perpendicular to the optical axis before movement shows prominent peaks corresponding to the movement direction.

[0156] On the other hand, Figures 26(B) and 26(C) show angle histograms of motion vectors detected from images captured by moving the camera diagonally relative to the optical axis direction before movement, as in Figures 23(A) and 23(B). The angle histograms of motion vectors detected from images captured with camerawork in which the camera is moved diagonally forward and backward relative to the optical axis direction before movement have a relatively wide frequency distribution and do not show any significant peaks.

[0157] As such, there is a significant relationship between the movement direction and the magnitude of the peaks in the angle histogram. Therefore, depending on whether the ratio of the maximum frequency (maximum frequency) to the total frequency in the angle histogram exceeds a predetermined value X (%), it is possible to determine whether the movement direction of the camera is perpendicular to the optical axis (up / down / left / right directions) or not (front / back (diagonal) directions).

[0158] The maximum frequency may be the frequency of one bin, or the sum of the frequencies of the bin with the maximum frequency and multiple bins adjacent to it that have a frequency above a certain level (for example, above the average frequency). A maximum number of bins (for example, 3 to 5) may be set for the sum of frequencies. Hereinafter, the terms "maximum frequency" and "maximum frequency" are used to refer to both the frequency of one bin and the sum of the frequencies of multiple bins.

[0159] In S2403, the camerawork determination unit 122b calculates the ratio of the maximum frequency (maximum frequency) to the total frequency in the angle histogram as a relative frequency. The relative frequency can be calculated as maximum frequency / total frequency, or maximum frequency / total frequency. The camerawork determination unit 122b then determines whether the calculated relative frequency exceeds a predetermined threshold value X (%). X (%) can be determined experimentally in advance, for example, and stored in, for example, the ROM 102.

[0160] If it is determined that the relative frequency exceeds X (%), the camerawork determination unit 122b executes S2404, and if not, executes S2405.

[0161] In S2404, the camera work determination unit 122b determines that the camera movement direction is the up / down / left / right direction perpendicular to the optical axis, and executes the corresponding camera movement direction determination process.

[0162] In S2405, the camera work determination unit 122b determines that the movement direction of the camera is in the forward / backward (diagonal) direction with respect to the optical axis, and executes the corresponding camera movement direction determination process.

[0163] Next, the camera movement direction determination process executed in S2404 will be described in detail using the flowchart shown in Fig. 27. As described above, when the camera movement direction is in the up, down, left, or right direction perpendicular to the optical axis, the angle histogram of the motion vectors detected from the captured image has a prominent peak. Since this peak appears at an angle corresponding to the movement direction, the camera movement direction can be determined by determining the angle with the highest frequency in the angle histogram.

[0164] The camerawork determination unit 122b determines the angle with the highest frequency in the angle histogram in steps S2701 to S2704, and specifies the camera movement direction as right, up, left, or down in steps S2705 to S2708 according to the determined angle.

[0165] Specifically, the camerawork determination unit 122b determines whether the angle θ (−180°≦θ<180°) with the highest frequency in the angle histogram is If -180°≦θ<-135° (S2701, Yes), turn right (S2705). If -135°≦θ< -45° (S2702, Yes), then move upward (S2706). If -45°≦θ< 45° (S2703, Yes), then move left (S2707). If 45°≦θ< 135° (S2704, Yes), then downward (S2708). If 135°≦θ< 180° (S2704, No), then turn right (S2705); The camera movement direction is determined as follows.

[0166] The angle θ with the highest frequency can be a representative value (for example, a median value) of the angle range corresponding to one or more bins for which the highest frequency is found.

[0167] Note that S2404 is executed when it is determined that the camera movement direction is up, down, left, or right. Therefore, in S2404, the camera movement direction may be identified based on the intersection of the motion vectors, as in the first embodiment.

[0168] Next, the camera movement direction determination process executed in S2405 will be described in detail with reference to FIGS.

[0169] In S2801 of Figure 28, the camerawork determination unit 122b determines the direction of a motion vector for each of multiple specific partial areas in a frame. The multiple specific partial areas are, for example, partial areas 3100 to 3103 shown in Figure 31. In this embodiment, for convenience, of the 64 areas obtained by dividing the entire frame into eight equal parts in each of the horizontal and vertical directions, 3 x 3 areas located in the four corners of the upper right, upper left, lower right, and lower left are defined as specific partial areas. Note that the specific partial areas may be set as ranges in which multiple motion vectors are detected within each quadrant of a Cartesian coordinate system whose origin is the center of the image.

[0170] In S2802, the camera work determination unit 122b determines the movement direction of the camera based on the direction of the motion vector determined for the partial region in S2801.

[0171] 29 is a flowchart showing the details of the motion vector direction calculation process in S2801. The camerawork determination unit 122b calculates the direction of the motion vector for each of the four partial areas 3100-3103.

[0172] In S2900, the camerawork determination unit 122b calculates the average angle and the standard deviation (variation) of the angles for the plurality of motion vectors detected in the target partial region, and stores them in the RAM 101, for example.

[0173] In S2901, the camerawork determination unit 122b determines whether or not the standard deviation of the angles calculated in S2900 is less than a predetermined value S. This is a determination as to whether or not a moving subject is included in the target partial region.

[0174] When a moving subject is present in a partial region, as in partial region 3200 in Fig. 32, it is highly likely that the direction of the motion vector caused by the camera movement is disturbed by the motion vector caused by the moving subject in the partial region. Therefore, a partial region with a large standard deviation of angles (greater than or equal to a predetermined value S) can be determined to contain a moving subject. The predetermined value S can be determined in advance, for example, experimentally, and stored in ROM 102.

[0175] If the camerawork determination unit 122b determines that the standard deviation of the angles is less than the predetermined value S, it executes S2903, and if not, it executes S2902.

[0176] In S2902, the camerawork determination unit 122b excludes the motion vector of the target partial region from the processes of S2903 to S2908 so that it does not affect the determination of the camera movement direction, and then executes S2909. In this way, the direction of the motion vector is not determined for the partial region for which S2902 has been executed.

[0177] In S2903 to S2905, the camerawork determination unit 122b determines the left and right directions of the motion vector of the target partial region. In S2906 to S2908, the camerawork determination unit 122b determines the up and down direction of the motion vector of the target partial region. In this way, the direction of the horizontal component and the direction of the vertical component of the motion vector are determined for each partial region.

[0178] Specifically, in S2903, the camerawork determination unit 122b determines whether the average angle θ calculated in S2900 is −90°<θ≦90°, and if it is determined that −90°<θ≦90°, it executes S2904, and if it is not determined that it is, it executes S2905. In S2904, the camerawork determination unit 122b determines that the horizontal component of the motion vector of the target partial region is to the right, and executes S2906. In S2905, the camerawork determination unit 122b determines that the horizontal component of the motion vector of the target partial region is to the left, and executes S2906.

[0179] In S2906, the camerawork determination unit 122b determines whether the average angle θ calculated in S2900 is 0°<θ≦180°, and if it is determined that 0°<θ≦180°, it executes S2907, and if it is not determined that 0°<θ≦180°, it executes S2908. In S2907, the camerawork determination unit 122b determines that the vertical component of the motion vector of the target partial region is upward, and executes S2909. In S2908, the camerawork determination unit 122b determines that the vertical component of the motion vector of the target partial region is downward, and executes S2909.

[0180] In S2909, the camerawork determination unit 122b determines whether processing has been performed for all partial areas, and if it is determined that processing has been performed, terminates processing (executes S2802), and if not, executes processing from S2900 for the unprocessed partial areas.

[0181] 30 is a flowchart showing the details of the camera movement direction determination process in S2802. In the camera movement direction determination process, the camera movement direction is determined based on the direction of each component of the motion vector determined in S2801. Note that partial areas for which the motion vector direction was not determined in S2801 are not taken into consideration in the camera movement direction determination process.

[0182] In S3001, the camerawork determination unit 122b determines whether the horizontal component of the motion vector determined in S2801 is leftward for the partial area set in the left half of the image (partial areas 3100 and 3101 in the example of FIG. 31). If the horizontal component of the motion vector is determined to be leftward, the camerawork determination unit 122b executes S3002; if not, it executes S3004.

[0183] If the motion vector direction has been determined for multiple partial regions set in the left half of the image, the camerawork determination unit 122b determines in S3001 whether all of the horizontal components of the motion vectors determined in S2801 are leftward. Therefore, if there is even one partial region in which the motion vector direction is determined to be a direction other than leftward, the camerawork determination unit 122b executes S3004. If the number of partial regions used for determination is large (a threshold or more), it may be determined whether a predetermined percentage or more (e.g., 80% or more) are determined to be leftward. The same applies to the other direction determination steps (S3002, S3004, S3005, S3007, S3008, S3009, S3010).

[0184] In S3002, the camerawork determination unit 122b determines whether the horizontal component of the motion vector determined in S2801 is rightward for the partial area set in the right half of the image (partial areas 3102 and 3103 in the example of FIG. 31). If the horizontal component of the motion vector is rightward, the camerawork determination unit 122b executes S3003; if not, it executes S3007.

[0185] In S3003, the camerawork determination unit 122b determines that the camera is moving forward. According to this embodiment, even when a motion vector such as that shown in Fig. 23(A) is detected, such as when a subject moving diagonally upward is being followed, the camerawork determination unit 122b can correctly determine that the camera is moving forward.

[0186] In S3004, the camerawork determination unit 122b determines whether the horizontal component of the motion vector determined in S2801 is rightward for the partial area set in the left half of the image (partial areas 3100 and 3101 in the example of FIG. 31). If the camerawork determination unit 122b determines that the horizontal component is rightward, it executes S3005; if not, it executes S3007.

[0187] In S3005, the camerawork determination unit 122b determines whether the horizontal component of the motion vector determined in S2801 is leftward for the partial area set in the right half of the image (partial areas 3102 and 3103 in the example of FIG. 31). If the horizontal component of the motion vector is leftward, the camerawork determination unit 122b executes S3006; if not, it executes S3007.

[0188] In S3006, the camerawork determination unit 122b determines that the camera is moving backward. When taking a lead shot of a subject by retreating diagonally downward, a motion vector in the opposite direction to that shown in Figure 23(A) is detected. Even in this case, according to this embodiment, the camera's moving direction can be correctly determined to be backward.

[0189] In S3007, the camerawork determination unit 122b determines whether the vertical component of the motion vector determined in S2801 is upward for the partial area set in the upper half of the image (partial areas 3100 and 3102 in the example of FIG. 31). If the vertical component is determined to be upward, the camerawork determination unit 122b executes S3008; if not, it executes S3009.

[0190] In S3008, the camerawork determination unit 122b determines whether the vertical component of the motion vector determined in S2801 is downward for the partial areas set in the lower half of the image (partial areas 3101 and 3103 in the example of FIG. 31). If the vertical component is determined to be downward, the camerawork determination unit 122b executes S3003; if not, it executes S3011.

[0191] In this way, even if the vertical component of the motion vector is determined to be upward in the upper half of the image and downward in the lower half, the moving direction of the camera is correctly determined to be forward.

[0192] In S3009, the camerawork determination unit 122b determines whether the vertical component of the motion vector determined in S2801 is downward for the partial area set in the upper half of the image (partial areas 3100 and 3102 in the example of FIG. 31). If the vertical component is determined to be downward, the camerawork determination unit 122b executes S3010, and if not, executes S3011.

[0193] In S3010, the camerawork determination unit 122b determines whether the vertical component of the motion vector determined in S2801 is upward for the partial areas set in the lower half of the image (partial areas 3101 and 3103 in the example of FIG. 31). If the vertical component is determined to be upward, the camerawork determination unit 122b executes S3006; if not, it executes S3011.

[0194] In this way, even if the vertical component of the motion vector is determined to be downward in the upper half of the image and upward in the lower half, the moving direction of the camera is correctly determined to be backward.

[0195] In S3011, the camerawork determination unit 122b determines that the camera movement direction is indefinite. This corresponds to a case where the camera movement direction cannot be determined from the motion vector alone, such as when the camera is stationary or a moving subject is detected across the entire image. In this case, the camerawork can be determined in more detail by determining the output signal of the device shake detection unit 123, etc.

[0196] (Other embodiments) In the above embodiment, a configuration for determining camerawork when shooting (recording) video with a digital camera has been described. However, camerawork determination does not have to be performed during shooting or recording. For example, if information about the movement of the digital camera that shot (recorded) the video data (e.g., the output signal of the device shake detection unit 123) is recorded in association with the video data, camerawork can be determined for recorded video data in the same manner.

[0197] In the above-described embodiment, angular velocities around three axes, namely the yaw axis, pitch axis, and roll axis, were used as movement information of the digital camera. However, angular velocity around the roll axis does not have to be used. Even when angular velocity around the roll axis is not used, the operation in the first embodiment remains unchanged. In S2201 of the second embodiment, it is sufficient to determine whether the gimbal control amount or rotation angle around the pitch axis is greater than the gimbal control amount or rotation angle around the yaw axis.

[0198] In the above-described embodiment, a configuration has been described in which one camerawork type is identified based on a motion vector and motion information. However, a configuration in which multiple camerawork types are identified and a reliability for each type is output may also be used. For example, the reliability can be calculated based on the proportion of candidate movement directions in the first embodiment, the angle histogram of motion vectors in the third embodiment, or the standard deviation of the angles of motion vectors in a partial region. In an application that acquires the identified camerawork, it becomes possible to change the control or display based on the calculated reliability along with the camerawork identification result.

[0199] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0200] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program. (Item 1) a detection means for detecting a motion vector from a video; a first acquisition means for acquiring information about a subject area detected in the moving image; a second acquisition means for acquiring information regarding the movement of the imaging device when capturing the video; a determination means for determining a moving direction of the imaging device based on the motion vector; a determination means for determining camera work of the imaging device when capturing the video based on the movement direction, information on the subject area, and information on the movement; 1. An image processing device comprising: (Item 2) the detection means detects a motion vector for each of a plurality of regions obtained by dividing the entire frame of the moving image; the determining means determines one of the plurality of motion vectors detected by the detecting means as a reference motion vector, obtains candidate movement directions for the plurality of different reference motion vectors, and determines the candidate movement direction with the highest frequency as the movement direction of the imaging device. 2. The image processing device according to item 1, (Item 3) The image processing device described in item 2 is characterized in that the determination means determines candidates for the movement direction based on a combination of the position in the frame of the intersection between the direction of the base motion vector and the direction of another motion vector and the sign of the base motion vector. (Item 4) 4. The image processing device according to item 3, wherein the position of the intersection within the frame is a type of area that includes the intersection among a plurality of areas predetermined for the frame. (Item 5) 5. The image processing device according to item 4, wherein the plurality of regions are predetermined in the center of the frame and peripheral regions above, below, left, and right of the frame. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the information about the movement includes angular velocities about axes of a pitch axis and a yaw axis of the imaging device. (Item 7) When the moving direction is right or left, the determining means the magnitude and sign of the angular velocity about the yaw axis; the position of the subject area, Magnitude of the motion vector in the subject area 7. The image processing device according to item 6, characterized in that the camera work is determined to be circle and pan and its direction based on the above. (Item 8) When the moving direction is right or left, the determining means the magnitude of the angular velocity around the yaw axis; the moving direction 8. The image processing device according to item 6 or 7, characterized in that dolly and its direction are determined as the camera work based on the above. (Item 9) If the imaging device uses a gimbal mechanism, Even if the movement direction is right or left, if the amplitude of the control amount of the gimbal mechanism about the pitch axis or the rotation angle of the gimbal mechanism is larger than the amplitude of the control amount of the gimbal mechanism about the yaw axis or the rotation angle of the gimbal mechanism, the position of the subject area, Change in size of subject area the moving direction 9. The image processing device according to any one of items 6 to 8, wherein the camera work is determined to be a push-in, a pull-out, a follow, or a lead based on the above. (Item 10) When the moving direction is up or down, the determining means the magnitude of the angular velocity about the pitch axis; Whether the movement direction is right or left 10. The image processing device according to any one of items 6 to 9, characterized in that tilt and elevator as the camera work and their directions are determined based on the above. (Item 11) If the imaging device uses a gimbal mechanism, Even if the movement direction is up or down, when a frequency component equal to or greater than a frequency threshold is detected as a main frequency component of the control amount of the gimbal mechanism about the pitch axis or the rotation angle of the gimbal mechanism, the discrimination means the position of the subject area, Change in size of subject area the moving direction Item 11. The image processing device according to item 10, characterized in that the camera work is determined to be a push-in, a pull-out, a follow, or a lead based on the above. (Item 12) When the moving direction is forward or backward, the determining means the position of the subject area, Change in size of subject area the moving direction 10. The image processing device according to any one of items 6 to 9, wherein the camera work is determined to be a push-in, a pull-out, a follow, or a lead based on the above. (Item 13) the detection means detects a motion vector for each of a plurality of regions obtained by dividing the entire frame of the moving image; The determination means If the ratio of the maximum frequency to the total frequency of the angular distribution of the motion vector detected by the detection means exceeds a threshold, the moving direction of the imaging device is determined based on the angle at which the maximum frequency occurs. 2. The image processing device according to item 1, (Item 14) Item 14. The image processing device according to item 13, wherein the determining means determines the moving direction of the imaging device as upward, downward, leftward, or rightward based on the angle with the highest frequency. (Item 15) When the ratio does not exceed the threshold value, the determination means determining a direction of a motion vector in terms of a horizontal component and a vertical component for each of a plurality of partial regions set for the entire frame of the moving image; 15. The image processing device according to item 13 or 14, characterized in that the moving direction of the imaging device is determined to be a forward direction or a backward direction based on the relationship between the positions of the plurality of partial regions and the direction of the determined motion vector. (Item 16) The determination means Item 16. The image processing device according to item 15, characterized in that the moving direction of the imaging device is determined to be forward when the horizontal component of the motion vector determined for a partial region set in the left half of the frame among the plurality of partial regions is leftward and the horizontal component of the motion vector determined for a partial region set in the right half of the frame is rightward, or when the vertical component of the motion vector determined for a partial region set in the upper half of the frame is upward and the vertical component of the motion vector determined for a partial region set in the lower half of the frame is downward. (Item 17) The determination means Item 17. The image processing device according to item 15 or 16, characterized in that the moving direction of the imaging device is determined to be backward when the horizontal component of the motion vector determined for a partial region set in the left half of the frame among the plurality of partial regions is rightward and the horizontal component of the motion vector determined for a partial region set in the right half of the frame is leftward, or when the vertical component of the motion vector determined for a partial region set in the upper half of the frame is downward and the vertical component of the motion vector determined for a partial region set in the lower half of the frame is upward. (Item 18) 18. The image processing device according to any one of items 15 to 17, wherein the determination means does not use, among the plurality of partial regions, partial regions in which the standard deviation of the angles of the motion vectors is equal to or greater than a predetermined value in determining the movement direction of the imaging device. (Item 19) 19. The image processing device according to any one of items 1 to 18, wherein the discrimination means outputs the reliability of each of a plurality of camera works instead of discriminating the camera work of the imaging device. (Item 20) an imaging unit that outputs video; 19. The image processing device according to any one of items 1 to 19, which uses the moving image; and recording means for recording the type of camerawork determined by the determination means in association with the video. (Item 21) An image processing method implemented by an image processing device, Detecting motion vectors from a video; Obtaining information about a subject area detected in the video; acquiring information about the movement of the imaging device when capturing the video; determining a direction of movement of the imaging device based on the motion vector; determining camera work of the imaging device when capturing the video based on the movement direction, information on the subject area, and information on the movement; An image processing method comprising: (Item 22) 20. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 19.

[0201] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0202] 1... digital camera, 11, 180... imaging unit, 14... image processing unit, 141... subject information detection unit, 142... motion vector detection unit, 12... video file generation unit, 122... shooting information data generation unit, 123... device shake detection unit

Claims

1. a detection means for detecting a motion vector from a video; a first acquisition means for acquiring information about a subject area detected in the moving image; a second acquisition means for acquiring information regarding the movement of the imaging device when capturing the video; a determination means for determining a moving direction of the imaging device based on the motion vector; a determination means for determining camera work of the imaging device when capturing the video based on the movement direction, information on the subject area, and information on the movement; 1. An image processing device comprising:

2. the detection means detects a motion vector for each of a plurality of regions obtained by dividing the entire frame of the moving image; the determining means determines one of the plurality of motion vectors detected by the detecting means as a reference motion vector, obtains candidate movement directions for the plurality of different reference motion vectors, and determines the candidate movement direction with the highest frequency as the movement direction of the imaging device.

2. The image processing device according to claim 1, wherein:

3. 3. The image processing device according to claim 2, wherein the determining means determines the candidate movement direction based on a combination of the position in the frame of the intersection between the direction of the reference motion vector and the direction of another motion vector and the sign of the reference motion vector.

4. 4. The image processing apparatus according to claim 3, wherein the position of the intersection within the frame is determined by the type of area that includes the intersection among a plurality of areas that are predetermined for the frame.

5. 5. The image processing device according to claim 4, wherein the plurality of regions are predetermined in the center of the frame and peripheral regions above, below, left and right of the frame.

6. The image processing device according to claim 1 , wherein the information about the movement includes angular velocities about a pitch axis and a yaw axis of the imaging device.

7. When the moving direction is right or left, the determining means the magnitude and sign of the angular velocity about the yaw axis; the position of the subject area, Magnitude of the motion vector in the subject area 7. The image processing device according to claim 6, wherein the camera work includes circle and pan, and the direction of the circle and pan is determined based on the above.

8. When the moving direction is right or left, the determining means the magnitude of the angular velocity around the yaw axis; the moving direction 7. The image processing apparatus according to claim 6, wherein dolly and its direction are determined as the camera work based on the above.

9. If the imaging device uses a gimbal mechanism, Even if the movement direction is right or left, if the amplitude of the control amount of the gimbal mechanism about the pitch axis or the rotation angle of the gimbal mechanism is larger than the amplitude of the control amount of the gimbal mechanism about the yaw axis or the rotation angle of the gimbal mechanism, the position of the subject area, Change in size of subject area the moving direction 7. The image processing apparatus according to claim 6, wherein the camera work is determined to be a push-in, a pull-out, a follow, or a lead based on the above.

10. When the moving direction is up or down, the determining means the magnitude of the angular velocity about the pitch axis; Whether the movement direction is right or left 7. The image processing apparatus according to claim 6, wherein tilt and elevator as the camera work and their directions are determined based on the above.

11. If the imaging device uses a gimbal mechanism, Even if the movement direction is up or down, when a frequency component equal to or greater than a frequency threshold is detected as a main frequency component of the control amount of the gimbal mechanism about the pitch axis or the rotation angle of the gimbal mechanism, the discrimination means the position of the subject area, Change in size of subject area the moving direction 11. The image processing device according to claim 10, wherein the camera work is determined to be a push-in, a pull-out, a follow, or a lead based on the above.

12. When the moving direction is forward or backward, the determining means the position of the subject area, Change in size of subject area the moving direction 7. The image processing apparatus according to claim 6, wherein the camera work is determined to be a push-in, a pull-out, a follow, or a lead based on the above.

13. the detection means detects a motion vector for each of a plurality of regions obtained by dividing a frame of the moving image; The determination means If the ratio of the maximum frequency to the total frequency of the angular distribution of the motion vector detected by the detection means exceeds a threshold, the moving direction of the imaging device is determined based on the angle at which the maximum frequency occurs.

2. The image processing device according to claim 1, wherein:

14. 14. The image processing device according to claim 13, wherein the determining means determines the moving direction of the imaging device as an upward direction, a downward direction, a leftward direction, or a rightward direction based on the angle with the highest frequency.

15. When the ratio does not exceed the threshold value, the determination means determining a direction of a motion vector for each of a plurality of partial regions set for a frame of the moving image, using a horizontal component and a vertical component; 14. The image processing device according to claim 13, wherein the moving direction of the imaging device is determined as a forward direction or a backward direction based on the relationship between the positions of the plurality of partial regions and the direction of the determined motion vector.

16. The determination means The image processing device of claim 15, characterized in that the moving direction of the imaging device is determined to be forward when the horizontal component of the motion vector determined for a partial region set in the left half of the frame among the plurality of partial regions is leftward and the horizontal component of the motion vector determined for a partial region set in the right half of the frame is rightward, or when the vertical component of the motion vector determined for a partial region set in the upper half of the frame is upward and the vertical component of the motion vector determined for a partial region set in the lower half of the frame is downward.

17. The determination means The image processing device described in claim 15, characterized in that the movement direction of the imaging device is determined to be backward when the horizontal component of the motion vector determined for a partial region set in the left half of the frame among the plurality of partial regions is rightward and the horizontal component of the motion vector determined for a partial region set in the right half of the frame is leftward, or when the vertical component of the motion vector determined for a partial region set in the upper half of the frame is downward and the vertical component of the motion vector determined for a partial region set in the lower half of the frame is upward.

18. 16. The image processing device according to claim 15, wherein the determining means does not use, among the plurality of partial regions, partial regions in which the standard deviation of the angles of the motion vectors is equal to or greater than a predetermined value in determining the movement direction of the imaging device.

19. 2. The image processing apparatus according to claim 1, wherein the determining means outputs a reliability of each of a plurality of camera works instead of determining the camera work of the imaging device.

20. an imaging unit that outputs video; an image processing device according to any one of claims 1 to 19, which uses the moving image; and a recording means for recording the type of camerawork determined by the determination means in association with the video.

21. An image processing method implemented by an image processing device, Detecting motion vectors from a video; Obtaining information about a subject area detected in the video; acquiring information about the movement of the imaging device when capturing the video; determining a direction of movement of the imaging device based on the motion vector; determining camera work of the imaging device when capturing the video based on the movement direction, information on the subject area, and information on the movement; An image processing method comprising:

22. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 19.

Images