Camera work detection device and program

Abstract

This invention provides a camera movement detection device that can detect camera movements accurately with less computational load than conventional devices. [Solution] The camera work detection device 1 includes a spatiotemporal image generation unit 10 that generates spatiotemporal images in the horizontal and vertical directions at predetermined intervals within a frame; an edge continuation row detection unit 20 that performs shift calculations using multiple different shift amounts for each row in the horizontal and vertical spatiotemporal images and performs a logical AND operation with the previous row, detecting rows where the maximum number of true values ​​exceeds a threshold as edge continuation rows; and a camera work determination unit 30 that determines the camera work from the length in the time direction, which is the total number of edge continuation rows, and the length in the spatial direction, which is the sum of the shift direction and shift amount in the shift calculations performed on the edge continuation rows.

Description

Technical Field

[0001] The present invention relates to a camera work detection device and a program thereof.

Background Art

[0002] Normally, when producing news videos, an editor edits the footage shot by a cameraman at the scene into a video according to the news time. In this case, even if the news video is a short video of about 1 to 2 minutes, the editor needs to check the entire footage and extract the parts to be used in the news video. Normally, in the camera operations (camera work) performed by a cameraman, pan and zoom operations are often performed according to changes in the situation. That is, the starting point of the camera work can be used as the starting point for the editor to check the footage. If the detection of the starting point of this camera work is automated, the editing work time of the news video can be shortened. Conventionally, the detection of camera work has been performed by obtaining the length and angle of optical flow vectors between temporally adjacent frames (see Patent Document 1 and Non-Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] Conventional camera movement detection methods require very heavy computational processing because they globally calculate the length and angle of the optical flow vector within the frame and then determine the mean, variance, etc. Furthermore, conventional methods do not distinguish between moving subjects in a video and stationary subjects. Therefore, conventional methods have the problem of not being able to recognize the difference between the movement of a moving subject and the camera work used to film a stationary subject.

[0006] Therefore, the object of the present invention is to provide a camera work detection device and program that can detect camera work accurately with less computational load than conventional devices. [Means for solving the problem]

[0007] To solve the aforementioned problems, the camera work detection device according to the present invention is a camera work detection device that detects camera work for each group of frames of a predetermined time length that constitute a camera image, and comprises a spatiotemporal image generation unit, an edge consecutive row detection unit, and a camera work determination unit.

[0008] In this configuration, the camera work detection device generates a horizontal spatiotemporal image in the left and right regions of the frame, by arranging rows of binarized pixels with edges in each horizontal block at predetermined vertical intervals, for the number of frames in the frame group. Furthermore, the camera work detection device generates a vertical spatiotemporal image in the upper and lower regions of the frame, by arranging rows of binarized pixels with edges in each vertical block at predetermined horizontal intervals, for the number of frames in the frame group.

[0009] The camera work detection device then uses an edge continuation row detection unit to perform shift operations on each row of the spatiotemporal images in the horizontal and vertical directions, using multiple different shift amounts, and also performs a logical AND operation with the previous row. Rows where the maximum number of true values ​​exceeds a threshold are detected as edge continuation rows.

[0010] The camera work detection device then determines the camera work based on the length in the time direction, which is the total number of consecutive edge rows, and the length in the spatial direction, which is the sum of the shift direction and shift amount in the shift calculation performed on the consecutive edge rows.

[0011] Furthermore, the camera work detection device detects camera work from spatiotemporal images with multiple spatial coordinates that differ. Therefore, even if the subject is moving in a given spatiotemporal image, the camera work can be accurately detected from the detection results in other spatiotemporal images within the same frame group where the subject is stationary. Furthermore, the camera work detection device can be operated using a program that causes the computer to function as one of the aforementioned components. [Effects of the Invention]

[0012] According to the present invention, by detecting edge continuity from spatiotemporal images with multiple spatial coordinates using light operations such as shift and logical AND operations, camera movement can be detected more accurately and with less computational load than in conventional methods. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing the configuration of a camera work detection device according to an embodiment of the present invention. [Figure 2] This is an explanatory diagram for describing the structure of camera footage and spatiotemporal images. [Figure 3] This figure shows an example of the arrangement of blocks that generate a horizontal spatiotemporal image. [Figure 4]This figure shows a specific example of block arrangement in a horizontal spatiotemporal image of 2K video. [Figure 5] This is an enlarged view of the block in Figure 4. [Figure 6] This is an explanatory diagram for describing the method of generating spatiotemporal images. [Figure 7] This figure shows an example of the arrangement of blocks that generate a vertical spatiotemporal image. [Figure 8A] This figure shows an example of the trajectory of a subject caused by zooming. [Figure 8B] This figure shows an example of a block that generates a horizontal spatiotemporal image used for zoom detection. [Figure 8C] This figure shows an example of a block that generates a vertical spatiotemporal image used for zoom detection. [Figure 9] This is an explanatory diagram illustrating a method for generating spatiotemporal images for zoom detection. [Figure 10] This is an explanatory diagram for illustrating the shift and AND processing in the edge consecutive row detection unit. (a) is an enlarged view of two rows of the spatiotemporal image, and (b) is a diagram showing a specific example of the shift and AND operation in (a). [Figure 11] This figure shows an example of a list of information generated by the edge continuity detection unit for each spatiotemporal image. [Figure 12] These are diagrams for defining straight lines and curves, where (a) is a diagram illustrating the concepts of the temporal and spatial lengths of the lines traced by edges, and (b) is a diagram illustrating an example in which straight lines and curves are defined by the temporal and spatial lengths of the lines traced by edges. [Figure 13] This is an explanatory diagram for explaining the camera work judgment criteria of the camera work judgment unit. [Figure 14] This is an explanatory diagram for explaining the pan detection method of the pan detection unit, where (a) is the detection result of consecutive edge rows, (b) is the result of performing a logical OR operation on the detection result of (a) for each shift direction, and (c) is a diagram showing the possible camera work that can be determined from the logical OR operation result of (b). [Figure 15]This diagram shows the configuration of the frame group to explain the search order in the bread detection unit. [Figure 16] This figure shows the numbers assigned to the spatiotemporal images in the frame group to explain the search order in the pan detection unit. [Figure 17] This diagram shows the search sequence in the bread detection unit. [Figure 18A] This is an explanatory diagram illustrating the process of recognizing moving and stationary objects in the pan detection unit to detect a state where no camera movement is performed and the camera is fixed, where (a) shows the previous frame group and (b) shows the current frame group. [Figure 18B] This is an explanatory diagram illustrating the process of recognizing moving and stationary objects in the bread detection unit to detect bread, where (a) shows the previous frame group and (b) shows the current frame group. [Figure 19] This is an explanatory diagram for explaining the zoom detection method of the zoom detection unit, where (a) is the detection result of consecutive edge rows, (b) is the result of adding the detection results of (a) for each shift direction, and (c) is a diagram showing the possible camera work that can be determined from the summation result of (b). [Figure 20] This is an explanatory diagram illustrating the process by which the judgment and synthesis unit determines the camera work of a group of frames based on the detection results in the horizontal and vertical directions. [Figure 21] This is a flowchart showing the operation of the camera work detection device according to an embodiment of the present invention. [Figure 22] Figure 21 is a flowchart showing the operation of the bread detection process. [Figure 23] Figure 21 is a flowchart showing the operation of the zoom detection process. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. [Configuration of the camera work detection device] Referring to Figure 1, the configuration of the camera work detection device 1 according to an embodiment of the present invention will be described.

[0015] The camera work detection device 1 detects camera work for each group of frames with a predetermined time length t that make up the camera image. Camera work refers to panning in the horizontal and vertical directions, and zooming in and out. The following explanation uses an example where the camera footage is 2K video (resolution: 1920×1080, frame rate: 30fps) and the predetermined time duration t is 2 seconds, but it is not limited to this example. The camera work detection device 1 comprises a spatiotemporal image generation unit 10, an edge continuous row detection unit 20, a camera work determination unit 30, and a storage unit 40.

[0016] The spatiotemporal image generation unit 10 generates a horizontal spatiotemporal image in which, in the left and right regions of the frame, a series of pixels with binarized edges are arranged in a number corresponding to the number of frames in the frame group, for each horizontal block with a predetermined vertical spacing. Furthermore, the spatiotemporal image generation unit 10 also generates a vertical spatiotemporal image in the upper and lower regions of the frame, by arranging rows of pixels with binarized edges for each vertical block spaced at predetermined horizontal intervals, corresponding to the number of frames in the frame group.

[0017] Furthermore, in order to reduce the amount of computation, the camera video input to the spatiotemporal image generation unit 10 uses only the luminance signal (Y), the difference between the luminance signal and the blue component (U), and the difference between the luminance signal and the red component (V) of the camera video. This input is performed by skipping one line each in the horizontal and vertical directions, and skipping one frame in the time direction.

[0018] As shown in Figure 2, the spatiotemporal image I is a camera image V in which frames in the spatial direction (horizontal direction [W direction], vertical direction [H direction]) are continuous in the temporal direction (T direction). CAM In this context, it is an image in which the pixel sequences in the spatial direction (W direction or H direction) are arranged in the temporal direction (T direction). Here, the image in which the pixel sequences in the W direction of the frame are arranged in the T direction is called a spatiotemporal image I. WTIt is denoted as such. Also, an image obtained by arranging pixel columns in the H direction of the frame in the T direction is referred to as a spatio-temporal image I HT It is denoted as such.

[0019] The spatio-temporal image generation unit 10 includes a spatio-temporal image generation unit 11 for pan detection and a spatio-temporal image generation unit 12 for zoom detection.

[0020] The spatio-temporal image generation unit 11 for pan detection generates a spatio-temporal image for detecting pan of camera work. As shown in FIG. 3, the spatio-temporal image generation unit 11 for pan detection generates a horizontal spatio-temporal image by generating pixel columns for one line by emphasizing edges, taking absolute values, binarizing, and arranging them in the time direction for each block B with a predetermined interval in the vertical direction in the left region S C excluding the predetermined central region S L and the right region S R of the frame F. Note that the binarization threshold is preferably adjustable appropriately according to an external setting. W For a specific example, it will be described with reference to FIGS. 4 to 6.

[0021] Here, as shown in FIG. 4, the frame F has a width W of 960 pixels and a height H of 540 pixels obtained by thinning out one line each in the horizontal and vertical directions of the 2K video frame F. In the frame F, the spatio-temporal image generation unit 11 for pan detection extracts the block B from the left region S C excluding the width Cnt of 100 pixels as the central region S L and the right region S R at an interval Int of 60 pixels in the vertical direction. Excluding the central region S W is to prevent misjudging the zoom as a state where the camera is fixed without camera work due to a straight line drawn in the central region in the spatio-temporal image where zooming is performed. Also, here, although the end regions with a width Ed of 60 pixels are excluded from the left end of the left region S C and the right end of the right region S L R respectively, this is not essential.​ Block B W As shown in Figure 5, this is an image with a range of 3 rows × 370 columns.

[0022] The spatiotemporal image generation unit 11 for bread detection is located in Block B. W In this process, edges that draw vertically extending lines are emphasized, their absolute values ​​are taken, and then the image is binarized to generate a 1x370 pixel array. For edge enhancement, general techniques can be used. For example, the spatiotemporal image generation unit 11 for pan detection can enhance edges that draw vertically extending lines using a 3x3 Sobel filter as shown in equation (1) below.

[0023]

number

[0024] Block B W The reason for using three rows is to utilize a 3x3 Sobel filter. The spatiotemporal image generation unit 11 for bread detection is located in block B, as shown in Figure 6. W By arranging the generated 1x370 pixel array for each step, a horizontal spatiotemporal image I is created, corresponding to a predetermined number of frames of a set time length. WT Generates. In this case, since frames are input skipping one frame at a time, only 30 frames, corresponding to a predetermined time length of 2 seconds, are arranged in each frame group, resulting in a horizontal spatiotemporal image I of 30 frames × 370 columns. WT Generates.

[0025] The spatiotemporal image generation unit 11 for pan detection further generates a predetermined central region S of frame F, as shown in Figure 7. C Upper region S excluding U and lower region S D In this, Block B is spaced horizontally at predetermined intervals. HFor each pixel, edges are emphasized, their absolute values ​​are taken, and then they are binarized to generate a line of pixels. These lines are then arranged in the time direction to generate a vertical spatiotemporal image. A specific example is the central region S. C The width Cnt, the interval Int for extracting blocks, and the width Ed of the excluded edge region are the same as the horizontal value, and block B H This is an image with a range of 160 rows x 3 columns. The generation of this vertical spatiotemporal image is the same process as the generation of the horizontal spatiotemporal image, only the direction is different, so we will omit the explanation.

[0026] Block B H In Figure 7, when edges that draw horizontally extending lines are emphasized, their absolute values ​​are taken, and then the image is binarized to generate a 160x1 pixel array, the spatiotemporal image generation unit 11 for pan detection can emphasize edges that draw horizontally extending lines using a 3x3 Sobel filter as shown in equation (2) below.

[0027]

number

[0028] The spatiotemporal image generation unit 11 for bread detection stores the generated horizontal and vertical spatiotemporal images for bread detection in the storage unit 40, associating them with the frame's time code. Furthermore, the spatiotemporal image generation unit 11 for pan detection notifies the spatiotemporal image generation unit 12 for zoom detection that it has stored the horizontal and vertical spatiotemporal images for pan detection in the storage unit 40.

[0029] The spatiotemporal image generation unit 12 for zoom detection generates spatiotemporal images for detecting zoom in camera movement. The spatiotemporal image generation unit 12 generates spatiotemporal images for zoom detection by logarithmically transforming the spatial coordinates of the horizontal and vertical spatiotemporal images for pan detection generated by the spatiotemporal image generation unit 11.

[0030] Furthermore, the spatiotemporal image generation unit 12 for zoom detection does not convert all horizontal and vertical spatiotemporal images, but rather converts only some spatiotemporal images according to the characteristics of the zoom. As shown in Figure 8A, when zooming, subjects within frame F will trace radial trajectories from the center of frame F. Therefore, due to the angle, blocks included in the central region of frame F are more likely to exhibit a zoom curve in the in-plane direction of the spatiotemporal image than blocks included in the edge regions. Therefore, the spatiotemporal image generation unit 12 for zoom detection generates the horizontal spatiotemporal image as shown in Figure 8B, specifically the upper and lower central region A within frame F. W Block B W The target is the horizontal spatiotemporal image generated from. In the example in Figure 8B, the upper and lower central region A W Block B W The study focuses on eight spatiotemporal images generated from [the source]. As will be described later, the zoom detection unit 32 performs a logical OR operation between the "presence or absence of correlation in the left region" and the "presence or absence of correlation in the right region" in the left and right regions of these eight spatiotemporal images, thereby combining information on zoom curves that span different spatiotemporal images with multiple H coordinates.

[0031] Furthermore, the spatiotemporal image generation unit 12 for zoom detection generates the spatiotemporal image in the vertical direction, as shown in Figure 8C, the left and right central region A within frame F. H Block B H The target is the vertical spatiotemporal image generated from. In the example in Figure 8C, the left and right central region A H Block B H The study focuses on six spatiotemporal images generated from [the source].

[0032] The processing of the spatiotemporal image generation unit 12 for zoom detection will be explained with reference to Figure 9. As shown in Figure 9, when zooming is performed, for example, the horizontal spatiotemporal image I WTThe lines drawn by the edges in this case change multiplicatively in distance from the frame center C, and the line groups (e.g., L1, L2) are not parallel. In that case, it becomes difficult to detect consecutive edge rows using the shift and processing in the edge consecutive row detection unit 20 described later.

[0033] Therefore, the spatiotemporal image generation unit 12 for zoom detection generates a horizontal spatiotemporal image I as shown in Figure 9. WT The W coordinate (column index) with the origin at the center C of the frame is W' = a * log b The horizontal spatiotemporal image I is obtained by logarithmically transforming (W) and rounding W' to obtain an integer value. ZWT This will be the column index. Here, the value of base b is arbitrary, within the range greater than 1 and less than half of Cnt as set in Figure 4. The value of multiplier a is a value that results in a sequential column index after rounding. Then, the spatiotemporal image generation unit 12 for zoom detection generates spatiotemporal image I WT For each column, spatiotemporal image I ZWT Move to the column of the corresponding column index. Spatiotemporal Image I WT From multiple columns, spatiotemporal image I ZWT When moving to a single column, move one column of the result of the logical OR operation of multiple columns. This allows for the creation of spatiotemporal image I ZWT Then, the line group (L1', L2') becomes almost parallel, making it possible to detect consecutive edge rows by shift and processing. The spatiotemporal image generation unit 12 for zoom detection also performs a logarithmic transformation on the vertical spatiotemporal image in the same manner to generate a vertical spatiotemporal image for zoom detection. Returning to Figure 1, we will continue our explanation of the configuration of the camera work detection device 1.

[0034] The spatiotemporal image generation unit 12 for zoom detection stores the generated horizontal and vertical spatiotemporal images for zoom detection in the storage unit 40, associating them with the frame's time code. Returning to Figure 1, we will continue our explanation of the configuration of the camera work detection device 1.

[0035] The edge continuation row detection unit 20 performs shift operations on each row using multiple different shift amounts in the horizontal and vertical spatiotemporal images (spatiotemporal image for pan detection, spatiotemporal image for zoom detection) stored in the memory unit 40, and also performs a logical AND operation with the previous row. Rows where the maximum number of true values ​​exceeds a threshold are detected as edge continuation rows. The edge continuation row detection unit 20 is instructed by the camera work determination unit 30, which will be described later, to perform operations on which spatiotemporal images and within what range the shift amounts should be varied.

[0036] Here, with reference to Figures 10 and 11, the processing of the edge continuity row detection unit 20 will be explained in detail. Note that here, the horizontal spatiotemporal image I for pan detection is used. WT This will be explained using the example, but the same applies to the vertical spatiotemporal image used for pan detection, and the horizontal and vertical spatiotemporal images used for zoom detection.

[0037] Figure 10(a) shows a single spatiotemporal image I WT And, spatiotemporal image I WT The image shows an enlarged view of the second row (row n and row n+1). Here, pixels shown in white are pixels detected as edges and have a pixel value of "1". Pixels shown in black are pixels that were not detected as edges and have a pixel value of "0". Figure 10(b) shows the state in which an AND row is generated by using the pixel column in the nth row of Figure 10(a) as a mask row, shifting the pixel column in the (n+1)th row horizontally (here, shifting by 1 pixel to the right), and performing an AND operation between the mask row and the (n+1)th row (hereinafter referred to as shift-and operation).

[0038] If the pixel value of a mask row is "1", the calculation result of the corresponding pixel value in the AND row will not change from the pixel value of the (n+1)th row. If the pixel value of a mask row is "0", the calculation result of the corresponding pixel value in the AND row will be "0". This allows for the detection of continuity of edges that are "1" as lines to the next row, and by setting the calculation result of non-contiguous pixels to "0", noise other than lines can be removed.

[0039] The more "1"s there are in an AND row, the higher the correlation between the mask row (the nth row or the AND row generated from the row before last and the row before that) and the (n+1)th row. The edge continuation row detection unit 20 performs shift and AND operations with multiple shift amounts in the left and right directions, and if the maximum number of "1"s in the AND row exceeds a threshold, the (n+1)th row is considered an edge continuation row where the edge is continuous from the nth row. This threshold is, for example, 60% of the number of "1"s in the mask row. If bread is performed, spatiotemporal image I WT The lines drawn by the edges in this region become parallel. When zooming occurs, spatiotemporal image I WT The lines traced by the edges in this case become parallel when logarithmically transformed, as described above. In shift and processing, the more parallel the lines are, the easier it is to detect them as a continuous row of edges. The edge contiguous row detection unit 20 performs this shift-and-process detection of edge contiguous rows for each spatiotemporal image, starting with the first row as the mask row, and repeating sequentially in the frame direction (time direction). If the maximum number of "1"s in the AND row does not exceed the threshold, the (n+1)th row is used as the new mask row and the process is repeated. If the maximum number of "1"s in the AND row exceeds the threshold, the AND row with the highest number of "1"s is used as the new mask row and the process is repeated.

[0040] The edge continuity row detection unit 20 stores in the storage unit 40, for each pair of spatiotemporal images of the left and right regions with the same H coordinate as shown in Figure 3, whether or not a continuity row of edges has been detected, the shift direction when the number of "1"s in the AND row reaches its maximum, and the amount of the shift. Similarly, the edge continuation row detection unit 20 stores in the storage unit 40, as a pair of spatiotemporal images of upper and lower regions with the same W coordinate as shown in Figure 7, whether or not a continuation row of edges has been detected, the shift direction when the number of "1"s in the AND row reaches its maximum, and the amount of the shift.

[0041] Figure 11 shows an example of the information that the edge continuity row detection unit 20 stores in the storage unit 40. Figure 11 shows the results detected in spatiotemporal images of a pair of left and right regions with the same H coordinate, in matrix form. "Left Region Correlation" indicates whether there is a correlation with the previous row in the left region spatiotemporal image in the left shift and right shift directions (Yes: "1", No: "0"). "Right Region Correlation" indicates whether there is a correlation with the previous row in the right region spatiotemporal image in the left shift and right shift directions (Yes: "1", No: "0"). "Shift Direction & Amount" indicates the shift direction and shift amount (positive for left shift, negative for right shift) when the number of "1"s in the AND row reaches its maximum in each of the "Left Region" and "Right Region" areas.

[0042] For example, in the first frame direction illustrated in Figure 11 (corresponding to the result of processing the first and second rows of the spatiotemporal image), no consecutive rows of edges are detected when shifting to the left and performing an AND operation in both the left and right regions (left shift: "0"), while in the left region, consecutive rows of edges are detected when shifting to the right and performing an AND operation (right shift: "1"). Furthermore, the maximum number of "1"s in the AND row occurs when the left region is shifted to the right by one pixel (-1).

[0043] The edge continuity row detection unit 20 generates the matrix shown in Figure 11 for each spatiotemporal image of a pair of left and right regions with different H coordinates, and stores it in the storage unit 40. Similarly, the edge continuity row detection unit 20 generates a matrix similar to that in Figure 11 for each spatiotemporal image of a pair of upper and lower regions with different W coordinates. In this case, since only the left and right regions in Figure 11 are replaced with the upper and lower regions, the illustration and explanation are omitted. Returning to Figure 1, we will continue our explanation of the configuration of the camera work detection device 1.

[0044] The camera work determination unit 30 determines the camera work based on the length in the time direction, which is the total number of consecutive edge rows detected by the edge consecutive row detection unit 20, and the length in the spatial direction, which is the sum of the shift direction and shift amount in the shift calculation performed on the consecutive edge rows. Furthermore, the camera work determination unit 30 uses edges that draw lines within the frame that extend perpendicular to the direction of camera movement in order to determine the camera work.

[0045] Now, with reference to Figures 12 and 13, the processing overview of the camera work determination unit 30 will be explained. First, referring to Figure 12, we will explain the temporal and spatial lengths of the lines drawn by the edges that serve as the basis for the camera work determination unit 30 to determine the camera work, as well as the straight and curved lines drawn by the edges. Figure 12 shows the spatiotemporal image I in the horizontal direction. WT Let's explain using this as an example.

[0046] As shown in Figure 12(a), the length in the time direction is the spatiotemporal image I WT This represents the number of consecutive frames in the time direction of the line EL traced by the edge shown above. This length in the time direction corresponds to the total number of rows with a correlation of "1" (consecutive edge rows) in the matrix shown in Figure 11. Furthermore, the spatial length is the sum of the shift amounts in the rows where correlation was detected (edge ​​contiguous rows) in the matrix shown in Figure 11.

[0047] As shown in Figure 12(b), the condition of whether the line EL drawn by the edge in Figure 12(a) is a straight line or a curve is set in advance. Here, a straight line is defined as having a length of 24 frames or more in the time direction and a length of 40 pixels or less in the spatial direction. In other words, a straight line here refers to a line whose shape extends across a predetermined number of frames in the time direction and whose spatial change is less than a predetermined number of pixels. Furthermore, when detecting panning, a curve must have a temporal length of 20 frames or more and a spatial length of 80 pixels or more. When detecting zooming, the temporal length must be 15 frames or more. In other words, a curve, as used here, refers to a line whose shape extends across a predetermined number of frames in the temporal direction, and whose spatial change in the case of panning is greater than or equal to a predetermined number of pixels.

[0048] Next, with reference to Figure 13, the criteria for determining the camera work judgment unit 30 will be explained. Note that here, the horizontal spatiotemporal image I WT and spatiotemporal image I ZWT The criteria for judgment are shown below. As shown in Figure 13, the camera work determination unit 30 determines the spatiotemporal image I of at least one of the left and right regions. WT In this case, if there is a straight line, it is determined that no camera movement is performed and the camera is fixed. Furthermore, the camera work determination unit 30 determines the spatiotemporal image I of the left and right regions. ZWT If curves in the opposite direction are present simultaneously, it is determined to be a "zoom" curve. Furthermore, the camera work determination unit 30 determines the spatiotemporal image I of the left and right regions. ZWT Therefore, spatiotemporal image I of the left and right regions where curves in opposite directions do not exist simultaneously and the H coordinates are the same. WT When combined, if there are curves in the same direction, it is determined to be a horizontal "pan". Furthermore, even if edges are continuous in the temporal and spatial directions, if their length is small and they are neither straight nor curved, the camera work determination unit 30 will determine it as "blur."

[0049] The camera work determination unit 30 will be described in detail below. The camera work determination unit 30 includes a pan detection unit 31, a zoom detection unit 32, a determination and synthesis unit 33, and a cut point detection unit 34.

[0050] The pan detection unit 31 detects the horizontal pan of the camera work from the direction and length of continuous edges in the horizontal spatiotemporal images generated from the left and right regions, and detects the vertical pan of the camera work from the direction and length of continuous edges in the vertical spatiotemporal images generated from the upper and lower regions. The bread detection unit 31 detects bread from spatiotemporal images for bread detection stored in the storage unit 40.

[0051] The pan detection unit 31 instructs the edge contiguous row detection unit 20 to perform a shift-and process for each spatiotemporal image in the frame group, stores in the storage unit 40 whether or not a contiguous row of edges was detected, the shift direction and shift amount (see Figure 11) when the number of "1"s in the AND row reached its maximum, and detects pan by measuring the length of the contiguous row of edges in the temporal and spatial directions. At this time, the pan detection unit 31 executes the edge continuous row detection unit 20 by specifying a predetermined shift amount for detecting straight lines and curves for each spatiotemporal image. For example, the pan detection unit 31 changes the shift amount within a range of ±25 pixels, including a shift of 0 pixels. The pan detection unit 31 executes the edge continuous row detection unit 20 according to a pre-set sequence of spatiotemporal images. This sequence of spatiotemporal images will be described later.

[0052] The pan detection unit 31 detects straight lines and curves in the spatiotemporal image of each block based on the total number of consecutive edge rows and the sum of the shift amounts of those consecutive edge rows. Then, as shown in Figure 13, the pan detection unit 31 determines that there is a horizontal "pan" if, in the spatiotemporal image of the left region and the spatiotemporal image of the right region, there are no curves extending in opposite directions simultaneously, and there are curves extending in the same direction.

[0053] Now, with reference to Figure 14, the horizontal bread detection method of the bread detection unit 31 will be explained. Figure 14(a) shows a portion of the presence or absence of consecutive edge rows detected in the spatiotemporal images of the pair of left and right regions shown in Figure 11. The pan detection unit 31 performs a logical OR operation on the values ​​of the detected consecutive edge rows for "left region correlation presence / absence" and "right region correlation presence / absence" in the same frame and the same shift direction, thereby generating a matrix in which the "left-shifted correlation logical OR" and "right-shifted correlation logical OR" shown in Figure 14(b) are arranged in the frame direction. This means that even if the temporal continuity of an edge is broken in one of the left or right regions, as long as it is not broken in the other region, the logical OR operation can maintain continuity.

[0054] The pan detection unit 31 then determines whether the line drawn by the edge is a straight line or a curve predetermined in Figure 12, based on the length in the time direction, which is the total number of rows of "1" in Figure 14(b), and the length in the spatial direction, which is the sum of the shift amounts in the shift direction where the "1"s are consecutive (see Figure 11). The bread detection unit 31 then detects the bread according to the criteria shown in Figure 13 and determines whether it is a rightward or leftward shifted bread, as shown in Figure 14(c), based on the direction of its shift.

[0055] In this pan detection process, we will refer to the process of generating spatiotemporal images of each block in a predetermined order and performing shift and processing on each spatiotemporal image until a straight line is detected as "search." Next, the search order of the horizontal spatiotemporal image by the pan detection unit 31 will be explained. To accurately determine panning, it is necessary to focus on the lines drawn by a stationary object (a motionless subject) in the spatiotemporal image. Due to the spatiotemporal continuity of the video, in a series of consecutive frames, a stationary object is likely to be in a spatiotemporal image at a nearby coordinate. Therefore, the pan detection unit 31 searches in order of proximity to the coordinates where the straight line or curve used for determining camera movement in the previous frame series was detected, and uses the first straight line or curve detected for the determination. Furthermore, as will be described later, if even one spatiotemporal image containing a straight line is detected in a given set of frames, it can be determined to be "fixed," and processing for that set of frames can be terminated. Therefore, if a series of consecutive frames are fixed, this search order allows for earlier detection of the straight line, thereby reducing computational load. As shown in Figure 15, the frame group (previous frame group GR) P , current frame group GR C Each of these consists of multiple (16 in this case) spatiotemporal images. To make the coordinate order easier to understand, the spatiotemporal images of the frame group GR are numbered L1 to L8 in the left region and R1 to R8 in the right region, as shown in Figure 16.

[0056] The pan detection unit 31 uses the previous frame group GR for the H coordinate. PThe system searches for points closest to the detected straight or curved line or curve, which was used to determine the camera's movement. Furthermore, the pan detection unit 31 detects the previous frame group GR for left and right regions with the same H coordinate. P The search begins in the area where a straight line or curve was detected, whichever was used to determine the camera movement. For example, the pan detection unit 31 is the previous frame group GR P If a straight line or curve used for camera work determination is detected from the spatiotemporal image of L5, the current frame group GR C The search begins by looking at the spatiotemporal images of L5.

[0057] Figure 17 shows the spatiotemporal sequence of images from the search start position to the search end position. For example, if L5 is the search start position, the pan detection unit 31 will then search in the following order: R5, L6, R6, L4, R4, L7, R7, L3, R3, L8, R8, L2, R2, L1, R1. If the search start position is in the right region, only the L and R in Figure 17 will be reversed.

[0058] Then, if the pan detection unit 31 detects a straight line in any of the spatiotemporal images in the current frame group, it determines that the image is "fixed" and terminates pan detection. Furthermore, if the pan detection unit 31 detects a curve in any of the spatiotemporal images in the current frame group, it then detects a straight line in another spatiotemporal image. When a straight line is detected, it determines that the image is "fixed" and terminates the pan detection. At this time, the pan detection unit 31 executes the edge continuous row detection unit 20 by specifying a predetermined shift amount for detecting only straight lines. For example, the pan detection unit 31 changes the shift amount within a range of ±4 pixels, including a shift of 0 pixels.

[0059] Furthermore, after detecting a curve, the pan detection unit 31 determines that there is a horizontal "pan" if it does not detect a straight line in any of the spatiotemporal images in the current frame group. Furthermore, the pan detection unit 31 determines that frames in which neither straight lines nor curves are detected are "blurred".

[0060] Here, with reference to Figures 18A and 18B, a specific example of the horizontal pan detection method of the pan detection unit 31 will be described. Figure 18A shows the state where the traffic light changes from red to blue, and from that point the stationary car (subject O) begins to move to the left, with the camera fixed. Figure 18B shows the state where the traffic light remains red, the car (subject O) is stationary, and the camera is panned to the right. In Figures 18A and 18B, the tree and the traffic light are static objects (stationary subjects).

[0061] Figures 18A(a) and (b) show the previous frame group GR. P and the current frame group GR C These are the initial frames of each. Note that, for the sake of clarity, Figure 18A(b) shows six spatiotemporal images I on both the left and right sides. As shown in Figure 18A(a), the previous frame group GR P The subject O is visible in the image. Also, the previous frame group GR P Let's assume that it was determined to be "fixed" in the spatiotemporal image I of R5. As shown in Figure 18A(b), the pan detection unit 31 detects the current frame group GR C In this case, when a curve is detected in the spatiotemporal image I of R5, it is not yet recognized at this stage whether the subject O is a moving object or a stationary object whose position has shifted due to panning. Therefore, the pan detection unit 31 continues searching for straight lines in other spatiotemporal images I, and if it detects a straight line from a stationary tree in the spatiotemporal image I of L3, it recognizes the subject O as a moving object, and the current frame group GR C This is determined to be "fixed".

[0062] Figures 18B(a) and (b) also show the previous frame group GR. P and the current frame group GR C These are the first frames of each. As shown in Figure 18B(a), the previous frame group GR P The subject O is visible in the image. Also, the previous frame group GR P Let's assume that it was determined to be "fixed" in the spatiotemporal image I of R5. As shown in Figure 18B(b), the pan detection unit 31 detects the current frame group GR C In this case, when a curve is detected in the spatiotemporal image I of R5, it is not yet recognized at this stage whether the subject O is a moving object or a stationary object whose position has changed due to panning. Therefore, the pan detection unit 31 continues searching for straight lines in other spatiotemporal images I, and if it does not detect straight lines in all other spatiotemporal images I, it recognizes the subject O as a stationary object and the current frame group GR C This is interpreted as a horizontal "pan".

[0063] In the two examples above, the position of subject O appears to be moving in the same way within the video, and accurate panning cannot be detected without recognizing whether it is a stationary or moving object. Therefore, the pan detection unit 31 focuses on the line drawn in the current frame group by the subject, which was stationary in the previous frame group, using a search mechanism that notifies the coordinates of the spatiotemporal image in which a straight line was detected in the previous frame group, i.e., the spatiotemporal image in which a stationary object was detected, to the current frame group. Furthermore, by searching for straight lines in all spatiotemporal images of the current frame group, we can distinguish between the example in Figure 18A, where a subject that was stationary in the previous frame group becomes a moving object in the current frame group, and the example in Figure 18B, where a subject that was stationary in the previous frame group remains stationary in the current frame group. This allows the pan detection unit 31 to correctly recognize whether the subject O is moving or stationary, and to accurately detect the pan.

[0064] When the pan detection unit 31 has finished searching the spatiotemporal image, it notifies the cut point detection unit 34 of the search order. In this explanation, the pan detection unit 31 has only described the detection of pan in the horizontal direction. However, regarding vertical panning, since it only differs in direction and uses spatiotemporal images of the upper and lower regions instead of spatiotemporal images of the left and right regions, the explanation is omitted. The pan detection unit 31 outputs the detected results (fixed, pan, shake) for each direction (horizontal and vertical) to the judgment and synthesis unit 33.

[0065] The zoom detection unit 32 detects the zoom of the camera work from the direction and length of continuous edges in the horizontal spatiotemporal image and the vertical spatiotemporal image for zoom detection. The zoom detection unit 32 detects the zoom level from the spatiotemporal image for zoom detection stored in the memory unit 40.

[0066] The zoom detection unit 32 instructs the edge contiguous row detection unit 20 to perform a shift-and process for each spatiotemporal image in the frame group, stores in the storage unit 40 whether or not a contiguous row of edges has been detected, the shift direction and shift amount (see Figure 11) when the number of "1"s in the AND row reaches its maximum, and detects zoom by measuring the length in the time direction, which is the total number of contiguous rows of edges. At this time, the zoom detection unit 32 executes the edge continuous row detection unit 20 by specifying a predetermined shift amount to detect only curves. For example, the zoom detection unit 32 changes the shift amount within a range of ±11 pixels, without including a shift of 0 pixels.

[0067] The zoom detection unit 32 detects curves in the spatiotemporal image of each block based on the total number of consecutive edge rows and the sum of the shift amounts of the consecutive edge rows. Then, as shown in Figure 13, the zoom detection unit 32 determines that "zoom" has occurred if curves extending in opposite directions are simultaneously present in the spatiotemporal image of the left region and the spatiotemporal image of the right region.

[0068] Here, with reference to Figure 19, the horizontal zoom detection method of the zoom detection unit 32 will be explained. First, the zoom detection unit 32 uses the "presence or absence of correlation in the left region" and the "presence or absence of correlation in the right region" in the spatiotemporal images of the pair of left and right regions shown in Figure 11 to determine the presence or absence of correlation in multiple spatiotemporal images (the upper and lower central region A in Figure 8B). W Block B W The spatiotemporal image generated from the above is subjected to a logical OR operation in the left and right regions, and the results are combined into the matrix shown in Figure 19(a). Figure 19(a) shows three examples in each row. The zoom detection unit 32 adds the detection results of consecutive edge rows in the same frame for "Left Shift" of "Left Region Correlation Presence / Absence" and "Right Shift" of "Right Region Correlation Presence / Absence" in Figure 19(a) to calculate the "Edge Direction Correlation Addition Value" shown in Figure 19(b). The zoom detection unit 32 also adds the detection results of consecutive edge rows in the same frame for "Right Shift" of "Left Region Correlation Presence / Absence" and "Left Shift" of "Right Region Correlation Presence / Absence" in Figure 19(a) to calculate the "Center Direction Correlation Addition Value" shown in Figure 19(b).

[0069] In this way, the zoom detection unit 32 generates an additive matrix (Figure 19(b)) from the spatiotemporal images of the left and right regions, indicating for each frame whether the curve extends towards the edges (away from the left and right) or towards the center (approaching the center).

[0070] Then, the zoom detection unit 32 determines that if the "edge-direction correlation sum" is "2", the frame may be part of the zoom-in curve, as shown in Figure 19(c). Furthermore, the zoom detection unit 32 determines that if the "central direction correlation sum value" after addition is "2", the frame may be part of the zoom-out curve, as shown in Figure 19(c). Furthermore, even if the "edge-direction correlation sum value" or the "center-direction correlation sum value" is "2", the zoom detection unit 32 determines that the frame is not part of the zoom curve (not zoomed) if there is a correlation in the same direction simultaneously in both the "left region correlation presence / absence" and the "right region correlation presence / absence". The zoom detection unit 32 then determines whether the line drawn by the edge is a curve predetermined in Figure 12, based on the length in the time direction, which is the total number of rows labeled "2" in Figure 19(b). The zoom detection unit 32 then detects the zoom according to the criteria shown in Figure 13, and determines whether to zoom in or zoom out based on the direction of the shift, as shown in Figure 19(c). In this way, the zoom detection unit 32 uses a matrix (Figure 19(a), Figure 19(b)) which combines information detected by multiple H coordinates in different spatiotemporal images to determine the presence or absence of a curve. In this case, the zoom detection unit 32 detected zoom from the spatiotemporal images of the left and right regions, but it also detects zoom similarly from the spatiotemporal images of the upper and lower regions. The zoom detection unit 32 outputs the detected results (zoom, blur) for each direction (horizontal direction, vertical direction) to the judgment and synthesis unit 33.

[0071] The zoom detection unit 32 may also add up the detection results of consecutive edge rows in the same frame in four directions (horizontal edge direction, horizontal center direction, vertical edge direction, and vertical center direction), and determine that the frame may be part of the zoom curve if the correlation sum is, for example, "2 or more".

[0072] The judgment and synthesis unit 33 determines the camera work of the frame group from the detection results of camera work detected in the horizontal and vertical directions, respectively. As shown in Figure 20, the judgment and synthesis unit 33 determines the camera work of the frame group from the detection results in the horizontal direction (W) and the detection results in the vertical direction (H).

[0073] For example, if the judgment and synthesis unit 33 determines that the camera movement of the frame group is "panning" when it determines that the movement is "fixed" in the W direction and "panning" in the H direction, it determines that the camera movement of the frame group is "panning". If "pan" and "zoom" are detected in the W direction and H direction, respectively, for example, if "pan" is detected in the W direction and "zoom" in the H direction, the zoom detection unit 32 in the H direction will consider the curves extending towards the edges or the center, based on the "presence or absence of left region correlation" and "presence or absence of right region correlation" explained in Figure 19(a). However, if there are curves extending simultaneously in the same direction, based on the "presence or absence of left region correlation" and "presence or absence of right region correlation" explained in Figure 14(a), the pan detection unit 31 in the W direction will consider the curves extending simultaneously in the same direction. In this case, it is unlikely to be a zoom. Therefore, the judgment synthesis unit 33 removes these frames that are unlikely to be zooms from the matrix explained in Figure 19(a) for the H direction. Subsequently, the judgment synthesis unit 33 removes frames that are considered zooms in the H direction from the matrix explained in Figure 14(a) for the W direction. Then, the judgment synthesis unit 33 uses the frame with the larger number of frames as the detection result for the frame group. The judgment and synthesis unit 33 outputs the detection results in the frame group as camera work detection results, associated with the time code.

[0074] The cut point detection unit 34 detects cut points from the camera footage. A cut point is the first frame of a scene (shot) in the video. Here, it is assumed that the camera footage being analyzed consists of multiple scenes. The cut point detection unit 34 detects as a cut point frames in either the horizontal or vertical spatiotemporal image directions, where, in two or more spatiotemporal images, the absolute value of the difference in the total number of pixels of edges between the current row and the previous row is greater than or equal to a threshold value. The total number of pixels of edges in a row of a spatiotemporal image is an approximate value of the spatial high-frequency components and can therefore be used for cut point detection. The cut point detection unit 34 detects the cut point using the first spatiotemporal image and the last spatiotemporal image in the search order notified by the pan detection unit 31. This is because the last spatiotemporal image is the spatiotemporal image that is furthest from the first spatiotemporal image within the frame.

[0075] Specifically, the cut point detection unit 34 calculates the difference in the total number of pixels of edges between adjacent rows in all rows (30 rows) of the first spatiotemporal image in the search order, and then calculates the absolute value of that difference. The cut point detection unit 34 then determines that if the ratio of the maximum absolute value to the average absolute value of the other absolute values ​​is less than a threshold, the frame corresponding to the maximum value is not a cut point.

[0076] On the other hand, if the ratio is greater than or equal to a threshold, the cut point detection unit 34 similarly calculates the difference in the total number of pixels of edges between adjacent rows in the last spatiotemporal image in the search order and obtains its absolute value. Then, if the frame corresponding to the maximum value of these absolute values ​​matches in the first spatiotemporal image and the last spatiotemporal image, the cut point detection unit 34 determines that frame is a "cut point". The cut point detection unit 34 outputs the timecode of the frame as the cut point. Returning to Figure 1, we will continue our explanation of the configuration of the camera work detection device 1.

[0077] The memory unit 40 stores various information generated during the operation of the camera work detection device 1. The memory unit 40 is composed of a general storage medium such as semiconductor memory. For example, the memory unit 40 stores the spatiotemporal images (spatiotemporal images for pan detection, spatiotemporal images for zoom detection) generated by the spatiotemporal image generation unit 10 in association with time codes. Furthermore, the storage unit 40 stores the detection results from the edge consecutive row detection unit 20.

[0078] With the configuration described above, the camera work detection device 1 can detect camera work without using a high-performance computer by detecting edge continuity through simple operations such as shift and logical AND, without having to determine the length or angle of the optical flow vector as in conventional methods.

[0079] Furthermore, the camera work detection device 1, in the pan detection unit 31, detects straight lines or curves from the spatiotemporal images of the current frame group in order of proximity to the coordinates of the spatiotemporal images in which the straight lines or curves used for determining the camera work in the previous frame group were detected. This allows the camera work detection device 1 to recognize whether the subject is moving or stationary, and by focusing on the lines drawn by the stationary object, it can accurately detect camera work. Furthermore, when detecting straight lines consecutively across a series of frames, the determination can be made with minimal computation using a small number of spatiotemporal images.

[0080] [Operation of the camera work detection device] Next, referring to Figure 21 (and Figure 1 as appropriate for the configuration), the operation of the camera work detection device 1 according to an embodiment of the present invention will be described.

[0081] In step S1, the spatiotemporal image generation unit 10 generates the first spatiotemporal image in a search order for each frame group of a predetermined time length from the camera image and stores it in the storage unit 40. The generation of spatiotemporal images is performed in step S1 and in steps S205 and S402, which will be described later. Here, the spatiotemporal image generation unit 10, along with the pan detection spatiotemporal image generation unit 11, generates a predetermined central region S of the frame F shown in Figure 3. C Left region S excluding L and right region S R In this, Block B is spaced at predetermined intervals in the vertical direction. W For each image, edges are emphasized, their absolute values ​​are taken, and then the image is binarized to generate a single line of pixels. These lines are then arranged in the time direction to generate a horizontal spatiotemporal image for pan detection.

[0082] Furthermore, the spatiotemporal image generation unit 10 generates a predetermined central region S of the frame F shown in Figure 7 using the spatiotemporal image generation unit 11 for pan detection. C Upper region S excluding U and lower region S D In this, Block B is spaced horizontally at predetermined intervals. H For each image, edges are emphasized, their absolute values ​​are taken, and then the image is binarized to generate a single line of pixels. These lines are then arranged in the time direction to generate a vertical spatiotemporal image for pan detection.

[0083] Furthermore, in step S402 described later, the spatiotemporal image generation unit 10 uses the zoom detection spatiotemporal image generation unit 12 to perform a logarithmic transformation on the horizontal and vertical spatiotemporal images for pan detection generated by the pan detection spatiotemporal image generation unit 11, as shown in Figure 9, to generate horizontal and vertical spatiotemporal images for zoom detection. At this time, the zoom detection spatiotemporal image generation unit 12 generates horizontal and vertical spatiotemporal images for zoom detection in the upper and lower central region A within frame F, as shown in Figures 8B and 8C. W and left and right central region A H Block B W ,B H The spatiotemporal image generated for bread detection is the target of the transformation.

[0084] In step S2, the bread detection unit 31 detects bread from the spatiotemporal image stored in the storage unit 40 (bread detection process).

[0085] Now, with reference to Figure 22, the pan detection process in step S2 will be described in detail. This pan detection process is performed sequentially (in search order), starting with spatiotemporal images that are closest in coordinate to the spatiotemporal image used to detect a straight line or curve in the previous frame group for determining camera movement.

[0086] In step S200, the pan detection unit 31 instructs the edge continuous row detection unit 20 to perform a shift and process on the spatiotemporal image for pan detection using a predetermined number of shift amounts (for "fixed" shifts). These "fixed" shift amounts are, for example, in the range of ±4 pixels, including a shift of 0 pixels.

[0087] In step S201, the pan detection unit 31 determines whether or not there is a straight line in the spatiotemporal image based on the temporal and spatial lengths of the consecutive edge rows detected in the shift and processing of step S200.

[0088] If it is determined that there is a straight line (Yes in step S201), the pan detection unit 31 proceeds to step S209. On the other hand, if it is determined that there are no straight lines (No in step S201), the pan detection unit 31 determines in step S202 whether or not there are curves in the spatiotemporal images searched up to this stage in the current frame group.

[0089] If a curve is present (Yes in step S202), the pan detection unit 31 proceeds to step S204. On the other hand, if there is no curve (No in step S202), the pan detection unit 31 proceeds to step S203.

[0090] In step S203, the pan detection unit 31 instructs the edge continuous row detection unit 20 to perform a shift and operation on the spatiotemporal image using a predetermined number of shift amounts ("for panning"). These "for panning" shift amounts are, for example, in the range of ±25 pixels, and do not include shifts in the range of 0 pixels to ±4 pixels.

[0091] In step S204, the pan detection unit 31 determines whether or not the determination has been completed for all spatiotemporal images in the current frame group. If the determination of all spatiotemporal images is completed at this point (Yes in step S204), the pan detection unit 31 proceeds to step S206. On the other hand, if the determination of all spatiotemporal images has not been completed (No in step S204), the pan detection unit 31 proceeds to step S205. In step S205, the pan detection unit 31 specifies the next spatiotemporal image in the frame group according to a preset search order, the spatiotemporal image generation unit 10 generates that spatiotemporal image, and stores it in the storage unit 40. Then, the pan detection unit 31 returns to step S200. Thus, if, in a set of frames, a curve is determined to exist in a spatiotemporal image in step S203, the pan detection unit 31 will only determine the presence or absence of a straight line in step S200 for subsequent spatiotemporal images and will skip step S203.

[0092] In step S206, the pan detection unit 31 determines whether there are no curves extending in opposite directions simultaneously in the spatiotemporal images of the left and right regions of the current frame group, and whether there are curves extending in the same direction. If the target spatiotemporal image is a spatiotemporal image of the upper and lower regions, the pan detection unit 31 performs the same determination for the spatiotemporal images of the upper and lower regions.

[0093] Here, if there are no curves extending in opposite directions at the same time, and there are curves extending in the same direction (Yes in step S206), the pan detection unit 31 sets the detection result to "pan" in step S207, terminates the pan detection process, and proceeds to step S3 (Figure 21). On the other hand, if there are curves extending in the opposite direction at the same time, or if there are no curves extending in the same direction (No in step S206), the pan detection unit 31 sets the detection result to "shake" in step S208, terminates the pan detection process, and proceeds to step S3 (Figure 21).

[0094] Furthermore, if the bread detection unit 31 detects a straight line in step S201 (Yes in step S201), in step S209, it sets the detection result to "fixed," terminates the bread detection process, and proceeds to step S3 (Figure 21). Returning to Figure 21, we will continue the explanation of the operation of the camera work detection device 1.

[0095] In step S3, the cut point detection unit 34 detects the cut point from the spatiotemporal image stored in the storage unit 40. Here, the cut point detection unit 34 detects the cut point using the first spatiotemporal image and the last spatiotemporal image searched by the pan detection unit 31. The cut point detection unit 34 detects the cut point from the absolute value of the difference in the total number of pixels of edges between adjacent rows in the spatiotemporal image. Then, the camera work detection device 1 proceeds to step S5.

[0096] In this example, the first spatiotemporal image generated in step S1 and the last spatiotemporal image generated in step S2 are used in step S3. However, it is also acceptable to generate the last spatiotemporal image before executing step S2, use that result to execute step S3 first, and then execute step S2. Furthermore, if step S2 finishes before generating the last spatiotemporal image, the last spatiotemporal image can be generated separately before executing S3.

[0097] In parallel with steps S2 and S3, in step S4, the zoom detection unit 32 detects the zoom level from the spatiotemporal image stored in the storage unit 40 (zoom detection process).

[0098] Now, with reference to Figure 23, the zoom detection process in step S4 will be described in detail. In step S400, the zoom detection unit 32 instructs the edge continuity row detection unit 20 to perform a shift-and-run operation on the spatiotemporal image for zoom detection using a predetermined number of shift amounts ("for zoom"). This "for zoom" shift amount is, for example, in the range of ±11 pixels and does not include a shift of 0 pixels. Then, the "presence or absence of left region correlation" and "presence or absence of right region correlation" in the spatiotemporal images of a pair of left and right regions are logically ORed in the left and right regions of the predetermined number of spatiotemporal images, and the results are combined into a matrix as shown in Figure 19(a). This predetermined spatiotemporal image is an image generated in the central region of the frame, as shown in Figures 8B and 8C.

[0099] In step S401, the zoom detection unit 32 determines whether or not the shift and processing has been completed for all predetermined spatiotemporal images.

[0100] If the shift and processing for all predetermined spatiotemporal images has not yet been completed (No in step S401), then in step S402, the spatiotemporal image generation unit 10 generates the next spatiotemporal image and stores it in the storage unit 40. Then, the zoom detection unit 32 returns to step S400. On the other hand, if the shift and processing for all predetermined spatiotemporal images is completed (Yes in step S401), the zoom detection unit 32 proceeds to step S403.

[0101] In step S403, the zoom detection unit 32 determines whether curves in opposite directions exist simultaneously in the spatiotemporal images of the left and right regions. If the target spatiotemporal image is a spatiotemporal image of the upper and lower regions, the zoom detection unit 32 performs the same determination for the spatiotemporal images of the upper and lower regions.

[0102] If there are curves in opposite directions simultaneously (Yes in step S403), the zoom detection unit 32 sets the detection result to "Zoom" in step S404, terminates the zoom detection process, and proceeds to step S5 (Figure 21). On the other hand, if there are no curves in the opposite direction simultaneously (No in step S403), the zoom detection unit 32 sets the detection result to "blur" in step S405, terminates the zoom detection process, and proceeds to step S5 (Figure 21). Returning to Figure 21, we will continue the explanation of the operation of the camera work detection device 1.

[0103] Steps S2, S3, and S4 will process the spatiotemporal images of the left and right regions and the spatiotemporal images of the top and bottom regions in parallel, respectively. As a result, the camera work detection device 1 detects pan, zoom, and cut points in both the horizontal and vertical directions.

[0104] In step S5, the judgment and synthesis unit 33 determines the horizontal and vertical camera movements based on the detection results from steps S2 and S4. If both detection results from steps S2 and S4 are "blurry", the camera movement in that direction is determined to be "blurry". Then, as explained in Figure 20, the judgment and synthesis unit 33 determines the camera work of the frame group from the detection results in the horizontal direction (W) and the detection results in the vertical direction (H), and outputs it as the detection result.

[0105] In step S6, the spatiotemporal image generation unit 10 determines whether or not there is a next frame. Then, if there is a next frame (Yes in step S6), the camera work detection device 1 returns to step S1 and continues its operation. On the other hand, if there is no next frame (No in step S6), the camera work detection device 1 terminates its operation. Through the above operations, the camera work detection device 1 can accurately detect camera work from the camera image using minimal calculations.

[0106] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and includes design changes and the like that that do not depart from the spirit of the present invention. For example, in this case, the camera work detection device 1 includes a judgment and synthesis unit 33 and outputs the detection result of the camera work for each frame group, but it may also output the detection results separately in the horizontal and vertical directions. In that case, the judgment and synthesis unit 33 can be omitted from the camera work detection device 1.

[0107] In this configuration, the camera work detection device 1 is equipped with a cut point detection unit 34. However, the camera work detection device 1 may not perform cut point detection, but only pan and zoom detection. In that case, the camera work detection device 1 can be configured without the cut point detection unit 34.

[0108] Furthermore, although the above-described embodiment assumed that the camera work detection device 1 is an independent piece of hardware, the present invention is not limited thereto. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the camera work detection device 1 described above. This program may be distributed via a communication line, or it may be written to a recording medium such as a CD-ROM or flash memory and distributed. [Explanation of Symbols]

[0109] 1. Camera work detection device 10. Spatiotemporal Image Generation Unit 11 Spatiotemporal image generation unit for bread detection 12 Spatiotemporal image generation unit for zoom detection 20 Edge continuous row detection unit 30 Camera work judgment unit 31 Bread detection unit 32 Zoom detection unit 33 Judgment synthesis section 34 Cut point detection unit 40 Storage section

Claims

1. A camera work detection device that detects camera work for each group of frames of a predetermined time length that constitutes a camera image, A spatiotemporal image generation unit generates a horizontal spatiotemporal image in which, in the left and right regions of the frame, a sequence of pixels with binarized edges is arranged for the number of frames in the frame group, with each horizontal block spaced at a predetermined vertical interval; and a vertical spatiotemporal image in which, in the upper and lower regions of the frame, a sequence of pixels with binarized edges is arranged for the number of frames in the frame group, with each vertical block spaced at a predetermined horizontal interval. An edge continuation row detection unit performs a shift operation with multiple different shift amounts for each row in the horizontal and vertical spatiotemporal images, and performs a logical AND operation with the previous row, detecting rows where the maximum number of true values ​​exceeds a threshold as edge continuation rows. A camera work detection device comprising: a camera work determination unit that determines the camera work based on the length in the time direction, which is the total number of consecutive edge rows, and the length in the spatial direction, which is the sum of the shift direction and shift amount in the shift calculation performed on the consecutive edge rows.

2. The camera work detection device according to claim 1, characterized in that the camera work determination unit detects the horizontal pan of the camera work from the direction and length of continuous edges in the horizontal spatiotemporal image generated from the left region and the right region, and the pan detection unit detects the vertical pan of the camera work from the direction and length of continuous edges in the vertical spatiotemporal image generated from the upper region and the lower region.

3. The camera work detection device according to claim 2, characterized in that the pan detection unit detects whether the edges are continuous in a straight line or a curve, starting from the spatiotemporal images of the current frame group that are closest to the coordinates of the spatiotemporal images in the previous frame group where the edges were detected as straight lines or curves, in each spatiotemporal image of the horizontal and vertical directions, and determines that no pan has occurred in the current frame group if the edges are continuous in a straight line in even one spatiotemporal image, and determines that pan has occurred in the current frame group if the edges are continuous in a curve and the edges are not continuous in a straight line in all other spatiotemporal images.

4. The spatiotemporal image generation unit generates spatiotemporal images for zoom detection by logarithmically transforming the spatial coordinates of the spatiotemporal images in the horizontal and vertical directions, respectively. The camera work detection device according to claim 1, wherein the camera work determination unit comprises a zoom detection unit that detects the zoom of the camera work from the direction and length of continuous edges in the horizontal and vertical spatiotemporal images for zoom detection.

5. The camera work detection device according to claim 1, further comprising a cut point detection unit that detects, as a cut point, frames corresponding to rows in two or more spatiotemporal images in either the horizontal or vertical spatiotemporal image direction, where the absolute value of the difference in the total number of pixels of edges with the previous row is equal to or greater than a threshold value.

6. A program for causing a computer to function as a camera work detection device according to any one of claims 1 to 5.

Images