Image processing apparatus, image processing method, and program

Abstract

To determine a more appropriate cut-out region when generating a cut-out image from the cut-out regions in a plurality of images included in a dynamic image.SOLUTION: An image processing device comprises: acquisition means for acquiring a dynamic image of a processing subject; detection means for detecting an object from images included in the dynamic image; determination means for determining a position of a region of interest in the image included in the dynamic image based on a detection result of the detection means with respect to the image; derivation means for deriving a cut-out locus, which is a locus corresponding to movement of the position of the cut-out region, based on a locus corresponding to movement of the position of the region of interest determined by the determination means regarding the dynamic image and a reference position for the cut-out region; and generation means for generating a cut-out image from each of a plurality of images included in the dynamic image from a cut-out region identified based on the cut-out locus in each of the plurality of images.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to image processing technology.

Background Art

[0002] There is a technique for generating a cut-out image by cutting out a cut-out area, which is a partial area to be cut out in an image, from the image. At this time, by sequentially changing the position (or size) of the cut-out area in the image, it is possible to virtually change the imaging range of the imaging device. The process of changing the position (or size) of such a cut-out area is called digital PTZ. Also, in digital PTZ, there is a technique for determining a cut-out area in the image from the position information of one or more objects (such as a person) detected from the image, and generating a cut-out image by cutting out the cut-out area from the image. At this time, due to the variation in the positions of the one or more objects, the position of the cut-out area varies finely in each of the images constituting the moving image, and as a result, a moving image (cut-out moving image) composed of a series of cut-out images may become an image that is difficult for the user to view and has an uncomfortable feeling. Therefore, Patent Document 1 discloses a description to the effect that the position of the cut-out area is moved along a regression line obtained by linear regression analysis based on the position of a person in the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, the position of the cut-out area depends only on the position of a person in the image, and depending on the behavior of the person included in the image, the cut-out area may be determined as an area in the image that the user does not originally intend.

[0005] Therefore, an object of the present invention is to be able to determine a more appropriate extraction region when generating extraction images from extraction regions in a plurality of images included in a moving image.

Means for Solving the Problems

[0006] To solve the above problems, an image processing apparatus of the present invention includes the following configuration. That is, an acquisition unit that acquires a moving image to be processed, a detection unit that detects an object from an image included in the moving image, and based on a detection result of the detection unit with respect to the image included in the moving image, a determination unit that determines the position of a region of interest in the image, a trajectory corresponding to the movement of the position of the region of interest determined by the determination unit for the moving image, and for an extraction region group Based on the reference position cut A specifying unit that specifies feature points used for deriving an extraction trajectory, and for the moving image, determined by the determination unit said Based on the trajectory corresponding to the movement of the position of the region of interest and the reference position said An extraction trajectory that is a trajectory corresponding to the movement of the position of the extraction region said A derivation unit that derives an extraction trajectory, and in a plurality of images included in the moving image, an extraction image is generated from an extraction region specified based on the extraction trajectory said A generation unit, and the specifying unit specifies the feature points based on an intersection of the trajectory corresponding to the movement of the region of interest and the reference position.

Effects of the Invention

[0007] According to the present invention, when generating extraction images from extraction regions in a plurality of images included in a moving image, a more appropriate extraction region can be determined.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] Also, in each of the following embodiments, the imaging of a sports scene is taken as an example for explanation, but it is not limited thereto, and it is also applicable to the imaging of various events, concerts, and lecture scenes. Further, in each of the following embodiments, an image processing apparatus that functions as an imaging apparatus (network camera) capable of connecting to a network and communicating with other devices is described. However, it is not limited thereto, and it is also applicable to an image processing apparatus that functions as an imaging apparatus that cannot be connected to a network. Also, in each of the following embodiments, the image processing apparatus is described as having an imaging function, but it is not limited to the image processing apparatus having an imaging function, and the imaging function may be realized by a device different from the image processing apparatus, and the image processing apparatus may acquire the captured image from the different device. Furthermore, the image thus acquired may be a composite of images captured by a plurality of imaging devices by stitching processing or the like.

[0011] (Embodiment 1) The image processing apparatus in this embodiment acquires a moving image obtained by imaging a sports scene, and generates a cut-out image from a cut-out area in each image included in the moving image using the result of detecting players and balls included in the moving image. Here, a basketball scene is used as an example of a use case for explanation. In a basketball game, generally, there are cases where players gather on either the left or right court and attack while passing the ball, and cases where the offense and defense switch and the players move within the court, which are repeated and developed. Even in such cases, according to this embodiment, it is possible to realize the movement of the position of the cut-out area in accordance with the game progress, which suppresses the fine variation of the position of the cut-out area due to the influence of the fine movement of the players and follows the players and the ball without delay when the offense and defense switch.

[0012] Here, FIG. 1 shows the schematic configuration of the system in this embodiment. The system in this embodiment includes an image processing apparatus 100 that also functions as an imaging device and a client device 200. The image processing apparatus 100 and the client device 200 are connected in a state where they can communicate with each other via a network 300. In this embodiment, it is assumed that the image processing apparatus 100 is connected to the network and can communicate with other devices (such as a network camera). However, it is not essential to be connectable to the network, and a configuration in which the image processing apparatus 100 and the client device 200 are directly connected by an HDMI (registered trademark) or SDI cable may also be possible. Alternatively, a configuration in which an image taken and saved in the past is acquired and analyzed to create a cut-out video is also possible.

[0013] Based on operations by the user, the client device 200 sends a delivery request command to request the image processing device 100 to deliver a video (image) stream, and a setting command for setting various parameters. In response to the delivery request command, the image processing device 100 delivers the video stream to the client device 200, and stores various parameters in response to the setting command. The configuration of the image processing device 100 will be described later. The client device 200 can be realized by installing a predetermined program on a computer such as a personal computer, a tablet terminal, or a smartphone.

[0014] Subsequently, with reference to FIG. 2, the image processing device 100 will be described in more detail. FIG. 2(A) shows an example of the functional blocks of the image processing device 100, and FIG. 2(B) shows an example of the hardware configuration of the image processing device 100. In FIG. 2(A), the image processing device 100 includes, as a functional configuration, an image acquisition unit 211, a detection unit 212, an ROI determination unit 213, a setting unit 214, a feature point specification unit 215, a trajectory derivation unit 216, a generation unit 217, and an output unit 218. Each function shown in FIG. 2(A) is realized, for example, as follows. That is, it is realized by the CPU (Central Processing Unit) of the imaging device 100 executing a computer program stored in the ROM (Read Only Memory) of the image processing device 100, which will be described later with reference to FIG. 2(B).

[0015] The image acquisition unit 211 acquires a moving image captured by the imaging unit 221, which will be described later, or acquires a moving image from an external device (not shown).

[0016] The detection unit 212 performs an object detection process for detecting an object with respect to a plurality of images constituting the moving image acquired by the image acquisition unit 211. In the present embodiment, the detection unit 212 uses, as an object to be detected, for example, an object to be detected such as a player or a ball included in the image. The detection unit 212 may use, for example, a method of generating a discriminator that has learned the features of the object to be detected by a machine learning method and detecting the object to be detected in the image using the discriminator. The detection unit 212 stores the image acquired from the image acquisition unit 211 and information regarding the object detected from the image (position information and size information of the object) in the storage unit 222.

[0017] The ROI determination unit 213 calculates a region of interest (ROI) in the image based on the position information of the object detected by the detection unit 212. Note that ROI means Region of Interest. The ROI calculation unit 213 acquires information on the center position of the ROI in the image and stores it in the storage unit 222.

[0018] The setting unit 214 sets a reference position for the cutout region. Details of the setting of the reference position will be described later. The information on the region set by the setting unit 214 is stored in the storage unit 222.

[0019] The feature point identification unit 215 extracts feature points based on the information on the center position of the ROI acquired by the ROI determination unit 213 and the region acquired by the setting unit 214. The extracted feature points are stored in the storage unit 222.

[0020] The trajectory derivation unit 216 derives a trajectory (cutout trajectory) representing the movement of the position of the cutout region based on the feature points acquired by the feature point identification unit 215. The information on the cutout trajectory derived by the trajectory derivation unit 216 is stored in the storage unit 222.

[0021] Based on the cut-out trajectory derived by the trajectory derivation unit 216, the generation unit 217 performs a cut-out process on each of the plurality of images included in the moving image held in the storage unit 222 to generate a series of cut-out images. Further, the generation unit 217 generates a moving image (hereinafter referred to as a cut-out moving image) composed of a series of cut-out images generated by the cut-out process for each of the images constituting the moving image.

[0022] The output unit 218 outputs the cut-out moving image generated by the generation unit 217 to an external device using an I / F 224 described later.

[0023] Next, a hardware configuration example of the image processing apparatus 100 will be described with reference to FIG. 2(B). In FIG. 2(B), the image processing apparatus 100 includes an imaging unit 221, a storage unit 222, a control unit 223, an I / F 224, and an accelerator unit 225 as a hardware configuration.

[0024] The imaging unit 221 receives light imaged through a lens by an image sensor, converts the received light into electric charges, and acquires a moving image. As the image sensor, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor can be used. Also, a CCD (Charge Coupled Device) image sensor may be used as the image sensor. Note that the imaging unit 221 is described by taking the case of being included in the hardware configuration in this embodiment as an example, but it is not essential as a hardware configuration, and a moving image photographed and stored in the past may be acquired through the network 300.

[0025] The storage unit 222 is composed of both, or either one of, a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores programs for performing various operations / functions of the image processing apparatus 100. Further, the storage unit 222 can store data (commands and image data) and various parameters acquired from an external apparatus such as the client apparatus 200 via the I / F 224. For example, for each of the images constituting the moving image captured by the imaging unit 221, the storage unit 222 stores information related to camera settings such as the pan / tilt / zoom values when the image was captured, and the white balance and exposure when the image was captured. Further, the storage unit 222 can also store parameters related to the moving image, including the frame rate of the moving image to be captured and the size (resolution) of the moving image.

[0026] Further, the storage unit 222 can provide a work area used when the control unit 223 executes various processes. Furthermore, the storage unit 222 can also function as a frame memory or a buffer memory. Note that, as the storage unit 222, in addition to memories such as a ROM and a RAM, storage media such as a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, and a DVD may be used.

[0027] The control unit 223 is composed of a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and controls the entire image processing apparatus 100 by executing the programs stored in the storage unit 222. Note that the control unit 223 may control the entire image processing apparatus 100 in cooperation with the programs stored in the storage unit 222 and an OS (Operating System). Note that the control unit 223 may be composed of a processor such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit).

[0028] I / F 224 transmits and receives wired or wireless signals to communicate with the client device 200 via the network 300.

[0029] The accelerator unit 225 has a CPU, GPU (Graphics Processing Unit), FPGA (field-programmable gate array), etc., and a storage unit, and is a processing unit added to the camera mainly for performing high-performance processing by DeepLearning.

[0030] Subsequently, with reference to FIGS. 3 and 4, the processing of the image processing apparatus 100 in the present embodiment will be described. In the present embodiment, the case where the image processing apparatus 100 performs analysis processing such as object detection processing is shown. However, regarding the analysis processing, it may be performed by an accelerator unit added from the outside via USB or the like, or may be executed by a dedicated device having a GPU or FPGA.

[0031] In the present embodiment, a use case for a sports competition as shown in FIG. 3 is assumed. FIG. 3 shows an image 30 captured by the image processing apparatus 100 so as to include a plurality of players 310 playing basketball and the entire basketball court 320. Further, a cutout region 330 determined by the processing described later based on the detection results of the players and the ball included in the image 30 is shown.

[0032] When performing object detection processing on players and balls for each image 30 that makes up a moving image and determining a cutout region based on the results, the position of the cutout region may change between images of frames that are temporally before and after. This is not only due to changes caused by players moving according to the progress of the game, but also due to errors based on misdetection or detection omissions, and those caused by dribbling or passing the ball, etc., and also reflects changes that the camera should not move according to the movement per se. The image processing apparatus 100 of the present embodiment executes the following processing in order to suppress fluctuations in the position of the cutout region that is the object to be cut out. That is, feature points are specified based on a trajectory corresponding to the movement of the center position of the attention region determined for each image that makes up the moving image. Then, a trajectory that smoothly connects the specified feature points is derived as a cutout trajectory, and by shifting the cutout region according to the cutout trajectory, a smooth cutout moving image is generated as the displayed video.

[0033] FIG. 4 is a flowchart of the image processing apparatus 100 in the present embodiment. The flowchart shown in FIG. 4 is realized by the CPU of the image processing apparatus 100 executing a computer program stored in the ROM of the image processing apparatus 100 and is executed by the functional blocks of the image processing apparatus 100 shown in FIG. 2. In the present embodiment, it is assumed that the following processing is executed for a pre-recorded moving image (the moving image stored in the recording unit 222) as the processing target. In the present embodiment, as an example, the case where the position of the cutout region is changed in the pan direction (horizontal direction) on the image will be described.

[0034] In S410, the image acquisition unit 211 acquires settings related to the image. For example, the image acquisition unit 211 acquires parameters related to the image from the storage unit 222. Parameters related to the image include information on the imaging direction of the image processing apparatus (imaging apparatus) 100, the frame rate, and information on the size (resolution) of the image. In the present embodiment, as an example, it is assumed that information on the size of the image being 1920×1080 pixels and the frame rate being 30 fps is acquired as parameters related to the image.

[0035] Next, in S420, the setting unit 214 sets the reference positions. In the case of this embodiment, the positions of the left and right goals of the basket and the position of the center of the court are set as the reference positions. FIG. 5 is a diagram showing the set reference positions. As shown in FIG. 5, a reference position 510 corresponding to the left court, a reference position 520 corresponding to the center of the court, and a reference position 530 corresponding to the right court are set. Here, among the reference positions 510 to 530 set in the image, in the pan direction (horizontal direction of the image), the reference position 510 located at one end and the reference position 530 located at the other end are called range reference positions. Although details will be described later, the center position of the cutout area can be located within the range in the pan direction (horizontal direction of the image) from the reference position 510 to the reference position 530, which is the range reference position. Note that the reference positions can be set manually by the user, or can be automatically set by detecting landmarks characteristic of the target sport such as the goal ring. Also, like the reference position 520 in FIG. 5, the center may not be set individually, but the reference position 520 may be derived as the center line after setting the reference position 510 and the reference position 530.

[0036] Note that the setting of the reference positions may be changed according to the use case. In face-to-face sports such as basketball, for example, as shown in FIG. 5, reference positions are set at the center and both sides. Also, as similar examples, volleyball, tennis, etc. can be considered. On the other hand, even for the same face-to-face type, when the ground is wider as in soccer or rugby, the reference positions may be set more finely.

[0037] In S430, the image acquisition unit 211 acquires each image constituting the moving image to be processed. Note that the moving image to be processed is a pre-recorded video, and for example, it may be acquired from the storage unit 222 or from another external device. Further, the acquired image is an overhead image that can overlook the entire sports game like the image 30 shown in FIG. 3, and a cut-out image will be generated from this. As shown in the example of the basketball in FIG. 3, the overhead image can be obtained by shooting the entire court with a wide-angle camera, by converting the video shot with a fish-eye camera, or by synthesizing videos from multiple cameras. In the case of sports played on a large court such as soccer or rugby, it is difficult to include the entire game within the angle of view with a single wide-angle camera, so a fish-eye camera or a synthesized video of multiple cameras is often used.

[0038] In S440, the detection unit 212 performs object detection processing on each image included in the moving image of the processing target acquired in S430, and detects a target object. Here, it is assumed that the scene is a basketball scene shown in FIG. 3, and the detection targets are players and balls. As a method of object detection processing, a method based on machine learning, particularly DeepLearning, is known as a method that achieves high accuracy and high-speed processing corresponding to real-time processing. Specifically, YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), etc. can be mentioned. Here, the case of using SSD is shown. SSD is one of the methods for detecting each object from an image in which a plurality of objects are captured. In order to construct a discriminator that detects players and balls using SSD, images in which people and balls are captured are collected from a plurality of images and prepared as learning data. Specifically, the regions of people and balls in the image are extracted, and a file describing the coordinates and size of the center position thereof is created. The learning data prepared in this way is learned to construct a discriminator that detects human bodies and balls. Using the constructed discriminator, objects such as people and balls are detected from the image, and position information indicating the position of the region of the detected object and size information indicating the size of the region are acquired. The position information of the region of the detected object is represented by the XY coordinates of the center position of the region of the detected object as coordinates with the upper left of the image as the origin. The size information of the region of the detected object is represented by the number of pixels of the width and height of the region.

[0039] In S450, the ROI determination unit 213 determines the center position of the region of interest (ROI) based on the detection result of the object detected in S440. Here, multiple methods for determining the region of interest can be considered according to the use case. For example, the ROI determination unit 213 determines the center of gravity position of each of the positions of one or more players and the ball detected from the image as the center position of the ROI. At this time, when calculating the center of gravity position, weighted average may be used to increase the weight of either the player or the ball. For example, the ROI determination unit 213 may perform a weighted average with more weight given to the ball when calculating the center of gravity position of the positions of one or more players and the ball. Also, the weight can be changed according to the progress of the game. Specifically, in scenes where the ball position is more important, such as free throws in basketball or goal kicks in soccer, it is conceivable to increase the weight given to the ball position in the calculation of the center of gravity position. Also, it is conceivable to change the weight according to the team of the player. For example, when a sponsor of one of the teams wants to create a cut-out video centered on the players of that team, or when generating two patterns of cut-out moving images with weights assigned to the players of each team for the user to select. Furthermore, it is also possible to assign weights to specific players or specific plays.

[0040] Thus, there are various methods for determining the region of interest, and the user may be allowed to select the determination method. Also, multiple determination methods may be used to generate cut-out images in multiple patterns according to each of the multiple determination methods. In this embodiment, the ROI determination unit 213 determines the size of the region of interest (corresponding to the magnification rate or the zoom magnification of the cut-out video) to a predetermined size (for example, a size that can accommodate a half court), but it is not limited to this.

[0041] In S460, the control unit 223 determines whether there is image data for which the attention area should be determined. If there is still an image for which the attention area should be determined (Yes in S460), the process returns to S430 and continues the process for the next image. If there is no image for which the attention area should be determined (No in S460), the process proceeds to S470.

[0042] In S470, the feature point identification unit 215 extracts feature points for the cutout area from the center position of the attention area for each image acquired in S450 and the reference position set in S420.

[0043] Fig. 6 shows a schematic diagram of feature point extraction. Fig. 6(A) shows the locus 610 of the movement of the center position of the attention area in the pan direction for each image constituting the moving image to be processed. The horizontal axis represents the number of frames of the image constituting the moving image to be processed, and it is assumed that there are 0 to 10,000 frames. Also, as described in Fig. 3, the image 30 to be subjected to the cutout process has a size of 1920x1080 (pixels; hereinafter referred to as px), and the range in which the center position of the cutout area of the digital PTZ can move in the pan direction corresponds to the range of 0 to 1920 (px) corresponding to the horizontal width of the image 30. Note that the transition of the center position of the attention area shown in Fig. 6(A) here is a graph obtained by performing a smoothing process on the plot of the center position of the attention area for each frame acquired in S440.

[0044] Here, Fig. 6(B) shows the trajectory 610 of the movement of the center position of the region of interest shown in Fig. 6(A) with lines corresponding to the reference positions 510 to 530 superimposed. Here, the reference position 510, which is the range reference position corresponding to the left goal in the image 30, corresponds to the pan position 600 (px) in the pan direction. Therefore, in Fig. 6(B), the line of the reference position 510 is superimposed on the pan position 600 (px). Also, the reference position 520 corresponding to the center of the court in the image 30 corresponds to the pan position 975 (px) in the pan direction. Therefore, in Fig. 6(B), the line of the reference position 520 is superimposed on the pan position 975 (px). Also, the reference position 530, which is the range reference position corresponding to the right goal in the image 30, corresponds to the pan position 1350 (px) in the pan direction. Therefore, in Fig. 6(B), the line of the reference position 530 is superimposed on the pan position 1350 (px).

[0045] Also, Fig. 6(B) further shows the state of specifying the feature points. In the present embodiment, the feature point specifying unit 215 extracts, as feature points, the intersections of the trajectory 610 of the center position of the region of interest in the image of each frame with each of the reference positions 510 to 530. Also, the feature point specifying unit 215 adds additional feature points (additional feature points) to the start position of the trajectory 610 (the position at frame number 0) and the end position of the trajectory 610 (the position at frame number 10000), respectively.

[0046] Next, in S480, the trajectory derivation unit 216 derives, as the trajectory of the center position of the cut-out region (cut-out trajectory), a trajectory passing through each feature point specified in S470. The cut-out trajectory calculated in this way is the cut-out trajectory 620 shown in Fig. 6(C). Note that, as shown in Fig. 6(C), the range in which the center position of the cut-out region can be taken is the range from the reference position 510, which is the range reference position, to the reference position 530.

[0047] In deriving the cut-out trajectory, the trajectory 610 of the center position of the region of interest in the image of each frame and the feature points that are the intersections of each of the reference positions 510 to 530 are used. However, it is not limited to this, and additional feature points (additional feature points) may be further used. Here, a method of adding additional feature points will be described. In this case, the feature point specifying unit 215 calculates the degree of deviation between the straight line connecting two adjacent feature points among the feature points that are the intersections of the trajectory 610 and each of the reference positions 510 to 530, and the trajectory 610 between the two feature points. Note that the feature point specifying unit 215 derives the difference value of the pan position (px) for each position of the straight line connecting the two feature points and each position of the trajectory 610 between the two feature points, and calculates the maximum value among the derived difference values at each position as the degree of deviation. Then, the feature point specifying unit 215 compares the degree of deviation calculated for the straight line connecting the two feature points and the trajectory 610 between the two feature points with the threshold value, and adds an additional feature point between the two feature points when the degree of deviation is greater than the threshold value. Here, referring to FIG. 6(D), a case of deriving an additional feature point from adjacent feature points 630 and 640 among the feature points that are intersections of the trajectory 610 of the center position of the region of interest and any of the reference positions 510 to 530 will be described. The feature point specifying unit 215 calculates the difference value of the pan position (px) at each corresponding position of the straight line connecting the feature points 630 and 640 and the trajectory 610 between the feature points 630 and 640. The feature point specifying unit 215 calculates the maximum value among the difference values of the pan position (px) calculated for each position of the straight line connecting the feature points 630 and 640 as the degree of deviation. Then, the feature point specifying unit 215 compares the calculated degree of deviation with the threshold value. In this case, it is assumed that the calculated degree of deviation is greater than the threshold value, and the feature point specifying unit 215 adds an additional feature point 650. At this time, the additional feature point 650 is added to the position on the trajectory 610 when the difference value is the largest (at the maximum value) among the difference values of the pan position (px) at each corresponding position of the straight line connecting the feature points 630 and 640 and the trajectory 610 between the feature points 630 and 640. Also, in the same way, another additional feature point 660 that is added is also shown in FIG. 6(D).After the additional feature points are added, the trajectory derivation unit 216 derives a trajectory passing through each of the extracted feature points (including the additional feature points) as the trajectory of the center position of the cut-out area (cut-out trajectory). The cut-out trajectory derived in this way is the cut-out trajectory 670 shown in FIG. 6(E).

[0048] Note that the range of the pan position (px) where additional feature points can be added is limited to the range from the reference position 510, which is the range reference position, to the reference position 530. In other words, the range of the pan position (px) where additional feature points can be added is limited to the range from the reference position 510 at the left end in the pan direction (horizontal direction) of the image 30 to the reference position 530 at the right end in the pan direction (horizontal direction). In the case of the example in FIG. 6(B), additional feature points are added only in the range from the pan position 600 (px) to the pan position 1350 (px). That is, no additional feature points are added in the range of the pan position exceeding the pan position of the reference line 530 and in the range of the pan position below the pan position of the reference line 510. From this, the center position of the cut-out area is limited to the range from the range reference position 510 to the reference line 530. By making such a limitation, when one of the teams is continuously attacking within the left and right coats shown in FIG. 5, the camera work in the digital PTZ (in other words, the position of the cut-out area) can be fixed.

[0049] Returning to the description of FIG. 4, at S480, the trajectory derivation unit 216 smoothly connects each of the feature points shown in FIG. 6(D) obtained at S470 to derive a cut-out trajectory. The cut-out trajectory derived in this way is the cut-out trajectory 670 shown in FIG. 6(E). At this time, as described above, the range where the center position of the cut-out area can be taken is limited to the range from the reference position 510, which is the range reference position, to the reference position 530.

[0050] Note that the cut-out trajectory 620 shown in FIG. 6(C) was calculated from each feature point shown in FIG. 6(B), and the cut-out trajectory 670 shown in FIG. 6(E) was derived from each feature point shown in FIG. 6(D). Here, the method for deriving the cut-out trajectory will be described in more detail. Various methods can be considered for connecting each feature point to derive the cut-out trajectory, but a method that is smooth and highly continuous is required. For example, the trajectory derivation unit 216 uses the piecewise cubic Hermite interpolation method to derive the cut-out trajectory from each feature point. The piecewise cubic Hermite interpolation method is a method of dividing the domain into small regions and approximating each region with a polynomial up to the third degree, enabling smooth connection interpolation without increasing the computational complexity or memory. Therefore, by using this method, for example, overshoots that deviate from the trajectory 610 up and down can be suppressed, and smooth connection can be achieved. Also, since a cut-out trajectory that deviates from the trajectory 610 up and down is not generated, for example, in FIG. 3, a case where the cut-out region reaches the end in the pan direction of the coat can be suppressed. In other words, it is possible to suppress the cut-out region from being located in an area that the user does not intend (for example, an end area that deviates from the coat).

[0051] In S490, the generation unit 217 cuts out the portion of the cut-out region specified based on the cut-out trajectory acquired in S480 from each frame image included in the moving image to be processed, and generates a cut-out image. Then, the generation unit 217 sets a series of cut-out images obtained from each of the frame images as a cut-out moving image. Note that in order to generate a cut-out image from an image, it is necessary to calculate the four vertices of the cut-out region (such as the cut-out region 330 shown in FIG. 5) in the image. In the present embodiment, the cut-out region is specified such that it becomes the imaging region when a camera installed at the same position as the imaging device that captured the image 30 captures an image toward the center position (for example, the pan position 600 (px)) of the cut-out region indicated by the cut-out trajectory.

[0052] Referring to FIG. 7 here, a method for calculating the four vertices of the cut-out region from the image of each frame included in the moving image to be processed will be described. FIG. 7(A) shows the relationship between the image (image 30) constituting the moving image to be processed captured by the imaging device and the spherical coordinates with the position O of the imaging device as the origin. In this embodiment, it is assumed that the position and imaging range of the imaging device for imaging the moving image are fixed, and the spherical coordinates shown in FIG. 7(A) and the positional relationship of each image constituting the moving image do not change. As shown in FIG. 7(A), the center position of the image constituting the moving image is indicated by R, the horizontal direction of the image is represented by the x-axis, and the vertical direction of the image is represented by the z-axis. FIG. 7(B) shows the definition of the spherical coordinates (r, θ, φ).

[0053] The generation unit 217 identifies which frame the current image to be processed is among the images constituting the moving image to be processed. Here, for example, it is identified that the image to be processed is the 4000th frame. The generation unit 217 identifies the center position of the cut-out region at the 4000th frame based on the derived cut-out trajectory. In the case of the example of the cut-out trajectory 670 shown in FIG. 6(E), the generation unit 217 identifies that the center position of the cut-out region is 1350 (px) for the image to be processed. Then, the generation unit 217 converts the identified center position of the cut-out region in the image shown in FIG. 7(A) into a point U(θc, φc) on the spherical coordinates by image conversion. The point U at this time is shown in FIG. 7(C). Then, with the point U as the center, the generation unit 217 sets the horizontal angular field of view corresponding to the size of the cut-out region as 2Δθ and the vertical angular field of view as 2Δφ, and obtains the positions of the four vertices (F1, F2, F3, F4) on the spherical coordinates as follows.

[0054] [Number]

[0055] Then, the generation unit 217 acquires, as the four vertices of the cutout region, the results of converting the positions of the four vertices of the cutout region on the spherical coordinates back into the coordinates on the image to be processed shown in FIG. 7(A) again. An example of the cutout region specified by such processing is the cutout region 330 shown in FIG. 3. Then, the generation unit 217 cuts out the cutout region specified on the image to be processed, and generates a cutout image by performing distortion correction processing such as projective transformation on the image after cutting. The above-described processing is executed for each frame image of the moving image to be processed, and a cutout image is generated from each image.

[0056] Then, the cutout moving image composed of a series of cutout images generated in S490 is transmitted by the output unit 218 to another external device.

[0057] In the above description, the case where the center position of the cutout region is changed in the pan direction (image horizontal direction) in the image has been described. However, the present invention is not limited to this, and the cutout region may be changed in the tilt direction (image vertical direction) in the image. In this case, the image processing apparatus 100 derives the locus of the position of the attention region in the tilt direction for each of the plurality of images included in the moving image to be processed. Then, as described with reference to FIG. 6, the image processing apparatus 100 specifies feature points from the preset reference position and the locus of the position of the attention region in the tilt direction, and derives a cutout locus connecting each feature point. Thereafter, the image processing apparatus 100 generates a series of cutout images from the moving image to be processed using the cutout locus.

[0058] As described above, the image processing apparatus 100 in the present embodiment determines the position of the attention area for a plurality of images included in the moving image, and specifies feature points based on the trajectory of the movement of the position of the attention area and the reference position. Then, the image processing apparatus 100 derives a cut-out trajectory based on the specified feature points, and generates a cut-out image from the cut-out area specified according to the cut-out trajectory in each of the plurality of images included in the moving image. Further, the image processing apparatus 100 acquires a cut-out moving image composed of a series of cut-out images obtained from each of the plurality of images included in the moving image. By doing the above, it is possible to generate a cut-out moving image that does not blur as an image even if there are fine movements or dribbles of a player, etc., and to generate a cut-out moving image that follows the player or the ball without delay when the offense and defense switch. Also, by restricting the range of positions where the cut-out area can be taken by the reference position, it is possible to suppress the cut-out area from being determined as an area unintended by the user.

[0059] (Embodiment 2) Next, Embodiment 2 will be described. Note that the description of the same parts as in Embodiment 1 will be omitted. In Embodiment 1, a method of deriving an optimal cut-out trajectory for a moving image of a relatively long time and generating a cut-out moving image was described. In this embodiment, a case where the process of deriving the cut-out trajectory shown in S470 and S480 is executed by being divided at certain time intervals will be described.

[0060] Hereinafter, the processing of the image processing apparatus 100 in the present embodiment will be described with reference to the flowchart of FIG. 8. The flowchart shown in FIG. 8 is executed by the functional blocks of the image processing apparatus 100 shown in FIG. 2 realized by the CPU of the image processing apparatus 100 executing a computer program stored in the ROM of the image processing apparatus 100, for example. Note that the processing other than S820, S860, S880, and S895 is the same as the processing of FIG. 4 described in the first embodiment, and thus the description thereof will be omitted.

[0061] In S820, the setting unit 214 sets a target section (for how many seconds of video to be analyzed) corresponding to the range of the number of image frames to be analyzed for generating the cut-out moving image. The longer the target section is, the smoother the camera work in the digital PTZ considering the overall flow can be obtained. However, accordingly, the generation of the cut-out moving image becomes slower, and it takes time until the user can view the video. For example, in basketball, there are the 24-second rule (when on offense, a shot must be taken within 24 seconds) and the 14-second rule (when the offense gets a rebound, the next shot must be taken within 14 seconds). These are rules to give the game a sense of speed. Considering these rules, the possibility of staying on one side of the court for more than one minute is low. Therefore, here, the target section is set to 1 minute, which is 1800 frames at 30 fps. Here, it is also possible to change the target section according to the situation. In the following, the case where the target section is constant will be described. However, when the game develops rapidly, it is conceivable to reduce the target section.

[0062] In S860, the control unit 223 determines whether there are still image frames for which the region of interest should be determined. If the determination of the regions of interest for 1800 frames has not been completed according to the target section set in S820 (Yes in S860), the process returns to S430 and continues the process for the next image frame. If the determination of the regions of interest for 1800 frames has been completed (No in S860), the process proceeds to S470.

[0063] In S880, the trajectory derivation unit 216 derives a cutout trajectory using the feature points identified from the moving image in the target section of the currently processed target. Not limited to this, the trajectory derivation unit 216 may also derive a cutout trajectory for the moving image in the target section of the currently processed target using the feature points identified from the moving image in the target section immediately preceding the current target section. Specifically, the trajectory derivation unit 216 may derive a cutout trajectory from the feature points identified for the trajectory of the region of interest corresponding to 1800 frames of the current target section and the feature points in the vicinity of the end of the immediately preceding target section (for example, frames from 50 frames before the frame at the end). By adding the feature points in the moving image of the previous target section in this way, it is possible to generate a cutout moving image that maintains the continuity of the position of the cutout region even at the boundary between target sections.

[0064] In S895, the control unit 223 determines whether there is data for which a cutout image should be generated. If there is still a moving image consisting of the images of unprocessed frames and the cutout process for the next target section is necessary (Yes in S895), the process returns to S820 to set the next target section. If there is no moving image consisting of the images of unprocessed frames and the process for the next target section is not necessary (No in S895), the process shown in FIG. 8 ends.

[0065] As described above, in the image processing apparatus 100 according to the present embodiment, by generating a cutout moving image at any time by dividing the acquired moving image into target sections, it is possible to generate a cutout moving image in a state close to real time.

[0066] (Other Embodiments) Furthermore, the present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-described embodiments. The program may be supplied to a system or apparatus having a processor via a network or a storage medium. Also, the present invention can be realized by a circuit (for example, ASIC) that realizes one or more functions of the above-described embodiments.

[0067] Furthermore, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention. For example, combinations of the embodiments are also included in the disclosure of this specification.

Explanation of Reference Numerals

[0068] 10 Image processing system 100 Image processing apparatus 200 Client apparatus 300 Network

Claims

1. An acquisition means for acquiring a moving image to be processed; A detection means for detecting an object from an image included in the moving image; A determination means for determining the position of a region of interest in the image based on the detection result of the detection means with respect to the image included in the moving image; A specifying means for specifying feature points used for deriving a cut-out trajectory based on a trajectory corresponding to the movement of the position of the region of interest determined by the determination means for the moving image and a reference position for a cut-out region; A deriving means for deriving the cut-out trajectory, which is a trajectory corresponding to the movement of the position of the cut-out region, based on a trajectory corresponding to the movement of the position of the region of interest determined by the determination means for the moving image and the reference position; A generation means for generating a cut-out image from the cut-out region specified based on the cut-out trajectory in a plurality of images included in the moving image; comprising The specifying means specifies the feature points based on an intersection of a trajectory corresponding to the movement of the region of interest and the reference position. An image processing apparatus characterized by the above.

2. Having a calculating means for calculating a degree of deviation between a straight line between the feature points specified by the specifying means and a trajectory between the feature points; When the degree of deviation calculated by the calculating means is greater than a threshold value, additional feature points are specified between the feature points. The image processing apparatus according to claim 1, characterized by the above.

3. The image processing apparatus according to any one of claims 1 to 2, characterized in that the reference position is set based on a user operation.

4. The image processing apparatus according to any one of claims 1 to 2, characterized in that the reference position is set based on the position of a specific object detected from an image by the detection means.

5. The image processing apparatus according to any one of claims 1 to 4, characterized in that the moving image acquired by the acquisition means is a pre-recorded moving image.

6. An acquisition step of acquiring a moving image to be processed; A detection step of detecting an object from an image included in the moving image; A determination step of determining the position of a region of interest in the image based on the detection result of the detection step with respect to the image included in the moving image; A specifying step of specifying feature points used for deriving a cut-out trajectory based on a trajectory corresponding to the movement of the position of the region of interest determined by the determination step for the moving image and a reference position for a cut-out region; A derivation step of deriving a cut-out trajectory, which is a trajectory corresponding to the movement of the position of the cut-out area, based on the trajectory corresponding to the movement of the position of the attention area determined in the determination step for the moving image and the reference position; A generation step of generating a cut-out image from the cut-out area specified based on the cut-out trajectory in a plurality of images included in the moving image; characterized by comprising: In the specifying step, the feature point is specified based on an intersection point between the trajectory corresponding to the movement of the attention area and the reference position. An image processing method.

7. A computer program for causing a computer to function as the image processing apparatus according to any one of Claims 1 to 5.

Images