Image processing apparatus, image processing method, and program

Abstract

To provide a crop image in which a blur generated in a predetermined subject is reduced.SOLUTION: An information processing apparatus includes: a composition determination unit for determining a composition of an image in accordance with a predetermined rule according to a subject; an important region determination unit for determining an important region of a crop image corresponding to the composition of the image; and a motion information calculation unit for calculating motion information relating to motion of a predetermined subject in the important region.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and in particular to a technology for cropping an image. [Background technology]

[0002] When a moving subject is captured by a camera, blurring (so-called motion blurring) may occur in the captured image. As a result, image processing devices have been proposed that perform processing to remove blurring from the entire image in which blurring has occurred (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-182576 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, there are cases where a part of a captured image is cut out (so-called cropping is performed). However, since the cropped image is only a part of the captured image, even if the above-mentioned blur removal process is performed on the entire image, there is a risk that the blur generated in the subject cannot necessarily be removed from the cropped image part.

[0005] Therefore, the present technology proposes a technology that can provide a cropped image in which blurring occurring in a specified subject is reduced. [Means for solving the problem]

[0006] An image processing device according to the present technology includes a composition determination unit that determines a composition for an image in accordance with a predetermined rule corresponding to a subject, an important area determination unit that determines an important area for a cropped image corresponding to the composition in the image, and a motion information calculation unit that calculates motion information regarding the movement of a predetermined subject in the important area. This makes it possible to reduce blurring of a specific subject and generate blurring in other areas for each cropped image in accordance with the movement information of the specific subject. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an image processing system. [Diagram 2] FIG. 1 illustrates a configuration of an image processing device. [Diagram 3] FIG. 2 is a diagram showing the functional configuration of a CPU. [Figure 4] 11 is a flowchart showing a flow of image processing performed by a CPU. [Diagram 5] FIG. 13 shows a GUI for switching between a composition, the number of compositions, and a method for determining an important area. [Figure 6] 11A and 11B are diagrams showing examples of rules for automatically determining compositions, the number of compositions, and important areas. [Figure 7] FIG. 1 is a diagram showing an example of a composition with a person as the subject. [Figure 8] 13A to 13C are diagrams showing examples of compositions when an animal, a car, or a motorcycle is the subject. [Figure 9] 13A and 13B are diagrams showing an example of determining a composition from an image. [Figure 10] FIG. 13 is a diagram showing an example of a cropped image after blur removal processing. [Figure 11] FIG. 13 is a diagram illustrating an example of a GUI for allowing a user to select a composition. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The embodiments will be described below in the following order. <1. Image processing system> 2. Configuration of image processing device 12 and overview of image processing <3. Specific Examples> <4. Modifications> <5. Summary> <6. This Technology>

[0009] In this technology, "image" includes both still images and videos. Furthermore, "image" does not only refer to the state in which it is displayed on the display unit, but also refers to image data that is not displayed on the display unit. Moreover, the term "subject" refers not only to an object captured by the fixed camera 11, but also includes the subject image captured in the image. Moreover, the term "subject" refers not only to a person, but also to various things such as animals, objects, and letters, and further includes parts (regions) of these. Moreover, the "composition" refers to a partial area (range) of an image. Moreover, the term "cropped image" refers to the portion of the entire image that corresponds to the composition. Moreover, the "important area" refers to an area that corresponds to a specific subject determined within the cropped image. Moreover, "motion information" refers to information about the motion (blur) in the important region.

[0010] <1. Image processing system> An image processing system to which the present technology can be applied will be described. FIG. 1 is a diagram illustrating an image processing system 1. As shown in FIG.

[0011] As shown in FIG. 1, the image processing system 1 includes a fixed camera 11, an image processing device 12, and a display device 13.

[0012] The fixed camera 11 is a camera that captures an image with a fixed position and a fixed angle of view, and captures a moving image in this embodiment. In the example of Fig. 1, persons 2a to 2c are provided as the subject 2, and the fixed camera 11 captures the subject 2 including these persons 2a to 2c. In Fig. 1, the house illustrated behind the persons 2a to 2c is the background.

[0013] The image processing device 12 receives an image captured by the fixed camera 11. The image processing device 12 cuts out (crops) a part of the image input from the fixed camera 11, removes (reduces) blurring of a predetermined subject 2 in the cut cropped image, and outputs the cropped image to the display device 13. The detailed processing performed by the image processing device 12 will be described later.

[0014] The display device 13 includes a display unit such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diodes) display, and displays the cropped image input from the image processing device 12. One or more display devices 13 are provided. The display device 13 may be the display unit 27 (see FIG. 2) of the image processing device 12.

[0015] In the image processing system 1, a fixed camera 11 and an image processing device 12, and the image processing device 12 and a display device 13 are each connected by wire, wirelessly or via the Internet.

[0016] For example, the image processing system 1 is assumed to be used in such a manner that the fixed camera 11 and the image processing device 12 are connected by wire, the image processing device 12 and the display device 13 are connected via the Internet, and cropped images captured by the fixed camera 11 and subjected to various processes by the image processing device 12 are distributed to a large number of display devices 13. However, this is just one example, and other examples of use may be used.

[0017] 2. Configuration of image processing device 12 and overview of image processing FIG. 2 is a diagram showing the configuration of the image processing device 12. As shown in FIG. The image processing device 12 is a device capable of information processing, particularly image processing, such as a computer device. Specific examples of the image processing device 12 include a personal computer, a workstation, a mobile terminal device such as a smartphone or a tablet, and a video editing device. The image processing device 12 may also be a computer device configured as a server device or a computing device in cloud computing. The image processing device 12 may also be provided in the fixed camera 11.

[0018] As shown in FIG. 2, the image processing device 12 includes a central processing unit (CPU) 21, a read only memory (ROM) 22, and a random access memory (RAM) 23.

[0019] The CPU 21 executes various processes according to a program stored in the ROM 22 or a program loaded from the storage unit 28 to the RAM 23. In addition, the RAM 23 also stores data and the like necessary for the CPU 21 to execute various processes as appropriate.

[0020] The CPU 21, the ROM 22, and the RAM 23 are interconnected via a bus 24. To this bus 24, an input / output interface 25 is also connected.

[0021] An input unit 26 consisting of operators and operation devices is connected to the input / output interface 25. For example, the input unit 26 may be various operators and operation devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, a remote controller, or the like. An operation by the user is detected by the input unit 26, and a signal corresponding to the input operation is interpreted by the CPU 21.

[0022] Furthermore, a display unit 27 such as an LCD or an OLED display is connected to the input / output interface 25 either integrally or separately. The display unit 27 is also capable of displaying various images, operation menus, icons, messages, etc., on the display screen based on instructions from the CPU 21, that is, as a GUI (Graphical User Interface).

[0023] A storage unit 28 configured with a HDD (Hard Disk Drive), solid-state memory, etc., and a communication unit 29 are also connected to the input / output interface 25. The storage unit 28 can store various data and programs.

[0024] The communication unit 29 performs communication processing via a transmission path such as the Internet, and communication with various devices such as the fixed camera 11 and the display device 13 via wired communication, wireless communication, bus communication, and the like.

[0025] Furthermore, a drive 30 is connected to the input / output interface 25 as required, and a removable recording medium 30a such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately loaded therein. Images, various computer programs, and the like can be read from the removable recording medium 30a by the drive 30. The read data is stored in the storage unit 28, and images and sounds contained in the data are output on the display unit 27. In addition, the computer programs, etc. read from the removable recording medium 30a are installed in the storage unit 28 as necessary.

[0026] In the image processing device 12, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 29 or via the removable recording medium 30a. Alternatively, the software may be stored in advance in the ROM 22, the storage unit 28, etc.

[0027] Fig. 3 is a diagram showing a functional configuration of the CPU 21. Fig. 4 is a flowchart showing the flow of image processing performed by the CPU 21. Note that the image processing shown in Fig. 4 is performed for each frame of image. However, this image processing may be performed appropriately every several frames instead of every frame.

[0028] As shown in FIG. 3, the CPU 21 functions as an image recognition unit 31, a composition determination unit 32, an important area determination unit 33, a movement information calculation unit , a blur removal unit 35, and an image output unit .

[0029] As shown in Fig. 4, the image recognition unit 31 acquires an image (one frame of image) from the fixed camera 11 in step S1, and performs image recognition processing on the image in step S2. The image recognition processing is processing for detecting and recognizing a subject 2 from an image, and performs various types of processing such as object detection, object recognition, individual recognition, skeletal detection, saliency recognition, moving object recognition, character recognition, pupil detection, face detection, emotion recognition, and speaker recognition. These processes are performed using known techniques, and a description of each process will be omitted here. Also, it is sufficient to perform one or more of these processes as appropriate, and it is not necessary to perform all of them.

[0030] For example, the image recognition unit 31 performs skeleton detection on the image and detects the joint positions and bones of a person in the image. The image recognition unit 31 also performs face detection on the image and links the skeleton detection result to the detected face area. The image recognition unit 31 also performs individual identification to identify an individual using the face detection result, and links the identified individual to the skeleton detection result.

[0031] When the image recognition process is completed, in step S3, the composition determination unit 32 performs a composition determination process for determining a composition for the image acquired from the fixed camera 11 according to a predetermined rule corresponding to the subject 2. In the composition determination process, the composition determination unit 32 may automatically determine the composition based on the result of the image recognition process, or may determine the composition according to a rule designated by the user. Note that "automatically" means that the CPU 21 determines the setting in accordance with a predetermined rule even if the user does not specify the rule, and this also applies hereinafter.

[0032] In the composition determination process, the composition determination unit 32 may automatically determine the number of compositions based on the result of the image recognition process, or may determine the number of compositions designated by the user. Therefore, in the composition determination process, one or more compositions are determined for one image. In addition, since the image portion corresponding to the composition determined by the composition determination process is output to the display device 13 as a cropped image, it can also be said that the composition determination unit 32 determines the cropped image from the image acquired from the fixed camera 11.

[0033] When the composition determination process is completed, in step S4, the important area determination unit 33 performs an important area determination process for each composition determined in step S3, i.e., for each cropped image, to determine an area of ​​the subject 2 from which blur is to be removed as an important area. In the important region determination process, the important region determination unit 33 may automatically determine the important region in accordance with the rules used when the composition was determined, or may determine the important region in accordance with rules specified by the user.

[0034] When the important region determination process is completed, in step S5, the motion information calculation unit 34 performs a motion information calculation process to calculate motion information (blur information) of the important region for each composition determined in step S3, that is, for each cropped image. Here, since the position and angle of view of the fixed camera 11 are fixed, no blurring occurs in the image when the subject 2 is not moving. On the other hand, when the subject 2 is moving, blurring occurs in the image according to the speed and direction of the subject 2's movement. Therefore, the motion information calculation unit 34 calculates, for example, a PSF (Point Spread Function) as the motion information in the important region. The PSF is a function that defines the extent of the blur that occurs depending on the speed and direction of movement of the subject 2. Therefore, by calculating the PSF through the motion information calculation process, it is possible to calculate the extent of the blur that occurs in the important region. Since the PSF can be calculated by a known method, a detailed description thereof will be omitted here.

[0035] When the motion information calculation process is completed, in step S6, the blur removal unit 35 performs a blur removal process for removing (reducing) blur for each composition, i.e., for each cropped image. In the blur removal process, the PSF calculated in step S6 is deconvolved for the entire cropped image. As a result, in the cropped image, blur in the important area (subject 2) is removed, and blur occurs in areas other than the important area (for example, the background). In this way, by generating a cropped image in which the blur of the subject 2 is removed and the background is blurred, the CPU 21 can generate a cropped image from the image acquired from the fixed camera 11, as if the moving subject 2 had been tracked and captured by a PTZ camera (pan-tilt-zoom camera).

[0036] Then, in step S7, image output unit 36 ​​outputs to display device 13 the cropped image in which the blur of subject 2 has been removed and the background has become blurred.

[0037] In the following, the composition determination process and the important area determination process will be mainly described in detail with reference to specific examples.

[0038] <3. Specific Examples> Fig. 5 shows a GUI for switching the method of determining the composition, the number of compositions, and the important area. In the composition determination process and the important area determination process, it is possible to switch between automatically determining the composition, the number of compositions, and the important area according to the result of the image recognition process, and determining the composition, the number of compositions, and the important area according to rules specified by the user. For example, before the above-mentioned image processing is started, the CPU 21 displays a GUI 41 as shown in Fig. 5 on the display unit 27 to allow the user to select which determination method to use. In Fig. 5, "AUTO" is an icon for selecting automatic determination, and "MANUAL" is an icon for selecting the user to specify the rule.

[0039] When "Auto" or "Manual" is selected for each by operating the input unit 26, the CPU 21 determines the composition, the number of compositions, and the important area by the determination method selected for each. Note that when "Manual" is selected, a GUI (not shown) for inputting the rule of the determination method is displayed on the display unit 27, and the user can specify each rule by operating the input unit 26.

[0040] [3-1. Specific Example 1] Next, the composition determination process and the important area determination process when "AUTO" is selected for the composition, the number of compositions, and the important area will be described. Fig. 6 is a diagram showing an example of rules for automatically determining the composition, the number of compositions, and important areas. Fig. 7 is a diagram showing an example of a composition with a person as the subject. Fig. 8 is a diagram showing an example of a composition with an animal, a car, or a motorcycle as the subject.

[0041] When "Auto" is selected, the composition determination unit 32 determines the composition according to a predetermined rule as shown in Fig. 6. Specifically, as shown in Fig. 6 and Fig. 7(a), compositions for one person include a close-up shot (face) composition 42a, a bust shot (above the chest) composition 42b, a waist shot (upper body) composition 42c, a knee shot (above the knees) composition 42d, and a full figure (whole body) composition 42e.

[0042] As shown in Fig. 6 and Fig. 7(b), a composition for a plurality of people includes a composition 42f including the whole bodies of the plurality of people. As shown in Fig. 6 and Fig. 8(a), a composition for an animal (a dog in the figure) includes a composition 42g of the animal's face, a composition 42h of the animal's upper body, and a composition 42i of the whole animal. As shown in Fig. 6 and Fig. 8(b) and (c), a composition for a car or a motorcycle includes a composition 42j of the whole car and a composition 42k of the whole motorcycle.

[0043] In this way, when the composition and the number of compositions are automatically determined, the composition determination unit 32 may receive the result of the image recognition process by the image recognition unit 31 and determine all possible compositions based on the above-mentioned rules, or may randomly select from among the possible compositions and determine a composition. Furthermore, for example, if the subject 2 is a single person, it may determine one of the compositions from a close-up shot to a full figure. Moreover, the composition determination section 32 may first determine the number of compositions, for example, randomly, and then determine the compositions based on the above-mentioned rules so as to achieve the determined number of compositions.

[0044] Fig. 9 is a diagram showing an example of determining a composition from an image. For example, when the subject 2 shown in Fig. 1 is imaged by the fixed camera 11, an image 43 as shown in Fig. 9(a) is acquired by the image processing device 12. In this example, it is assumed that the person 2a is not moving, and the people 2b and 2c are moving. It is also assumed that the person 2c is moving faster than the person 2b.

[0045] In such a case, in the image 43 captured by the fixed camera 11, the person 2a and the background are not blurred, but the people 2b and 2c are blurred. Also, in the image 43, the person 2c is more blurred than the person 2b.

[0046] Assume that image recognition unit 31 performs image recognition processing on such image 43, and as a result of the image recognition processing, for example, persons 2a to 2c are identified. Here, persons 2a to 2c are recognized as persons, and their body parts such as skeletons, faces, and eyes are also detected.

[0047] Then, the composition determination unit 32 determines compositions 44a to 44c as shown in Fig. 9(b), for example, according to the result of the image recognition process and the composition rules shown in Fig. 6. Composition 44a is determined to be a waist shot of person 2a. Composition 44b is determined to be a full figure of person 2c. Composition 44c is determined to be a composition including the whole bodies of persons 2a and 2b. Also, the number of compositions is determined to be three.

[0048] Once the composition is determined in this manner, important region determination unit 33 determines the important region according to the rules shown in Fig. 6. Specifically, the important region is determined to be the person's eyes in composition 42a which is a close-up shot, the person's face in composition 42b which is a bust shot, the person's face in composition 42c which is a waist shot, the upper body in composition 42d which is a knee shot, and the entire body in composition 42e which is a full figure.

[0049] In composition 42f including the whole bodies of multiple people, a specific person or multiple people are determined as the important region. In composition 42j of an entire car, the headlights or the entire car are determined as the important region. In composition 42k of an entire motorcycle, the entire motorcycle is determined as the important region.

[0050] Therefore, in the example shown in Figure 9(b), in composition 44a, the face of person 2a is determined as important region 45a, in composition 44b, the entire body of person 2c is determined as important region 45b, and in composition 44c, for example, the entire body of person 2b is determined as important region 45c.

[0051] When the important region is determined for each composition, the motion information calculation section 34 calculates a PSF for the important region for each composition.

[0052] 10 is a diagram showing an example of a cropped image after blur removal processing. When the PSF is calculated for the important region for each composition, the blur remover 35 performs deconvolution of the calculated PSF for the entire cropped image (entire composition).

[0053] For example, as shown in FIG. 10(a), in composition 44a, person 2a is not moving to begin with, and no blurring occurs in the cropped image corresponding to composition 44a. Therefore, in cropped image 46a generated by performing the blur removal process, no blurring occurs in person 2a or the background.

[0054] In composition 44b, person 2c is moving and the entire body of person 2c is blurred. Therefore, in cropped image 46b generated by performing blur removal processing, blurring of person 2c, which has been determined to be in important area 45b, is removed, and blurring occurs in the background (and person 2a).

[0055] In composition 44c, person 2b is moving and the entire body of person 2b is blurred. Therefore, in cropped image 46c generated by performing blur removal processing, blurring of person 2b, which has been determined to be in important area 45c, is removed, and blurring occurs in person 2a and the background.

[0056] In this way, when "Auto" is selected for the composition, number of compositions, and important area, a series of image processing operations is performed automatically by the CPU 21 without user operation, and as a result, one or more cropped images are generated in which blur of the specified subject 2 is removed (reduced) and blur is generated (increased) in the background (areas other than the specified subject 2).

[0057] [3-2. Specific Example 2] Next, the composition determination process and the important area determination process when "manual" is selected for the composition, the number of compositions, and the important area will be described.

[0058] When "Manual" is selected, the composition determination unit 32 determines the composition and the number of compositions according to rules previously specified by the user via the input unit 26. The rules specified by the user can specify various rules, such as a composition targeting a specific person (individual) or a composition targeting a person closest to a specific object among multiple people.

[0059] In addition, when the composition determination unit 32 is unable to determine the compositions for the number of compositions specified by the user according to the rules specified by the user, the composition determination unit 32 may determine the remaining compositions according to predetermined rules such as those described in the case of “Auto”.

[0060] 11 is a diagram illustrating an example of a GUI for allowing a user to select a composition. When "manual" is selected, the user may specify that the composition determination unit 32 extracts composition candidates according to the rules described in "auto" and determines the composition by selecting one of the extracted candidates.

[0061] In this case, as shown in FIG. 11, the composition determination unit 32 determines multiple composition candidates (here, candidates 48a to 48g) from the image 43 in accordance with the rules shown in FIG. 6, and displays the determined candidates on the display unit 27 as a selectable GUI 47.

[0062] Then, when a candidate is selected by the user via the input unit 26, the composition determination unit 32 determines (finally determines) the candidate composition selected by the user as the official composition. This allows the user to simply select a desired composition from the candidate compositions, and allows the user to determine a composition that is more desirable than the "auto" option.

[0063] Furthermore, the important region determination unit 33 determines an important region for each composition according to a rule previously specified by the user via the input unit 26. In accordance with the rule specified by the user, the face of only a specific person (individual) can be specified as an important region, and the upper body can be specified as an important region for other people.

[0064] If the important region determining unit 33 cannot determine the important region according to the rules specified by the user, the important region determining unit 33 may determine the important region according to predetermined rules such as those described in the case of "auto."

[0065] Furthermore, when “Manual” is selected, the user may specify that important area determination unit 33 extracts important area candidates according to the rules described for “Auto” and determines the important area by selecting from among the extracted candidates, just as in the case where composition candidates are displayed.

[0066] In this case, the important region determination unit 33 determines an important region for each composition according to the rules shown in Fig. 6. Then, when multiple important regions can be determined for one composition, the candidates are displayed on the display unit 27 as a selectable GUI.

[0067] Then, when a candidate is selected by the user via the input unit 26, the important region determining unit 33 determines (finally determines) the candidate important region selected by the user as a formal important region. This allows the user to simply select a desired important region from the candidate important regions, and allows the user to determine the intended important region more easily than with "Auto."

[0068] <4. Modifications> It should be noted that the embodiment is not limited to the specific example described above, and various modified configurations may be adopted.

[0069] For example, in the above embodiment, the image processing device 12 acquires images from the fixed camera 11. However, the image processing device 12 may acquire images from a PTZ camera. The PTZ camera is capable of capturing images by tracking a specific subject 2, but when the movement of the specific subject 2 is too fast to be tracked, for example, the image processing of the embodiment can be performed to remove blurring of the subject 2.

[0070] In the above embodiment, when a knee shot is determined as the composition, the important region determination unit 33 automatically determines the important region according to the composition (in the case of "auto") and determines the upper body as the important region. However, it is possible that the head, body, and arms of the upper body move separately. In this way, when it is expected that multiple subjects 2 (each body part, multiple people, etc.) will move separately, the important region determination unit 33 may determine each of the multiple subjects 2 (head, body, arms, multiple people, etc.) as an important region. Then, the motion information calculation unit 34 may calculate the PSF for a plurality of important regions and set the average value of the calculated PSFs as the PSF (motion information) of the composition. This makes it possible to remove (reduce) blur based on the average motion information of the subjects 2 that move in different ways.

[0071] Furthermore, the motion information calculation section 34 may calculate PSFs for a plurality of important regions, and may add a predetermined weighting to the calculated PSFs to obtain the PSF (motion information) of the composition. This allows, for example, the PSF of a subject 2 with a large area to be weighted more heavily than the PSFs of other subjects 2, thereby making it possible to more significantly reflect the influence of the subject 2 with a larger area than the other subjects 2 and remove (reduce) blur. Also, for example, the PSF of the subject 2 recognized as a speaker may be weighted more heavily than the PSFs of other subjects 2, so that the influence of the subject 2 recognized as a speaker is largely reflected to remove (reduce) blur. Also, for example, a large weight can be applied to the PSF of subject 2 recognized as a specific person designated by the user as an important person, and blur can be removed (reduced) by largely reflecting the influence of subject 2. In addition, for example, if the head, body, and arms of the same person are determined as multiple important regions, the head can be weighted at 70%, the body at 20%, and the arms at 10%, so that the weighting increases in the order of importance (subject 2), and blur can be removed (reduced) to reflect the influence of the important regions to a greater extent.

[0072] Furthermore, if the fixed camera 11 can record sound simultaneously with the video, the CPU 21 may perform a recognition process on the recorded sound, and the voice recognition result may be used to calculate the composition and important areas in the same way as the image recognition result. This allows, for example, a speaker to be identified from the voice recognition result, and the PSF of the identified speaker to be weighted more heavily than the PSFs of other subjects 2, so that the influence of the identified speaker is more significantly reflected to remove (reduce) blur. Also, for example, when a specific keyword (person's name, object name, etc.) is recognized, a composition corresponding to the keyword may be calculated with priority. Specifically, a composition may be determined to include a subject 2 corresponding to the recognized keyword, or a composition may be included to include a subject 2 corresponding to a keyword that has been recognized more times than a predetermined number among the recognized keywords. Also, weighting may be applied to the PSF of the subject 2 corresponding to the recognized keyword so that the weight increases in the order of the number of times the keyword has been recognized among the recognized keywords.

[0073] <5. Summary> According to the above embodiment, the following effects can be obtained. The image processing device 12 of the embodiment includes a composition determination unit 32 that determines the composition of an image in accordance with a predetermined rule corresponding to a subject 2, an important area determination unit 33 that determines an important area for a cropped image corresponding to the composition of the image, and a motion information calculation unit 34 that calculates motion information regarding the movement of a predetermined subject 2 in the important area. This enables the image processing device 12 to reduce blurring according to the motion information of the predetermined subject 2 for each cropped image. Therefore, the image processing device 12 can provide a cropped image in which blurring occurring in the predetermined subject 2 is reduced.

[0074] The image processing device 12 also includes a blur removal unit 35 that performs blur removal processing on the entire cropped image in accordance with the motion information. This makes it possible to provide a cropped image in which the blur of the specified subject 2 is reduced and the background is blurred, as if the specified subject 2 had been tracked and captured by a PTZ camera. Here, while a PTZ camera can usually capture only one image, the image processing device 12 can provide multiple cropped images at once by setting multiple compositions. Therefore, since there is no need to prepare multiple PTZ cameras, the cost of the image processing system 1 as a whole can be reduced.

[0075] Furthermore, when the subject 2 in the cropped image is moving, the blur removal unit 35 performs blur removal processing on the entire cropped image according to the motion information, thereby reducing the blur of the subject 2 in the cropped image and increasing the blur of areas other than the subject 2. As a result, when the subject 2 is moving, a cropped image that looks as if it was captured by tracking the subject 2 can be provided.

[0076] The image output unit 36 ​​outputs the cropped image that has been subjected to the blur removal process. This makes it possible to display the cropped image on the external display device 13, and for example, it is possible to display the cropped image on a plurality of display devices 13 simultaneously.

[0077] Moreover, the important region determining unit 33 determines an important region according to the rules set when the composition is determined. This makes it possible to determine the subject 2 that is most suitable for each composition as the important region, so that blurring of the subject 2 can be reduced with high accuracy.

[0078] Furthermore, when there are multiple subjects 2 moving differently in the cropped image, the important region determining section 33 determines each of the multiple subjects 2 as an important region. This makes it possible to reduce blurring depending on the parts that move differently.

[0079] Furthermore, when a plurality of important regions are determined for a cropped image, the motion information calculation section 34 calculates an average value of the motion information of the plurality of important regions as the motion information of the cropped image. As a result, when the subject 2 moves differently for each part or when a plurality of subjects 2 move differently, blurring can be reduced according to the average movement of these subjects.

[0080] Furthermore, when a plurality of important regions are determined in a cropped image, the motion information calculation section 34 calculates a value obtained by adding predetermined weights to the motion information of the plurality of important regions as the motion information of the cropped image. As a result, when the subject 2 moves differently for each part or when a plurality of subjects 2 move differently, blurring can be optimally reduced by adjusting the weighting.

[0081] The image processing device 12 also includes an image recognition unit 31 that performs image recognition processing to recognize the subject 2 from an image. This makes it possible to easily determine the composition and important areas, for example, without having to manually extract the subject 2.

[0082] Moreover, the composition determination unit 32 determines the composition based on the result of the image recognition process. This reduces the burden on the user because the composition is automatically determined when an image is captured.

[0083] Moreover, the composition determination unit 32 determines the number of compositions based on the result of the image recognition process. This reduces the burden on the user because the number of compositions is automatically determined when an image is captured.

[0084] Moreover, the composition determination unit 32 determines the composition based on rules specified by the user. This makes it possible to provide a cropped image in which the subject 2 is captured as intended by the user.

[0085] Moreover, the composition determination unit 32 determines the number of compositions designated by the user. This makes it possible to provide the user with the number of cropped images that he or she desires.

[0086] Furthermore, the composition determination unit 32 is capable of switching between determining the composition based on the result of the image recognition process or based on rules specified by the user. This makes it possible to provide a cropped image based on an automatically determined composition when, for example, there is no subject 2 that the user focuses on, and to provide a cropped image based on a composition specified by the user when there is a subject 2 that the user focuses on. Therefore, it is possible to reduce the burden on the user and provide a cropped image that reflects the user's intention.

[0087] Furthermore, the composition determination unit 32 determines a plurality of composition candidates for the image, allows the user to select one of the determined plurality of candidates, and finally determines the selected composition. This makes it possible, for example, to automatically present candidate compositions and then allow the user to select the composition that he or she desires most, thereby further reducing the burden on the user and providing a cropped image that meets the user's intentions.

[0088] Furthermore, the important region determining unit 33 determines a plurality of candidates for important regions for the cropped image, allows the user to select one of the determined plurality of important regions, and finally determines the selected important region. This makes it possible, for example, to automatically present candidates for important areas and then allow the user to select the important area that they most desire, thereby further reducing the burden on the user and providing a cropped image that is in line with the user's intentions.

[0089] Moreover, the image is captured by a fixed camera 11 . This makes it possible to provide a cropped image in which blurring of a specified subject 2 is reduced and background is blurred, as if the specified subject 2 was tracked and captured by a PTZ camera, for an image captured by a fixed camera 11 with a fixed position and angle of view.

[0090] Moreover, the image is captured by a PTZ camera. As a result, even if the PTZ camera cannot track the movement of the subject 2, it is possible to provide a cropped image in which the blur of the specified subject 2 is reduced and the background is blurred.

[0091] The image processing method also determines a composition for the image according to a predetermined rule corresponding to the subject 2, determines an important area for a cropped image corresponding to the composition of the image, and calculates motion information relating to the movement of the specified subject 2 in the important area. The program also causes the computer to execute a process of determining a composition for the image in accordance with predetermined rules corresponding to the subject 2, determining an important area for a cropped image of the image that corresponds to the composition, and calculating movement information relating to the movement of the specified subject 2 in the important area.

[0092] The program of the embodiment is a program that causes a processor such as a CPU or a DSP, or a device including these, to execute the above image processing.

[0093] Such a program can be pre-recorded in a HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU. Also, such a program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium can be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0094] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0095] <6. This Technology> The present technology can also be configured as follows. (1) a composition determination unit that determines a composition of an image according to a predetermined rule corresponding to a subject; an important region determining unit that determines an important region for a cropped image of the image that corresponds to the composition; a motion information calculation unit that calculates motion information related to a motion of the predetermined subject in the important region; An image processing device comprising: (2) A blur removal unit that performs blur removal processing on the entire cropped image in accordance with the motion information. The image processing device according to (1) is provided. (3) The blur removal unit is When the subject in the cropped image is moving, the blur removal process is performed on the entire cropped image according to the motion information, thereby reducing the blur of the subject in the cropped image and increasing the blur of areas other than the subject. An image processing device as described in (2). (4) an image output unit that outputs the cropped image that has been subjected to the blur removal processing; The image processing device according to (2) or (3), (5) The important region determination unit determines the important region according to a rule used when the composition is determined. An image processing device according to any one of (1) to (4). (6) The important region determination unit When there are a plurality of the objects that move differently in the cropped image, the plurality of the objects are determined as the important region. An image processing device according to any one of (1) to (5). (7) The motion information calculation unit When a plurality of the important regions are determined for the cropped image, an average value of the motion information of the plurality of the important regions is calculated as the motion information of the cropped image. An image processing device according to (6). (8) The motion information calculation unit When a plurality of the important regions are determined in the cropped image, a value obtained by adding the motion information of the plurality of the important regions with a predetermined weight is calculated as the motion information of the cropped image. An image processing device according to (6). (9) an image recognition unit that performs image recognition processing to recognize the subject from the image; The image processing device according to any one of (1) to (8), comprising: (10) The composition determination unit determines the composition based on a result of the image recognition processing. An image processing device according to (9). (11) The composition determination unit determines the number of compositions based on a result of the image recognition processing. An image processing device according to (9). (12) The composition determination unit determines the composition based on a rule specified by a user. An image processing device according to any one of (1) to (9). (13) The composition determination unit determines the number of compositions designated by a user. An image processing device according to any one of (1) to (9) and (12). (14) The composition determination unit is capable of switching between determining the composition based on the result of the image recognition process or based on a rule specified by a user. An image processing device according to (9). (15) The composition determination unit determines a plurality of composition candidates for the image, allows a user to select one of the determined plurality of candidates, and finally determines the selected composition. An image processing device according to any one of (1) to (14). (16) The important area determination unit determines a plurality of candidates for the important area in the cropped image, allows a user to select one of the determined plurality of important areas, and finally determines the selected important area. An image processing device according to any one of (1) to (15). (17) The image is captured by a fixed camera. An image processing device according to any one of (1) to (16). (18) The image is captured by a PTZ camera. An image processing device according to any one of (1) to (16). (19) The composition of the image is determined according to a set of rules based on the subject, determining an area of ​​importance for a cropped image of the image that corresponds to the composition; Calculating motion information regarding the motion of the predetermined subject in the important region Image processing methods. (20) The composition of the image is determined according to a set of rules based on the subject, determining an area of ​​importance for a cropped image of the image that corresponds to the composition; Calculating motion information regarding the motion of the predetermined subject in the important region A program that causes a computer to carry out processing. [Explanation of symbols]

[0096] 1. Image Processing System 11 Fixed Camera 12 Image processing device 13 Display device 31 Image Recognition Unit 32 Composition determination section 33 Important area determination department 34 Motion information calculation unit 35 Blur removal section 36 Image output section

Claims

1. a composition determination unit that determines a composition of an image according to a predetermined rule corresponding to a subject; an important region determining unit that determines an important region for a cropped image of the image that corresponds to the composition; a motion information calculation unit that calculates motion information related to a motion of the predetermined subject in the important region; An image processing device comprising:

2. A blur removal unit that performs blur removal processing on the entire cropped image in accordance with the motion information. The image processing device according to claim 1 .

3. The blur removal unit is When the subject in the cropped image is moving, the blur removal process is performed on the entire cropped image according to the motion information, thereby reducing the blur of the subject in the cropped image and increasing the blur of areas other than the subject. The image processing device according to claim 2 .

4. an image output unit that outputs the cropped image that has been subjected to the blur removal processing; The image processing device according to claim 2 .

5. The important region determination unit determines the important region according to a rule used when the composition is determined. The image processing device according to claim 1 .

6. The important region determination unit When there are a plurality of the objects that move differently in the cropped image, the plurality of the objects are determined as the important region. The image processing device according to claim 1 .

7. The motion information calculation unit When a plurality of the important regions are determined for the cropped image, an average value of the motion information of the plurality of the important regions is calculated as the motion information of the cropped image. The image processing device according to claim 6.

8. The motion information calculation unit When a plurality of the important regions are determined in the cropped image, a value obtained by adding the motion information of the plurality of the important regions with a predetermined weight is calculated as the motion information of the cropped image. The image processing device according to claim 6.

9. an image recognition unit that performs image recognition processing to recognize the subject from the image; The image processing device according to claim 1 .

10. The composition determination unit determines the composition based on a result of the image recognition processing. The image processing device according to claim 9 .

11. The composition determination unit determines the number of compositions based on a result of the image recognition processing. The image processing device according to claim 9 .

12. The composition determination unit determines the composition based on a rule specified by a user. The image processing device according to claim 1 .

13. The composition determination unit determines the number of compositions designated by a user. The image processing device according to claim 1 .

14. The composition determination unit is capable of switching between determining the composition based on the result of the image recognition process or based on a rule specified by a user. The image processing device according to claim 9 .

15. The composition determination unit determines a plurality of composition candidates for the image, allows a user to select one of the determined plurality of candidates, and finally determines the selected composition. The image processing device according to claim 1 .

16. The important area determination unit determines a plurality of candidates for the important area in the cropped image, allows a user to select one of the determined plurality of important areas, and finally determines the selected important area. The image processing device according to claim 1 .

17. The image is captured by a fixed camera. The image processing device according to claim 1 .

18. The image is captured by a PTZ camera. The image processing device according to claim 1 .

19. The composition of the image is determined according to a set of rules based on the subject, determining an area of ​​importance for a cropped image of the image that corresponds to the composition; Calculating motion information regarding the motion of the predetermined subject in the important region Image processing methods.

20. The composition of the image is determined according to a set of rules based on the subject, determining an area of ​​importance for a cropped image of the image that corresponds to the composition; Calculating motion information regarding the motion of the predetermined subject in the important region A program that causes a computer to carry out processing.

Images