Image processing device

Abstract

To segment a desired region out of a captured image with high accuracy.SOLUTION: An image processing device comprises: detection means which detects a subject region out of a captured image; acquisition means which acquires a distance map corresponding to the captured image; and determination means which determines a segmentation region being a region to segment out of the captured image, based on the subject region, the distance map, and a position designated in segmentation of the region out of the captured image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image processing device, and more particularly to a technique for cutting out an area from a captured image (a captured image). [Background technology]

[0002] Techniques for extracting a subject area from a captured image have been proposed. Non-Patent Document 1 discloses image matting, which is a trained model for extracting a subject area from a captured image. Also proposed is a technique for extracting a subject area from a captured image based on the distribution of defocus amounts in the captured image. Non-Patent Document 2 discloses a trained model for generating a distance map (two-dimensional information indicating the distribution of defocus amounts) from a captured image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Deep image matting”, N.Xu, B.Price, S.Cohen, and T.Huang, Proc. IEEE Conference on CVPR, 2017 [Non-patent document 2] “Depth map prediction from a single image using a multi-scale deep network”,David Eigen, Christian Puhrsch, and Rob Fergus, NeurIPS,2014 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional methods are unable to accurately extract desired regions. For example, with a method using image matting, even if a user wishes to extract the region of a person and their belongings, only the region of the person may be extracted, and the regions of the belongings may not be extracted. Also, even if a user wishes to extract the region of only one of multiple people (a specific person), the regions of multiple people may be extracted. With a method using a distance map, regions that are intricately blended into the background, such as a region of a person's hair or a region of animal fur, cannot be accurately extracted.

[0005] An object of the present invention is to provide a technique that enables a desired area to be cut out from a captured image with high accuracy. [Means for solving the problem]

[0006] A first aspect of the present invention is an image processing device characterized by having a detection means for detecting a subject area from a captured image, an acquisition means for acquiring a distance map corresponding to the captured image, and a determination means for determining a cut-out area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image.

[0007] A second aspect of the present invention is an image processing method characterized by having a detection step of detecting a subject area from a captured image, an acquisition step of acquiring a distance map corresponding to the captured image, and a determination step of determining a cut-out area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image.

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device. A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device. It is a medium. [Effects of the Invention]

[0009] According to the present invention, it is possible to extract a desired area from a captured image with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 2 is a block diagram of an area detection unit according to the first embodiment. [Figure 3] 3A and 3B are schematic diagrams of image matting and a distance map according to the first embodiment. [Figure 4] 5A to 5C are schematic diagrams illustrating processing of a distance map according to the first embodiment. [Figure 5] FIG. 2 is a block diagram of a cutout processing unit according to the first embodiment. [Figure 6] FIG. 10 is a schematic diagram of a clipped region superimposed image before updating according to the first embodiment. [Figure 7] 10 is a flowchart of a cutout process according to the first embodiment. [Figure 8] 5A to 5C are schematic diagrams illustrating processing of a distance map according to the first embodiment. [Figure 9] FIG. 4 is a schematic diagram of a small value selection process according to the first embodiment. [Figure 10] FIG. 10 is a schematic diagram of an updated clipped region superimposed image according to the first embodiment. [Figure 11] FIG. 10 is a block diagram of an area detection unit according to a second embodiment. [Figure 12] FIG. 10 is a schematic diagram of a small value selection process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Example 1 A first embodiment of the present invention will be described below. Fig. 1 is a block diagram showing the configuration of an image processing device 101 according to the first embodiment. The image processing device 101 is, for example, an imaging device such as a smartphone or a digital camera that captures an image of a subject.

[0012] The image processing device 101 has an optical system 102, an imaging unit 103, an A / D conversion unit 104, an image processing unit 105, a storage unit 106, a display unit 107, a control unit 108, a ROM 109, a RAM 110, an area detection unit 111, and a cutout processing unit 112. These are connected to a bus so that they can communicate with each other.

[0013] The optical system 102 is a group of lenses including a zoom lens and a focus lens, and forms a subject image (optical image of the subject) on the imaging unit 103. The imaging unit 103 is an imaging sensor such as a CCD or CMOS sensor, and performs photoelectric conversion on the subject image (optical image of the subject) formed on the imaging unit 103 by the optical system 102. The imaging unit 103 then outputs an analog signal (analog signal indicating the subject image) obtained by the photoelectric conversion to the A / D conversion unit 104. The arrangement of the multiple image sensors in the imaging unit 103 is, for example, a Bayer arrangement of RGB (red, green, and blue). The A / D conversion unit 104 converts the input analog image signal into digital image data, and outputs the obtained digital image data to the RAM 110.

[0014] The image processing unit 105 applies various image processing such as white balance adjustment, noise reduction, color interpolation (debayering), and gamma processing to the image data recorded in the RAM 110. The image processing unit 105 can obtain (generate) a main image and a thumbnail image as images after image processing. The thumbnail image is, for example, an image in which the resolution of the main image is reduced to VGA size.

[0015] The storage unit 106 is, for example, a memory card that is detachable from the image processing device 101. The image that has undergone image processing by the image processing unit 105 is stored in the storage unit 106 via the RAM 110 as a recorded image.

[0016] The display unit 107 is a display unit that can accept touch operations. For example, the display unit 107 is The image processing device 101 has a liquid crystal display and a touch screen. The touch screen detects contact of an operating object such as a finger or a stylus pen. For example, a capacitive method or a resistive film method is used to detect contact of the operating object with the touch screen. The image processing device 101 (control unit 108) determines the type of touch operation based on the position and time of contact detected by the touch screen. A touch operation is an operation performed on the touch screen by the operating object. Examples of touch operations include touch, release, tap, drag, swipe, flick, pinch in, and pinch out. When a touch operation is performed, the image processing device 101 (control unit 108) performs processing (control) according to the performed touch operation. The liquid crystal display displays images recorded in the RAM 110 or the storage unit 106, and displays an operation user interface for receiving instructions from the user.

[0017] For example, the user touches a subject that the user wants to cut out in the captured image (captured image) displayed on the display unit 107. Then, the control unit 108 controls the cutout processing unit 112 based on the touch position (touched position) to cut out the subject area at the touch position (the area of ​​the touched subject). Note that although an example of specifying a position by touch has been described, the method of specifying a position is not particularly limited, and for example, the user may specify a position by moving a cursor using a four-way key. Furthermore, the captured image may be an image (main image or thumbnail image) after image processing by the image processing unit 105, or may be an image (image output from the A / D conversion unit 104) before image processing by the image processing unit 105.

[0018] The control unit 108 controls the entire image processing device 101. For example, the control unit 108 is a CPU that reads a program from the ROM 109, expands it into the RAM 110, and executes it to control each unit of the image processing device 101. The ROM 109 is a non-volatile memory that stores data (for example, programs and parameters) required to control each unit of the image processing device 101. The RAM 110 is a volatile memory that temporarily stores data required to control each unit of the image processing device 101.

[0019] The region detection unit 111 determines a cutout region, which is a region to be cut out from the captured image, and outputs a cutout map indicating the cutout region.

[0020] The clipping processing unit 112 acquires the clipping map output from the region detection unit 111, and clips out the clipping region indicated by the clipping map from the captured image. The clipping processing unit 112 superimposes the contour line of the clipping region on the captured image, and outputs the captured image with the contour line superimposed. This makes it possible to display the captured image so that the clipping region is identifiable. Note that there is no particular limitation on the method for making the clipping region identifiable. For example, the entire clipping region may be emphasized, rather than just the contour of the region.

[0021] 2 is a block diagram showing the configuration of area detection unit 111. Area detection unit 111 has an image matting unit 201, a distance measurement unit 202, a distance map processing unit 203, an average distance acquisition unit 204, and a clipping map generation unit 205.

[0022] The image matting unit 201 detects a subject region from a captured image. For example, the image matting unit 201 detects the subject region using image matting (a trained model) disclosed in Non-Patent Document 1. By using image matting, it is possible to accurately detect regions that are intricately mixed into the background, such as a region of a person's hair or a region of animal fur. The image matting unit 201 outputs an image matting map indicating the detected subject region to the clipping map generation unit 205.

[0023] The distance measurement unit 202 acquires a distance map corresponding to the captured image (a map indicating the subject distance at each position in the captured image). The method for acquiring the distance map is not particularly limited. For example, the distance map may be acquired from the captured image using a trained model as in the method described in Non-Patent Document 2, or the distance map may be acquired showing the distribution of defocus amounts by performing phase-difference AF. The distance map may be included in advance as metadata in the file of the captured image, or may be acquired from the metadata. A distance map associated with the captured image may be stored in advance separately from the captured image, and the stored distance map may be acquired. The distance map may also be acquired from an external device.

[0024] 3A is a schematic diagram of an image matting map obtained by the image matting unit 201, and FIG. 3B is a schematic diagram of a distance map obtained by the distance measuring unit 202. FIGS. 3A and 3B show maps obtained (generated) from captured images of three people lined up at different distances from the image processing device 101 (image capture device). For example, an image is generated as the image matting map in which the detected subject area is bright (the signal level of the detected subject area is high) and the remaining area is dark (the signal level of the remaining area is low). Then, an image is generated in which the shorter the subject distance is, the brighter the image is (the shorter the subject distance is, the higher the signal level is) and the longer the subject distance is, the darker the image is (the longer the subject distance is, the lower the signal level is). In the first embodiment, it is assumed that an area in the distance map whose brightness (signal level) is equal to or greater than a threshold is the subject area indicated by the distance map.

[0025] Distance measurement unit 202 outputs the acquired distance map to distance map processing unit 203 and average distance acquisition unit 204. Average distance acquisition unit 204 uses the distance map acquired by distance measurement unit 202 to acquire the average value (average distance) of multiple subject distances indicated in the distance map for the user's touch area. The touch area is an area that includes the position designated (touched) by the user and its surrounding area, and may be interpreted as the area touched by the user. Distance map processing unit 203 processes the distance map acquired by distance measurement unit 202 based on the average distance acquired by average distance acquisition unit 204. Distance map processing unit 203 outputs the processed distance map to clip map generation unit 205.

[0026] An example of the operation of the distance map processing unit 203 and the average distance acquisition unit 204 will be described with reference to FIGS. 4(A) to 4(C).

[0027] 4(B) is a schematic diagram of a distance map obtained by distance measurement unit 202. In the distance map of FIG. 4(B), the subject distance of region 402 of a person's face is long (the brightness (signal level) of region 402 is low), and region 402 is not shown as a subject region. When a user touches region 402, average distance acquisition unit 204 acquires the average value (average distance) of multiple subject distances indicated in the distance map of FIG. 4(B) for region 402.

[0028] The distance map processing unit 203 generates a gain function 401 shown in FIG. 4A based on the average distance obtained by the average distance acquisition unit 204. The gain function 401 indicates the correspondence between the object distance and the gain value (coefficient by which the object distance is multiplied) of the object distance. Instead of the gain function 401, a table indicating the correspondence between the object distance and the gain value may be generated. The horizontal axis of FIG. 4A indicates the object distance, and the vertical axis of FIG. 4A indicates the gain value. As the object distance increases from t1 to t3, the gain value increases linearly from 0 to 2, and as the object distance increases from t3 to t4, the gain value decreases linearly from 2 to 0. The object distance t3 is the average distance obtained by the average distance acquisition unit 204, and the object distances t1 and t4 are determined based on the object distance t3. The object distance t2 is the object distance in region 403 of FIG. 4B.

[0029] The distance map processing unit 203 performs gain processing using the generated gain function to The distance map output from 202 is processed. In the gain processing, the object distance at each position is multiplied by a gain value corresponding to the object distance. FIG. 4C is a schematic diagram of the distance map after processing the distance map of FIG. 4B using the gain function of FIG. 4A. By using the gain function of FIG. 4A, the brightness (signal level) of region 402 in FIG. 4B is increased. As a result, in the distance map of FIG. 4C, region 404 corresponding to region 402 in FIG. 4B is shown as the object region. In the distance map of FIG. 4B, the brightness (signal level) of region 402 is lower than that of region 403. In contrast, in the distance map of FIG. 4C, the brightness (signal level) of region 404 corresponding to region 402 in FIG. 4B is higher than that of region 405 corresponding to region 403 in FIG. 4B.

[0030] In this way, the distance map is processed based on the multiple object distances in the touch area. While the example described above involves determining a gain function with a large gain value in a distance range that includes the average value of the multiple object distances in the touch area, a gain function with a small gain value in a distance range that includes the average value of the multiple object distances in the touch area may also be determined (details will be described later). Although the example described above involves using the average value of the multiple object distances in the touch area, the method of processing the distance map is not limited to this. The distance map may also be processed based on the maximum and minimum values ​​of the multiple object distances in the touch area. For example, the maximum and minimum values ​​of the multiple object distances in the touch area may be obtained, and a gain function with a large (or small) gain value in the distance range from the maximum to the minimum may be determined. The distance map may also be processed based on a histogram of the multiple object distances in the touch area. For example, the distance map may be processed using a large (or small) gain value in a region of a histogram similar to the histogram of the touch area. The distance map may also be processed using a process other than gain processing. The processed distance map may be a binary (1-bit) map in which the signal level at each position indicates whether or not the area is an object area.

[0031] The clipping map generation unit 205 generates a clipping map by a synthesis process that synthesizes the image matting map output from the image matting unit 201 and the distance map output from the distance map processing unit 203. In the synthesis process, for example, a large value selection process, a small value selection process, an addition process, a subtraction process, a multiplication process, and a division process are selectively performed on the two maps. In the synthesis process, the signal level of the image matting map or the distance map may be selected as the signal level of the clipping map based on the result of comparing the signal level (pixel value) of the distance map with a threshold. For example, if the signal level of the distance map is less than the threshold, the signal level of the image matting map may be selected as the signal level of the clipping map. Then, if the signal level of the distance map is equal to or greater than the threshold, the signal level of the distance map may be selected as the signal level of the clipping map.

[0032] 5 is a block diagram showing the configuration of the cutout processing unit 112. The cutout processing unit 112 includes a cutout unit 501 and a cutout region superimposing unit 502.

[0033] The clipping unit 501 clips out the clipping area indicated by the clipping map from the captured image. For example, the clipping unit 501 generates and outputs, as a clipped image, an image in which pixel values ​​in the region of the clipping map where the signal level is equal to or higher than a threshold are the same as those in the captured image and pixel values ​​in the region of the clipping map where the signal level is equal to or higher than the threshold are a predetermined value (for example, black pixel value).

[0034] The clipping area superimposing unit 502 detects the contour of the clipping area indicated by the clipping map, superimposes it on the captured image, and outputs the captured image after the contour has been superimposed (clip-out area superimposed image). The method for detecting the contour is not particularly limited, and any known method may be used.

[0035] FIG. 6 is a schematic diagram of the cutout region superimposed image. In FIG. 6, the image processing device 101 (imaging The contour of the cutout area is superimposed on a captured image of three people lined up at different distances from the camera (device). In FIG. 6, the area of ​​the three people is determined as the cutout area, and a gray contour surrounding the three people is superimposed as the contour of the cutout area. By superimposing and displaying the contour of the cutout area on the captured image, the user can be informed of the cutout area. Note that the color of the contour is not particularly limited.

[0036] Fig. 7 is a flowchart of the cut-out process performed by the image processing device 101. The cut-out process in Fig. 7 is realized by the control unit 108 expanding a program stored in the ROM 109 into the RAM 110 and executing it. For example, when a user issues an instruction to execute a predetermined image editing application, the cut-out process in Fig. 7 starts.

[0037] In step S701, the control unit 108 selects a captured image specified by the user from the plurality of captured images stored in the storage unit 106, and displays the selected captured image on the display unit 107. At this time, the image processing device 101 is set to an image display mode, which is an operation mode for displaying the captured image on the display screen.

[0038] In step S702, in response to a long tap on the display unit 107 (touch screen), the control unit 108 switches the set operation mode from the image display mode to a cutout mode, which is an operation mode for cutting out an area from a captured image. Then, the control unit 108 controls the area detection unit 111 to determine a cutout area, and controls the cutout area superimposition unit 502 to superimpose the outline of the cutout area on the captured image, and switches the image to be displayed to the cutout area superimposed image.

[0039] Image matting can determine the area of ​​a single subject with high accuracy. Therefore, in step S702, the cutout area is determined using only the image matting unit 201. In other words, the subject area detected by the image matting unit 201 is determined as the cutout area.

[0040] Note that the predetermined touch operation for setting the cut-out mode is not limited to a long tap. For example, it may be a double tap, a swipe, a flick, or a drag. A long tap is a touch operation in which the display surface is touched for longer than a first time period and then released, and a double tap is a touch operation in which, after a tap (a touch operation in which the display surface is touched for shorter than the first time period and then released), the touch operation is performed again before a second time period has elapsed. A swipe is a touch operation in which the display surface is traced, a flick is a touch operation in which the display surface is flicked, and a drag is a touch operation in which a point on the display surface is touched for longer than a third time period and then the touch position is moved.

[0041] The user views the clipping area superimposed image, confirms whether the desired area has been determined as the clipping area, and inputs the confirmation result to the image processing device 101. In step S703, the control unit 108 determines whether the user's desired area has been determined as the clipping area based on the input from the user. If the user's desired area has been determined as the clipping area, the process proceeds to step S712, and if the user's desired area has not been determined as the clipping area, the process proceeds to step S704.

[0042] If the area desired by the user has not been determined as the cut-out area, the user touches (specifies) the area to be corrected. In step S704, control unit 108 accepts the touch (specification) of the area to be corrected. The area to be corrected is the area to be added to the cut-out area or the area to be removed from the cut-out area. When the cut-out area superimposed image of FIG. 6 is displayed and the user wants to cut out only the person in the middle, the user touches the person at the forefront and the person at the back.

[0043] In step S705, the control unit 108 controls the average distance acquisition unit 204 to acquire the average value (average distance) of the multiple subject distances indicated by the distance map for the user touch area. At this time, it is assumed that the distance map has already been obtained by the distance measurement unit 202.

[0044] In step S706, the control unit 108 determines whether the position specified by the user (the portion to be corrected) is included in the current clipping area (the subject area detected by the image matting unit 201). If the specified position is not included in the current clipping area, the process proceeds to step S707; if the specified position is included in the current clipping area, the process proceeds to step S709. As will be described in detail later, if the process proceeds to step S707, a clipping area including a distance area based on the specified position is determined. The distance area based on the specified position is an area in which the subject distance within the distance range based on the specified position is indicated by the pre-processing distance map. In other words, the subject area at the specified position is added to the clipping area. If the process proceeds to step S709, a clipping area not including the distance area based on the specified position is determined. In other words, the subject area at the specified position is excluded from the clipping area.

[0045] In step S707, control unit 108 controls distance map processing unit 203 to process the distance map so as to indicate a subject area including a distance area based on a position (a location to be corrected) designated by the user. As described above, the distance area based on the designated position is an area in which the subject distance in the distance range based on the designated position is indicated by the distance map before processing. For example, as shown in FIG. 4(A), a gain function having a large gain value for a distance range including an average value of multiple subject distances in the touch area is determined, and the distance map is processed using this gain function.

[0046] In step S708, the control unit 108 controls the clipping map generation unit 205 to update the clipping area. The clipping area superimposed image to be displayed is also updated in accordance with the update of the clipping area. The control unit 108 determines the updated clipping area based on the current clipping area (the subject area detected by the image matting unit 201) and the subject area indicated by the processed distance map. For example, the control unit 108 determines an area included in at least one of the current clipping area and the subject area indicated by the processed distance map as the updated clipping area. Such an updated clipping area can be determined, for example, by a large value selection process that selects the larger of the signal level of the current clipping map (image matting map) and the signal level of the processed distance map.

[0047] In step S709, control unit 108 controls distance map processing unit 203 to process the distance map so as to indicate a subject area that does not include a distance area based on the position (location to be corrected) specified by the user. As described above, the distance area based on the specified position is an area in which the subject distance in the distance range based on the specified position is indicated by the distance map before processing. For example, as shown in FIG. 8(A), a gain function with a small gain value for the distance range that includes the average value of multiple subject distances in the touch area is determined, and the distance map is processed using this gain function.

[0048] FIG. 8(A) is a schematic diagram of a gain function, FIG. 8(B) is a schematic diagram of a distance map before processing, and FIG. 8(C) is a schematic diagram of a distance map after processing. With the cropped area superimposed image of FIG. 6 displayed, suppose the user touches areas 802 and 803 (the backmost and frontmost people) in FIG. 8(B) to crop out only the person in the middle. In the gain function 801 of FIG. 8(A), the gain value is 0 in the distance range including the subject distance t1 and the distance range including the subject distance t3, and the gain value is 2 between these distance ranges. The subject distance t1 is the subject distance of area 802 in FIG. 8(B), and the subject distance t3 is the subject distance of area 803 in FIG. 8(B). The subject distance t2 is the subject distance of the person in the middle. In the gain function 801 of FIG. 8(A), By processing the distance map of FIG. 8(B) using the In function, it is possible to obtain the distance map of FIG. 8(C) which shows only the area of ​​the person in the middle.

[0049] In step S709, the control unit 108 controls the clipping map generation unit 205 to update the clipping area. The clipping area superimposed image to be displayed is also updated in accordance with the update of the clipping area. The control unit 108 determines the updated clipping area based on the current clipping area (the subject area detected by the image matting unit 201) and the subject area indicated by the processed distance map. For example, the control unit 108 determines the area included in both the current clipping area and the subject area indicated by the processed distance map as the updated clipping area. Such an updated clipping area can be determined, for example, by a minimum value selection process that selects the smaller of the signal level of the current clipping map (image matting map) and the signal level of the processed distance map.

[0050] Figure 9 is a schematic diagram showing the small value selection process for the image matting map and the processed distance map. The image matting map has large values ​​in the areas of all three people, while the processed distance map has large values ​​only in the area of ​​the middle person. In this case, the small value selection process results in a clipping map with large values ​​only in the area of ​​the middle person.

[0051] Fig. 10 is a schematic diagram of the clipping region superimposed image after updating. By updating the clipping map to show only the area of ​​the person in the middle, the clipping region superimposed image in Fig. 10 shows a contour line showing only the area of ​​the person in the middle.

[0052] In step S711, the control unit 108 determines whether the region desired by the user has been determined as the cutout region based on the input from the user. If the region desired by the user has been determined as the cutout region, the process proceeds to step S711. If the region desired by the user has not been determined as the cutout region, the process proceeds to step S704.

[0053] In step S712, the control unit 108 cuts out the cut-out region in accordance with an instruction from the user (for example, a copy instruction).

[0054] As described above, according to the first embodiment, by determining the cut-out area based on the image matting map, the distance map, and the position specified when cutting out the area from the captured image, it becomes possible to cut out the desired area from the captured image with high accuracy.

[0055] <Example 2> A second embodiment of the present invention will be described. In the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described. In the first embodiment, the subject area is detected by image matting. In the second embodiment, the subject area is detected by a method different from image matting.

[0056] 11 is a block diagram showing the configuration of an area detection unit 111 according to the second embodiment. The area detection unit 111 according to the second embodiment has a subject detection unit 1100 instead of the image matting unit 201 of the first embodiment.

[0057] The subject detection unit 1100 detects a subject area from a captured image. For example, the subject detection unit 1100 detects a person's face area and human body area, and determines a rectangular area that encompasses them as the subject area. There are no particular limitations on the method for detecting the face area and human body area, and for example, the face area may be detected by a known face detection process. The face area may also be detected using information about the face (for example, skin color information, information about facial parts such as the eyes, nose, and mouth). The subject detection unit 1100 is a classifier that detects a face area using a learning algorithm typified by a neural network. The human body region may also be detected by a known human body detection process. The subject detection unit 1100 outputs a subject detection map indicating the detected subject region to the clipping map generation unit 205.

[0058] 12 is a schematic diagram showing the smallest value selection process for the subject detection map and the processed distance map. The subject detection map has large values ​​in the three rectangular regions that each contain three people, while the processed distance map has large values ​​only in the region of the central person. In this case, the smallest value selection process results in an extraction map with large values ​​only in the region of the central person, that is, an extraction map similar to that in Example 1 (FIG. 9).

[0059] As described above, even with the configuration of the second embodiment, which detects a rectangular subject area using a method different from image matting, it is possible to extract a desired area from a captured image with high accuracy, just like the first embodiment.

[0060] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0061] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0062] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0063] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0064] The disclosure of this embodiment includes the following configuration, method, program, and medium. (Configuration 1) a detection means for detecting a subject area from a captured image; an acquisition means for acquiring a distance map corresponding to the captured image; a determining means for determining a cutout area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image; 1. An image processing device comprising: (Configuration 2) The detection means detects the subject region using a trained model. 2. The image processing device according to configuration 1, (Configuration 3) a setting means for switching the operation mode to be set from a first mode for displaying the captured image on a display surface to a second mode for cutting out an area from the captured image in response to a predetermined touch operation being performed on the display surface while the first mode is set; Further having 3. The image processing device according to configuration 1 or 2. (Configuration 4) The predetermined touch operation is a long tap, a double tap, a swipe, a flick, or a drag. 4. The image processing device according to configuration 3. (Configuration 5) The determining means processing the distance map based on a plurality of distances indicated by the distance map for an area including the specified position and its surroundings; The cut-out area is determined based on the subject area and the processed distance map. 5. The image processing device according to any one of configurations 1 to 4. (Configuration 6) The determining means processes the distance map based on an average value of the plurality of distances. 6. The image processing device according to configuration 5. (Configuration 7) The determining means processes the distance map based on a histogram of the plurality of distances. 6. The image processing device according to configuration 5. (Configuration 8) a control means for controlling, when cutting out an area from the captured image, to display the captured image so that the cut-out area can be identified; Further having 8. The image processing device according to any one of configurations 1 to 7, wherein: (Configuration 9) The determining means When the specified position is included in the subject area, a cutout area is determined in which a distance in a distance range based on the specified position does not include an area indicated by the distance map; If the specified position is not included in the subject area, a distance range based on the specified position determines a cutout area that includes the area indicated by the distance map. 9. The image processing device according to any one of configurations 1 to 8. (Configuration 10) The determining means When the specified position is included in the object area, modifying the distance map so that a distance range based on the specified position indicates a second object area that does not include the area indicated by the distance map; If the specified position is not included in the object region, modifying the distance map so that a distance range based on the specified position indicates a second object region including the region indicated by the distance map; 10. The image processing device according to any one of configurations 1 to 9, wherein the clipping area is determined based on the subject area and the second subject area. (Configuration 11) The determining means When the specified position is included in the subject area, an area included in both the subject area and the second subject area is determined as the cut-out area; 11. The image processing device according to claim 10, wherein, when the specified position is not included in the subject area, an area included in at least one of the subject area and the second subject area is determined as the cut-out area. (method) a detection step of detecting a subject area from a captured image; an acquisition step of acquiring a distance map corresponding to the captured image; a determining step of determining a cutout region, which is a region to be cut out from the captured image, based on the subject region, the distance map, and a position specified when cutting out a region from the captured image; An image processing method comprising: (program) 12. A program for causing a computer to function as each means of the image processing device according to any one of configurations 1 to 11. (medium) 12. A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0065] 101: Image processing device 108: Control unit 111: Area detection unit 201: Image matting section 202: Distance measurement section 203: Distance map processing unit 204: Average distance acquisition unit 205: Crop map generation unit 1100: Object detection unit

Claims

1. a detection means for detecting a subject area from a captured image; an acquisition means for acquiring a distance map corresponding to the captured image; a determining means for determining a cutout area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image; 1. An image processing device comprising:

2. The detection means detects the subject region using a trained model.

2. The image processing device according to claim 1, wherein:

3. a setting means for switching the operation mode to be set from a first mode for displaying the captured image on a display surface to a second mode for cutting out an area from the captured image in response to a predetermined touch operation being performed on the display surface while the first mode is set; Further having 2. The image processing device according to claim 1, wherein:

4. The predetermined touch operation is a long tap, a double tap, a swipe, a flick, or a drag.

4. The image processing device according to claim 3.

5. The determining means processing the distance map based on a plurality of distances indicated by the distance map for an area including the specified position and its surroundings; The cut-out area is determined based on the subject area and the processed distance map.

2. The image processing device according to claim 1, wherein:

6. The determining means processes the distance map based on an average value of the plurality of distances.

6. The image processing device according to claim 5,

7. The determining means processes the distance map based on a histogram of the plurality of distances.

6. The image processing device according to claim 5,

8. a control means for controlling, when cutting out an area from the captured image, to display the captured image so that the cut-out area can be identified; Further having 2. The image processing device according to claim 1, wherein:

9. The determining means When the specified position is included in the subject area, a cutout area is determined in which a distance in a distance range based on the specified position does not include an area indicated by the distance map; If the specified position is not included in the subject area, a distance range based on the specified position determines a cutout area that includes the area indicated by the distance map.

2. The image processing device according to claim 1, wherein:

10. The determining means If the specified position is included in the subject area, modifying the distance map to indicate a second object region whose distance range does not include the region indicated by the distance map; If the specified position is not included in the object region, modifying the distance map so that a distance range based on the specified position indicates a second object region including the region indicated by the distance map; The image processing device according to claim 1 , wherein the cropping area is determined based on the subject area and the second subject area.

11. The determining means When the specified position is included in the subject area, an area included in both the subject area and the second subject area is determined as the cut-out area; 11. The image processing device according to claim 10, wherein, when the specified position is not included in the subject area, an area included in at least one of the subject area and the second subject area is determined as the crop area.

12. a detection step of detecting a subject area from a captured image; an acquisition step of acquiring a distance map corresponding to the captured image; a determining step of determining a cutout region, which is a region to be cut out from the captured image, based on the subject region, the distance map, and a position specified when cutting out a region from the captured image; An image processing method comprising:

13. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

14. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images