Image processing device

Abstract

This image processing device comprises: a detection means for detecting a subject region from a captured image; an acquisition means for acquiring a distance map corresponding to the captured image; and a determination means for determining a cutout region, which is a region to be cut out from the captured image, on the basis of the subject region, the distance map, and a position designated when cutting out the region from the captured image.

Description

Image processing device

[0001] The present invention relates to an image processing device, and more particularly to a technique for cutting out an area from a captured image (a captured image).

[0002] Techniques for extracting a subject region from a captured image have been proposed. Non-Patent Document 1 discloses image matting, which is a trained model for extracting a subject region from a captured image. Also, a technique for extracting a subject region from a captured image based on the distribution of defocus amounts in the captured image has been proposed. Non-Patent Document 2 discloses a trained model for generating a distance map (two-dimensional information indicating the distribution of defocus amounts) from a captured image.

[0003] “Deep image matting”, N.Xu, B.Price, S.Cohen, and T.Huang, Proc. IEEE Conference on CVPR, 2017 “Depth map prediction from a single image using a multi-scale deep network”, David Eigen, Christian Puhrsch, and Rob Fergus, NeurIPS,2014

[0004] However, conventional methods are unable to accurately extract desired regions. For example, with a method using image matting, even if a user wishes to extract the region of a person and their belongings, only the region of the person may be extracted, and the regions of the belongings may not be extracted. Furthermore, even if a user wishes to extract the region of only one of multiple people (a specific person), the regions of multiple people may be extracted. With a method using a distance map, regions that are intricately blended into the background, such as a region of a person's hair or an animal's fur, may not be extracted accurately.

[0005] An object of the present invention is to provide a technique that enables a desired area to be cut out from a captured image with high accuracy.

[0006] A first aspect of the present invention is an image processing device characterized by having a detection means for detecting a subject area from a captured image, an acquisition means for acquiring a distance map corresponding to the captured image, and a determination means for determining a cut-out area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image.

[0007] A second aspect of the present invention is an image processing method characterized by having a detection step of detecting a subject area from a captured image, an acquisition step of acquiring a distance map corresponding to the captured image, and a determination step of determining a cut-out area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image.

[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device.A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device.

[0009] According to the present invention, it is possible to extract a desired area from a captured image with high accuracy.

[0010] FIG. 1 is a block diagram of an image processing device according to a first embodiment. FIG. 2 is a block diagram of a region detection unit according to the first embodiment. FIG. 3A is a schematic diagram of image matting according to the first embodiment, and FIG. 3B is a schematic diagram of a distance map according to the first embodiment. FIGS. 4A to 4C are schematic diagrams of processing the distance map according to the first embodiment. FIG. 5 is a block diagram of a clipping processing unit according to the first embodiment. FIG. 6 is a schematic diagram of a clipped region superimposed image before updating according to the first embodiment. FIG. 7 is a flowchart of the clipping process according to the first embodiment. FIGS. 8A to 8C are schematic diagrams of processing the distance map according to the first embodiment. FIG. 9 is a schematic diagram of small value selection processing according to the first embodiment. FIG. 10 is a schematic diagram of a clipped region superimposed image after updating according to the first embodiment. FIG. 11 is a block diagram of a region detection unit according to the second embodiment. FIG. 12 is a schematic diagram of small value selection processing according to the second embodiment.

[0011] First Embodiment A first embodiment of the present invention will be described below. Fig. 1 is a block diagram showing the configuration of an image processing device 101 according to the first embodiment. The image processing device 101 is, for example, an imaging device such as a smartphone or a digital camera that captures an image of a subject.

[0012] The image processing device 101 has an optical system 102, an imaging unit 103, an A / D conversion unit 104, an image processing unit 105, a storage unit 106, a display unit 107, a control unit 108, a ROM 109, a RAM 110, an area detection unit 111, and a cutout processing unit 112. These are connected to a bus so that they can communicate with each other.

[0013] The optical system 102 is a group of lenses including a zoom lens and a focus lens, and forms a subject image (optical image of the subject) on the imaging unit 103. The imaging unit 103 is an imaging sensor such as a CCD or CMOS sensor, and performs photoelectric conversion of the subject image (optical image of the subject) formed on the imaging unit 103 by the optical system 102. The imaging unit 103 then outputs an analog signal (analog signal indicating the subject image) obtained by the photoelectric conversion to the A / D conversion unit 104. The arrangement of the multiple image sensors in the imaging unit 103 is, for example, an RGB (red, green, and blue) Bayer arrangement. The A / D conversion unit 104 converts the input analog image signal into digital image data and outputs the obtained digital image data to the RAM 110.

[0014] The image processing unit 105 applies various image processing such as white balance adjustment, noise reduction, color interpolation (debayering), and gamma processing to the image data recorded in the RAM 110. The image processing unit 105 can obtain (generate) a main image and a thumbnail image as images after image processing. The thumbnail image is, for example, an image in which the resolution of the main image is reduced to VGA size.

[0015] The storage unit 106 is, for example, a memory card that is detachable from the image processing apparatus 101. The image that has undergone image processing by the image processing unit 105 is stored in the storage unit 106 via the RAM 110 as a recorded image.

[0016] The display unit 107 is capable of receiving touch operations. For example, the display unit 107 includes a liquid crystal display and a touch screen. The touch screen detects contact with an operating object such as a finger or a stylus pen. For example, a capacitive or resistive film method is used to detect contact of the operating object with the touch screen. The image processing device 101 (control unit 108) determines the type of touch operation based on the position and time of contact detected by the touch screen. A touch operation is an operation performed on the touch screen by the operating object. Examples of touch operations include touch, release, tap, drag, swipe, flick, pinch-in, and pinch-out. When a touch operation is performed, the image processing device 101 (control unit 108) performs processing (control) according to the performed touch operation. The liquid crystal display displays images stored in the RAM 110 or the storage unit 106, and displays an operation user interface for receiving instructions from a user.

[0017] For example, the user touches a subject that the user wants to cut out in the captured image (captured image) displayed on the display unit 107. Then, the control unit 108 controls the cutout processing unit 112 based on the touch position (touched position) to cut out the subject area at the touch position (the area of ​​the touched subject). Note that although an example of specifying a position by touch has been described, the method of specifying a position is not particularly limited. For example, the user may specify a position by moving a cursor using a four-way key. Furthermore, the captured image may be an image (main image or thumbnail image) after image processing by the image processing unit 105, or an image (image output from the A / D conversion unit 104) before image processing by the image processing unit 105.

[0018] The control unit 108 controls the entire image processing device 101. For example, the control unit 108 is a CPU that reads a program from the ROM 109, expands it into the RAM 110, and executes it to control each unit of the image processing device 101. The ROM 109 is a non-volatile memory that stores data (e.g., programs and parameters) necessary for controlling each unit of the image processing device 101. The RAM 110 is a volatile memory that temporarily stores data necessary for controlling each unit of the image processing device 101.

[0019] The region detection unit 111 determines a region to be cut out from the captured image, and outputs a cutout map indicating the cutout region.

[0020] The clipping processing unit 112 acquires the clipping map output from the region detection unit 111 and clips out the clipping region indicated by the clipping map from the captured image. The clipping processing unit 112 superimposes the contour line of the clipping region on the captured image and outputs the captured image with the contour line superimposed. This makes it possible to display the captured image so that the clipping region is identifiable. Note that there is no particular limitation on the method for making the clipping region identifiable. For example, the entire clipping region may be emphasized rather than just the contour of the clipping region.

[0021] 2 is a block diagram showing the configuration of the area detection unit 111. The area detection unit 111 has an image matting unit 201, a distance measurement unit 202, a distance map processing unit 203, an average distance acquisition unit 204, and a clipping map generation unit 205.

[0022] The image matting unit 201 detects a subject region from a captured image. For example, the image matting unit 201 detects the subject region using image matting (a trained model) disclosed in Non-Patent Document 1. By using image matting, it is possible to accurately detect regions that are intricately mixed into the background, such as a region of a person's hair or a region of animal fur. The image matting unit 201 outputs an image matting map indicating the detected subject region to the clipping map generation unit 205.

[0023] The ranging unit 202 acquires a distance map corresponding to the captured image (a map indicating the subject distance at each position in the captured image). The method for acquiring the distance map is not particularly limited. For example, the distance map may be acquired from the captured image using a trained model, as in the method described in Non-Patent Document 2, or the distance map may be acquired by performing phase-difference AF to indicate the distribution of defocus amounts. The distance map may be included in advance as metadata in the file of the captured image, or may be acquired from the metadata. A distance map associated with the captured image may be stored in advance separately from the captured image, and the stored distance map may be acquired. The distance map may also be acquired from an external device.

[0024] 3A is a schematic diagram of an image matting map obtained by the image matting unit 201, and FIG. 3B is a schematic diagram of a distance map obtained by the distance measuring unit 202. FIGS. 3A and 3B show maps obtained (generated) from captured images of three people lined up at different distances from the image processing device 101 (imaging device). For example, an image is generated as the image matting map in which the detected subject area is bright (the signal level of the detected subject area is high) and the remaining area is dark (the signal level of the remaining area is low). Then, an image is generated in which the shorter the subject distance is, the brighter the image is (the shorter the subject distance is, the higher the signal level is) and the longer the subject distance is, the darker the image is (the longer the subject distance is, the lower the signal level is). In Example 1, it is assumed that an area in the distance map whose brightness (signal level) is equal to or greater than a threshold is the subject area indicated by the distance map.

[0025] The distance measuring unit 202 outputs the acquired distance map to the distance map processing unit 203 and the average distance acquisition unit 204. The average distance acquisition unit 204 uses the distance map acquired by the distance measuring unit 202 to acquire the average value (average distance) of multiple subject distances indicated in the distance map for the user's touch area. The touch area is an area including the position designated (touched) by the user and its surrounding area, and may be interpreted as the area touched by the user. The distance map processing unit 203 processes the distance map acquired by the distance measuring unit 202 based on the average distance acquired by the average distance acquisition unit 204. The distance map processing unit 203 outputs the processed distance map to the cutout map generation unit 205.

[0026] An example of the operation of the distance map processing unit 203 and the average distance acquisition unit 204 will be described with reference to FIGS. 4A to 4C.

[0027] 4B is a schematic diagram of a distance map obtained by distance measurement unit 202. In the distance map of FIG. 4B, the subject distance of a person's face region 402 is long (the brightness (signal level) of region 402 is low), and region 402 is not shown as a subject region. When a user touches region 402, average distance acquisition unit 204 acquires the average value (average distance) of multiple subject distances indicated by the distance map of FIG. 4B for region 402.

[0028] The distance map processing unit 203 generates a gain function 401 shown in FIG. 4A based on the average distance obtained by the average distance acquisition unit 204. The gain function 401 indicates the correspondence between the object distance and the gain value (coefficient by which the object distance is multiplied) of the object distance. Instead of the gain function 401, a table indicating the correspondence between the object distance and the gain value may be generated. The horizontal axis of FIG. 4A indicates the object distance, and the vertical axis of FIG. 4A indicates the gain value. As the object distance increases from t1 to t3, the gain value increases linearly from 0 to 2, and as the object distance increases from t3 to t4, the gain value decreases linearly from 2 to 0. The object distance t3 is the average distance obtained by the average distance acquisition unit 204, and the object distances t1 and t4 are determined based on the object distance t3. The object distance t2 is the object distance in region 403 of FIG. 4B.

[0029] The distance map processing unit 203 processes the distance map output from the distance measurement unit 202 through gain processing using the generated gain function. In the gain processing, the object distance at each position is multiplied by a gain value corresponding to the object distance. FIG. 4C is a schematic diagram of the distance map after the distance map of FIG. 4B has been processed using the gain function of FIG. 4A. By using the gain function of FIG. 4A, the brightness (signal level) of region 402 in FIG. 4B is increased. As a result, in the distance map of FIG. 4C, region 404 corresponding to region 402 in FIG. 4B is shown as the object region. In the distance map of FIG. 4B, the brightness (signal level) of region 402 is lower than the brightness (signal level) of region 403. In contrast, in the distance map of FIG. 4C, the brightness (signal level) of region 404 corresponding to region 402 in FIG. 4B is higher than the brightness (signal level) of region 405 corresponding to region 403 in FIG. 4B.

[0030] In this way, the distance map is processed based on multiple object distances in the touch area. While the example described above involves determining a gain function with a large gain value in a distance range including the average value of multiple object distances in the touch area, a gain function with a small gain value in a distance range including the average value of multiple object distances in the touch area may also be determined (details will be described later). Although the example described above involves using the average value of multiple object distances in the touch area, the method of processing the distance map is not limited to this. The distance map may also be processed based on the maximum and minimum values ​​of multiple object distances in the touch area. For example, the maximum and minimum values ​​of multiple object distances in the touch area may be obtained, and a gain function with a large (or small) gain value in the distance range from the maximum to the minimum may be determined. The distance map may also be processed based on a histogram of multiple object distances in the touch area. For example, the distance map may be processed using a large (or small) gain value in a region of a histogram similar to the histogram of the touch area. Furthermore, the distance map may also be processed using a process other than gain processing. The processed distance map may be a binary (1-bit) map in which the signal level at each position indicates whether or not the area is an object area.

[0031] The clipping map generation unit 205 generates a clipping map by a synthesis process that synthesizes the image matting map output from the image matting unit 201 and the distance map output from the distance map processing unit 203. The synthesis process selectively performs, for example, large value selection, small value selection, addition, subtraction, multiplication, and division of the two maps. The synthesis process may select the signal level of the image matting map or the distance map as the signal level of the clipping map based on the result of comparing the signal level (pixel value) of the distance map with a threshold. For example, if the signal level of the distance map is less than the threshold, the signal level of the image matting map may be selected as the signal level of the clipping map. If the signal level of the distance map is equal to or greater than the threshold, the signal level of the distance map may be selected as the signal level of the clipping map.

[0032] 5 is a block diagram showing the configuration of the cutout processing unit 112. The cutout processing unit 112 includes a cutout unit 501 and a cutout region superimposing unit 502.

[0033] The cropping unit 501 crops out the crop area indicated by the crop map from the captured image. For example, the cropping unit 501 generates and outputs, as a cropped image, an image in which pixel values ​​in areas of the crop map where the signal level is equal to or higher than a threshold are the same as those in the captured image and pixel values ​​in areas of the crop map where the signal level is equal to or higher than the threshold are a predetermined value (for example, black pixel values).

[0034] The clipping area superimposing unit 502 detects the contour of the clipping area indicated by the clipping map, superimposes it on the captured image, and outputs the captured image after the contour has been superimposed (clip-out area superimposed image). The method for detecting the contour is not particularly limited, and any known method may be used.

[0035] Fig. 6 is a schematic diagram of a clipping area superimposed image. In Fig. 6, the contour of the clipping area is superimposed on a captured image in which three people are lined up at different distances from the image processing device 101 (image capture device). In Fig. 6, the area of ​​the three people is determined as the clipping area, and a gray contour surrounding the three people is superimposed as the contour of the clipping area. By superimposing and displaying the contour of the clipping area on the captured image, it is possible to inform the user of the clipping area. Note that the color of the contour is not particularly limited.

[0036] Fig. 7 is a flowchart of the cut-out process performed by the image processing device 101. The cut-out process in Fig. 7 is realized by the control unit 108 expanding a program stored in the ROM 109 into the RAM 110 and executing the program. For example, when a user issues an instruction to execute a predetermined image editing application, the cut-out process in Fig. 7 starts.

[0037] In step S701, the control unit 108 selects a captured image specified by the user from the plurality of captured images stored in the storage unit 106, and displays the selected captured image on the display unit 107. At this time, the image processing device 101 is set to an image display mode, which is an operation mode for displaying the captured image on the display screen.

[0038] In step S702, in response to a long tap on the display unit 107 (touch screen), the control unit 108 switches the set operation mode from the image display mode to a cutout mode, which is an operation mode for cutting out an area from a captured image. Then, the control unit 108 controls the area detection unit 111 to determine a cutout area, and controls the cutout area superimposition unit 502 to superimpose the outline of the cutout area on the captured image, thereby switching the image to be displayed to the cutout area superimposed image.

[0039] Image matting can determine the area of ​​a single subject with high accuracy. Therefore, in step S702, the cutout area is determined using only the image matting unit 201. In other words, the subject area detected by the image matting unit 201 is determined as the cutout area.

[0040] Note that the predetermined touch operation for setting the cutout mode is not limited to a long tap. For example, it may be a double tap, a swipe, a flick, or a drag. A long tap is a touch operation in which the display surface is touched for longer than a first time period and then released, and a double tap is a touch operation in which a tap (a touch operation in which the display surface is touched for shorter than the first time period and then released) is followed by another tap before a second time period has elapsed. A swipe is a touch operation in which the display surface is traced, a flick is a touch operation in which the display surface is flicked, and a drag is a touch operation in which a point on the display surface is touched for longer than a third time period and then the touch position is moved.

[0041] The user views the clipping region superimposed image, confirms whether the desired region has been determined as the clipping region, and inputs the confirmation result to the image processing device 101. In step S703, the control unit 108 determines whether the user's desired region has been determined as the clipping region based on the user's input. If the user's desired region has been determined as the clipping region, the process proceeds to step S712, and if the user's desired region has not been determined as the clipping region, the process proceeds to step S704.

[0042] If the area desired by the user has not been determined as the cut-out area, the user touches (specifies) the area to be corrected. In step S704, the control unit 108 accepts the touch (specification) of the area to be corrected. The area to be corrected is the area to be added to the cut-out area or the area to be removed from the cut-out area. When the cut-out area superimposed image of FIG. 6 is displayed and the user wants to cut out only the person in the middle, the user touches the person at the forefront and the person at the back.

[0043] In step S705, the control unit 108 controls the average distance acquisition unit 204 to acquire the average value (average distance) of the multiple subject distances indicated by the distance map for the user touch area. At this time, it is assumed that the distance map has already been obtained by the distance measurement unit 202.

[0044] In step S706, the control unit 108 determines whether the position specified by the user (the portion to be corrected) is included in the current crop area (the subject area detected by the image matting unit 201). If the specified position is not included in the current crop area, the process proceeds to step S707; if the specified position is included in the current crop area, the process proceeds to step S709. As will be described in detail later, if the process proceeds to step S707, a crop area including a distance area based on the specified position is determined. The distance area based on the specified position is an area in which the subject distance within the distance range based on the specified position is indicated by the pre-processing distance map. In other words, the subject area at the specified position is added to the crop area. If the process proceeds to step S709, a crop area not including the distance area based on the specified position is determined. In other words, the subject area at the specified position is excluded from the crop area.

[0045] In step S707, the control unit 108 controls the distance map processing unit 203 to process the distance map so as to indicate a subject area including a distance area based on a position (a location to be corrected) designated by the user. As described above, the distance area based on the designated position is an area in which the subject distance within the distance range based on the designated position is indicated by the distance map before processing. For example, as shown in FIG. 4A , a gain function having a large gain value for a distance range including an average value of multiple subject distances in the touch area is determined, and the distance map is processed using the gain function.

[0046] In step S708, the control unit 108 controls the clipping map generation unit 205 to update the clipping area. As the clipping area is updated, the displayed clipping area superimposed image is also updated. The control unit 108 determines the updated clipping area based on the current clipping area (the subject area detected by the image matting unit 201) and the subject area indicated by the processed distance map. For example, the control unit 108 determines an area included in at least one of the current clipping area and the subject area indicated by the processed distance map as the updated clipping area. Such an updated clipping area can be determined, for example, by a large value selection process that selects the larger of the signal level of the current clipping map (image matting map) and the signal level of the processed distance map.

[0047] In step S709, the control unit 108 controls the distance map processing unit 203 to process the distance map so as to indicate a subject area that does not include a distance area based on the position (location to be corrected) specified by the user. As described above, the distance area based on the specified position is an area in which the subject distance in the distance range based on the specified position is indicated by the distance map before processing. For example, as shown in FIG. 8A , a gain function with a small gain value for a distance range that includes the average value of multiple subject distances in the touch area is determined, and the distance map is processed using the gain function.

[0048] FIG. 8A is a schematic diagram of a gain function, FIG. 8B is a schematic diagram of a distance map before processing, and FIG. 8C is a schematic diagram of a distance map after processing. With the cropped area superimposed image of FIG. 6 displayed, assume that a user touches areas 802 and 803 (the rearmost and foremost people) in FIG. 8B to crop out only the person in the middle. In the gain function 801 of FIG. 8A, the gain value is 0 in a distance range including subject distance t1 and a distance range including subject distance t3, and the gain value is 2 between these distance ranges. Subject distance t1 is the subject distance of area 802 of FIG. 8B, and subject distance t3 is the subject distance of area 803 of FIG. 8B. Subject distance t2 is the subject distance of the person in the middle. By processing the distance map of FIG. 8B using the gain function of FIG. 8A, the distance map of FIG. 8C showing only the area of ​​the person in the middle can be obtained.

[0049] In step S709, the control unit 108 controls the clipping map generation unit 205 to update the clipping area. The clipping area superimposed image to be displayed is also updated in conjunction with the update of the clipping area. The control unit 108 determines the updated clipping area based on the current clipping area (the subject area detected by the image matting unit 201) and the subject area indicated by the processed distance map. For example, the control unit 108 determines the area included in both the current clipping area and the subject area indicated by the processed distance map as the updated clipping area. Such an updated clipping area can be determined, for example, by a minimum value selection process that selects the smaller of the signal level of the current clipping map (image matting map) and the signal level of the processed distance map.

[0050] 9 is a schematic diagram showing the small value selection process for the image matting map and the processed distance map. The image matting map has large values ​​in the areas of all three people, while the processed distance map has large values ​​only in the area of ​​the middle person. In this case, the small value selection process results in a clipping map with large values ​​only in the area of ​​the middle person.

[0051] Fig. 10 is a schematic diagram of the clipping region superimposed image after updating. By updating the clipping map to one that shows only the area of ​​the person in the middle, the clipping region superimposed image in Fig. 10 shows a contour line that shows only the area of ​​the person in the middle.

[0052] In step S711, the control unit 108 determines whether the region desired by the user has been determined as the cut-out region based on the input from the user. If the region desired by the user has been determined as the cut-out region, the process proceeds to step S711. If the region desired by the user has not been determined as the cut-out region, the process proceeds to step S704.

[0053] In step S712, the control unit 108 cuts out the cut-out area in accordance with an instruction from the user (for example, a copy instruction).

[0054] As described above, according to Example 1, by determining the cut-out area based on the image matting map, the distance map, and the position specified when cutting out the area from the captured image, it becomes possible to cut out the desired area from the captured image with high accuracy.

[0055] <Embodiment 2> Embodiment 2 of the present invention will be described. Note that, hereinafter, a description of the same points as in embodiment 1 (for example, the same configuration and processing as in embodiment 1) will be omitted, and differences from embodiment 1 will be described. In embodiment 1, the subject area is detected by image matting. In embodiment 2, the subject area is detected by a method different from image matting.

[0056] 11 is a block diagram showing the configuration of an area detection unit 111 according to the second embodiment. The area detection unit 111 according to the second embodiment has a subject detection unit 1100 instead of the image matting unit 201 of the first embodiment.

[0057] The subject detection unit 1100 detects a subject region from a captured image. For example, the subject detection unit 1100 detects a person's face region and human body region, and determines a rectangular region that encompasses them as the subject region. The method for detecting the face region and human body region is not particularly limited, and for example, the face region may be detected by a known face detection process. The face region may also be detected using information about the face (for example, skin color information, information about facial parts such as the eyes, nose, and mouth). The subject detection unit 1100 may be a classifier that detects the face region using a learning algorithm such as a neural network. The human body region may also be detected by a known human body detection process. The subject detection unit 1100 outputs a subject detection map indicating the detected subject region to the clipping map generation unit 205.

[0058] 12 is a schematic diagram showing the small value selection process for the subject detection map and the processed distance map. The subject detection map has large values ​​in three rectangular regions that respectively include the three people, while the processed distance map has large values ​​only in the region of the central person. In this case, the small value selection process results in an extraction map with large values ​​only in the region of the central person, i.e., an extraction map similar to that of Example 1 (FIG. 9).

[0059] As described above, even with the configuration of Example 2, which detects rectangular subject areas using a method different from image matting, it is possible to extract desired areas from a captured image with high accuracy, just like Example 1.

[0060] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0061] The processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, a central processing unit (CPU), a micro processing unit (MPU), and a digital signal processor (DSP). Dedicated processors include, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a programmable logic device (PLD). Examples of programmable logic devices include a field programmable gate array (FPGA) and a complex programmable logic device (CPLD).

[0062] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0063] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0064] The disclosure of the present embodiments includes the following configurations, methods, programs, and media. (Configuration 1) An image processing device comprising: a detection means for detecting a subject region from a captured image; an acquisition means for acquiring a distance map corresponding to the captured image; and a determination means for determining a cut-out region to be cut out from the captured image based on the subject region, the distance map, and a position specified when cutting out a region from the captured image. (Configuration 2) The image processing device of Configuration 1, wherein the detection means detects the subject region using a trained model. (Configuration 3) The image processing device of Configuration 1 or 2, further comprising: a setting means for switching a set operation mode from a first mode in which the captured image is displayed on a display surface to a second mode in which a region is cut out from the captured image in response to a predetermined touch operation being performed on the display surface while the first mode is set. (Configuration 4) The image processing device of Configuration 3, wherein the predetermined touch operation is a long tap, a double tap, a swipe, a flick, or a drag. (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the determination means processes the distance map based on a plurality of distances indicated in the distance map for an area including the specified position and its periphery, and determines the cut-out area based on the subject area and the processed distance map. (Configuration 6) The image processing device according to configuration 5, wherein the determination means processes the distance map based on an average value of the plurality of distances. (Configuration 7) The image processing device according to configuration 5, wherein the determination means processes the distance map based on a histogram of the plurality of distances. (Configuration 8) The image processing device according to any one of configurations 1 to 7, further comprising control means for controlling, when a region is cut out from the captured image, to display the captured image in a manner that allows the cut-out area to be identified.(Configuration 9) The image processing device according to any one of configurations 1 to 8, wherein the determining means, when the specified position is included in the subject area, determines a clipping area in which a distance in a distance range based on the specified position does not include an area indicated by the distance map, and when the specified position is not included in the subject area, determines a clipping area in which a distance in a distance range based on the specified position includes an area indicated by the distance map. (Configuration 10) The image processing device according to any one of configurations 1 to 9, wherein the determining means, when the specified position is included in the subject area, processes the distance map so that a distance range based on the specified position indicates a second subject area that does not include an area indicated by the distance map, and when the specified position is not included in the subject area, processes the distance map so that a distance range based on the specified position indicates a second subject area that includes an area indicated by the distance map, and determines the clipping area based on the subject area and the second subject area. (Configuration 11) The image processing device of Configuration 10, wherein the determination means determines, when the specified position is included in the subject area, an area included in both the subject area and the second subject area as the clipping area, and when the specified position is not included in the subject area, determines, when the specified position is not included in the subject area, an area included in at least one of the subject area and the second subject area as the clipping area. (Method) An image processing method comprising: a detection step of detecting a subject area from a captured image; an acquisition step of acquiring a distance map corresponding to the captured image; and a determination step of determining a clipping area to be clipped from the captured image based on the subject area, the distance map, and the position specified when clipping a area from the captured image. (Program) A program for causing a computer to function as each means of the image processing device of any of Configurations 1 to 11. (Medium) A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing device of any of Configurations 1 to 11.

[0065] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0066] This application claims priority based on Japanese Patent Application No. 2024-043309, filed March 19, 2024, the entire contents of which are incorporated herein by reference.

[0067] 101: Image processing device 108: Control unit 111: Area detection unit 201: Image matting unit 202: Distance measurement unit 203: Distance map processing unit 204: Average distance acquisition unit 205: Clipping map generation unit 1100: Subject detection unit

Claims

1. An image processing device comprising: a detection means for detecting a subject area from a captured image; an acquisition means for acquiring a distance map corresponding to the captured image; and a determination means for determining a cut-out area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image.

2. The image processing device according to claim 1, wherein the detection means detects the subject area using a trained model.

3. The image processing device according to claim 1, further comprising a setting means for switching the operating mode to be set from a first mode for displaying the captured image on a display surface to a second mode for cutting out an area from the captured image in response to a predetermined touch operation being performed on the display surface while the first mode for displaying the captured image on the display surface is set.

4. The image processing device according to claim 3, wherein the predetermined touch operation is a long tap, a double tap, a swipe, a flick, or a drag.

5. The image processing device described in claim 1, characterized in that the determination means processes the distance map based on multiple distances indicated in the distance map for an area including the specified position and its surroundings, and determines the cut-out area based on the subject area and the processed distance map.

6. The image processing device according to claim 5, wherein said determining means processes said distance map based on an average value of said plurality of distances.

7. The image processing device according to claim 5, wherein said determining means processes said distance map based on a histogram of said plurality of distances.

8. The image processing device according to claim 1, further comprising control means for controlling, when cutting out an area from the captured image, the captured image to be displayed so that the cut-out area can be identified.

9. The image processing device described in claim 1, characterized in that the determination means determines a cutout area in which the distance range based on the specified position does not include the area indicated by the distance map when the specified position is included in the subject area, and determines a cutout area in which the distance range based on the specified position includes the area indicated by the distance map when the specified position is not included in the subject area.

10. The image processing device described in claim 1, characterized in that the determination means processes the distance map so that, when the specified position is included in the subject area, the distance range based on the specified position indicates a second subject area that does not include the area indicated by the distance map, and when the specified position is not included in the subject area, processes the distance map so that the distance range based on the specified position indicates a second subject area that includes the area indicated by the distance map, and determines the cut-out area based on the subject area and the second subject area.

11. The image processing device described in claim 10, characterized in that the determination means determines, when the specified position is included in the subject area, an area included in both the subject area and the second subject area as the cut-out area, and when the specified position is not included in the subject area, determines, when the specified position is not included in the subject area, an area included in at least one of the subject area and the second subject area as the cut-out area.

12. An image processing method comprising: a detection step of detecting a subject area from a captured image; an acquisition step of acquiring a distance map corresponding to the captured image; and a determination step of determining a cut-out area, which is an area to be cut out from the captured image, based on the subject area, the distance map, and a position specified when cutting out an area from the captured image.

13. A program for causing a computer to function as each means of the image processing device according to any one of claims 1 to 11.

14. A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing device according to any one of claims 1 to 11.

Images