Image processing device
By combining image matting and distance map processing techniques, the problem of difficulty in accurately cutting out the subject area in the captured image in existing technologies has been solved, and precise cutting of the area desired by the user has been achieved.
Patent Information
- Application Number
- CN202480089524.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-03-19
- Filing Date
- 2024-12-05
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies struggle to precisely cut out the desired subject area from captured images, especially when dealing with complex backgrounds such as human hair or animal fur, often resulting in incorrect or incomplete cutting.
By combining image matting and distance map processing techniques, the subject area is detected, and the cutout area is determined based on the user-specified position and distance map processing. The cutout image is then generated using gain function and compositing processing.
It achieves high-precision cutting out of the user's desired area from the captured image, improving the cutting accuracy for complex background areas.
Smart Images

Figure CN123125890A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing apparatus, and more particularly to a technique for cutting out regions from a captured image (an already captured image). Background Technology
[0002] Techniques for extracting subject regions from captured images have been proposed. Non-Patent Document 1 discloses an image matting method, which is a learned model for extracting subject regions from captured images. Furthermore, techniques for extracting subject regions from captured images based on the distribution of defocus in the captured image have been proposed. Non-Patent Document 2 discloses a learned model for generating a distance map (two-dimensional information representing the distribution of defocus) from a captured image. Existing technical documents Non-patent literature
[0003] Non-patent literature 1: “Deep image matting”, N. Xu, B. Price, S. Cohen, and T. Huang, Proc. IEEE Conference on CVPR, 2017 Non-patent document 2: "Depth map prediction from a single image using a multi-scale deep network", David Eigen, Christian Puhrsch, and Rob Fergus, NeurIPS, 2014 Summary of the Invention The technical problem that the invention aims to solve
[0004] However, traditional methods cannot accurately extract the desired region. For example, in image matting methods, even if the user expects to extract the region of a person and their possessions, it may only extract the region of the person and not the region of the possessions. Furthermore, even if the user expects to extract the region of only one of multiple people (a specific person), it may extract the regions of multiple people. In methods using distance maps, regions that are complexly blended with the background, such as the region of a person's hair or the region of an animal's fur, are not accurately extracted.
[0005] This invention provides a technique for cutting out a desired region from a captured image with high precision. Solution to the problem
[0006] The present invention provides, in a first aspect, an image processing apparatus comprising: a detection unit configured to detect a subject region from an captured image; an acquisition unit configured to acquire a distance map corresponding to the captured image; and a determination unit configured to determine a cutout region based on the subject region, the distance map, and a position specified when cutting out a region from the captured image, the cutout region being the region to be cut out from the captured image.
[0007] The present invention provides, in a second aspect, an image processing method comprising: a detection step for detecting a subject region from a captured image; an acquisition step for acquiring a distance map corresponding to the captured image; and a determination step for determining a cutout region based on the subject region, the distance map, and a position specified when cutting out a region from the captured image, the cutout region being the region to be cut out from the captured image.
[0008] In a third aspect, the present invention provides a program that enables a computer to function as various units of the aforementioned image processing apparatus. In a fourth aspect, the present invention provides a computer-readable medium storing a program that enables a computer to function as various units of the aforementioned image processing apparatus. Advantages of the present invention
[0009] According to the present invention, a desired region can be cut out from a captured image with high precision. Attached Figure Description
[0010] Figure 1 This is a block diagram of an image processing apparatus according to the first embodiment. Figure 2 This is a block diagram of the region detection unit according to the first embodiment. Figure 3 (A) is a schematic diagram of image matting according to the first embodiment, and Figure 3 (B) is a schematic diagram of a distance map according to the first embodiment. Figure 4 (A) to (C) are schematic diagrams of the distance map processing according to the first embodiment. Figure 5 This is a block diagram of the cutting processing unit according to the first embodiment. Figure 6 This is a schematic diagram of the overlay image of the cut-out region before the update according to the first embodiment. Figure 7 This is a flowchart of the cutting process according to the first embodiment. Figure 8 (A) to (C) are schematic diagrams of the distance map processing according to the first embodiment. Figure 9This is a schematic diagram of the minimum value selection process according to the first embodiment. Figure 10 This is a schematic diagram of the updated overlay image of the cut-out region according to the first embodiment. Figure 11 This is a block diagram of the region detection unit according to the second embodiment. Figure 12 This is a schematic diagram of the minimum value selection process according to the second embodiment. Detailed Implementation
[0011] <First Embodiment> The first embodiment of the present invention is described. Figure 1 This is a block diagram illustrating the configuration of the image processing apparatus 101 according to the first embodiment. The image processing apparatus 101 is, for example, a camera device such as a smartphone or digital camera that captures images of a subject.
[0012] The image processing device 101 includes an optical system 102, a camera unit 103, an A / D conversion unit 104, an image processing unit 105, a storage unit 106, a display unit 107, a control unit 108, a ROM 109, a RAM 110, a region detection unit 111, and a cutout processing unit 112. These components are connected to a bus for communication with each other.
[0013] The optical system 102 is a lens group including a zoom lens and a focusing lens, and forms an image of the subject (an optical image of the subject) on the imaging unit 103. The imaging unit 103 is an imaging sensor such as a CCD or CMOS sensor, and performs photoelectric conversion on the image of the subject (the optical image of the subject) formed by the optical system 102 on the imaging unit 103. Then, the imaging unit 103 outputs the analog signal (an analog signal representing the image of the subject) obtained through photoelectric conversion to the A / D conversion unit 104. The arrangement of the multiple imaging elements in the imaging unit 103 is, for example, an RGB (red, green, and blue) Bayer arrangement. The A / D conversion unit 104 converts the input analog image signal into digital image data and outputs the obtained digital image data to the RAM 110.
[0014] The image processing unit 105 applies various image processing techniques, such as white balance adjustment, noise reduction, color interpolation (de-Bayer), and gamma processing, to the image data recorded in the RAM 110. The image processing unit 105 can obtain (generate) a main image and a thumbnail image as the processed image. The thumbnail image is, for example, an image obtained by reducing the resolution of the main image to VGA size.
[0015] Storage unit 106 is, for example, a memory card removable from image processing device 101. The image processed by image processing unit 105 is recorded as a recording image in storage unit 106 via RAM 110.
[0016] Display unit 107 is a display unit capable of receiving touch operations. For example, display unit 107 includes a liquid crystal display (LCD) and a touch screen. The touch screen detects contact with an operating object such as a finger or stylus. As a method for detecting contact between the operating object and the touch screen, for example, a capacitive method or a resistive film method is used. Image processing device 101 (control unit 108) determines the type of touch operation based on the location and time of the contact detected by the touch screen. A touch operation is an operation performed by an operating object on the touch screen. As a touch operation, for example, touching, releasing, tapping, dragging, sliding, swiping, pinching, or splitting. When a touch operation is performed, image processing device 101 (control unit 108) performs processing (control) corresponding to the performed touch operation. The LCD displays an image recorded in RAM 110 or storage unit 106, or the LCD displays an operation user interface for receiving instructions from the user.
[0017] For example, a user touches the subject to be cut out from the captured image (the captured image) displayed on the display unit 107. Then, the control unit 108 controls the cutout processing unit 112 to cut out the subject area (the area of the touched subject) based on the touch position (the touched position). Note that although an example of specifying a position by touch has been described, the method of specifying a position is not particularly limited, and for example, a user can specify a position by moving the cursor using the four arrow keys. Furthermore, the captured image can be an image after image processing using the image processing unit 105 (the main image or a thumbnail image), or it can be an image before image processing using the image processing unit 105 (the image output from the A / D conversion unit 104).
[0018] Control unit 108 controls the entire image processing apparatus 101. For example, control unit 108 is a CPU, and it controls the various units of the image processing apparatus 101 by reading a program from ROM 109, loading the program into RAM 110, and executing the program. ROM 109 is a non-volatile memory that stores data (e.g., programs and parameters) required to control the various units of the image processing apparatus 101. RAM 110 is a volatile memory that temporarily stores data required to control the various units of the image processing apparatus 101.
[0019] The region detection unit 111 determines the cutout region as the region to be cut out from the captured image and outputs a cutout map representing the cutout region.
[0020] The cutting processing unit 112 acquires the cutting map output from the region detection unit 111 and cuts out the cutting region represented by the cutting map from the captured image. The cutting processing unit 112 overlays the outline of the cutting region onto the captured image and outputs the captured image with the overlaid outline. Thus, the captured image can be displayed, making the cutting region recognizable. Note that there are no particular limitations on the method used to make the cutting region recognizable. For example, the entire cutting region can be emphasized, rather than just the outline of the cutting region.
[0021] Figure 2 This is a block diagram illustrating the configuration of the region detection unit 111. The region detection unit 111 includes an image matting unit 201, a distance measurement unit 202, a distance map processing unit 203, an average distance acquisition unit 204, and a cutout map generation unit 205.
[0022] Image matting unit 201 detects subject regions from the captured image. For example, image matting unit 201 uses image matting (learned model) disclosed in Non-Patent Document 1 to detect subject regions. By using image matting, regions that are complexly blended with the background, such as areas of human hair or animal fur, can also be accurately detected. Image matting unit 201 outputs an image matting map representing the detected subject regions to cutout map generation unit 205.
[0023] The distance measurement unit 202 acquires a distance map (a map representing the distances of the subject at various locations in the captured image) corresponding to the captured image. There are no particular limitations on the method for acquiring the distance map, and it can be acquired from the captured image using a learned model, as described in Non-Patent Document 2, or by performing phase-detection AF to acquire a distance map representing the distribution of defocus. The distance map can be pre-included as metadata in the captured image file, or it can be retrieved from that metadata. In addition to the captured image, a distance map associated with the captured image can be pre-stored, and the stored distance map can be retrieved. The distance map can be retrieved from an external device.
[0024] Figure 3 (A) is a schematic diagram of the image matting image obtained by the image matting unit 201, and Figure 3 (B) is a schematic diagram of the distance map obtained by the distance measurement unit 202. Figure 3(A) and (B) illustrate images obtained (generated) from a captured image in which three people standing in a row have different distances from the image processing device 101 (camera device). For example, as an image matting image, an image is generated where the detected subject area is bright (the signal level of the detected subject area is high) and the remaining area is dark (the signal level of the remaining area is low). Then, as a distance image, an image is generated in which the shorter the subject distance, the brighter the image (the shorter the subject distance, the higher the signal level), and the longer the subject distance, the darker the image (the longer the subject distance, the lower the signal level). In the first embodiment, the area in the distance image where the brightness (signal level) is equal to or greater than a threshold is assumed to be the subject area represented by the distance image.
[0025] The distance measurement unit 202 outputs the acquired distance map to the distance map processing unit 203 and the average distance acquisition unit 204. The average distance acquisition unit 204 uses the distance map acquired by the distance measurement unit 202 to obtain the average distance (average distance) of multiple subject distances represented by the distance map for the user's touch area. The touch area includes the location specified (touched) by the user and its surrounding area, and can be interpreted as the area touched by the user. The distance map processing unit 203 processes the distance map acquired by the distance measurement unit 202 based on the average distance obtained by the average distance acquisition unit 204. The distance map processing unit 203 outputs the processed distance map to the cutout image generation unit 205.
[0026] Reference Figure 4 Sections (A) to (C) describe operational examples of the distance map processing unit 203 and the average distance acquisition unit 204.
[0027] Figure 4 (B) is a schematic diagram of the distance map acquired by the distance measurement unit 202. Figure 4 In the distance diagram (B), the subject distance of region 402 (the brightness (signal level) of region 402) is long, and region 402 is not shown as a subject area. When the user touches region 402, the average distance acquisition unit 204 acquires the distance from region 402. Figure 4 The distance map in (B) represents the average distance (mean distance) of multiple subjects.
[0028] Distance map processing unit 203 generates a distance map based on the average distance obtained by average distance acquisition unit 204. Figure 4 The gain function 401 of (A) represents the correspondence between the subject distance and the gain value of the subject distance (multiplied by a coefficient of the subject distance). Instead of the gain function 401, a table representing the correspondence between the subject distance and the gain value can be generated. Figure 4The horizontal axis of (A) represents the distance to the subject, and Figure 4 The vertical axis of (A) represents the gain value. As the subject distance increases from t1 to t3, the gain value increases linearly from 0 to 2, and as the subject distance increases from t3 to t4, the gain value decreases linearly from 2 to 0. The subject distance t3 is the average distance obtained by the average distance acquisition unit 204, and the subject distances t1 and t4 are determined based on the subject distance t3. The subject distance t2 is... Figure 4 The distance of the subject in region 403 of (B).
[0029] The distance map processing unit 203 processes the distance map output from the distance measurement unit 202 by using gain processing of the generated gain function. In the gain processing, the subject distance at each location is multiplied by a gain value corresponding to that subject distance. Figure 4 (C) is in use Figure 4 Gain function processing of (A) Figure 4 A schematic diagram of the distance map following the distance map of (B). By using... Figure 4 The gain function of (A), Figure 4 The brightness (signal level) of region 402 in area (B) increases. As a result, in Figure 4 In the distance graph of (C), with Figure 4 Region 404, corresponding to region 402 of (B), is exemplified as the subject region. Figure 4 In the distance map (B), the brightness (signal level) of region 402 is lower than that of region 403. On the other hand, in Figure 4 In the distance graph of (C), with Figure 4 The brightness (signal level) of region 404 corresponding to region 402 of (B) is higher than that of region 404. Figure 4 The brightness (signal level) of region 405 corresponding to region 403 of (B).
[0030] In this way, the distance map is processed based on the distances of multiple subjects within the touch area. Note that although an example has been described of determining a gain function with a large gain value within a distance range that includes the average distances of multiple subjects within the touch area, it is also possible to determine a gain function with a small gain value within a distance range that includes the average distances of multiple subjects within the touch area (details will be described later). Furthermore, although an example using the average distances of multiple subjects within the touch area has been described, the method of processing the distance map is not limited to this. The distance map can be processed based on the maximum and minimum values of the multiple subject distances within the touch area. For example, the maximum and minimum values of the multiple subject distances within the touch area can be obtained, and a gain function with a large (or small) gain value within a distance range from the maximum to the minimum value can be determined. The distance map can be processed based on a histogram of the multiple subject distances within the touch area. For example, a distance map can be processed using a large (or small) gain value in a region of a histogram similar to the histogram of the touch area. Furthermore, the distance map can be processed using a different process than gain processing. The processed distance map can be a binary (1-bit value) map that indicates whether a location is a subject area by the signal level at each location.
[0031] The cutout image generation unit 205 generates a cutout image by performing a compositing process that combines the image matting image output from the image matting unit 201 and the distance map output from the distance map processing unit 203. In the compositing process, for example, selective processing such as maximum value selection, minimum value selection, addition, subtraction, multiplication, and division are performed on the two images. In the compositing process, the signal level of either the image matting image or the distance map can be selected as the signal level of the cutout image based on a comparison between the signal level (pixel value) of the distance map and a threshold. For example, when the signal level of the distance map is less than the threshold, the signal level of the image matting image can be selected as the signal level of the cutout image. Then, when the signal level of the distance map is equal to or greater than the threshold, the signal level of the distance map can be selected as the signal level of the cutout image.
[0032] Figure 5 This is a block diagram illustrating the configuration of the cutout processing unit 112. The cutout processing unit 112 includes a cutout unit 501 and a cutout area overlay unit 502.
[0033] The cutting unit 501 cuts out a cut-out region represented by a cut-out image from the captured image. For example, the cutting unit 501 generates and outputs an image as a cut-out image in which the pixel values of the regions in the cut-out image whose signal level is equal to or greater than a threshold are the same as the pixel values in the captured image, and the pixel values of the regions in the cut-out image whose signal level is equal to or greater than the threshold are predetermined values (e.g., black pixel values).
[0034] The cutout region overlay unit 502 detects the contour line of the cutout region represented by the cutout map, overlays the contour line onto the captured image, and outputs the captured image with the overlaid contour line (cutout region overlay image). There are no particular restrictions on the method for detecting the contour line, and known methods can be used.
[0035] Figure 6 This is a schematic diagram of the overlay of images from the cropped regions. Figure 6 In the image, the outline of the cut-out area is superimposed on the captured image, in which three figures standing in a row at different distances from the image processing device 101 (camera device) are depicted. Figure 6 In the image, the area containing the three figures is defined as the cutout region, and a gray outline surrounding these three figures is overlaid as the cutout region's outline. By overlaying the cutout region's outline onto the captured image and displaying that image, the user is notified of the cutout region. Note that there are no specific restrictions on the outline's color.
[0036] Figure 7 This is a flowchart of the cutout process performed by the image processing device 101. Figure 7 The cut-out process is achieved by the control unit 108 loading the program stored in ROM 109 into RAM 110 and executing the program. For example, when the user instructs the execution of a predetermined image editing application, Figure 7 The cutting process begins.
[0037] In step S701, the control unit 108 selects a user-specified image from a plurality of captured images stored in the storage unit 106 and displays the selected image on the display unit 107. At this time, an image display mode is set in the image processing device 101, which is an operation mode for displaying captured images on the display surface.
[0038] In step S702, in response to a long tap on the display unit 107 (touchscreen), the control unit 108 switches the operating mode to be set from the image display mode to the cutout mode, which is an operating mode for cutting out a region from the captured image. Then, the control unit 108 controls the region detection unit 111 to determine the cutout region, controls the cutout region overlay unit 502 to overlay the outline of the cutout region onto the captured image, and switches the image to be displayed as the cutout region overlay image.
[0039] In image matting, a region of a subject can be determined with high precision. Therefore, in step S702, it is assumed that only the image matting unit 201 is used to determine the cut-out region. That is, it is assumed that the subject region detected by the image matting unit 201 is determined as the cut-out region.
[0040] Note that the predefined touch operation used to set the switch mode is not limited to a long tap. For example, the predefined touch operation can be a double tap, swipe, flick, or drag. A long tap is a touch operation that touches the display surface for a time longer than a first time and then releases the display surface, and a double tap is a touch operation that performs a tap (touches the display surface for a time shorter than the first time and then releases the display surface) and then taps again before a second time has elapsed. A swipe is a touch operation that moves across the display surface, a flick is a touch operation that flicks the display surface, and a drag is a touch operation that moves the touch position after touching a point on the display surface for a time longer than a third time.
[0041] The user views the overlay image of the cut-out region, confirms whether the desired region has been identified as the cut-out region, and inputs the confirmation result into the image processing device 101. In step S703, the control unit 108 determines whether the user's desired region has been identified as the cut-out region based on the input from the user. When the user's desired region is identified as the cut-out region, the process proceeds to step S712, and when the user's desired region is not identified as the cut-out region, the process proceeds to step S704.
[0042] When the user's desired area is not identified as the cutout area, the user touches (designates) the portion to be corrected. In step S704, the control unit 108 receives the touch (designation) of the portion to be corrected. The portion to be corrected is either the portion to be added to the cutout area or the portion to be removed from the cutout area. When displayed... Figure 6 When the cropped area is overlaid with an image and the user expects only the middle character to be cropped, the user touches the foremost and last characters.
[0043] In step S705, the control unit 108 controls the average distance acquisition unit 204 to acquire the average value (average distance) of multiple subject distances represented by a distance map for the user's touch area. At this time, it is assumed that the distance map has already been obtained by the distance measurement unit 202.
[0044] In step S706, the control unit 108 determines whether the location specified by the user (the part to be corrected) is included in the current cutout area (the subject area detected by the image matting unit 201). When the specified location is not included in the current cutout area, the process proceeds to step S707, and when the specified location is included in the current cutout area, the process proceeds to step S709. Although details will be described later, when the process proceeds to step S707, it is determined that the cutout area includes a distance region based on the specified location. The distance region based on the specified location is the area represented by the distance map before processing based on the distance of the subject within the distance range of the specified location. That is, the area of the subject at the specified location is added to the cutout area. When the process proceeds to step S709, it is determined that the cutout area does not include the distance region based on the specified location. That is, the area of the subject at the specified location is removed from the cutout area.
[0045] In step S707, the control unit 108 controls the distance map processing unit 203 to process the distance map to represent a subject area including a distance region based on a user-specified location (the portion to be corrected). As described above, the distance region based on the specified location is the area represented by the unprocessed distance map based on the subject distance within the distance range of the specified location. For example, such as Figure 4 As illustrated in (A), a gain function with a large gain value is determined in the distance range that includes the average distance of multiple subjects in the touch area, and the gain function is used to process the distance map.
[0046] In step S708, the control unit 108 controls the cutout image generation unit 205 to update the cutout region. Along with the update of the cutout region, the overlay image of the cutout region to be displayed is also updated. The control unit 108 determines the updated cutout region based on the current cutout region (the subject region detected by the image matting unit 201) and the subject region represented by the processed distance map. For example, the control unit 108 determines the updated cutout region as the region included in at least one of the current cutout region and the subject region represented by the processed distance map. Such an updated cutout region can be determined, for example, by a maximum value selection process, which selects the larger of the signal level of the current cutout image (image matting image) and the signal level of the processed distance map.
[0047] In step S709, the control unit 108 controls the distance map processing unit 203 to process the distance map to represent a subject area excluding distance regions based on a user-specified location (the portion to be corrected). As described above, the distance region based on the specified location is the area represented by the unprocessed distance map based on the subject distance within the distance range of the specified location. For example, such as Figure 8As illustrated in (A), a gain function with a small gain value is determined in the distance range that includes the average distance of multiple subjects in the touch area, and the gain function is used to process the distance map.
[0048] Figure 8 (A) is a schematic diagram of the gain function. Figure 8 (B) is a schematic diagram of the distance map before processing, and Figure 8 (C) is a schematic diagram of the processed distance map. Assuming that in the display... Figure 6 When the cropped area is overlaid with an image, the user touches... Figure 8 Areas 802 and 803 of (B) (the last character and the first character) are cut out only to show the middle character. Figure 8 In the gain function 801 of (A), the gain value is 0 in the distance range including subject distance t1 and the distance range including subject distance t3, and the gain value is 2 between these distance ranges. Subject distance t1 is Figure 8 The subject distance in region 802 of (B), and the subject distance t3 is Figure 8 The subject distance in region 803 of (B). Subject distance t2 is the subject distance of the middle person. By using Figure 8 The gain function of (A) Figure 8 By processing the distance map of (B), we can obtain the area that only represents the middle character. Figure 8 Distance plot of (C).
[0049] In step S709, the control unit 108 controls the cutout image generation unit 205 to update the cutout region. Along with the update of the cutout region, the overlay image of the cutout region to be displayed is also updated. The control unit 108 determines the updated cutout region based on the current cutout region (the subject region detected by the image matting unit 201) and the subject region represented by the processed distance map. For example, the control unit 108 determines the updated cutout region as the area included in both the current cutout region and the subject region represented by the processed distance map. Such an updated cutout region can be determined, for example, by a minimum value selection process used to select the smaller of the signal level of the current cutout image (image matting image) and the signal level of the processed distance map.
[0050] Figure 9 This is a schematic diagram illustrating the minimum value selection process for the image matting map and the processed distance map. In the image matting map, the median value is large in the regions of all three figures, while in the processed distance map, the median value is large only in the region of the middle figure. In this case, by using minimum value selection processing, a cutout map can be obtained where the median value is large only in the region of the middle figure.
[0051] Figure 10 This is a schematic diagram of the updated overlaid image of the cut-out region. Because the cut-out image has been updated to only represent the area of the central figure, therefore... Figure 10 The cutout region in the overlay image is an example that only represents the outline of the area of the person in the middle.
[0052] In step S711, the control unit 108 determines whether the user's desired area is identified as a cut-out area based on the input from the user. When the user's desired area is identified as a cut-out area, the process proceeds to step S711, and when the user's desired area is not identified as a cut-out area, the process proceeds to step S704.
[0053] In step S712, the control unit 108 cuts out the cut-out area according to the instruction from the user (e.g., copy instruction).
[0054] As described above, according to the first embodiment, by determining the cutout region based on the image matting map, the distance map, and the position specified when cutting out the region from the captured image, it becomes possible to cut out the desired region from the captured image with high precision.
[0055] <Second Embodiment> A second embodiment of the invention will now be described. Note that, in the following text, descriptions of points identical to those in the first embodiment (e.g., configurations and processes identical to those in the first embodiment) will be omitted, and points different from those in the first embodiment will be described. In the first embodiment, detection of the subject region by image matting has been described. In the second embodiment, the subject region is detected by a method different from image matting.
[0056] Figure 11 This is a block diagram illustrating the configuration of the region detection unit 111 according to the second embodiment. The region detection unit 111 according to the second embodiment includes a subject detection unit 1100 instead of the image matting unit 201 of the first embodiment.
[0057] Subject detection unit 1100 detects subject regions from the captured image. For example, subject detection unit 1100 detects facial and body regions of a person and identifies a rectangular region including the facial and body regions as the subject region. There are no particular limitations on the method for detecting facial and body regions, and facial regions can be detected, for example, using known facial detection processing. Facial regions can be detected using information related to the face (e.g., skin color information and information related to facial organs such as eyes, nose, and mouth). Subject detection unit 1100 can be a discriminator that uses a learning algorithm, such as a neural network, to detect facial regions. Similarly, body regions can be detected using known human detection processing. Subject detection unit 1100 outputs a subject detection map representing the detected subject regions to cropping image generation unit 205.
[0058] Figure 12 This is a schematic diagram illustrating the minimum value selection process for a subject detection map and a processed distance map. In the subject detection map, the median value is large in three rectangular regions that respectively include three people, and in the processed distance map, the median value is large only in the region of the middle person. In this case, by using the minimum value selection process, a cutout map can be obtained where the median value is large only in the region of the middle person (i.e., compared with the first embodiment). Figure 9 (Similar to the cutout image).
[0059] As described above, even with the configuration of the second embodiment that uses a different method than image matting to detect the rectangular subject region, it becomes possible to cut out the desired region from the captured image with high precision, similar to the first embodiment.
[0060] Note that the various types of control described above can be performed by a single piece of hardware (e.g., a processor or circuitry) or by processing in other ways. Processing can be distributed among multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits), thereby performing control over the entire device.
[0061] Furthermore, the processors mentioned above are processors in a broad sense, encompassing both general-purpose and special-purpose processors. Examples of general-purpose processors include central processing units (CPUs), microprocessor units (MPUs), and digital signal processors (DSPs). Examples of special-purpose processors include graphics processing units (GPUs), application-specific integrated circuits (ASICs), and programmable logic devices (PLDs). Examples of PLDs include field-programmable gate arrays (FPGAs) and complex programmable logic devices (CPLDs).
[0062] The above embodiments (including variations) are merely examples. Any configurations obtained by appropriately modifying or changing some configurations of the embodiments within the scope of this disclosure are also included in this disclosure. This disclosure also includes other configurations obtained by appropriately combining various features of the embodiments.
[0063] <Other Embodiments> The present invention can also be implemented by supplying a program for implementing one or more of the functions of the above embodiments to a system or device via a network or storage medium, and causing one or more processors of the computer of the system or device to read and execute the program. Furthermore, the present invention can also be implemented by a circuit (e.g., an ASIC) for implementing one or more functions.
[0064] The disclosure of this embodiment includes the following configurations, methods, procedures, and media. (Configuration 1) An image processing apparatus, comprising: A detection unit is configured to detect a subject region from an captured image; An acquisition unit, configured to acquire a distance map corresponding to the captured image; and A determining unit is configured to determine a cutout region based on the subject region, the distance map, and a position specified when cutting out a region from the captured image, the cutout region being the region to be cut out from the captured image. (Configuration 2) According to the image processing apparatus described in configuration 1, wherein, The detection unit uses a learned model to detect the subject region. (Configuration 3) The image processing apparatus according to configuration 1 or 2 further includes: The setting unit is configured to switch the operation mode to be set from the first mode to a second mode for cutting out a region from the captured image in response to a predetermined touch operation performed on the display surface while a first mode for displaying the captured image on the display surface is set. (Configuration 4) According to the image processing apparatus described in configuration 3, wherein... The predetermined touch operation is a long tap, a double tap, a swipe, a flick, or a drag. (Configuration 5) The image processing apparatus according to any one of configurations 1 to 4, wherein, The determining unit processes the distance map based on multiple distances represented by the distance map, targeting the specified location and its surrounding area. The determining unit determines the cutout area based on the subject area and the processed distance map. (Configuration 6) According to the image processing apparatus described in configuration 5, wherein... The determining unit processes the distance map based on the average of the plurality of distances. (Configuration 7) According to the image processing apparatus described in configuration 5, wherein... The determining unit processes the distance map based on the histograms of the plurality of distances. (Configuration 8) The image processing apparatus according to any one of configurations 1 to 7 further includes: A control unit controls the display of the captured image when a region is cut out from the captured image, such that the cut-out region is recognizable. (Configuration 9) The image processing apparatus according to any one of configurations 1 to 8, wherein, When the specified location is included in the subject area, the determining unit determines a cutout area that does not include the area where distances within a distance range based on the specified location are represented by the distance map. If the specified location is not included in the subject area, the determining unit determines a cutout area including the distance range based on the specified location represented by the distance map. (Configuration 10) The image processing apparatus according to any one of configurations 1 to 9, wherein, If the specified location is included in the subject area, the determining unit processes the distance map to represent a second subject area excluding the area where the distance range based on the specified location is represented by the distance map. If the specified location is not included in the subject area, the determining unit processes the distance map to represent a second subject area including the distance range based on the specified location, where the distance range is represented by the distance map. The determining unit determines the cutout area based on the subject area and the second subject area. (Configuration 11) According to the image processing apparatus described in configuration 10, wherein, When the specified location is included in the subject area, the determining unit determines the area included in both the subject area and the second subject area as the cut-out area, and If the specified location is not included in the subject area, the determining unit will determine the area included in at least one of the subject area and the second subject area as the cut-out area. (method) An image processing method, comprising: The detection step is used to detect the subject region from the captured image; The acquisition step is used to acquire a distance map corresponding to the captured image; and The determination step is used to determine the cutout region based on the subject region, the distance map, and a position specified when cutting out a region from the captured image, the cutout region being the region to be cut out from the captured image. (program) A program that enables a computer to be used as a unit of an image processing apparatus according to any one of configurations 1 to 11. (medium) A computer-readable medium storing programs for enabling a computer to function as various units of an image processing apparatus according to any one of configurations 1 to 11.
[0065] This invention is not limited to the embodiments described above, and various modifications and alterations can be made to this invention without departing from its spirit and scope. Therefore, the appended claims are added to disclose the scope of the invention.
[0066] This application claims priority to Japanese Patent Application 2024-043309, filed on March 19, 2024, all of which are disclosed therein and are incorporated herein by reference. List of reference numerals
[0067] 101 Image Processing Device 108 Control Unit 111 Area Detection Unit 201 Image matting unit 202 Distance Measurement Unit 203 Distance Map Processing Unit 204 Average Distance Acquisition Units 205 Cutout Image Generation Unit 1100 Subject Detection Unit
Claims
1. An image processing apparatus, comprising: A detection unit is configured to detect a subject region from an captured image; An acquisition unit is configured to acquire a distance map corresponding to the captured image; as well as A determining unit is configured to determine a cutout region based on the subject region, the distance map, and a position specified when cutting out a region from the captured image, the cutout region being the region to be cut out from the captured image.
2. The image processing apparatus according to claim 1, wherein, The detection unit uses a learned model to detect the subject region.
3. The image processing apparatus according to claim 1, further comprising: The setting unit is configured to switch the operation mode to be set from the first mode to a second mode for cutting out a region from the captured image in response to a predetermined touch operation performed on the display surface while a first mode for displaying the captured image on the display surface is set.
4. The image processing apparatus according to claim 3, wherein, The predetermined touch operation is a long tap, a double tap, a swipe, a flick, or a drag.
5. The image processing apparatus according to claim 1, wherein, The determining unit processes the distance map based on multiple distances represented by the distance map, targeting the specified location and its surrounding area. The determining unit determines the cutout area based on the subject area and the processed distance map.
6. The image processing apparatus according to claim 5, wherein, The determining unit processes the distance map based on the average of the plurality of distances.
7. The image processing apparatus according to claim 5, wherein, The determining unit processes the distance map based on the histograms of the plurality of distances.
8. The image processing apparatus according to claim 1, further comprising: A control unit controls the display of the captured image when a region is cut out from the captured image, such that the cut-out region is recognizable.
9. The image processing apparatus according to claim 1, wherein, When the specified location is included in the subject area, the determining unit determines a cutout area that does not include the area where distances within a distance range based on the specified location are represented by the distance map. If the specified location is not included in the subject area, the determining unit determines a cutout area including the distance range based on the specified location represented by the distance map.
10. The image processing apparatus according to claim 1, wherein, If the specified location is included in the subject area, the determining unit processes the distance map to represent a second subject area excluding the area where the distance range based on the specified location is represented by the distance map. If the specified location is not included in the subject area, the determining unit processes the distance map to represent a second subject area including the distance range based on the specified location, where the distance range is represented by the distance map. The determining unit determines the cutout area based on the subject area and the second subject area.
11. The image processing apparatus according to claim 10, wherein, When the specified location is included in the subject area, the determining unit determines the area included in both the subject area and the second subject area as the cut-out area, and If the specified location is not included in the subject area, the determining unit determines the area included in at least one of the subject area and the second subject area as the cut-out area.
12. An image processing method, comprising: The detection step is used to detect the subject region from the captured image; The acquisition step is used to acquire a distance map corresponding to the captured image; as well as The determination step is used to determine the cutout region based on the subject region, the distance map, and a position specified when cutting out a region from the captured image, the cutout region being the region to be cut out from the captured image.
13. A program that enables a computer to function as various units of an image processing apparatus according to any one of claims 1 to 11.
14. A computer-readable medium storing a program for causing a computer to function as various units of an image processing apparatus according to any one of claims 1 to 11.
Citation Information
Patent Citations
Medicine progress management device, medicine progress management system and medicine progress management method
JP2024043309A
