Information processing device, processing method of information processing device, and program

Abstract

To provide an information processing device capable of acquiring highly-accurate distance information on a region of a subject.SOLUTION: An information processing device includes: subject extraction means 107 for extracting a region of a subject from an image so as to generate region information on the subject; distance information calculation means 106 for calculating distance information corresponding to the image on the basis of information related to the distance to the image; and region extraction means 109 for extracting distance information on the region of the subject from the distance information calculated by the distance information calculation means by using the region information on the subject, which has been generated by the subject extraction means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a processing method for an information processing device, and a program. [Background technology]

[0002] In mixed reality, where virtual space information is superimposed on real space in real time and displayed to the user, it is necessary to estimate the 3D shape of the real space. In particular, a high-quality mixed reality can be achieved by quickly and accurately estimating the specific subject area in real space that is to be superimposed on the virtual space and its 3D shape.

[0003] Patent Document 1 discloses a technology in which corresponding points are detected across the entire stereo image to estimate the depth, and based on the results, the depth is measured again using the distance from the subject to be measured to the camera as the estimated distance range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-45283 A Summary of the Invention [Problem to be solved by the invention]

[0005] When acquiring only distance information to a specific subject that is the subject of imaging distance measurement, it is difficult to extract the specific subject area with high accuracy from the overall distance information. For example, if the distance to the subject and the distance to the background are close, it is difficult to determine whether the subject is the subject based on the distance information alone, even if the approximate distance to the subject is known in advance.

[0006] In addition, distance measurement errors can cause the system to measure the distance to a part of the background as being within the distance of the subject, or to measure the distance to the subject as being within the distance of the background, resulting in the problem of outputting erroneous extraction results that include the background area or omit the subject area that should actually be included.

[0007] An object of the present disclosure is to make it possible to obtain highly accurate distance information of a subject area. [Means for solving the problem]

[0008] The information processing device has a subject extraction means that extracts a subject area from an image and generates subject area information, a distance information calculation means that calculates distance information corresponding to the image based on information about the distance of the image, and an area extraction means that uses the subject area information generated by the subject extraction means to extract distance information of the subject area from the distance information calculated by the distance information calculation means. [Effects of the Invention]

[0009] According to the present disclosure, highly accurate distance information of a subject area can be obtained. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] 10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 5] 10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 6] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 7]10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 8] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 9] 10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 10] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 11] 10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 12] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 13] 10 is a flowchart illustrating an example of processing executed by an information processing device. [Figure 14] FIG. 2 is a block diagram illustrating an example of a functional configuration of the imaging system. [Figure 15] 10 is a flowchart illustrating an example of processing executed by an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. The configurations described in each embodiment are representative examples, and the invention is not necessarily limited to those specific configurations.

[0012] (First embodiment) The first embodiment will be described with reference to the drawings. The first embodiment is configured to acquire only distance information to a specific subject.

[0013] 1 is a block diagram showing an example of the functional configuration of an imaging system 100 according to the first embodiment. As shown in Fig. 1, the imaging system 100 includes an imaging device 101, a distance measuring device 103, and an information processing device 105. In the imaging system 100, the imaging device 101 and the distance measuring device 103 are connected to the information processing device 105.

[0014] The imaging device 101 has an imaging section 102, and outputs an image captured by the imaging section 102 to an information processing device 105. The imaging section 102 has a lens (not shown) and one or more imaging elements.

[0015] The distance measuring device 103 has a distance measuring unit 104, and outputs distance measurement information acquired by the distance measuring unit 104 to the information processing device 105. The distance measurement information is an example of information relating to the distance of the image captured by the imaging unit .

[0016] The information processing device 105 includes a distance information calculation unit 106 , a subject extraction unit 107 , a region information storage unit 108 , and a region extraction unit 109 .

[0017] The distance information calculation unit 106 receives the distance measurement information input from the distance measurement device 103 and calculates the distance information according to the type of distance measurement information. The distance information is information that indicates the distance from the distance measurement unit 104 to the subject. The distance information calculation unit 106 outputs the calculated distance information to the area extraction unit 109.

[0018] The subject extraction unit 107 receives an image input from the imaging device 101, determines the subject based on predetermined conditions, and acquires area information of the subject. For example, the subject extraction unit 107 may perform the determination based on the color or brightness of the captured image. The subject extraction unit 107 outputs the acquired area information of the subject to the area information storage unit 108.

[0019] The region information holding unit 108 holds the region information of the subject acquired by the subject extraction unit 107. Then, the region information holding unit 108 outputs the held region information of the subject to the region extraction unit 109.

[0020] The area extraction unit 109 extracts the distance information calculated by the distance information calculation unit 106 within the range of the subject, using the area information of the subject stored in the area information storage unit 108. Then, the area extraction unit 109 outputs the distance information extracted within the area of ​​the subject to the outside of the information processing device 105.

[0021] Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device 105 shown in Fig. 1. The information processing device 105 includes a bus 200, a CPU 201, a RAM 202, a ROM 203, an input I / F 204, and an output I / F 205.

[0022] A CPU 201 accesses a RAM (random access memory) 202 and a ROM (read only memory) 203 via a bus 200, and executes programs stored in the RAM 202 and the ROM 203 and references and writes data.

[0023] The input I / F 204 inputs the captured image and distance measurement information from the image capturing device 101 and distance measuring device 103 in a format that can be processed by the information processing device 105 .

[0024] The output I / F 205 outputs the distance information processed by the information processing device 105 in a format that can be processed by an external output destination.

[0025] Fig. 3 is an example of a flowchart showing the processing up to outputting subject extraction distance information from a captured image and distance measurement information by the information processing device 105 in Fig. 1. Fig. 3 shows a processing method of the information processing device 105.

[0026] In step S300, the CPU 201 receives the distance measurement information acquired by the distance measuring device 103 via the input I / F 204. Then, the CPU 201 temporarily stores the received distance measurement information in the RAM 202 or the like.

[0027] In step S301, the distance information calculation unit 106 calculates distance information based on the distance measurement information acquired by the distance measuring device 103. The distance measurement information is an example of information relating to the distance of an image captured by the imaging unit 102. The distance information is distance information corresponding to the image captured by the imaging unit 102.

[0028] For example, the distance measurement unit 104 is a Light Detection and Ranging (LiDAR) device including a laser element and a light receiving element (not shown). The distance information calculation unit 106 calculates distance information to the target based on the distance measurement information acquired by the LiDAR. The distance measurement information and distance information are, for example, information on a two-dimensional region corresponding to a two-dimensional image captured by the imaging device 101.

[0029] In this case, the distance information calculation unit 106 may perform a projection transformation on the distance information before calculating the distance information in order to align the positions of the captured image and the distance information due to differences in the viewpoint and angle of view of the imaging device 101 and the distance measuring device 103.

[0030] For example, if the approximate distance to a specific subject is known in advance, the distance information calculation unit 106 may calculate distance information by treating only distances within a predetermined range from that distance as valid values. In this case, for the range outside the valid values, the distance information calculation unit 106 may output a predetermined value that was set in advance.

[0031] In step S302, the CPU 201 receives an image captured by the imaging device 101 via the input I / F 204. Then, the CPU 201 temporarily stores the received image in the RAM 202 or the like.

[0032] In step S303, the subject extraction unit 107 extracts a region of a specific subject from the image captured by the imaging unit 102, and generates and outputs region information of the subject. For example, if the specific subject has a characteristic shape, the subject extraction unit 107 may store information about the characteristic of the subject in advance and determine the region having the characteristic as the subject region.

[0033] For example, if a specific subject has a characteristic color, the subject extraction unit 107 may register the color of the subject in advance and determine the subject area based on whether the color is that color. For example, the subject extraction unit 107 may have a function to recognize and detect subjects, and may determine the subject area based on the results of the recognition and detection.

[0034] In step S304, the region information holding unit 108 stores the region information of the subject output by the subject extraction unit 107. For example, the region information holding unit 108 can store the region information of the subject in the same image where each pixel is an 8-bit object. In comparison with this case, to reduce the amount of RAM 202 used, the region information holding unit 108 may store the region information of the subject as 1-bit information for each pixel, such as 1 for a subject region and 0 for a non-subject region.

[0035] For example, if it is assumed that the object area is not complex in shape, the boundaries of the object area may be regarded as edges and object area information may be stored to reduce the amount of RAM used in the RAM 202. In this case, the object area information may be configured by combining multiple straight lines, each of which connects two pixels.

[0036] Note that the above steps S300 to S301 and steps S302 to S304 may be executed in parallel, or one of them may be executed sequentially.

[0037] In step S305, area extraction unit 109 uses the area information of the subject stored in area information storage unit 108 to extract distance information of the subject area from the distance information calculated by distance information calculation unit 106, and outputs the subject extraction distance information. The subject extraction distance information is distance information of the area of ​​a specific subject. For example, in order to clearly indicate areas other than the specific subject, area extraction unit 109 may output the distance information included in the subject area as is, and may overwrite the distance information included in areas other than the subject area with a separately set value and output it.

[0038] As described above, according to this embodiment, the information processing device 105 can output distance information of only a specific subject by extracting distance information of the subject area from the calculated distance information.

[0039] (Second embodiment) The second embodiment will be described with reference to the drawings. Hereinafter, the same components as those in the first embodiment will be designated by the same reference numerals, and the description thereof will be omitted.

[0040] Generally, the resolution of the image sensor of the imaging unit 102 is greater than the resolution of the distance measurement sensor such as LiDAR of the distance measurement unit 104. In particular, when focusing on the subject, there is a problem that the contour of the subject in the distance information is rougher and less accurate than the contour of the subject in the image information.

[0041] In order to address this issue, the first embodiment is configured to adjust the resolution of the distance information before subject extraction, thereby improving the accuracy of the contour of the distance information of the subject that is finally extracted.

[0042] 4 is a block diagram showing an example of the functional configuration of an image capturing system 400 according to the second embodiment. The image capturing system 400 includes an image capturing device 101, a distance measuring device 103, and an information processing device 401.

[0043] The information processing device 401 has a distance information calculation unit 106, a subject extraction unit 107, a region information storage unit 108, a region extraction unit 109, and further has a distance information enlargement / reduction unit 402. The distance information enlargement / reduction unit 402 enlarges or reduces the resolution of the distance information calculated by the distance information calculation unit 106 and outputs it to the region extraction unit 109.

[0044] 5 is a flowchart showing an example of processing performed by the information processing device 401 up to outputting subject extraction distance information from a captured image and distance measurement information.

[0045] In Fig. 5, step S501 is added to Fig. 3. Step S501 is performed after step S301. After step S501, the process of step S305 is performed.

[0046] In step S501, the distance information scaling unit 402 scales the resolution of the distance information calculated by the distance information calculation unit 106 to a predetermined resolution. The interpolation algorithm used for scaling may be a known nearest neighbor method or bilinear method.

[0047] For example, if the resolution of the distance measurement information output by the distance measuring device 103 is lower than the resolution of the image information output by the imaging device 101, the distance information enlargement / reduction unit 402 may enlarge the distance information to match the resolution of the image information so that the subject can be appropriately extracted by the area extraction unit 109. In this case, it is desirable to match the resolution of the image information with the resolution of the distance information, but they do not necessarily have to match.

[0048] In step S305, the area extraction unit 109 uses the area information of the subject stored in the area information storage unit 108 to extract distance information of the subject area from the distance information enlarged or reduced by the distance information enlargement / reduction unit 402, and outputs the subject extraction distance information.

[0049] In step S305, if the resolution of the image information is greater than the resolution of the distance information, the area extraction unit 109 may adjust the area of ​​the subject extraction distance information so as to output results corresponding to the resolution of the distance information and to output only the range where both the image information and the distance information are valid.

[0050] In the second embodiment, the area information of the subject may be enlarged or reduced as shown in FIG.

[0051] 6 is a block diagram showing an example of the functional configuration of an imaging system 600. The imaging system 600 has an imaging device 101, a distance measuring device 103, and an information processing device 601. The information processing device 601 is obtained by adding a region information enlargement / reduction unit 602 to the information processing device 401 in FIG. 4. The region information enlargement / reduction unit 602 enlarges or reduces the resolution of the region information of the subject stored in the region information holding unit 108, and outputs the result to the region extraction unit 109.

[0052] FIG. 7 is an example of a flowchart showing the processing up to when the information processing device 601 outputs subject extraction distance information from a captured image and distance measurement information.

[0053] In Fig. 7, step S701 is added to Fig. 5. Step S701 is performed after step S304. After step S701, the process of step S305 is performed.

[0054] In step S701, the region information enlargement / reduction unit 602 enlarges or reduces the resolution of the region information of the subject stored in the region information storage unit 108 to the resolution of the subject extraction distance information to be output. For example, if the resolution of the subject extraction distance information to be output does not match the resolution of the captured image, the region information enlargement / reduction unit 602 enlarges or reduces the resolution of the subject region information to match the resolution of the subject extraction distance information, and outputs it to the region extraction unit 109.

[0055] In step S305, the area extraction unit 109 uses the area information of the subject enlarged or reduced by the area information enlargement / reduction unit 602 to extract distance information of the area of ​​the subject from the distance information enlarged or reduced by the distance information enlargement / reduction unit 402, and outputs the subject extraction distance information.

[0056] As described above, according to this embodiment, the information processing devices 401 and 601 enlarge the distance information to fit the subject image and then extract it from the subject area, regardless of whether they have the configurations shown in Figures 4 or 6. This allows the information processing devices 401 and 601 to output distance information with high accuracy of the subject contour even when the resolution of the image information is higher than the resolution of the distance information.

[0057] (Third embodiment) The third embodiment will be described with reference to the drawings. Hereinafter, the same components as those in the above-described embodiments will be designated by the same reference numerals, and the description thereof will be omitted.

[0058] The configuration of the second embodiment has a problem in that when projection transformation is performed to calculate distance, taking into account the difference in position between the image capture device 101 and the distance measurement device 103, an erroneous calculation of the distance to the background area other than the subject can result in the background area being projected onto the subject area, which can cause an error in the distance information of the subject. The third embodiment has a configuration that addresses this problem.

[0059] 8 is a block diagram showing an example of the functional configuration of an imaging system 800 according to the third embodiment. The imaging system 800 includes an imaging device 101, a distance measuring device 103, and an information processing device 801. The information processing device 801 is configured by adding a distance information calculation unit 802 instead of the distance information calculation unit 106 to the information processing device 601 in FIG. 6. The distance information calculation unit 802 selects a region of the distance measurement information using region information of the subject output by the subject extraction unit 107, calculates distance information, and outputs the distance information to the distance information enlargement / reduction unit 402.

[0060] FIG. 9 is an example of a flowchart showing the processing up to when the information processing device 801 outputs subject extraction distance information from a captured image and distance measurement information.

[0061] 9 is obtained by adding step S901 to FIG. 7. After steps S300 and S302, the process of step S303 is performed. Then, the processes of steps S901 and S304 are performed. After step S901, the process of step S501 is performed.

[0062] In step S901, the distance information calculation unit 802 selects a region of the ranging information using the region information of the subject output by the subject extraction unit 107, and calculates distance information. Therefore, the distance information calculation unit 802 calculates and outputs distance information from the ranging information in the region of the subject, and does not calculate distance information in the region other than the region of the subject, but outputs a predetermined value. For example, the distance information output in the region other than the region of the subject may be a fixed value such as 0 or infinity, regardless of the pixel position. Furthermore, when performing projection transformation processing, the distance information calculation unit 802 may treat the distance information in the region other than the region of the subject as infinity and perform the projection transformation processing. In this case, the distance information calculation unit 802 calculates distance information by the projection transformation processing.

[0063] As described above, according to this embodiment, the information processing device 801 can use the area information of the subject obtained by extracting the subject to obtain distance information that reduces the influence of distance information of areas other than the subject area.

[0064] (Fourth embodiment) The fourth embodiment will be described with reference to the drawings. Hereinafter, the same components as those in the above-described embodiments will be designated by the same reference numerals, and the description thereof will be omitted.

[0065] In the fourth embodiment, a distance measuring camera is used instead of the imaging device 101 and distance measuring device 103 of the first to third embodiments, and block matching is performed from the image captured by the distance measuring camera to calculate distance information.

[0066] Generally, when the block of block matching is large, even if the pixel of interest is in an area other than the subject, it may be matched with the subject area within the block, and distance information for the pixel of interest may not be calculated correctly. This poses a problem that the contour of the subject area in the distance information may differ from the contour of the subject area in the captured image. The fourth embodiment is configured to address this problem.

[0067] 10 is a block diagram showing an example of the functional configuration of an image capturing system 1000 according to the fourth embodiment. The image capturing system 1000 includes a distance measuring camera 1001 and an information processing device 1003.

[0068] The distance measuring camera 1001 has an image capturing unit 1002 and outputs an image captured by the image capturing unit 1002 to an information processing device 1003 .

[0069] The information processing device 1003 is configured by providing a distance information calculation unit 1004 instead of the distance information calculation unit 106 in the information processing device 601 of FIG.

[0070] The distance information calculation unit 1004 is connected to a distance measurement camera 1001 that serves both as the imaging device 101 and the distance measurement device 103. The distance information calculation unit 1004 calculates distance information using an image captured by the distance measurement camera 1001 as distance measurement information.

[0071] The subject extraction unit 107 receives an image input from the distance measurement camera 1001, determines the subject based on predetermined conditions, and acquires area information of the subject.

[0072] FIG. 11 is an example of a flowchart showing the processing up to when the information processing device 1003 outputs subject extraction distance information from a captured image.

[0073] 11 is obtained by deleting steps S300 and S301 from FIG. 7 and adding step S1101.

[0074] In step S302, the CPU 201 receives an image captured by the distance measurement camera 1001 via the input I / F 204 and temporarily stores the received image in the RAM 202 or the like. Thereafter, the processes of steps S1001 and S303 are performed. After step S1101, the process of step S501 is performed.

[0075] In step S303, the object extraction unit 107 extracts the object area from the image captured by the distance measurement camera 1001, and generates object area information.

[0076] In step S1101, the distance information calculation unit 1004 calculates distance information using an image captured by the distance measurement camera 1001. The image captured by the distance measurement camera 1001 is an example of information about the distance of an image captured by the imaging unit 1002.

[0077] For example, the distance measurement camera 1001 is a stereo camera. The distance information calculation unit 1004 calculates distance information by block matching using the principle of triangulation.

[0078] Also, for example, the image capturing unit 1002 is configured with pixels each having a plurality of divided photoelectric conversion units. The distance information calculation unit 1004 calculates distance information by block matching that takes into account the image plane phase difference.

[0079] As described above, according to this embodiment, the information processing device 1003 calculates distance information and extracts the subject area using the image acquired by the imaging unit 1002, even when using the ranging camera 1001. This allows the information processing device 1003 to cancel out erroneous calculation of distance information due to block matching and output distance information with high accuracy of the contour of the subject area.

[0080] (Fifth embodiment) The fifth embodiment will be described with reference to the drawings. Hereinafter, the same components as those in the above-described embodiments will be designated by the same reference numerals, and the description thereof will be omitted.

[0081] Generally, in image block matching, corresponding portions are estimated by comparing the features of blocks. However, depending on the imaging conditions and the state of the subject, there is a problem that the subject may be matched with an object other than the subject or with the background, making it impossible to correctly calculate distance information for the subject portion. The fifth embodiment has a configuration that addresses this problem.

[0082] 12 is a block diagram showing an example of the functional configuration of an image capturing system 1200 according to the fifth embodiment. The image capturing system 1200 includes a distance measuring camera 1001 and an information processing device 1201.

[0083] The information processing device 1201 is configured by providing an object extraction unit 1202 and a distance information calculation unit 1203 instead of the object extraction unit 107 and the distance information calculation unit 1004 of the information processing device 1003 in FIG.

[0084] The object extraction unit 1202 extracts an object region from the image captured by the ranging camera 1001 and generates object region information. In addition, the object extraction unit 1202 uses the object region information to extract an image of the object region from the image captured by the ranging camera 1001 and outputs it as an extracted object image. The object extraction unit 1202 outputs the object region information to the region information holding unit 108 and outputs the extracted object image to the distance information calculation unit 1203.

[0085] The distance information calculation unit 1203 calculates distance information using the extracted object image output by the object extraction unit 1202 as distance measurement information, and outputs the distance information to the distance information enlargement / reduction unit 402 .

[0086] FIG. 13 is an example of a flowchart showing the processing up to when the information processing device 1201 outputs subject extraction distance information from a captured image.

[0087] 13 is obtained by deleting steps S303 and S1101 and adding steps S1301 and S1302 from FIG. 11. After step S302, the process of step S1301 is performed. After step S1301, the processes of steps S1302 and S304 are performed. After step S1302, the process of step S501 is performed.

[0088] In step S1301, the object extraction unit 1202 extracts an object region from the image captured by the ranging camera 1001 using a predetermined method set in advance, and generates object region information. Then, the object extraction unit 1202 uses the object region information to extract an object extraction image of the object region from the image captured by the ranging camera 1101. For example, in the object extraction image, pixels in the object region output the pixel values ​​of the object image as they are, and pixels in regions other than the object region output a fixed value such as 0. The object extraction unit 1202 outputs the object region information to the region information storage unit 108, and outputs the object extraction image to the distance information calculation unit 1203.

[0089] In step S 1302 , the distance information calculation unit 1203 calculates distance information based on the extracted object image output by the object extraction unit 1202 , and outputs the calculated distance information to the distance information enlargement / reduction unit 402 .

[0090] As described above, according to this embodiment, the information processing device 1201 calculates distance information using an object extraction image in which pixel values ​​of areas other than the object are set to 0. This enables the information processing device 1201 to output distance information from an extracted object area while reducing the influence of matching between the object and objects other than the object.

[0091] (Sixth embodiment) The sixth embodiment will be described with reference to the drawings. Hereinafter, the same components as those in the above-described embodiments will be designated by the same reference numerals, and the description thereof will be omitted.

[0092] Generally, image block matching requires a larger amount of processing as the resolution and block size of the image to be processed increase. In the sixth embodiment, a configuration is adopted in which the processing amount is reduced to cope with this increase while maintaining the accuracy of the contours of the subject area in the depth information.

[0093] 14 is a block diagram showing an example of the functional configuration of an image capturing system 1400 according to the sixth embodiment. The image capturing system 1400 includes a distance measuring camera 1001 and an information processing device 1401.

[0094] 12 , an information processing device 1401 further includes an object extraction image reduction unit 1402. The object extraction image reduction unit 1402 reduces the resolution of the object extraction image output by the object extraction unit 1202 to a predetermined resolution, and outputs the reduced resolution to the distance information calculation unit 1203.

[0095] FIG. 15 is an example of a flowchart showing the processing up to when the information processing device 1401 outputs the subject extraction distance information from the captured image.

[0096] Figure 15 is obtained by adding step S1501 to Figure 13. After step S1301, the processes of steps S1501 and S304 are performed. After step S1501, the process of step S1302 is performed.

[0097] In step S1501, the object extraction image reducing unit 1402 reduces the resolution of the object extraction image output by the object extraction unit 1202 to a predetermined resolution that has been set in advance. For example, the object extraction image reducing unit 1402 determines the reduction rate according to the required processing performance, processing time, and calculation scale. For example, if the reduction rate is 1 / 2, the processing time required to calculate distance information is reduced to 1 / 2 or less.

[0098] Furthermore, distance information enlargement / reduction section 402 determines the enlargement rate of the distance information according to the reduction rate and the resolution of the subject region information. For example, if the reduction rate of subject extraction image reduction section 1402 is 1 / 2 and the resolution of the captured image is 1 / 2 of the resolution of the subject extraction distance information to be output, the enlargement rate of distance information enlargement / reduction section 402 is 4, because the reduction is doubled to cancel out the reduction and then doubled again to match the output resolution.

[0099] In step S 1302 , the distance information calculation unit 1203 calculates distance information based on the extracted object image reduced by the extracted object image reduction unit 1402 .

[0100] As described above, according to this embodiment, the information processing device 1401 maintains the resolution of the object region information while reducing the amount of processing required to calculate distance information by reducing the object extracted image using the object extracted image reducing unit 1402. This enables the information processing device 1401 to output distance information with high accuracy for the contours of the object region.

[0101] According to the first to sixth embodiments, the information processing device can obtain distance information while maintaining high accuracy of the contours for the area of ​​the subject of the distance information.

[0102] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0103] The present disclosure is not limited to the above-described embodiment, and various embodiments are possible without departing from the spirit of the present disclosure.

[0104] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an object extraction means for extracting an object area from an image and generating object area information; distance information calculation means for calculating distance information corresponding to the image based on information about the distance of the image; an area extraction means for extracting distance information of the area of ​​the subject from the distance information calculated by the distance information calculation means, using the area information of the subject generated by the object extraction means; An information processing device comprising: (Configuration 2) a distance information scaling means for scaling up or down the resolution of the distance information calculated by the distance information calculation means; The information processing device according to configuration 1, wherein the area extraction means uses the area information of the subject generated by the subject extraction means to extract distance information of the area of ​​the subject from the distance information enlarged or reduced by the distance information enlargement / reduction means. (Configuration 3) further comprising area information enlargement / reduction means for enlarging or reducing the resolution of the area information of the object generated by the object extraction means; 2. The information processing device according to configuration 1, wherein the area extraction means extracts distance information of the area of ​​the subject using area information of the subject enlarged or reduced by the area information enlargement / reduction means. (Configuration 4) further comprising area information enlargement / reduction means for enlarging or reducing the resolution of the area information of the object generated by the object extraction means; The information processing device according to configuration 2, wherein the area extraction means uses the area information of the subject enlarged or reduced by the area information enlargement / reduction means to extract distance information of the area of ​​the subject from the distance information enlarged or reduced by the distance information enlargement / reduction means. (Configuration 5) The information processing device according to any one of configurations 1 to 4, characterized in that the distance information calculation means uses the area information of the subject generated by the subject extraction means to select information of the area of ​​the subject from information relating to the distance of the image, and calculates the distance information. (Configuration 6) 6. The information processing device according to configuration 5, wherein the distance information calculation means calculates the distance information by projective transformation. (Configuration 7) the subject extraction means extracts a subject area from an image captured by an imaging unit and generates subject area information; 7. The information processing device according to any one of configurations 1 to 6, wherein the information about the distance of the image is distance measurement information acquired by a distance measurement unit. (Configuration 8) the subject extraction means extracts a subject area from an image captured by a distance measuring camera and generates subject area information; 7. The information processing device according to any one of configurations 1 to 6, wherein the information about the distance of the image is an image captured by the distance measuring camera. (Configuration 9) the distance measuring camera is a stereo camera, 9. The information processing device according to configuration 8, wherein the distance information calculation means calculates the distance information based on triangulation. (Configuration 10) the distance measuring camera is a camera in which each pixel has a plurality of photoelectric conversion units, 9. The information processing device according to configuration 8, wherein the distance information calculation means calculates the distance information based on an image plane phase difference. (Configuration 11) the subject extraction means extracts an image of the subject area from the image captured by the distance measuring camera using the subject area information; the information about the distance of the image is an image of the area of ​​the subject; 11. The information processing device according to any one of configurations 8 to 10, wherein the distance information calculation means calculates the distance information based on an image of the area of ​​the subject. (Configuration 12) The image processing device further includes an image reduction means for reducing the resolution of the image of the region of the object extracted by the object extraction means, 12. The information processing apparatus according to configuration 11, wherein the distance information calculation means calculates the distance information based on the image of the area of ​​the subject reduced by the image reduction means. (Method 1) an object extraction step of extracting an object area from the image and generating object area information; a distance information calculation step of calculating distance information corresponding to the image based on information about the distance of the image; an area extraction step of extracting distance information of the area of ​​the object from the distance information calculated in the distance information calculation step using the area information of the object generated in the object extraction step; 1. A processing method for an information processing device, comprising: (Program 1) 13. A program for causing a computer to function as the information processing device according to any one of the first to second aspects. [Explanation of symbols]

[0105] 101 imaging device, 102 imaging unit, 103 distance measuring device, 104 distance measuring unit, 105 information processing device, 106 distance information calculation unit, 107 object extraction unit, 108 region information storage unit, 109 region extraction unit

Claims

1. an object extraction means for extracting an object area from an image and generating object area information; distance information calculation means for calculating distance information corresponding to the image based on information about the distance of the image; an area extraction means for extracting distance information of the area of ​​the subject from the distance information calculated by the distance information calculation means, using the area information of the subject generated by the object extraction means; An information processing device comprising:

2. a distance information scaling means for scaling up or down the resolution of the distance information calculated by the distance information calculation means; 2. The information processing device according to claim 1, wherein the area extraction means uses the area information of the subject generated by the subject extraction means to extract distance information of the area of ​​the subject from the distance information enlarged or reduced by the distance information enlargement / reduction means.

3. further comprising area information enlargement / reduction means for enlarging or reducing the resolution of the area information of the object generated by the object extraction means; 2. The information processing apparatus according to claim 1, wherein the area extraction means extracts distance information of the area of ​​the subject by using the area information of the subject enlarged or reduced by the area information enlargement / reduction means.

4. further comprising area information enlargement / reduction means for enlarging or reducing the resolution of the area information of the object generated by the object extraction means; 3. The information processing device according to claim 2, wherein the area extraction means uses the area information of the subject enlarged or reduced by the area information enlargement / reduction means to extract distance information of the area of ​​the subject from the distance information enlarged or reduced by the distance information enlargement / reduction means.

5. 2. The information processing device according to claim 1, wherein the distance information calculation means uses the area information of the subject generated by the subject extraction means to select information of the area of ​​the subject from information relating to the distance of the image and calculates the distance information.

6. 6. The information processing apparatus according to claim 5, wherein the distance information calculation means calculates the distance information by projective transformation.

7. the subject extraction means extracts a subject area from an image captured by an imaging unit and generates subject area information; 2. The information processing apparatus according to claim 1, wherein the information about the distance of the image is distance measurement information acquired by a distance measurement unit.

8. the subject extraction means extracts a subject area from an image captured by a distance measuring camera and generates subject area information; 2. The information processing apparatus according to claim 1, wherein the information about the distance of the image is an image captured by the distance measuring camera.

9. the distance measuring camera is a stereo camera, 9. The information processing apparatus according to claim 8, wherein the distance information calculation means calculates the distance information based on triangulation.

10. the distance measuring camera is a camera in which each pixel has a plurality of photoelectric conversion units, 9. The information processing apparatus according to claim 8, wherein the distance information calculation means calculates the distance information based on an image plane phase difference.

11. the subject extraction means extracts an image of the subject area from the image captured by the distance measuring camera using the subject area information; the information about the distance of the image is an image of the area of ​​the subject; 9. The information processing apparatus according to claim 8, wherein the distance information calculation means calculates the distance information based on an image of the area of ​​the subject.

12. The image processing device further includes an image reduction means for reducing the resolution of the image of the region of the object extracted by the object extraction means, 12. The information processing apparatus according to claim 11, wherein the distance information calculation means calculates the distance information based on the image of the area of ​​the subject reduced by the image reduction means.

13. an object extraction step of extracting an object area from the image and generating object area information; a distance information calculation step of calculating distance information corresponding to the image based on information about the distance of the image; an area extraction step of extracting distance information of the area of ​​the object from the distance information calculated in the distance information calculation step using the area information of the object generated in the object extraction step; 1. A processing method for an information processing device, comprising:

14. A program for causing a computer to function as the information processing device according to any one of claims 1 to 12.

Images