Information processing device, information processing method, and program

Abstract

An information processing device according to the present invention includes a segmentation unit, a refinement unit, and an image generation unit. The segmentation unit performs segmentation on an input image. The refinement unit performs refinement so that the contour of each segment obtained by the segmentation matches the contour of the corresponding object in the input image. The image generation unit generates a segmentation map by integrating the segments after the refinement.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] Object recognition technology using segmentation is known. Segmentation refers to the process of identifying objects in an image and classifying them by color or other means. Hereinafter, an image in which objects are classified by color or other means will be referred to as a segmentation map.

[0003] Special table 2018-537742 publication

[0004] Using a segmentation map, it is possible to grasp the detailed shape of an object. For example, in a distance measurement sensor using pulsed light such as ToF (Time of Flight), the distance measurement points tend to be sparse. Therefore, it is conceivable to upsample the distance measurement data using the contour of the object obtained from the segmentation map as a clue. However, current segmentation maps do not provide contour information with sufficient accuracy to be used as guide information.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that can provide a segmentation map having highly accurate contour information.

[0006] According to the present disclosure, there is provided an information processing device having a segmentation unit that performs segmentation on an input image, a refinement unit that performs refinement so that the contours of each segment obtained by the segmentation match the contours of a corresponding object in the input image, and an image generation unit that integrates each refined segment to generate a segmentation map. Also, according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer, and a program that causes a computer to realize the information processing of the information processing device.

[0007] FIG. 1 is a diagram illustrating an example of the configuration of a depth estimation system using the segmentation method of the present disclosure; FIG. 2 is a diagram illustrating an example of a segmentation result using SAM; FIG. 3 is a diagram illustrating an example of generation of a segmentation map; FIG. 4 is a diagram illustrating re-learning using refined segment data; FIG. 5 is a diagram illustrating an example of a method for reducing the load of refinement processing; and FIG. 6 is a diagram illustrating an example of the hardware configuration of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Configuration of depth estimation system] [2. Example of generation of segmentation map] [3. Re-learning using refined segment data] [4. Reduction of load of refinement processing] [5. Example of hardware configuration] [6. Effects]

[0010] 1 is a diagram showing an example of the configuration of a depth estimation system 1 using the segmentation method of the present disclosure. The depth estimation system 1 includes an information processing device 10, an image sensor 20, and a distance measurement sensor 30.

[0011] The image sensor 20 captures an RGB image of the subject SB. The distance measurement sensor 30 acquires distance measurement data of the subject SB. The distance measurement data includes distances to each point on the subject SB as distance measurement information. For example, the distance measurement sensor 30 is a direct ToF distance measurement sensor. The distance measurement sensor 30 outputs spot-shaped pulsed light and calculates the distance to the subject SB based on the time of flight of the pulsed light.

[0012] The relative positional relationship between the distance measurement sensor 30 and the image sensor 20 is fixed. The distance measurement range of the distance measurement sensor 30 and the imaging range of the image sensor 20 are calibrated. The correspondence between the distance measurement points of the distance measurement sensor 30 and each pixel of the image sensor 20 is known. Spot-shaped pulsed light is generally sparse pulsed light. Therefore, the density of the distance measurement points is smaller than the density of the pixels of the image sensor 20.

[0013] The information processing device 10 acquires an RGB image and distance measurement data of a subject SB from an image sensor 20 and a distance measurement sensor 30, respectively. The information processing device 10 upsamples the sparse distance measurement data based on the high-resolution RGB image to acquire high-resolution depth information. For example, the information processing device 10 includes a segmentation unit 11, a refinement unit 12, an image generation unit 13, and a depth estimation unit 14.

[0014] The segmentation unit 11 converts the RGB image of the subject SB from the image sensor 20 into an input image IM. Ａ The segmentation unit 11 acquires the input image IM Ａ Segmentation is performed on the input image IM Ａ A plurality of segments SG corresponding to each object in the image are acquired. As a segmentation method, a known method such as semantic segmentation can be adopted. In the present disclosure, for example, the segmentation method used is the Segment Anything Model (SAM) described in the following document 1.

[0015] [Reference 1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything, arXiv preprint arXiv:2304.02643, 2023

[0016] 2 is a diagram showing an example of a segmentation result using SAM. In the example of FIG. 2, an input image IM Ａ is a bust shot of the person who is the subject SB. The segmentation unit 11 uses SAM to segment the input image IM Ａ23 segments SG are generated from the image. The segments SG are the parts displayed in white. The person's face, hair, torso, arms, headphones, etc. are extracted as separate segments SG. The background of the person (walls, metal fittings, etc.) is also extracted as separate segments SG.

[0017] The refinement unit 12 refines the contours of each segment SG obtained by segmentation in the input image IM. Ａ In FIG. 1, the segment SG before refinement is denoted by the symbol "SG Ａ " and the refined segment SG is denoted by "SG Ｂ " is indicated.

[0018] The refinement can be performed using a cost volume, which is the input image IM Ａ represents the probability that a pixel in the input image IM belongs to the segment SG. Ａ The refinement unit 12 calculates a cost volume for each pixel of the segment SG. Based on the cost volume, the refinement unit 12 obtains a label indicating whether the pixel belongs to the segment SG.

[0019] For example, the refinement unit 12 prepares cost volumes of two types of layers for each pixel to be processed based on the following formula (1): Ａ The pixels classified into the segment SG Ａ The cost volume of the second layer indicates the probability that the segment belongs to the segment SG Ａ The pixels classified into the segment SG Ａ indicates the probability that the

[0020]

[0021] In formula (1), "C" represents the cost volume. "k" represents the segment SG Ａ "p" indicates the pixel to be processed. "t" indicates the layer of the cost volume. "t=0" indicates the first layer. "t=1" indicates the second layer.

[0022] Next, the refinement unit 12 sets an initial value (initial cost volume) of the cost volume for each pixel "p" based on the result of the segmentation performed by the segmentation unit 11. The refinement unit 12 adds the initial value of the cost volume (initial cost volume) to the initial cost volume generated based on the segmentation result. Ａ The cost volume (corrected cost volume) is calculated by applying an edge-preserving filter guided by the following. As the edge-preserving filter, a guided filter described in the following document 2 can be used.

[0023] [Reference 2] K. He, J. Sun, and X. Tang, “Guided Image Filtering”, European Conference on Computer Vision (ECCV), 2010

[0024] For example, the refinement unit 12 calculates the segment SG based on the following formula (2): Ａ The refinement unit 12 assigns "0" as the initial cost volume of the first layer to the pixels classified as "0" and "c" as the initial cost volume of the second layer to the pixels classified as "0" and "c" in the segment SG Ａ For pixels not classified as "0", "0" is assigned as the initial cost volume of the second layer, and "c" is assigned as the initial cost volume of the first layer.

[0025]

[0026] In equation (2), "p=p_segment" means that the pixel "p" to be processed is located in the segment SG Ａ "Otherwise" indicates that the pixel "p" to be processed is classified as a pixel in the segment SG Ａ "c" indicates that the pixel is not classified as ". "c" is a default setting value given to pixels that meet certain conditions. The setting value "c" can be set arbitrarily by the system developer.

[0027] Next, the refinement unit 12 calculates the input image IM based on the following equation (3): ＡCost volume filtering is performed using an edge-preserving filter "W" that uses the above as a guide. The edge-preserving filter "W" calculates the pixel value of pixel "p" by referring to the pixel values ​​of pixels in a predetermined block. In equation (3), "q" indicates the pixel position in the block that is referenced by the edge-preserving filter. "I" is the input image IM that serves as a guide. Ａ "C'" indicates the cost volume after filtering (corrected cost volume).

[0028]

[0029] The refinement unit 12 calculates the segment SG for each pixel “p” based on the following equation (4): Ａ For example, if the cost of the first layer is lower than the cost of the second layer, the refinement unit 12 assigns a label "f" to the pixel "p" to be processed, indicating whether it belongs to the segment SG. Ａ If the cost of the first layer is higher than the cost of the second layer, the refinement unit 12 assigns a label "f" to the pixel "p" to be processed, which indicates that the pixel "p" belongs to the segment SG. Ａ The label "f" is assigned to the object "not belonging to the group".

[0030]

[0031] The refinement unit 12 Ａ The set of pixels labeled "f" as "belonging to" is the refined segment SG Ｂ The refinement unit 12 refines all the segments SG Ａ The above processing is performed on

[0032] The image generating unit 13 generates the refined segments SG Ｂ The segmentation map IM is created by integrating Ｂ For example, the image generating unit 13 generates each refined segment SG Ｂ The image generating unit 13 assigns different colors to the segments SG Ｂ Each segment SG in descending order of area Ｂsegmentation map IM Ｂ This pastes all the segments SG Ｂ Segmentation map IM Ｂ is obtained.

[0033] The depth estimation unit 14 estimates the input image IM Ａ A low-resolution depth image DP including distance measurement information of the subject SB Ａ segmentation map IM Ｂ For example, the depth estimation unit 14 performs upsampling based on the segmentation map IM Ｂ The depth estimation unit 14 upsamples the distance measurement data using the extracted object contour as a clue. As a result, the depth estimation unit 14 obtains a low-resolution depth image DP Ａ High-resolution depth image DP with higher resolution than Ｂ Get.

[0034] As a result, high-density depth information can be obtained from sparse ranging data with few ranging points. This depth information can be applied to next-generation 3D entertainment content such as stereoscopic vision, AR / VR, and the Metaverse. Because excessive computation is not required, real-time processing may be possible in the future. If real-time processing becomes possible, this method can be applied to in-vehicle sensors. Conventional in-vehicle stereo cameras could only obtain depth information for the edge of an object, but this method makes it possible to obtain dense depth information in real time, which may contribute to improving the accuracy of recognizing objects outside the vehicle.

[0035] [2. Example of Segmentation Map Generation] FIG. 3 shows a segmentation map IM Ｂ 3 is a diagram illustrating an example of generating the segmentation map IM Ｂ is the input image IM shown in FIG. Ａ The bottom left of Figure 3 shows the human head segment SG without refinement. Ａ The bottom right diagram in FIG. 3 shows the segment SG of the human head after refinement. Ｂ Shows.

[0036] As shown in the example at the bottom left of FIG. 3, without refinement, the head segment SG Ａ is identified by a rough outline that ignores the bounciness of the hair. A guide image (segmentation map IM Ｂ ), the contour of the segment SG must match the contour of the subject SB in the RGB image. Ａ If the contours are inaccurate, accurate depth information cannot be obtained when the distance measurement data is upsampled using the contour information as a clue.

[0037] On the other hand, as shown in the example at the bottom right of Figure 3, when refinement is performed, the shape of each individual hair is reproduced almost faithfully. Ｂ The contour of the segment SG matches well with the contour of the subject SB in the RGB image. Ｂ Since the contours are accurate, accurate depth information can be obtained when the distance measurement data is upsampled using the contour information as a clue.

[0038] 3. Re-learning using refined segment data FIG. 4 is a diagram illustrating re-learning using refined segment data.

[0039] As described above, conventional segmentation techniques cannot obtain segments SG with sufficient accuracy. Therefore, it is conceivable to retrain artificial intelligence (AI) for segmentation based on information on refined segments SG. For example, the segmentation unit 11 may Ａ is the student image, and the refined segment SG Ｂ The DNN coefficients retrained using the training image are obtained as the DNN coefficients of the AI ​​for segmentation.

[0040] By using the AI ​​after re-learning, the segmentation unit 11 can output more accurate segmentation results. The refinement unit 12 can output even more accurate data of the segments SG using the acquired segmentation results. Because the accuracy of segmentation is improved by re-learning, it is possible to omit the refinement process by the refinement unit 12 and reduce the amount of calculation.

[0041] 4. Reducing the Load of Refinement Processing FIG. 5 is a diagram illustrating an example of a method for reducing the load of refinement processing.

[0042] When high-resolution segmentation of an 8K RGB image is required, it is necessary to create and process a DNN network that supports 8K, but this is difficult to implement in terms of processing time and memory consumption. Therefore, it is conceivable to reduce the load of refinement processing by down-converting and up-converting the data.

[0043] For example, the information processing device 10 includes a downconverter 15 and an upconverter 16. The downconverter 15 converts a high-resolution image IM Ｈ is reduced in resolution to the input image IM Ａ The segmentation unit 11 obtains the input image IM Ａ Segmentation is performed on the low-resolution segment SG Ｌ The up-converter 16 generates the input image IM Ａ The low-resolution segment SG obtained from Ｌ is upsampled to obtain a high-resolution segment SG Ａ Get.

[0044] For example, the downconverter 15 receives an 8K high-resolution RGB image (high-resolution image IM Ｈ The downconverter 15 converts the high-resolution RGB image into a low-resolution RGB image such as an HD image (input image IM ＡThe segmentation unit 11 obtains a low-resolution segmentation result using a DNN network that supports low resolutions such as HD.

[0045] The upconverter 16 linearly upsamples the segmentation result to the original high resolution (8K). The refinement unit 12 refines the upsampled segmentation result. This may enable high-resolution segmentation results, such as 8K, to be obtained in a short processing time and with a small memory capacity.

[0046] 5. Example of Hardware Configuration FIG. 6 is a diagram showing an example of the hardware configuration of the information processing device 10. As shown in FIG.

[0047] The information processing of the information processing device 10 is realized by, for example, a computer 1000. The computer 1000 has a central processing unit (CPU) 1100, a random access memory (RAM) 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0048] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0049] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0050] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0051] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0052] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0053] For example, when the computer 1000 functions as the information processing device 10 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0054] 6. Effects The information processing device 10 includes a segmentation unit 11, a refinement unit 12, and an image generation unit 13. The segmentation unit 11 generates a segment of an input image IM Ａ The refinement unit 12 refines each segment SG obtained by the segmentation. Ａ The contour of the input image IM Ａ The image generator 13 refines each segment SG after refinement so that it matches the contour of the corresponding object in the segment SG. Ｂ The segmentation map IM is created by integrating Ｂ In the information processing method of the present disclosure, the processing of the information processing device 10 is executed by the computer 1000. The program of the present disclosure causes the computer 1000 to realize the processing of the information processing device 10.

[0055] According to this configuration, a segmentation map IM having highly accurate contour information is generated. Ｂ is provided.

[0056] The refinement unit 12 refines the input image IM Ａ The cost volume is calculated for each pixel in the segment SG. Ａ The refinement unit 12 determines the probability that a pixel belongs to the segment SG based on the cost volume. Ａ Get a label indicating whether it belongs to the group or not.

[0057] This configuration improves the accuracy of estimating the contour.

[0058] The refinement unit 12 adds the input image IM to the initial cost volume generated based on the segmentation result. Ａ An edge-preserving filter guided by is applied to calculate the cost volume.

[0059] This configuration improves the accuracy of estimating the contour.

[0060] The information processing device 10 includes a depth estimation unit 14. The depth estimation unit 14 estimates an input image IM Ａ A low-resolution depth image DP including distance measurement information of the subject SB Ａ segmentation map IM Ｂ and upsampling the low-resolution depth image DP Ａ High-resolution depth image DP with higher resolution than Ｂ Get.

[0061] This configuration improves the accuracy of depth estimation.

[0062] The segmentation unit 11 segments an input image IM Ａ is the student image, and the refined segment SG Ｂ The DNN coefficients retrained using the training image are obtained as the DNN coefficients of the AI ​​for segmentation.

[0063] This configuration increases the accuracy of the segmentation.

[0064] The information processing device 10 includes a downconverter 15 and an upconverter 16. The downconverter 15 converts a high-resolution image IM Ｈ is reduced in resolution to the input image IM Ａ The up-converter 16 obtains the input image IM Ａ The low-resolution segment SG obtained from Ｌ is upsampled to obtain a high-resolution segment SG Ａ Get.

[0065] According to this configuration, highly accurate segmentation results can be obtained with a small amount of calculation.

[0066] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0067] [Additional Notes] The present technology can also be configured as follows. (1) An information processing device having: a segmentation unit that performs segmentation on an input image; a refinement unit that performs refinement so that the contours of each segment obtained by the segmentation match the contours of a corresponding object in the input image; and an image generation unit that integrates each refined segment to generate a segmentation map. (2) The information processing device described in (1), wherein the refinement unit calculates, for each pixel of the input image, a cost volume that represents the probability that the pixel belongs to the segment, and obtains a label indicating whether the pixel belongs to the segment based on the cost volume. (3) The information processing device described in (2), wherein the refinement unit calculates the cost volume by applying an edge-preserving filter guided by the input image to an initial cost volume generated based on the segmentation result. (4) The information processing device according to any one of (1) to (3), further comprising: a depth estimation unit that upsamples a low-resolution depth image including ranging information of the subject of the input image based on the segmentation map, and acquires a high-resolution depth image having a higher resolution than the low-resolution depth image. (5) The information processing device according to any one of (1) to (4), further comprising: a downconverter that reduces the resolution of a high-resolution image to acquire the input image; and an upconverter that upsamples the low-resolution segment obtained from the input image to acquire a high-resolution segment to be refined.(7) An information processing method executed by a computer, comprising: segmenting an input image, refining the contours of each segment obtained by the segmentation so that the contours of each segment match the contours of the corresponding object in the input image, and integrating the refined segments to generate a segmentation map. (8) A program that causes a computer to perform the following steps: segmenting an input image, refining the contours of each segment obtained by the segmentation so that the contours of each segment match the contours of the corresponding object in the input image, and integrating the refined segments to generate a segmentation map.

[0068] REFERENCE SIGNS LIST 10 Information processing device 11 Segmentation unit 12 Refinement unit 13 Image generation unit 14 Depth estimation unit 15 Downconverter 16 Upconverter DP Ａ Low-resolution depth image DP Ｂ High resolution depth image IM Ａ Input image IM Ｂ Segmentation Map IM Ｈ High resolution image SB Subject SG, SG Ａ , S.G. Ｂ , S.G. Ｌ segment

Claims

1. An information processing device having: a segmentation unit that performs segmentation on an input image; a refinement unit that performs refinement so that the contours of each segment obtained by the segmentation match the contours of the corresponding object in the input image; and an image generation unit that integrates each refined segment to generate a segmentation map.

2. The information processing device according to claim 1, wherein the refinement unit calculates, for each pixel of the input image, a cost volume representing the probability that the pixel belongs to the segment, and obtains a label indicating whether the pixel belongs to the segment based on the cost volume.

3. The information processing device according to claim 2, wherein the refinement unit calculates the cost volume by applying an edge-preserving filter using the input image as a guide to an initial cost volume generated based on the segmentation result.

4. The information processing device according to claim 1, further comprising a depth estimation unit that upsamples a low-resolution depth image including ranging information of the subject of the input image based on the segmentation map, and obtains a high-resolution depth image having a higher resolution than the low-resolution depth image.

5. The information processing device according to claim 1, wherein the segmentation unit acquires DNN coefficients re-trained using the input image as a student image and the refined segment as a teacher image as DNN coefficients of the AI ​​for segmentation.

6. An information processing device as described in claim 1, comprising: a downconverter that reduces the resolution of a high-resolution image to obtain the input image; and an upconverter that upsamples a low-resolution segment obtained from the input image to obtain a high-resolution segment that is the subject of the refinement.

7. An information processing method executed by a computer, comprising: segmenting an input image; refining the contours of each segment obtained by the segmentation so that the contours of each segment match the contours of a corresponding object in the input image; and integrating the refined segments to generate a segmentation map.

8. A program that causes a computer to perform the following steps: segment an input image; refine the contours of each segment obtained by the segmentation so that they match the contours of the corresponding object in the input image; and integrate the refined segments to generate a segmentation map.

Images