Information processing apparatus, information processing method, information processing program

Abstract

Provided are an information processing device, an information processing method, and an information processing program that can extract a region in an image or a depth map of a subject, even when colors are the same or similar. The information processing device comprises: a difference map generation unit that generates a difference map from a first depth map of a subject and a second depth map of the subject, said depth maps having been acquired using a ToF sensor; and a region extraction unit that extracts a region in the first depth map on the basis of the difference map.

Description

Technical Field

[0001] This technology relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Various techniques for extracting a specific area from an image or depth map of an object have been proposed. For example, by using a so-called RGB camera capable of shooting an RGB (Red, Green, Blue) image, an area with a different color can be extracted.

[0003] In addition, a technique called ToF (Time Of Flight) can also be used for area extraction. There is a technique called ToF that acquires distance information (depth information) by measuring the reflection time of pulsed light irradiated on an object for each pixel.

[0004] Regarding ToF, in order to acquire distance information (depth information) more accurately, a technique has been proposed for generating accurate distance information of an object using stereo distance calculated according to a stereo method using two images and ToF (Patent Document 1).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, even when using an RGB camera, there is a problem that areas that are different but have the same color or similar colors cannot be accurately extracted from an image or depth map of an object. Also, even if the accuracy of distance information generation by ToF is improved, there is the same problem.

[0007] The present technology has been made in view of such points, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of extracting regions having the same color or similar colors in an image or a depth map of an object.

Means for Solving the Problems

[0008] In order to solve the above-described problems, a first technology includes a difference map generation unit that generates a difference map from a first depth map of an object acquired by a ToF sensor and a second depth map of the object, and based on the difference map, Extract a region on a first depth map having a depth difference corresponding to a peak in the histogram of the difference map an information processing apparatus including a region extraction unit.

[0009] Further, a second technology The difference map generation unit generates a difference map from a first depth map of an object acquired by a ToF sensor and a second depth map of the object, and The region extraction unit based on the difference map, Extract a region on a first depth map having a depth difference corresponding to a peak in the histogram of the difference map is an information processing method.

[0010] Furthermore, a third technology generates a difference map from a first depth map of an object acquired by a ToF sensor and a second depth map of the object, and based on the difference map, Extract a region on a first depth map having a depth difference corresponding to a peak in the histogram of the difference map is an information processing program for causing a computer to execute an information processing method.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

[0012] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order. <1. First Embodiment> [1-1. Configuration of Information Processing System 10] [1-2. Configuration of Information Processing Apparatus 100] [1-3. Processing by Information Processing Apparatus 100] <2. Second Embodiment> [2-1. Configuration of Information Processing Apparatus 200] [2-1. Processing by Information Processing Apparatus 200] <3. Third Embodiment> [3-1. Configuration of Information Processing System 30] [3-2. Configuration of Information Processing Apparatus 300] [3-3. Processing by Information Processing Apparatus 300] <4. Modification Example>

[0013] <1. First Embodiment> [1-1. Configuration of Information Processing System 10] With reference to FIG. 1, the configuration of the information processing system 10 in the first embodiment of the present technology will be described. The information processing system 10 includes an information processing apparatus 100, a ToF sensor 500, and a distance measurement sensor 600.

[0014] The information processing apparatus 100 performs region extraction processing based on the depth map of the object generated by the ToF sensor 500 and the image or depth map of the object generated by the distance measurement sensor 600.

[0015] The object is an object to be subjected to region extraction processing by the information processing apparatus 100, and includes two or more distinguishable objects, or an object that is one as an object but has a plurality of regions on its surface that are composed of different materials, materials, substances (hereinafter referred to as materials, etc.).

[0016] Examples of two or more distinguishable objects include "a hand (skin) holding a spoon" consisting of a spoon and a hand (skin), and "food on a plate" consisting of a plate and food. Examples of an object that is one as an object but has a plurality of regions on its surface that are composed of different materials, etc. include a spoon composed of a tip part (bowl) made of metal and a handle made of wood, and a cardboard box and characters, figures, decorations, etc. attached to the surface of the box and made of a material different from the material of the cardboard box.

[0017] Note that even if the names of the objects are the same, objects with different materials, etc. are regarded as "two or more distinguishable objects". Therefore, for example, a wooden spoon and a metal spoon are the same spoon but are two or more distinguishable objects.

[0018] When two or more regions are extracted by the information processing apparatus 100, there may be cases where there are two or more objects and each object is extracted as a region, or there may be cases where there is one object but there are two or more distinguishable regions on the surface of the object. Furthermore, there may be cases where there are two or more objects and there are two or more distinguishable regions on the surface of the object.

[0019] The ToF sensor 500 is a sensor that utilizes ToF to acquire distance information (first depth map) to an object that is the target of processing by the information processing apparatus 100. There are two types of ToF, iToF (indirect Time of Flight) and dToF (direct Time of Flight), and the ToF sensor 500 may be of either type. iToF is a method of obtaining depth from the phase difference of a periodic signal. dToF is a method of obtaining depth by measuring the time when a pulsed laser emitted from a light source is sent and the time when it returns, and calculating the difference therebetween.

[0020] In ToF, a phenomenon called multipath occurs. ToF measures the time it takes for light to be reflected by an object and return and be received, but depending on the object, the light does not totally reflect on the object surface, and part of it enters the object, reflects repeatedly inside, and then the light returns to the ToF sensor 500. Therefore, depending on the object, the time it takes for the ToF sensor 500 to receive the light becomes longer, and the distance is detected as being farther than the actual distance. The reflectivity of this object varies depending on the material of the object and the like.

[0021] Before performing processing by the information processing apparatus 100, the user needs to generate a first depth map for the object using the ToF sensor 500.

[0022] The distance measurement sensor 600 is an RGB stereo camera composed of a first RGB camera 610 and a second RGB camera 620. The first RGB camera 610 is a camera capable of capturing an RGB (Red, Green, Blue) image, corresponding to the first imaging device in the claims. The first RGB image obtained by capturing with the first RGB camera 610 corresponds to the first image in the claims.

[0023] The second RGB camera 620 is a camera capable of capturing an RGB (Red, Green, Blue) image, corresponding to the second imaging device in the claims. The second RGB image obtained by capturing with the second RGB camera 620 corresponds to the second image in the claims. Unless it is necessary to separately describe the first RGB camera 610 and the second RGB camera 620 in the following description, the description will be made as the distance measurement sensor 600.

[0024] Before the user performs the processing by the information processing apparatus 100, it is necessary to acquire the first RGB image and the second RGB image of the object using the distance measurement sensor 600.

[0025] [1-2. Configuration of Information Processing Apparatus 100] Next, the configuration of the information processing apparatus 100 will be described. The information processing apparatus 100 includes a control unit 150, a storage unit 160, an interface 170, and a display unit 180.

[0026] The control unit 150 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The CPU executes various processes according to the programs stored in the ROM and issues commands to control the entire information processing apparatus 100 and each part.

[0027] The memory unit 160 is a large-capacity storage medium such as a hard disk or a flash memory, for example. The memory unit 160 stores programs, calibration data, tables, etc. used in the processing in the information processing apparatus 100.

[0028] The interface 170 is an interface for communicating with the ToF sensor 500 and the distance measurement sensor 600. The interface 170 may include a wired or wireless communication interface. More specifically, the wired or wireless communication interface may include cellular communication such as 3GPP, Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), HDMI (registered trademark) (High-Definition Multimedia Interface), USB (Universal Serial Bus), etc. When the information processing apparatus 100 and the ToF sensor 500 and the distance measurement sensor 600 are hardware-connected, the interface 170 may include connection terminals between the devices, buses inside the devices, etc. When the information processing apparatus 100 is realized by being distributed over a plurality of devices, the interface 170 may include different types of interfaces for each device. For example, the interface 170 may include both a communication interface and an interface inside the device.

[0029] The display unit 180 is a display device constituted by, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, etc.

[0030] As shown in FIG. 3, the information processing apparatus 100 is configured to include functional blocks such as a depth map generation unit 101, a difference map generation unit 102, a region extraction unit 103, and an image processing unit 104. Each of these units is a function realized by a control unit 150. Also, the transmission and reception of data and information between each unit and the ToF sensor 500 and the distance measurement sensor 600 are performed using an interface 170.

[0031] The depth map generation unit 101 generates a second depth map by performing triangulation such as pattern matching using a first RGB image captured by a first RGB camera 610 constituting the distance measurement sensor 600 and a second RGB image captured by a second RGB camera 620.

[0032] The difference map generation unit 102 generates a difference map using the first depth map generated by the ToF sensor 500 and the second depth map generated by the depth map generation unit 101.

[0033] The region extraction unit 103 extracts a region on the first depth map using the first depth map and the difference map.

[0034] The image processing unit 104 generates a display image for showing the extraction result by the region extraction unit 103.

[0035] The information processing apparatus 100 is configured as described above. Note that the functional blocks in the information processing apparatus 100 may be realized by the execution of a program, and a personal computer, a tablet terminal, a smartphone, a server device, etc. may be made to function as the information processing apparatus 100 by the execution of that program. The program may be installed in advance in a device such as a personal computer, or may be distributed by download, a storage medium, etc., and installed by the user himself / herself.

[0036] [1-3. Processing by Information Processing Apparatus 100] Next, the processing by the information processing apparatus 100 will be described with reference to the flowchart of FIG. 4.

[0037] First, in step S101, the information processing apparatus 100 acquires a first depth map of the object from the ToF sensor 500.

[0038] Also, in step S102, the information processing apparatus 100 acquires a first RGB image and a second RGB image from the distance measurement sensor 600. Note that steps S101 and S102 do not necessarily have to be performed in this order, and the reverse order or almost simultaneously is also acceptable.

[0039] Next, in step S103, the depth map generation unit 101 generates a second depth map from the first RGB image and the second RGB image by performing triangulation such as pattern matching using the first calibration data.

[0040] The first calibration data is data indicating the relative positional relationship between the first RGB camera 610 and the second RGB camera 620. Since the viewpoints of the first RGB camera 610 and the second RGB camera 620 are different, it is necessary to match the viewpoints using the first calibration data in order to generate the second depth map.

[0041] Also, in order to perform pattern matching, it is necessary to consider the distortion (distortion) of the first RGB camera 610 and the second RGB camera 620. It is advisable to include the distortion correction data in the first calibration data as well.

[0042] Note that the first calibration data may be stored in the storage unit 160 in advance, or may be held by the information processing apparatus 100 in advance. Also, the first calibration data may be stored in an external server or the like, and the information processing apparatus 100 may access and read the data from the server via the interface 170.

[0043] Next, in step S104, the difference map generation unit 102 generates a difference map based on the first depth map and the second depth map.

[0044] In generating the difference map, first, the second depth map is projected onto the first depth map using the second calibration data to generate a map for generating the difference map.

[0045] The second calibration data is data indicating the relative positional relationship between the ToF sensor 500 and the distance measurement sensor 600. Since the ToF sensor 500 and the distance measurement sensor 600 are separate and have different viewpoints, the second calibration data is used to align the viewpoints and project the second depth map onto the first depth map. Then, the difference map is generated by calculating the difference between the map for generating the difference map for each pixel constituting the first depth map.

[0046] Note that the second calibration data may be stored in the storage unit 160 in advance, or may be held in the information processing apparatus 100 in advance. Also, the second calibration data may be stored in an external server or the like, and the information processing apparatus 100 may access the server via the interface 170 to read it out.

[0047] Next, in step S105, the region extraction unit 103 generates a histogram of the difference map and detects peaks from the histogram. The peaks in the histogram are detected as shown in, for example, FIG. 5A. Peaks are detected for each region. For example, when there are two objects, two peaks are detected. Even if there is one object, if there are two distinguishable regions on the surface of the object, two peaks are detected.

[0048] Next, in step S106, the region extraction unit 103 extracts regions by extracting all the pixels having the depth difference of the peaks detected from the histogram from among all the pixels constituting the first depth map.

[0049] Alternatively, as shown in FIG. 5B, the region extraction unit 103 extracts a region by extracting all pixels having a depth difference within a predetermined width centered on the peak detected from the histogram from the first depth map.

[0050] The predetermined width centered on the peak is a width set based on the depth variation due to the performance of the ToF sensor 500 and the ranging sensor 600, the depth variation for each object, etc., and is set in advance in the information processing apparatus 100. By extracting all pixels having a depth difference within the predetermined width, it is possible to appropriately extract a region according to the depth variation due to the performance of the ToF sensor 500 and the ranging sensor 600, the depth variation for each object, etc.

[0051] Whether to perform region extraction based on the peak or based on a predetermined width centered on the peak may be selectable by the user, or the information processing apparatus 100 may set the extraction method based on the performance of the ToF sensor 500 and the ranging sensor 600, etc.

[0052] The depth that can be obtained by the ToF sensor 500 deviates from the true value depending on the material of the object and the like due to the multi-path described above. Therefore, by comparing the depth obtained by the ToF sensor 500 with the depth obtained by another method (the ranging sensor 600 in this embodiment) for the same object and grasping the deviation, it is possible to perform region extraction for each material and the like.

[0053] Next, in step S107, the image processing unit 104 generates a display image in which the region extracted from the first depth map is drawn. The generation of the display image is performed by drawing the extracted region on the first depth map. Note that the display image may be generated by drawing on any of the second depth map, the first RGB image, and the second RGB image. By displaying the generated display image on the display unit 180, the user can confirm the extracted region.

[0054] For example, assume that the object is "the hand of a person holding a spoon", the spoon is wooden, and the color of the spoon is similar to that of the person's hand. Here, "the colors are similar" means that the values of hue, saturation, and lightness, which are the elements constituting the color of the spoon, and the values of hue, saturation, and lightness, which are the elements constituting the color of the hand (skin), are within a predetermined approximation range, or that the RGB values representing the color of the spoon and the RGB values representing the color of the hand (skin) are within a predetermined approximation range.

[0055] The grayscale image of the object "spoon and hand" is shown in FIG. 6A, and the first depth map of the object is shown in FIG. 6B. Assume that the region extraction unit 103 extracts the spoon and the hand as different regions. In this case, as shown in FIG. 6C, the display image is an image representing that the spoon and the hand extracted as regions are drawn in different colors, indicating that they are extracted as different regions.

[0056] By extracting regions in this way, there may be cases where two or more objects are extracted, or cases where there is one object but multiple regions composed of different materials, etc. on the surface of the object are extracted. Furthermore, there may be cases where two or more objects are extracted and two or more distinguishable regions are extracted on the surface of the object.

[0057] The processing in the first embodiment is performed as described above. According to the first embodiment, since regions are extracted based on the depth difference, regions that cannot be extracted only by the RGB image because the colors are the same or similar can be extracted.

[0058] <2. Second Embodiment> [2-1. Configuration of Information Processing Apparatus 200] Next, a second embodiment of the present technology will be described. The configuration of the information processing system 10 is the same as that of the first embodiment. As shown in FIG. 7, the information processing apparatus 200 in the second embodiment is configured to include functional blocks such as a depth map generation unit 101, a difference map generation unit 102, a region extraction unit 103, a type identification unit 201, and an image processing unit 104. Since the depth map generation unit 101, the difference map generation unit 102, the region extraction unit 103, and the image processing unit 104 are the same as those in the first embodiment, their descriptions will be omitted.

[0059] The type identification unit 201 refers to a table in which the depth difference peaks and the types of objects are associated in advance in the depth map acquired by the ToF sensor 500 and the depth map generated from the image acquired by the distance measurement sensor 600, as shown in FIG. 8, to identify the type of material or the like that constitutes the object. Note that the table may be possessed by the type identification unit 201, or may be stored in advance in the storage unit 160, and the type identification unit 201 may read the table in the storage unit 160. Alternatively, the table may be stored in an external server or the like, and the information processing apparatus 200 may access the server via the interface 170 to read the table. The table shown in FIG. 8 is presented for convenience of explanation, and it does not mean that the depth differences of wood, food, skin, and cloth described in FIG. 8 are the values shown in FIG. 8.

[0060] A type is a classification of things that have common properties, forms, etc. according to a certain criterion, and each is grouped together. There are various types, such as metal, plant, food, living thing, cloth, synthetic resin, mineral, paper, and so on. Note that the types may be classified in more detail. For example, food may be classified into vegetables and meat in more detail, or metal may be further classified into iron, copper, gold, etc.

[0061] [2-1. Processing by Information Processing Apparatus 200] Next, the processing of the information processing apparatus 200 in the second embodiment will be described with reference to the flowchart of FIG. 9.

[0062] Since the processes from step S101 to step S106 are the same as those in the first embodiment, the description thereof is omitted.

[0063] When the region extraction unit 103 extracts a region on the first depth map in step S106, next, in step S201, the type identification unit 201 identifies the type of the region extracted by the region extraction unit 103.

[0064] For example, when a table is such that the depth difference and the types such as the material constituting an object are associated as shown in FIG. 8, and the histogram of the object generated by the region extraction unit 103 is as shown in FIG. 10. Then, by comparing the histogram of the table and the object, when the position of the peak is included in a certain type range in the table, it is identified as the type such as the material constituting the region.

[0065] In the table of FIG. 8 and the histogram of FIG. 10, since the two peaks of the histogram of FIG. 10 are respectively included in "wood" and "skin" in the table of FIG. 8, the types of the extracted regions can be identified as the wood region and the skin region.

[0066] Next, in step S202, the image processing unit 104 generates a display image in which the region extracted from the first depth map and the type of the region are drawn. The display image is generated by drawing the extracted region on the first depth map. Note that the display image may be generated by drawing on any one of the second depth map, the first RGB image, and the second RGB image.

[0067] Assume that the grayscale image of the object "spoon and hand (skin)" is as shown in FIG. 11A, the first depth map of the object is as shown in FIG. 11B, and the spoon and the hand (skin) are extracted as different regions by the region extraction unit 103. In this case, as shown in FIG. 11C, the display image is an image representing that the spoon and the hand (skin) extracted as different regions are drawn in different colors to indicate that they are extracted as different regions.

[0068] Furthermore, in the display image shown in FIG. 11C, for each of the extracted regions, the type of each identified region is indicated by a character. By displaying this display image on the display unit 180, the user can confirm the extracted region and the type of that region. Note that the method of indicating the type of region in the display image may be a method as shown in FIG. 11D, and any method may be used as long as the user can know the type of region. In FIG. 11D, it is assumed that the spoon is represented in blue and the hand (skin) is represented in red.

[0069] The processing in the second embodiment is performed as described above. According to the second embodiment, in addition to detecting the regions in the object, it is possible to specify the type of material or the like that constitutes the region.

[0070] Note that in order to perform detailed type specification (for example, specifying the types such as meat and vegetables in food, and specifying daikon radish, green onion, carrot, etc. in vegetables), experiments may be conducted to obtain depth differences and a table may be prepared.

[0071] <3. Third Embodiment> [3-1. Configuration of Information Processing System 30] Next, with reference to FIG. 12, the configuration of the information processing system 30 in the third embodiment of the present technology will be described. The information processing system 30 includes an information processing apparatus 300 and a ToF sensor 500. Since the ToF sensor 500 is the same as that in the first embodiment, its description will be omitted.

[0072] In the third embodiment, it is premised that on the surface of the object, there are regions that are different in material or the like from the object and have the same or similar color. Such regions may originally exist on the surface of the object, or may be something like a marker composed of characters, figures, etc. attached to the surface of the object by the user or the like. "Similar color" is the same as that described in the first embodiment.

[0073] [3-2. Configuration of Information Processing Apparatus 300] Next, the configuration of the information processing apparatus 300 in the third embodiment will be described. Since the configuration other than the functional blocks is the same as that in the first embodiment, the description thereof will be omitted.

[0074] As shown in FIG. 13, the information processing apparatus 300 in the third embodiment is configured to include functional blocks such as a state estimation unit 301, a depth map generation unit 101, a difference map generation unit 102, a region extraction unit 103, and an image processing unit 104. Since the depth map generation unit 101, the difference map generation unit 102, the region extraction unit 103, and the image processing unit 104 are the same as those in the first embodiment, the description thereof will be omitted.

[0075] The state estimation unit 301 estimates the state of the object using the three-dimensional shape data. The state of the object refers to the shape, posture, and size of the object. When there is one type of shape of the object, the state of the object may be only the posture. Also, the size is not essential, and the state of the object may be a combination of the posture and the shape.

[0076] The three-dimensional shape data is data showing a plurality of three-dimensional shapes such as a sphere, a cylinder, a cone, a square, a rectangle, a hexagonal prism, a triangular pyramid, a triangular prism, and a flat plate as shown in FIGS. 14A to 14I, for example. However, the three-dimensional shape data is not limited to these, and any three-dimensional shape information may be used. In order to enable detection of regions of objects with more various shapes, it is advisable to prepare more three-dimensional shape data in advance.

[0077] The three-dimensional shape data may be possessed by the state estimation unit 301, or may be stored in advance in the storage unit 160, and the state estimation unit 301 may read the three-dimensional shape data in the storage unit 160. Alternatively, a table may be stored in an external server or the like, and the information processing apparatus 300 may access the server via the interface 170 to read the three-dimensional shape data.

[0078] The information processing apparatus 300 is configured as described above.

[0079] [Processing by Information Processing Apparatus 300] Next, with reference to the flowchart of FIG. 15, the processing of the information processing apparatus 300 in the third embodiment will be described. Note that detailed descriptions of the same processing as in the first embodiment will be omitted.

[0080] In step S101, when the information processing apparatus 300 acquires a first depth map of the object from the ToF sensor 500, next, in step S301, the state estimation unit 301 estimates the state of the object.

[0081] To estimate the shape, posture, and size of the object, first, for one three-dimensional shape data, the posture and size at a certain viewpoint in the three-dimensional space are tentatively determined, and a depth map for state estimation of the three-dimensional shape viewed from that viewpoint is obtained. The depth map for state estimation is compared with the first depth map of the object. Then, this comparison process is performed for all possible patterns of postures and a plurality of sizes for each of all the three-dimensional shape data, and the one with the smallest depth difference is estimated as the shape, posture, and size of the object. Note that the range of the size is defined in advance, and a tentative size is determined within that range.

[0082] At that time, for example, if it is a sphere, the depth map is the same regardless of the posture, but in that case, it is advisable to select one as a representative.

[0083] Next, in step S302, the depth map generation unit 101 generates a second depth map from the viewpoint of the ToF sensor 500 based on the estimated posture of the object and in accordance with the estimated size of the object for the three-dimensional shape data estimated to be the state of the object in the process of step S301.

[0084] Next, in step S303, the difference map generation unit 102 generates a difference map based on the first depth map and the second depth map. The generation of the difference map is performed in the same manner as in the first embodiment by calculating the depth difference from the second depth map at each pixel constituting the first depth map.

[0085] Next, in step S304, the region extraction unit 103 generates a histogram of the difference map and detects peaks from the histogram.

[0086] Next, in step S305, the region extraction unit 103 extracts a region by extracting all the pixels having the depth difference of the peak detected in the histogram from among all the pixels constituting the first depth map.

[0087] Alternatively, the region extraction unit 103 extracts a region by extracting all the pixels having a depth difference included in a predetermined width centered on the peak detected in the histogram from the first depth map. The method of region extraction is the same as that in the first embodiment.

[0088] Next, in step S306, the image processing unit 104 generates a display image on which the extraction region extracted from the first depth map is drawn. The display image is generated by drawing the extraction region on the first depth map. Note that the display image may be generated by drawing on the second depth map or the three-dimensional shape data. By displaying the generated display image on the display unit 180, the user can confirm the extracted region.

[0089] The processing in the third embodiment is performed as described above.

[0090] According to the third embodiment, for example, when a marker is attached to any surface of an object whose orientation cannot be determined only by its shape and looks the same from any direction (up, down, left, or right), and the marker is extracted by the present technology, the orientation and surface of the object can be easily grasped.

[0091] Specifically, as shown in FIG. 16A, a marker (the character "front" in FIG. 16A) made of a material different from that of the object is provided as a mark on the front surface of the box, which is the object. Then, by extracting and recognizing the marker as a region by the present technology, it can be grasped that the surface with the marker is the front surface of the box.

[0092] Since the marker can be extracted even if its color is the same as or similar to that of the object, it does not interfere with the appearance of the object, and furthermore, it is possible to prevent people who view the object from recognizing the presence of the marker.

[0093] Also, as shown in Fig. 16B, if secret information is described in an area on the surface of the object that has a different material from the object and the same or similar color, the secret information can be read only by using this technology. As a result, the exchange of secret information can be carried out without being known to other people. In Fig. 16B, the words "confidential information" are described in characters as an area on the paper which is the object.

[0094] <4. Modification Example> As described above, the embodiments of the present technology have been specifically described. However, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

[0095] In the embodiment, an RGB stereo camera is used as the distance measurement sensor 600. However, any camera or sensor may be used as long as the distance measurement sensor can acquire depth information and generate a depth map. For example, the distance measurement sensor 600 may be a stereo IR camera composed of two IR (Infrared) cameras, or a triangulation using one IR camera and Structured Light.

[0096] In the embodiment, an image is acquired from the distance measurement sensor 600 and the information processing device 100 generates a depth map. However, a depth map may be generated by the distance measurement sensor 600 or an external device, and the information processing device 100 may acquire it.

[0097] Also, in any of the embodiments, the information processing device may operate in a server or the cloud. In that case, the information processing device receives and processes the first depth map generated by the ToF sensor 500, the image generated by the distance measurement sensor 600, etc. via a network.

[0098] The present technology can also adopt the following configuration. (1) A difference map generation unit that generates a difference map from a first depth map of an object acquired by a ToF sensor and a second depth map of the object, and a region extraction unit that extracts a region on the first depth map based on the difference map. An information processing apparatus. (2) The information processing apparatus according to (1), further comprising a depth map generation unit that generates the second depth map. (3) The depth map generation unit generates the second depth map based on a first image acquired by a first imaging device constituting a stereo camera as a distance measurement sensor and a second image acquired by a second imaging device constituting the stereo camera. The information processing apparatus according to (2). (4) The difference map generation unit generates the difference map by calculating a depth difference between the second depth map and each pixel constituting the first depth map. The information processing apparatus according to any one of (1) to (3). (5) The region extraction unit extracts a region on the first depth map having a depth difference corresponding to a peak in a histogram of the difference map. The information processing apparatus according to any one of (1) to (4). (6) The region extraction unit extracts a region on the first depth map having a depth difference included in a predetermined width centered on a peak in a histogram of the difference map. The information processing apparatus according to any one of (1) to (5). (7) The information processing apparatus according to any one of (1) to (6), further comprising an image processing unit that generates an image showing the region extracted by the region extraction unit. (8) An information processing apparatus according to any one of (1) to (7), comprising a type specifying unit that specifies the type of the object by referring to a table in which the depth difference and the type of the object are associated in advance. (9) The information processing apparatus according to (8), comprising an image processing unit that generates an image showing the region extracted by the region extraction unit and the type of the object specified by the type specifying unit. (10) The information processing apparatus according to any one of (1) to (9), comprising a state estimation unit that estimates the state of the object based on the first depth map and a depth map for state estimation generated from three-dimensional shape data. (11) In the information processing apparatus according to (10), the state of the object is the posture of the object. (12) In the information processing apparatus according to (11), the state of the object is the shape of the object. (13) In the information processing apparatus according to (11) or (12), the state of the object is the size of the object. (14) The depth map generation unit generates a second depth map for the three-dimensional shape data with respect to the viewpoint of the ToF sensor based on the state of the object, The information processing apparatus according to (10), wherein the difference map generation unit generates the difference map from the first depth map and the second depth map. (15) Generate a difference map from the first depth map of the object acquired by the ToF sensor and the second depth map of the object, Extract a region on the first depth map based on the difference map Information processing method. (16) Generate a difference map from the first depth map of the object acquired by the ToF sensor and the second depth map of the object, Extract a region on the first depth map based on the difference map An information processing program that causes a computer to execute an information processing method.

Explanation of Signs

[0099] 100, 200, 300 ··· Information processing apparatus 101 ··· Depth map generation unit 102 ··· Difference map generation unit 103 ··· Region extraction unit 104 ··· Image processing unit 201 ··· Type identification unit 301 ··· State estimation unit 500 ··· ToF sensor 600 ··· Stereo camera

Claims

1. A difference map generation unit that generates a difference map from a first depth map of an object acquired by a ToF sensor and a second depth map of the object; An area extraction unit that extracts an area on the first depth map having a depth difference corresponding to a peak in the histogram of the difference map based on the difference map; An information processing apparatus comprising:

2. The information processing apparatus according to claim 1, further comprising a depth map generation unit that generates the second depth map.

3. The depth map generation unit generates the second depth map based on a first image acquired by a first imaging device constituting a stereo camera as a distance measurement sensor and a second image acquired by a second imaging device constituting the stereo camera. The information processing apparatus according to claim 2.

4. The difference map generation unit generates the difference map by calculating a depth difference between each pixel constituting the first depth map and the second depth map. The information processing apparatus according to claim 1.

5. The area extraction unit extracts an area on the first depth map having a depth difference included in a predetermined width centered on a peak in the histogram of the difference map. The information processing apparatus according to claim 1.

6. The information processing apparatus according to claim 1, further comprising an image processing unit that generates an image showing the area extracted by the area extraction unit. The information processing apparatus according to claim 1.

7. The information processing apparatus according to claim 1, further comprising a type identification unit that identifies the type of the object by referring to a table in which a depth difference and the type of the object are associated in advance. The information processing apparatus according to claim 1.

8. The information processing apparatus according to claim 7, further comprising an image processing unit that generates an image showing the area extracted by the area extraction unit and the type of the object identified by the type identification unit. The information processing apparatus according to claim 7.

9. The information processing apparatus according to claim 1, further comprising a state estimation unit that estimates the state of the object based on the first depth map and a depth map for state estimation generated from three-dimensional shape data. The information processing apparatus according to claim 1.

10. The state of the object is the posture of the object. The information processing apparatus according to claim 9.

11. The state of the object is the shape of the object. The information processing apparatus according to claim 10.

12. The state of the object is the size of the object. The information processing apparatus according to claim 10.

13. ​ The depth map generation unit generates the second depth map for the three-dimensional shape data based on the viewpoint of the ToF sensor based on the state of the object, The difference map generation unit generates the difference map from the first depth map and the second depth map. The information processing apparatus according to claim 9.

14. The difference map generation unit generates a difference map from a first depth map for an object acquired by a ToF sensor and a second depth map for the object, The region extraction unit extracts a region on the first depth map having a depth difference corresponding to a peak in the histogram of the difference map based on the difference map. Information processing method.

15. Generate a difference map from a first depth map for an object acquired by a ToF sensor and a second depth map for the object, Based on the difference map, extract a region on the first depth map having a depth difference corresponding to a peak in the histogram of the difference map. An information processing program for causing a computer to execute an information processing method.

Images