DETECTION DEVICE, INFORMATION PROCESSING DEVICE, DETECTION METHOD, DETECTION PROGRAM, AND DETECTION SYSTEM

Abstract

To detect the shape or attitude of the whole or part of an object with high accuracy.SOLUTION: A detection device pertaining to one embodiment includes a detection unit that detects an object from one viewpoint. The detection device includes a model calculation unit that calculates the model information of the object using the detection results of the detection unit. The detection device includes a probability calculation unit that calculates probability information on the basis of the detection results that includes probability for each part of the object. The detection device includes an identification unit that identifies a first part among probability information in one viewpoint, the probability of which is low compared to a reference. The detection device includes an instruction generation unit that generates instruction information, which instructs next object detection by the detection unit to a user, on the basis of the first part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a detection device, an information processing device, a detection method, a detection program, and a detection system. [Background technology]

[0002] As a technique for detecting an object, for example, there is a technique described in Patent Document 1 below, but it is desired to detect the shape or orientation of all or a part of an object with high accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-134546 A Summary of the Invention

[0004] A detection device according to one aspect of the present invention ,one From a viewpoint Distance to the object The detection device includes a detection unit that detects an object. The detection device includes a model calculation unit that calculates model information of the object using a detection result from the detection unit. Detection section The detection device includes an accuracy calculation unit that calculates accuracy information including an accuracy for each part of the object based on the detection result. The detection device includes an identification unit that identifies a first part having a low accuracy relative to a standard from the accuracy information in one viewpoint. The detection device includes an instruction generation unit that generates instruction information that instructs a user on the next detection of the object by the detection unit based on the first part. The teaching information includes a detection direction, and the detection direction is determined based on a direction of a normal to the first portion calculated using a detection result of the detection unit.

[0005] An information processing device according to one aspect of the present invention includes a model calculation unit that calculates model information of an object obtained by detecting the object from one viewpoint. The information processing device includes an accuracy calculation unit that calculates accuracy information including an accuracy for each part of the object based on the result of detecting the object from one viewpoint. The information processing device includes an identification unit that identifies a first part having an accuracy lower than a standard from the accuracy information in one viewpoint. The information processing device calculates model information of the object based on the first part. ,versusThe apparatus further includes an instruction generating unit that generates instruction information for instructing a user on the next detection of an object. The instruction information includes a detection direction, which is determined based on a direction of a normal to a first portion calculated using a result of detecting the object from one viewpoint.

[0008] A detection program according to an embodiment of the present invention causes a computer to detect an object from one viewpoint. The detection program causes the computer to calculate model information of the object using a result of the detection. The detection program causes the computer to calculate accuracy information including an accuracy for each part of the object based on the result of the detection. The detection program causes the computer to identify a first part of the accuracy information in one viewpoint, the first part having an accuracy lower than a standard. The detection program causes the computer to generate instruction information for instructing a user on the next detection of the object based on the first part. The teaching information includes a detection direction, and the detection direction is determined based on a direction of a normal to the first portion calculated using a result of detecting the object.

[0009] A detection system according to one aspect of the present invention includes the above-mentioned detection device, and a processing device for processing information output from the detection device. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a functional block diagram illustrating an example of a detection device according to the first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a detection device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a detection unit according to the first embodiment. [Figure 4] 5A to 5C are diagrams illustrating an example of a display on a display unit according to the first embodiment. [Diagram 5] 6A to 6C are diagrams illustrating other display examples of the display unit according to the first embodiment. [Figure 6] 6A to 6C are diagrams illustrating other display examples of the display unit according to the first embodiment. [Figure 7] 4 is a flowchart showing an example of a detection method according to the first embodiment. [Figure 8] 6(A) and 6(B) are flowcharts showing another example of the detection method according to the first embodiment. [Figure 9] FIG. 11 is a functional block diagram illustrating an example of a detection device according to a second embodiment. [Figure 10] 13A to 13C are diagrams illustrating an example of a detection method performed by the detection device according to the second embodiment. [Figure 11] 13A and 13B are diagrams illustrating a display example of a display unit according to a second embodiment. [Figure 12] 10 is a flowchart showing an example of a detection method according to the second embodiment. [Figure 13] FIG. 11 is a functional block diagram illustrating an example of a detection device according to a third embodiment. [Figure 14] 13A to 13C are diagrams illustrating an example of a detection method using a detection device according to a third embodiment. [Figure 15] 13 is a diagram showing an example of calculation of a normal direction by a direction calculation unit according to the third embodiment; FIG. [Figure 16] 13A and 13B are diagrams illustrating a display example of a display unit according to a third embodiment. [Figure 17] 13A and 13B are diagrams illustrating another display example of the display unit according to the third embodiment. [Figure 18] 13A and 13B are diagrams illustrating another display example of the display unit according to the third embodiment. [Figure 19] 13 is a flowchart showing an example of a detection method according to the third embodiment. [Figure 20] FIG. 13 is a functional block diagram illustrating an example of an information processing device according to a fourth embodiment. [Figure 21] FIG. 13 is a functional block diagram illustrating an example of an information processing device according to a fifth embodiment. [Figure 22] FIG. 13 is a functional block diagram illustrating an example of an information processing device according to a sixth embodiment. [Diagram 23] 13A and 13B show an example of a moving device according to a sixth embodiment, in which (A) is a diagram showing a state before movement, and (B) is a diagram showing a state after movement. [Figure 24] 13A and 13B show another example of the moving device according to the sixth embodiment, in which (A) shows a state before movement, and (B) shows a state after movement. [Diagram 25] FIG. 13 is a functional block diagram illustrating an example of a detection system according to a seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described with reference to the drawings. In the drawings, in order to explain the embodiments, some parts are enlarged or emphasized, and the scale is appropriately changed. In the drawings, there is a diagram explaining the directions in the drawings using an XYZ coordinate system. In this XYZ coordinate system, a plane parallel to the horizontal plane is the XZ plane. One direction in this XY plane is written as the X direction, and a direction perpendicular to the X direction is written as the Y direction. A direction perpendicular to the XY plane is written as the Z direction. The X direction, Y direction, and Z direction will be explained assuming that the direction indicated by the arrow in the drawing is the + direction, and the opposite direction is the - direction.

[0013] [First embodiment] A first embodiment will be described. FIG. 1 is a functional block diagram showing an example of the configuration of a detection device 1. FIG. 2 is a conceptual diagram showing an example of the detection device 1 according to this embodiment. As shown in FIG. 1 and FIG. 2, the detection device 1 is, for example, an imaging device, and detects an object OB in a detection area A1 (for example, a field of view). The detection device 1 may be, for example, a fixed camera, a camera whose field of view can be changed manually or automatically, or a portable information terminal (for example, a smartphone, a tablet, a mobile phone with a camera). The detection device 1 performs a calculation process on information related to the object OB using the result of detecting the object OB. The detection device 1 models at least a part of the object OB by the calculation process of its own terminal, and calculates model information (model data). For example, the detection device 1 performs computer graphic processing (CG processing) on ​​at least a part of the object OB by the calculation process, and calculates model information (for example, CG model data).

[0014] The model information includes, for example, at least one of shape information indicating the three-dimensional shape of the object OB and texture information indicating the pattern on the surface of the object OB. Also, for example, the model information includes at least one of three-dimensional point coordinates, related information of the point coordinates, texture information of the surface defined by the three-dimensional point coordinates and the related information, spatial information of the image such as the lighting conditions and light source information of the entire image, and polygon data as shape information. The texture information includes, for example, at least one of information on characters, figures, patterns, and information defining unevenness on the surface of the object OB, a specific image, and color (e.g., chromatic color, achromatic color). For example, the detection device 1 calculates model information indicating the object OB viewed from a viewpoint Vp (e.g., a predetermined viewpoint, one viewpoint, a single viewpoint, one direction). Also, for example, the detection device 1 calculates model information indicating the object OB viewed from one viewpoint at a predetermined angle to the object OB.

[0015] The detection device 1 includes, for example, a detection unit 2, a probability calculation unit 3, a model calculation unit 4, an identification unit 5, a control unit 6, a storage unit 7, a communication unit 8, a display control unit 9, and a main body unit 10. The main body unit 10 is, for example, a camera body, a case, a housing, etc. The detection unit 2, the probability calculation unit 3, the model calculation unit 4, the identification unit 5, the control unit 6, the storage unit 7, the communication unit 8, and the display control unit 9 are, for example, provided in the main body unit 10 to be unitized. The detection unit 2 optically detects an object OB from a viewpoint Vp (one viewpoint). For example, the detection unit 2 captures an image of the object OB from the viewpoint Vp, and outputs image data of the object OB as a detection result.

[0016] FIG. 3 is a diagram showing an example of the detection unit 2. The detection unit 2 optically detects the object OB from, for example, a viewpoint Vp. The detection unit 2 acquires at least one of an image of the object OB viewed from one viewpoint (viewpoint Vp) and a distance from one viewpoint (viewpoint Vp) to each point on the object OB. The detection unit 2 may detect the object OB at, for example, a predetermined visual angle. The detection unit 2 may detect the object OB at, for example, a predetermined line of sight (e.g., a single line of sight). The detection unit 2 includes, for example, an imaging unit 21 and a distance measuring unit 22. The imaging unit 21 captures an image of the object OB from the viewpoint Vp and outputs image data of the object OB as a detection result. The detection result of the detection unit 2 includes, for example, image data of the detected object OB. The distance measuring unit 22 detects a distance from the viewpoint Vp (see FIG. 2) to each point on the object OB. Note that the detection unit 2 does not need to include the imaging unit 21 or the distance measuring unit 22.

[0017] The imaging unit 21 includes an imaging optical system 13 and an image sensor 14. The imaging optical system 13 forms an image of an object OB. The imaging optical system 13 is held, for example, in a lens barrel, and is attached to the main body 10 (see FIG. 2) together with the lens barrel. The imaging optical system 13 and the lens barrel are, for example, interchangeable lenses, and are removable from the main body 10. The imaging optical system 13 and the lens barrel may be built-in lenses, and for example, the lens barrel may be part of the main body 10, or may not be removable from the main body 10.

[0018] The imaging element 14 is, for example, a CMOS image sensor in which a plurality of pixels are two-dimensionally arranged, or a CCD image sensor. The imaging element 14 is housed, for example, in the main body 10. The imaging element 14 captures an image formed by the imaging optical system 13. The imaging result (detection result) of the imaging element 14 includes, for example, information on the gradation value for each color of each pixel (e.g., RGB data). The imaging element 14 outputs the imaging result, for example, in a data format of a full-color image.

[0019] The distance measuring unit 22 detects the distance from one viewpoint to each point on the object OB (on the surface of the object OB). The distance measuring unit 22 detects the distance by, for example, a time of flight (TOF) method. The distance measuring unit 22 may detect the distance by other methods. For example, the distance measuring unit 22 may include a laser scanner and detect the distance by laser scanning. For example, the distance measuring unit 22 may project a predetermined pattern onto the object OB and measure the distance based on the detection result of this pattern. The distance measuring unit 22 may also include a phase difference sensor and detect the distance by a phase difference method. The distance measuring unit 22 may also detect the distance by a depth from defocus (DFD) method. When the DFD method is used, the distance measuring unit 22 may use at least one of the imaging optical system 13 and the image sensor 14 of the imaging unit 21.

[0020] The distance measuring unit 22 includes, for example, an irradiation unit 15, an imaging optical system 16, an image sensor 17, and a controller 18. The irradiation unit 15 can irradiate the object OB with infrared light. The irradiation unit 15 is controlled by the controller 18. The controller 18 changes the intensity of the infrared light irradiated from the irradiation unit 15 over time (for example, amplitude modulates). The imaging optical system 16 forms an image of the object OB. As with the imaging optical system 13, the imaging optical system 16 may be at least a part of an interchangeable lens or at least a part of a built-in lens. The image sensor 17 is, for example, a CMOS image sensor or a CCD image sensor. The image sensor 17 has sensitivity to at least the wavelength band of the light irradiated by the irradiation unit 15. The image sensor 17 is, for example, controlled by the controller 18, and detects the infrared light reflected and scattered by the object OB. For example, the image sensor 17 captures an image formed by the imaging optical system 16.

[0021] The controller 18 detects the distance (depth) from each point on the surface of the object OB to the image sensor 17 using the detection result by the image sensor 17. For example, the flight time of light incident on the image sensor 17 from a point on the surface of the object OB changes according to the depth of the point. The output of the image sensor 17 changes according to the flight time, and the controller 18 calculates the depth based on the output of the image sensor 17, for example. The controller 18 calculates the depth for each partial area (e.g., one pixel, multiple pixels) of the image captured by the image sensor 17, and calculates (generates) depth information by associating the position of the area with the depth. The depth information includes, for example, information that associates the position of a point on the surface of the object OB with the distance (depth, depth) from the point to the detection device 1. The depth information includes, for example, information (e.g., depth image) indicating the distribution of depths in the object OB (e.g., depth map).

[0022] The point cloud data generating unit 23 calculates point cloud data including coordinates of a plurality of points on the object OB based on the detection result of the detecting unit 2. The point cloud data generating unit 23 executes a point cloud data calculation process to calculate point cloud data using the detection result (e.g., depth information) of the distance measuring unit 22. For example, the point cloud data generating unit 23 calculates the point cloud data by perspective transformation from a distance image (depth image) indicated by the depth information to a planar image. In this case, for example, the point cloud data is a plurality of points that are arranged on a plane corresponding to a plurality of regions (e.g., one pixel, a plurality of pixels) in the depth image and can connect adjacent or nearby regions. Note that, when the imaging unit 21 and the distance measuring unit 22 have different fields of view, the point cloud data generating unit 23 may convert the detection result of the distance measuring unit 22 into a result of detecting the object OB from the field of view of the imaging unit 21 by perspective transformation (projection transformation), for example.

[0023] The point cloud data generating unit 23 may perform perspective transformation using, for example, parameters that depend on the positional relationship (e.g., the position of the viewpoint, the direction of the line of sight) between the field of view of the imaging unit 21 and the field of view of the distance measuring unit 22. The point cloud data generating unit 23 stores the calculated point cloud data in the storage unit 7, for example. Note that the point cloud data generating unit 23 does not have to be included in the detection unit 2. The point cloud data generating unit 23 may be provided outside the detection unit 2. The point cloud data generating unit 23 may be included in, for example, the model calculation unit 4. That is, the model calculation unit 4 may calculate the point cloud data based on the detection result of the detection unit 2.

[0024] The accuracy calculation unit 3 calculates accuracy information including accuracy for each part of the object OB based on the detection result of the detection unit 2. The accuracy calculation unit 3 executes an accuracy calculation process to calculate accuracy information of the object OB. The accuracy information includes at least one of the probability that an object (object OB) exists at each position of an area including the object OB (e.g., the detection area A1 of the detection unit 2, the field of view), a probability distribution, and an expected value. This probability distribution is, for example, information indicating the relationship between the accuracy of a part of the detection area of ​​the detection unit 2 and the accuracy of the surrounding area. The accuracy calculation unit 3 may calculate the accuracy information based on, for example, device characteristic information. The device characteristic information is information indicating the characteristics of each part provided in the detection device 1. The device characteristic information includes, for example, the optical characteristics (e.g., aberration) of the optical system (e.g., the imaging optical system 13, the imaging optical system 16) provided in the detection unit 2, the distance measurement range of the distance measurement unit 22, and other characteristics.

[0025] The imaging optical system 13 (see FIG. 3) of the imaging unit 21 has a larger aberration, for example, the further away from the optical axis 13a. For example, the optical axis 13a of the imaging optical system 13 corresponds to the visual field center 14a of the imaging unit 21, and the reliability of the detection result using the imaging unit 21 decreases as the area (e.g., one pixel, multiple pixels) in the imaging element 14 is farther from the visual field center 14a. The captured image of the imaging unit 21 is more affected by aberration, for example, the farther away from the optical axis 13a, and the reproducibility of the subject (reliability of the detection result) decreases due to blurring, etc. The accuracy calculation unit 3 may calculate accuracy information based on the distance from the visual field center 14a for each area (e.g., one pixel, multiple pixels) of the captured image.

[0026] The relationship between the accuracy and the distance from the visual field center 14a of the imaging unit 21 is determined based on, for example, the distribution of aberration on the imaging element 14 caused by the imaging optical system 13. The accuracy may change nonlinearly with respect to the distance from the visual field center 14a, or may change linearly with respect to the distance from the visual field center 14a. The accuracy may also change discontinuously (for example, stepwise) with respect to the distance from the visual field center 14a. Information indicating the relationship between the accuracy and the distance from the visual field center 14a of the imaging unit 21 on the captured image (hereinafter referred to as first relationship information) is included in, for example, the above device characteristic information, and is stored in advance in the storage unit 7 (see FIG. 1). The accuracy calculation unit 3 calculates, for example, the distance from the visual field center 14a for each region (for example, one pixel, multiple pixels) of the captured image. The accuracy calculation unit 3 compares the calculated distance with the first relationship information to calculate the accuracy in this region. The accuracy calculation unit 3 may calculate, for example, information associating the position of each region with the accuracy as the accuracy information.

[0027] The likelihood calculation unit 3 may calculate the likelihood information based on the aberration (e.g., distribution of aberration) of the imaging optical system 13. The likelihood calculation unit 3 may also calculate the likelihood information based on the aberration of the imaging optical system 13 and the distance from the center of the field of view. The likelihood calculation unit 3 may also calculate the likelihood information for the detection result of the distance measurement unit 22 based on at least one of the distance from the center of the field of view of the distance measurement unit 22 and the aberration of the imaging optical system 16.

[0028] Also, the brightness of the captured image may be saturated in some areas due to, for example, reflected light from the object OB. Also, when there is a shadow of another object on the object OB, a dark area may be formed when viewed from the viewpoint Vp. The accuracy calculation unit 3 may calculate the accuracy information based on, for example, the brightness (light and dark) in the captured image. The relationship between the brightness and the accuracy is arbitrarily set based on, for example, an experiment or a simulation. For example, the accuracy may change nonlinearly or linearly with respect to the brightness, or may change discontinuously (e.g., in steps).

[0029] Information indicating the relationship between brightness and accuracy (hereinafter referred to as second relationship information) is stored in advance in, for example, the storage unit 7 (see FIG. 1). The accuracy calculation unit 3 compares the brightness (e.g., gradation value) of each region (e.g., one pixel, multiple pixels) of the captured image with the second relationship information, and calculates the accuracy of this region. The accuracy calculation unit 3 may calculate, for example, information that associates the position of each region with the accuracy as the accuracy information.

[0030] The information associating the position and accuracy of each region may be stored in the same file as the captured image data. For example, the data structure (data type, data format) of this information may be a structure in which the gradation value of a pixel (gradation value of each of R, G, and B) and the accuracy are paired. The information associating the position and accuracy of each region may be stored in a file other than the captured image data. For example, the data structure of the information associating the position and accuracy of a region may be a structure in which the accuracy value of each pixel is arranged in correspondence with the pixel arrangement of the captured image. Note that the accuracy calculation unit 3 does not need to calculate the accuracy information based on the optical characteristics of the optical system (e.g., the imaging optical system 13, the imaging optical system 16) provided in the detection unit 2. For example, the accuracy calculation unit 3 does not need to calculate the information associating the position and accuracy of the above-mentioned region.

[0031] Also, the accuracy calculation unit 3 may calculate the accuracy information based on the distance detected by the distance measurement unit 22, for example. The reliability (e.g., detection accuracy) of the detection result of the distance measurement unit 22 may decrease as the position is farther from the viewpoint Vp. In such a case, the accuracy calculation unit 3 may calculate, for example, an accuracy value having a negative correlation with the distance as the accuracy information. The accuracy decreases as the distance from the viewpoint Vp increases. Information indicating the relationship between the accuracy and the distance from the viewpoint Vp (hereinafter, third relationship information) is stored, for example, in the storage unit 7 (see FIG. 1) in advance. For example, for each region (e.g., one pixel, multiple pixels) on the depth image, which is information indicating the distribution of depths (e.g., a depth map), the accuracy calculation unit 3 compares the distance to the viewpoint Vp in the region with the third relationship information to calculate the accuracy of the region. For example, the accuracy calculation unit 3 may calculate, as the accuracy information, information associating the position of each region with the accuracy.

[0032] The distance measuring unit 22 has, for example, a predetermined distance measuring range, and the reliability of the detection result (e.g., detection accuracy) may decrease as the distance from the center of the distance measuring range increases. In such a case, the accuracy calculation unit 3 may calculate the accuracy information based on, for example, the amount of deviation between the center of the distance measuring range and the distance from a predetermined viewpoint. The accuracy is maximum at the center of the distance measuring range, and decreases as the distance deviates from the center of the distance measuring range. The relationship between the distance and the accuracy is appropriately set according to, for example, the characteristics of the distance measuring unit 22. The accuracy calculation unit 3 may also calculate the accuracy information based on the distribution of the distances detected by the distance measuring unit 22.

[0033] The reliability of the detection result of the distance measuring unit 22 decreases, for example, at a position where the change in distance is greater than a threshold (for example, near a step). In such a case, the accuracy calculation unit 3 may calculate, for example, an accuracy according to the change in distance as the accuracy information. For example, the accuracy calculation unit 3 may calculate, as the accuracy information, an accuracy of a negative correlation value with respect to the change in distance. For example, the accuracy calculation unit 3 may calculate an accuracy indicating that the reliability of the detection result of the distance measuring unit 22 is high for an area where the change in distance is relatively small. Also, for example, the accuracy calculation unit 3 may calculate an accuracy indicating that the reliability of the detection result of the distance measuring unit 22 is low for an area where the change in distance is relatively large.

[0034] The amount of change in distance is, for example, an amount according to the inclination between the line of sight passing through the viewpoint of the detection unit 2 and the surface on the object OB. For example, when the amount of change in distance is large, the surface on the object OB is nearly parallel to the line of sight, and when the amount of change in distance is small, the surface on the object OB is nearly perpendicular to the line of sight. The accuracy calculation unit 3, for example, calculates the amount of change in distance (e.g., the difference between the depths in two adjacent areas) for each area of ​​the depth image, and compares the calculated amount of change with a threshold. For example, when the amount of change in distance is equal to or smaller than the threshold, the accuracy calculation unit 3 sets the accuracy of this area to be high. When the amount of change in distance is greater than the threshold, the accuracy calculation unit 3 sets the accuracy of this area to be low.

[0035] The accuracy is constant in a range where the change in distance is equal to or less than a threshold, and decreases in a range where the change in distance exceeds the threshold. The accuracy may change nonlinearly, linearly, or discontinuously (e.g., stepwise) with respect to the change in distance. Information indicating the relationship between the accuracy and the change in distance from the viewpoint Vp (hereinafter, fourth relationship information) may be stored in advance in the storage unit 7 (see FIG. 1), for example. The accuracy calculation unit 3 calculates the change in distance to the viewpoint Vp for each region (e.g., one pixel, multiple pixels) on the depth image, and calculates the accuracy of each region by collating this change with the fourth relationship information. The accuracy calculation unit 3 may calculate, for example, information that associates the position of each region with the accuracy as the accuracy information.

[0036] The information associating the position and accuracy of each region described above may be stored in the same file as the depth information. For example, the data structure (data format) of this information may be a structure in which accuracy and depth are paired for each part of the detection region A1. For example, this information may be expressed in a format in which a depth value and accuracy are paired for each region of the depth image (e.g., one pixel, multiple pixels). Also, the information associating the position and accuracy of each region may be stored in a file separate from the depth information. For example, the information associating the position and accuracy of each region may be a structure in which the accuracy value of each region is arranged in correspondence with the data arrangement of the distance of each region in the depth information.

[0037] The accuracy calculation unit 3 may not calculate information associating the position of each area with the accuracy. The accuracy calculation unit 3 may not calculate accuracy information based on the distance detected by the detection unit 2 (e.g., the distance measurement unit 22). The accuracy calculation unit 3 may calculate an accuracy that combines two or more accuracy. For example, the accuracy calculation unit 3 may calculate the accuracy by weighting (calculating a weighted average) using a plurality of pieces of information associating the position of each area with the accuracy. The accuracy calculation unit 3 may calculate at least one of the arithmetic mean and the geometric mean of two or more pieces of information associating the position of each area with the accuracy as the accuracy.

[0038] The point cloud data generating unit 23 may calculate the point cloud data using the accuracy information. For example, the point cloud data generating unit 23 may select an area in the depth image with a relatively high accuracy, and perform perspective transformation from the depth image to a planar image. For example, the point cloud data generating unit 23 may omit perspective transformation of at least a part of an area in the depth image with a relatively low accuracy. In addition, the point cloud data generating unit 23 may use an area in the depth image with a relatively high accuracy to interpolate an area with a relatively low accuracy, and perform perspective transformation from the depth image to a planar image. The point cloud data generating unit 23 may generate information that associates the point cloud data with the accuracy information, for example. For example, the point cloud data generating unit 23 may calculate information that associates three-dimensional point coordinates included in the point cloud data with the accuracy (point accuracy information) of a point on the depth image corresponding to the point. The data structure of this information may be, for example, a structure in which three-dimensional point coordinates and accuracy are paired. The point cloud data generating unit 23 may, for example, store in the storage unit 7 information that associates the point cloud data with the point accuracy information.

[0039] The accuracy calculation unit 3 calculates accuracy information (point accuracy information) regarding information of points included in the point cloud data, for example, by using the point cloud data generated by the point cloud data generation unit 23. The accuracy information includes, for example, the reliability of the detection result of the detection unit 2 for each position of the area (e.g., the detection area and field of view of the detection unit 2) including the object OB. For example, the accuracy is a numerical representation of the accuracy (accuracy, precision, reliability) or uncertainty (vagueness, unclearness) of the data at each position, and the accuracy information includes a distribution of accuracy at multiple positions. For example, high accuracy corresponds to high accuracy or low uncertainty. For example, low accuracy corresponds to low accuracy or high uncertainty. The accuracy information includes a collection of accuracy (multiple accuracy), for example, an accuracy map. The accuracy information includes accuracy (i.e., multiple accuracy) calculated for each part of the object OB.

[0040] The accuracy calculation unit 3 calculates accuracy information (accuracy information of point cloud data) regarding information of points included in the point cloud data, for example, by using the point cloud data generated by the point cloud data generation unit 23. For example, the accuracy calculation unit 3 may calculate the accuracy information based on distance information between two predetermined points included in the point cloud data. For example, the accuracy calculation unit 3 may select two adjacent points from the point cloud data, compare the distance between these two points with a threshold, and calculate the accuracy information. In addition, the accuracy calculation unit 3 may calculate the accuracy information based on spatial frequency information (e.g., density of spatial distribution of points) of multiple points included in the point cloud data.

[0041] The accuracy calculation unit 3 may also calculate the accuracy information based on vector information connecting two points included in the point cloud data. For example, the accuracy calculation unit 3 may select two adjacent points from the point cloud data, and calculate the accuracy information using a vector connecting these two points and position information of the detection device 1. The position information of the detection device 1 includes, for example, the direction of the viewpoint Vp (detection direction, line of sight, direction of the optical axis of the optical system). The accuracy calculation unit 3 may calculate the accuracy information according to, for example, an angle between the vector connecting two points included in the point cloud data and the direction of the viewpoint Vp. For example, when the angle between the vector and the direction of the viewpoint Vp is close to 0° or 180°, the accuracy calculation unit 3 may relatively lower the accuracy of at least one of the start point and the end point of the vector. For example, when the angle between the vector and the direction of the viewpoint Vp is close to 90° or 270°, the accuracy calculation unit 3 may relatively increase the accuracy.

[0042] The information used by the accuracy calculation unit 3 to calculate the accuracy information (point accuracy information) may be one type of information or two or more types of information among the above-mentioned distance information, spatial frequency information, vector information, and position information of the detection device 1. The point cloud data used by the accuracy calculation unit 3 to calculate the point accuracy information may be generated by the point cloud data generation unit 23 using the accuracy information, or may be generated by the point cloud data generation unit 23 without using the accuracy information. In addition, the accuracy calculation unit 3 may store the point accuracy information calculated using the point cloud data in the storage unit 7.

[0043] The accuracy calculation unit 3 may also calculate accuracy information (line accuracy information) related to line information, for example. For example, the accuracy calculation unit 3 may calculate the line accuracy information by arithmetic mean, geometric mean, or weighted average using point accuracy information of two points corresponding to the end points of the line. For example, the accuracy calculation unit 3 may compare the lower value of the accuracy of the two end points with a threshold to calculate (evaluate) the accuracy of the line. The accuracy calculation unit 3 may also calculate the line accuracy information using the line information. For example, the accuracy calculation unit 3 may calculate the line accuracy information using at least one of the above-mentioned distance information, vector information, and position information of the detection device 1, similar to the calculation of the point accuracy information using the above-mentioned point cloud data. The accuracy calculation unit 3 may store the calculated line accuracy information in the storage unit 7.

[0044] The accuracy calculation unit 3 may also calculate accuracy information (surface accuracy information) regarding the information of the surface. For example, the accuracy calculation unit 3 may calculate the surface accuracy information by arithmetic mean, geometric mean, or weighted average using line accuracy information of a line corresponding to the perimeter line of the surface. The accuracy calculation unit 3 may also calculate the accuracy information by using the normal direction of the surface and position information of the detection device 1 (for example, the direction of the viewpoint Vp). For example, the accuracy calculation unit 3 may calculate the surface accuracy information according to the angle between the normal vector of the surface and the direction of the viewpoint Vp. For example, the accuracy calculation unit 3 may relatively lower the accuracy of the surface when the angle between the normal vector of the surface and the direction of the viewpoint Vp is close to 0° or 180° (for example, when the surface and the line of sight are nearly parallel). For example, the accuracy calculation unit 3 may relatively increase the accuracy of the surface when the angle between the normal vector of the surface and the direction of the viewpoint Vp is close to 90° or 270° (when the surface and the line of sight are nearly perpendicular). The accuracy calculation unit 3 may store the calculated surface accuracy information in the storage unit 7.

[0045] The accuracy calculation unit 3 may calculate the accuracy information using, for example, the surface information generated by the model calculation unit 4. Furthermore, the accuracy calculation unit 3 may store the accuracy information calculated using the surface information in the storage unit 7. When the accuracy calculation unit 3 calculates the accuracy information using the surface information, the model calculation unit 4 may recalculate at least a part of the surface information (e.g., line information, face information) using this accuracy information.

[0046] The accuracy calculation unit 3 may calculate accuracy information related to the texture information, for example, using image data on which the texture information is based. The accuracy calculation unit 3 may calculate the accuracy information, for example, using at least one of the luminance (tone value) of R, G, and B of each pixel included in the image data and spatial frequency information of the texture calculated from the image data. The accuracy calculation unit 3 may calculate the accuracy information, for example, using at least a part of the characteristics (e.g., illumination field, aberration) of the optical system used to acquire the image data, the optical characteristics (e.g., reflectance, transmittance, absorptance) of the object, and information on the light source of the light that illuminates the object.

[0047] The accuracy calculation unit 3 calculates accuracy information for each region of the object OB (e.g., each point of the point cloud data, an element of the surface information). The accuracy information may include, for example, information weighted on the detection result of the detection unit 2 in order to calculate shape information or texture information. The accuracy information may include, for example, information weighted on the detection result of the detection unit 2 or information generated from the detection result as information used in calculating shape information or texture information. The coefficient used for this weighting may be a value according to the accuracy included in the accuracy information acquired prior to the weighting process.

[0048] The model calculation unit 4 calculates model information of the object OB using the detection result of the detection unit 2. The model calculation unit 4 executes a model calculation process to calculate the model information of the object OB. The model calculation unit 4 calculates, for example, at least one of shape information and texture information of the object OB when viewed from the viewpoint Vp. The model calculation unit 4 calculates the model information of the object OB using, for example, point cloud data generated by the point cloud data generation unit 23. The model information includes at least one of shape information and texture information of the object OB at the viewpoint Vp.

[0049] The model calculation unit 4 generates surface information including, for example, coordinates of a plurality of points on the object OB and connection information between the plurality of points as shape information based on the detection result of the detection unit 2. The surface information is, for example, polygon data, vector data, draw data, etc. The connection information includes, for example, information that associates points at both ends of a line corresponding to a ridge line (e.g., an edge) of the object OB with each other, and information that associates a plurality of lines corresponding to the contour of the surface of the object OB with each other. The model calculation unit 4 may generate the surface information using, for example, the accuracy information calculated by the accuracy calculation unit 3.

[0050] The model calculation unit 4 generates line information using, for example, point cloud data. The model calculation unit 4 generates surface information using the line information. The model calculation unit 4 estimates a surface between a point (e.g., a point corresponding to a pixel in a depth image) selected from a plurality of points included in the point cloud data and a point nearby the point, and converts the point cloud data into polygon data having plane information between the points (surface processing). The model calculation unit 4 converts the point cloud data into polygon data using, for example, an algorithm using the least squares method. This algorithm may be, for example, an algorithm published in a point cloud processing library. The model calculation unit 4 may, for example, associate the accuracy information calculated for each region of the object OB with at least shape information. The model calculation unit 4 may store the calculated surface information in the storage unit 7.

[0051] The model calculation unit 4 may calculate texture information using the accuracy information calculated by the accuracy calculation unit 3, for example. The model calculation unit 4 may not use the accuracy information when generating surface information. For example, the model calculation unit 4 may not use the accuracy information when the point cloud data generation unit 23 generates point cloud data using the accuracy information and generates surface information based on this point cloud data. The model calculation unit 4 may not use the line accuracy information when it generates line information using point accuracy information and generates surface information based on this line information.

[0052] The model calculation unit 4 calculates the texture information by, for example, an inverse rendering technique. The texture information includes, for example, at least one item of information from pattern information indicating the pattern on the surface of the object OB, light source information of the light illuminating the object OB, and optical property information indicating the optical properties (e.g., reflectance, scattering rate, transmittance) of the surface of the object OB. The light source information includes, for example, at least one item of information from the position of the light source, the direction in which light is irradiated from the light source to the object, the wavelength of the light irradiated from the light source, and the type of the light source.

[0053] The model calculation unit 4 calculates light source information using, for example, a model assuming Lambertian reflection, a model including an albedo estimation, or the like. For example, the model calculation unit 4 estimates a component derived from light diffused by the object OB and a component specularly reflected by the object OB among the pixel values ​​of each pixel of the image captured by the imaging unit 21. In addition, the model calculation unit 4 calculates a direction in which light is incident from the light source to the object OB using, for example, an estimation result of the component specularly reflected by the object OB and shape information. For example, the model calculation unit 4 estimates the reflection characteristics of the object OB using the calculated light source information and shape information, and calculates optical characteristic information including the estimation result of the reflection characteristics. In addition, the model calculation unit 4 removes the influence of the illumination light from the data of the visible light image using, for example, the calculated light source information and optical characteristic information, and calculates texture information (pattern information). The model calculation unit 4 may store the calculated texture information in the storage unit 7.

[0054] The model calculation unit 4 may, for example, selectively use an area with a relatively high degree of accuracy from among the detection results (captured image) of the detection unit 2 (e.g., the imaging unit 21) to calculate the texture information. The model calculation unit 4 may not use an area with a relatively low degree of accuracy from among the captured image by the imaging unit 21 to calculate the texture information, in which case it is possible to reduce the processing load and the amount of data of the generated texture information. Furthermore, the model calculation unit 4 may, for example, calculate the texture information by interpolating information from an area with a relatively low degree of accuracy using information from an area with a relatively high degree of accuracy.

[0055] The identification unit 5 identifies a first portion AR1 in which the accuracy of each point in the object OB is low compared to a reference level, from among the accuracy information calculated by the accuracy calculation unit 3. The identification unit 5 executes a first portion identification process to identify the first portion AR1. The object OB is, for example, arranged with the protrusion OB1 facing the +X direction in the XYZ coordinate system. In this state, as shown in FIG. 2, when the accuracy information is calculated based on the information acquired by the detection unit 2 from the viewpoint Vp, the portion in the shadow of the object OB as seen from the viewpoint Vp (the portion that cannot be seen from the viewpoint Vp, including the portion in the shadow of the protrusion OB1) becomes the first portion AR1 in which the accuracy is low compared to a reference level. The first portion AR1 is an area in which the accuracy is low compared to a reference level. For example, the first portion AR1 is a low-precision area.

[0056] The object OB includes a first portion AR1 and a second portion AR2 different from the first portion AR1. The second portion AR2 is a part or all of the object OB excluding the first portion AR1. The second portion AR2 is an area having a higher accuracy than the accuracy of each point of the first portion AR1. The second portion AR2 is an area having a high accuracy with respect to a reference.

[0057] The identification unit 5 sets a threshold (predetermined value) as a criterion, and identifies a portion with a certainty smaller than the threshold as the first portion AR1. The identification unit 5 also identifies a portion excluding a portion with a certainty larger than the threshold as the first portion AR1. The threshold may be a value set in advance, or may be a value obtained by calculation or the like. The threshold may be a value set by the user looking at the model information generated by the model calculation unit 4. When the user sets the threshold by looking at the model information, for example, the user may set the threshold by surrounding (or painting) an area to be the first portion AR1 with an input device (e.g., a touch pen, etc.) for the model information displayed on the screen DPA (see FIG. 4, etc.) of the display unit DP, so that the surrounded area (or the painted area) is the first portion AR1 (i.e., a threshold that is a value larger than the certainty included in the surrounded area). The criterion (threshold) set in the identification unit 5 can be determined according to the shape of the object OB (e.g., when it includes complex unevenness), the detection accuracy in the detection unit 2, and the like.

[0058] The identification unit 5 may set a plurality of regions (a plurality of portions) for the second portion AR2 based on the accuracy information. The identification unit 5 may set, for example, a second threshold (a second predetermined value) and set the second portion AR2 to two regions (portions) AR21 and AR22 (see FIG. 5). For example, the region AR21 is a region with a higher accuracy than the second threshold, and the region AR22 is a region with a lower accuracy than the second threshold. The second threshold may be a value set in advance, or may be a value obtained by calculation or the like. The second threshold may be a value set by the user looking at the model information generated by the model calculation unit 4. The identification unit 5 may set, for example, the second portion AR2 to three or more regions. The identification unit 5 may calculate a ratio between the first portion AR1 and the second portion AR2. After identifying the first portion AR1, the identification unit 5 may calculate a ratio between the first portion AR1 and the second portion AR2, and store the calculation result in the storage unit 7.

[0059] The control unit 6 controls each unit of the detection device 1, for example, by a command (control signal) from a user or an external device. For example, the control unit 6 causes the detection unit 2 to execute the above-mentioned detection process. This detection process includes, for example, an imaging process by the imaging unit 21. For example, the control unit 6 causes at least a part of the detection result of the detection unit 2 to be stored in the storage unit 7. For example, the control unit 6 causes the accuracy calculation unit 3 to calculate accuracy information, and causes at least a part of the accuracy information calculated by the accuracy calculation unit 3 to be stored in the storage unit 7. For example, the control unit 6 causes the model calculation unit 4 to calculate model information, and causes at least a part of the model information calculated by the model calculation unit 4 to be stored in the storage unit 7. For example, the control unit 6 controls the communication unit 8 to cause the detection device 1 to transmit a command (control signal) or setting information. For example, the control unit 6 causes the information received by the communication unit 8 to be stored in the storage unit 7.

[0060] The storage unit 7 may be included in the control unit 6, or may be disposed outside the control unit 6. The storage unit 7 may be, for example, a non-volatile memory such as a USB memory or a memory card. The storage unit 7 may include a storage device built into the detection device 1, or may include a port to which a storage device that can be released from the detection device 1 can be connected. The storage unit 7 stores, for example, the detection result of the detection unit 2 (e.g., captured image data, depth information, point cloud data), the calculation result of the model calculation unit 4 (e.g., shape information, surface information, texture information), the calculation result of the accuracy calculation unit 3 (accuracy information), various setting information, and a program for executing processing.

[0061] The communication unit 8 transmits and receives various data (e.g., model information) to and from devices, equipment, information terminals, etc. outside the main body unit 10 wirelessly or by wire. The communication unit 8 includes at least one of an I / O port such as a USB port, and a communicator that performs wireless communication by radio waves or infrared rays. The communication unit 8 is controlled by the control unit 6, reads information stored in the storage unit 7, and transmits the read information to an external device. For example, the communication unit 8 transmits at least a part of the calculation result (e.g., accuracy information) of the accuracy calculation unit 3 and the calculation result (e.g., model information) of the model calculation unit 4 to an external device (e.g., an information processing device). The communication unit 8 also receives information including, for example, a command from an external device. The communication unit 8 can store the received information in the storage unit 7 and supply the received information to the control unit 6.

[0062] Moreover, the detection device 1 can output at least a part of the model information to a digital device that can input and output digital information such as a barcode or a two-dimensional code. Such a digital device can display or print digital information including at least a part of the model information on another display device (display device) or paper. A reader device equipped with a reader unit (e.g., an optical reader) that can read the displayed or printed digital information can store the digital information read by the reader unit in a storage area of ​​the device. The detection device 1 may also be configured to include the above-mentioned digital device and reader device. When the detection device 1 includes the above-mentioned digital device, the communication unit 8 may transmit at least a part of the information (e.g., model information) to the above-mentioned digital device. The above-mentioned digital device may also generate digital information based on the received model information and output the digital information to a medium such as paper.

[0063] When displaying the model information of the object OB on the display unit DP, the display control unit 9 emphasizes a portion of the model information corresponding to the first portion AR1. The display control unit 9 executes a rendering process using the model information calculated by the model calculation unit 4. The display control unit 9 includes a rendering processing unit. The display control unit 9 (rendering processing unit) includes, for example, a graphics processing unit (GPU). Note that the display control unit 9 (rendering processing unit) may be in a form in which a CPU and a memory execute each process according to an image processing program. The rendering processing unit executes, for example, at least one process of drawing processing, texture mapping processing, and shading processing.

[0064] In the drawing process, the display control unit 9 (rendering processing unit) can calculate, for example, an estimated image (e.g., a reconstructed image) of a shape defined in the shape information of the model information viewed from an arbitrary viewpoint position. In the following description, the shape indicated by the shape information is referred to as a model shape. For example, the display control unit 9 (rendering processing unit) can reconstruct a model shape (e.g., an estimated image) from the model information (e.g., shape information) by the drawing process. For example, the display control unit 9 (rendering processing unit) stores data of the calculated estimated image in the storage unit 7. In addition, in the texture mapping process, the display control unit 9 (rendering processing unit) can calculate an estimated image in which an image indicated by the texture information of the model information is pasted onto the surface of an object (e.g., object OB) on the estimated image.

[0065] The display control unit 9 (rendering processing unit) can calculate an estimated image in which a texture different from that of the target object is pasted onto the surface of the object in the estimated image. In the shading process, the display control unit 9 (rendering processing unit) can calculate an estimated image in which, for example, a shadow formed by a light source indicated by light source information in the model information is added to the object in the estimated image. In addition, in the shading process, the display control unit 9 (rendering processing unit) can calculate an estimated image in which, for example, a shadow formed by an arbitrary light source is added to the object in the estimated image.

[0066] The display control unit 9 executes a highlighting process for highlighting a portion of the model information that corresponds to the first portion AR1. Highlighting the first portion AR1 on the display unit DP includes displaying the portion so that the portion corresponding to the first portion AR1 can be specified when the user looks at the screen DPA of the display unit DP. When displaying the model information of the object OB on the display unit DP, the display control unit 9 highlights the portion of the model information that corresponds to the first portion AR1 by a specific display. The specific display may be, for example, at least one of the color, saturation, and brightness of the portion of the model information of the object OB that corresponds to the first portion AR1 is different from the portion that corresponds to the second portion AR2. In addition, the specific display may be changed over time when displaying the model information of the object OB on the display unit DP. For example, the display control unit 9 may blink the display that corresponds to the first portion AR1 as the specific display. In this way, in this embodiment, model information that can distinguish (or can be specified and visually recognized by the user) the portion that corresponds to the first portion AR1 from other portions (for example, the portion that corresponds to the second portion AR2) is generated and displayed.

[0067] 4 to 6 are diagrams showing display examples of the display unit DP according to the first embodiment. FIG. 4 shows an example displayed on the screen DPA of the display unit DP. As shown in FIG. 4, the model information of the object OB is displayed on the screen DPA of the display unit DP by a rendering process performed by the display control unit 9. At this time, the display control unit 9 performs a highlighting process as the specific display S by changing the color of the part corresponding to the first part AR1 from the color of the second part AR2, that is, by making the first part AR1 a highlighted display as seen by the user.

[0068] The display control unit 9 displays the first part AR1 in red or reddish color as the specific display S, for example, while displaying the second part AR2 in blue or bluish color. In this way, by performing the specific display in which the color of the first part AR1 is changed from that of the second part AR2, the first part AR1 is emphasized. That is, by highlighting the first part AR1, the user can easily confirm that the accuracy of the first part AR1 is lower than the standard (threshold value) and that the model information of the first part AR1 is uncertain. As a result, the user can easily recognize that it is necessary to obtain information on the object OB for the first part AR1. The color of the first part AR1 can be set arbitrarily. In addition, the display control unit 9 may display the second part AR2 in a light color and the first part AR1 in a dark color, or may display the second part AR2 in a dark color and the first part AR1 in a light color, for example, when displaying the model information in grayscale (or in a single color).

[0069] The display control unit 9 may increase (brighten) the luminance of the first portion AR1 relative to the second portion AR2 as the specific display S. Since the first portion AR1 is displayed brighter than the second portion AR2, the first portion AR1 is emphasized. The display control unit 9 may increase (brighten) the luminance of the second portion AR2 relative to the first portion AR1 as the specific display S. The display control unit 9 may increase (brighten) the luminance of the second portion AR2 relative to the first portion AR1 as the specific display S. The display control unit 9 may increase (brighten) the luminance of the second portion AR2 relative to the first portion AR1 as the specific display S. In other words, the display control unit 9 may perform a dark color process on the second portion AR2 while performing a light color process on the first portion AR1. In this case, the second portion AR2 is vivid (clear, vivid), whereas the first portion AR1 has a light color (dull), and the first portion AR1 is emphasized. The display control unit 9 may increase (brighten) the luminance of the first portion AR1 relative to the second portion AR2 as the specific display S.

[0070] The display control unit 9 may combine two or more of the above-mentioned specific processing. For example, the color of the first part AR1 may be changed with respect to the second part AR2, and the luminance of the first part AR1 and the luminance of the second part AR2 may be changed. The display control unit 9 may change the first part AR1 as the specific display S over time, or may change the second part AR2 as the specific display S over time. The change over time may be, for example, a form in which the first part AR1 blinks, a form in which the color of the first part AR1 is gradually changed, or a form in which the saturation of the color of the first part AR1 is changed, for example, by making the color lighter or darker. The display control unit 9 may continuously or intermittently display the specific display S or model information including the specific display S by performing the above-mentioned processing in real time, etc.

[0071] 5 shows another example displayed on the screen DPA of the display unit DP according to the first embodiment. As shown in FIG. 5, the screen DPA of the display unit DP displays model information of the object OB by rendering processing by the display control unit 9, as in FIG. 4. When the identification unit 5 sets the areas AR21 and AR22 for the second part AR2 based on the accuracy information and the second threshold, the display control unit 9 may change at least one of the color, saturation, and brightness of the areas AR21 and AR22. For example, when the area AR21 is a part with a higher accuracy than the area AR22, the display control unit 9 may make the color of the area AR21 darker or the brightness higher than the area AR22.

[0072] The user can easily recognize the part of the second part AR2 with high accuracy, which is higher than the standard accuracy information. The same applies to the case where the second part AR2 is divided into three or more parts. The display of the areas AR21 and AR22 shown in FIG. 5 may be selectively displayed by the user. For example, the display may be configured such that the user can operate an input unit (not shown) to select between a mode in which the areas AR21 and AR22 are displayed on the screen DPA of the display unit DP and a mode in which the areas AR21 and AR22 are not displayed. Alternatively, the display may be configured such that either one of the areas AR21 and AR22 is displayed on the screen DPA of the display unit DP by the user's operation.

[0073] FIG. 6 shows another example displayed on the screen DPA of the display unit DP according to the first embodiment. As shown in FIG. 6, the screen DPA of the display unit DP displays model information of the object OB by rendering processing by the display control unit 9, similar to FIG. 4. When the identification unit 5 calculates the ratio (proportion) between the first part AR1 and the second part AR2, the display control unit 9 may display a graph G indicating the model information and the accuracy ratio on the screen DPA of the display unit DP. That is, the model information and the ratio may be displayed on one screen. The form of the graph G is arbitrary, and may be another form (e.g., a bar graph) indicating the ratio between the first part AR1 and the second part AR2.

[0074] The user can easily recognize the ratio of the first part AR1 to the entire model information by displaying the model information and the graph G on the screen DPA of the display unit DP. That is, the user can easily confirm the extent to which the accuracy does not reach the standard (threshold) with respect to the entire model information. Note that in the graph G, the part corresponding to the second part AR2 may be displayed in the color used to display the second part AR2 of the model information, and the part corresponding to the first part AR1 may be displayed in the color used to display the first part AR1 of the model information. This display allows the user to intuitively grasp the comparison between the first part AR1 (or the second part AR2) of the model information and the first part AR1 (or the second part AR2) of the graph G.

[0075] The graph G shown in Fig. 6 may be selectively displayed by the user. For example, the user may operate an input unit (not shown) to select between a mode in which the graph G is displayed together with the model information on the screen DPA of the display unit DP and a mode in which the graph G is not displayed. The user may operate the screen DPA to switch from a mode in which the model information is displayed to a mode in which the graph G is displayed. When the areas AR21 and AR22 shown in Fig. 5 are displayed, the area of ​​the second part AR2 of the graph G may be divided into the area AR21 and the area AR22 and displayed.

[0076] Next, an example of a detection method according to an embodiment will be described based on the configuration of the detection device 1 described above. Fig. 7 and Fig. 8 are flowcharts showing an example of the detection method according to the embodiment. The detection method according to this embodiment includes detecting an object from one viewpoint, calculating model information of the object using the detection result, calculating accuracy information including the accuracy of each part of the object based on the detection result, identifying a first part of the accuracy information in one viewpoint that has a lower accuracy than a standard, and displaying model information in which a part of the model information corresponding to the first part is emphasized on a display unit.

[0077] As shown in FIG. 7, in step S1, the detection unit 2 detects an object OB from a predetermined viewpoint (single viewpoint). For example, the imaging unit 21 of the detection unit 2 images the object OB from a viewpoint Vp. In step S2, the model calculation unit 4 calculates model information of the object OB using the detection result of the detection unit 2. The model calculation unit 4 calculates, for example, at least one of shape information and texture information of the object OB when viewed from the viewpoint Vp. In addition, the model calculation unit 4 calculates surface information of the object OB using, for example, point cloud data.

[0078] In step S3, the accuracy calculation unit 3 calculates accuracy information including the accuracy of each part of the object OB based on the detection result of the detection unit 2. For example, the accuracy calculation unit 3 calculates point accuracy information (accuracy information, accuracy information of point cloud data) regarding the information of points included in the point cloud data using the point cloud data generated by the point cloud data generation unit 23. The accuracy calculation unit 3 may calculate line accuracy information or surface accuracy information. In step S4, the identification unit 5 identifies the first part AR1 having a low accuracy compared to a standard among the accuracy information in one viewpoint. For example, the identification unit 5 identifies the first part AR1 having a smaller accuracy than a preset threshold value (predetermined value) based on this threshold value. In addition, the identification unit 5 sets a plurality of regions (e.g., regions AR21, AR22) for the second part AR2 based on a second threshold value.

[0079] In step S5, the display control unit 9 displays the model information on the screen DPA of the display unit DP, in which the part of the model information corresponding to the first part AR1 is highlighted. For example, as shown in Fig. 4, when the display control unit 9 displays the model information on the screen DPA of the display unit DP, the display control unit 9 highlights the first part AR1 by a specific display such as changing the color of the part corresponding to the first part AR1 from the second part AR2. When the user looks at the model information on the screen DPA, the user can easily recognize the first part AR1, whose accuracy is lower than the reference value.

[0080] 8(A) and (B) are flowcharts showing another example of the detection method according to the embodiment. In FIG. 8(A) and (B), steps S1 to S5 are the same as those in the flowchart shown in FIG. 7, and therefore the description will be simplified or omitted. As shown in FIG. 8(A), after step S4, in step S6, the display control unit 9 determines whether or not to display the second part AR2 in a plurality of regions in the model information displayed on the screen DPA of the display unit DP. When the identification unit 5 sets the regions AR21 and AR22 for the second part AR2 based on the accuracy information and the second threshold, for example, the display control unit 9 may determine to display the regions AR21 and AR22. In addition, the display control unit 9 may determine to display the regions AR21 and AR22 when a display instruction for the regions AR21 and AR22 is received from the user.

[0081] If the display control unit 9 determines that the second part AR2 is to be displayed in a plurality of regions (YES in step S6), then in step S7, the display control unit 9 causes the second part AR2 to be displayed in, for example, two (a plurality of) regions AR21 and AR22 when displaying model information on the screen DPA of the display unit DP (see FIG. 5). The regions AR21 and AR22 are displayed on the screen DPA of the display unit DP with at least one of color, saturation, and brightness different from each other, for example. If the display control unit 9 determines that the second part AR2 is not to be displayed in a plurality of regions (NO in step S6), then in step S5, the display control unit 9 causes the model information to be displayed on the screen DPA of the display unit DP.

[0082] 8(B), after step S4, in step S8, the display control unit 9 determines whether or not to display the ratio graph G on the screen DPA of the display unit DP. When the identification unit 5 has calculated, for example, the ratio (proportion) of the first part AR1 and the second part AR2 to the entire model information, the display control unit 9 may determine to display the ratio graph G. In addition, the display control unit 9 may determine to display the ratio graph G when an instruction to display the ratio graph G is received from the user.

[0083] When the display control unit 9 determines that the ratio graph G is to be displayed (YES in step S8), in step S9, the display control unit 9 displays the ratio graph G together with the model information on the screen DPA of the display unit DP (see FIG. 6). When the display control unit 9 determines that the ratio graph G is not to be displayed (NO in step S8), in step S5, the display control unit 9 displays the model information on the screen DPA of the display unit DP. Note that the flowchart shown in FIG. 8(A) and the flowchart shown in FIG. 8(B) may be performed simultaneously. That is, when step S6 is YES and step S8 is YES, the display control unit 9 may display the second part AR2 in the areas AR21 and AR22 and display the ratio graph G when displaying the model information on the screen DPA of the display unit DP.

[0084] Thus, according to this embodiment, the first part AR1 of the model information, which has a lower accuracy than the standard, is highlighted and displayed, so that the user can easily confirm the position or range, etc., of the first part AR1, which is a part with a low accuracy (a part whose shape, etc. is uncertain). Furthermore, highlighting the first part AR1 can provide a suggestion for increasing the accuracy of the first part AR1. In other words, the user can easily recognize that it is possible to increase the accuracy of the first part AR1 (improve the accuracy of the model information as a whole) by separately (newly) acquiring information on the first part AR1.

[0085] In the above-described embodiment, the detection device 1 includes, for example, a computer system. The detection device 1 reads out a detection program stored in the storage unit 7, and executes various processes according to the detection program. The detection program causes the computer to execute, for example, the following: detect an object OB from a predetermined viewpoint Vp (single viewpoint); calculate model information of the object OB using the detection result of the detection unit 2; specify a first portion AR1 having a lower accuracy than a reference from the accuracy information at the viewpoint Vp; and display, on the display unit DP, model information in which a portion of the model information corresponding to the first portion AR1 is emphasized. The detection program may be provided by being recorded in a computer-readable storage medium.

[0086] [Second embodiment] A second embodiment will be described. FIG. 9 is a functional block diagram showing an example of a detection device according to the second embodiment. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. As shown in FIG. 9, the detection device 1A is, for example, an imaging device, and detects an object OB in a plurality of detection regions (for example, a field of view). The detection device 1A may be, for example, a fixed camera, a camera whose field of view can be changed manually or automatically, or a portable information terminal (for example, a smartphone, a tablet, a mobile phone with a camera). The detection device 1A performs a calculation process of information related to the object OB using the result of detecting the object OB. The detection device 1A models at least a part of the object OB by the calculation process of its own terminal, and calculates model information (model data). For example, the detection device 1A performs computer graphic processing (CG processing) on ​​at least a part of the object OB by the calculation process, and calculates model information (for example, CG model data).

[0087] The detection device 1A calculates model information indicating the object OB viewed from a plurality of viewpoints (e.g., different viewpoints, a plurality of directions), for example. In this embodiment, model information indicating the object OB viewed from two viewpoints Vp1 and Vp2 is calculated, but the present invention is not limited to this form, and model information indicating the object OB viewed from three or more viewpoints may be calculated. The detection device 1A includes, for example, detection units 2 and 2A, a probability calculation unit 3, a model calculation unit 4, an identification unit 5, a control unit 6, a storage unit 7, a communication unit 8, a display control unit 9, a main unit 10, and a model integration unit 30. The detection units 2 and 2A, the probability calculation unit 3, the model calculation unit 4, the identification unit 5, the control unit 6, the storage unit 7, the communication unit 8, the display control unit 9, and the model integration unit 30 are unitized by being provided, for example, in the main unit 10. The two detection units 2 and 2A may be provided in one main unit 10, or at least one of the detection units 2 and 2A may be separable from the main unit 10.

[0088] The detection unit 2 optically detects the object OB from a viewpoint Vp1 (one viewpoint). For example, the detection unit 2 captures an image of the object OB from the viewpoint Vp1 and outputs image data of the object OB as the detection result. The detection unit 2A optically detects the object OB from a viewpoint Vp2 (one viewpoint). For example, the detection unit 2 captures an image of the object OB from the viewpoint Vp1 and outputs image data of the object OB as the detection result. The viewpoints Vp1 and Vp2 are different viewpoints (positions). The detection unit 2 and the detection unit 2A may have the same configuration or different configurations.

[0089] FIG. 10 is a diagram showing an example of a detection method by the detection device 1A according to the second embodiment. The detection unit 2 optically detects the object OB from, for example, a viewpoint Vp1. The detection unit 2 acquires at least one of an image of the object OB viewed from the viewpoint Vp1 and a distance from the viewpoint Vp1 to each point on the object OB. The detection unit 2A optically detects the object OB from a viewpoint Vp2. The detection unit 2A acquires at least one of an image of the object OB viewed from the viewpoint Vp2 and a distance from the viewpoint Vp2 to each point on the object OB. Each of the detection units 2 and 2A includes, for example, an imaging unit 21 and a distance measuring unit 22. The arrangement of the viewpoints Vp1 and Vp2 is not limited to the arrangement shown in FIG. 10. For example, the angle between the direction in which the object OB is viewed from the viewpoint Vp1 and the direction in which the object OB is viewed from the viewpoint Vp2 may be smaller than 90° or larger than 90° (e.g., 180°).

[0090] The imaging unit 21 of the detection unit 2 captures an image of the object OB from the viewpoint Vp1, and outputs image data of the object OB as a detection result. The distance measurement unit 22 of the detection unit 2 detects the distance from the viewpoint Vp1 to each point on the object OB. The imaging unit 21 of the detection unit 2A captures an image of the object OB from the viewpoint Vp2, and outputs image data of the object OB as a detection result. The distance measurement unit 22 of the detection unit 2A detects the distance from the viewpoint Vp2 to each point on the object OB. Note that the detection units 2 and 2A do not necessarily include the imaging unit 21 or the distance measurement unit 22. The point cloud data generation unit 23 of the detection unit 2 generates point cloud data based on information acquired from the viewpoint Vp1. The point cloud data generation unit 23 of the detection unit 2A generates point cloud data based on information acquired from the viewpoint Vp2. The point cloud data generated by the detection unit 2A may be transmitted to the detection device 1A, for example, via the communication unit 8 (see FIG. 9), or may be sent to the detection device 1A via a portable recording medium such as a USB memory.

[0091] 10, when model information is calculated by the model calculation unit 4 (see FIG. 9) based on point cloud data created based on information acquired by the detection unit 2 from the viewpoint Vp1, the model information is created with a lower accuracy than the standard for the part (first part AR1) in the shadow of the object OB as viewed from the viewpoint Vp1. Also, when model information is calculated by the model calculation unit 4 based on point cloud data created based on information acquired by the detection unit 2A from the viewpoint Vp2, the model information is created with a lower accuracy than the standard for the part (first part AR11) in the shadow of the object OB as viewed from the viewpoint Vp2.

[0092] The first part AR1, whose accuracy is lower than the standard when viewed from the viewpoint Vp1, and the first part AR11, whose accuracy is lower than the standard when viewed from the viewpoint Vp2, are in a misaligned state because the viewpoints Vp1 and Vp2 are different. Therefore, the first part AR1 viewed from the viewpoint Vp1, excluding the first part AR11 viewed from the viewpoint Vp2, is included in the information acquired by the detection unit 2A from the viewpoint Vp2. Also, the first part AR11 viewed from the viewpoint Vp2, excluding the first part AR1 viewed from the viewpoint Vp1, is included in the information acquired by the detection unit 2 from the viewpoint Vp1. The part (first part AR12) that is in the shadow of the object OB when viewed from both the viewpoints Vp1 and Vp2 is not included in the information acquired by the detection units 2 and 2A.

[0093] The model integration unit 30 shown in FIG. 9 integrates a plurality of pieces of model information calculated based on information acquired from different viewpoints (e.g., viewpoints Vp1 and Vp2). The model integration unit 30 executes an integrated model calculation process (model integration process) that integrates a plurality of pieces of model information to generate integrated model information. The integrated model calculation process integrates, for example, first model information generated based on a result of detecting the object OB from the viewpoint Vp1 (first detection result) and second model information generated based on a result of detecting the object OB from the viewpoint Vp2 (second detection result) to generate integrated model information (integrated shape information). The model integration unit 30 may also acquire third model information generated from a viewpoint other than the detection units 2 and 2A and integrate one or both of the first model information and the second model information to calculate the integrated model information.

[0094] The model integration unit 30 integrates the first model information and the second model information, for example, by matching a first feature point of the shape indicated by the first model information with a second feature point of the shape indicated by the second model information. The first feature point is a part that can be distinguished from other parts of the shape indicated by the first model information. For example, a part defined as a face in the surface information can be distinguished from other faces with fewer bumps due to a shape with fine bumps. The second feature point is a part that can be distinguished from other parts of the shape indicated by the second model information.

[0095] Moreover, the model integration unit 30 detects a first feature point using at least one of the first model information and the first texture information associated with the first model information, and calculates first feature point data indicating the position of the first feature point and the feature amount (first feature amount) of the feature point. The model integration unit 30, for example, stores the first feature point data in the storage unit 7 as one item of information of the first model information. The model integration unit 30 also detects a second feature point using at least one of the second model information and the second texture information associated with the second model information, and calculates second feature point data indicating the position of the second feature point and the feature amount (second feature amount) of the feature point. The model integration unit 30, for example, stores the second feature point data in the storage unit 7 as one item of information of the second model information.

[0096] The model integration unit 30 calculates the feature points and the feature amounts by, for example, a Scale-Invariant Feature Transform (SIFT) method or a Speeded Up Robust Features (SURF) method. Note that the above feature point data (e.g., the first feature point data and the second feature point data) is data including at least the feature amounts of the feature points.

[0097] The model integration unit 30 calculates integrated model information including third feature data obtained by matching and integrating first feature data having a first feature amount of the first model information and second feature data having a second feature amount of the second model information. The third feature data is data obtained by matching the first feature data (e.g., first feature amount) and the second feature data (e.g., second feature amount). The third feature data includes, for example, a portion of the object OB where the first feature data and the second feature data overlap, and is data of a wider range than the first feature points and the second feature points. Note that the shape indicated by the integrated model information may be partial model information indicating a part of the object OB.

[0098] The model integration unit 30 may integrate the model information by, for example, weighting the first model information and the second model information (calculating a weighted average). The weighted average includes, for example, an arithmetic average, a geometric average, or an exponential average (exponentially weighted moving average). When there is an overlapping portion between the first model information and the second model information, the weighting coefficient may be determined based on, for example, the accuracy of the first model information in the overlapping portion and the accuracy of the second model information in the overlapping portion.

[0099] The accuracy calculation unit 3 (see FIG. 9) calculates accuracy information on the first model information (or the point cloud data on which the first model information is calculated) and accuracy information on the second model information (or the point cloud data on which the second model information is calculated) before calculating the integrated model information, and calculates the accuracy information on the integrated model information from these calculation results. The accuracy calculation unit 3 may calculate the accuracy information on the integrated model information after the model integration unit 30 calculates the integrated model information. When the shape information of the first model information obtained from the information from the viewpoint Vp1 and the shape information of the second model information obtained from the information from the viewpoint Vp2 partially overlap, the accuracy calculation unit 3 may select points with relatively high accuracy in the overlapping part (removing points with low accuracy) and use the result as the accuracy of each point in the overlapping part of the integrated model information.

[0100] Furthermore, when integrating a plurality of pieces of model information, the accuracy calculation unit 3 is not limited to selecting the higher accuracy obtained in each piece of model information as the accuracy of the overlapping portion. For example, when there is an overlapping portion in a plurality of pieces of model information, even if the accuracy of each point in the overlapping portion is low in each of the plurality of pieces of model information, if the direction of the normal of the surface indicated by the plurality of points in the overlapping portion of one piece of model information coincides with or is close to the direction of the normal of the surface indicated by the plurality of points in the overlapping portion of the other piece of model information, the accuracy of each point in the overlapping portion in the integrated model information may be set high.

[0101] That is, in an overlapping portion of a plurality of model information pieces obtained from a plurality of viewpoints of a single object OB, all points have low accuracy, but since the normal directions are the same or similar, it is possible to assume that the surface defined by one model information piece and the surface defined by the other model information piece in the overlapping portion are the same or nearly the same surface, and to set the accuracy of each point in the overlapping portion higher than the initial accuracy. As a result, the set accuracy becomes higher than the accuracy calculated for any of the plurality of model information pieces.

[0102] After the accuracy information is calculated for the integrated model information, the identification unit 5 (see FIG. 9) identifies the first part AR12, whose accuracy is lower than the standard, from the accuracy information for the integrated model information calculated by the accuracy calculation unit 3. The first part AR12 is an overlapping part of the first part AR1 and the first part AR11, as shown in FIG. 10. The first part AR12 is a region whose accuracy is lower than the standard (e.g., a threshold value, a predetermined value). The object OB includes the first part AR12 and a second part AR13 different from the first part AR12. The second part AR13 is a part or the whole of the part of the object OB excluding the first part AR12. The second part AR13 is a region having an accuracy higher than the accuracy of each point of the first part AR12. The second part AR13 is a region having an accuracy higher than the standard. The identification unit 5 may also identify the part excluding the part having an accuracy higher than the standard (e.g., a threshold value, a predetermined value) as the first part AR12.

[0103] When the display control unit 9 displays the integrated model information of the object OB on the display unit DP, the display control unit 9 emphasizes a portion of the integrated model information that corresponds to the first portion AR12. When the display control unit 9 displays the integrated model information of the object OB on the display unit DP, the display control unit 9 may display the integrated model information and the pre-integration model information on the single screen DPA of the display unit DP. When the display control unit 9 displays the pre-integration model information on the display unit DP, the display control unit 9 may display one of the multiple pre-integration model information on the single screen DPA, or may display multiple pre-integration model information on the single screen DPA. When the display control unit 9 displays the pre-integration model information, the display control unit 9 may display a viewpoint (for example, a viewpoint Vp1 in the case of displaying the above-mentioned first model information) on the screen DPA in association with the displayed pre-integration model information.

[0104] In addition, when the display control unit 9 displays the integrated model information of the object OB on the display unit DP, it is not necessary to display the model information before integration within the single screen DPA. Furthermore, when the display control unit 9 receives an instruction to switch the display from a user from an input device (not shown), for example, the display control unit 9 may switch between the integrated model information and the model information before integration and display them on the display unit DP. Furthermore, when the display control unit 9 displays the integrated model information and the model information before integration, the display control unit 9 may display them side by side with the same or almost the same dimensions within the single screen DPA of the display unit DP. The display control unit 9 is capable of performing rendering processing on each of the integrated model information and the model information.

[0105] FIG. 11 is a diagram showing a display example of the display unit DP according to the second embodiment. FIG. 11 shows an example of displaying integrated model information and one of the pieces of model information before integration in one screen DPA of the display unit DP. As shown in FIG. 11, the integrated model information and the model information before integration are displayed side by side on one screen DPA of the display unit DP by a rendering process by the display control unit 9. At this time, the display control unit 9 performs a highlighting process as a specific display S by changing the color of the first part AR1 in the model information before integration and the part corresponding to the first part AR12 in the integrated model information from the color of the second parts AR2 and AR13, that is, by making each of the first parts AR1 and AR12 into a highlighted display as seen by the user.

[0106] The specific display S for the first part AR1 and the specific display S for the first part AR12 may be the same form (e.g., color, brightness, blinking, etc.) or different forms. By looking at the integrated model information and the model information before integration, the user can easily confirm that the area with a lower accuracy than the standard (threshold) has changed from the first part AR1 to the first part AR12 (e.g., has shrunk or decreased). Note that it is optional whether or not the detection device 1A includes a display control unit 9. The detection device 1A does not need to include the display control unit 9.

[0107] Next, an example of a detection method according to the embodiment will be described based on the configuration of the above-mentioned detection device 1A. FIG. 12 is a flowchart showing an example of a detection method according to the embodiment. As shown in FIG. 12, in step S11, the detection unit 2 detects an object OB from a plurality of viewpoints (e.g., viewpoints Vp1 and Vp2). For example, the imaging unit 21 of the detection unit 2 images the object OB from the viewpoint Vp1, and the imaging unit 21 of the detection unit 2A images the object OB from the viewpoint Vp2. In step S12, the model calculation unit 4 calculates model information of the object OB using the detection results of the detection units 2 and 2A. The model calculation unit 4 calculates, for example, at least one of shape information and texture information of the object OB when viewed from the viewpoint Vp1, and at least one of shape information and texture information of the object OB when viewed from the viewpoint Vp2. In addition, the model calculation unit 4 calculates surface information of the object OB using, for example, point cloud data.

[0108] In step S13, the model integration unit 30 integrates multiple pieces of model information. For example, the model integration unit 30 integrates model information obtained from multiple viewpoints (e.g., viewpoints Vp1 and Vp2) to calculate integrated model information. In step S14, the accuracy calculation unit 3 calculates accuracy information including the accuracy of each part of the object OB based on the integrated model information. In step S15, the identification unit 5 identifies a first part AR12 whose accuracy is lower than a standard among the accuracy information related to the integrated model information. In addition, the identification unit 5 may set multiple areas (e.g., areas AR21 and AR22) for the second part AR13 based on a second threshold value.

[0109] In step S16, the display control unit 9 judges whether or not to display model information that displays model information before integration within the single screen DPA when displaying the integrated model information. The display control unit 9 may judge to display, within the single screen DPA, the model information that was calculated first among the multiple pieces of model information before integration together with the integrated model information. Furthermore, when the display control unit 9 receives an instruction to display the model information before integration from the user, it may judge to display the model information before integration within the single screen DPA together with the integrated model information. Furthermore, the model information before integration to be displayed within the single screen DPA may be specified by the user.

[0110] When the display control unit 9 determines that the model information before integration is to be displayed in the single screen DPA together with the integrated model information (YES in step S16), in step S17, the display control unit 9 displays the integrated model information and the model information before integration in the single screen DPA of the display unit DP (see FIG. 11). The display control unit 9 displays the integrated model information and the model information before integration side by side in the single screen DPA with the same or almost the same dimensions. The user can easily compare the integrated model information on the screen DPA with the model information before integration, and can easily confirm that the area with the accuracy lower than the standard (threshold) has changed from the first part AR1 to the first part AR12 (e.g., has been reduced or decreased). In addition, when the display control unit 9 determines that the model information before integration is not to be displayed in the single screen DPA together with the integrated model information (NO in step S16), in step S18, the display control unit 9 displays the integrated model information on the screen DPA of the display unit DP.

[0111] [Third embodiment] A third embodiment will be described. FIG. 13 is a functional block diagram showing an example of a detection device according to the third embodiment. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. As shown in FIG. 13, the detection device 1B is, for example, an imaging device, and detects an object OB in a plurality of detection regions (for example, a field of view). The detection device 1B may be, for example, a fixed camera, a camera whose field of view can be changed manually or automatically, or a portable information terminal (for example, a smartphone, a tablet, a mobile phone with a camera). The detection device 1B performs a calculation process of information on the object OB using the result of detecting the object OB. The detection device 1B models at least a part of the object OB by the calculation process of its own terminal, and calculates model information (model data). For example, the detection device 1B performs computer graphic processing (CG processing) on ​​at least a part of the object OB by the calculation process, and calculates model information (for example, CG model data).

[0112] The detection device 1B includes, for example, a detection unit 2, an accuracy calculation unit 3, a model calculation unit 4, an identification unit 5, a control unit 6, a storage unit 7, a communication unit 8, a display control unit 9, a main body unit 10, an instruction generation unit 40, an instruction generation unit 41, and an orientation sensor 42. The detection unit 2, the accuracy calculation unit 3, the model calculation unit 4, the identification unit 5, the control unit 6, the storage unit 7, the communication unit 8, the display control unit 9, the instruction generation unit 40, the orientation calculation unit 41, and the orientation sensor 42 are, for example, provided in the main body unit 10 to be unitized. The instruction generation unit 40 generates instruction information for instructing the user on the next detection of the object OB by the detection unit 2 based on, for example, the first part AR1 identified by the identification unit. The instruction information includes detection conditions related to the next detection of the object OB by the detection unit 2 (for example, detection direction, detection count, detection time, detection target in the object OB, environmental conditions such as a light source, etc.).

[0113] The instruction generating unit 40 executes an instruction generating process for generating instruction information for instructing the user on the next detection of the object OB by the detection unit 2 based on the position information. The instruction information generated by the instruction generating process includes, for example, a detection direction from a predetermined viewpoint (predetermined position) different from one viewpoint with respect to the first part AR1 of the object OB. The predetermined viewpoint is, for example, a viewpoint from which the first part AR1 can be seen, and includes a detection direction (for example, a data acquisition direction, a shooting direction) that eliminates a low-precision area (for example, the first part AR1) acquired at the one viewpoint. The predetermined viewpoint may be a viewpoint from which a part of the second part AR2 can be seen together with the first part AR1. The detection direction may be in a form indicated by an arrow on the screen DPA of the display unit DP, or in a form indicated by a straight line, a dotted line, or the like.

[0114] FIG. 14 is a diagram showing an example of a detection method by the detection device 1B according to the third embodiment. When the detection device 1B acquires information of the object OB from a viewpoint Vp1, the model information (point cloud data) includes the first part AR1 with a degree of accuracy higher than a reference (threshold value). Therefore, as shown in FIG. 14, by acquiring information of the object OB from a viewpoint Vp3, it is possible to increase the accuracy of the first part AR1. The viewpoint Vp3 is a predetermined viewpoint different from the viewpoint Vp1, which is one viewpoint. The viewpoint Vp3 can be set to any viewpoint that can see all or part of the first part AR1.

[0115] In FIG. 14, the information of the object OB (object OB including the first part AR1) is acquired from the viewpoint Vp3 using the detection device 1B, but the present invention is not limited to this. For example, the information of the object OB including the first part AR1 may be acquired from the viewpoint Vp3 by a detection device other than the detection device 1B, and the information may be acquired by the detection device 1B via the communication unit 8 (see FIG. 13). Alternatively, the detection unit 2 of the detection device 1B may be detached from the main body unit 10, and the information of the object OB may be acquired from the viewpoint Vp3 by the detection unit 2. The instruction generating unit 40 generates instruction information that instructs the user to perform the next detection, for example, to detect the object OB from a viewpoint (e.g., viewpoint Vp3) where the first part AR1 can be seen, or from a detection direction including the viewpoint Vp3.

[0116] The direction calculation unit 41 calculates the detection direction as teaching information using the detection result of the detection unit 2. For example, the direction calculation unit 41 may calculate the direction of the normal line in the first part AR1 as the detection direction as teaching information using depth information obtained from the detection result of the distance measurement unit 22 (see FIG. 13) included in the detection unit 2. The direction calculation unit 41 may also calculate the direction of the normal line in the first part AR1 as the detection direction as teaching information using point cloud data of the object OB generated by the point cloud data generation unit 23 using the depth information obtained from the detection result of the distance measurement unit 22.

[0117] FIG. 15 is a diagram showing an example of calculating the direction of a normal line by the direction calculation unit 41 according to the third embodiment. The direction calculation unit 41 selects planes PL1 to PLn including at least three points from among the points in the first part AR1, for example, using one or both of the depth information and the point cloud data. The direction calculation unit 41 calculates the directions VE1 to VEn (directions VE1 to VE7 in FIG. 15) of the normal line in each plane (planes PL1 to PL7 in FIG. 15). The three points (or four or more points) selected in the first part AR1 are selected, for example, by a combination of three points that are the shortest distance from a certain point. The direction calculation unit 41 selects, for example, two (or three or more) points in ascending order of distance from a certain point to form a set of three points (or four or more points), and selects a plane (for example, planes PL1 to PL7) defined by the points of this set.

[0118] The direction calculation unit 41 may select a surface (e.g., surfaces PL1 to PL7) for the entire first part AR1, or may select a surface for a portion of the first part AR1. The portion of the first part AR1 may be determined by the direction calculation unit 41 or may be specified by the user. When the direction calculation unit 41 determines a portion of the first part AR1, for example, the first part AR1 may be divided into a plurality of areas (e.g., three or four), and a surface may be selected and a normal direction may be calculated for the area farthest from the second part AR2 among the divided areas.

[0119] The direction calculation unit 41 calculates the most common normal direction VE among the multiple normal directions (e.g., directions VE1 to VE7) on each surface (e.g., surfaces PL1 to PL7). The direction calculation unit 41 may use all normal directions to calculate the most common normal direction VE, or may use some normal directions to calculate the most common normal direction VE. The direction calculation unit 41 may also calculate a direction other than the most common normal direction (a direction other than the direction VE) among the multiple normal directions on each surface. The direction calculation unit 41 may also calculate a direction obtained by averaging the multiple normal directions. The direction calculation unit 41 calculates the normal direction VE as a detection direction as teaching information. The detection direction includes a predetermined viewpoint (e.g., viewpoint Vp3) different from one viewpoint (e.g., viewpoint Vp1). The direction calculation unit 41 may calculate multiple detection directions as teaching information. It is optional whether the detection device 1B includes the direction calculation unit 41 or not. The detection device 1B does not necessarily have to include the direction calculation unit 41.

[0120] When the display control unit 9 causes the display unit DP to display the model information of the object OB, the display control unit 9 causes the display unit DP to display the teaching information generated by the teaching generation unit 40 together with the model information on the screen DPA of the display unit DP. When the display control unit 9 causes the display unit DP to display the model information of the object OB, the display control unit 9 may not display the teaching information. In addition, when the display control unit 9 receives an instruction to switch the display from a user from an input device (not shown), for example, the display control unit 9 may display the teaching information on the display unit DP together with the model information. In addition, the display control unit 9 is capable of performing a rendering process on each of the model information and the teaching information, for example. The display control unit 9 may cause the display unit DP to display the detection direction calculated by the direction calculation unit 41 as the teaching information. The display control unit 9 may cause the display unit DP to display an arrow, a line, or the like as the detection direction.

[0121] FIG. 16 is a diagram showing a display example of the display unit DP according to the third embodiment. FIG. 16 shows an example of displaying model information and teaching information T within a screen DPA of the display unit DP. As shown in FIG. 16, the model information and an arrow T1 indicating the detection direction as teaching information T are displayed on the screen DPA of the display unit DP by a rendering process performed by the display control unit 9. The display control unit 9 displays the arrow T1 so that it is directed in the direction in which the first part AR1 is viewed. The arrow T1 is displayed in association with the model information. Therefore, when the orientation (posture) of the model information changes, the display control unit 9 changes the display position and orientation of the arrow T1 to match the orientation (posture) of the model information.

[0122] The arrow T1 instructs the user on the next detection by the detection device 1B. The user can easily confirm the direction to be detected next by looking at the instruction information T (arrow T1). The arrow T1 may be displayed in an emphasized manner with respect to the model information, for example, by blinking or adjusting the brightness. In addition, when the direction calculation unit 41 calculates a plurality of detection directions as instruction information, the display control unit 9 may display a plurality of arrows on the display unit DP. In this case, numbers (e.g., recommended numbers) may be displayed in order from the most recommended arrow as a detection direction among the plurality of arrows.

[0123] The orientation sensor 42 (see FIG. 13) detects the orientation of the detection device 1B. The orientation sensor 42 can detect the detection direction by the detection unit 2. Therefore, the display control unit 9 (or the control unit 6) can calculate the relationship between the detection direction by the detection unit 2 and the detection direction that is the teaching information T by using the detection result of the orientation sensor 42. The display control unit 9 may also cause the display unit DP to display the detection direction by the detection unit 2 by the orientation sensor 42. The detection result of the orientation sensor 42 is included in the teaching information. When the detection result of the orientation sensor 42 shows that the detection direction by the detection unit 2 and the detection direction that is the teaching information T match or almost match, the display control unit 9 (or the control unit 6) may display on the display unit DP a display indicating that the two match or almost match (for example, a character display such as OK or a symbol display such as a circle) or may output a sound. The user can easily recognize that the detection direction by the detection unit 2 and the detection direction that is the teaching information T match or almost match with each other by this display, sound, or other signal.

[0124] The detection direction of the detection unit 2 detected by the orientation sensor 42 may be displayed on the display unit DP by, for example, an arrow, a line, or the like. The user can align the detection direction of the detection unit 2 with the detection direction that is the teaching information T while looking at the detection direction of the detection unit 2 detected by the orientation sensor 42 and the detection direction that is the teaching information T on the display unit DP. That is, by displaying the detection direction of the detection unit 2 detected by the orientation sensor 42 on the display unit DP, the user can be guided as to the next detection direction by the detection device 1B (detection unit 2). In addition, it is optional whether or not the detection device 1B includes the orientation sensor 42. The detection device 1B does not need to include the orientation sensor 42.

[0125] Moreover, the teaching information generated by the teaching generation unit 40 may be teaching model information that is model information when a predetermined viewpoint (e.g., viewpoint Vp3) is used. The teaching generation unit 40 may generate teaching model information when a predetermined viewpoint (e.g., viewpoint Vp3) is used, or may generate teaching model information using all or part of model information generated elsewhere (e.g., generated by the model calculation unit 4). The display control unit 9 displays the teaching model information generated by the teaching generation unit 401 on the screen DPA of the display unit DP. Moreover, the teaching generation unit 40 may display a silhouette of the object OB as the teaching model information. For example, the user may detect the object OB from a predetermined viewpoint that overlaps at least a part of the teaching model information. The display control unit 9 may display the model information and the teaching model information simultaneously or by switching between them on one screen DPA of the display unit DP.

[0126] FIG. 17 is a diagram showing a display example of the display unit DP according to the third embodiment. FIG. 17 shows an example of displaying model information and teaching model information T2, which is teaching information T, in one screen DPA of the display unit DP. As shown in FIG. 17, the model information and teaching model information T2 are displayed as teaching information T on the screen DPA of the display unit DP by a rendering process by the display control unit 9. The teaching model information T2 is a silhouette of the object OB seen from a predetermined viewpoint (e.g., viewpoint Vp3) from which the first part AR1 can be seen, unlike one viewpoint (e.g., viewpoint Vp1). The teaching model information T2 does not have to be a silhouette of the object OB. The teaching model information T2 may be model information of the object OB seen from a predetermined viewpoint (e.g., viewpoint Vp3).

[0127] Furthermore, when the display control unit 9 displays the teaching model information T2, the display control unit 9 may display a viewpoint (e.g., viewpoint Vp3) for viewing the first part AR1 in the screen DPA in association with the teaching model information T2. ​​When the display control unit 9 displays the model information of the object OB on the display unit DP, the display control unit 9 may not display the teaching model information T2 on the single screen DPA. When the display control unit 9 receives an instruction to switch the display from a user via an input device (not shown), for example, the display control unit 9 may display the model information and the teaching model information T2 on the single screen DPA, or may switch between the model information and the teaching model information T2 and display them on the screen DPA. When the display control unit 9 displays the model information and the teaching model information T2, the display control unit 9 may display them side by side with the same or almost the same dimensions on the single screen DPA of the display unit DP.

[0128] The display control unit 9 can perform rendering processing on each of the model information and the teaching model information T2. ​​The display control unit 9 may emphasize the display of the teaching model information T2 relative to the model information. For example, the display control unit 9 may change the display form (e.g., color, brightness, blinking, etc.) of the teaching model information T2 relative to the model information to emphasize the teaching model information T2. ​​By looking at the teaching model information T2, the user can easily confirm the next detection direction (the direction to be detected next).

[0129] FIG. 18 is a diagram showing a display example of the display unit DP according to the third embodiment. FIG. 18 shows an example of displaying model information and assumed model information T3, which is teaching information T, in one screen DPA of the display unit DP. As shown in FIG. 18, the display control unit 9 displays model information and assumed model information T3 as teaching information T on the screen DPA of the display unit DP by rendering processing by the display control unit 9. The assumed model information T3 is model information assumed when integrating model information based on information from one viewpoint (e.g., viewpoint Vp1) and model information calculated from information from a predetermined viewpoint (e.g., viewpoint Vp3) from which the first part AR1 can be expected. The assumed model information T3 may be generated by the teaching generation unit 40 or may be generated elsewhere (e.g., generated by the model calculation unit 4).

[0130] Furthermore, when the display control unit 9 displays the assumed model information T3, the display unit DP may display the first part AR15 and the second part AR16, which are assumed to have a smaller accuracy than the reference (threshold), in association with the assumed model information T3. The first part AR15 is smaller than the first part AR1. When the display control unit 9 displays the model information of the object OB on the display unit DP, the display control unit 9 may not display the assumed model information T3 within the single screen DPA. When the display control unit 9 receives an instruction to switch the display from a user from an input device (not shown), for example, the display control unit 9 may display the model information and the assumed model information T3 on the single screen DPA, or may switch between the model information and the assumed model information T3 and display them on the screen DPA. When the display control unit 9 displays the model information and the assumed model information T3, the display control unit 9 may display them side by side with the same or almost the same dimensions within the single screen DPA of the display unit DP.

[0131] The assumed model information T3 may be model information assumed when information is acquired from three or more viewpoints. The viewpoint for generating the assumed model information T3 may be set by the instruction generating unit 40 or may be set by the user. When the instruction generating unit 40 sets the viewpoint, a viewpoint (e.g., a viewpoint on the opposite side of the viewpoint Vp1) away from the viewpoint (e.g., viewpoint Vp1) from which the model information is acquired first may be set. The display control unit 9 may also emphasize the display of the assumed model information T3 with respect to the model information. For example, the display control unit 9 may change the display form (e.g., color, brightness, blinking, etc.) of the assumed model information T3 with respect to the model information to emphasize the assumed model information T3. The user can easily confirm that the accuracy of the model information is increased by looking at the assumed model information T3. It is optional whether the detection device 1B includes the display control unit 9. The detection device 1B does not need to include the display control unit 9.

[0132] Next, an example of a detection method according to the embodiment will be described based on the configuration of the above-mentioned detection device 1B. The detection method according to the present embodiment includes detecting an object from one viewpoint, calculating model information of the object using the detection result, calculating accuracy information including the accuracy of each part of the object based on the detection result, identifying a first part of the accuracy information in one viewpoint that has a lower accuracy than a reference, and generating instruction information for instructing a user on the next detection of the object based on the first part. FIG. 19 is a flowchart showing an example of a detection method according to the embodiment. Note that in the flowchart of FIG. 19, the same reference numerals are used for the same processes as those in the flowchart of FIG. 7, and the description thereof is omitted or simplified.

[0133] As shown in FIG. 19, in step S1, the detection unit 2 detects an object OB from a viewpoint Vp1 which is a predetermined viewpoint (single viewpoint). In step S2, the model calculation unit 4 calculates model information of the object OB using the detection result of the detection unit 2. In step S3, the accuracy calculation unit 3 calculates accuracy information including the accuracy of each part of the object OB based on the detection result of the detection unit 2. In step S4, the identification unit 5 identifies a first part AR1 whose accuracy is lower than a standard (threshold value) from the accuracy information at one viewpoint (e.g., viewpoint Vp1).

[0134] In step S20, the instruction generating unit 40 generates instruction information T (e.g., an arrow T1, instruction model information T2, and assumed model information T3) for instructing the user on the next detection of the object OB based on the first part AR1 identified by the identifying unit 5. The instruction information T generated by the instruction generating unit 40 may be displayed on the display unit DP by the display control unit 9, or may be transmitted to the outside of the detection device 1B and displayed together with the model information of the object OB on the display unit of another device. For example, the user can easily confirm the direction to be detected next by looking at the instruction information T on the screen DPA.

[0135] In this way, according to the present embodiment, the teaching information T is generated based on the first portion AR1 of the model information, the accuracy of which is smaller than the reference value, so that the user can be easily instructed on the next detection of the object OB. That is, the user can easily confirm the direction to be detected next with respect to the object OB, for example, by viewing the teaching information T associated with the model information of the object OB on the screen DPA of the display unit DP. In addition, the detection instructed by the teaching information T is performed to calculate the model information, and the model information is integrated with the previously acquired model information, so that the accuracy of the model information as a whole can be improved.

[0136] In the above-described embodiment, the detection device 1B includes, for example, a computer system. The detection device 1B reads out a detection program stored in the storage unit 7 and executes various processes according to the detection program. The detection program causes the computer to execute, for example, the following: detect an object OB from a predetermined viewpoint Vp (single viewpoint); calculate model information of the object OB using the detection result of the detection unit 2; specify a first portion AR1 of the accuracy information at the viewpoint Vp, the accuracy of which is lower than a reference; and generate instruction information T for instructing a user on the next detection of the object OB based on the first portion AR1. The detection program may be provided by being recorded in a computer-readable storage medium.

[0137] [Fourth embodiment] A fourth embodiment will be described. FIG. 20 is a functional block diagram showing an example of an information processing device according to the fourth embodiment. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. As shown in FIG. 20, the information processing device 100 includes a probability calculation unit 3, a model calculation unit 4, an identification unit 5, a control unit 6, a storage unit 7, and a communication unit 8. For example, information on an object OB is sent to the information processing device 100 from a detection unit 200. Note that the information processing device 100 does not need to include at least one of the control unit 6, the storage unit 7, and the communication unit 8.

[0138] The detection unit 200 includes an imaging unit 21, a distance measurement unit 22, and a point cloud data generation unit 23. The detection unit 200 is, for example, an imaging device, and detects an object OB within a plurality of detection regions (for example, a field of view). The detection unit 200 may be, for example, a fixed camera, a camera whose field of view can be changed manually or automatically, or a portable information terminal (for example, a smartphone, a tablet, a mobile phone with a camera). The detection unit 200 may have the same configuration as the detection unit 2 in the above-mentioned embodiment, or may have a different configuration from the detection unit 2.

[0139] The information processing device 100 receives, as information on the object OB, depth information detected by the distance measuring unit 22 from the detection unit 200, or point cloud data generated by the point cloud data generating unit 23 from this depth information, via the communication unit 8, for example, wirelessly or wired. The accuracy calculation unit 3 of the information processing device 100 calculates accuracy information from the transmitted information on the object OB. The model calculation unit 4 calculates model information of the object OB from the transmitted information on the object OB. The identification unit 5 identifies a first part AR1 whose accuracy is smaller than a reference (threshold value, predetermined value) from the accuracy information calculated by the accuracy calculation unit 3.

[0140] The information processing device 100 is capable of outputting model information of the object OB and information relating to the first part AR1 identified by the identification unit 5. A display control unit 9 provided outside the information processing device 100 performs, for example, rendering processing on the model information and information relating to the first part AR1 output from the information processing device 100 and displays them on the display unit DP. The display control unit 9 displays the model information of the object OB on the display unit DP and highlights the part corresponding to the first part AR1. The point cloud data generation unit 23 provided in the detection unit 200 may be provided in the information processing device 100. Furthermore, one or both of the display control unit 9 and the display unit DP may be provided in the information processing device 100.

[0141] Thus, according to this embodiment, a first part AR1 of the model information whose certainty is less than a standard is identified, so that a user can easily confirm the position, range, etc. of the first part AR1, which is a part with a low certainty (a part whose shape, etc. is uncertain), by viewing the information output from the information processing device 100, for example, on the display unit DP.

[0142] [Fifth embodiment] A fifth embodiment will be described. FIG. 21 is a functional block diagram showing an example of an information processing device according to the fifth embodiment. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. As shown in FIG. 21, the information processing device 100A includes a probability calculation unit 3, a model calculation unit 4, a specification unit 5, a control unit 6, a storage unit 7, a communication unit 8, and an instruction generation unit 40. For example, information on an object OB is sent from the detection unit 200 to the information processing device 100A. Note that the information processing device 100A does not need to include at least one of the control unit 6, the storage unit 7, and the communication unit 8.

[0143] The information processing device 100A receives, as information on the object OB, depth information detected by the distance measuring unit 22 from the detection unit 200, or point cloud data generated by the point cloud data generating unit 23 from the depth information, for example wirelessly or via wired communication via the communication unit 8. The accuracy calculation unit 3 of the information processing device 100A calculates accuracy information from the information on the object OB sent. The model calculation unit 4 calculates model information of the object OB from the information on the object OB sent. The identification unit 5 identifies a first portion AR1 having an accuracy smaller than a reference (threshold, predetermined value) from the accuracy information calculated by the accuracy calculation unit 3. The instruction generation unit 40 generates instruction information for instructing the user on the next detection of the object OB by the detection unit 200 based on the first portion AR1 of the object OB.

[0144] The information processing device 100A can output model information of the object OB and teaching information generated by the teaching generation unit 40. A display control unit 9 provided outside the information processing device 100A performs, for example, rendering processing on the model information and teaching information output from the information processing device 100A and displays them on the display unit DP. The display control unit 9 displays the model information and teaching information of the object OB on the display unit DP. Note that the point cloud data generation unit 23 provided in the detection unit 200 may be provided in the information processing device 100A. Also, one or both of the display control unit 9 and the display unit DP may be provided in the information processing device 100A.

[0145] Thus, according to this embodiment, since instruction information is generated by the instruction generation unit 40, the user can easily confirm the next detection of the object OB by the detection unit 200 by viewing the information output from the information processing device 100A, for example, on the display unit DP.

[0146] [Sixth embodiment] A sixth embodiment will be described. FIG. 22 is a functional block diagram showing an example of an information processing device according to the sixth embodiment. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. As shown in FIG. 22, the information processing device 100B includes a probability calculation unit 3, a model calculation unit 4, a specification unit 5, a control unit 6, a storage unit 7, a communication unit 8, and a movement calculation unit 50. For example, the information processing device 100B receives information on the object OB from the detection unit 200. Note that the information processing device 100B does not need to include at least one of the model calculation unit 4, the control unit 6, the storage unit 7, and the communication unit 8.

[0147] The information processing device 100B receives, as information on the object OB, depth information detected by the distance measuring unit 22 from the detection unit 200, or point cloud data generated by the point cloud data generating unit 23 from this depth information, via the communication unit 8, for example, wirelessly or wired. The accuracy calculation unit 3 of the information processing device 100B calculates accuracy information from the transmitted information on the object OB. The model calculation unit 4 calculates model information on the object OB from the transmitted information on the object OB. The identification unit 5 identifies a first part AR1 whose accuracy is smaller than a reference (threshold value, predetermined value) from the accuracy information calculated by the accuracy calculation unit 3.

[0148] The movement calculation unit 50 calculates the movement amount or movement position of the detection unit 200 in the next detection of the object OB by the detection unit 200 based on the first part AR1 of the object OB. The movement calculation unit 50 calculates the movement amount (e.g., movement distance, movement angle) or movement position (e.g., coordinate value) of the detection unit 200 to move in order to detect the object OB from a predetermined viewpoint different from one viewpoint. The predetermined viewpoint is, for example, a viewpoint from which the first part AR1 can be expected. The predetermined viewpoint may be a viewpoint from which a part of the second part AR2 can be expected together with the first part AR1. The information processing device 100B according to this embodiment may be configured to include, for example, the direction calculation unit 41 according to the above-mentioned embodiment. The movement calculation unit 50 may calculate the movement amount or movement position of the detection unit 200 based on the direction calculated by the direction calculation unit 41.

[0149] The movement calculation unit 50 executes a movement calculation process for calculating movement information related to the movement amount or movement position of the detection unit 200 in the next detection of the object OB by the detection unit 200. The movement information calculated by the movement calculation process is sent to the movement device 60.

[0150] The moving device 60 moves the object OB and the detection unit 200 relatively in a predetermined moving direction based on the movement information. The moving device 60 moves the object OB and the detection unit 200 relatively by, for example, rotating the object OB. Also, the moving device 60 moves the object OB and the detection unit 200 relatively by, for example, moving the detection unit 200 around the object OB. The moving device 60 may be configured to move either one of the object OB and the detection unit 200, or may be configured to move both the object OB and the detection unit 200. The moving device 60 includes a drive unit 61 and a drive control unit 62 that controls the drive unit 61. Based on the movement information calculated by the movement calculation unit 50, the drive control unit 62 drives the drive unit 61 to relatively move the object OB and the detection unit 200, and disposes the detection unit 200, for example, at a viewpoint from which the first part AR1 can be seen (for example, a position where the first part AR1 is within the field of view of the detection unit 200). That is, the detection unit 200 is disposed by the movement device 60, for example, at an optimal position (optimum viewpoint) in the next detection.

[0151] FIG. 23 shows an example of the moving device 60, where (A) shows a state before movement and (B) shows a state after movement. In the example shown in FIG. 23, the moving device 60 is a turntable 60A. As shown in FIG. 23(A), an object OB is placed on the turntable 60A. The turntable 60A is rotatable around an axis AX1 in the vertical direction. When the turntable 60A rotates, the object OB and the detection unit 200 move relatively. The turntable 60A is rotated by a drive unit (see drive unit 61 in FIG. 22), not shown. The amount of rotation or the rotation position of the turntable 60A is controlled by a drive control unit (see drive control unit 62 in FIG. 22), not shown. As shown in FIG. 23(B), the turntable 60A rotates based on the amount of rotation (movement information) calculated by the movement calculation unit 50, or rotates to the rotation position (movement information) calculated by the movement calculation unit 50, to position the detection unit 200 at a position that views the first part AR1 of the object OB.

[0152] The drive control unit of the turntable 60A notifies the information processing device 100B that it has rotated to a position based on the movement information (i.e., that the detection unit 200 has been placed at a position that sees the first portion AR1 of the object OB), for example. Based on this notification, the information processing device 100B instructs the detection unit 200 to detect the object OB. The detection unit 200 can detect a part or the whole of the first portion AR1 of the object OB by detecting the object OB according to an instruction from the information processing device 100B. Note that the turntable 60A is not limited to rotating the object OB. For example, the detection unit 200 may be held on the turntable 60A and rotate around the object OB.

[0153] FIG. 24 shows another example of the moving device 60, where (A) shows a state before movement and (B) shows a state after movement. In the example shown in FIG. 24, the moving device 60 is a robot arm 60B. As shown in FIG. 24(A), the robot arm 60B includes a holding unit HL that holds the detection unit 200, and a plurality of arms AM connected to the holding unit HL. The robot arm 60B can move the detection unit 200 held by the holding unit HL, for example, in a direction of six degrees of freedom. By moving the detection unit 200 by the robot arm 60B, the object OB and the detection unit 200 move relatively. The robot arm 60B moves the detection unit 200 held by the holding unit HL by a drive unit (see drive unit 61 in FIG. 22) that drives the holding unit HL and the arm AM. The movement amount or movement position of the holding unit HL (detection unit 200) is controlled by a drive control unit (see drive control unit 62 in FIG. 22) not shown. As shown in Figure 24 (B), the robot arm 60B moves the holding unit HL based on the movement amount (movement information) calculated by the movement calculation unit 50, or moves the holding unit HL to the movement position (movement information) calculated by the movement calculation unit 50, and positions the detection unit 200 at a position that views the first part AR1 of the object OB.

[0154] The drive control unit of the robot arm 60B notifies the information processing device 100B that the holding unit HL has moved to a position based on the movement information (i.e., the detection unit 200 has been placed in a position that sees the first part AR1 of the object OB), for example. Based on this notification, the information processing device 100B instructs the detection unit 200 to detect the object OB. The detection unit 200 can detect a part or the whole of the first part AR1 of the object OB by detecting the object OB according to an instruction from the information processing device 100B. Note that the robot arm 60B is not limited to the holding unit HL holding and moving the object OB. For example, the object OB may be held by the holding unit HL and moved relative to the detection unit 200.

[0155] The information processing device 100B can output model information of the object OB. The display control unit 9 provided outside the information processing device 100B performs, for example, rendering processing on the model information output from the information processing device 100B and displays it on the display unit DP. The display control unit 9 displays the model information of the object OB on the display unit DP. The point cloud data generation unit 23 provided in the detection unit 200 may be provided in the information processing device 100B. One or both of the display control unit 9 and the display unit DP may be provided in the information processing device 100B. The information processing device 100B may be integrated with the detection unit 200 by being housed in one housing, for example. The form in which the detection unit 200 and the information processing device 100 are integrated is a detection device.

[0156] As described above, according to this embodiment, since movement information is generated by the movement calculation unit 50, by driving the movement device 60 based on this movement information, for example, the detection unit 200 can be easily placed at an appropriate position (appropriate viewpoint) for the next detection. Therefore, the user does not need to move the detection unit 200 for the next detection, and detection of the object OB from multiple viewpoints can be automated.

[0157] [Seventh embodiment] A seventh embodiment will be described. In this embodiment, the same components as those in the above-mentioned embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 25 is a diagram showing an example of a detection system SYS according to this embodiment. The detection system SYS includes detection devices 1, 1A, and 1B, and a processing device 300 that processes information output from the detection devices 1, 1A, and 1B. The processing device 300 is provided with, for example, an input device 301 and a display device 302. The detection system SYS may include one or more of the detection devices 1, 1A, and 1B, or may include the same multiple detection devices (e.g., detection device 1).

[0158] The processing device 300 acquires information from the detection device 1 etc. through communication with the detection devices 1, 1A, 1B. The processing device 300 executes a rendering process using, for example, the information acquired from the detection device 1 etc. (e.g., model information, accuracy information, information related to the first part AR1, teaching information). For example, the processing device 300 calculates data of an estimated image of the object OB viewed from a viewpoint based on setting information of the viewpoint input by a user to the input device 301. The processing device 300, for example, sends the data of the estimated image to the display device 302 and causes the display device 302 to display the estimated image.

[0159] The input device 301 includes at least one of, for example, a keyboard, a mouse, a touch panel, a sensor such as an acceleration sensor, a voice input device, a touch-type pen, etc. The input device 301 is connected to the processing device 300. The input device 301 accepts information input, for example, from a user, and supplies the input information to the processing device 300. The display device 302 includes, for example, a liquid crystal display or a touch-panel display, and is connected to the processing device 300. The display device 302 displays an image (for example, an estimated image by rendering processing) based on image data supplied from the processing device 300, for example.

[0160] As shown in FIG. 25, the processing device 300 includes a communication unit 303, a storage unit 304, a rendering processing unit 305, and a control unit 306. The communication unit 303 includes at least one of a USB port, a network card, and a communication device that performs wireless communication using radio waves or infrared rays. The communication unit 303 can communicate with the communication units 8 of the detection devices 1, 1A, and 1B. The storage unit 304 includes a removable storage medium such as a USB memory, and a large-capacity storage device such as an external or built-in hard disk. The storage unit 304 stores, for example, at least a part of the data of the information received via the communication unit 303, an imaging control program for controlling the detection devices 1, 1A, and 1B, a processing program for executing each process of the processing device 300, and the like.

[0161] The rendering processing unit 305 includes, for example, a graphics processing unit (GPU). The rendering processing unit 305 may be configured such that a CPU and a memory execute each process according to an image processing program. The rendering processing unit 305 executes at least one process of, for example, a drawing process, a texture mapping process, and a shading process. The control unit 306 controls, for example, each unit of the processing device 300, the detection devices 1, 1A, and 1B, the input device 301, and the display device 302. The control unit 306 controls, for example, the communication unit 303 to transmit commands (control signals) and setting information to the detection devices 1, 1A, and 1B. The control unit 306 stores, for example, information received by the communication unit 303 from the detection devices 1, 1A, and 1B in the storage unit 304. The control unit 306 controls, for example, the rendering processing unit 305 to execute a rendering process.

[0162] The detection system SYS may not include the input device 301. For example, the detection system SYS may be configured such that various commands and information are input via the communication unit 303. The detection system SYS may not include the display device 302. For example, the detection system SYS may output data of an estimated image generated by a rendering process to an external display device, which may display the estimated image. The processing device 300 may also include a model integration unit. This model integration unit integrates multiple pieces of model information (shape information) sent from the detection devices 1, 1A, and 1B.

[0163] Although the embodiments have been described above, the technical scope of the present invention is not limited to the aspects described in the above-mentioned embodiments. One or more of the requirements described in the above-mentioned embodiments may be omitted. In addition, the requirements described in the above-mentioned embodiments may be combined as appropriate. In addition, to the extent permitted by law, the disclosures of all documents cited in this specification are incorporated by reference and made part of the description in this text. [Explanation of symbols]

[0164] REFERENCE SIGNS LIST 1, 1A, 1B... detection device, 2, 2A... detection section, 3... accuracy calculation section, 4... model calculation section, 5... identification section, 9... display control section, 21... imaging section, 22... ranging section, 23... point cloud data generation section, 30... model integration section, 40... teaching generation section, 41... direction calculation section, 42... orientation sensor, 100, 100A, 100B... information processing device, 300... processing device, OB... object, AR1... first part, AR2... second part, DP... display section, SYS... detection system, T... teaching information, T1... arrow, T2... teaching model information, Vp1, Vp2, Vp3... viewpoint, VE... normal

Claims

1. A detection unit that detects a distance from a viewpoint to a target object; a model calculation unit that calculates model information of the object using a detection result from the detection unit; an accuracy calculation unit that calculates accuracy information including an accuracy for each part of the object based on a detection result of the detection unit; an identification unit that identifies a first portion of the certainty information in the one viewpoint, the first portion having a low certainty relative to a reference; an instruction generating unit configured to generate instruction information for instructing a user on a next detection of the object by the detection unit based on the first portion, The teaching information includes a detection direction, A detection device, wherein the detection direction is determined based on a direction of a normal to the first portion calculated using the detection result of the detection unit.

2. The detection device according to claim 1 , further comprising a display control unit that causes the teaching information generated by the teaching generation unit to be displayed on a display unit together with the model information.

3. The detection device according to claim 2 , wherein the display control unit causes the display unit to display the detected direction.

4. The detection device according to claim 2 or 3, wherein the detection direction is indicated by an arrow on the display unit.

5. a point cloud data generation unit that generates point cloud data of the object using a detection result of the detection unit, The detection device according to claim 1 , wherein the direction of the normal in the first portion is calculated using the point cloud data.

6. The detection device according to claim 1 , wherein the detection direction is determined based on directions of a plurality of the normals calculated for a plurality of portions included in the first portion.

7. The detection device according to claim 5 , wherein the detection direction is a direction of the normal line that is most prevalent among the plurality of normal lines.

8. The teaching information is teaching model information, which is the model information when viewed from a predetermined viewpoint, The detection device according to claim 2 , wherein the display control unit causes the display unit to display the teaching model information.

9. The detection device according to claim 8 , wherein the predetermined viewpoint is a viewpoint from which the first portion can be seen.

10. The detection device according to claim 8 or 9, wherein the teaching model information is a silhouette of the object.

11. The detection device according to claim 8 , wherein the display control unit causes the model information and the teaching model information to be displayed within one screen of the display unit.

12. The detection device according to claim 1 , wherein the teaching information includes at least one of the detection direction of the object, a detection time, and a light source used for detection.

13. a model calculation unit that calculates model information of an object obtained by detecting the object from one viewpoint; an accuracy calculation unit that calculates accuracy information including an accuracy for each part of the object based on a result of detecting the object from the one viewpoint; an identification unit that identifies a first portion of the certainty information in the one viewpoint, the first portion having a low certainty relative to a reference; an instruction generating unit that generates instruction information for instructing a user on a next detection of the object based on the first portion, The teaching information includes a detection direction, An information processing device, wherein the detection direction is determined based on a direction of a normal to the first portion calculated using a result of detecting the object from the one viewpoint.

14. On the computer, Detecting an object from a single viewpoint; Calculating model information of the object using a result of detecting the object; Calculating accuracy information including an accuracy for each part of the object based on a result of detecting the object; Identifying a first portion of the certainty information in the one viewpoint, the certainty being lower than a reference; generating instruction information for instructing a user on a next detection of the object based on the first portion; The teaching information includes a detection direction, A detection program, wherein the detection direction is determined based on a direction of a normal in the first portion calculated using a result of detecting the object.

15. A detection device according to any one of claims 1 to 12; A detection system comprising: a processing device that processes information output from the detection device.

Images