Object attitude estimation apparatus and object attitude estimation method

Abstract

To provide an object attitude estimation apparatus and an object attitude estimation method for accurately estimating an attitude of an object even if an object included in an input image is small or low in resolution.SOLUTION: An attitude estimation apparatus 1 which estimates an attitude of an object from image data is configured to: acquire a depth image; acquire a normal image from the depth image; generate a super-resolution image of the object clipped out from the depth image; extract features of the object included in the image data, the super-resolution image, and the normal image; generate a corrected image obtained by restoring the shape of the object based on the features extracted from the super-resolution image and the normal image; estimate a parameter related to a state of the object based on the corrected image; estimate a position of the object from the image data; and estimate the attitude of the object based on the parameter related to the state of the object and the position of the object.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for analyzing an object contained in video data and image data to estimate its pose. [Background technology]

[0002] In recent years, advanced technologies such as artificial intelligence (AI) and information and communications technology (ICT) have been utilized to put into practical use technologies that efficiently perform processes such as learning, estimation, recognition, and judgment. One of these learning methods implemented by AI is machine learning. In machine learning, a machine (computer) learns on its own using a large amount of data provided, and based on the learning results (trained model), optimizes output data for input data and performs estimations and predictions based on the output data. Such machine learning includes processing technologies such as convolutional neural networks (CNNs), which perform convolutional processing based on input data. Machine learning is used in a variety of fields, including image recognition, speech recognition, natural language processing, and machine translation.

[0003] A technology for estimating the pose of an object using machine learning as described above is described in Patent Document 1. Patent Document 1 discloses an object pose estimation and matching system that aims to accurately estimate and match the pose of an object image regardless of conditions such as the pose and brightness of the object. The system in Patent Document 1 generates multiple pose candidates based on an input image. Based on the multiple pose candidates, a 3D object model is projected onto a 2D image, and multiple comparison images with lighting and other conditions similar to those of the input image are generated. A feature (sharpness feature) reflecting the sharpness of each generated comparison image is extracted. Then, a difference (weighted difference) is calculated for each comparison image by weighting the sharpness feature to the difference between the input image and the multiple comparison images. Then, based on the multiple weighted differences, a comparison image that is closest to the input image is selected from the multiple comparison images, and an optimal (closest) pose is estimated from the selected comparison image. The system in Patent Document 1 takes into account sharpness features that reflect the sharpness of the comparison image, which means that if the posture or object does not match between the 3D object model and the input image, large discrepancies are likely to occur, thereby improving the accuracy of posture estimation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4692773 Summary of the Invention [Problem to be solved by the invention]

[0005] The system of Patent Document 1 estimates the pose based on an input image to generate multiple comparison images, and estimates the pose of an object based on the feature values ​​of the comparison images and the degree of difference between the input image and the comparison image. However, when a camera is installed at a position far from the object or when the target object is small, the object contained in the input image may be small or the resolution of the object may be low. In such cases, the feature values ​​obtained from the input image may be reduced, and the shape of the object, such as its unevenness, may be erroneously estimated. Therefore, even when taking into account the sharpness, the system of Patent Document 1 may not be able to estimate the pose of the object with the desired accuracy depending on the size and resolution of the object in the input image.

[0006] The present invention has been made in light of the above technical problems, and aims to provide an object posture estimation device and posture estimation method that are capable of accurately estimating the posture of an object even when the object included in the input image is small or the resolution of the object is low. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention provides an object posture estimation device that analyzes image data of an object using a convolutional neural network and estimates the posture of the object contained in the image data, the device comprising: a depth data acquisition unit that acquires a depth image containing depth data, which is data relating to the depth of the object contained in the image data; a normal data acquisition unit that acquires a normal image by determining normal vectors of the surface of the object from the depth image; a super-resolution image generation unit that generates a super-resolution image of the object cut out from the depth image with an increased resolution; and a method for estimating the posture of the object by analyzing the image data, the super-resolution image, and the normal image. a corrected image generation unit that generates a corrected image in which the shape of the object is restored based on the feature amounts extracted from the super-resolution image and the normal image; a state estimation unit that estimates parameters related to the state of the object based on the corrected image; a position estimation unit that estimates the position of the object from the image data; and a posture estimation unit that estimates the posture of the object based on the parameters related to the state of the object estimated by the state estimation unit and the position of the object estimated by the position estimation unit.

[0008] In addition, the posture estimation device of the present invention may further include an optimization unit that performs fitting of output data of the position of the object estimated by the position estimation unit and the posture of the object estimated by the posture estimation unit to actual data regarding the position and posture of the object by a Gauss-Newton method.

[0009] Furthermore, the corrected image generation unit in the posture estimation device of the present invention may be configured to generate a composite feature by combining the feature of the object extracted from the depth image and the feature of the object extracted from the normal image in the channel direction, and to restore the object by performing an upsampling process that interpolates missing parts using surrounding copy values ​​based on the composite feature, thereby generating the corrected image.

[0010] On the other hand, the present invention is a method for estimating the posture of an object, which analyzes image data of an object using a convolutional neural network and estimates the posture of the object contained in the image data, the method comprising the steps of: acquiring a depth image including depth data, which is data related to the depth of the object contained in the image data; acquiring a normal image from the depth image by calculating a normal vector of the surface of the object; generating a super-resolution image of the object cut out from the depth image with increased resolution; extracting feature amounts that quantitatively represent the features of the object contained in the image data, the super-resolution image, and the normal image; generating a corrected image that restores the shape of the object based on the feature amounts extracted from the super-resolution image and the normal image; estimating parameters related to the state of the object based on the corrected image; estimating the position of the object from the image data; and estimating the posture of the object based on the estimated parameters related to the state of the object and the estimated position of the object.

[0011] Furthermore, the posture estimation method of the present invention may be a method of fitting output data relating to the estimated position and posture of the object and actual data relating to the position and posture of the object using a Gauss-Newton method.

[0012] Furthermore, the posture estimation method of the present invention may be a method of generating a composite feature by combining the feature of the object extracted from the depth image and the feature of the object extracted from the normal image in the channel direction, and restoring the object by performing an upsampling process that interpolates missing parts using surrounding copy values ​​based on the composite feature, thereby generating the corrected image.

[0013] Furthermore, an object posture estimation system that analyzes image data of an object using a convolutional neural network and estimates the posture of the object contained in the image data includes a camera that acquires a depth image including depth data that is data related to the depth of the object, and a posture estimation device that estimates the posture of the object, wherein the posture estimation device includes a normal data acquisition unit that acquires a normal image by determining normal vectors of the surface of the object from the depth image, a super-resolution image generation unit that generates a super-resolution image with a higher resolution of the object cut out from the depth image, and a processing unit that performs processing on the image data, the super-resolution image, and the normal image. a corrected image generation unit that generates a corrected image in which the shape of the object is restored based on the feature amounts extracted from the super-resolution image and the normal image; a state estimation unit that estimates parameters related to the state of the object based on the corrected image; a position estimation unit that estimates the position of the object from the image data; and a posture estimation unit that estimates the posture of the object based on the parameters related to the state of the object estimated by the state estimation unit and the position of the object estimated by the position estimation unit. [Effects of the Invention]

[0014] In an object posture estimation device and posture estimation method according to an embodiment of the present invention, a depth image and a normal vector image (normal image) of an object are generated based on image data of the input object captured from a bird's-eye view. A super-resolution image is generated from the generated depth image by performing super-resolution processing to increase the resolution of the object included in the image data. Then, feature quantities of the object are extracted from each of the super-resolution image and the normal image. A corrected image in which the shape of the object is restored based on the extracted feature quantities is generated. Then, data related to the shape of the object itself, such as the object's size, label (classification), rotation amount (rotation angle), and thickness, is estimated from the object feature quantities extracted from the corrected image, and the result is output. Furthermore, data related to the object's position is estimated from the input image data of the object captured from a bird's-eye view. The posture of the object is then estimated based on the data related to the object's shape and the data related to the object's position estimated in this way. That is, the shape of the object is restored with high accuracy using a super-resolution image obtained by super-resolutioning input image data (depth image) and a normal image of the object generated from the super-resolution image, so that the shape of the object can be estimated with high accuracy. Furthermore, the orientation of the object is estimated together with the position of the object estimated from image data obtained by photographing the object from a bird's-eye view, so that the orientation of the object can be estimated with high accuracy.

[0015] For example, if an object is unavoidably placed far from the camera due to the location of the camera, the captured image data may contain a small image of the object. In such cases, the resolution of the object may be low, resulting in fewer object features, or the distance between the positions where the features are extracted may be so long that the object's shape is recognized as a plane, making it impossible to obtain sufficient features for estimating the object's pose from the image data. Even in such cases, the pose estimation device or pose estimation method according to the present embodiment can estimate the object's pose with high accuracy.

[0016] Furthermore, as described above, the accuracy of the data such as the estimated shape and position of the object is high, so by having the learning model learn based on that data, learning by the learning model can be performed effectively and the output result, i.e., the estimation accuracy of the object's posture, can be further improved. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is an explanatory diagram for explaining a state in which video data to be analyzed by a posture estimation device according to an embodiment of the present invention is being acquired; [Figure 2] 1 is an explanatory diagram illustrating the overall configuration of a posture estimation device according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating a functional configuration of a posture estimation device according to an embodiment of the present invention. [Figure 4] 4A and 4B are diagrams showing images of an object output by processing image data performed by the functional configuration of the posture estimation device shown in FIG. 3, in which (a) is a diagram showing an image of a segment image in which an object is detected or extracted from the image data, (b) is a diagram showing an image in which the area of ​​the object detected from the segment image is converted into a rectangle, and (c) is a diagram showing an image of a masked image in which parts other than the segment area of ​​the object are masked. [Figure 5] 4A and 4B are diagrams showing images of objects output by processing image data performed by the functional configuration of the posture estimation device shown in FIG. 3, where (a) is a diagram showing an image of a cut-out image obtained by cutting out only the object included in the masking image, (b) is a diagram showing an image of a super-resolution image obtained by super-resolving the object included in the cut-out image, and (c) is a diagram showing an image of a normal image representing the surface orientation of the object included in the cut-out image. [Figure 6]4A and 4B are diagrams showing an image of an object output by processing image data performed by the functional configuration of the posture estimation device shown in FIG. 3, where (a) is a diagram showing an image of an object input to a convolutional neural network to extract features of the object included in the super-resolution image, (b) is a diagram showing an image of an object input to a convolutional neural network to extract features of the object included in the normal image, and (c) is a diagram showing an image of a corrected image generated based on the extracted features. [Figure 7] 1 is a flowchart (process chart) showing an example of the contents and steps of arithmetic processing (control) used by the posture estimation device and posture estimation method according to an embodiment of the present invention. [Figure 8] 8 is a flowchart showing an example of the contents and steps of the calculation process used by the posture estimation device and posture estimation method according to the embodiment of the present invention, and is a flowchart for explaining the processing and steps subsequent to the flowchart shown in FIG. 7. [Figure 9] 9 is a flowchart showing an example of the contents and steps of the calculation processing used by the posture estimation device and posture estimation method according to an embodiment of the present invention, and is a flowchart for explaining the processing and steps subsequent to the flowcharts shown in FIGS. 7 and 8. DETAILED DESCRIPTION OF THE INVENTION

[0018] Next, the present invention will be described based on the embodiments shown in the drawings. Note that the embodiments described below are merely examples of specific embodiments of the present invention, and are not intended to limit the present invention.

[0019] In an object pose estimation device and method according to an embodiment of the present invention, a convolutional neural network is used to perform a convolution operation based on a large amount of collected information (object data) to extract features (feature amounts, feature points) of the information source. For example, feature amounts are extracted from image data (object data) of an object such as a workpiece or a transported object, and the position and orientation of the object are estimated based on the feature amounts. In an object pose estimation device and method according to an embodiment of the present invention, a convolutional neural network is used to estimate the orientation of an object (workpiece) contained in image data captured by an imaging device. For example, the orientation of an object such as a tool placed at a work site is estimated.

[0020] 1 and 2, the posture estimation device 1 estimates the posture of an object by analyzing image data captured by a camera 2. In the example shown in FIG. 1, the camera 2 is installed to capture an overhead image of the entire work site 3 or an object (workpiece) 4, etc., and as a result, the object 4 is located at a position relatively far from the camera 2.

[0021] Camera 2 is an imaging device that captures an image of a predetermined area in work site 3 from above. Camera 2 is fixed to, for example, the ceiling of work site 3, and is positioned so as to capture an image of the entire work site 3 from a bird's-eye view. Camera 2 may be configured similarly to conventionally known cameras 2, and is configured to be able to detect or estimate the depth or depth of object 4 from the captured image data.

[0022] For example, camera 2 may be a stereo camera having two cameras and capable of calculating the distance to object 4 based on the parallax based on image data captured by each camera, the distance between the cameras, and the focal length of each camera. Alternatively, camera 2 may be a monocular camera capable of sensing depth. For example, camera 2 may be configured with a camera that moves camera 2 and detects the depth of object 4 based on image data before and after the movement and the movement difference, a camera that detects the distance to object 4 based on the time between irradiating light onto object 4 and detecting the reflected light, or a camera that irradiates object 4 with a predetermined light pattern such as a striped pattern and detects the distance to object 4 based on the distortion of the predetermined light pattern.

[0023] The posture estimation device 1 mainly comprises a processor, a communication unit, and a storage unit. The posture estimation device 1 is configured to perform calculations according to a predetermined program using data acquired from an external source and pre-stored data, and to output the results of the calculations as control command signals. For example, the posture estimation device 1 executes functions that meet a predetermined purpose by having the processor load a program stored on a recording medium into a working area of ​​the storage unit and execute the program, and by performing various controls through the execution of the program.

[0024] The processor is, for example, a CPU or a DSP. This processor is configured to control the posture estimation device 1 and perform various information processing operations. The processor may also include a GPU (Graphics Processing Unit) capable of high-speed image processing. The storage unit includes, for example, a RAM and a ROM. As described above, the storage unit has a working area for the processor to execute programs. The storage unit also includes an auxiliary storage unit such as an EPROM or a hard disk drive. This auxiliary storage unit may also include a portable recording medium, i.e., a removable medium. The auxiliary storage unit freely stores various programs, various data, and various tables in the recording medium by reading or writing them. The auxiliary storage unit may also store an operating system. The communication unit is a wireless communication circuit connected to an external communication device or the above-mentioned camera 2 via wireless communication so as to be able to communicate data. The posture estimation device 1 is configured to perform machine learning processing using a neural network using appropriate elements from the above-mentioned components, such as a detection unit, a calculation unit, and a learning unit.

[0025] Next, the configuration of pose estimation device 1 according to an embodiment of the present invention will be described with reference to Figures 2 and 3. As described above, pose estimation device 1 is able to estimate the pose of object 4 contained in image data by analyzing the image data. Figure 2 is a conceptual diagram showing the overall configuration of a learning model or architecture for estimating the pose of object 4, and Figure 3 is a block diagram showing the functional configuration.

[0026] 2, the posture estimation device 1 has, as functions for estimating the posture of the object 4, a first processing unit 5 that estimates the characteristics of the object 4 itself, such as the size of the object 4 and the type of work (label) the object 4 is, and a second processing unit 6 that estimates the position of the object 4 relative to the camera 2. The posture estimation device 1 is configured to estimate the posture of the object 4 based on the characteristics of the object 4 itself and the position of the object 4 estimated by the first processing unit 5 and the second processing unit 6.

[0027] Next, specific details of the first processing unit 5 and the second processing unit 6 and a specific functional configuration for estimating the orientation of the object 4 will be described using Fig. 3. As shown in Fig. 3, the orientation estimation device 1 includes an image acquisition unit 7, an image information preprocessing unit 8, a super-resolution processing unit 9, a normal image generation unit 10, a corrected image generation unit 11, an object information estimation unit 12, an object position estimation unit 13, an object orientation estimation unit 14, and an output unit 15. Note that a file containing data on the position and orientation of the object 4 has been input or learned in advance into the orientation estimation device 1 or the learning model. Also, images generated by the processing executed by this functional configuration or diagrams showing the image of the object 4 output are shown in Figs. 4, 5, and 6.

[0028] The image acquisition unit 7 acquires image data or video data captured by the camera 2. When acquiring image data, the image acquisition unit 7 may be configured so that any image data is input by the user, or may be configured so that the image data is automatically acquired from the camera 2.

[0029] The image information preprocessing unit 8 performs processing to obtain more information about the object 4 from the acquired image data and to make it easier to extract data about the object 4. For example, the image information preprocessing unit 8 performs preprocessing such as normalizing the image data, adjusting the size, and cutting out the object 4. The normalization processing performed here is a preprocessing of the image data, and is a process of adjusting the distribution of the data by adjusting the mean and variance by scaling the pixel values ​​of the image to a predetermined range.

[0030] Specifically, the image information preprocessing unit 8 first acquires depth information of the object 4 based on the image data and information from the camera 2. The image information preprocessing unit 8 generates a depth image of the object 4 based on the depth and perspective of the object 4 detected by the camera 2. Although not shown, the depth image is an image that can acquire data related to the three-dimensional spatial coordinates of the object 4 relative to the camera 2 by outputting the depth image by assigning colors to distance values ​​or by changing the color shading. The image included in the first processing unit 5 shown in FIG. 2 is an example of a depth image.

[0031] By pre-training depth images for training, it is possible to generate depth images based on input image data, and random noise is added to the generated depth images. Random noise is added to ensure the robustness and consistency of the image data, and training with such noise applied to the depth images can improve the predictive performance of the training model for data that it has not learned before. The mechanisms for randomly adding noise to such image data and normalizing the image data are publicly available and provided (shared) via an API (Application Programming Interface).

[0032] The image information preprocessing unit 8 identifies the objects 4 included in the depth image thus generated, and performs segmentation to label each object 4 on a pixel-by-pixel basis. Through segmentation, segment areas are generated in which the areas occupied by the objects 4 in the image data are identified. If the image data includes multiple objects in addition to the object 4 whose pose is to be estimated, segment areas are generated for all of the objects.

[0033] The image information preprocessing unit 8 then converts the generated segment area of ​​each object into a rectangle. For example, the image information preprocessing unit 8 converts the segment area of ​​each object into a rectangle by generating a rectangular area (bounding box) for each object. Note that the objects identified at this time may be only those objects 4 that are recognized as objects 4 (workpieces) in the label list set in the learning model, that is, objects 4 that are to be identified or learned by the learning model.

[0034] The image information preprocessing unit 8 then masks and cuts out the segmented image in which a segment area has been added to each object. That is, it masks out the other objects, leaving only the object 4 whose pose has been estimated and to which a segment area has been added. Then, it generates a cut-out image by cropping only the object 4 to which the segment area has been added from the masked image in which everything other than the object 4 has been masked.

[0035] The super-resolution processor 9 uses a convolutional neural network to super-resolution the clipped image generated by the image information preprocessor 8. Super-resolution is a process performed to restore the resolution of image data to a higher resolution, and is performed by a convolutional neural network such as SRCNN (Super-Resolution Convolutional Neural Network) that can convert such image data into high-resolution image data.

[0036] SRCNN is constructed by combining a convolutional layer and a ReLU activation function (ReLU function). In SRCNN, an object 4 included in image data is enlarged by bicubic interpolation, which interpolates a pixel by taking a weighted average of the pixel values ​​of 16 pixels surrounding a pixel of interest. A high-resolution image is then generated by a neural network composed of a convolutional layer and a ReLU function. SRCNN uses mean squared error as its loss function. The super-resolution processor 9 corresponds to the super-resolution image generator in this embodiment of the present invention.

[0037] The normal image generation unit 10 generates a normal vector image (normal image) from the super-resolution image, which is a super-resolved depth image generated by the super-resolution processing unit 9. The normal image is an image that shows the orientation of the surface of the object 4 by adding a normal vector perpendicular to the surface of the object 4. The normal image generation unit 10 generates a normal image in which a normal vector for representing the contour of the object 4 is added to the super-resolution image. Note that a mechanism for obtaining such normal vectors of the surface of the object 4 uses a publicly available API. The normal image generation unit 10 corresponds to a normal data acquisition unit in the embodiment of the present invention.

[0038] The corrected image generation unit 11 generates a corrected image based on the super-resolution image and normal image generated by the super-resolution processing unit 9 and normal image generation unit 10, respectively. The corrected image generation unit 11 first extracts feature quantities of the object 4 from each of the super-resolution image and normal image. The feature quantities are values ​​that quantitatively represent the qualitative features of the object 4 contained in each of the super-resolution image and normal image. The feature quantities are extracted by using deep learning to extract elements for identifying the object 4, such as acquiring data on brightness distribution, edges, and color appearance rates. The feature quantities are extracted by generating a feature map using a learning model. It is preferable to previously train the learning model on additional depth images for learning that include the object 4 whose pose is to be estimated. By extracting feature quantities from the super-resolution image, data related mainly to the distance (dimensions) of the object 4 is extracted. By extracting feature quantities from the normal image, data related to the surface and contour of the object 4 is extracted.

[0039] In the corrected image generation unit 11, the super-resolution image and the normal image are input to a convolutional neural network for image classification, whereby feature extraction is performed. An example of the convolutional neural network for image classification is a densely connected convolutional network (DenseNet).

[0040] DenseNet is characterized by a repeated densely connected structure and convolutional layers. DenseNet has an architecture called Dense Block, which is a convolutional block that inputs a feature map that concatenates all of the outputs of the layer preceding each layer. DenseNet has convolutional layers and pooling layers between the Dense Blocks. It also has a Transition-Layer, which is a layer for compressing (downsampling) the number of channels that has increased due to the Dense Block. In other words, DenseNet reduces information loss between layers by directly inputting the feature map that extracts the features of object 4 generated in each layer to the next layer, thereby alleviating the so-called vanishing gradient problem and enabling enhanced deep feature transmission and efficient feature utilization.

[0041] The corrected image generation unit 11 then generates a corrected image in which the object 4 has been corrected based on the feature amounts extracted or generated by inputting each of the super-resolution image and the normal image into DenseNet. In other words, the corrected image generation unit 11 generates a corrected image in which the features of the object 4 are more clearly expressed based on the feature amounts extracted by the image classification convolutional neural network.

[0042] Specifically, the corrected image generation unit 11 combines, in the channel direction, the feature amounts of the object 4 extracted from the super-resolution image generated by the super-resolution processing unit 9 and the feature amounts of the object 4 extracted from the normal image generated by the normal image generation unit 10. As an example of such combining, first, a feature map spatially indicating the feature amounts based on the features of the object 4 extracted from the super-resolution image in deep learning, and a feature map spatially indicating the feature amounts based on the features of the object 4 extracted from the normal image are generated. The feature tensors having the respective feature maps generated from the respective images are combined in the channel direction, i.e., values ​​with the same data dimensions. The feature tensor is, for example, a multidimensional tensor including a set of feature maps and other features, and is composed of multiple dimensions such as the batch size and the number of channels.

[0043] The corrected image generation unit 11 then generates a first combined feature, which is a feature obtained by combining the feature tensors in the channel direction. The generated first combined feature is then input to a decoder to restore the image data. The decoder mainly increases the size by reducing the number of channels that increased due to the extraction and combination of the features described above. That is, the size of the feature map is reduced by performing a transposed convolution (deconvolution) or the like. That is, upsampling is performed to restore the downsampled feature map. The image data restored by reconstructing the features of the object 4 in this way is generated as a corrected image (restored image).

[0044] When generating the corrected image, similar weights (filters) of each image are shared, i.e., so-called weight sharing, in order to improve learning efficiency or reduce processing load. Furthermore, the process of correcting the surface (curved surface) of the object 4 from the super-resolution image and normal image is performed by a block composed of multiple layers, such as convolutional layers and activation functions, for such processing. The corrected image generation unit corresponds to the feature extraction unit and corrected image generation unit in the embodiment of the present invention.

[0045] The object information estimation unit 12 estimates the features of the object 4 itself from the corrected image generated by the corrected image generation unit 11. First, the object information estimation unit 12 extracts the feature quantities of the object 4 contained in the corrected image. In the object information estimation unit 12, the feature quantities of the object 4 are extracted by image recognition processing using a convolutional neural network such as DenseNet, similar to the image processing used by the image information preprocessing unit 8. That is, the corrected image is input to the CNN, and the feature quantities of the object 4 are extracted by repeating convolution processing and pooling processing. Note that by extracting the feature quantities from the corrected image, data mainly related to the shape, size, and orientation of the object 4 is extracted. Furthermore, the process of extracting the feature quantities of the object 4 from the corrected image of the object 4 is performed by a block (an intermediate layer block) composed of multiple layers, such as a convolutional layer and an activation function, for such processing.

[0046] Furthermore, the object information estimation unit 12 outputs the final characteristics of the object 4 itself. Based on the extracted features, the object information estimation unit 12 estimates the state of the object 4, such as its size (dimensions), label (class or category), rotation amount (rotation angle), and thickness. The size indicates the size of the object 4, the label indicates the type of object 4 (workpiece) such as a wrench, and the rotation amount (rotation angle) indicates how much the object 4 is rotated relative to the camera 2. The object information estimation unit 12 outputs the data in a representation appropriate to each parameter. For example, the size and rotation amount are represented by coordinates (x, y, z or x, y, z, w), and the label is represented by a matrix (1xN). When each parameter is output, a different output header is used depending on the parameter.

[0047] For example, as shown in FIG. 2, each parameter is estimated by a Scale-header that estimates the size, a Label-header that estimates the label, and a Rotational-header that estimates the amount of rotation. The Rotational-header is constructed by combining a linear transformation layer (Linear layer) and an activation function such as a ReLU function. As shown in FIG. 2, the amount of rotation (rotation angle) of the image is configured to perform fitting of the output result by a least squares method such as the Gauss-Newton method described later. The object information estimation unit 12 corresponds to a state estimation unit in the embodiment of the present invention.

[0048] The above-described process estimates what type of workpiece the object 4 is, the size of the workpiece, etc. Next, a process for estimating the position of the object 4 using the image generated in the above-described process will be described.

[0049] The object position estimation unit 13 extracts features from the masked image generated by the image information preprocessing unit 8 and estimates the position of the object 4. The object position estimation unit 13 first acquires the masked image generated by the image information preprocessing unit 8. As described above, the masked image is an image in which the object 4, the orientation of which is desired to be estimated in the image data (depth image), is masked except for the object 4. That is, in the masked image, random noise is added to the object 4, the segment area of ​​the object 4 is converted into a rectangle, and the masked image is a depth image in which no other objects 4 are included.

[0050] The object position estimation unit 13 extracts feature amounts of the object 4 from the masked image. As described above, feature amounts are values ​​that quantitatively represent qualitative features of the object 4. The feature amounts are extracted, for example, by a CNN in which convolutional layers and pooling layers are alternately arranged. Specifically, the feature amounts are extracted by a convolution process that multiplies a filter that responds to a specific shape of the image data, and a pooling process that extracts one numerical value from a numerical value in a predetermined range of window in the convolved values ​​to reduce the image data. Then, based on the multiple features extracted in this way, feature amounts, or a feature map that spatially represents feature amounts, or a feature tensor that represents a collection of feature maps, are generated, thereby extracting the feature amounts.

[0051] Furthermore, the object position estimation unit 13 generates a second combined feature by combining, in the channel direction, the feature extracted from the masked image and the feature extracted from the corrected image generated by the corrected image generation unit 11. That is, a feature map or feature tensor based on the feature of the object 4 extracted from the masked image, and a feature map or feature tensor based on the feature of the object 4 extracted from the corrected image are generated. The two feature maps or feature tensors are combined in the channel direction, that is, by combining values ​​with the same data dimensions, to generate the second combined feature.

[0052] Then, the object position estimation unit 13 estimates the position of the object 4 from the generated second combined feature. The object position estimation unit 13 estimates the position of the object 4 using an existing algorithm for image recognition. That is, the position of the object 4 is estimated by the above-mentioned DenseNet or Residual Network (Residual Network) which is a CNN used for image recognition. Note that ResNet has an architecture also called a residual block, and by performing a skip connection in which a signal (image data) input to a predetermined layer is added to the output of a layer higher than the predetermined layer, the propagated error can be transmitted without attenuation, thereby enabling highly accurate image recognition.

[0053] Thereafter, the object position estimation unit 13 estimates and outputs the position of the object 4 based on the generated second combined feature. The position of the object 4 is represented by three-dimensional coordinates (x, y, z) relative to the camera 2. When the coordinates are output, a Transformer Header (Trans Header) or the like is used, which is an output header for estimating the position of the object 4, as shown in FIG. 2. By inputting the second combined feature into the Transformer Header, the position of the object 4 relative to the camera 2 is estimated and output. That is, the object position estimation unit 13 estimates the position based on the first combined feature generated from the corrected image or the like and the second combined feature generated by the object position estimation unit 13. Note that the process of extracting the feature of the object 4 from the overhead image of the object 4 is performed by a block configured of multiple layers, such as a convolutional layer and an activation function, for such processing. Note that the object position estimation unit 13 corresponds to the position estimation unit in the embodiment of the present invention.

[0054] The object pose estimation unit 14 estimates the pose of the object 4 based on the feature data regarding the size, label, and amount of rotation of the object 4 estimated by the object information estimation unit 12 described above, and the position data regarding the three-dimensional position of the object 4 from the camera 2 estimated by the object position estimation unit 13. The object pose estimation unit 14 estimates what kind of work the object 4 is, how it is placed, and in what orientation, based on these parameters. The results of this estimation are output, for example, as a numerical value expressed as a probability. In other words, as a result of the estimation, a high probability is assigned to the most likely label or pose, and conversely, a low probability is assigned to the least likely label or pose.

[0055] Furthermore, the object pose estimation unit 14 performs fitting based on the obtained feature data and position data. That is, the object pose estimation unit 14 adjusts the parameters of the learning model of the object 4 and the function used to estimate the object 4 based on the input data (Grand Truth) related to the actual object 4 and the output result related to the estimated object 4. At this time, a process of constructing a highly accurate model is executed by optimizing the fit between the input data and the learning model using the least squares method, for example, the Gauss-Newton method, which is a nonlinear least squares method. Furthermore, the object pose estimation unit 14 is configured to obtain (evaluate) the degree of deviation (error) between the output result (prediction result) and the actual parameters (teaching data) by comparing and evaluating them. The object pose estimation unit 14 corresponds to the position estimation unit and optimization unit in the embodiment of the present invention.

[0056] The output unit 15 displays the results estimated by the object pose estimation unit 14. The output unit 15 displays, for example, the most likely item regarding the estimated position, label, orientation, etc. of the object 4. At this time, the possibility that the item is the most likely item is displayed quantitatively as a probability. The output unit 15 may be, for example, a PC monitor or a display of a mobile terminal.

[0057] Next, we will explain the processing or steps (processes) executed by the posture estimation device 1 and the posture estimation method according to the embodiment of the present invention. Figures 7, 8, and 9 show a flowchart (process diagram) executed when estimating the posture of an object 4 located relatively far from the camera 2 at a work site 3, as an example of the processing or steps.

[0058] 7, first, in step S1, initial settings are made in the learning model. The initial settings made in step S1 include setting data related to the object 4 and the target object, setting data related to the degree of learning in the learning model, and the like.

[0059] After the initial settings of the learning model are completed, the process proceeds to step S2, where image data is read. In step S2, image data captured by camera 2 is input. The image data may be input manually by a user or may be automatically transferred from camera 2. In step S2, preprocessing such as normalization is performed on the input image data.

[0060] After the image data is input, the process proceeds to step S3, where noise is added to the image data. As described above, the camera 2 can estimate the depth or depth of the object 4 from the image data, and such information is added to the image data. In other words, the image data input in step S2 is also a depth image (depth_img) to which the depth or depth-related data of the object 4 is added. In step S3, noise is randomly added to the image data that is the depth image or to the object 4 included in the depth image. Adding noise to the depth image improves the performance and robustness of the learning model.

[0061] After noise is added to the depth image, the process proceeds to step S4, where the region of the object 4 included in the image data is estimated. In step S4, first, based on the label included in the data of the object 4 set at the time of initialization, a segment region of the object 4 corresponding to the label is generated or added. That is, in step S4, as shown in FIG. 4(a), segment regions are added to all of the workpieces, etc., set in the learning model as objects whose poses are to be estimated. In addition, in step S4, the segment region of the object 4 is generated, for example, by detecting each object based on the feature values ​​of each object including the object 4 that has been learned in advance, or by detecting the pixel region of the object 4 in the image data.

[0062] Then, in step S4, the generated segment area of ​​each object 4 is converted into a rectangle. For example, in step S4, as shown in FIG. 4(b), a square area (bounding box) of each object 4 is generated by detecting the object 4 or the like, thereby converting the segment area of ​​each target object into a rectangle.

[0063] After the segment image (seg_img) is generated in step S4, the process proceeds to step S5, where a masking process is performed on the depth image generated in step S3. In step S5, as shown in FIG. 4(c), a masked image (headview_img) is generated in which, among the multiple objects that have been segmented, the area of ​​a specific object 4 whose pose is to be estimated is left and the other parts are masked. For example, if multiple objects (workpieces) are detected in step S4, the multiple objects 4 are each in a different layer, and therefore the layer of the specific object 4 whose pose is to be estimated is extracted.

[0064] After the image data is masked, the process proceeds to step S6, where the depth image is cropped to a segment area of ​​the predetermined object 4. That is, in step S6, as shown in Fig. 5(a), a cropped image (crop_depth) is generated in which the cropping process is performed so as to leave only the predetermined object 4 included in the depth image.

[0065] After the clipping process is performed in this manner, the process proceeds to step S7, where the clipped image generated in step S6 is super-resolved. In step S7, the above-mentioned SRCNN, which is a neural network for super-resolving the object 4, is used. As a result, features are extracted from the clipped image input to the SRCNN, and missing (defective) data is complemented, resulting in the generation of a super-resolved clipped image as shown in FIG. 5(b). As described above, if the object 4 is located relatively far from the camera 2, the captured image data may not capture the three-dimensionality of the object 4, or holes and gaps in the object 4 may be blurred. In step S7, the clipped image is super-resolved, whereby the three-dimensionality and holes of the object 4 are restored as a point cloud with relatively high accuracy.

[0066] After the super-resolved cropped image is generated, the process proceeds to step S8, where a normal image (crop_norm) of object 4 is generated from the super-resolved image. The normal image is an image that indicates the orientation of the surface of object 4 by adding data related to a vector perpendicular to the surface of object 4, as shown in FIG. 5(c). In step S7, the cropped image is super-resolved, so that the surface of object 4 is restored with relatively high accuracy. Therefore, the normal image generated in step S8 can also represent the surface of object 4 with relatively high accuracy.

[0067] After the normal image is generated, the process proceeds to step S9, where a feature (feature 1) of object 4 is extracted from the super-resolution image. In step S9, characteristic parts of object 4 included in the super-resolution image are extracted, and qualitative parameters are extracted as feature quantities, which are quantitative parameters. This process is performed using a convolutional neural network for image classification, such as DenseNet. The extracted feature quantities are mainly used to detect or estimate the distance between camera 2 and object 4. Note that FIG. 6(a) shows an example of data input to the convolutional neural network for image classification.

[0068] After the feature quantities of object 4 are extracted from the super-resolution image, the process proceeds to step S10, where a feature quantity (feature quantity 2) of object 4 is extracted from the normal image generated in step S8. In step S10, similar to the process executed in step S9, a CNN for image classification is used to extract characteristic parts of object 4 contained in the normal image, and qualitative parameters are extracted as feature quantities, which are quantitative parameters. The feature quantities extracted at this time are mainly features for detecting or estimating the surface and contour of object 4. Note that FIG. 6(b) shows an example of data input to the convolutional neural network for image classification.

[0069] After the feature of the object 4 is extracted from the normal image, the process proceeds to step S11, where a first combined feature (feature 3) is generated by combining the feature extracted in steps S9 and S10. In step S11, the feature extracted from the super-resolution image and the feature extracted from the normal image are combined. When combining, the generated feature tensors are combined in the channel direction, that is, with values ​​that have the same data dimensions. In this way, the features of the super-resolution image and the normal image are combined in the channel direction to generate the first combined feature.

[0070] After the first synthesized feature is generated, the process proceeds to step S12, where a corrected image is generated from the first synthesized feature. In step S12, a corrected image (depth_imprior) in which a predetermined object 4 has been corrected is generated based on the first synthesized feature. In step S12, a corrected image such as that shown in FIG. 6(c) is generated by inputting the first synthesized feature to a decoder, etc. That is, when the object 4 is expanded by deconvolving the first synthesized feature, missing parts are interpolated using neighboring copy values. That is, by performing an upsampling process (upsampling) that enlarges (expands) the feature map, the object 4 is restored and a corrected image is generated.

[0071] After the corrected image is generated, the process proceeds to step S13, where feature quantities (feature quantities 4) of object 4 are extracted from the generated corrected image. In step S13, similar to the processes executed in steps S9 and S10, feature quantities are extracted by extracting characteristic parts of object 4 included in the corrected image using a CNN used in image processing such as DenseNet. The feature quantities extracted in step S13 are mainly feature quantities for detecting or estimating the shape, size, orientation, etc. of object 4.

[0072] After the feature quantities of object 4 are extracted from the corrected image, the process proceeds to step S14, where detailed information about object 4 is estimated from the extracted feature quantities of object 4 in the corrected image. In step S14, data such as the size (dimensions), label (classification), and rotation amount of object 4 are estimated from the feature quantities of object 4. That is, estimated values ​​of this data are output as the final output based on the feature quantities of object 4 extracted from the corrected image. The output values ​​are represented by coordinates or matrices. For example, the size of object 4 is represented by coordinates in the X-axis, Y-axis, and Z-axis directions, and the rotation amount of object 4 is represented by coordinates that further add the W-axis perpendicular to the Z-axis. Furthermore, the labels are represented by matrices that correspond to the number of estimated labels of object 4, etc. The size, label, and rotation amount of object 4 are estimated by inputting them into a header network. That is, detailed data about object 4 is estimated using a header that estimates the size, label, and rotation amount from the input feature quantities.

[0073] After the data of the object 4 is estimated, the process proceeds to estimating the position of the object 4. After the process of step S14, the process proceeds to step S15, where feature quantities (feature quantities 5) of the object 4 are extracted from a masked image obtained by masking the image data generated in step S5. In step S15, similar to the process executed in step S9 and the like, the masked image is input to DenseNet, whereby characteristic parts of a predetermined object 4 included in the masked image are extracted, and qualitative parameters are extracted as feature quantities, which are quantitative parameters. The feature quantities extracted at this time mainly relate to the position of the object 4 in the angle of view of the image data, that is, features related to the position relative to the camera 2.

[0074] After the feature amounts related to the position of the object 4 are extracted, the process proceeds to step S16, where the extracted feature amounts of the object 4 are synthesized or combined. In step S16, a second synthesized feature amount (feature amount 6) is obtained by synthesizing the feature amount related to the position of the object 4 extracted in step S15 and the feature amount related to the object 4 itself extracted in step S13. In step S16, similar to the process in step S11, the generated feature amount tensors are combined in the channel direction, that is, values ​​with aligned data dimensions.

[0075] After the second synthetic feature is generated, the process proceeds to step S17, where the position of the object 4 in the image data is estimated. In step S17, the position of the object 4 is estimated based on the second synthetic feature of the object 4 generated in step S16. In step S17, the position of the object 4 is estimated and output from each patch obtained by dividing the image data using, for example, a Transformer-based architecture. The position of the object 4 is output using coordinates in the X-axis, Y-axis, and Z-axis directions.

[0076] In this way, by processing steps S1 to S17, the size (dimensions), label (type), amount of rotation (rotation angle) and position of object 4 contained in the image data are detected or estimated, and the posture of object 4 is thereby estimated.

[0077] After the orientation of the object 4 is estimated, the process proceeds to step S18, where fitting is performed based on the estimated position and orientation of the object 4. In fitting, the parameters of a learning model of the object 4 and the parameters of a function used to estimate the object 4 are adjusted based on the parameters of the object 4 based on the input image data and the parameters of the object 4 based on the estimated output result. Specifically, the parameters of the learning model are adjusted to minimize the error function between the input actual data (grand truth) and the estimated result using a least squares method such as the Gauss-Newton method. In this way, in step S18, a process is executed to build a highly accurate model by optimizing the fit between the input data and the learning model.

[0078] As described above, in the posture estimation device 1 and posture estimation method for an object 4 according to an embodiment of the present invention, when estimating the posture of an object 4 included in image data, a depth image and a normal vector image (normal image) of the object 4 are first generated based on the input image data. From the generated depth image, a super-resolution image is generated using a convolutional neural network (SRCNN) that increases the resolution of the object 4 included in the image data. Features of the object 4 are extracted from each of the super-resolution image and the normal image using a CNN suitable for image classification, such as DenseNet. First composite features are generated by aligning and combining the extracted features in the channel direction, and a corrected image is generated based on the first composite features. Then, features of the object 4 itself, such as size (dimensions), label (classification), and rotation amount (rotation angle), are estimated from the features of the object 4 extracted from the corrected image, and the results are output.

[0079] Then, features of object 4 are extracted from a masked image in which only object 4 is left from the input image data and the depth is displayed. A second combined feature is generated by combining the feature extracted from the masked image and the feature of object 4 extracted from the corrected image in the channel direction. The position of object 4 from camera 2 is estimated from the second combined feature. In other words, the posture of object 4 is estimated from data output based on the first combined feature and the second combined feature.

[0080] For example, if the object 4 is inevitably placed far from the camera 2 due to circumstances at the work site 3, the captured image data will contain a small image of the object 4. In such cases, the resolution of the object 4 may be low, resulting in fewer feature amounts for the object 4, or the distance between the positions where feature amounts are extracted may be so long that the shape of the object 4 is recognized as a plane, and so on, and it may not be possible to obtain sufficient feature amounts to estimate the orientation of the object 4.

[0081] Even in such a case, in the posture estimation device 1 or posture estimation method according to the embodiment of the present invention, the shape of the object 4 is restored with high accuracy using a super-resolution image obtained by super-resolving the input image data (depth image), a normal vector image of the object 4 generated from the super-resolution image, and each normalized image. Therefore, the surface, contour, thickness, distance, etc. of the object 4 are restored with high accuracy, and the features of the object 4 itself can be estimated with high accuracy.

[0082] Furthermore, the position of object 4 is estimated based on a masked image in which everything other than object 4 is masked from image data (depth image) captured from a bird's-eye view, and the above-mentioned corrected image. The above-mentioned super-resolution image is a segmented image in which only object 4 is cut out, so although the features of object 4 itself are extracted with high accuracy, the positional relationship between camera 2 and object 4 cannot be extracted with high accuracy. Therefore, by further using the feature amounts of object 4 extracted from a masked image in which everything other than object 4 is masked from image data of input object 4 captured from a bird's-eye view, the learning model can accurately learn the actual positional relationship, and the estimation accuracy of the posture of object 4 can be further improved.

[0083] Although the embodiments of the present invention have been described above, the present invention is not limited to the above examples and may be modified as appropriate within the scope of achieving the object of the present invention. For example, the input image data is not limited to the depth image described above, but may be point cloud data in which the shape of the object 4 is represented by a point cloud obtained by 3D scanning or the like, or image data to which three RGB channel data representing the color of the object 4 is added in addition to the depth image. Furthermore, the depth image is not limited to a configuration in which depth data is acquired by the camera 2, but may be configured to acquire data regarding the depth of the object (depth data) by a depth data acquisition unit that analyzes the acquired image data and extracts depth data. Furthermore, the present invention may be configured as a posture estimation system including the posture estimation device 1 and camera 2 shown in FIG. 1. [Explanation of symbols]

[0084] 1 Posture estimation device 2 Cameras 3. Work site 4 objects 5. First Processing Section 6. Second Processing Section 7 Image acquisition unit 8 Image information preprocessing section 9 Super-resolution processing section 10 Normal image generation unit 11 Corrected image generation unit 12 Object information estimation section 13. Object position estimation section 14. Object posture estimation section 15 Output Department

Claims

1. An object posture estimation device that analyzes image data of an object using a convolutional neural network and estimates the posture of the object contained in the image data, a depth data acquisition unit that acquires a depth image including depth data that is data related to the depth of the object included in the image data; a normal data acquisition unit that acquires a normal image by calculating a normal vector of the surface of the object from the depth image; a super-resolution image generating unit that generates a super-resolution image in which the resolution of the object cut out from the depth image is increased; a feature extraction unit that extracts feature amounts that quantitatively represent features of the object included in the image data, the super-resolution image, and the normal image; a corrected image generation unit that generates a corrected image in which the shape of the object is restored based on the feature amounts extracted from the super-resolution image and the normal image; a state estimation unit that estimates parameters related to a state of the object based on the corrected image; a position estimation unit that estimates a position of the object from the image data; a posture estimation unit that estimates a posture of the object based on the parameters related to the state of the object estimated by the state estimation unit and the position of the object estimated by the position estimation unit. The object pose estimation device is characterized by:

2. The object posture estimation device according to claim 1, The apparatus further includes an optimization unit that performs fitting of output data of the object position estimated by the position estimation unit and the object posture estimated by the posture estimation unit to actual data regarding the object position and posture by a Gauss-Newton method. The object pose estimation device is characterized by:

3. 3. The object posture estimation device according to claim 1, The corrected image generation unit generating a composite feature by combining the feature of the object extracted from the depth image and the feature of the object extracted from the normal image in a channel direction; The object is restored by performing an upsampling process that interpolates missing portions using surrounding copy values ​​based on the synthesized feature amount, and the corrected image is generated. The object pose estimation device is characterized by:

4. A method for estimating an object's posture by analyzing image data of an object using a convolutional neural network and estimating the posture of the object contained in the image data, acquiring a depth image including depth data that is data relating to the depth of the object included in the image data; obtaining a normal image by calculating a normal vector of the surface of the object from the depth image; generating a super-resolution image by increasing the resolution of the object cropped from the depth image; extracting feature amounts that quantitatively represent features of the object included in the image data, the super-resolution image, and the normal image; generating a corrected image by restoring the shape of the object based on the feature amounts extracted from the super-resolution image and the normal image; estimating parameters relating to the state of the object based on the corrected image; Estimating the position of the object from the image data; Estimating a pose of the object based on the parameters related to the estimated state of the object and the estimated position of the object.

2. A method for estimating the posture of an object, comprising:

5. The object pose estimation method according to claim 4, The output data relating to the estimated object position and the estimated object pose are fitted to the actual data relating to the object position and pose by a Gauss-Newton method.

2. A method for estimating the posture of an object, comprising:

6. 6. The method for estimating an object posture according to claim 4 or 5, generating a composite feature by combining the feature of the object extracted from the depth image and the feature of the object extracted from the normal image in a channel direction; Based on the synthesized feature, the object is restored by performing an upsampling process to interpolate missing parts using surrounding copy values, and the corrected image is generated.

2. A method for estimating the posture of an object, comprising:

Images