Camera device and camera system

Abstract

The present invention provides a photographing device and a photographing system, wherein the photographing device performs machine learning processing related to the estimation of a distance image, wherein the estimation of the distance image is to estimate a distance image related to the object that is closer to reality than the distance image based on the distance image related to the object captured by a photographing sensor, and the photographing device comprises: a data acquisition unit that acquires distance image data related to the object; and a preprocessing unit that generates input data based on the distance image data related to the object, and uses the input data to perform machine learning processing for estimating distance image data related to the object that is closer to reality based on the distance image data related to the object.

Description

Technical Field

[0001] The present invention relates to an imaging device and an imaging system, and in particular to an imaging device and an imaging system capable of acquiring height information of an object. Background Art

[0002] In the past, systems have been developed for the purpose of detecting the three-dimensional position, posture, and shape of an object. For example, a three-dimensional vision sensor is known that can capture a distance image having distance information as a pixel value. The three-dimensional vision sensor obtains an image from two sensors (cameras) configured to have parallax for analysis. The three-dimensional vision sensor uses a triangulation technique that utilizes parallax to find out how far the position corresponding to each pixel is from the sensor. Thus, the three-dimensional vision sensor restores the three-dimensional shape of the object (for example, Japanese Patent Publication No. 2013-024653, etc.).

[0003] However, sometimes the shape of the object is distorted in the distance image captured by the three-dimensional vision sensor using a stereo method or the like. For example, when a portion of an object whose distance from the three-dimensional vision sensor changes dramatically is obtained by two sensors of the three-dimensional vision sensor, it is difficult to accurately match the portion captured in each image with each other. Therefore, even for pixels representing portions of an object located at positions where the distances from the three-dimensional vision sensor are equal to each other, sometimes different distance information is calculated between the pixels. In this case, the shape of the object in the distance image is distorted. Summary of the invention

[0004] Therefore, there is a need for an imaging device and an imaging system that can estimate a distance image of an object that is closer to the real object from a distance image of an object with a distorted shape.

[0005] One aspect of the present disclosure uses machine learning to repair a range image in which the shape of an object is distorted through the following process.

[0006] Process 1) Obtain CAD (Computer Aided Design) data of the object.

[0007] Step 2) Obtaining a range image of the object Using a known analysis method, the position and posture of the object on the range image are obtained.

[0008] Process 3) transforms the position and posture of the object on the range image into the position and posture of the object observed by the three-dimensional vision sensor.

[0009] Process 4) The CAD data of the object and the three-dimensional vision sensor are configured in the virtual space through the positional relationship between the three-dimensional vision sensor and the object.

[0010] Process 5) Generates a distance image of the object projected by the three-dimensional visual sensor in the virtual space that is close to reality.

[0011] Process 6) sets a data set with real-world distance images as input data and virtual-space distance images as label data as learning data.

[0012] Step 7) Perform steps 1 to 6 for each object. The objects may be of different types.

[0013] Process 8) The acquired set of learning data is used as a learning data set and is learned by a learner such as a neural network. When a distance image of the real world is input as input data, the learner transforms the distance image into a distance image close to reality.

[0014] Furthermore, one method of the present disclosure is a photographing device that performs machine learning processing, wherein the machine learning processing involves estimating a distance image related to the object that is closer to reality than the distance image based on a distance image related to the object captured by a photographing sensor, and the photographing device comprises: a data acquisition unit that acquires distance image data related to the object; and a preprocessing unit that generates input data based on the distance image data related to the object, and uses the input data to perform machine learning processing for estimating distance image data related to the object that is closer to reality based on the distance image data related to the object.

[0015] Another embodiment of the present disclosure is a machine learning processing method for a photographing device that performs machine learning processing, wherein the machine learning processing is used to infer a distance image related to the object that is closer to reality based on a distance image related to the object captured by a photographing sensor, and the machine learning processing method comprises: a first step of obtaining distance image data related to the object; a second step of generating input data based on the distance image data related to the object; and a third step of using the input data to perform machine learning processing for inferring distance image data related to the object that is closer to reality based on the distance image data related to the object.

[0016] Another aspect of the present disclosure is an imaging system that connects a plurality of devices to each other via a network, the plurality of devices including an imaging device having at least a learning unit.

[0017] According to one aspect of the present disclosure, it is possible to estimate a distance image of a target object that is closer to the actual target object than the distance image, based on a distance image in which the shape of the target object is distorted. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects and features of the present invention will become clear by describing the following embodiments with reference to the accompanying drawings. In these drawings:

[0019] Figure 1 This is a schematic hardware configuration diagram of a photographing device having a machine learning device according to one embodiment.

[0020] Figure 2 This is a schematic functional block diagram of the imaging device according to the first embodiment.

[0021] Figure 3 The distance image data of the object and the distance image data close to reality are exemplified.

[0022] Figure 4 This is a schematic functional block diagram of the imaging device according to the second embodiment.

[0023] Figure 5 This example shows a three-layer system consisting of a cloud server, fog computer, and edge computer.

[0024] Figure 6 This is a schematic diagram of the hardware structure of the imaging device installed on the computer.

[0025] Figure 7 It is a schematic configuration diagram of an imaging system according to a third embodiment.

[0026] Figure 8 It is a schematic configuration diagram of an imaging system according to a fourth embodiment.

[0027] Fig. 9 It is a schematic configuration diagram of an imaging system according to a fifth embodiment. DETAILED DESCRIPTION

[0028] The following, with the attached Figure 1 The embodiments of the present invention will now be described.

[0029] Figure 1 The schematic hardware structure diagram of an imaging device with a machine learning device according to one embodiment is shown. The imaging device 1 of this embodiment is installed in a computer such as a personal computer installed in a factory, a unit computer for managing machinery installed in a factory, a host computer, an edge computer, or a cloud server. Figure 1 This shows an example in which the imaging device 1 is installed in a personal computer installed in a factory.

[0030] The CPU (Central Processing Unit) 11 of the imaging device 1 of this embodiment is a processor that controls the imaging device 1 as a whole. The CPU 11 reads the system program stored in the ROM (Read Only Memory) 12 via the bus 20. The CPU 11 controls the entire imaging device 1 according to the system program. Temporary calculation data, various data input by the operator via the input device 71, etc. are temporarily stored in the RAM (Random Access Memory) 13.

[0031] The non-volatile memory 14 is composed of, for example, a memory supported by a battery not shown or an SSD (Solid State Drive). The storage state of the non-volatile memory 14 is maintained even if the power supply of the shooting device 1 is turned off. A setting area for storing setting information related to the operation of the shooting device 1 is ensured in the non-volatile memory 14. The non-volatile memory 14 stores programs and data input from the input device 71, CAD data read via an external storage device or network not shown, etc. The programs and various data stored in the non-volatile memory 14 can be expanded in the RAM 13 when executed / used. In addition, a well-known analysis program for analyzing a learning data set and a system program for controlling data exchange with the machine learning device 100 described later are pre-written in the ROM 12.

[0032] The photographing sensor 4 is, for example, a three-dimensional visual sensor that generates distance image data of an object (data with two-dimensional or one-dimensional arrangement of distance information as a value) by photographing the object. The three-dimensional visual sensor as the photographing sensor 4 can be a triangulation method having multiple CCD (Charge Coupled Device) sensors, a time-of-flight method, or a focusing method. The photographing sensor 4 is, for example, installed on the manipulator of a robot not shown in the figure. The photographing sensor 4 moves to a photographing position for photographing the object to be identified by the robot. The photographing sensor 4 forwards the distance image data obtained by photographing the object to the CPU 11 via the interface 19. In addition, the photographing sensor 4 can also be fixedly set at a predetermined position so that the object grasped by the manipulator of the robot moves to a position where it can be photographed by the photographing sensor 4, thereby photographing the distance image of the object. The photographing sensor 4 can also be controlled by executing a program by the photographing device 1 to photograph the object. The photographing sensor 4 can also be controlled by a robot controller or other device that controls the robot to photograph the object.

[0033] The display device 70 displays various data read into the memory, data obtained as a result of executing the program, etc. In addition, the distance image data of the object obtained by the imaging sensor 4, the data output from the machine learning device 100 described later, etc. are output and displayed on the display device 70 via the interface 17. In addition, the input device 71 composed of a keyboard and a pointing device, etc. receives instructions, data, etc. based on the operator's operation. The input device 71 forwards these instructions, data, etc. to the CPU 11 via the interface 18.

[0034] The interface 21 connects the photographing device 1 to the machine learning device 100. The machine learning device 100 has: a processor 101 that controls the entire machine learning device 100, and a ROM 102 that stores a system program, etc. The machine learning device 100 has a RAM 103 for temporary storage in various processes for machine learning. The machine learning device 100 has a non-volatile memory 104 for storing learning models, etc. The machine learning device 100 can observe various information that can be obtained by the photographing device 1 (for example, distance image data, CAD data, etc.) via the interface 21. In addition, the photographing device 1 obtains the processing results output from the machine learning device 100 via the interface 21.

[0035] Figure 2 This is a schematic functional block diagram of a learning mode of the imaging device 1 and the machine learning device 100 according to the first embodiment. Figure 1 The CPU 11 of the photographing device 1 and the processor 101 of the machine learning device 100 execute their respective system programs to control the operations of the photographing device 1 and the machine learning device 100, thereby achieving Figure 2 The functions of the functional blocks shown.

[0036] The imaging device 1 of this embodiment includes a data acquisition unit 30, an object detection unit 32, and a preprocessing unit 34. The machine learning device 100 included in the imaging device 1 includes a learning unit 110. CAD data related to an object (there may be a plurality of objects) used for learning is stored in advance in the CAD data storage unit 50 provided in the nonvolatile memory 14 via an external storage device (not shown) or a wired / wireless network.

[0037] The data acquisition unit 30 is a functional unit that acquires distance image data related to the object from the imaging sensor 4 or via an external storage device or a wired / wireless network (not shown). The distance image data acquired by the data acquisition unit 30 is data in which the distance from the imaging sensor 4 is set for each pixel on the two-dimensional imaging plane.

[0038] The object detection unit 32 is a functional unit that obtains the position and posture of the object in the distance image data based on the distance image data related to the object obtained by the data acquisition unit 30. The object detection unit 32 determines the shape of the object by, for example, reading the CAD data related to the object from the CAD data storage unit 50. The object detection unit 32 performs matching processing between the CAD data and the distance image data while changing the position and posture of the CAD data. In this way, the object detection unit 32 determines the position and posture of the object from the distance image data.

[0039] The preprocessing unit 34 generates training data used in the learning of the machine learning device 100 based on the distance image data related to the object and the CAD data related to the object. The preprocessing unit 34 generates the training data T using the distance image data related to the object as input data and using the distance image data related to the object generated based on the CAD data that is close to reality (data of a distance image that is closer to reality than the distance image related to the object acquired by the data acquisition unit 30, data of an ideal distance image related to the object) as output data. Figure 3 An example of a distance image related to an object and a distance image related to an object that is close to reality is shown. As described above, according to the distance image generation algorithm, a distance detection error occurs in the portion of the object where the distance to the imaging sensor 4 changes. Therefore, this portion is prone to distortion. For example, the preprocessing unit 34 configures CAD data at the same position and posture as the position and posture of the object relative to the imaging sensor 4 for the viewpoint position in the virtual space, and generates a distance image data close to reality related to the object based on the shape of the CAD data observed from the viewpoint position at that time. The distance image close to reality related to the object generated in this way will not produce an error even in the portion where the distance to the viewpoint changes. Therefore, the image will not be distorted. In addition, sometimes a fixture for fixing the object is also captured in the distance image data related to the object. In this case, the preprocessing unit 34 can also obtain CAD data related to the fixture in advance, configure the CAD data related to the fixture in the virtual space like the CAD data of the object, and include it in the distance image data close to reality of the object.

[0040] The learning unit 110 is a functional unit that performs supervised learning using the training data T generated by the preprocessing unit 34. The learning unit 110 generates (learns) a learned model, which is used to estimate the distance image data related to the object that is close to reality based on the distance image data related to the object. The learning unit 110 of the present embodiment can be configured to perform supervised learning using a neural network as a learning model, for example. In this case, a neural network having three layers, namely an input layer, an intermediate layer, and an output layer, can be used as a learning model. In addition, a neural network forming more than three layers can also be used as a learning model. That is, a deep learning method can be used. In this case, more efficient learning and inference are performed. The learned model generated by the learning unit 110 is stored in the learning model storage unit 130 provided in the non-volatile memory 104. The learned model is used for the estimation processing of the distance image data related to the object that is close to reality performed by the estimation unit 120.

[0041] The learning unit 110 uses the distance image data related to various objects (different objects) acquired by the data acquisition unit 30, and the distance image data related to the object captured by changing the shooting conditions (lighting position, etc.) to repeat the above-mentioned learning. As a result, the learning unit 110 generates a learned model for estimating the distance image data related to the object that is close to reality based on the distance image data related to the object. The generated learned model is independent of the object and the shooting conditions, and is used to obtain a distance image that is closer to reality. That is, by using the generated learned model, no matter how the object in the distance image obtained from the shooting sensor 4 is distorted, a distance image that is closer to reality is obtained.

[0042] In a variation of the photographing device 1 of the present embodiment, the data acquisition unit 30 acquires not only the distance image data related to the object, but also acquires other forms of image data such as brightness image data related to the object captured in the same positional relationship as the positional relationship between the photographing sensor 4 and the object when the distance image is acquired. The data acquisition unit 30 uses the acquired image data in other forms as auxiliary image data to assist the distance image data for learning. In this case, the preprocessing unit 34 generates training data T using the distance image data related to the object and the auxiliary image data as input data. The learning unit 110 performs supervised learning using the generated training data T. Thus, a learned model for estimating the distance image data close to reality related to the object is constructed based on more information related to the object. By using the learned model, the accuracy of the estimation of the distance image data close to reality performed by the estimation unit 120 is improved.

[0043] In another variation of the photographing device 1 of the present embodiment, for example, realistic distance image data related to the object used when generating the training data T is generated based on an image obtained using a high-precision photographing sensor. As high-precision photographing sensors, there are illustrated photographing sensors with high resolution, sensors that detect brightness or other physical quantities in addition to the distance image, and sensors that can detect the distance with higher accuracy based on this information. Such photographing sensors are expensive sensors, but are only required at the stage of building a learned model, and have the advantage of being used for other purposes after the learned model is built. In addition, such photographing sensors have the advantage of being used to build a learned model when the CAD data of the object is not available.

[0044] Figure 4 This is a schematic functional block diagram of the estimation mode of the imaging device 1 and the machine learning device 100 according to the second embodiment. Figure 1 The CPU 11 of the photographing device 1 and the processor 101 of the machine learning device 100 execute their respective system programs to control the operations of the photographing device 1 and the machine learning device 100, thereby achieving Figure 4 The functions of the functional blocks shown.

[0045] In the estimation mode, the imaging device 1 of this embodiment estimates distance image data closer to reality than the distance image related to the object based on the distance image data related to the object acquired by the data acquisition unit 30. In the imaging device 1 of this embodiment, the functions of the data acquisition unit 30 and the object detection unit 32 are the same as those of the first embodiment.

[0046] The preprocessing unit 34 generates input data used for estimation by the machine learning device 100 based on the range image data about the object.

[0047] The estimation unit 120 estimates a realistic image related to the object using the learned model stored in the learning model storage unit 130 based on the input data input from the preprocessing unit 34. The estimation unit 120 of the present embodiment inputs the input data (distance image data related to the object) input from the preprocessing unit 34 to the learned model (with determined parameters) generated by the learning unit 110 through supervised learning. Thus, the estimation unit 120 estimates (calculates) realistic distance image data related to the object. For example, the realistic distance image data related to the object estimated by the estimation unit 120 is displayed on the display device 70. In addition, the realistic distance image data related to the object estimated by the estimation unit 120 can be output to a host computer or a cloud computer via a wired / wireless network not shown in the figure for use.

[0048] In the imaging device 1 of the present embodiment configured as described above, realistic distance image data about an object is estimated using a learned model obtained from a plurality of training data obtained by imaging various objects under various imaging conditions.

[0049] In the following third to fifth embodiments, an embodiment in which the camera 1 of the first embodiment is installed as a part of a system interconnected with a plurality of devices including a cloud server or a host computer, a fog computer, an edge computer (robot controller, control device, etc.) via a wired / wireless network is described. Figure 5 As illustrated, in the following third to fifth embodiments, it is assumed that a system is constituted by logically dividing a plurality of devices into three layers, namely, a layer including devices such as a cloud server 6, a layer including devices such as a fog computer 7, and a layer including devices such as an edge computer 8 (a robot controller, a control device, etc. included in a unit 9) while a plurality of devices are respectively connected to a network. In such a system, a photographing device 1 according to one embodiment of the present invention can be installed in one of the cloud server 6, the fog computer 7, and the edge computer 8. The photographing device 1 can share data used in processing related to machine learning with each of the plurality of devices via a network to perform distributed learning. In addition, the photographing device 1 can collect the generated learning model to the fog computer 7 or the cloud server 6 for large-scale analysis. Furthermore, the photographing device 1 can reuse the generated learning models, etc. Figure 5 In the illustrated system, multiple units 9 are installed in factories in various places, and each unit 9 is managed by a fog computer 7 at a higher level in a predetermined unit (factory unit, multiple factories of the same manufacturer, etc.). In addition, the data collected and analyzed by these fog computers 7 are also collected and analyzed by the cloud server 6 at the higher level. The information obtained as a result is used for control in each edge computer 8, etc.

[0050] Figure 6 This is a schematic diagram of the hardware structure when the camera is installed on a computer such as a cloud server or fog computer.

[0051] The CPU 311 of the photographing device 1' installed in the computer of this embodiment is a processor that controls the photographing device 1' as a whole. The CPU 311 reads the system program stored in the ROM 312 via the bus 320. The CPU 311 controls the entire photographing device 1' according to the system program. Temporary calculation data and display data, various data input by the operator via an input unit not shown, etc. are temporarily stored in the RAM 313.

[0052] The non-volatile memory 314 is supported, for example, by a battery not shown. The non-volatile memory 314 is a memory that maintains the storage state even if the power of the imaging device 1' is turned off. The non-volatile memory 314 stores programs input via the input device 371 and various data obtained from the imaging sensor 4 and the like via various parts of the imaging device 1' or the network 5. The programs and various data stored in the non-volatile memory 314 can be expanded in the RAM 313 when executed / used. In addition, various system programs such as well-known analysis programs (including system programs for controlling data exchange with the machine learning device 100 described later) are pre-written in the ROM 312.

[0053] The camera 1' is connected to the wired / wireless network 5 via the interface 319. The network 5 is connected to at least one camera sensor, other cameras 1, edge computers 8, fog computers 7, cloud servers 6, etc. These devices exchange data with the camera 1'.

[0054] Data read into the memory and data obtained as a result of executing a program are output and displayed on the display device 370 via the interface 317. The input device 371 composed of a keyboard and a pointing device transfers commands and data based on operator operations to the CPU 311 via the interface 318.

[0055] The interface 321 is an interface for connecting the photographing device 1' to the machine learning device 100. The machine learning device 100 has Figure 1 The structure is the same as the machine learning device 100 described above.

[0056] Thus, when the imaging device 1 ′ is installed in a computer such as a cloud server or a fog computer, the functions of the imaging device 1 ′ are the same as those described in the first and second embodiments except that information from the imaging sensor 4 is exchanged via the network 5 .

[0057] Figure 7 This is a schematic diagram of the configuration of an imaging system according to a third embodiment having an imaging device 1 ′. The imaging system 500 includes a plurality of imaging devices 1 and 1 ′, a plurality of imaging sensors 4 , and a network 5 that interconnects the imaging devices 1 and 1 ′ and the imaging sensors 4 .

[0058] In the photographing system 500, the photographing device 1' having the machine learning device 100 uses the learning result of the learning unit 110 to infer a close-to-reality distance image related to the object photographed by the photographing sensor 4. In addition, at least one photographing device 1' learns a close-to-reality distance image related to the object common to all the photographing devices 1, 1' based on the training data T respectively obtained by the other plurality of photographing devices 1, 1'. The photographing system 500 is configured so that the learning result is shared by all the photographing devices 1, 1'. Therefore, in the photographing system 500, a more diverse data set (including the training data T) is used as input, and the speed and reliability of learning are improved.

[0059] Figure 8 1 is a schematic diagram of a system according to a fourth embodiment in which a machine learning device and a photographing device are installed in different devices. The photographing system 500' has at least one machine learning device 100 installed as part of a computer such as a cloud server, a host computer, or a fog computer (in Figure 8 In the example, a plurality of cameras 1" and a network 5 connecting the cameras 1" and the computer are shown. Figure 6 The hardware of the imaging device 1 ′ shown is similar to that of a CPU 311 , a RAM 313 , a nonvolatile memory 314 , and other hardware that a general computer has, which are connected via a bus 320 .

[0060] In the shooting system 500' having the above-described structure, the machine learning device 100 learns the correlation between the distance image of the object common to all the shooting devices 1" and the distance image of the object that is close to reality, based on the training data T obtained for each of the plurality of shooting devices 1". By using the learning result in the shooting system 500', it is possible to infer the distance image that is close to reality based on the image of the object captured by each shooting sensor 4. According to the structure of the shooting system 500', the required number of shooting devices 1" can be connected to the machine learning device 100 when needed, regardless of the location and date and time.

[0061] Fig. 9 1 is a schematic diagram of a photographing system 500" of a fifth embodiment having a machine learning device 100' and a photographing device 1. The photographing device 500" comprises: at least one machine learning device 100' (in Fig. 9 , an example is shown in which a machine learning device 100' is installed as part of a fog computer 7), multiple photographing devices 1, and a wired / wireless network 5 that interconnects these photographing devices 1 and a computer.

[0062] In the photographing system 500" having the above-mentioned structure, the fog computer 7 having a machine learning device 100' obtains a learning model obtained as a result of machine learning of the machine learning device 100 possessed by each photographing device 1. Furthermore, the machine learning device 100' regenerates an optimized or streamlined learning model by optimizing or streamlining knowledge based on these multiple learning models. The machine learning device 100' distributes the generated learning model to each photographing device 1.

[0063] As an example of optimization or efficiency improvement of the learning model performed by the machine learning device 100', generation of a distillation model based on multiple learning models obtained from each shooting device 1 is listed. At this time, the machine learning device 100' of this embodiment generates input data to the learning model. The machine learning device 100' uses the output obtained as a result of inputting the input data to each learning model to start learning from 1. The machine learning device 100' generates a new learning model (distillation model) by such a method. As described above, the distillation model generated in this way is distributed to the shooting device 1 or other computers via an external storage medium or network 5 for use.

[0064] Next, other examples of optimization or efficiency improvement of the learning model performed by the machine learning device 100' are shown. In the process of distilling multiple learning models obtained from each shooting device 1, the distribution of output data of each learning model relative to the input data is analyzed by a general statistical method (for example, outlier detection, etc.). Outliers of a group of input data and output data are extracted, and distillation is performed using the group of input data and output data from which the outliers have been removed. Through such a process, exceptional estimation results are removed from the group of input data and output data obtained from each learning model. As a result, a distillation model is generated using the group of input data and output data from which the exceptional estimation results have been removed. The distillation model generated in this way is used as a more general learning model than the learning models generated by multiple shooting devices 1.

[0065] In addition, other general learning model optimization or efficiency improvement methods (analyzing each learning model and optimizing the hyperparameters of the learning model based on the analysis results) may be appropriately introduced.

[0066] In the photographing system 500" of the present embodiment, for example, a machine learning device 100' is configured in a fog computer 7 provided for a plurality of photographing devices 1 serving as an edge computer. In the photographing system 500", the learning models generated by the respective photographing devices 1 are collected and stored in the fog computer 7. In the photographing system 500", optimization or efficiency is performed based on the plurality of stored learning models, and the optimized or efficient learning models are redistributed to the respective photographing devices 1 as needed.

[0067] In addition, in the "shooting system 500" of the present embodiment, for example, the learning models stored on the fog computer 7 or the learning models optimized or streamlined on the fog computer 7 can also be collected on a higher-level host computer or cloud server. In this case, these learning models are used for intelligent operations of factories or manufacturers of shooting devices 1 (construction and redistribution of more general learning models in the upper-level server, assistance in maintenance operations based on the analysis results of the learning models, analysis of the performance of each shooting device 1, applications developed for new equipment, etc.).

[0068] As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, It is possible to implement in various forms by adding appropriate changes.

[0069] For example, the learning algorithm executed by the machine learning device 100 , the calculation algorithm executed by the machine learning device 100 , the control algorithm executed by the imaging device 1 , etc. are not limited to the above, and various algorithms may be used.

[0070] In addition, in the above embodiment, the photographing device 1 and the machine learning device 100 are described as devices having different CPUs (processors), but the machine learning device 100 can also be implemented by the CPU 11 of the photographing device 1 and the system program stored in the ROM 12.

Claims

1. An imaging device that performs machine learning processing, the machine learning processing involving estimating a distance image related to an object that is closer to reality than the distance image based on a distance image related to the object captured by an imaging sensor, characterized in that: The photographing device comprises: a data acquisition unit that acquires distance image data related to the object; and a preprocessing unit that generates input data based on range image data related to the object, Using the input data, performing machine learning processing for estimating realistic distance image data related to the object based on the distance image data related to the object; The preprocessing unit generates training data using distance image data related to the object as input data and distance image data related to the object that is close to reality as output data. The imaging device further includes a learning unit that performs supervised learning related to the machine learning process based on the training data to generate a learned model for estimating realistic range image data related to the object based on the range image data related to the object; The data acquisition unit acquires brightness image data of the object in addition to the distance image data related to the object. The preprocessing unit receives distance image data and brightness image data about the object as input data.

2. The photographing device according to claim 1, characterized in that: The imaging device further includes an object detection unit configured to detect a position and a posture of the object from the range image data on the object acquired by the data acquisition unit.

3. The photographing device according to claim 2, characterized in that: The imaging device further includes a CAD data storage unit, the CAD data storage unit storing CAD data related to the object. The preprocessing unit generates realistic distance image data of the object based on the CAD data related to the object stored in the CAD data storage unit.

4. The photographing device according to claim 1, characterized in that: The photographing device also has: an object detection unit that detects a position and a posture of the object from the range image data related to the object acquired by the data acquisition unit; and A CAD data storage unit stores CAD data related to the object. The preprocessing unit generates realistic distance image data of the object based on the CAD data on the object stored in the CAD data storage unit and the position and posture of the object detected by the object detection unit.

5. The photographing device according to claim 2, characterized in that: The pre-processing unit generates realistic distance image data of the object based on a distance image of the object captured by an imaging sensor having higher accuracy than the imaging sensor.

6. The photographing device according to claim 1, characterized in that: The photographing device also has: a learning model storage unit storing a learned model for estimating realistic distance image data related to the object based on the distance image data related to the object; as well as The estimating unit estimates realistic range image data about the object from the range image data about the object using the input data and the learned model as the machine learning process.

7. A machine learning processing method for a photographing device, wherein the photographing device performs machine learning processing, the machine learning processing being used to estimate a distance image related to an object that is closer to reality based on a distance image related to the object photographed by a photographing sensor, characterized in that: The machine learning processing method has: The first step is to obtain distance image data related to the object; The second step is to generate input data according to the range image data related to the object; as well as A third step is to perform a machine learning process for estimating realistic distance image data related to the object based on the distance image data related to the object using the input data; The second step is a step of generating training data by using the distance image data related to the object as input data and the distance image data related to the object that is close to reality as output data. In the third step, as the machine learning process, supervised learning is performed based on the training data to generate a learned model for estimating realistic distance image data related to the object from the distance image data related to the object; In the first step, in addition to obtaining the distance image data related to the object, the brightness image data of the object is also obtained. In the second step, distance image data and brightness image data regarding the object are used as input data.

8. The machine learning processing method of the camera according to claim 7, characterized in that: In the third step, as the machine learning processing, the input data and the learned model for inferring the close-to-reality distance image data related to the object based on the distance image data related to the object are used to infer the close-to-reality distance image data related to the object based on the distance image data related to the object.

9. A photographing system, which is a system in which a plurality of devices are connected to each other via a network, characterized in that: The plurality of devices at least include the imaging device according to any one of claims 2 to 4, namely, the first imaging device.

10. The photographing system according to claim 9, characterized in that: The plurality of devices include a computer having a machine learning device, The computer obtains a learning model as a result of at least one of the learnings of the first imaging device, The machine learning device included in the computer performs optimization or efficiency based on the acquired learning model.

11. The photographing system according to claim 9, characterized in that: The plurality of devices include a second camera different from the first camera, The learning result of the first shooting device is shared with the second shooting device.

12. The photographing system according to claim 9, characterized in that: The plurality of devices include a second camera different from the first camera, The data observed by the second imaging device can be used in the learning of the first imaging device via the network.

Images