Generation method and estimation device

Abstract

To make it possible to appropriately estimate a type and a posture of an object using machine learning.SOLUTION: A generation method is provided that acquires first and second images, first information indicating a posture and a type of a subject in the first image, and second information indicating a posture and a type of a subject in the second image, generates a third image by correcting the first image, and generates a trained model configured to estimate a posture and a type of a subject in a fourth image based on the third image, the second image, the first information, and the second information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a generation method and an estimation device. [Background technology]

[0002] Conventionally, a technique for estimating the type and orientation of an object using machine learning is known (see, for example, Patent Document 1). Patent Document 1 provides a detection method including a first estimation step of estimating an inclusion area surrounding an object and the type of the object from an image area of ​​color image data, an extraction step of removing background areas other than the object from the inclusion area using information on the external area of ​​the inclusion area and extracting an object area surrounded by the contour of the object, a generation step of generating a distance distribution image by matching the object area extracted in the extraction step to the image area of ​​distance image data, a calculation step of calculating the position of the object using the distance distribution image generated in the generation step, and a second estimation step of estimating the orientation of the object by matching the type of the object estimated in the first estimation step and the distance distribution image generated in the generation step with a matching model prepared in advance. According to the technique described in Patent Document 1, the type, position, and orientation of an object captured by an imaging unit can be detected with high accuracy within a practical time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-128897 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art leaves room for improvement in order to be able to properly estimate the type and pose of an object using, for example, machine learning.

[0005] An object of the present disclosure is to provide a technology that can appropriately estimate the type and orientation of an object. [Means for solving the problem]

[0006] In a first aspect of the present disclosure, a generation method is provided that acquires a first image and a second image, first information indicating a posture and type of a subject of the first image, and second information indicating the posture and type of the subject of the second image, generates a third image by correcting the first image, and generates a trained model that estimates the posture and type of the subject of a fourth image based on the third image, the second image, the first information, and the second information.

[0007] In addition, a second aspect of the present disclosure provides an estimation device having an acquisition unit that acquires a fourth image, and an estimation unit that estimates a posture and type of a subject of the fourth image based on the fourth image and a trained model, wherein the trained model is generated based on a third image obtained by correcting a first image, a second image, first information indicating the posture and type of the subject of the first image, and second information indicating the posture and type of the subject of the second image. Effect of the Invention

[0008] According to one aspect, the type and orientation of an object can be appropriately estimated. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to an embodiment. [Diagram 2] 10 is a flowchart illustrating an example of processing of the information processing device according to the embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a learning DB (database) according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The principles of the present disclosure are described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the present disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0011] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. <Configuration> The configurations of an information processing device 10 that executes a learning phase according to the embodiment, and an information processing device 20 (estimation device, detection device, inference device) that executes an inference (estimation, detection) phase according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 that executes a learning phase according to the embodiment, and the information processing device 20 that executes an inference phase according to the embodiment. The information processing device 10 and the information processing device 20 may be connected to be able to communicate with each other via a network or the like, or may be the same (integrated) device.

[0013] <<Configuration of information processing device 10>> 1, the information processing device 10 includes an acquisition unit 11, a generation unit 12, and a generation unit 13. Each of these units may be realized by cooperation of one or more programs installed in the information processing device 10 and hardware such as a processor and a memory of the information processing device 10.

[0014] The acquisition unit 11 acquires a data set for machine learning. The data set for machine learning may include a plurality of data sets each of which is a combination of an image captured by a photographing device (camera) and information indicating the posture and type of each subject in the image.

[0015] The generation unit 12 generates an image by correcting the image for machine learning acquired by the acquisition unit 11. The generation unit 13 performs machine learning based on at least a part of the data set for machine learning acquired by the acquisition unit 11 and the image generated by the generation unit 12, thereby generating a trained model that estimates the posture and type of a subject of an image captured by the imaging device.

[0016] <<Configuration of information processing device 20>> 1, the information processing device 20 includes an acquisition unit 21 and an estimation unit 22. These units may be realized by cooperation of one or more programs installed in the information processing device 20 and hardware such as a processor and a memory of the information processing device 20.

[0017] The acquisition unit 21 acquires an image captured by an image capture device. The estimation unit 22 estimates the posture and type of a subject of the image based on the image and a trained model generated by the information processing device 10.

[0018] Based on the estimation results of the posture and type, the information processing device 20 may determine whether a component such as a sensor is properly assembled, for example, in a factory, etc. Also, based on the estimation results of the posture and type, the information processing device 20 may pick up a component transported on a cart, for example, in a factory, etc., with a robot arm, etc.

[0019] <Processing> Next, an example of processing of the information processing device 10 according to the embodiment will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a flowchart showing an example of processing of the information processing device 10 according to the embodiment. Fig. 3 is a diagram showing an example of a learning DB (database) 301 according to the embodiment.

[0020] The process of Fig. 2 may be executed for each record of the learning data, for example. The process of Fig. 2 may be executed a number of times designated by a user (an operator of the information processing device 10), for example. Note that the processes of Fig. 2 may be executed in different orders as long as they are not inconsistent.

[0021] In step S1, the acquisition unit 11 acquires a combination (record) of data associated with one image (or image ID) included in a machine learning dataset. Here, the acquisition unit 11 may acquire, from the learning DB 301, a record that has not yet been used for learning, among the machine learning datasets recorded in the learning DB 301 in FIG.

[0022] Among the machine learning datasets recorded in the learning DB301, an image contained in a record associated with one image ID is an example of a "first image," and an image contained in another record is an example of a "second image."

[0023] In the example of Fig. 3, images (image data) are recorded in the learning DB 301 in association with an image ID. Information indicating the posture (center position, rotation, size) and type is also recorded in association with a combination of the image ID and the in-image subject ID. The image ID is identification information for the image. The in-image subject ID is identification information for each subject in the image.

[0024] The attitude in the present disclosure is, for example, the central position, rotation, and size of the subject in three-dimensional space. The attitude may include information on the rotation of the subject indicated by, for example, roll, pitch, and yaw. The attitude may also include information on the central position of the subject in three-dimensional space relative to the imaging device, and the size of the subject in three-dimensional space. The type is the type of the subject. The type may be, for example, the type of parts (work, work in progress) assembled in a factory. The learning DB301 may be recorded in a storage device inside the information processing device 10, or may be recorded in a recording device outside the information processing device 10.

[0025] The image ID and the image recorded in the learning DB 301 may be recorded in advance by, for example, a user, etc. The information indicating the in-image subject ID, posture, and type recorded in the learning DB 301 may be recorded in advance by, for example, a user, etc.

[0026] Furthermore, the information indicating the in-image subject ID, posture, and type recorded in the learning DB 301 may be information calculated by, for example, AI, etc. In this case, for example, the information for each item may be calculated using a technology that takes a relatively long time to infer the posture, etc.

[0027] The acquisition unit 11 may determine whether or not the size of the image recorded in the learning DB 301 (e.g., the number of vertical pixels×the number of horizontal pixels) is different from the size of the image to be input to a convolutional neural network (CNN) for machine learning. If they are different, the acquisition unit 11 may acquire the image recorded in the learning DB 301 by changing (resizing) it to the size of the image to be input to the CNN.

[0028] Next, the generating unit 12 determines whether or not to modify the image acquired by the acquiring unit 11 (step S2). Here, the generating unit 12 may determine whether or not to modify the image acquired by the acquiring unit 11 based on, for example, a generated random number or the like (randomly). In this case, the generating unit 12 may determine whether or not to modify the image randomly so that the proportion of the image to be modified is approximately a specific proportion. This allows, for example, the same convolutional neural network to perform learning that is more suitable for both the posture estimation and the type estimation of the subject. Note that the specific proportion may be set in advance by a user of the information processing device 10 or the like.

[0029] If it is determined that the image is not to be modified (NO in step S2), the generation unit 12 outputs the unmodified image acquired by the acquisition unit 11 to the generation unit 13 as an image for learning (step S3), and proceeds to processing in step S5.

[0030] On the other hand, when it is determined that the image is to be modified (YES in step S2), the generating unit 12 outputs an image ("third image") obtained by modifying the image ("first image") acquired by the acquiring unit 11 to the generating unit 13 as an image for learning (step S4). Here, the generating unit 12 may generate a third image in which, for example, the average value and standard deviation of at least one of the values ​​indicating the color (e.g., RGB values), the brightness, the saturation, and the hue of each pixel of the first image are changed.

[0031] In this case, the generating unit 12 may generate a third image in which each pixel value of the first image is changed so that the average value and the standard deviation are each a specific value. As a result, a third image is generated in which the color of the object is changed but the contour of the object is made clearer compared to the first image. As a result, for example, even for an image with strong shadows, the type of the object can be estimated with higher accuracy. Also, for example, even for an object that is nearly transparent or an object whose color is close to the color of the object in the background, the type of the object can be estimated with higher accuracy.

[0032] The generating unit 12 may randomly determine the average value and the standard deviation based on random numbers, for example. The value indicating the color of a pixel may be, for example, an RGB (Red, Green, Blue) value. The luminance value of a pixel may be calculated by a formula such as 0.3R+0.6G+0.1B.

[0033] Furthermore, the generating unit 12 may average each pixel value of the first image with a pixel width determined randomly or the like. This allows learning even with an image in which the contour of an object is unclear, for example. Therefore, even if the size (number of pixels) of an image captured by a photographing device is relatively small when estimating a posture or the like in the information processing device 20, estimation can be performed with higher accuracy. Furthermore, the generating unit 12 may change (process) each pixel value of the first image with a motion blur strength determined randomly or the like. This allows estimation with higher accuracy even if the blurring due to vibration of an image captured by a photographing device is relatively large when estimating a posture or the like in the information processing device 20.

[0034] Next, the generating unit 13 performs machine learning using the same convolutional neural network based on the image output by the generating unit 12 and the information indicating the posture and type according to each in-image subject ID included in the record acquired by the acquiring unit 11 (step S5). Here, the generating unit 13 may use, for example, Residual Network (ResNet), EfficientNet, DenseNet, or Fast R-CNN as the convolutional neural network. Here, the generating unit 13 may use the image as training data and learn the posture and type of one or more subjects as a correct answer label. As a result, a trained model is generated that estimates the posture and type of each subject in the fourth image using the same convolutional neural network based on the fourth image captured by the imaging device.

[0035] This also makes it possible to estimate (recognize) the postures and types of multiple subjects (objects) based on a single image without cropping the image, thereby reducing the amount of calculations compared to a technique that performs a process of cropping the subject areas in an image and then performs estimation based on the cropped image.

[0036] <Other> For example, it is possible to machine-learn the posture and type of an object based on an image that has been color-converted to make color boundaries clearer. In this case, since the color (hue) of the object cannot be learned, the accuracy of estimating the type of object may decrease for objects with a different color from that at the time of machine learning.

[0037] In addition, when machine learning is used to estimate the pose and type of an object using an image with the original color that has not been converted, the accuracy of estimating the pose of the object may decrease if the contour of the object is unclear due to shadows or blur. In addition, when estimating the types of multiple objects from one image, the method of cutting out the area in which each subject is captured or the method of calculating the feature region (ROI, Region of Interest) increases the amount of calculation required for the estimation process, which increases the time required.

[0038] For example, in the case of a posture estimation method based on a 3D model, even if the accuracy of posture estimation can be improved, the time required for estimation processing increases due to an increase in the amount of calculation. In addition, in actual situations, the point cloud data itself may be noisy due to occlusion problems or disturbances, resulting in a decrease in accuracy. On the other hand, in the present disclosure, learning is performed in two cases: based on an image with corrected color and based on an image with uncorrected color. This makes it possible to appropriately estimate the type and posture of an object using machine learning.

[0039] <Hardware configuration> Fig. 4 is a diagram showing an example of the hardware configuration of each of the information processing devices 10 and 20 according to the embodiment. In the example of Fig. 4, the information processing devices 10 and 20 (computer 100) include a processor 101, a memory 102, and a communication interface 103. These units may be connected by a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0040] When the program 104 is executed by the processor 101, the memory 102, etc. in cooperation with each other, the computer 100 performs at least some processing of the embodiment of the present disclosure. The memory 102 may be of any type. The memory 102 may be a non-transitory computer-readable storage medium, as a non-limiting example. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as a non-limiting example. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0041] The embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor or other computing device.

[0042] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0043] The program codes for carrying out the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus. When the program codes are executed by the processor or controller, the functions / operations in the flowcharts and / or the block diagrams they implement are performed. The program codes are executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0044] The program can be stored and provided to the computer using various types of non-transitory computer-readable media. The non-transitory computer-readable media include various types of tangible recording media. Examples of the non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, semiconductor memories, and the like. The magnetic recording media include, for example, flexible disks, magnetic tapes, hard disk drives, and the like. The magneto-optical recording media include, for example, magneto-optical disks, and the like. The optical disk media include, for example, Blu-ray disks, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), CD-RW (ReWritable), and the like. The semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (random access memories), and the like. The program may also be provided to the computer by various types of temporary computer-readable media. Examples of the temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0045] <Modification> The information processing devices 10 and 20 may be devices contained in one housing, but the information processing devices 10 and 20 of the present disclosure are not limited to this. Each unit of the information processing devices 10 and 20 may be realized by cloud computing configured by one or more computers, for example.

[0046] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0047] 10. Information processing device 11 Acquisition Department 12 Generation part 13 Generation part 20 Information processing device 21 Acquisition Department 22 Estimation part

Claims

1. Obtaining a first image and a second image, first information indicating a posture and a type of a subject of the first image, and second information indicating a posture and a type of a subject of the second image; generating a third image by modifying the first image; generating a trained model for estimating a posture and a type of a subject in a fourth image based on the third image, the second image, the first information, and the second information; Generation method.

2. generating the third image by modifying at least one of the average value and the standard deviation of the color value, the brightness, the saturation, and the hue of each pixel of the first image; The method of claim 1 .

3. Randomly determining whether the first image or the third image is to be used to generate the trained model; The method of claim 1 or 2.

4. A trained model is generated that estimates a posture and a type of a subject in a fourth image based on the third image, the second image, the first information, and the second information using the same Convolutional Neural Network (CNN). The method of claim 1 or 2.

5. an acquisition unit that acquires a fourth image; An estimation unit that estimates a posture and a type of a subject of the fourth image based on the fourth image and a trained model, The trained model is generated based on a third image obtained by correcting the first image, a second image, first information indicating the posture and type of the subject of the first image, and second information indicating the posture and type of the subject of the second image. Estimation device.

Images