Learning data generation support device, movement controller, article acquisition arrangement system, learning data generation support method, and program

Abstract

To provide a learning data generation support device, a movement controller, an article acquisition arrangement system, a learning data generation support method, and a program with which correct answer data can be obtained with less manual operation.SOLUTION: A method using a learning data generation support device executes: first extraction processing; first determination processing; second determination processing; second extraction processing; third determination processing; and machine learning processing. The first extraction processing extracts a first extraction region corresponding to a detection object from a photographic image. The first determination processing determines reliability of the first extraction region. The second determination processing determines reliability of the first extraction region at a standpoint different from that of the first determination processing. The second extraction processing extracts a second extraction region corresponding to a detection object by using a method different from that of the first extraction processing, from a photographic image including the first extraction region having the reliability of a reference or higher. The third determination processing determines reliability of the second extraction region. The machine learning processing includes the second extraction region having the reliability of a reference or higher, in learning data of a machine learning model for extracting a detection object.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a learning data generation support device, a movement control device, an item acquisition and placement system, a learning data generation support method, and a program. [Background technology]

[0002] Conventionally, there is a technology that uses an arm or the like to appropriately pick up the necessary parts from a pile of parts and send them to the manufacturing assembly process. Image recognition technology is used to recognize the parts. The desired parts are recognized from images captured by a camera, and the position and orientation of the arm are determined based on the part's position and orientation. Machine learning models are effective for this type of image recognition. Machine learning models improve recognition accuracy by learning using captured images of the object to be recognized and ground truth data that indicates the correct shape.

[0003] However, for components positioned in various three-dimensional directions, especially when the shape is complex, accurately generating correct answer data is difficult, requires a huge amount of effort, and is significantly affected by the skill level of the person who creates the correct answer data. Therefore, it is difficult to easily obtain accurate correct answer data. In response to this, Patent Document 1 discloses a technology in which correct answer data is mechanically initially generated and its reliability is determined, and only unreliable correct answer data is manually checked or corrected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-038990 Summary of the Invention [Problem to be solved by the invention]

[0005] However, such corrections ultimately require the manual labor of an experienced person, and there is a problem in that the degree of labor reduction is only small.

[0006] An object of the present invention is to provide a learning data generation support device, a movement control device, an item acquisition and placement system, a learning data generation support method, and a program that can obtain correct data with less manpower. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention provides a first extraction means for extracting a first extraction region corresponding to the detection target from a captured image including the detection target; a first determination means for determining the certainty that the first extracted region is accurate; a second determination means for determining the certainty of the first extracted region, the certainty of which is determined by the first determination means to be equal to or higher than a standard, from a different viewpoint than that of the first determination means; a second extraction means for extracting a second extraction area corresponding to the detection target by a method different from that of the first extraction means from the captured image including the first extraction area for which the second determination means has determined that the certainty is equal to or higher than a standard; a third determination means for determining the certainty that the second extracted region is accurate; a learning data generation means for determining the second extraction area, for which the third determination means has determined that the certainty is equal to or higher than a standard, as learning data for a machine learning model that extracts the detection target; The learning data generation support device includes: [Effects of the Invention]

[0008] According to the present invention, This has the following effect. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an item acquisition and placement system. [Figure 2] FIG. 10 is a diagram illustrating an example of a target part. [Figure 3] FIG. 10 is a diagram showing an example of target parts to be bulk-stacked; [Figure 4]FIG. 1 is a diagram illustrating a flow of extracting learning data and improving a machine learning model. [Figure 5] 10 is a flowchart showing the procedure of an initial learning process. [Figure 6] 10 is a flowchart showing the procedure of a learning data candidate generation process. [Figure 7] 10 is a flowchart showing a control procedure for a learning data generation process. [Figure 8] 10 is a flowchart showing a control procedure for a model learning process. [Figure 9] 10 is a flowchart showing a control procedure of an item placement control process. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing the configuration of an item acquisition and placement system 100 according to this embodiment. This item acquisition and placement system 100 is a pick-and-place system that grasps a target item, a workpiece, from bulk-stacked parts with an arm 62 and moves and places it at a specified position. The item acquisition and placement system 100 includes an information processing device 10 as a movement control device and a learning data generation support device, an imaging device 50, and a moving and placing unit 60 (moving operation mechanism).

[0011] The information processing device 10 is an electronic computer, and may be a normal PC (Personal Computer). The information processing device 10 includes a CPU (Central Processing Unit) 11 (controller), a RAM (Random Access Memory) 12, a storage unit 13, a communication unit 14, a display unit 15, an operation reception unit 16, and the like.

[0012] The CPU 11 is a hardware processor that performs arithmetic processing and controls the overall operation of the information processing device 10. There may be one hardware processor, or multiple processors that operate independently or in parallel depending on the application.

[0013] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data. The RAM 12 is, for example, a DRAM, but is not limited to this.

[0014] The storage unit 13 is a non-volatile memory that stores a program 130 and various setting data. The non-volatile memory may be, for example, a flash memory or a hard disk drive (HDD). The program 130 includes a first extraction unit 131 (first extraction means) and a second extraction unit 132 (second extraction means). The first extraction unit 131 and the second extraction unit 132 are functions, subroutines, software modules, or combinations thereof of the program 130. The first extraction unit 131 includes a trained model 1311. The second extraction unit 132 includes a trained model 1321. The machine learning model 135 is an image recognition program that identifies a detection target part (workpiece) from an image captured by the imaging device 50. The training data 136 is data for training the machine learning model 135. The first judgment criterion 137, the second judgment criterion 138, and the third judgment criterion 139 are data representing conditions for acquiring the training data 136. The training of the machine learning model 135 by the program 130 and the use of the trained machine learning model 135 will be described later.

[0015] The communication unit 14 controls communication with external devices. The control target may be, for example, communication via an internet line, a LAN (Local Area Network), or a wireless LAN. The receiving unit 14 may have a connection terminal for direct communication, such as a USB (Universal Serial Bus) terminal.

[0016] Display unit 15 includes, for example, a digital display screen. Display unit 15 displays various information on the digital display screen under the control of CPU 11. The digital display screen may be, for example, a liquid crystal display screen or an organic EL (Electro-Luminescent) display screen.

[0017] The operation reception unit 16 receives an input operation from outside and outputs an operation signal corresponding to the content of the received input operation to the CPU 11. The operation reception unit 16 may include, for example, a pointing device such as a mouse or a touch panel. The operation reception unit 16 may also include a keyboard.

[0018] The display unit 15 and the operation reception unit 16 may be attached as peripheral devices to the main body of a computer including at least the CPU 11. The storage unit 13 does not have to be built into the computer's housing. The storage unit 13 here may include an external auxiliary storage device, network storage, a cloud server, etc. The RAM 12 and the communication unit 14 may also be attached externally to the main body of the computer as needed.

[0019] The image capturing device 50 captures images of the bulk-stacked parts (items) at a timing instructed by the CPU 11 or periodically at predetermined time intervals, and outputs the captured image data to the CPU 11. The image capturing device 50 is, for example, a digital image capturing device having a CMOS sensor or the like. The image capturing range, magnification, focal length, etc. of the image capturing device 50 may be fixed. Alternatively, the image capturing range of the image capturing device 50 may be movable under the control of the CPU 11.

[0020] The moving and placing unit 60 has a driving unit 61 and an arm 62. The moving and placing unit 60 uses the arm 62 to grasp and acquire a necessary part from a pile of parts, and then moves the part to a specified position and releases it to place it. For example, the moving and placing unit 60 may be capable of placing a part to be mounted on a board on a conveyor belt in an appropriate position and orientation, such as during a product assembly process. The driving unit 61 is a mechanism that moves and operates the arm 62 under the control of the CPU 11. The arm 62 may be one or more. Note that the hardware processor that operates the driving unit 61 may be different from the hardware processor of the CPU 11 that controls the overall operation. Furthermore, this hardware processor may not be a general-purpose processor, but may have a configuration specialized for controlling the operation of the arm 62.

[0021] Next, image recognition of images captured by the image capturing device 50 and the associated learning will be described. The item acquisition and placement system 100 retrieves a target item from bulk items stored in a tray, box, or the like, and places the target item in a predetermined position in the correct orientation. The target item may be, for example, a component (target part) to be assembled during an assembly process. The material of the target item does not deform when the item is normally bulk stacked. The information processing device 10 detects the target part to be detected from the image captured by the imaging device 50 and accurately identifies its outer shape and orientation. Based on the identification result, the information processing device 10 determines the target part to be retrieved and causes the moving and placement unit 60 to perform an operation to retrieve the target part. In bulk stacking, many parts are positioned overlapping each other. Therefore, most of the parts are partially or completely covered by other parts. The arm 62 does not need to dig out such parts. The item acquisition and placement system 100 may identify only the target part located at the top and not covered by other parts as the detection target.

[0022] 2 and 3 are diagrams showing examples of target parts to be bulk loaded. As shown in Fig. 2, the target part P may have a complex shape with irregularities and holes depending on its function and its positional relationship with other parts. A large number of parts including such target parts P are piled up in bulk as shown in Fig. 3.

[0023] The target part may be in any orientation within the bulk pile. The target part may also be tilted relative to the imaging surface. In particular, the target part may be upside down. Depending on these, the part of the target part that is illuminated by external lighting, etc., and the part from which reflected light is directed toward the imaging device 50, may change. Taking these into consideration, the CPU 11 accurately recognizes the position and orientation of the target part, grasps the appropriate position of the target part with the arm 62, rotates the target part to the correct orientation, and mounts it in the specified position. Note that the target part is not limited to one type. Multiple types of parts may be set as target parts, and each may be mounted in a specified position.

[0024] The information processing device 10 uses a machine learning model 135 to recognize target parts from captured images. For the machine learning model 135 to accurately recognize target parts, it must be properly trained. The training data 136 includes captured images of bulk piles and correct answer data, which is a contour mask indicating the area of ​​the target parts in the captured images. The correct answer data may also include additional information indicating whether the front or back side of the target parts is being photographed. Furthermore, in the case of multiple types of target parts, as described above, the additional information includes identification information for the multiple types of target parts. Conventionally, correct answer data, i.e., annotations, are obtained by manual input operations on images. However, accurately tracing the contours of parts with complex shapes is often difficult. In particular, the task of tracing the contours of bulk piles while reliably distinguishing them from other parts requires experience and attention. Therefore, manually generating the training data 136 requires a great deal of effort. In this embodiment, correct answer data is generated with less manual effort than conventional methods.

[0025] The information processing device 10 initially generates simple trained models 1311 and 1321. Both trained models 1311 and 1321 have an image recognition algorithm related to region segmentation that extracts the range of the target part. This image recognition algorithm outputs a probability distribution of the range of the target part, and the range of the target part is obtained by binarizing the probability distribution. Conditions related to extraction accuracy are applied to the output results of these trained models 1311 and 1321, and training data 136 for the target part is collected. By generating and improving a machine learning model 135 using the collected training data 136, the detection accuracy of the target part by the machine learning model 135 is improved. Furthermore, as the trained models 1311 and 1321 are successively improved, the collection accuracy of the training data 136 is also improved.

[0026] It would be pointless to take time and effort to generate the learning data for training and generating these trained models 1311 and 1321. The initial model of the trained model 1311 used to generate the initial learning data may be obtained, for example, based on images of a single target part (detection target) photographed from multiple directions. In such a simple state, correct data can be easily obtained by using, for example, a simple contour detection algorithm. At this stage, accurate detection of the target part from the bulk pile is not required. Note that if the trained model 1311 is capable of extracting the ranges of multiple types of target articles, it is sufficient to perform learning based on photographed images of a single target article for each of the multiple types of target articles.

[0027] The photographed image of a single target part may be generated outside the item acquisition and placement system 100 and input to the information processing device 10. Alternatively, if the photographing direction of the photographing device 50 is changeable, the photographing device 50 may photograph a single target part while sequentially changing the orientation of the target part to a preset orientation using the arm 62.

[0028] The trained model 1321 may be an initial model that has been trained using, for example, transfer learning. The original trained model may be a model that has been trained to enable some kind of region division (segmentation). The original region division target of the original trained model is not particularly limited, but materials or objects that are as similar as possible may be selected to easily avoid negative transfer. Regardless of the above, transfer learning may also be applied to the trained model 1311.

[0029] The trained model 1321 may be capable of determining not only the range of the target object but also the pattern within the outline. The specific pattern may be set based on a user's natural language input. The trained model 1321 may be sequentially improved and trained using the obtained training data 136. This gradually improves the accuracy of the trained model 1321 in detecting the target part.

[0030] FIG. 4 is a diagram illustrating the flow of extracting learning data and improving the machine learning model 135. First, data of a target image is prepared. The target image is an image of a bulk pile at a predetermined position photographed by the photographing device 50. However, in the initial stage, the target image data may be an image prepared specifically for learning, as will be described later.

[0031] Data of a target image is input to the first extraction unit 131 in the first extraction process P1. As described above, the first extraction unit 131 has a trained model 1311. In the first extraction process P1, the range of the target part in the target image (first extraction area) is output together with its reliability. The reliability may be a conventionally known reliability score. The first determination process P2 (first determination means) excludes target images for which the range of the detected target part is unlikely to be detected, based on a first determination criterion 137. The first determination criterion 137 may be a lower limit of an acceptable reliability score, for example. Furthermore, the second determination process P3 (second determination means) excludes target images that are unlikely to be the detected target part, from among the target images not excluded in the first determination process P2, based on a second determination criterion 138.

[0032] The second judgment criterion 138 determines whether the detection of the target part meets or exceeds a certainty standard based on a different perspective from the first judgment criterion 137, particularly a perspective unrelated to a machine learning model. For example, the second judgment criterion 138 may be determined based on human-recognizable visual shape features, such as the size and contour shape of an extracted area in a captured image. In other words, the visual shape features referred to here do not refer to multidimensional features of four or more dimensions obtained using a neural network or the like. As described above, when photographing bulk piles, the positional relationship between the camera 50 and the bulk pile, the photographing range and magnification of the camera 50, etc. are generally fixed, so the size of the photographed parts is also approximately constant. Therefore, differences in actual size can be distinguished based on the size in the captured image. Even if the magnification rate is changed, differences in actual size can be distinguished depending on the magnification rate. Furthermore, visual shape features may include, for example, the number of corners (vertices), shape features of curves, angular widths of corners and arcs, height information of protruding parts, and contour features such as the positional relationship of these shape parts. The similarity of these features as a whole may be determined, for example, by pattern matching. Furthermore, the visual shape characteristics may include the shape, number, and positional relationship of unevenness and holes obtained by edge detection within the contour, etc. The image used as the object of judgment in such second judgment process P3 may be obtained in advance from a photographed image of one target part or a processed image thereof.

[0033] The target image that is not excluded in the first determination process P2 or the second determination process P3 is set as an intermediate candidate. The intermediate candidate image may be a range of a predetermined size (predetermined range) that includes a portion of the target image that has been subjected to the first extraction process P1 and in which it has been determined that the target part has been detected. The predetermined size may be determined according to the size of the input image to the second extraction unit 132 in the second extraction process P4. The predetermined range may be determined, for example, based on the position of the center of gravity of the range of the extracted target part. The position of the center of gravity is determined with equal weighting for all pixels within the range of the target part. Alternatively, the centroid position may be determined relative to a pixel on the contour of the target part. The centroid position may be included in a predetermined range, and in particular, may be the center position thereof.

[0034] The intermediate candidate image is further used as an input image to be input to the second extraction unit 132 in the second extraction process P4. The second extraction unit 132 has the trained model 1321 as described above. The trained model 1321 is an image segmentation model having a structure different from that of the trained model 1311. The trained model 1321 may use the same algorithm as the trained model 1311 but with a different number of layers. The trained model 1321 has a larger number of layers than the trained model 1311. Therefore, the extraction accuracy of the second extraction unit 132 can be higher than the extraction accuracy of the first extraction unit 131, but a greater amount of learning is required to improve the accuracy of extracting target parts from bulk piles.

[0035] In third determination process P5 (third determination means), examples in which there is little certainty that the range of the target part (second extraction area) extracted by second extraction unit 132 in second extraction process P4 is accurate are excluded based on third determination criterion 139. Third determination criterion 139 may be a reference value of the reliability score, similar to first determination criterion 137, or may be a value different from first determination criterion 137. Information on the remaining target images and the range of the target parts in the target images is used as learning data 136. First determination criterion 137, second determination criterion 138, and third determination criterion 139 may be variable depending on the situation.

[0036] The trained model 1311 of the first extraction unit 131 has the same structure as the machine learning model 135. When machine learning of the machine learning model 135 is performed (P6), the trained model 1311 is also updated with the obtained model. This gradually improves the trained model 1311, and the detection accuracy of the range of the target part by the machine learning model 135 also improves within the generated range of the training data 136.

[0037] As described above, in this embodiment, a two-stage judgment is performed, and the judgment criteria include a second judgment criterion 138 that does not directly depend on the trained model. As a result, the trained model obtained by training the machine learning model 135 using the training data 136 for which correct answer data is mechanically determined has higher accuracy than the individual trained models 1311 and 1321. In particular, the training accuracy of the machine learning model 135 is further improved by performing a set of a process of updating the trained model 1311 using the training results of the machine learning model 135 and a process of generating the training data 136 multiple times. As described above, the trained model 1321 is larger in scale than the trained model 1311, and therefore the accuracy of the trained model 1311 initially improves more quickly. On the other hand, when the updates of the trained models 1311 and 1321 and the generation of the training data 136 are repeated, the accuracy of the trained model 1321 significantly improves. Accordingly, more appropriate training data 136 is obtained. As a result, the training accuracy of the machine learning model 135 is also likely to improve.

[0038] 5 is a flowchart showing the procedure of the initial learning process for obtaining the initial trained model 1311. Here, a case where the initial learning process is performed in the item acquisition and placement system 100 will be described.

[0039] CPU 11 causes arm 62 to grasp one target part (S1). CPU 11 causes arm 62 to hold the target part at a set position and orientation using drive unit 61 (S2). At this time, the background of the target part may be a plain, monochromatic plane with no pattern, for example. The orientation may be determined, for example, by rotating the target part by a unit angle within a certain angle range with respect to a first axis direction perpendicular to the imaging surface and a second axis direction parallel to the imaging surface. As an example, the angle range with respect to the first axis direction may be ±45 degrees, and the angle range with respect to the second axis direction may be ±30 degrees, with the unit angle being 5 degrees, for example. CPU 11 causes imaging device 50 to capture an image of the target part while changing the orientation of the target part in this way, and acquires the captured image. Gain (S3).

[0040] The CPU 11 detects the range of the target part from the captured image and acquires a contour mask (S4). As described above, the range of the target part may be detected using any of a variety of simple detection algorithms that are different from machine learning models. The interior of the detected closed boundary is set as a contour mask. The CPU 11 generates initial learning data from a pair of the captured image and the contour mask (S5).

[0041] The CPU 11 determines whether the target part has been photographed in all of the set directions (S7). If it is determined that the target part has not been photographed in all directions, that is, that there is an angular direction in which the target part has not been photographed (S6; N), the processing by the CPU 11 returns to step S2. If it is determined that the target part has been photographed in all of the set directions (S6; Y), the CPU 11 performs initial training on the machine learning model of the first extraction unit 131 using the obtained data set of multiple initial training data (S7).

[0042] The CPU 11 evaluates the error of the trained machine learning model and determines whether the error is equal to or less than the standard (S8). If it is determined that the error is not equal to or less than the standard (S8; N), the processing of the CPU 11 returns to step S1. That is, the CPU 11 additionally photographs another target part and generates initial learning data.

[0043] If it is determined that the error is equal to or smaller than the reference value (S8; Y), the CPU 11 registers the obtained trained model as the trained model 1311 of the first extraction unit 131 (S9). Then, the CPU 11 ends the initial training process.

[0044] 6 is a flowchart showing the procedure of the learning data candidate generation process, which corresponds to the first extraction process P1, the first determination process P2, and the second determination process P3 described above.

[0045] The CPU 11 acquires a bulk image including target parts (S11). The CPU 11 detects the target parts from the acquired bulk image using the first extraction unit 131, and acquires the detection range and its reliability score (S12; first extraction means). The CPU 11 determines whether the reliability score satisfies the first determination criterion 137 (S13; first determination means).

[0046] If it is determined that the reliability score does not satisfy the first criterion 137 (S13; N), the CPU 11 excludes the detection range from the intermediate candidates (S20). Then, the processing of the CPU 11 proceeds to step S17. If it is determined that the reliability score satisfies the first criterion 137 (S13; Y), the CPU 11 analyzes the image of the detection range and calculates a parameter according to the second criterion (S14). As described above, the parameter may be, for example, a value related to the size of the detection range, such as the number of pixels. The CPU 11 determines whether the detection range satisfies the second criterion (S15; second determination means).

[0047] If it is determined that the detection range does not satisfy the second criterion (S14; N), the CPU 11 proceeds to step S20. If it is determined that the detection range satisfies the second criterion (S14; Y), the CPU 11 sets an image area of ​​a predetermined size that includes the detection range as an intermediate candidate (S16). Then, the CPU 11 proceeds to step S17.

[0048] In step S17, the CPU 11 determines whether all prepared bulk pile images have been acquired (S17). If it is determined that all bulk pile images have not been acquired, that is, that there are images that have not been acquired (S17; N), the processing of the CPU 11 returns to step S11. If it is determined that all bulk pile images have been acquired (S17; Y), the CPU 11 sets a list of the set intermediate candidates and stores and holds them together (S18). Then, the CPU 11 ends the learning data candidate generation process.

[0049] 7 is a flowchart showing the control procedure for the learning data generation process. This process is performed after the learning data candidate generation process. This process corresponds to the second extraction process P4 and the third determination process P5 described above.

[0050] The CPU 11 acquires an image of an intermediate candidate from the intermediate candidate list (S21). The CPU 11 causes the second extraction unit 132 to detect the range of the target part from the acquired image of the intermediate candidate, and acquires its reliability score (S22; second extraction means).

[0051] The CPU 11 determines whether the reliability score satisfies the third criterion 139 (S23; third determination means). If it is determined that the reliability score does not satisfy the third criterion 139 (S23; N), the processing of the CPU 11 proceeds to step S25. If it is determined that the reliability score satisfies the third criterion 139 (S23; Y), the CPU 11 sets the detected range as the correct answer data as a contour mask. The CPU 11 associates this correct answer data with the intermediate candidate image and sets it as learning data (S24). Then, the processing of the CPU 11 proceeds to step S25.

[0052] In step S25, the CPU 11 determines whether all images of the intermediate candidates have been acquired (S25). If it is determined that not all images of the intermediate candidates have been acquired, that is, that there are images of the intermediate candidates that have not been acquired (S25; N), the processing of the CPU 11 returns to step S21. If it is determined that all images of the intermediate candidates have been acquired (S25; Y), the CPU 11 generates a training data set that compiles the set sets of training data (S26). Then, the CPU 11 ends the training data generation processing. Steps S24 and S26 correspond to the learning data generating means of this embodiment.

[0053] 8 is a flowchart showing the control procedure of the model learning process for learning the machine learning model 135. This process is executed after the learning data set is generated by the learning data generation process.

[0054] The CPU 11 acquires unacquired learning data from the learning dataset (S31). The CPU 11 inputs images of the acquired learning data into the machine learning model 135 and acquires the output result (S32). The CPU 11 compares the acquired result with the correct answer data to calculate an error (S33).

[0055] The CPU 11 feeds back the error to the parameters of the machine learning model 135 (S34). An appropriate loss function may be used to calculate the error. The feedback may be performed by a conventionally known method, for example, by backpropagation.

[0056] The CPU 11 determines whether all the learning data has been acquired (S35). If it is determined that all the learning data has not been acquired, that is, that there is learning data that has not been acquired (S35; N), the processing of the CPU 11 returns to step S31. If it is determined that all the learning data has been acquired (S35; Y), the CPU 11 sets the trained machine learning model 135 as a trained model (S36). The CPU 11 updates the trained model 1311 of the first extraction unit 131 with this trained model (S37). Then, the CPU 11 ends the model learning process.

[0057] 9 is a flowchart showing the control procedure of the item placement control process using the obtained trained model. This process is used in the actual operation of the item acquisition and placement system 100.

[0058] The CPU 11 causes the photographing device 50 to photograph the bulk pile and acquire the photographed image (S41). The CPU 11 inputs the acquired photographed image into the trained model and acquires the detection result of the range of the target part (S42).

[0059] The CPU 11 determines whether or not a target part has been detected (S43). If it is determined that a target part has been detected (S43; Y), the CPU 11 selects one of the detected target parts and sets the position and orientation of the part to be gripped by the arm 62 according to the position and orientation of the selected target part (S44). The CPU 11 causes the drive unit 61 to operate the arm 62 in accordance with the settings, thereby causing the arm 62 to acquire the selected target part. The CPU 11 causes the arm 62 to place the acquired target part at the set position in the set orientation (S45).

[0060] CPU 11 acquires information on the state of the target parts after they have been moved to their set positions, and determines whether the target parts have been properly placed (S46; state determination means). CPU 11 may, for example, be equipped with a photographing device that photographs the set positions. CPU 11 stores the determination result in storage unit 13 as a placement history (S47). Note that target parts that have not been placed accurately may be corrected and placed in another process. Furthermore, a captured image of the parts piled up in a case where they have not been placed accurately may be stored in storage unit 13 as a determination failure image. This determination failure image may be included in the images acquired in step S11 of the repeated learning data candidate generation process until first determination criterion 137 to third determination criterion 139 are satisfied. Then, the process of CPU 11 returns to step S41.

[0061] If it is determined in the determination process of step S43 that no target part has been detected (S43; N), CPU 11 causes display unit 15 or another notification unit to perform an announcing operation indicating that there is no target part to be acquired (S48). CPU 11 analyzes the placement history stored in memory unit 13, calculates the accuracy of the trained model, and determines and evaluates its quality (S49; accuracy determination means). The evaluation result is stored in memory unit 13. The evaluation result may also be displayed and output by display unit 15, or may be output as data or printed externally as a report. Then, CPU 11 ends the item placement control process.

[0062] As described above, the information processing device 10 serving as the learning data generation support device of this embodiment includes a CPU 11. The CPU 11 extracts a first extracted region corresponding to the detection target from a captured image containing the detection target using the first extraction unit 131 of the program 130. The CPU 11, as a first determination means, determines the certainty that the first extracted region is accurate. The CPU 11, as a second determination means, determines the certainty of a first extracted region determined to have a certainty equal to or higher than a standard from a perspective different from that used by the first determination means. The CPU 11 extracts a second extracted region corresponding to the detection target from a captured image containing a first extracted region determined to have a certainty equal to or higher than a standard by the second extraction unit 132 of the program 130 using a method different from that used by the first extraction unit 131. Here, the structure of the second extraction unit 132 is different from that of the first extraction unit 131. The CPU 11, as a third determination means, determines the certainty that the second extracted region is accurate. The CPU 11, as the learning data generating means, determines the second extraction area, the third determination means having determined that the certainty is equal to or higher than the standard, as learning data for a machine learning model that extracts the detection target. Therefore, the information processing device 10 can obtain correct answer data with less human labor. As a result, the information processing device 10 can easily achieve the same level of accuracy for the correct answer data. Therefore, learning data can be generated more easily, and the machine learning model can be trained appropriately.

[0063] Furthermore, the CPU 11 as the second determination means may determine the reliability of the first extracted region based on the visual shape characteristics of the detection target. That is, for the trained model 1311 at a stage where learning is insufficient, another determination method that does not use a machine learning model is used in combination. In this way, the information processing device 10 can equalize the level of discrimination accuracy while reducing the decrease in discrimination accuracy of the target part. It is possible.

[0064] Furthermore, the CPU 11, which serves as the second determination means, may determine the reliability of the second extraction region based on the size of the object to be detected. If the object to be detected is an item of a fixed size, size information can be an important determination criterion. However, since the three-dimensional orientation of the object to be detected is not fixed, the captured image may be slightly tilted, or may partially overlap with other objects. Therefore, the information processing device 10 can easily and clearly exclude first extraction regions of clearly different sizes as erroneous extractions based on the size of the detected object in the captured image.

[0065] Furthermore, the CPU 11, which serves as the second determination means, may determine the reliability of the second extraction region using pattern matching or edge detection of the shape of the object to be detected. Because the object to be detected is fixed, the contour and the geometric features within the contour are fixed. Therefore, by using a technology that easily extracts these features, the information processing device 10 can add information about clear structural differences to the extraction results of a trained model that has not yet been trained.

[0066] Furthermore, in the processing by the first extraction unit 131, the CPU 11 may use a machine learning model having the same structure as the machine learning model 135. The CPU 11 extracts the first extraction region based on the probability distribution to be detected using the trained model 1311 of this machine learning model. The CPU 11, acting as the first determination means, may determine the certainty of the first extraction region based on the obtained probability distribution. That is, the information processing device 10 can obtain more accurate learning data by using another method in combination with the trained model 1311 at a certain stage of the machine learning model 135 to be learned. This makes it possible to improve the accuracy of the machine learning model 135 compared to the original trained model 1311 without manual intervention in generating correct answer data.

[0067] Furthermore, CPU 11 as the first extraction means may obtain a trained model of machine learning model 135 trained using acquired training data 136, and set it as trained model 1311. CPU 11 may extract the first extraction region using trained model 1311. That is, by generating training data 136 and training machine learning model 135, the trained model 1311 is improved to have higher accuracy without requiring human intervention to generate correct answer data.

[0068] Furthermore, the information processing device 10 of this embodiment may generate training data and train the machine learning model multiple times. The CPU 11, serving as the first extraction means, may acquire a trained model for each training of the machine learning model 135 and update the trained model 1311. When extracting the first extracted region for the first time, an initial model trained based on a captured image of a single detection target may be used. Even when gradually improving the trained model 1311, it is necessary to prepare an initial trained model 1311 before training the machine learning model 135. In the present disclosure, since the target object is fixed from the beginning, the target object can be trained to a level where it can be detected independently without machine learning, rather than being piled up in bulk. At this level of training, correct answer data can be obtained through simple image processing without manual intervention, thereby improving the training accuracy of the machine learning model 135 without manual intervention for generating correct answer data.

[0069] Furthermore, the visual shape features may be determined based on a captured image of a single detection target. That is, the visual shape features used for the second judgment criterion 138, which is different from the trained model, may be determined based on a captured image of a single detection target article. This allows the visual shape feature criteria to be obtained with high accuracy and minimal human intervention.

[0070] Furthermore, the CPU 11 as the second extraction means may determine, as the input image, a range based on at least the center of gravity of the first extraction area in the captured image. The second extraction unit 132 may extract an appropriate range including the target item from the bulk image and learn it. At this time, the size of the extracted image is determined based on the center of gravity of the target item, so that the size of the extracted image can be easily determined. An appropriate image range can be determined easily and accurately regardless of the orientation or tilt.

[0071] Furthermore, the CPU 11 as the first extraction means may extract a first extraction area and set a classification for the detection target. In this case, the CPU 11 as the second extraction means may extract a second extraction area and set a classification for the detection target. The information processing device 10 can not only simply identify the range of the target item from the image, but also obtain type information such as the condition of the target item as additional information. This allows the information processing device 10 to improve the accuracy of identifying the target item from bulk piles.

[0072] The classification may also include information about the front and back of the detected object. In bulk loading, the target object may be photographed both on its front and its back. It may be necessary to identify the front and back so that the movement placement unit 60 operates differently depending on which side is located on the front side. The additional information indicating the classification of the front and back allows the information processing device 10 to more easily and appropriately handle the target object related to the extracted area.

[0073] Furthermore, the CPU 11 may be capable of extracting extraction areas relating to multiple types of detection targets using the first extraction unit 131 and the second extraction unit 132. In this case, the classification may include multiple types of identification information. That is, the target items to be detected may not be of one type. In this case, by including information identifying the extracted target items as additional information, the information processing device 10 can more easily and appropriately handle the target items relating to the extracted areas.

[0074] Furthermore, the information processing device 10 as a movement control device of this embodiment includes a trained model obtained by learning using the generated learning data described above, and a CPU 11. The CPU 11 extracts the detection target item using the trained model obtained by training the machine learning model 135 from a photographed image of a bulk pile of multiple items photographed by the photographing device 50. The CPU 11 then causes the moving and arranging unit 60 to acquire the items related to the extracted area from the bulk pile and move them to a specified position. This information processing device 10 can accurately obtain the bulk piled target items, including information on their position and orientation. Therefore, the information processing device 10 can more appropriately control the operation of the moving and arranging unit 60, which actually moves the target items, while reducing the effort required for training the machine learning model 135.

[0075] Furthermore, the CPU 11 of the information processing device 10 may function as a state determination means to acquire information about the state of the target item after it has been moved by the moving and arranging unit 60. The CPU 11 may also determine whether the state of the moved target item, i.e., whether its position, orientation, or front and back is appropriate. The CPU 11 may also function as an accuracy determination means to determine the accuracy of the machine learning model 135 after learning based on the above-mentioned determination result. By making such a determination during operation, the information processing device 10 can determine whether the machine learning model 135 has been learned with a practically usable accuracy and determine whether improvement is necessary.

[0076] The item acquisition and placement system 100 of this embodiment also includes an information processing device 10 as the movement control device, a photographing device 50, and a moving and placing unit 60. The photographing device 50 photographs the bulk pile to obtain a photographed image. The moving and placing unit 60 acquires the target item from the bulk pile and moves it to a specified position in accordance with the control of the information processing device 10 based on the photographed image. Such an item acquisition and placement system 100 can more easily improve the accuracy of moving the target item from the bulk pile to the desired position.

[0077] The item acquisition and placement system 100 of this embodiment includes an information processing device 10, a moving and placing unit 60, and a photographing device 50. The information processing device 10 includes a CPU 11 and a machine learning model 135 trained using the training data 136 generated as described above. U11 extracts a target item from an image of a bulk pile of multiple items captured by the camera device 50 using the machine learning model 135. CPU 11 causes the arm 62 to acquire the extracted item from the bulk pile and move it to a specified position. CPU 11 may cause the camera device 50 to capture the target item acquired from the bulk pile while changing the orientation of the target item relative to the camera device 50 using the moving and arranging unit 60. CPU 11 may obtain an initial model of the trained model 1311 based on the acquired images captured in multiple orientations. The item acquisition and placement system 100 is capable of aligning the orientation of each item using the arm 62. Utilizing this, CPU 11 may also be able to cause the camera device 50 to capture the target item while changing its orientation. This allows the item acquisition and placement system 100 to easily obtain training data for generating an initial model of the trained model 1311.

[0078] The training data generation support method of this embodiment also includes the following steps: (1) a first extraction step of extracting a first extracted region corresponding to the detection target from a captured image containing the detection target; (2) a first determination step of determining the accuracy of the first extracted region; (3) a second determination step of determining the accuracy of a first extracted region determined by the first determination means to be at or above a certain level from a different perspective than that used by the first determination means; (4) a second extraction step of extracting a second extracted region corresponding to the detection target from a captured image including the first extracted region determined to be at or above a certain level by the second determination means, using a different method than that used by the first extraction means; (5) a third determination step of determining the accuracy of the second extracted region; and (6) a training data generation step of defining the second extracted region determined by the third determination means to be at or above a certain level as training data for a machine learning model that extracts the detection target. This training data generation support method makes it possible to obtain correct training data for a machine learning model 135 that extracts target items from bulk loads with less human labor. As a result, the accuracy level of the correct data in the training data can be easily aligned. Therefore, by using this training data generation support method, training data can be more easily generated and the machine learning model can be trained appropriately.

[0079] The program 130 relating to the training data generation support method can be easily installed on a computer. By executing the program 130, it is possible to obtain the correct answer data for the training data 136 through software processing using a normal electronic computer with less human labor.

[0080] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the first extraction unit 131 may not have a machine learning model in the initial state. Once the machine learning model 135 to be learned is learned and a trained model is obtained, the trained model may be set as the trained model 1311 in the first extraction unit 131.

[0081] Furthermore, the learning data for the initial model of the trained model 1311 may be obtained outside the item acquisition and placement system 100. Alternatively, the initial model of the trained model 1311 itself may be generated outside the item acquisition and placement system 100.

[0082] Furthermore, although the target item has been described above as being a part of some configuration, the present invention is not limited to this. The above technology may be used for sorting or aligning target items. It may also be used outside of manufacturing sites such as factories. Alternatively, the detection target of the machine learning model 135 may not be an item in a captured image. Furthermore, the captured image may not be a visible light image.

[0083] In addition, although the above description has been given with an example of extracting target objects having a plate-like front and back, the present invention is not limited to this. Objects may have a more three-dimensional structure. Alternatively, if the front and back structures are symmetrical and there is no need to distinguish between the front and back, classification of the front and back may not be performed.

[0084] In the above, the image range based on the center of gravity of the first extraction area is set as the input range of the second extraction means. However, the reference position and range may be determined so as to appropriately include the first extracted region.

[0085] Furthermore, the training and updating of the trained model 1321 may be performed at a frequency or timing different from that of the training and updating of the trained model 1311. The training and updating may be performed when the necessary training data is collected.

[0086] The second determination process P3 may be performed using a method other than pattern matching or edge detection.

[0087] In the above description, the reliability score is used as the judgment criterion for the first and third judgment units, but this is not limiting, and other parameters may be used to judge the reliability.

[0088] 9 may be performed by a configuration external to the item acquisition and placement system 100. That is, an imaging device and an information processing device for quality evaluation may exist separately from the item acquisition and placement system 100 to perform quality evaluation.

[0089] In addition, in the item acquisition and placement system 100, the information processing device 10, the photographing device 50, and the moving and placing unit 60 may be integrated, or may be a combination of originally separate configurations. The information processing device 10 may be used in a mobile environment, such as Jetson (registered trademark). In addition, the information processing device 10 may have different configurations as a learning data generation support device and a movement control device.

[0090] In the above description, the storage unit 13, which is composed of a nonvolatile memory such as an HDD or flash memory, has been used as an example of a computer-readable medium for storing the program 130 related to the generation control of supervised data of the present invention, but this is not limited to this. Other computer-readable media that can be used include other nonvolatile memories such as MRAM, and portable recording media such as CD-ROMs and DVD discs. Furthermore, a carrier wave can also be used as a medium for providing the program data of the present invention via a communication line. In addition, the specific configurations, contents and procedures of the processing operations, etc. shown in the above embodiments can be modified as appropriate without departing from the spirit of the present invention. The scope of the present invention includes the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0091] 10. Information processing equipment 11 CPU 12 RAM 13 Storage section 130 Programs 131 1st extraction part 132 Second extraction part 1311, 1321 trained models 135 Machine Learning Models 136 training data 137 First judgment criterion 138 Second criterion 139 Third criterion 14 Communications Department 15 Display section 16 Operation reception section 50 Imaging equipment 60 Moving placement part 61 Drive unit 62 Arm 100 Item Acquisition and Placement System P Target parts

Claims

1. a first extraction means for extracting a first extraction region corresponding to the detection target from a captured image including the detection target; a first determination means for determining the certainty that the first extracted region is accurate; a second determination means for determining the certainty of the first extracted area, the certainty of which is determined by the first determination means to be equal to or higher than a standard, from a different viewpoint than that of the first determination means; a second extraction means for extracting a second extraction area corresponding to the detection target by a method different from that of the first extraction means from the captured image including the first extraction area for which the second determination means has determined that the certainty is equal to or higher than a standard; a third determination means for determining the certainty that the second extracted region is accurate; a learning data generation means for determining the second extraction area, for which the third determination means has determined that the certainty is equal to or higher than a standard, as learning data for a machine learning model that extracts the detection target; A learning data generation support device comprising:

2. 2. The learning data generation support device according to claim 1, wherein the second determination means determines the certainty of the first extracted region based on visual shape characteristics of the detection target.

3. The learning data generation support device according to claim 1 , wherein the second determination means determines the certainty of the second extracted region based on a size of the detection target.

4. 2. The learning data generation support device according to claim 1, wherein the second determination means determines the certainty of the second extracted region by using pattern matching or edge detection of the shape of the detection object.

5. the first extraction means extracts the first extraction region based on a probability distribution of the detection target using a machine learning model having the same structure as the machine learning model; the first determination means determines the certainty of the first extracted region based on the probability distribution.

2. The learning data generation support device according to claim 1.

6. 6. The learning data generation support device according to claim 5, wherein the first extraction means acquires a trained model of the machine learning model trained using the learning data, and extracts the first extraction region using the trained model.

7. A set of generating the learning data and learning the machine learning model is performed multiple times; 7. The learning data generation support device according to claim 6, wherein the first extraction means acquires and updates the trained model for each learning, and when extracting the first extraction area for the first time, uses an initial model trained based on a photographed image of one of the detection targets.

8. the second determination means determines the certainty of the first extracted region based on visual shape characteristics of the detection object; The shape feature is determined based on a captured image of the one detection target.

8. The learning data generation support device according to claim 7.

9. 2. The learning data generation support device according to claim 1, wherein said second extraction means takes as an input image at least a range of said photographed image based on the position of the center of gravity of said first extraction area.

10. The first extraction means extracts the first extraction region and classifies the detection target. Set The learning data generation support device according to claim 1 , wherein the second extraction means extracts the second extraction region and sets a classification related to the detection target.

11. The learning data generation support device according to claim 10 , wherein the classification includes information on the front and back of the detection object.

12. the first extraction means and the second extraction means are capable of extracting extraction regions relating to a plurality of types of the detection target, and the classification includes the plurality of types of identification information; The learning data generation support device according to claim 10.

13. A trained model obtained by training using training data generated by the training data generation support device according to any one of claims 1 to 12, and a control unit, The control unit extracts the detection target item from a photographed image of a bulk pile of multiple items photographed by an imaging device using the trained model, and causes a moving operation mechanism to obtain the extracted item from the bulk pile and move it to a specified position. Movement control device.

14. a state determination means for obtaining information about the state of the movement and determining whether the state is appropriate for the detection target; an accuracy determination means for determining the accuracy of the trained model based on the determination result of the state determination means; The movement control device of claim 13, comprising:

15. A movement control device according to claim 13; an imaging device that captures an image of the bulk pile; a movement operation mechanism that acquires the detection target item from the bulk pile and moves it to the specified position in accordance with control of the movement control device based on the captured image; An item acquisition and placement system comprising:

16. a learned model obtained by learning using the learning data generated by the learning data generation support device according to claim 7; and a control unit; The control unit extracts the detection target item from a photographed image of a bulk pile of multiple items photographed by an imaging device using the trained model, and causes a moving operation mechanism to obtain the extracted item from the bulk pile and move it to a specified position. A movement control device; an imaging device that captures an image of the bulk pile; a movement operation mechanism that acquires the detection target item from the bulk pile and moves it to the specified position in accordance with control of the movement control device based on the captured image; Equipped with the control unit obtains the initial model based on a captured image obtained by causing the photographing device to photograph the object to be detected acquired from the bulk pile while changing the orientation of the object with respect to the photographing device using the moving operation mechanism; Item acquisition and placement system.

17. a first extraction step of extracting a first extraction region corresponding to the detection target from a captured image including the detection target; a first determining step of determining the certainty that the first extracted region is accurate; a second determination step of determining the certainty of the first extracted area, the certainty of which is determined to be equal to or higher than a standard in the first determination step, from a different viewpoint than that in the first determination step; a second extraction step of extracting a second extraction area corresponding to the detection target by a method different from that of the first extraction step from the captured image including the first extraction area for which the certainty is determined to be equal to or higher than a standard in the second determination step; a third determining step of determining whether the second extracted region is accurate; a learning data generation step of determining the second extraction area, for which the certainty is determined to be equal to or greater than a standard in the third determination step, as learning data for a machine learning model that extracts the detection target; A method for supporting generation of learning data, including:

18. Computer, a first extraction means for extracting a first extraction region corresponding to the detection target from a captured image including the detection target; a first determination means for determining the certainty that the first extracted region is accurate; a second determination means for determining the certainty of the first extracted region, the certainty of which is determined by the first determination means to be equal to or higher than a standard, from a different viewpoint than that of the first determination means; a second extraction means for extracting a second extraction area corresponding to the detection target by a method different from that of the first extraction means from the captured image including the first extraction area for which the second determination means has determined that the certainty is equal to or higher than a standard; a third determination means for determining the certainty that the second extracted region is accurate; a learning data generation means for determining the second extraction area, for which the third determination means has determined that the certainty is equal to or higher than a standard, as learning data for a machine learning model that extracts the detection target; A program that functions as a

Images