Processing surface determination device, processing surface determination program, processing surface determination method, processing system, inference device, and machine learning device

Abstract

The processing surface determination device (7) includes: a classification result acquisition unit (70A) that acquires a classification result for classifying the state of the processing surface (100) into one of a plurality of processing states for a plurality of small image regions (430) into which the determination image region (420) of the determination image (42) is divided; and a determination result inference unit (71) that infers a determination result for the determination image (42) by inputting the classification result for the plurality of small image regions (430) into a determination learning model (2) for the determination, wherein the determination learning model (2) is obtained by performing machine learning on the correlation between the classification result for a plurality of learning image regions (410) corresponding to the plurality of small image regions (430) and the determination result when the state of the processing surface (100) in the plurality of learning image regions (410) is determined based on the classification result.

Description

Technical Field

[0001] This invention relates to a machining surface determination device, a machining surface determination program, a machining surface determination method, a machining system, an inference device, and a machine learning device. Background Technology

[0002] In recent years, the development of devices that automatically determine product quality using various sensors, replacing manual visual judgment by operators, has been progressing in the manufacturing process of various products. For example, Patent Document 1 discloses an inspection device that inspects the shape of an impeller by binarizing an image of the impeller being inspected.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-51664 Summary of the Invention

[0006] As one of the indicators for judging product quality, for example, the state of the machined surface after various processing processes such as grinding, milling, cutting, or casting can be listed. The state of the machined surface includes various judgment items such as roughness, unevenness, ripples, warping, patterns, creases, and undulations.

[0007] However, the inspection device disclosed in Patent Document 1 is used to inspect the shape of the object being inspected, but it cannot determine the condition of the machined surfaces within the object. Furthermore, when the condition of the machined surfaces is determined by an operator, it relies on the operator's skill level and experience (including tacit knowledge). Therefore, individual differences among operators become greater, making it difficult to guarantee product quality.

[0008] In view of the above-mentioned problems, the present invention aims to provide a machining surface determination device, machining surface determination program, machining surface determination method, machining system, inference device, and machine learning device that can automatically determine the state of the machining surface of a determination object.

[0009] To achieve the above objective, one aspect of the present invention provides a machining surface determination device that determines the state of the machining surface based on a determination image obtained by photographing the machining surface of a determination object. The machining surface determination device includes:

[0010] The classification result acquisition unit acquires, on a per-small-image-region basis, a classification result for classifying the state of the processing surface into one of multiple processing states, based on multiple small-image-region segments formed by dividing the decision image region of the decision image; and

[0011] The determination result inference unit infers the determination result for the determination image by inputting the classification results for the multiple small image regions into a determination learning model, wherein the determination learning model is obtained by machine learning the correlation between the classification results for multiple learning image regions corresponding to the multiple small image regions and the determination result when the state of the processing surface in the multiple learning image regions is determined based on the classification results.

[0012] Invention Effects

[0013] According to the processing surface determination apparatus of the present invention, the determination result inference unit infers the determination result for the determination image by inputting the classification results for each small image region when the determination image region of the determination image is divided into multiple small image regions into a determination learning model. Therefore, the state of the processing surface of the object to be determined can be determined automatically.

[0014] Other issues, components, and effects not mentioned above will become clear through the specific implementation methods described below. Attached Figure Description

[0015] Figure 1 This is a schematic configuration diagram showing an example of a machining system 1 equipped with the machining surface determination device 7 of the first embodiment.

[0016] Figure 2 This is a hardware configuration diagram showing an example of a computer 200 that constitutes a machine learning device 6 and a processing surface determination device 7.

[0017] Figure 3 This is a block diagram illustrating an example of the machine learning device 6 according to the first embodiment.

[0018] Figure 4 This is a data structure diagram representing an example of data used in first-class classification learning.

[0019] Figure 5 This is a data structure diagram representing an example of data used for decision-making learning.

[0020] Figure 6 This is a schematic diagram representing an example of an inference model 20 applied to the first classification learning model 2A.

[0021] Figure 7 This is a schematic diagram representing an example of an inference model 20 applied to the decision-making learning model 2.

[0022] Figure 8 This is a block diagram showing an example of the processing surface determination device 7 of the first embodiment.

[0023] Figure 9 This is a functional illustration diagram showing an example of classification result acquisition processing performed by the classification result acquisition unit 70A.

[0024] Figure 10 This is a functional illustration diagram showing an example of the judgment result inference processing performed based on the judgment result inference unit 71.

[0025] Figure 11 This is a flowchart illustrating an example of a machining surface determination method based on the machining surface determination device 7 of the first embodiment.

[0026] Figure 12 This is a block diagram illustrating an example of the machine learning device 6 according to the second embodiment.

[0027] Figure 13 This is a data structure diagram representing an example of data used in second-class classification learning.

[0028] Figure 14 This is a schematic diagram representing an example of an inference model 20B applied to the learning model 2B for the second classification.

[0029] Figure 15 This is a block diagram illustrating an example of the processing surface determination device 7 of the second embodiment.

[0030] Figure 16 This is a functional illustration diagram showing an example of classification result acquisition processing performed by the classification result acquisition unit 70B.

[0031] Figure 17 This is a flowchart illustrating an example of a machining surface determination method based on the machining surface determination device 7 of the second embodiment. Detailed Implementation

[0032] Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. The scope of the description illustrating the purpose of the present invention is shown schematically, primarily focusing on the scope necessary for describing corresponding parts of the invention; for parts omitted from the description, they are based on prior art.

[0033] (First Embodiment)

[0034] Figure 1 This is a schematic configuration diagram showing an example of a machining system 1 equipped with the machining surface determination device 7 of the first embodiment.

[0035] The processing system 1 includes: a processing unit 3 that processes and determines an object 10; an imaging unit 4 that captures images of the processed surface 100 of the object 10; a processed surface determination device 7 that determines the state of the processed surface 100 of the object 10 using a first classification learning model 2A and a determination learning model 2; and a control device 5 that controls the processing unit 3, the imaging unit 4, and the processed surface determination device 7. Additionally, the processing system 1 includes, as an additional component, a machine learning device 6 that generates the first classification learning model 2A and the determination learning model 2.

[0036] The object to be judged 10 is, for example, any article formed of any material such as metal, resin, or ceramic, and which is the object of processing in the processing unit 3. As a specific example, the object to be judged 10 is fluid machinery or a fluid component constituting fluid machinery. Furthermore, there are no particular limitations on the three-dimensional shape, surface properties, color, size, etc. of the object to be judged 10.

[0037] The processed surface 100 is, for example, the surface of the object to be judged 10 when the object to be judged 10 has been processed by the processing unit 3. The processed surface 100 can be any surface of the object to be judged 10, either the entire surface of the object to be judged 10 or a part thereof.

[0038] The machining unit 3 consists of various robot manipulators and machine tool machining mechanisms that operate using electricity or fluid pressure as a drive source. Based on control commands from the control device 5, the machining unit 3 performs machining operations such as grinding, milling, cutting, or casting. Furthermore, the machining unit 3 can perform any machining operation, or combine multiple machining operations, as long as it is used to process or form the surface of the target object 10.

[0039] exist Figure 1 In the machining system 1 shown, the machining unit 3 consists of a robot manipulator with a replaceable grinding wheel mounted on its top, used to perform the grinding process. Furthermore, the object to be determined, 10, is a fluid component constituting the pump, which is an impeller with multiple blades, and the machining surface 100 is the surface of each blade that has been machined by the grinding process based on the machining unit 3.

[0040] The imaging unit 4 is a camera that captures images of the processed surface 100, and is composed of an image sensor such as a CMOS sensor or a CCD sensor. The imaging unit 4 is installed at a predetermined position capable of capturing images of the processed surface 100. If the processing unit 3 is, for example, a robot manipulator, the imaging unit 4 can be installed on the top of the robot manipulator or fixed above the platform (including movable ones) on which the object to be judged 10 is placed. Furthermore, if the processing unit 3 is, for example, a machining mechanism of a machine tool, the imaging unit 4 can be installed inside the protective cover of the machine tool or fixed above a worktable separate from the machine tool.

[0041] The imaging unit 4 is installed in the aforementioned designated position and its position and orientation are adjusted so that the processed surface 100 is within the viewing angle of the imaging unit 4. Furthermore, as... Figure 1 As shown, the imaging unit 4 can have separate imaging units connected to the machine learning device 6 and the processing surface determination device 7, or it can be a single imaging unit connected to both the machine learning device 6 and the processing surface determination device 7 and shared by both. Furthermore, the imaging unit 4 can also have a pan-tilt-zoom function. Moreover, the imaging unit 4 is not limited to using a single camera to photograph the processing surface 100; multiple cameras can also be used for imaging.

[0042] Control device 5 includes, for example, a control board 50, which is powered by a general-purpose or special-purpose computer (see below). Figure 2 It consists of a microcontroller or similar component; and an operation display panel 51, which consists of a touch panel display, switches, buttons, etc.

[0043] The control board 50 is connected to the actuator and sensors (not shown) of the machining unit 3. Based on the machining action parameters used to perform the machining process and the detection signals from the sensors, it sends control commands to the actuator, thereby controlling the machining process performed by the machining unit 3. The control board 50 sends a shooting command to the imaging unit 4, and as a result, receives the image captured by the imaging unit 4. The control board 50 sends the captured image as a determination image to the machining surface determination device 7, and as a result, receives the state of the machining surface 100 determined by the machining surface determination device 7. Furthermore, the control board 50 can also send the captured image to the machine learning device 6.

[0044] The operation display panel 51 accepts the operations of the workers and outputs various information through display and sound.

[0045] The machine learning device 6 operates as the main body of the learning phase in machine learning. Based on images captured by the imaging unit 4, the machine learning device 6 acquires learning data and generates a first classification learning model 2A and a decision learning model 2 based on this learning data. The machine learning device 6 provides the learned first classification learning model 2A and decision learning model 2 to the processing surface determination device 7 via any communication network, recording medium, etc. Details of the machine learning device 6 will be described later.

[0046] The machining surface determination device 7 operates as the main body of the inference stage in machine learning. Using the first classification learning model 2A and the determination learning model 2 generated by the machine learning device 6, the machining surface determination device 7 uses the image of the machining surface 100 captured by the imaging unit 4 as the determination image to determine the state of the machining surface 100 of the object 10. Details of the machining surface determination device 7 will be described later.

[0047] Furthermore, the components of the machining system 1 can be housed in a single casing to form, for example, a machine tool. In this case, at least one of the machine learning device 6 and the machining surface determination device 7 can be integrated into the control device 5. Alternatively, the components of the machining system 1 can consist of a machining device with a machining section 3 and an inspection device with an imaging section 4 and the machining surface determination device 7. In this case, the functions of the control device 5 can be distributed between the machining device and the inspection device. Moreover, the components of the machining system 1 can be connected via a wireless or wired network, allowing at least one of the machine learning device 6 and the machining surface determination device 7 to be located in a location separate from the machining site where the machining section 3 and the imaging section 4 are located. In this case, the control device 5 can be located either at the machining site or elsewhere.

[0048] Figure 2 This is a hardware configuration diagram showing an example of a computer 200 that constitutes a machine learning device 6 and a processing surface determination device 7.

[0049] The machine learning device 6 and the processing surface determination device 7 are each composed of a general-purpose or special-purpose computer 200. For example... Figure 2 As shown, the computer 200 includes a bus 210, a processor 212, a memory 214, an input device 216, a display device 218, a storage device 220, a communication I / F (interface) unit 222, an external device I / F unit 224, an I / O (input / output) device I / F unit 226, and a media input / output unit 228 as its main components. Furthermore, the aforementioned components may be appropriately omitted depending on the intended use of the computer 200.

[0050] The processor 212 consists of one or more arithmetic processing devices (CPU, MPU, GPU, DSP, etc.) and operates as the control unit for the overall computer 200. The memory 214 stores various data and programs 230, and is composed of volatile memory (DRAM, SRAM, etc.) that functions as main memory and non-volatile memory (ROM, flash memory, etc.).

[0051] Input device 216 may be composed of, for example, a keyboard, mouse, numeric keypad, electronic pen, etc. Display device 218 may be composed of, for example, a liquid crystal display, organic EL display, electronic paper, projector, etc. Input device 216 and display device 218 may also be integrated as a touch panel display. Storage device 220 may be composed of, for example, an HDD, SSD, etc., storing various data required for the execution of operating system and program 230.

[0052] The communication I / F unit 222 connects to a network 240, such as the Internet or an intranet, via wired or wireless means, and transmits and receives data with other computers according to a specified communication standard. The external device I / F unit 224 connects to external devices 250, such as printers or scanners, via wired or wireless means, and transmits and receives data with these external devices 250 according to a specified communication standard. The I / O device I / F unit 226 connects to various I / O devices 260, such as sensors and actuators, and transmits and receives various signals and data, such as sensor detection signals and control signals to actuators, with these I / O devices. The media input / output unit 228 is composed of a drive device such as a DVD drive or CD drive, and reads and writes data to media 270 such as DVDs and CDs.

[0053] In the computer 200 with the above configuration, the processor 212 loads the program 230 into the workpiece storage area of ​​the memory 214 and executes it, controlling various parts of the computer 200 via the bus 210. Alternatively, the program 230 can be stored in the storage device 220 instead of the memory 214. The program 230 can also be recorded in the form of an installable file or an executable file on a non-transitory recording medium such as a CD or DVD, and provided to the computer 200 via the media input / output unit 228. The program 230 can also be provided to the computer 200 by downloading it via the network 240 through the communication I / F unit 222. Furthermore, the computer 200 can also utilize hardware such as an FPGA or ASIC to implement various functions achieved by the processor 212 executing the program 230.

[0054] Computer 200 may be a fixed-type computer or a portable computer, and is an electronic device of any form. Computer 200 may be a client computer, a server computer, or a cloud computer. Computer 200 may also be used in devices other than machine learning device 6 and processing surface determination device 7.

[0055] (Machine Learning Device 6)

[0056] Figure 3 This is a block diagram illustrating an example of the machine learning device 6 according to the first embodiment.

[0057] The machine learning device 6 includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a learned model storage unit 63. The machine learning device 6, for example, comprises... Figure 2 The computer 200 shown is configured as follows. In this case, the learning data acquisition unit 60 is configured with a communication I / F unit 222 or an I / O device I / F unit 226 and a processor 212, the machine learning unit 62 is configured with a processor 212, and the learning data storage unit 61 and the learning completed model storage unit 63 are configured with a storage device 220.

[0058] The learning data acquisition unit 60 is an interface unit that connects to various external devices via a communication network and establishes corresponding learning data by acquiring input and output data. External devices include, for example, the imaging unit 4, the processing surface determination device 7, and the operator's terminal 8.

[0059] The learning data storage unit 61 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 60. The learning data includes first-class learning data for generating the first-class learning model 2A and decision-making learning data for generating the decision-making learning model 2. Furthermore, the specific configuration of the database constituting the learning data storage unit 61 can be appropriately designed.

[0060] The machine learning unit 62 performs machine learning using the learning data stored in the learning data storage unit 61. Specifically, the machine learning unit 62 inputs multiple sets of first-class learning data into the first-class learning model 2A, causing the first-class learning model 2A to learn the correlation between the input and output data contained in the first-class learning data, thereby generating the first-class learning model 2A. Similarly, the machine learning unit 62 inputs multiple sets of decision learning data into the decision learning model 2, causing the decision learning model 2 to learn the correlation between the input and output data contained in the decision learning data, thereby generating the decision learning model 2.

[0061] The learned model storage unit 63 is a database storing the first classification learning model 2A and the decision learning model 2 generated by the machine learning unit 62. The first classification learning model 2A and the decision learning model 2 stored in the learned model storage unit 63 are provided to a physical system (e.g., a processing surface determination device 7) via any communication network, recording medium, etc. Furthermore, the first classification learning model 2A and the decision learning model 2 can also be provided to an external computer (e.g., a server computer, a cloud computer) and stored in the storage unit of an external computer. Additionally, in Figure 3 In this context, the learning data storage unit 61 and the learning completed model storage unit 63 are represented as independent storage units, but they can also be composed of a single storage unit.

[0062] Figure 4 This is a data structure diagram representing an example of data used in first-class classification learning.

[0063] The first classification learning data includes the learning image 41 as input data and the classification result of classifying the state of the processing surface 100 contained in the learning image 41 into one of the multiple processing states as output data. These input data and output data are established in correspondence.

[0064] The learning image 41, which serves as input data, is each of a plurality of images generated by dividing a captured image 40, which has a predetermined captured image region 400, obtained by the capturing unit 4 capturing the processing surface 100 of the judgment object 10, into a learning image region 410.

[0065] The image area 400 of the captured image 40 is the area captured by the imaging unit 4 and is determined according to the viewing angle of the imaging unit 4. Figure 4 The image capture area 400 shown is set in such a way that it includes a part of a blade of the impeller, which is the object to be determined 10. Furthermore, in Figure 4 In the captured image 40 shown, not only the processing surface 100 is captured, but the background 110 is also captured. However, the captured image area 400 can also be set without capturing the background 110.

[0066] like Figure 4 As shown, the learning image region 410 of the learning image 41 is formed by dividing the capturing image region 400 of the capturing image 40 into a grid pattern, with each learning image region 410 being a square. Furthermore, the number of images, shape, size, and aspect ratio of the learning image region 410 can be appropriately changed; for example, it can be rectangular or other shapes. Additionally, the segmentation method for dividing the capturing image region 400 into the learning image region 410 can be appropriately changed; for example, it can be segmented into a jagged pattern or segmented according to other criteria.

[0067] In supervised learning, the classification result as output data is referred to as training data or correct label, for example. When multiple processing states are classified using two classes, such as "good" and "poor," the classification result is represented by either "good" or "poor." When multiple processing states are classified using three classes, such as "good," "acceptable," and "poor," the classification result is represented by either "good," "acceptable," or "poor." Furthermore, the multiple processing states when classifying the states of processing surface 100 are not limited to the above categories; for example, they can be classified into four or more categories, or classified from other perspectives.

[0068] Furthermore, when the edge of the processing surface 100 or the background 110 other than the processing surface 100 is present in the learning image region 410, a classification of "excluding the judgment object" can be added. For classification results where the edge of the processing surface 100 or the background 110 other than the processing surface 100 is present in the learning image region 410, and the result is classified as "excluding the judgment object," in the examples of the two categories mentioned above, represented by one of "good," "bad," and "excluding the judgment object." In the examples of the three categories mentioned above, such as... Figure 4 As shown, it is represented by one of "Good", "Acceptable", "Poor" and "Not a Target". In addition, when both the processing surface 100 and the background 110 are captured in the learning image 41, for example, it can be classified as "Not a Target" if the ratio of the background 110 is higher than the specified ratio, or it can never be classified as "Not a Target".

[0069] Figure 5 This is a data structure diagram representing an example of data used for decision-making learning.

[0070] The learning data includes the classification result of classifying the state of the processing surface 100 into one of the multiple processing states for each of the multiple learning image regions 410 as input data, and the determination result of determining the state of the processing surface 100 in the multiple learning image regions 410 based on the classification result as output data. These input data and output data are established in correspondence.

[0071] When the state of the processing surface 100 is classified as, for example, "good", "acceptable", "bad" and "not a judgment object", the classification results of the multiple learning image regions 410 used as input data are represented by integer values ​​such as "0", "1", "2" and "3".

[0072] The judgment result as output data is referred to as training data or correct label in supervised learning. The judgment result is obtained by judging the overall state of the processing surface 100 with regard to multiple learning image regions 410, that is, the image region 400 before it was segmented into multiple learning image regions 410.

[0073] The determination result is obtained by judging at least one of the following as the state of the machined surface 100: whether reprocessing is required to perform the same processing steps as when the machined surface 100 was processed; whether other processing is required to perform processing steps different from when the machined surface 100 was processed; whether finishing processing of the machined surface 100 by the operator is required; and the processing range of the machined surface 100 that is the object of reprocessing, other processing, or finishing. Furthermore, the determination result may also be, as an alternative to or addition to the above, a result obtained by judging whether the machined surface 100 as a whole includes at least one of multiple processing states of "good" and "bad".

[0074] The learning data acquisition unit 60 can employ various methods to acquire and determine the first category of learning data. For example, the learning data acquisition unit 60 acquires a captured image 40 obtained by the imaging unit 4 of the object to be determined after processing by the processing unit 3, and generates multiple learning images 41 by segmenting the captured image 40. Then, the learning data acquisition unit 60, for example, overlays the frame lines constituting each learning image region 410 with the captured image 40, thereby displaying the multiple learning images 41 on the display screen of the operator's terminal 8 in a state that allows for differentiation.

[0075] The operator visually identifies each of the learning images 41 on the display screen, inputs the results (classification results) obtained by classifying the states of the processing surfaces 100 contained in each of the multiple learning images 41 into multiple processing states (categories), and inputs the results (judgment results) obtained by judging the states of the processing surfaces 100 contained in the captured image 40 using the operator's terminal 8. The learning data acquisition unit 60 accepts the operator's input and establishes a correspondence between the learning images 41 (input data) and the classification results (output data) of the input operation on the learning images 41, thereby acquiring multiple first-category learning data. Furthermore, the learning data acquisition unit 60 acquires judgment learning data by establishing a correspondence between the classification results (input data) of the multiple learning image regions 410 of each learning image 41 and the judgment results (output data) of the input operation on the captured image 40.

[0076] Therefore, the learning data acquisition unit 60 can acquire a number of first-class learning data from a single captured image 40 that is equivalent to the number of segments when segmenting into multiple learning images 41, and by repeatedly performing the above operation, it can acquire the desired number of first-class learning data. Furthermore, the learning data acquisition unit 60 can acquire judgment learning data in conjunction with the acquisition of the first-class learning data. Therefore, it is easy to collect both the first-class learning data and the judgment learning data.

[0077] Figure 6 This is a schematic diagram representing an example of an inference model 20A applied to the learning model 2A for the first classification.

[0078] The inference model 20A employs a convolutional neural network (CNN) as the specific machine learning method. The inference model 20A has an input layer 21, an intermediate layer 22, and an output layer 23.

[0079] The input layer 21 has a number of neurons corresponding to the number of pixels in the learning image 41, which is used as input data, and the pixel value of each pixel is input to each neuron.

[0080] The intermediate layer 22 consists of a convolutional layer 22a, a pooling layer 22b, and a fully connected layer 22c. For example, multiple convolutional layers 22a and pooling layers 22b are alternately provided. The convolutional layers 22a and pooling layers 22b extract feature values ​​from the image input via the input layer 21. The fully connected layer 22c, for example, uses an activation function to transform the feature values ​​extracted from the image by the convolutional layers 22a and pooling layers 22b, and outputs them as a feature vector. Furthermore, the fully connected layer 22c may also have multiple layers.

[0081] Output layer 23 outputs data containing the classification results based on the feature vector output from fully connected layer 22c. Alternatively, the output data may include, for example, a rating indicating the confidence level of the classification results, in addition to the classification results.

[0082] Synapses are arranged between the layers of the inference model 20A to connect the neurons between the layers respectively, and the weights correspond to the synapses of the convolutional layer 22a and the fully connected layer 22c of the intermediate layer 22.

[0083] The machine learning unit 62 inputs the first classification learning data into the inference model 20A, enabling the inference model 20A to perform machine learning on the correlation between the learning image 41 and the classification result. Specifically, the machine learning unit 62 inputs the learning image 41, which constitutes the first classification learning data, as input data into the input layer 21 of the inference model 20A. Furthermore, the machine learning unit 62 can also perform pre-processing on the learning image 41 by performing prescribed image adjustments (e.g., image format, image size, image filter, image mask, etc.) as pre-processing before inputting the learning image 41 into the input layer 21.

[0084] The machine learning unit 62 uses an error function that compares the classification result (inference result) shown by the output data from the output layer 23 with the classification result (training data) constituting the first classification learning data. It repeatedly adjusts the weights corresponding to each synapse (backpropagation) to reduce the evaluation value of the error function. Furthermore, when the machine learning unit 62 determines that the learning termination conditions have been met, such as repeating the above series of processes a predetermined number of times and the evaluation value of the error function being smaller than the allowable value, it terminates the machine learning process and saves the inference model 20A (with all weights corresponding to each synapse) as the first classification learning model 2A in the learned model storage unit 63.

[0085] Figure 7 This is a schematic diagram representing an example of an inference model 20 applied to the decision-making learning model 2.

[0086] Inference Model 20 and Figure 6 The inference model 20A shown also employs a convolutional neural network as the specific method for machine learning. The following will compare it with... Figure 6 The differences of the inference model 20A shown are the focus of the explanation of inference model 20.

[0087] The input layer 21 has a number of neurons corresponding to the number of segments when the captured image region 400 is divided into multiple learning image regions 410, and the classification results (e.g., integer values ​​of 0, 1, 2, 3) for each learning image region 410 are input to each neuron respectively.

[0088] Output layer 23 outputs output data containing the decision result based on the feature vector output from fully connected layer 22c. In addition to the decision result, the output data may also include, for example, a score indicating the confidence level of the decision result.

[0089] The machine learning unit 62 inputs the decision learning data into the inference model 20, enabling the inference model 20 to perform machine learning on the correlation between the classification results and the decision results for multiple learning image regions 410. Specifically, the machine learning unit 62 inputs the classification results for multiple learning image regions 410, which constitute the decision learning data, as input data into the input layer 21 of the inference model 20.

[0090] The machine learning unit 62 uses an error function that compares the judgment result (inference result) shown in the output data from the output layer 23 with the judgment result (training data) constituting the judgment learning data, and repeatedly adjusts the weights corresponding to each synapse (backpropagation) to reduce the evaluation value of the error function. Furthermore, when the machine learning unit 62 determines that the learning termination conditions, such as having repeatedly performed the above series of processes a predetermined number of times and the evaluation value of the error function being smaller than the allowable value, have been met, the machine learning unit 62 terminates the machine learning and stores the inference model 20 at this time (with all weights corresponding to each synapse) as the judgment learning model 2 in the learned model storage unit 63.

[0091] (Machined surface determination device 7)

[0092] Figure 8 This is a block diagram showing an example of the processing surface determination device 7 of the first embodiment.

[0093] The machining surface determination device 7 includes a classification result acquisition unit 70A, a determination result inference unit 71, a learned model storage unit 72, and an output processing unit 73. The machining surface determination device 7, for example, comprises... Figure 2 The computer 200 shown is configured in this case. The classification result acquisition unit 70A is configured by a communication I / F unit 222 or an I / O device I / F unit 226 and a processor 212, the judgment result inference unit 71 and the output processing unit 73 are configured by the processor 212, and the learned model storage unit 72 is configured by a storage device 220.

[0094] The classification result acquisition unit 70A performs the following classification result acquisition processing (see below). Figure 9 ): For multiple small image regions 430 obtained by dividing the decision image region 420 of the decision image 42, the classification result of classifying the state of the processing surface 100 into one of the multiple processing states is obtained in units of small image regions 430.

[0095] The classification result acquisition unit 70A includes, as a specific configuration, an image acquisition unit 700 connected to the imaging unit 4, which acquires an image obtained by the imaging unit 4 of the processing surface 100 of the object to be determined 10 as a determination image 42 having a determination image region 420; a small image generation unit 701, which generates multiple small images 43 from the determination image 42 by dividing the determination image region 420 into multiple small image regions 430; and a first classification result inference unit 702A, which infers the classification result for the multiple small image regions 430 by inputting the multiple small images 43 into a first classification learning model 2A in units of small image regions 430.

[0096] The first classification result inference unit 702A records, for example, the positional relationship of each small image region 430 relative to the decision image region 420 as additional information of the small image 43, so that the decision image 42 before segmentation can be reconstructed from multiple small images 43.

[0097] The determination result inference section 71 performs the following determination result inference processing (see below). Figure 10 The classification results for multiple small image regions 430 obtained by the classification result acquisition unit 70A are input into the decision learning model 2, thereby inferring the decision result for the decision image region 420.

[0098] The judgment result inferred from the judgment result inference unit 71 is obtained by judging whether reprocessing is needed, whether other processing is needed, whether finishing is needed, and at least one of the processing ranges of the processing surface 100 that are objects to be reprocessed, processed, or finished. The judgment result may also be an alternative to or an addition to the above, obtained by judging whether the processing surface 100 as a whole includes at least one of the multiple processing states of "good" and "bad".

[0099] Furthermore, part or all of the classification result acquisition unit 70A and the determination result inference unit 71 may be replaced by the processor of an external computer (e.g., a server computer or a cloud computer), and part or all of the classification result acquisition processing performed by the classification result acquisition unit 70A and the determination result inference processing performed by the determination result inference unit 71 may also be executed by an external computer.

[0100] The learned model storage unit 72 is a database storing the learned first classification learning model 2A used in the inference processing of the classification result acquisition unit 70A and the learned judgment learning model 2 used in the inference processing of the judgment result inference unit 71. Furthermore, the number of the first classification learning model 2A and the judgment learning model 2 stored in the learned model storage unit 72 is not limited to one. For example, it can store multiple learned models with different conditions, such as machine learning methods, processing steps performed by the processing unit 3, and the judgment object 10, and can be selectively used. Alternatively, the learned model storage unit 72 can be replaced by the storage unit of an external computer (e.g., a server computer or a cloud computer). In this case, the classification result acquisition unit 70A and the judgment result inference unit 71 can also perform the aforementioned classification result acquisition processing and judgment result inference processing by accessing the external computer.

[0101] The output processing unit 73 performs output processing for outputting the determination result inferred by the determination result inference unit 71. Various methods can be employed for outputting the determination result. For example, based on the determination result, the output processing unit 73 may send reprocessing or other processing operation instructions to the processing unit 3 via the control board 50, or notify the operator of the fine processing implementation via the operation display panel 51 or the operator terminal 8 using display and sound, or store the processing unit 3's operation history in the storage unit of the control board 50. Furthermore, the output processing unit 73 may output (send, notify, store) only the determination result based on the determination result inference unit 71, or it may further output (send, notify, store) the classification results for multiple small image regions 430 based on the classification result acquisition unit 70A, in addition to the determination result based on the determination result inference unit 71.

[0102] Figure 9 This is a functional illustration diagram showing an example of the classification result acquisition process performed by the classification result acquisition unit 70A.

[0103] The determination image area 420 of the determination image 42 is the area captured by the imaging unit 4 and is determined according to the viewing angle of the imaging unit 4. Figure 9 The determination is made using image region 420 and... Figure 4 Similarly, the image capture area 400 shown is set in such a way that it includes a blade of the impeller that is the object to be determined 10. In addition, the image area 420 for determination can be set at a different position than the image capture area 400, and the number of images, shape, size and aspect ratio of the two can also be different.

[0104] like Figure 9As shown, the small image region 430 of the small image 43 is formed by dividing the decision image region 420 of the decision image 42 into a grid pattern in such a way that each small image region 430 is a square. The small image region 430 of the small image 43 is equivalent to the learning image region 410 of the learning image 41 when the first classification learning model 2A is generated in the machine learning device 6, and the number of images, shape, size and aspect ratio of the two are preferably the same or to the same extent.

[0105] Therefore, as long as the number of images, shape, size, and aspect ratio of the small image region 430 are comparable to those of the learning image region 410, the segmentation method for dividing the decision image region 420 into the small image region 430 can be appropriately modified. For example, it can be segmented into a jagged shape, or it can be segmented according to other criteria. In this case, the segmentation method for dividing the decision image region 420 into the small image region 430 can be the same as or different from the segmentation method for dividing the captured image region 400 into the learning image region 410.

[0106] Here, the first classification learning model 2A is obtained by the machine learning device 6 through machine learning of the correlation between a learning image 41 having a learning image region 410 corresponding to a small image region 430, and the classification result of classifying the state of the processing surface 100 contained in the learning image 41 into one of multiple processing states. Therefore, the first classification result inference unit 702A functions as a classifier, which classifies the state of the processing surface 100 within the small image region 430 into one of multiple processing states by inputting multiple small images 43 into the first classification learning model 2A in units of small image regions 430. When two categories (good, bad) are used as multiple processing states, the classification result is represented by two categories (good, bad); when three categories (good, acceptable, bad) are used as multiple processing states, the classification result is represented by three categories (good, acceptable, bad).

[0107] Alternatively, the first classification learning model 2A can be obtained by having the machine learning device 6 perform machine learning on the correlation between the learning image 41 and the classification result. The learning image 41 captures at least one of the processing surface 100 and the background 110 outside the processing surface 100. The classification result is a classification result that classifies the state of the processing surface 100 captured in the learning image 41 into one of a plurality of processing states, or classifies it as an object of judgment on the grounds that the edge of the processing surface 100 or the background 110 outside the processing surface 100 exists in the small image region 430 of the learning image 41. In this case, the first classification result inference unit 702A of the classification result acquisition unit 70A functions as a classifier. This classifier inputs multiple small images 43 into the first classification learning model 2A in units of small image regions 430, thereby classifying the state of the processing surface 100 within the small image region 430 into one of multiple processing states, or classifying it as an object of judgment based on the presence of an edge of the processing surface 100 or a background 110 outside the processing surface 100 within the small image region 430. Since the object of judgment is further added to the multiple processing states in the classification result, the classification result in the above example is represented by three categories (good, bad, object of judgment) or four categories (good, acceptable, bad, object of judgment).

[0108] Furthermore, the classification results for the small image region 430 can also include scores (confidence levels) for each category. In this case, if the classification results are represented by four categories (Good, Acceptable, Poor, and Not Subject to Judgment), then the scores for each category for a specific small image region 430 would be output as, for example, "0.02", "0.10", "0.95", and "0.31". The scoring can be used in any way; for example, the category with the highest score (in the above example, "Poor" with a score of "0.95") can be used as the classification result, or the category can be used as the classification result if the score of a specified category exceeds a specified benchmark value (in the above example, the score of "Poor" category "0.95" exceeds the benchmark value "0.80").

[0109] Furthermore, the classification results for the small image region 430 are preferably stored in the learned model storage unit 72 or other storage device (not shown). Past classification results can be used as first-class learning data for online learning and relearning, for example, to further improve the inference accuracy of the learned first-class learning model 2A.

[0110] Figure 10This is a functional illustration diagram showing an example of the judgment result inference processing performed by the judgment result inference unit 71. Hereinafter, the following scenario will be described: by dividing the judgment image region 420 into 60 small image regions 430, such as... Figure 10 As shown, a decision image 42 is segmented into 60 smaller images 43.

[0111] The decision learning model 2 is obtained by performing machine learning on the correlation between the classification results for multiple learning image regions 410 corresponding to multiple small image regions 430 and the decision results when the state of the processing surface 100 within the multiple learning image regions 410 is determined based on the classification results. Therefore, the decision result inference unit 71 infers the state of the processing surface 100 within the multiple small image regions 430, i.e., the decision result for the processing surface 100 within the decision image region 420, by inputting the classification results for the multiple small image regions 430 obtained by the classification result acquisition unit 70A into the decision learning model 2.

[0112] When the processing surface 100 is defined as whether reprocessing, other processing, or finishing is required, the determination result is, for example, a real value with a range of 0 to 1, where the closer to "0" is to "no" (not required) and the closer to "1" is to "required" (required). Furthermore, when the processing range is defined as the processing surface 100, the processing range is determined as a classification result for multiple small image regions 430, for example, the range containing at least the small image regions 430 classified as "poor".

[0113] Furthermore, the determination result inference unit 71 can also perform prescribed post-processing on the determination result inferred from the determination learning model 2 as described above. For example, the determination result inference unit 71 can also perform the following post-processing: compare the values ​​of the determination results for whether reprocessing is needed, the values ​​of the determination results for whether other processing is needed, and the values ​​of the determination results for whether finishing is needed, and select the processing with the largest determination result value as the final determination result.

[0114] (Method for determining machined surfaces)

[0115] Figure 11 This is a flowchart illustrating an example of a machining surface determination method based on the machining surface determination apparatus 7 of the first embodiment. Furthermore, Figure 11The series of machining surface determination methods shown are repeatedly executed by the machining surface determination device 7 at predetermined time intervals. The predetermined time interval can be arbitrary; for example, it can be after the completion of a machining operation performed by the machining unit 3, during the machining operation, or when a predetermined event occurs (such as during operator operation or instruction from the production management system). The following describes the case where the machining surface determination method is executed on the object 10 that has been processed by the machining unit 3 after the completion of the machining operation performed by the machining unit 3.

[0116] First, in step S100, if the processing step performed by the processing unit 3 is completed, the imaging unit 4 takes a picture of the processed surface 100 of the judgment object 10 that has been processed by the processing step, and sends the captured image to the processing surface judgment device 7 via the control device 5. Thereby, the image acquisition unit 700 of the classification result acquisition unit 70A acquires the captured image as the judgment image 42.

[0117] Next, in step S110, as a preprocessing step for the determination image 42, the small image generation unit 701 generates a plurality of small images 43 from the determination image 42 by dividing the determination image region 420 of the determination image 42 into a plurality of small image regions 430.

[0118] Next, in steps S120 to S128, the first classification result inference unit 702A, with the number of segments of the multiple small images 43 set to K and the consecutive numbers (1≤n≤K) assigned to the multiple small images 43 respectively, increases the variable i from "1" to "K", thereby performing loop processing.

[0119] Specifically, in step S120, the first classification result inference unit 702A initializes variable i with "1". Next, in step S122, the first classification result inference unit 702A selects the i-th small image 43 and inputs it to the input layer 21 of the first classification learning model 2A, thereby inferring the classification result output from the output layer 23 of the first classification learning model 2A.

[0120] Next, in step S126, variable i is incremented, and in step S128, it is determined whether variable i exceeds the segmentation number K. Furthermore, the first classification result inference unit 702A repeatedly performs the above steps S122 and S126 until variable i exceeds the segmentation number K, thereby obtaining the classification result for multiple small image regions 430.

[0121] Next, in step S130, the determination result inference unit 71 inputs the classification results for multiple small image regions 430 to the input layer 21 of the determination learning model 2, thereby inferring the determination result output from the output layer 23 of the determination learning model 2 (e.g., whether reprocessing is needed, whether other processing is needed, whether fine processing is needed, processing range, etc.).

[0122] Next, in step S140, the output processing unit 73 outputs information corresponding to the determination result deduced by the determination result inference unit 71 to the output unit (e.g., control device 5, operator terminal 8, etc.). Then, the process ends. Figure 11 The following is a series of processing surface determination methods. In the processing surface determination method, step S100 is equivalent to the image acquisition process, steps S100 to S128 are equivalent to the classification result acquisition process, step S130 is equivalent to the determination result inference process, and step S140 is equivalent to the output processing process.

[0123] As described above, according to the processing surface determination apparatus 7 and processing surface determination method of this embodiment, the classification result acquisition unit 70A inputs each of the plurality of small images 43 generated from the determination image 42 by dividing the determination image region 420 into small image regions 430 to the first classification learning model 2A, thereby inferring the classification result for the plurality of small image regions 430. Furthermore, the determination result inference unit 71 inputs the classification result for the plurality of small image regions 430 to the determination learning model 2, thereby inferring the state of the processing surface 100 as the determination result.

[0124] Therefore, by inputting each of the multiple small images 43 obtained from the segmentation decision image 42, and inferring the classification result based on the first classification learning model 2A on a unit basis of small image regions 430, the collection of learning data required for machine learning is easier compared to inputting a single decision image 42 into the first classification learning model 2A, and the accuracy of the first classification learning model 2A can be improved. Furthermore, by inputting the classification results for the multiple small image regions 430 obtained based on the first classification learning model 2A into the decision learning model 2, the state of the processing surface 100 contained in the decision image 42 is determined. Therefore, the state of the processing surface 100 possessed by the decision object 10 can be determined automatically.

[0125] (Second Implementation)

[0126] In the processing system 1 of the first embodiment, the case where the first classification learning model 2A and the decision learning model 2 are used in the learning and inference phases of machine learning is described. In contrast, in the processing system 1 of the second embodiment, the case where the second classification learning model 2B and the decision learning model 2 are used is described. Furthermore, the basic structure and operation of the processing system 1 of the second embodiment are the same as those of the first embodiment. Therefore, the following description focuses on the part related to the second classification learning model 2B, which is a difference from the first embodiment.

[0127] (Machine Learning Device 6)

[0128] Figure 12 This is a block diagram illustrating an example of the machine learning device 6 according to the second embodiment.

[0129] The machine learning device 6, like the first embodiment, includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a learning completed model storage unit 63.

[0130] The learning data acquisition unit 60 is an interface unit that connects to various external devices via a communication network and acquires learning data. The learning data storage unit 61 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 60. The learning data includes second-classification learning data for generating the second-classification learning model 2B and determination learning data similar to that in the first embodiment.

[0131] The machine learning unit 62 inputs multiple sets of second-class classification learning data into the second-class classification learning model 2B, thereby enabling the second-class classification learning model 2B to perform machine learning on the correlation between the input data and the output data contained in the second-class classification learning data, thus generating the second-class classification learning model 2B. Furthermore, similarly to the first embodiment, the machine learning unit 62 uses the decision learning data to generate a decision learning model 2.

[0132] The learning-completed model storage unit 63 is a database that stores the second classification learning model 2B and the decision learning model 2 generated by the machine learning unit 62.

[0133] Figure 13 This is a data structure diagram representing an example of data used in second-class classification learning.

[0134] The second classification learning data includes pixel classification results for multiple learning pixel regions 411 obtained from the learning image 41 as input data, and classification results obtained by classifying the state of the processing surface 100 contained in the learning image 41 into one of multiple processing states as output data. These input data and output data are established in correspondence.

[0135] Regarding the pixel classification results for multiple learning pixel regions 411 as input data, for the multiple learning pixel regions 411 constituting the learning image 41, a pixel classification result representing the classification result for the learning pixel regions 411 is obtained on a per-learning pixel region 411 basis, based on the pixel values ​​within the learning pixel regions 411.

[0136] The learning pixel region 411 is a region equivalent to 1 pixel. The pixel values ​​within the learning pixel region 411 are represented, for example, by RGB values, grayscale values, brightness values, etc. When using four categories, such as "good", "acceptable", "poor" and "not a target", as multiple processing states, for the pixel classification results, for example, the pixel values ​​within the learning pixel region 411 are compared with three specified thresholds (the third threshold < the second threshold < the first threshold). If the pixel value is above the first threshold, a classification result of "good" (0) is assigned; if the pixel value is below the first threshold but above the second threshold, a classification result of "acceptable" (1) is assigned; if the pixel value is below the second threshold but above the third threshold, a classification result of "poor" (2) is assigned; and if the pixel value is below the third threshold, a classification result of "not a target" (3) is assigned.

[0137] As in the first embodiment, the classification result of the output data is also the classification result for the processing surface 100 within the learning image region 410, for example, Figure 13 The text is represented by one of the following: "Good", "Acceptable", "Poor", and "Not applicable".

[0138] The learning data acquisition unit 60 can employ various methods to acquire and determine the second category of learning data. For example, similar to the first embodiment, the learning data acquisition unit 60 acquires a captured image 40 obtained by the imaging unit 4 of the determination object 10 after the processing unit 3 has performed a processing step, generates a plurality of learning images 41 by segmenting the captured image 40, and displays the plurality of learning images 41 on the display screen of the operator's terminal 8.

[0139] The operator visually identifies each of the learning images 41 on the display screen, inputs the result (classification result) obtained by classifying the state of the processing surface 100 contained in each of the multiple learning images 41 into multiple processing states (categories), and inputs the result (determination result) obtained by judging the state of the processing surface 100 contained in the captured image 40 using the operator's terminal 8. Furthermore, the learning data acquisition unit 60 accepts the operator's input operation and establishes a correspondence between the pixel classification result (input data) of the multiple learning pixel regions 411 acquired from the learning images 41 and the classification result (output data) of the input operation performed on the learning images 41, thereby acquiring multiple second-class learning data. Additionally, the learning data acquisition unit 60 acquires judgment learning data by establishing a correspondence between the classification result (input data) of the multiple learning image regions 410 possessed by each of the learning images 41 and the judgment result (output data) of the input operation performed on the captured image 40.

[0140] Therefore, the learning data acquisition unit 60 can acquire a number of second-class learning data equivalent to the number of segments when a single captured image 40 is segmented into multiple learning images 41, and can acquire the desired number of second-class learning data by repeatedly performing the above-described operation. Furthermore, the learning data acquisition unit 60 can acquire judgment learning data in conjunction with acquiring the second-class learning data. Therefore, both second-class learning data and judgment learning data can be easily collected.

[0141] Figure 14 This is a schematic diagram representing an example of an inference model 20B applied to the learning model 2B for the second classification.

[0142] Inference Model 20B and Figure 6 The inference model 20A shown also employs a convolutional neural network as the specific method for machine learning. The following will compare it with... Figure 6 The differences of the inference model 20A shown are the focus of the explanation of inference model 20B.

[0143] The input layer 21 has a number of neurons corresponding to the number of pixels in the learning image 41, which is used as input data, and inputs the pixel classification results for multiple learning pixel regions 411 to each neuron respectively.

[0144] Output layer 23 outputs output data containing the classification results based on the feature vector output from fully connected layer 22c. Alternatively, the output data may include, in addition to the classification results, a rating indicating the confidence level of the classification results.

[0145] The machine learning unit 62 inputs the second classification learning data into the inference model 20B, enabling the inference model 20B to perform machine learning on the correlation between the pixel classification results and the classification results for multiple learning pixel regions 411. Specifically, the machine learning unit 62 inputs the pixel classification results for multiple learning pixel regions 411, which constitute the second classification learning data, as input data into the input layer 21 of the inference model 20B.

[0146] The machine learning unit 62 uses an error function that compares the classification result (inference result) shown by the output data from the output layer 23 with the classification result (training data) constituting the second classification learning data. It repeatedly adjusts the weights corresponding to each synapse (backpropagation) to reduce the evaluation value of the error function. Furthermore, when the machine learning unit 62 determines that the learning termination conditions have been met, such as repeating the above series of processes a predetermined number of times and the evaluation value of the error function being smaller than the allowable value, it terminates the machine learning process and saves the inference model 20B (with all weights corresponding to each synapse) as the second classification learning model 2B in the learned model storage unit 63.

[0147] (Machined surface determination device 7)

[0148] Figure 15 This is a block diagram illustrating an example of the processing surface determination device 7 of the second embodiment.

[0149] The processing surface determination device 7, like the first embodiment, includes a classification result acquisition unit 70B, a determination result inference unit 71, a learning completed model storage unit 72, and an output processing unit 73.

[0150] The classification result acquisition unit 70B performs the following classification result acquisition processing (see below). Figure 16 ): For the multiple small image regions 430 that are divided into the decision image region 420 of the decision image 42, the classification result of classifying the state of the processing surface 100 into one of the multiple processing states is obtained in units of small image regions 430.

[0151] The classification result acquisition unit 70B includes: an image acquisition unit 700 and a small image generation unit 701, which are the same as those in the first embodiment; a pixel classification result acquisition unit 703, which acquires a pixel classification result representing the classification result for a pixel region based on the pixel value in the pixel region for each of the plurality of small images 43; and a second classification result inference unit 702B, which infers the classification result for the plurality of small image regions 430 by inputting the pixel classification result for the plurality of pixel regions into the second classification learning model 2B in units of small image regions 430.

[0152] The determination result inference unit 71 performs the following determination result inference processing: by inputting the classification results for multiple small image regions 430 obtained by the classification result acquisition unit 70B into the determination learning model 2, the determination result for the determination image region 420 is inferred.

[0153] The learned model storage unit 72 is a database that stores the learned second classification learning model 2B used in the inference processing of the classification result acquisition unit 70B and the learned judgment learning model 2 used in the inference processing of the judgment result inference unit 71.

[0154] Figure 16 This is a functional illustration diagram showing an example of the classification result acquisition process performed by the classification result acquisition unit 70B.

[0155] The determination image region 420 of the determination image 42 is, similarly to that in the first embodiment, the region captured by the imaging unit 4. The small image region 430 of the small image 43 is obtained by dividing the determination image region 420 of the determination image 42 into a grid pattern. The small image region 430 of the small image 43 corresponds to the learning image region 410 of the learning image 41. The plurality of pixel regions 431 constituting the small image 43 corresponds to the plurality of learning pixel regions 411 constituting the learning image 41.

[0156] Here, the second classification learning model 2B is obtained by machine learning the correlation between the pixel classification results for multiple learning pixel regions 411 corresponding to multiple pixel regions 431 and the classification results when classifying the state of the processing surface 100 within the multiple learning pixel regions 411 into one of multiple processing states based on the pixel classification results. Therefore, the second classification result inference unit 702B functions as a classifier, which inputs the pixel classification results for the multiple pixel regions 431 constituting each of the multiple small images 43 into the second classification learning model 2B in units of small image regions 430, thereby classifying the state of the processing surface 100 within the small image region 430 into one of the multiple processing states.

[0157] (Method for determining machined surfaces)

[0158] Figure 17 This is a flowchart illustrating an example of a machining surface determination method based on the machining surface determination device 7 of the second embodiment.

[0159] First, in step S100, the image acquisition unit 700 of the classification result acquisition unit 70B acquires the determination image 42.

[0160] Next, in step S110, the small image generation unit 701, as a preprocessing step for the determination image 42, generates multiple small images 43 from the determination image 42 by dividing the determination image region 420 of the determination image 42 into multiple small image regions 430.

[0161] Then, in step S112, the pixel classification result acquisition unit 703 acquires a pixel classification result for each of the multiple pixel regions 431 constituting each of the multiple small images 43, based on the pixel values ​​within the pixel regions 431.

[0162] Next, in steps S120 to S128, the second classification result inference unit 702B, with the number of segments of the multiple small images 43 set to K and the consecutive numbers (1≤n≤K) assigned to the multiple small images 43 respectively, increases the variable i from "1" to "K", thereby performing loop processing.

[0163] Specifically, in step S120, the second classification result inference unit 702B initializes variable i with "1". Next, in step S124, the second classification result inference unit 702B selects the i-th small image 43 and inputs the pixel classification results for the multiple pixel regions 431 constituting the small image 43 to the input layer 21 of the second classification learning model 2B, thereby inferring the classification result output from the output layer 23 of the second classification learning model 2B.

[0164] Next, in step S126, variable i is incremented, and in step S128, it is determined whether variable i exceeds the segmentation number K. Furthermore, the second classification result inference unit 702B repeatedly performs the above steps S124 and S126 until variable i exceeds the segmentation number K, thereby obtaining the classification result for multiple small image regions 430.

[0165] Next, in step S130, the determination result inference unit 71 infers the determination result (e.g., whether reprocessing is needed, whether other processing is needed, whether fine processing is needed, processing range, etc.) output from the output layer 23 of the determination learning model 2 by inputting the classification results for multiple small image regions 430 to the input layer 21 of the determination learning model 2.

[0166] Next, in step S140, the output processing unit 73 outputs information corresponding to the determination result deduced by the determination result inference unit 71 to the output unit (e.g., control device 5, operator terminal 8, etc.). Then, the process ends. Figure 17The following is a series of processing surface determination methods. In the processing surface determination method, step S100 is equivalent to the image acquisition process, steps S100 to S128 are equivalent to the classification result acquisition process, step S130 is equivalent to the determination result inference process, and step S140 is equivalent to the output processing process.

[0167] As described above, according to the processing surface determination apparatus 7 and processing surface determination method of this embodiment, the classification result acquisition unit 70B generates a plurality of small images 43 from the determination image 42 by dividing the determination image region 420 into small image regions 430, and inputs the pixel classification results of the plurality of pixel regions 431 constituting each of the plurality of small images 43 into the second classification learning model 2B, thereby inferring the classification result for the plurality of small image regions 430. Furthermore, the determination result inference unit 71 infers the state of the processing surface 100 as the determination result by inputting the classification results for the plurality of small image regions 430 into the determination learning model 2.

[0168] Therefore, by inputting each of the multiple small images 43 formed from the segmentation decision image 42, the classification result based on the second classification learning model 2B is inferred on a unit basis of small image regions 430. Thus, compared to inputting a single decision image 42 into the second classification learning model 2B, the collection of learning data required for machine learning is easier, and the accuracy of the second classification learning model 2B can be improved. Furthermore, by inputting the classification results for the multiple small image regions 430 obtained based on the second classification learning model 2B into the decision learning model 2, the state of the processing surface 100 contained in the decision image 42 is determined. Therefore, the state of the processing surface 100 possessed by the decision object 10 can be determined automatically.

[0169] (Other implementation methods)

[0170] This invention is not limited to the above-described embodiments and can be implemented with various modifications without departing from the spirit of the invention. Furthermore, all of these modifications are encompassed within the technical concept of this invention.

[0171] For example, in the above embodiment, the determination image area 420 is set such that a portion of a blade of the impeller of the determination object 10 is included as the processing surface 100 of the determination object. Conversely, the determination image area 420 can also be set such that by enlarging the impeller as a whole, it includes multiple processing surfaces 100 of multiple blades of the impeller as determination objects. That is, when the determination object 10 has multiple processing surfaces 100 that have been processed by the processing unit 3 through different processing steps, the determination image area 420 can also be set such that it includes multiple processing surfaces 100.

[0172] In this case, the classification result acquisition unit 70B acquires a determination image 42 obtained by capturing multiple processing surfaces 100, and sets a determination image region 420 for each processing surface in such a way that the determination image 42 is separated at the boundaries of the multiple processing surfaces 100. The boundaries of the processing surfaces 100 can be set in advance or by image processing performed on the determination image 42. Furthermore, the classification result acquisition unit 70B acquires a classification result for multiple small image regions 430 formed by dividing the determination image region 420 of each processing surface, on a unit basis of small image regions 430. Next, the determination result inference unit 71 infers the determination result for each processing surface determination image 42 by inputting the classification results for the multiple small image regions 430 into the determination learning model 2 according to the processing surface.

[0173] Furthermore, in the above implementation method, the use of CNN (refer to...) Figure 6 , Figure 7 The specific method of machine learning performed by the Machine Learning Department 62 has been described, but the Machine Learning Department 62 may also employ any other machine learning method. Examples of other machine learning methods include, for instance, tree-based methods such as decision trees and regression trees, ensemble learning methods such as bagging algorithms and boosting algorithms, neural network types such as recurrent neural networks and convolutional neural networks (including deep learning), hierarchical clustering, non-hierarchical clustering, clustering types such as k-nearest neighbors and k-means, multivariate analysis such as principal component analysis, factor analysis, and logistic regression, and support vector machines.

[0174] (Machining Surface Judgment Procedure)

[0175] This invention enables Figure 2 The computer 200 shown is provided as a program (machining surface determination program) 230 that functions the various parts of the machining surface determination device 7 described in the above embodiment. Furthermore, the present invention can also be used to... Figure 2 The computer 200 shown is provided in the form of a program (machining surface determination program) 230 for each process of the machining surface determination method of the above embodiment.

[0176] (Inference apparatus, inference method, and inference procedure)

[0177] This invention can be provided not only in the form of the processing surface determination apparatus 7 (processing surface determination method or processing surface determination program) of the above-described embodiments, but also in the form of an inference apparatus (inference method or inference program) used to determine the state of the processing surface 100. In this case, the inference apparatus (inference method or inference program) includes a memory and a processor, wherein the processor is capable of performing a series of processes. The series of processes includes: a classification result acquisition process (classification result acquisition step), which acquires a classification result for classifying the state of the processing surface 100 into one of a plurality of processing states for a plurality of small image regions 430 divided from the determination image region 420 of the determination image 42, on a unit of small image regions 430; and a determination result inference process (determination result inference step), which infers the state of the processing surface 100 contained in the determination image 42 as the determination result for the determination image 42 if a classification result for the plurality of small image regions 430 is acquired in the classification result acquisition process.

[0178] By providing it in the form of an inference device (inference method or inference procedure), it can be easily applied to various devices compared to the case where the machining surface determination device 7 is installed. Those skilled in the art will certainly understand that when the inference device (inference method or inference procedure) infers the state of the machining surface 100, the inference method implemented by the determination result inference unit 71 of the machining surface determination device 7 using the learned determination learning model 2 generated by the machine learning device 6 of the above embodiment can also be applied.

[0179] Industrial applicability

[0180] This invention can be applied to a machining surface determination device, a machining surface determination program, a machining surface determination method, a machining system, an inference device, and a machine learning device.

[0181] Explanation of reference numerals in the attached figures

[0182] 1: Processing system; 2: Learning model for judgment; 2A: Learning model for first classification; 2B: Learning model for second classification; 3: Processing unit; 4: Imaging unit; 5: Control device; 6: Machine learning device; 7: Processing surface judgment device; 8: Terminal for operator; 10: Object to be judged; 20, 20A, 20B: Inference model; 21: Input layer; 22: Intermediate layer; 22a: Convolutional layer; 22b: Pooling layer; 22c: Fully connected layer; 23: Output layer; 40: Image captured; 41: Image for learning; 42: Image for judgment; 43: Small image; 50: Control panel; 51: Operation display panel; 60: Data acquisition unit for learning; 61: 62: Learning data storage unit; 63: Machine learning unit; 70A, 70B: Classification result acquisition unit; 71: Judgment result inference unit; 72: Model storage unit; 73: Output processing unit; 100: Processing surface; 110: Background; 200: Computer; 400: Image capture area; 410: Learning image area; 411: Learning pixel area; 420: Judgment image area; 430: Small image area; 431: Pixel area; 700: Image acquisition unit; 701: Small image generation unit; 702A: First classification result inference unit; 702B: Second classification result inference unit; 703: Pixel classification result acquisition unit.

Claims

1. A processing surface determination device, which determines the state of the processing surface based on a determination image obtained by photographing the processing surface of a determination object, the processing surface determination device comprising: The classification result acquisition unit acquires, on a unit basis, a classification result for classifying the state of the processing surface into one of multiple processing states, based on multiple small image regions formed by segmenting the decision image region of the decision image; and a determination result inference unit that infers a determination result for the determination image by inputting the classification results for the plurality of small image regions to a determination-use learning model, wherein The decision learning model is obtained by machine learning the correlation between the classification results of multiple learning image regions corresponding to multiple small image regions and the decision results when the state of the processing surface in the multiple learning image regions is determined based on the classification results.

2. The processing surface determination device according to claim 1, wherein, The classification result acquisition unit has the following functions: An image acquisition unit acquires the determination image having the determination image region; The small image generation unit generates multiple small images from the determination image by dividing the determination image region into multiple small image regions; as well as The first classification result inference unit infers the classification result for the multiple small image regions by inputting the multiple small image regions into the first classification learning model as a unit. The first classification learning model is obtained by machine learning the correlation between the learning image having the learning image region and the classification result when the state of the processing surface contained in the learning image is classified into one of the multiple processing states.

3. The processing surface determination device according to claim 1, wherein, The classification result acquisition unit has the following functions: An image acquisition unit acquires the determination image having the determination image region; The small image generation unit generates multiple small images from the determination image by dividing the determination image region into multiple small image regions; The pixel classification result acquisition unit acquires a pixel classification result representing the classification result for each of the multiple pixel regions constituting each of the multiple small images, based on the pixel values ​​within the pixel regions, on a unit basis of the pixel regions. as well as The second classification result inference unit infers the classification result for the multiple small image regions by inputting the pixel classification results for the multiple pixel regions into the second classification learning model as a unit of the small image regions. The second classification learning model is obtained by machine learning the correlation between the pixel classification results for multiple learning pixel regions corresponding to the multiple pixel regions and the classification results when classifying the state of the processing surface in the multiple learning pixel regions into one of the multiple processing states based on the pixel classification results.

4. The processing surface determining device according to any one of claims 1 to 3, wherein, The classification result acquisition unit acquires the classification result on a unit basis, by dividing the determination image region of each of the processing surfaces obtained from the determination image obtained by capturing multiple processing surfaces into multiple small image regions. The determination result inference unit infers the determination result for each of the processing surfaces by inputting the classification results for multiple small image regions into the determination learning model according to the processing surface.

5. The processing surface determining device according to any one of claims 1 to 4, wherein, The classification result acquisition unit acquires the following classification result for each of the plurality of small image regions, on a unit basis: the classification result is the classification of the state of the processing surface in the small image region into one of the plurality of processing states that includes at least good and bad, or the classification result is the classification of the processing surface in the small image region into a non-judgment object on the grounds that there is an edge of the processing surface or a background other than the processing surface in the small image region.

6. The processing surface determining device according to any one of claims 1 to 5, wherein, The inference section of the determination result infers at least one of the following as the determination result: Whether or not reprocessing is required, performing the same processing steps as when the surface was processed; Whether or not other processing is required, which is different from the processing steps performed on the processed surface; Whether or not the operator needs to perform finishing on the machined surface; and The processing area in the processing surface is the object to be processed, the other processing, or the finishing.

7. The processing surface determining device according to any one of claims 1 to 6, wherein, The processed surface is the surface of the object being judged when it has been processed using a grinding, milling, cutting, or casting process.

8. The processing surface determination device according to any one of claims 1 to 7, wherein, The object of determination is fluid machinery or fluid components constituting the fluid machinery.

9. A program product comprising a machining surface determination program, the machining surface determination program enabling a computer to function as a machining surface determination device according to any one of claims 1 to 8.

10. A method for determining a machined surface, which determines the state of the machined surface based on a determination image obtained by photographing the machined surface of a determination object. The method for determining the processed surface includes the following steps: The classification result acquisition process involves, for each of the multiple small image regions segmented from the judgment image region, acquiring a classification result for classifying the state of the processing surface into a specific processing state from among multiple processing states, using each small image region as a unit; and The judgment result inference process involves inputting the classification results for multiple small image regions into a judgment learning model to infer the judgment result for the judgment image, wherein... The decision learning model is obtained by machine learning the correlation between the classification results of multiple learning image regions corresponding to multiple small image regions and the decision results when the state of the processing surface in the multiple learning image regions is determined based on the classification results.

11. A processing system comprising: The processing surface determination device according to any one of claims 1 to 8; The processing department processes the object to be determined; The imaging unit, which images the processed surface of the object to be judged; and The control unit controls the processing surface determination device, the processing unit, and the imaging unit.

12. An inference device for determining the state of a processed surface based on a determination image obtained by photographing a processed surface of a determination object. The inference device includes a memory and a processor. The processor performs the following processing: The classification result acquisition process involves, for each of the multiple small image regions segmented from the determination image region, acquiring the classification result for classifying the state of the processing surface into one of multiple processing states, using each small image region as a unit; and In the determination result inference process, if the classification result acquisition process obtains the classification result for multiple small image regions, then the state of the processing surface contained in the determination image is inferred as the determination result for the determination image.

13. A machine learning apparatus that generates a decision learning model for use in a machining surface determination apparatus, which determines the state of the machining surface based on a decision image obtained by photographing the machining surface of a determination object. The machine learning device includes: The learning data storage unit stores multiple sets of learning data. The learning data includes the following classification results as input data and the following judgment results as output data, wherein... The classification result is the classification result when the state of the processing surface is classified into one of the multiple processing states for each of the multiple learning image regions corresponding to the multiple small image regions divided into the decision image region of the decision image. The determination result is the determination result when the state of the processing surface in the multiple learning image regions is determined based on the classification result. The machine learning unit learns a decision learning model for inferring the correlation between the input data and the output data by inputting multiple sets of the learning data. as well as After the model storage unit has completed learning, it stores the decision-making learning model obtained by the machine learning unit.

Images