Processing surface determination device, processing surface determination program, processing surface determination method, and processing system

Abstract

The processing surface determination device (7) includes: an image acquisition unit (70) that acquires a determination image (42) having a predetermined determination image region in which a processing surface of a determination target object is captured; an inference unit (71) that inputs each of a plurality of small images (43) generated from the determination image (42) by dividing the determination image region into small image regions to a learning model (2), and thereby infers a classification result for the plurality of small images (43) in units of the small images, wherein the learning model (2) machine-learns a correlation between a learning image having a learning image region corresponding to the small image region and a classification result of classifying a state of a processing surface included in the learning image into one of a plurality of processing states; a classification result processing unit (73) that obtains a processing result by processing the classification result for the plurality of small images (43) inferred in units of the small images; and a determination unit (75) that determines the state of the processing surface based on the processing result.

Description

Technical Field

[0001] This invention relates to a machining surface determination device, a machining surface determination procedure, a machining surface determination method, and a machining system. Background Technology

[0002] In recent years, the manufacturing process of various products has seen advancements in the development of devices that automatically determine product quality using various sensors, replacing the need for operators to visually assess product quality. For example, Patent Document 1 discloses an inspection device that inspects the shape of an impeller by performing binarization processing relative to an image captured of the impeller as the object of inspection.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: JP 2008-51664 Summary of the Invention

[0006] As one of the indicators for judging product quality, the condition of the machined surface after various processing steps such as grinding, milling, cutting, or casting can be cited as an example. The condition of the machined surface includes various judgment items such as roughness, unevenness, undulation, warping, pattern, creases, and wrinkles.

[0007] However, the inspection device disclosed in Patent Document 1 is for inspecting the external shape of the object being inspected, but it cannot determine the condition of the machined surfaces within the object. Furthermore, if the condition of the machined surfaces is determined by an operator, it relies on the operator's skill or experience (including tacit knowledge), thus increasing the individual differences among operators and making it difficult to guarantee product quality.

[0008] In view of the above-mentioned problems, the present invention aims to provide a processing surface determination device, a processing surface determination program, a processing surface determination method, and a processing system capable of automatically determining the state of the processing surface of a determination object.

[0009] To achieve the above objectives, one aspect of the machining surface determination apparatus of the present invention is a machining surface determination apparatus for determining the state of a machining surface of a determination object, characterized in that it comprises:

[0010] The image acquisition unit acquires a determination image having a predetermined determination image area that captures the processed surface;

[0011] The inference unit inputs each of a plurality of small images generated from the decision image by dividing the decision image region into small image regions into a learning model, thereby inferring a classification result for the plurality of small images on a unit basis. The learning model learns the correlation between a learning image having a learning image region corresponding to the small image region and the classification result that classifies the state of the processing surface contained in the learning image into one of a plurality of processing states.

[0012] The classification result processing unit calculates a processing result by processing the classification results inferred from the small images on a per-small-image basis; and

[0013] The determination unit determines the state of the processed surface based on the processing result.

[0014] Invention Effects

[0015] According to the processing surface determination apparatus of the present invention, the inference unit inputs each of a plurality of small images generated from a determination image by dividing a determination image region into small image regions into a learning model, and infers a classification result for the plurality of small images on a unit basis. The classification result processing unit processes the classification result inferred for the plurality of small images on a unit basis to obtain a processing result, and the determination unit determines the state of the processing surface based on the processing result. Therefore, the state of the processing surface of a determination object can be automatically determined.

[0016] Other issues, structures, and effects beyond those described above become clear through the implementation methods used to carry out the invention described later. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating an example of a machining system 1 with a machining surface determination device 7 according to an embodiment.

[0018] Figure 2 This is a hardware configuration diagram showing an example of a computer 200 that constitutes a machine learning device 6 and a processing surface determination device 7.

[0019] Figure 3 This is a block diagram illustrating an example of a machine learning device 6 according to an implementation method.

[0020] Figure 4 This is a data structure diagram illustrating an example of data used for learning.

[0021] Figure 5 This is a schematic diagram illustrating an example of inference model 20 applied to learning model 2.

[0022] Figure 6This is a block diagram showing an example of the processing surface determination device 7 according to an embodiment.

[0023] Figure 7 This is a functional illustration diagram showing an example of inference processing based on inference unit 71.

[0024] Figure 8 This is a functional illustration diagram showing an example of the classification result processing unit 73 and the determination unit 74 when the ratio is set as the benchmark.

[0025] Figure 9 This is a functional illustration diagram showing an example of the classification result processing unit 73 and the determination unit 74 when the distribution is set as the basis.

[0026] Figure 10 This is a flowchart illustrating an example of a machining surface determination method of the machining surface determination apparatus 7 based on the embodiment. Detailed Implementation

[0027] Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. The scope required to achieve the objectives of the present invention is illustrated schematically below, primarily focusing on the scope necessary for describing corresponding parts of the invention; omissions in the description are considered prior art.

[0028] (Implementation Method)

[0029] Figure 1 This is a schematic diagram illustrating an example of a machining system 1 with a machining surface determination device 7 according to an embodiment.

[0030] The processing system 1 includes: a processing unit 3 for processing the object to be determined 10; an imaging unit 4 for photographing the processed surface 100 of the object to be determined 10; a processing surface determination device 7 for determining the state of the processed surface 100 of the object to be determined using a learning model 2; and a control device 5 for controlling the processing unit 3, the imaging unit 4, and the processing surface determination device 7. Additionally, the processing system 1 includes, as an additional component, a machine learning device 6 for generating the learning model 2.

[0031] The object to be judged 10 is formed from any material such as metal, resin, or ceramic, and is any article that can be processed based on the processing unit 3. Specifically, the object to be judged 10 is fluid machinery or a fluid component constituting fluid machinery. Furthermore, the three-dimensional shape, surface properties, color, size, etc., of the object to be judged 10 are not particularly limited.

[0032] The processing surface 100 is, for example, the surface of the determination object 10 when the determination object 10 is processed by the processing unit 3. The processing surface 100 can be any surface of the determination object 10, the entire surface of the determination object 10, or a part of it.

[0033] The machining unit 3 is composed of various robotic arms or machine tools that operate using electricity, fluid pressure, or other power sources. Based on control commands from the control device 5, the machining unit 3 performs machining processes such as grinding, milling, cutting, or casting. Furthermore, the machining unit 3 can perform any machining process as long as it can process or form the surface of the target object 10, and multiple machining processes can be combined.

[0034] exist Figure 1 In the machining system 1 shown, the machining unit 3 is composed of a robot arm with a replaceable grinding wheel mounted at its front end, used to perform the grinding process. In addition, the object to be determined 10 is a fluid component constituting the pump, which is an impeller with multiple blades, and the machining surface 100 is the surface of each blade processed by the grinding process based on the machining unit 3.

[0035] The imaging unit 4 is a camera for capturing images of the processing surface 100, and is constructed using an image sensor such as a CMOS sensor or a CCD sensor. The imaging unit 4 is mounted at a predetermined position capable of capturing images of the processing surface 100. If the processing unit 3 is, for example, a robotic arm, the imaging unit 4 can be mounted on the front end of the robotic arm or fixed above the mounting table (including movable ones) on which the object to be judged 10 is placed. Alternatively, if the processing unit 3 is, for example, a processing mechanism of a machine tool, the imaging unit 4 can be mounted inside the safety cover of the machine tool or fixed above a worktable separate from the machine tool.

[0036] The imaging unit 4 is installed in the aforementioned designated position, and its position or orientation is adjusted so that the processed surface 100 falls within the viewing angle of the imaging unit 4. Furthermore, the imaging unit 4, as... Figure 1 As shown, a camera unit 4 connected to the machine learning device 6 and a camera unit 4 connected to the processing surface determination device 7 can be separately provided. A single camera unit 4 can also be connected to both the machine learning device 6 and the processing surface determination device 7 and shared. Furthermore, the camera unit 4 can have translation / tilt / zoom functions. Moreover, the camera unit 4 is not limited to using a single camera to capture the processing surface 100; multiple cameras can also be used for capturing images.

[0037] Control device 5, for example, is equipped with a general-purpose or special-purpose computer (see below). Figure 2 A control panel 50 consisting of a microcontroller or similar component, and an operation display panel 51 consisting of a touch panel display, switches, buttons, etc.

[0038] The control panel 50 is connected to an actuator or sensor (not shown) of the machining unit 3. It controls the machining process based on the machining unit 3 by sending control commands to the actuator in accordance with the machining action parameters used to perform the machining process or the detection signals from the sensor. The control panel 50 sends an image capture command to the imaging unit 4, and as a result, receives the image captured by the imaging unit 4. The control panel 50 sends the captured image as a determination image to the machining surface determination device 7, and as a result, receives the state of the machining surface 100 determined by the machining surface determination device 7. Furthermore, the control panel 50 can also send the captured image to the machine learning device 6.

[0039] The operation display panel 51 accepts the operator's operations and outputs various information through display or sound.

[0040] Machine learning device 6 acts as the main body of the learning phase in machine learning. Machine learning device 6 generates learning model 2 and provides it to processing surface determination device 7 via any communication network or recording medium. Learning model 2 learns the correlation between the learning image and the classification result after classifying the state of the processing surface 100 contained in the learning image. Machine learning device 6 will be described in detail later.

[0041] The machining surface determination device 7 operates as the main body of the inference stage in machine learning. Using the learning model 2 generated by the machine learning device 6, the machining surface determination device 7 uses the image of the machining surface 100 captured by the imaging unit 4 as the determination image to determine the state of the machining surface 100 of the object 10. Details of the machining surface determination device 7 will be explained later.

[0042] Furthermore, each component of the processing system 1 can be configured as a working machine by embedding it into a frame. In this case, at least one of the machine learning device 6 and the processing surface determination device 7 can be embedded in the control device 5. Alternatively, each component of the processing system 1 can be configured as a processing device having a processing section 3 and an inspection device having an imaging section 4 and the processing surface determination device 7. In this case, the function of the control device 5 can be distributed between the processing device and the inspection device. Moreover, each component of the processing system 1 is connected together via a wireless or wired network. Therefore, at least one of the machine learning device 6 and the processing surface determination device 7 can be located away from the processing site where the processing section 3 and the imaging section 4 are located. In this case, the control device 5 can be located at the processing site or at another location.

[0043] Figure 2 This is a hardware configuration diagram showing an example of a computer 200 that constitutes a machine learning device 6 and a processing surface determination device 7.

[0044] The machine learning device 6 and the processing surface determination device 7 are each composed of a general-purpose or special-purpose computer 200. The computer 200, such as... Figure 2 As shown, its main components include a bus 210, a processor 212, a memory 214, an input device 216, a display device 218, a storage device 220, a communication I / F (interface) unit 222, an external machine I / F unit 224, an I / O (input / output) device I / F unit 226, and a media input / output unit 228. Furthermore, the aforementioned components may be appropriately omitted depending on the intended use of the computer 200.

[0045] The processor 212 consists of one or more arithmetic processing devices (CPU, MPU, GPU, DSP, etc.) and acts as the control unit that coordinates the overall operation of the computer 200. The memory 214 stores various data and programs 230, and is composed of volatile memory (DRAM, SRAM, etc.) that functions as main memory and non-volatile memory (ROM, flash memory, etc.).

[0046] Input device 216 may consist of, for example, a keyboard, mouse, keypad, or electronic pen. Display device 218 may consist of, for example, a liquid crystal display (LCD), an organic EL display (OLED), electronic paper, or a projector. Input device 216 and display device 218 may also be integrated as in a touch panel display. Storage device 220 may consist of, for example, an HDD or SSD, and stores various data required for the execution of the operating system or program 230.

[0047] The communication I / F unit 222 connects to a network 240, such as the Internet or an intranet, via wired or wireless means, and transmits and receives data with other computers according to prescribed communication specifications. The external machine I / F unit 224 connects to external machines 250, such as printers or scanners, via wired or wireless means, and transmits and receives data with the external machines 250 according to prescribed communication specifications. The I / O device I / F unit 226 connects to I / O devices 260, such as sensors and actuators, and transmits and receives various signals or data, such as sensor-based detection signals or actuator control signals, between the I / O devices 260. The media input / output unit 228 is configured as a drive device, such as a DVD drive or CD drive, and reads and writes data to media 270 such as DVDs or CDs.

[0048] In the computer 200 with the above configuration, the processor 212 executes the program 230 by calling the working memory area of ​​the sub-memory 214, and controls various parts of the computer 200 via the bus 210. Furthermore, the program 230 can be stored in the storage device 220 instead of the memory 214. The program 230 can be recorded in the form of an installable file or an executable file on a non-transitory recording medium such as a CD or DVD and provided to the computer 200 via the media input / output unit 228. The program 230 can also be provided to the computer 200 by downloading it via the network 240 through the communication I / F unit 222. Additionally, the computer 200 can also utilize hardware such as an FPGA or ASIC to implement various functions achieved by the processor 212 executing the program 230.

[0049] Computer 200 may be configured as a stationary computer or a portable computer, and is an electronic device of any form. Computer 200 may be a client computer, a server computer, or a cloud computer. Computer 200 may also be used in devices other than machine learning device 6 and processing surface determination device 7.

[0050] (Machine Learning Device 6)

[0051] Figure 3 This is a block diagram illustrating an example of a machine learning device 6 according to an implementation method.

[0052] The machine learning device 6 includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a learned model (already learned model) storage unit 63. The machine learning device 6, for example, comprises... Figure 2 The computer 200 shown is configured in this case. The learning data acquisition unit 60 is configured by a communication I / F unit 222 or an I / O device I / F unit 226 and a processor 212, the machine learning unit 62 is configured by the processor 212, and the learning data storage unit 61 and the learned model storage unit 63 are configured by a storage device 220.

[0053] The learning data acquisition unit 60 is connected to various external devices via a communication network, serving as an interface unit for acquiring learning data that correlates input and output data. External devices include, for example, the imaging unit 4, the processing surface determination device 7, and the operator's terminal 8.

[0054] The learning data storage unit 61 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 60. Furthermore, the specific configuration of the database constituting the learning data storage unit 61 can be appropriately designed.

[0055] The machine learning unit 62 uses the learning data stored in the learning data storage unit 61 to perform machine learning. That is, the machine learning unit 62 inputs multiple sets of learning data into the learning model 2, causing the learning model 2 to learn the correlation between the input data and the output data contained in the learning data, thereby generating the learning model 2.

[0056] The learned model storage unit 63 is a database storing the learned model 2 generated by the machine learning unit 62. The learned model 2 stored in the learned model storage unit 63 is provided to the actual system (e.g., the processing surface determination device 7) via any communication network or recording medium. Furthermore, the learned model 2 can also be provided to an external computer (e.g., a server computer or a cloud computer) and stored in the external computer's storage unit. Additionally, in Figure 3 The diagram shows that the learning data storage unit 61 and the learned model storage unit 63 are separate storage units, but these storage units can also be constructed using a single storage unit.

[0057] Figure 4 This is a data structure diagram illustrating an example of data used for learning.

[0058] The learning data includes the learning image 41 as input data and the classification result of the state of the processing surface 100 contained in the learning image 41 into one of multiple processing states as output data. These input data and output data are associated with each other.

[0059] The learning image 41, which serves as input data, is each of a plurality of images generated by dividing a captured image 40 having a defined captured image region 400 that captures the processing surface 100 of the object to be determined 10 using the capturing unit 4 into a learning image region 410.

[0060] The image area 400 of the image 40 is the area captured by the imaging unit 4 and is defined by the viewing angle of the imaging unit 4. Figure 4 The image area 400 shown is configured to include a portion of a blade of the impeller, which is the object of determination 10. Furthermore, in Figure 4 The captured image 40 shown captures not only the processed surface 100 but also the background 110, but the captured image area 400 can also be set in a way that does not capture the background 110.

[0061] Learning image 41, learning image region 410, such as Figure 4The image region 400 of the captured image 40 is divided into a grid pattern, with each of the learning image regions 410 being a square. Furthermore, the number, shape, size, and aspect ratio of the learning image regions 410 can be appropriately changed; for example, it can be rectangular or other shapes. Additionally, the method of dividing the captured image region 400 into the learning image regions 410 can be appropriately changed; for example, it can be divided in an alternating pattern or according to other criteria.

[0062] In supervised learning, the classification results as output data are referred to as teacher data or correct answer labels, for example. When multiple processing states are classified using a two-level system, such as "good" and "poor," the classification result is represented by one of "good" or "poor." When multiple processing states are classified using a three-level system, such as "good," "acceptable," and "poor," the classification result is represented by one of "good," "acceptable," and "poor." Furthermore, the multiple processing states when classifying the state of processing surface 100 are not limited to the above-mentioned levels; for example, they can be classified into four or more levels, or other perspectives can be used for classification.

[0063] Furthermore, it is also possible to add a "judgment object outside" level to the background 110 outside the processing surface 100 captured in the learning image 41 for classification. In this case, the classification result is represented by one of "good", "bad" and "judgment object outside" in the example of level 2 above. In the example of level 3 above, such as Figure 4 The results are indicated by one of the following: “Good,” “Acceptable,” “Poor,” and “Not Subject to Judgment.” Furthermore, when learning to capture both the processed surface 100 and the background 110 in image 41, for example, if the ratio of the background 110 is higher than the specified ratio, it can be classified as “Not Subject to Judgment,” and it will always be classified as “Not Subject to Judgment.”

[0064] The learning data acquisition unit 60 can employ various methods to acquire learning data. For example, the learning data acquisition unit 60 acquires a photographic image 40 of the object to be judged 10 after the processing step has been performed by the processing unit 3, captured by the photographing unit 4. This image is then displayed on the display screen of the operator's terminal 8, which generates multiple learning images 41 by dividing the photographic image 40. Then, if the operator visually confirms each of the learning images 41 on the display screen and inputs the classification result (classification result) for each of the learning images 41 using the operator's terminal 8, the learning data acquisition unit 60 acquires the learning data by correspondingly associating the learning images 41 (input data) on the display screen with the classification result (input data) of the input operation.

[0065] Therefore, the learning data acquisition unit 60 can acquire a number of learning data points equivalent to the number of divisions when dividing a single captured image 40 into multiple learning images 41, and by repeatedly performing the above operation, a desired number of learning data points can be acquired. Thus, learning data can be easily collected.

[0066] Figure 5 This is a schematic diagram illustrating an example of inference model 20 applied to learning model 2.

[0067] The inference model 20 employs a convolutional neural network (CNN) model as a specific method of machine learning. The inference model 20 has an input layer 21, an intermediate layer 22, and an output layer 23.

[0068] The input layer 21 has a number of neurons corresponding to the number of learning images 41 used as input data, and the pixel value of each pixel is input to each neuron respectively.

[0069] The intermediate layer 22 consists of a convolutional layer 22a, a pooling layer 22b, and a fully connected layer 22c. For example, multiple layers of convolutional layers 22a and pooling layers 22b are alternately provided. The convolutional layers 22a and pooling layers 22b extract feature values ​​from the image input via the input layer 21. The fully connected layer 22c, for example, uses an activation function to transform the feature values ​​extracted from the image by the convolutional layers 22a and pooling layers 22b, and outputs them as a feature vector. Furthermore, multiple layers of the fully connected layer 22c can also be provided.

[0070] Output layer 23 outputs output data, including the classification result, based on the feature vector output from fully connected layer 22c. In addition to the classification result, the output data may also include, for example, a score representing the reliability of the classification result.

[0071] Synapses are provided between the layers of the inference model 20 to connect the neurons between the layers. The synapses and weights in the convolutional layer 22a and the fully connected layer 22c of the intermediate layer 22 are associated with each other.

[0072] The machine learning unit 62 inputs the learning data into the inference model 20, enabling the inference model 20 to machine learn the correlation between the learning image 41 and the classification result. Specifically, the machine learning unit 62 inputs the learning image 41, which constitutes the learning data, as input data into the input layer 21 of the inference model 20. Furthermore, as a preprocessing step when inputting the learning image 41 into the input layer 21, the machine learning unit 62 can perform prescribed image adjustments on the learning image 41 (e.g., image format, image size, image filter, image mask, etc.).

[0073] The machine learning unit 62 repeatedly adjusts the weights associated with each synapse (backpropagation) by comparing the classification result (inference result) represented by the output data from the output layer 23 with the classification result (teacher data) constituting the learning data, thereby reducing the evaluation value of the error function. Then, the machine learning unit 62 terminates the machine learning when it determines that the prescribed learning termination conditions, such as having performed the above series of processes a predetermined number of times and the evaluation value of the error function becoming smaller than the allowable value, have been met. The inference model 20 at this time (with all weights associated with each synapse) is stored as the learning model 2 in the learned model storage unit 63.

[0074] (Machined surface determination device 7)

[0075] Figure 6 This is a block diagram showing an example of the processing surface determination device 7 according to an embodiment.

[0076] The machining surface determination device 7 includes an image acquisition unit 70, an inference unit 71, a learned model storage unit 72, a classification result processing unit 73, a determination unit 74, and an output processing unit 75. The machining surface determination device 7, for example, comprises... Figure 2 The computer 200 shown is configured in this case. The image acquisition unit 70 is configured by a communication I / F unit 222 or an I / O device I / F unit 226 and a processor 212, the inference unit 71, the classification result processing unit 73, the determination unit 74 and the output processing unit 75 are configured by the processor 212, and the learned model storage unit 72 is configured by a storage device 220.

[0077] The image acquisition unit 70 is an interface unit connected to the imaging unit 4. This interface unit acquires an image of the processing surface 100 of the object to be judged 10 captured by the imaging unit 4 and uses it as a judgment image 42 with a predetermined judgment image area.

[0078] Inference unit 71 inputs each of the multiple small images 43 generated from decision image 42 by dividing the decision image region into small image regions into learning model 2, and performs inference processing on a small image-by-small image basis to infer the classification results for the multiple small images 43 (see below). Figure 7 At this time, the inference unit 71 records the positional relationship of each small image region 430 relative to the decision image region 420, for example, as additional information of the small image 43, in a manner that allows the decision image 42 before division to be reconstructed from multiple small images 43. Furthermore, part or all of the inference unit 71 can be replaced by the processor of an external computer (e.g., a server computer or a cloud computer), and part or all of the inference processing based on the inference unit 71 can also be executed by an external computer.

[0079] The learned model storage unit 72 is a database storing the learned models 2 that have been learned and are used in the inference process of the inference unit 71. Furthermore, the number of learned models 2 stored in the learned model storage unit 72 is not limited to one; for example, multiple learned models can be stored and selectively utilized based on machine learning methods, processing steps of the processing unit 3, and conditions such as the determination of the object 10. Alternatively, the learned model storage unit 72 can be replaced by the storage unit of an external computer (e.g., a server computer or a cloud computer), in which case the inference unit 71 can perform the aforementioned inference process by accessing that external computer.

[0080] The classification result processing unit 73 obtains the processing result by processing the classification results inferred from multiple small images 43 on a small image-by-small image basis (see below). Figure 8 At this time, when the inference unit 71 functions as a classifier that classifies the background 110 captured in the small image 43 as not being a determination object, the classification result processing unit 73 can also obtain a processing result by processing the classification results for the small images 43 other than those classified as determination objects from the classification results for the multiple small images 43.

[0081] The classification result processing unit 73 can process the classification results for multiple small images 43 using any method based on ratios, distributions, statistics, etc. Furthermore, multiple bases can be combined, or other bases can be used.

[0082] For example, with the ratio as a benchmark, the classification result processing unit 73 calculates, as a processing result, the ratio of the total number of small images 43 classified as a specified processing state among multiple processing states to the total number of small images 43 captured on the processing surface 100. That is, the classification result processing unit 73 calculates the above ratio in order to evaluate the state of the processing surface 100 from the perspective of what proportion of the small images 43 classified as the specified processing state are relative to the entire determination image 42 before division.

[0083] Furthermore, with the distribution as a benchmark, the classification result processing unit 73, as the processing result, calculates the distribution of the set of small images 43 classified into a specified processing state among multiple processing states relative to the set of small images 43 captured on the processing surface 100. In other words, the classification result processing unit 73 calculates the aforementioned distribution in order to evaluate the state of the processing surface 100, considering the degree of deviation between the small images 43 classified into the specified processing state and their positions relative to the overall determination image 42 before classification.

[0084] The determination unit 74 determines the state of the processing surface 100 based on the processing result obtained by the classification result processing unit 73. Specifically, the determination unit 74 determines, based on the state of the processing surface 100, whether further processing is required, whether other processing is required, whether final processing is required, and whether it is at least one of the processing ranges that is subject to further processing, other processing, or final processing of the processing surface 100. In the case of further processing, the same processing steps as when the processing surface 100 was processed are performed again. In the case of other processing, different processing steps than when the processing surface 100 was processed are performed. In the case of final processing, the operator performs final processing relative to the processing surface 100.

[0085] The output processing unit 75 performs output processing for outputting the determination result determined by the determination unit 74. The specific output means for outputting the determination result can take various forms. For example, the output processing unit 75 may send reprocessing or other processing operation instructions to the processing unit 3 via the control panel 50 based on the determination result, or notify the operator of the final processing implementation via the operation display panel 51 or the operator's terminal 8 through display or sound, or store it in the control panel 50 as the operation history of the processing unit 3. Furthermore, the output processing unit 75 may only output (send, notify, store) the determination result of the processing surface 100 based on the determination unit 74, or in addition to the determination result of the processing surface 100, it may also output (send, notify, store) the classification result of multiple small images 43 based on the inference unit 71, or the processing result based on the classification result processing unit 73.

[0086] Figure 7 This is a functional illustration diagram showing an example of the inference processing of the inference section 71.

[0087] The determination image region 420 of the determination image 42 is the region captured by the imaging unit 4 and is defined by the viewing angle of the imaging unit 4. Figure 7 The determination is shown using image region 420 and... Figure 4 Similarly, the image capture area 400 shown is configured to include a portion of a blade of the impeller that serves as the judgment object 10. Furthermore, the judgment image area 420 can be set at a different location than the image capture area 400, and the number of images, shape, size, and aspect ratio of the two can also differ.

[0088] Small image 43, small image region 430, such as Figure 7As shown, the decision image region 420 of the decision image 42 is divided into a grid pattern, with each of the small image regions 430 being a square. The small image region 430 of the small image 43 is equivalent to the learning image region 410 of the learning image 41 when the learning model 2 is generated using the machine learning device 6, and the number, shape, size, and aspect ratio of the two images are preferably the same or equivalent.

[0089] Therefore, if the number of images, shape, size, and aspect ratio of the small image region 430 are equivalent to those of the learning image region 410, the method for dividing the decision image region 420 into the small image region 430 can be appropriately changed. For example, it can be divided in an alternating manner or according to other criteria. In this case, the method for dividing the decision image region 420 into the small image region 430 can be the same as or different from the method for dividing the captured image region 400 into the learning image region 410.

[0090] Here, the learning model 2 enables the machine learning device 6 to learn the correlation between the learning image 41, which has a learning image region 410 corresponding to a small image region 430, and the classification result that classifies the state of the processing surface 100 contained in the learning image 41 into one of multiple processing states. Therefore, the inference unit 71 functions as a classifier that classifies the state of the processing surface 100 captured in the small image 43 into one of multiple processing states on a small image basis by inputting each of the small images 43 into the learning model 2. When two levels (good, bad) are used as multiple processing states, the classification result based on the inference unit 71 is expressed as two levels (good, bad); when three levels (good, acceptable, bad) are used as multiple processing states, the classification result based on the inference unit 71 is expressed as three levels (good, acceptable, bad).

[0091] Additionally, the learning model 2 can also enable the machine learning device 6 to machine learn the correlation between a learning image 41 capturing at least one of the processing surface 100 or the background 110 outside the processing surface 100, and a classification result that categorizes the state of the processing surface 100 captured in the learning image 41 into one of multiple processing states and classifies the background 110 captured in the learning image 41 as outside the judgment object. In this case, the inference unit 71 functions as a classifier that, by inputting each of the small images 43 into the learning model 2, classifies the state of the processing surface 100 captured in the small image 43 into one of multiple processing states on a small image basis and classifies the background 110 captured in the small image 43 as outside the judgment object. In the classification result, outside the judgment object is further added to the multiple processing states; therefore, the classification result based on the inference unit 71 in the above example is expressed as a 3-level (good, bad, outside the judgment object) or a 4-level (good, acceptable, bad, outside the judgment object).

[0092] Furthermore, the classification result for the small image 43 based on the inference unit 71 can include scores (reliability) for each level. In this case, if the classification result is expressed using 4 levels (good, acceptable, poor, not a judgment object), the score for each level of the specific small image 43 can be output, for example, in the form of "0.02", "0.10", "0.95", or "0.31". The scores can be used in any way; for example, the highest score level (in the above example, "poor" with a score of "0.95") can be used as the classification result for the small image 43. If the score of a specified level exceeds a specified score benchmark value (in the above example, the score of "poor" level "0.95" exceeds the score benchmark value "0.80"), that level can also be used as the classification result for the small image 43.

[0093] Furthermore, the classification results for the small image 43 based on the inference unit 71 are preferably stored in the learned model storage unit 72 or other storage device (not shown). Past classification results, for example, further improve the inference accuracy of the learned model 2 after learning, and thus can be used as learning data for online learning or relearning.

[0094] Figure 8 This is a functional illustration diagram showing an example of the classification result processing unit 73 and the determination unit 74 when the ratio is set as the benchmark. Figure 9 This is a functional illustration diagram showing an example of the classification result processing unit 73 and the determination unit 74 when the distribution is set as the basis.

[0095] For specific examples of the classification result processing unit 73 and the determination unit 74, such as Figure 8 , Figure 9As shown, let's assume and explain the case where a decision image 42 is divided into 60 smaller images 43.

[0096] like Figure 8 As shown, when using the processing result based on the ratio, the classification result processing unit 73, for example, calculates the remaining small images 43 excluding the 11 small images 43 classified as "excluding the judgment object," that is, the total number of small images 43 captured on the processing surface 100 is 49. Furthermore, the classification result processing unit 73 calculates a first ratio of the total number of small images 43 classified as "good" (25 images) to the total number (49 images), a second ratio of the total number of small images 43 classified as "good" and "acceptable" (40 images) to the total number (49 images), or a third ratio of the total number of small images 43 classified as "poor" (11 images) to the total number (49 images). In addition, the classification result processing unit 73 can calculate all of the first to third ratios, calculate a single ratio, or calculate a ratio based on other calculation formulas.

[0097] The determination unit 74 determines the state of the processed surface 100 by comparing the ratio obtained as a processing result in the above manner with a predetermined ratio reference value. For example, if the first ratio is 80% or higher (ratio reference value), the determination unit 74 may determine that "no further processing is needed," "no other processing is needed," or "no final processing is needed." Furthermore, if the second ratio is 70% or lower, it may determine that "reprocessing is needed." Moreover, if the third ratio is 50% or higher, it may determine that "other processing is needed," if the third ratio is between 10% and 50%, it may determine that "reprocessing is needed," and if the third ratio is between 5% and 10%, it may determine that "final processing is needed." In addition, if the determination unit 74 determines that "reprocessing is needed," "other processing is needed," or "final processing is needed," it may, for example, determine that the area including at least the small image 43 classified as "defective" is the processing area.

[0098] like Figure 9 As shown, when using the processing result with the distribution as the basis, the classification result processing unit 73, for example, calculates the set of small images 43 remaining excluding the 11 small images 43 classified as "excluding the judgment object", that is, the set of small images 43 captured on the processing surface 100 (processing surface set 120). Then, the classification result processing unit 73, for example, calculates the distribution of the set of small images 43 classified as "defective" (defect set 130) relative to its processing surface set 120. As for the distribution of the defect set 130 relative to the processing surface set 120, for example, it is assumed that the 9 small images 43 classified as "defective" are adjacent, presenting a first distribution situation of clustering in a specific part of the processing surface set 120. Figure 9(on the left), or presenting a second distribution of 3 small images 43 classified as "defective" that are not adjacent to each other and are scattered throughout the processing surface set 120. Figure 9 (Right side).

[0099] Then, the determination unit 74 determines the state of the processed surface 100 by comparing the distribution obtained as a processing result in the above manner with a predetermined distribution benchmark. For example, if the distribution of the defect set 130 relative to the processed surface set 120 is a first distribution condition, the determination unit 74 may determine that "reprocessing is required," "other processing is required," or "final processing is not required." Furthermore, if the distribution is a second distribution condition, it may determine that "reprocessing is not required," "other processing is not required," or "final processing is required." Moreover, if the determination unit 74 determines that "reprocessing is required," "other processing is required," or "final processing is required," it may determine that the area containing at least the defect set 130 is the processing area.

[0100] Furthermore, if the classification result for the small image 43 based on the inference unit 71 includes scores for each level, the classification result processing unit 73 can also use these scores to calculate the processing result. For example, the classification result processing unit 73 can calculate the ratio of the total number of small images 43 whose scores for the "Good" level exceed a predetermined score benchmark value (e.g., "0.90") to the total number of images, and it can also calculate the distribution of the set of small images 43 whose scores for the "Good" level exceed a predetermined score benchmark value (e.g., "0.90") relative to the processing surface set 120.

[0101] (Method for determining machined surfaces)

[0102] Figure 10 This is a flowchart illustrating an example of a machining surface determination method of the machining surface determination apparatus 7 based on the embodiment. Furthermore, the machining surface determination apparatus 7 repeatedly executes [the method] at a predetermined time interval. Figure 10 A series of processing surface determination methods are shown. The specified timing can be arbitrary, for example, after the processing step based on processing unit 3 is completed, during the processing step, or when a specified event occurs (when the operator operates, when an instruction is received from the production management system, etc.). Hereinafter, the case where the processing surface determination method is performed relative to the determination object 10 processed by the processing step based on processing unit 3 is described.

[0103] First, in step 100, if the processing step based on the processing unit 3 is completed, the processing surface 100 of the object to be judged 10 processed by the processing step is captured by the imaging unit 4, and the captured image is sent to the processing surface judgment device 7 via the control device 5. Thereby, the image acquisition unit 70 of the processing surface judgment device 7 acquires the captured image as the judgment image 42.

[0104] Next, in step S110, the inference unit 71, as a preprocessing step based on the decision image 42, generates multiple small images 43 by dividing the decision image region 420 of the decision image 42 into small image regions 430.

[0105] Next, in steps S120 to S126, the inference unit 71 sets the number of divisions of the multiple small images 43 to K. While assigning sequence numbers (1≤n≤K) to the multiple small images 43 respectively, the variable i is gradually increased from "1" to "K" to perform loop processing.

[0106] Specifically, in step S120, the inference unit 71 initializes variable I to "1". Next, in step S122, the inference unit 71 selects the i-th small image 43, inputs it to the input layer 21 of the learning model 2, and obtains the classification result output from the output layer 23 of the learning model 2.

[0107] Next, in step S124, variable i is incremented, and in step S126, it is determined whether variable i exceeds the number of divisions K. Then, the inference unit 71 repeats the above steps S122 and S124 until variable i exceeds the number of divisions K, and infers the classification result for multiple small images 43 on a small image basis.

[0108] Next, in step S130, the classification result processing unit 73, as a post-processing unit for the classification results, calculates the processing result (e.g., ratio, distribution, statistics, etc.) by processing the classification results for multiple small images 43 inferred on a small image basis.

[0109] Next, in step S140, the determination unit 74 determines the state of the processed surface 100 (e.g., whether further processing is required, whether other processing is required, whether final processing is required, processing range, etc.) based on the processing result obtained by the classification result processing unit 73.

[0110] Next, in step S150, the output processing unit 75 outputs information corresponding to the determination result determined by the determination unit 74 to an output means (e.g., control device 5, operator terminal 8, etc.). Then, the process ends. Figure 10A series of processing surface determination methods are shown. In the processing surface determination method, step S100 is equivalent to the image acquisition process, steps S110 to S126 are equivalent to the inference process, step S130 is equivalent to the classification result processing process, step S140 is equivalent to the determination process, and step S150 is equivalent to the output processing process.

[0111] As described above, according to the processing surface determination device 7 and processing surface determination method of this embodiment, the inference unit 71 divides the determination image region 420 into small image regions 430 and inputs each of the multiple small images 43 generated from the determination image 42 to the learning model 2. Thereby, the classification result for the multiple small images 43 is inferred on a small image basis. The classification result processing unit 73 processes the classification result for the multiple small images 43 inferred on a small image basis to obtain the processing result. The determination unit 74 determines the state of the processing surface 100 based on the processing result.

[0112] Therefore, by inputting each small image 43 after dividing the judgment image 42, the classification result based on the learning model 2 is inferred on a small image basis. Thus, compared to inputting a single judgment image into the learning model 2, it is easier to collect the learning data required for machine learning, and the accuracy of the learning model 2 can be improved. Then, as post-processing for the classification result based on the learning model 2, the state of the processing surface 100 is determined based on the processing result obtained by processing the classification results for multiple small images 43. Therefore, the state of the processing surface 100 possessed by the judgment object 10 can be automatically determined.

[0113] (Other implementation methods)

[0114] This invention is not limited to the embodiments described above, and various modifications can be made to implement it without departing from the spirit of the invention. All of these are then included within the technical concept of this invention.

[0115] For example, in the above embodiments, Figure 7 The image region 420 for determination shown is configured to include a portion of a blade of the impeller of the object to be determined 10 as the processed surface 100 of the object to be determined. Conversely, by expanding to the entire impeller, the image region 420 for determination is configured to include multiple blades of the impeller as multiple processed surfaces 100 of the object to be determined. That is, when the object to be determined 10 has multiple processed surfaces 100 that have been processed by the processing unit 3 through different processing steps, the image region 420 for determination is configured to include multiple processed surfaces 100.

[0116] In this case, the image acquisition unit 70 acquires a determination image 42 capturing multiple processing surfaces 100. Then, the classification result processing unit 73 processes the classification results for each processing surface against the multiple small images 43 to obtain a processing result, and the determination unit 74 determines the state of the multiple processing surfaces 100 for each processing surface based on the processing result. Furthermore, the inference unit 71, similar to the embodiment described above, simply infers the classification result for the multiple small images 43 generated from the determination image 42 by inputting each of them into the learning model 2, on a small image basis. Additionally, the boundaries of the multiple processing surfaces 100 in the determination image 42 can be predetermined, set using image processing of the determination image 42, or set based on the classification result.

[0117] Furthermore, in the above embodiment, as a specific method for machine learning based on the machine learning unit 62, a CNN (see [reference]) was described. Figure 5 However, the Machine Learning Department 62 can also employ any other machine learning methods. Examples of other machine learning methods include tree-based methods such as decision trees and regression trees, ensemble learning such as bagging and boosting, neural network-based methods such as recurrent neural networks and convolutional neural networks (including deep learning), clustering types such as hierarchical clustering, non-hierarchical clustering, k-nearest neighbors, and k-means, multivariate analysis such as principal component analysis, factor analysis, and logistic regression, and support vector machines.

[0118] (Machining Surface Judgment Procedure)

[0119] The present invention can provide a program (machining surface determination program) 230 in which each part of the machining surface determination device 7 of the above embodiment functions. Figure 2 The computer 200 is shown. Furthermore, the present invention can also be used to make... Figure 2 The computer 200 shown is provided in the form of a program (machining surface determination program) 230 for each process of the machining surface determination method of the above embodiment.

[0120] (Inference apparatus, inference method, and inference procedure)

[0121] The present invention can be provided not only in the form of a machining surface determination device 7 (machining surface determination method or machining surface determination program) based on the above-described embodiments, but also in the form of an inference device (inference method or inference program) used for determining the state of the machining surface 100. In this case, the inference device (inference method or inference program) includes a memory and a processor, wherein the processor is capable of performing a series of processes. The series of processes includes: image acquisition processing (image acquisition step), which acquires a judgment image 42 having a defined judgment image region 420 that captures the processing surface 100; inference processing (inference step), which, when the judgment image 42 is acquired by the image acquisition processing, infers a classification result that classifies the state of the processing surface 100 into one of a plurality of processing states relative to each of the plurality of small images 43 generated from the judgment image 42 by dividing the judgment image region 420 into small image regions 430, on a small image basis; classification result processing (classification result processing step), which obtains a processing result by processing the classification result inferred on a small image basis for the plurality of small images 43; and judgment processing (judgment step), which determines the state of the processing surface 100 based on the processing result.

[0122] By providing it in the form of an inference device (inference method or inference procedure), it can be easily applied to various devices compared to the case where the machining surface determination device 7 is installed. Those skilled in the art will certainly understand that when the inference device (inference method or inference procedure) infers the state of the machining surface 100, the learned learning model 2 generated by the machine learning device 6 of the above embodiment can be used to apply the inference method implemented by the inference unit 71 of the machining surface determination device 7.

[0123] Industrial availability

[0124] This invention can be used in a machining surface determination device, a machining surface determination program, a machining surface determination method, and a machining system.

[0125] Explanation of reference numerals in the attached figures

[0126] 1. Machining system; 2. Learning model; 3. Machining unit; 4. Imaging unit; 5. Control device

[0127] 6 Machine learning device, 7 Processing surface determination device, 8 Operator terminal, 10 Judgment

[0128] Target object,

[0129] 20 Inference Model, 21 Input Layer, 22 Intermediate Layer, 22a Convolutional Layer

[0130] 22b Pooling layer, 22c Fully connected layer, 23 Output layer

[0131] 40. Images taken, 41. Images for learning, 42. Images for judgment, 43. Small images

[0132] 50 Control Panel, 51 Operation Display Panel

[0133] 60 Learning Data Acquisition Department, 61 Learning Data Storage Department

[0134] 62 Machine Learning Department, 63 Model Storage Department

[0135] 70 Image Acquisition Unit, 71 Inference Unit, 72 Model Storage Unit,

[0136] 73 Classification Result Processing Department, 74 Judgment Department, 75 Output Processing Department

[0137] 100 machined surface, 110 background, 120 machined surface set, 130 defect set

[0138] 200 computers

[0139] 400 Image capture area, 410 Image learning area

[0140] 420 Image region for determination, 430 Small image region.

Claims

1. A processing surface determination device, which determines the state of the processing surface of a processing object, characterized in that it comprises: The image acquisition unit acquires a determination image having a predetermined determination image area that captures the processed surface; The inference unit inputs each of a plurality of small images generated from the decision image by dividing the decision image region into small image regions into a learning model, thereby inferring a classification result for the plurality of small images on a unit basis. The learning model learns the correlation between a learning image having a learning image region corresponding to the small image region and the classification result that classifies the state of the processing surface contained in the learning image into one of a plurality of processing states. The classification result processing unit calculates a processing result by processing the classification results inferred for multiple small images on a per-small-image basis. as well as The determination unit, based on the processing result, determines the state of the processed surface. The inference unit inputs each of the plurality of small images into a learning model that has machine-learned the correlation between a learning image that captures at least one of the processing surface or the background outside the processing surface, and a classification result that classifies the state of the processing surface captured in the learning image as one of a plurality of processing states and classifies the background captured in the learning image as outside the determination object. Thus, the inference unit infers the classification result for the plurality of small images on a per-image basis. The classification result processing unit processes the classification results for multiple small images inferred on a per-small-image basis, excluding the small images classified as the determination object, to obtain the processing result.

2. The processing surface determination device according to claim 1, characterized in that, The classification result processing unit calculates the ratio of the total number of small images classified into a specified processing state among the multiple processing states to the total number of small images of the processing surface captured, and uses this as the processing result.

3. The processing surface determination device according to claim 1, characterized in that, The classification result processing unit calculates the distribution of the set of small images classified into a specified processing state among the multiple processing states relative to the set of small images captured on the processing surface, and uses this as the processing result.

4. The processing surface determination device according to any one of claims 1 to 3, characterized in that, The image acquisition unit acquires images of multiple processing surfaces for determination. The classification result processing unit processes the classification results of the multiple small images for each processing surface to obtain a processing result. The determination unit determines the state of multiple processing surfaces for each processing surface based on the processing result.

5. The processing surface determination device according to any one of claims 1 to 3, characterized in that, As the state of the processed surface. The determination unit determines whether further processing is required, whether other processing is required, whether final processing is required, and at least one of the processing surfaces that constitutes the processing range for the further processing, other processing, or final processing. In the reprocessing, the same processing steps as when the surface was processed are performed again. In the other processing steps, different processing steps are performed than when the surface being processed was processed. In the final processing, the operator performs the final processing relative to the processing surface.

6. The processing surface determination device according to any one of claims 1 to 3, characterized in that, The processed surface is the surface of the object being judged when it has been processed using a grinding process, a milling process, a cutting process, or a casting process.

7. The processing surface determination device according to any one of claims 1 to 3, characterized in that, The object of determination is a fluid machine or a fluid component constituting the fluid machine.

8. A processing surface determination device for determining the state of a processing surface of an object, characterized in that it comprises: The image acquisition unit acquires a determination image having a predetermined determination image area that captures the processed surface; The inference unit inputs each of a plurality of small images generated from the decision image by dividing the decision image region into small image regions into a learning model, thereby inferring a classification result for the plurality of small images on a unit basis. The learning model learns the correlation between a learning image having a learning image region corresponding to the small image region and the classification result that classifies the state of the processing surface contained in the learning image into one of a plurality of processing states. The classification result processing unit calculates a processing result by processing the classification results inferred for multiple small images on a per-small-image basis. as well as The determination unit, based on the processing result, determines the state of the processed surface. As to the state of the processed surface, the determination unit determines whether further processing is required, whether other processing is required, whether final processing is required, and at least one of the processing ranges of the processed surface that becomes the object to be further processed, the other processing, or the final processing. In the reprocessing, the same processing steps as when the surface was processed are performed again. In the other processing steps, different processing steps are performed than when the surface being processed was processed. In the final processing, the operator performs the final processing relative to the processing surface.

9. A program product comprising a processing surface determination program, characterized in that, The machining surface determination program enables the computer to function as the machining surface determination device according to any one of claims 1 to 8.

10. A method for determining a machined surface, which determines the machined surface of an object to be determined, characterized in that the method comprises the following steps: The image acquisition process acquires a determination image having a defined determination image area that captures the processing surface; The inference process involves inputting each of a plurality of small images generated from the decision image by dividing the decision image region into small image regions into a learning model, thereby inferring a classification result for the plurality of small images on a unit basis. The learning model learns the correlation between a learning image having a learning image region corresponding to the small image region and the classification result that classifies the state of the processing surface contained in the learning image into one of a plurality of processing states. The classification result processing step obtains the processing result by processing the classification results inferred from the small images as units for multiple small images; as well as The determination process involves assessing the state of the processed surface based on the processing results. In the inference process, each of the plurality of small images is input into a learning model that has machine-learned the correlation between a learning image that captures at least one of the processing surface or the background outside the processing surface, and a classification result that categorizes the state of the processing surface captured in the learning image as one of a plurality of processing states and classifies the background captured in the learning image as outside the judgment object. Thus, the classification result for the plurality of small images is inferred on a per-small-image basis. In the classification result processing step, the processing result is obtained by processing the classification results for multiple small images inferred on a unit basis, excluding the small images classified as the determination object.

11. A processing system, characterized in that, have: The processing surface determination device according to any one of claims 1 to 8; A processing unit that processes the object to be determined; A camera unit that captures images of the processed surface of the object to be determined; as well as A control unit that controls the processing surface determination device, the processing unit, and the imaging unit.

Images