PROGRAM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING APPARATUS
Patent Information
- Application Number
- JP2023580606
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2023-12-26
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2043-12-26
AI Technical Summary
Existing methods for detecting tool wear in cutting tools, such as those described in Patent Document 1, often fail to accurately identify wear areas due to issues like coating peeling, leading to inconsistent and potentially inaccurate assessments of tool deterioration.
A program that utilizes a trained learning model to analyze images of cutting tool blades from multiple angles, employing models for top, oblique, outer periphery, and rake face surfaces to detect wear and chipping areas with high precision, integrating with a tool management system for accurate deterioration state determination.
The system enables precise and automated detection of tool deterioration, reducing human error and workload, allowing timely replacement and improving machining efficiency by ensuring accurate assessment of blade condition.
Abstract
Description
[Technical field]
[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]
[0002] At manufacturing processing sites, management of tools used in processing is an important task. For example, in cutting tools used in cutting processing, the chipping and wear of the cutting edge affect the processing accuracy, so the deterioration state of the cutting edge must be appropriately judged and replaced with a new one as necessary. Currently, in many sites, skilled workers visually check the deterioration state of the cutting edge of the cutting tool and judge whether it is necessary to replace it with a new one. In addition, Patent Document 1 discloses a technology for analyzing the wear state of a tool by photographing the cutting edge of the tool after cutting processing at different angles and analyzing it by image processing. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2023-9980 Summary of the Invention [Problem to be solved by the invention]
[0004] In the technology disclosed in Patent Document 1, multiple image files are taken during one revolution around the tool, and each image file is binarized to detect the wear area from the image, and the amount of wear is measured from the detected wear area. In Patent Document 1, the coating of the worn part is peeled off, exposing the internal metal and shining white, and the white part in the binarized image is detected as the wear area. However, depending on the tool, there are cases where the wear area cannot be detected properly.
[0005] In one aspect, an object of the present invention is to provide a program or the like capable of detecting the deterioration state of a tool with high accuracy. [Means for solving the problem]
[0006] A program relating to one aspect causes a computer to execute a process of acquiring an image of a blade portion provided on a cutting tool, inputting the acquired image into a learning model that has been trained to output information regarding the deterioration state of the blade portion when the image of the blade portion of the cutting tool is input, and acquiring information regarding the deterioration state of the blade portion of the cutting tool in the captured image. Effect of the Invention
[0007] According to one aspect, the deterioration state of a tool can be detected with high accuracy. [Brief description of the drawings]
[0008] [Figure 1] FIG. 2 is a block diagram showing an example of the configuration of a tool management system. [Figure 2A] FIG. 2 is an explanatory diagram showing an example of the appearance of a tool to be photographed. [Figure 2B] FIG. 2 is an explanatory diagram showing an example of the appearance of a tool to be photographed. [Figure 3A] FIG. [Figure 3B] FIG. [Figure 3C] FIG. [Figure 4] FIG. 11 is an explanatory diagram illustrating an example of a record layout of a tool DB. [Figure 5A] FIG. 13 is an explanatory diagram showing a configuration example of an upper surface model. [Figure 5B] FIG. 13 is an explanatory diagram showing a configuration example of a diagonal model. [Figure 5C] FIG. 13 is an explanatory diagram showing a configuration example of a periphery model. [Figure 5D] FIG. 13 is an explanatory diagram showing a configuration example of a rake face model. [Figure 6] 13 is a flowchart showing an example of a degradation state determining process procedure. [Figure 7] 13 is a flowchart showing an example of a degradation state determining process procedure. [Figure 8] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 9] FIG. 2 is an explanatory diagram showing an example of the appearance of the imaging device. [Figure 10A] FIG. 2 is an explanatory diagram showing an example of the configuration of an imaging device. [Figure 10B] FIG. 2 is an explanatory diagram showing an example of the configuration of an imaging device. [Figure 11] FIG. 11 is an explanatory diagram showing an example of a record layout of a shooting condition DB. [Figure 12] 10 is a flowchart showing an example of a photographing process procedure performed by the photographing device. [Figure 13] 11 is a flowchart showing an example of a photographing process procedure by the photographing device of the third embodiment. [Figure 14] FIG. 11 is an explanatory diagram of an adjustment process of a shooting condition. [Figure 15] 13 is a flowchart showing an example of a registration process procedure of an imaging condition. [Figure 16] FIG. 11 is an explanatory diagram of a registration process of shooting conditions. [Figure 17] 13 is a flowchart showing an example of a procedure for estimating a deteriorated area. [Figure 18A] FIG. 11 is an explanatory diagram of a process of estimating the size of a degraded area. [Figure 18B] FIG. 11 is an explanatory diagram of a process of estimating the size of a degraded area. [Figure 19] FIG. 13 is an explanatory diagram illustrating an example of a record layout of a discrimination threshold DB. [Figure 20] 23 is a flowchart showing an example of a procedure for estimating a deteriorated area according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] The program, information processing method, and information processing device of the present disclosure will be specifically described below with reference to the drawings showing the embodiments. In the following embodiments, a process for determining the deterioration state of the blade of an end mill, which is a cutting tool used in cutting processing, will be described as an example. However, the object to be determined is not limited to an end mill, and may be a cutting tool such as a drill, a tap, or a milling cutter, or may be a grinding tool used in grinding processing. Furthermore, in the cutting tool, a replaceable cutting edge tool (throw-away tool) whose cutting edge (tip) is detachable from a body (handle) may be the object to be determined. In addition, when a replaceable cutting edge tool is the object to be determined, the captured image used to determine the deterioration state of the cutting edge may be an image captured with the cutting edge detached from the body, or may be an image captured with the cutting edge attached to the body.
[0010] (Embodiment 1) FIG. 1 is a block diagram showing an example of the configuration of a tool management system. In this embodiment, a tool management system that determines the deterioration state of a cutting blade provided on a cutting tool based on an image of the cutting blade photographed by a photographing device 20 will be described. The tool management system of this embodiment includes a photographing device 20 that photographs the cutting blade of the tool and a server 10, and the photographing device 20 and the server 10 are connected to each other so as to be able to communicate with each other. The photographing device 20 and the server 10 may be configured to directly communicate with each other by wired communication or wireless communication via a cable, or may be configured to communicate with each other via a network. The network here may be the Internet or a public telephone line network, or may be a LAN (Local Area Network) constructed in a facility where the photographing device 20 and the server 10 are installed.
[0011] The server 10 is an information processing device capable of various information processing and transmitting and receiving information, and is composed of a server computer or a personal computer. The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and the like, and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), a TPU (Tensor Processing Unit), or an AI chip (AI semiconductor). The control unit 11 executes the program P stored in the storage unit 12 as appropriate to execute the information processing and control processing to be performed by the server 10. In addition, when the control unit 11 includes multiple processors, the control unit 11 may execute each process by a different processor.
[0012] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores a program P (program product) executed by the control unit 11 and various data. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the program P. The storage unit 12 also stores a learning model that has learned training data by, for example, machine learning. The learning model is expected to be used as a program module constituting artificial intelligence software. The learning model performs a predetermined calculation on an input value and outputs the calculation result, and the storage unit 12 stores data such as coefficients and thresholds of a function that specifies this calculation as the learning model. In this embodiment, the storage unit 12 stores, as the learning model, a top surface model M1, an oblique model M2, a periphery model M3, and a rake face model M4, which will be described later. The storage unit 12 further stores a tool DB 12a, which will be described later. A part of the storage unit 12 may be another storage device connected to the server 10, or another storage device with which the server 10 can communicate.
[0013] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to and from other devices. In this embodiment, the communication unit 13 may be configured to communicate directly with the photographing device 20, or may be configured to communicate via a network. The input unit 14 accepts operation input by a user, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.
[0014] The reading unit 16 reads information stored in a portable storage medium 10a such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, or an SD (Secure Digital) card. The program P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. The program P and various data may be written in the storage unit 12 during the manufacturing stage of the server 10, or may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.
[0015] In this embodiment, the server 10 may be a multi-computer consisting of multiple computers, may be a virtual machine virtually constructed by software in one device, or may be a cloud server. The input unit 14 and the display unit 15 are not essential for the server 10, and the server 10 may be configured to accept operations through a connected computer, or may be configured to output information to be displayed to an external display device. The program P may be executed on a single computer, or may be executed on multiple computers connected to each other via a network.
[0016] The photographing device 20 is an information processing device having a photographing function, and includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a photographing unit 26, a photographing direction control unit 27, etc., and these units are connected via a bus. The control unit 21, the storage unit 22, the communication unit 23, the input unit 24, and the display unit 25 of the photographing device 20 each have the same configuration as the control unit 11, the storage unit 12, the communication unit 13, the input unit 14, and the display unit 15 of the server 10, and therefore a description of the configuration will be omitted.
[0017] The photographing unit 26 has an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and a lens unit. The photographing unit 26 is a so-called industrial camera, and is a high-resolution camera having an imaging element of about 2 million pixels (for example, 1600 pixels x 1200 pixels) in the case of monochrome, and an imaging element of about 12 million pixels (4024 pixels x 3036 pixels x 3 colors) in the case of color. The photographing unit 26 also has a light source unit including a light source such as an LED (Light Emitting Diode) and an illumination lens, and photographs the tool to be photographed by the illumination light irradiated by the light source unit to obtain a photographed image. The light irradiated by the light source unit is, for example, white light, but may be light of a predetermined wavelength such as green light or near-infrared light, and the image photographed by the photographing unit 26 may be a color image or a black-and-white image. The photographing unit 26 photographs according to an instruction from the control unit 21 and obtains, for example, one piece of image data (still image). The image data (photographed image) acquired by the photographing unit 26 is stored in the storage unit 22.
[0018] 2A and 2B are explanatory diagrams showing an example of the appearance of a tool to be photographed, in which FIG. 2A shows a side view of the tool 30 viewed from the lateral direction, and FIG. 2B shows a tip surface of the cutting portion 31 of the tool 30 viewed from the tip side (upper side in FIG. 2A). In this embodiment, an end mill as shown in FIG. 2A and FIG. 2B is the subject of photography by the photography device 20. Although an example of a square end mill is shown in FIG. 2A and FIG. 2B, the subject of photography may be a ball end mill, a radius end mill, a tapered end mill, or the like. For example, the cutting portion 31 and the shank 32 (handle portion) are integrally formed in the tool 30, and a plurality of cutting edges 31a are provided on the cutting portion 31. Although four cutting edges 31a are provided on the tool 30 in FIG. 2A and FIG. 2B, about two to six cutting edges 31a may be provided.
[0019] The photographing direction control unit 27 controls the photographing direction of the photographing unit 26 with respect to the tool to be photographed according to an instruction from the control unit 21. FIG. 3A to FIG. 3C are explanatory diagrams of the photographing direction. In this embodiment, as shown in FIG. 3A to FIG. 3C, the tool 30 with the blade portion 31 facing upward is fixed to a pedestal installed at a predetermined position, and the position of the photographing unit 26 with respect to the blade portion 31 of the tool 30 is changed to photograph the blade portion 31 of the tool 30 from a plurality of photographing directions. In this embodiment, as shown in FIG. 3A, the tip surface of the blade portion 31 is photographed from above to obtain a top surface image, as shown in FIG. 3B, each blade 31a is photographed from an obliquely upward direction to obtain an oblique image, and as shown in FIG. 3C, the side surface of each blade 31a is photographed from the side to obtain an outer periphery image and a rake face image. The top surface image includes a top surface whole image (photographed image of the tip surface of the blade portion 31) photographed in a state including all the blades 31a, and a top surface individual image (photographed image of the tip surface of each blade 31a) photographed individually of the tip surface of each blade 31a. The oblique image is captured from a direction inclined by about 5 degrees to 30 degrees with respect to the tip of the blade 31a to be captured, for example, with the top of the tool 30 as a reference. The angle of the capturing direction can be changed by an operation input via the input unit 24, and is set according to the type of the tool 30 to be captured and the shape of the blade 31a. The outer peripheral image is an image captured by focusing on the flank surface of the blade 31a to be captured (the area surrounded by the dashed line in FIG. 3C), and the rake face image is an image captured by focusing on the rake face of the blade 31a to be captured (the area surrounded by the dashed line in FIG. 3C). The rake face of the blade 31a is the surface that comes into contact with the chips, and the flank surface of the blade 31a is the surface of the tool 30 opposite to the rake face. The outer periphery image and the cutting face image can be captured from the same shooting direction relative to the tool 30, but in the outer periphery image, the blade 31a captured in the center of the image (the blade surrounded by a dashed line in Figure 3C) is the blade being captured, and in the cutting face image, the blade 31a captured on the left side of the image (the blade surrounded by a dotted line in Figure 3C) is the blade being captured.
[0020] The base is configured to rotate around the axial direction of the tool 30 (the vertical direction in Figs. 3A to 3C), and when capturing individual top surface images, oblique images, outer peripheral images, and rake face images, the photographing direction control unit 27 rotates the base by a predetermined angle. This allows each blade 31a of the tool 30 to be moved within the photographing range of the photographing unit 26, and photographing of each blade 31a can be performed. When the subject to be photographed is a four-blade tool 30 as shown in Figs. 3A to 3C, the photographing direction control unit 27 rotates the base by 90 degrees. In addition to changing the photographing direction of the tool 30 (blade 31a) to be photographed by rotating the base or the tool 30 placed on the base, the blade 31a to be photographed may be changed by moving the photographing unit 26 around the tool 30 (blade 31a).
[0021] In the tool management system configured as described above, the photographing direction of the tool to be photographed is registered in advance in the photographing device 20, and when the photographing device 20 starts photographing processing for one tool, the control unit 21 executes photographing by the photographing unit 26 while sequentially switching the photographing direction of the photographing unit 26 to the registered photographing directions by the photographing direction control unit 27. For example, the control unit 21 photographs one whole top surface image from above the tool as shown in Fig. 3A, photographs individual top surface images of each blade while rotating the base, then photographs oblique images of each blade from obliquely above the tool as shown in Fig. 3B while rotating the base, and finally photographs an outer peripheral image and a rake face image of each blade from the side of the tool as shown in Fig. 3C while rotating the base and further adjusting the focus. This makes it possible to obtain multiple photographed images of the blade portion and each blade of the tool from multiple photographing directions registered in advance.
[0022] The control unit 21 can specify information on the blade to be photographed and the photographing direction with respect to the blade for the photographed image obtained by the photographing unit 26, based on the result of the photographing direction control unit 27 controlling the photographing direction. Therefore, the control unit 21 adds information indicating the specified contents to the photographed image. For example, the control unit 21 may give a file name reflecting the specified contents to the photographed image. In this way, the blade to be photographed and the photographing direction for each photographed image can be known by the file name. The control unit 21 may also add the date and time of photographing to the photographed image as the photographing date and time. Note that, for example, the blade to be photographed may be a first blade for a tool fixed to a base, and each blade to be photographed according to the rotation direction of the tool by the base may be a second blade to a fourth blade. In addition, when the tool is fixed to the base in a predetermined direction, the allocation of the first blade to the fourth blade to each blade is always the same. The photographing device 20 transmits the photographed image to which a file name reflecting information on the blade to be photographed and the photographing direction with respect to the blade is given to the server 10 and registers it. The server 10 of this embodiment performs a process of determining the deterioration state of each blade of a tool based on an image of the tool captured by the imaging device 20. The server 10 of this embodiment uses models M1 to M4 when determining the deterioration state of each blade of the tool in the captured image of the tool from the captured image of the tool.
[0023] In the tool management system of this embodiment, the photographing device 20 having the above-mentioned configuration can be equipped with an industrial camera capable of photographing at high resolution, like a general tool presetter, and can photograph the blade of the tool to be photographed by switching between multiple photographing directions.
[0024] FIG. 4 is an explanatory diagram showing an example of a record layout of the tool DB 12a. The tool DB 12a is a database that stores information on tools managed by the tool management system. The tool DB 12a includes, for example, a tool ID column, a tool information column, a photographing date column, a photographing target column, a top surface image column, a diagonal image column, a peripheral image column, a rake face image column, and a comprehensive judgment result column, and stores information on each tool and photographed images photographed by the photographing device 20 in association with a tool ID uniquely assigned to each tool. The tool information column stores information on the tool, including, for example, the type, manufacturer, model number, and usage history of the tool. The photographing date column stores the date and time when the photographing was performed by the photographing device 20. The photographing target column stores information on the blade of the photographing target. In this embodiment, the four-blade tool 30 is the subject of the photographing, so the information on the blade of the photographing target is information indicating the entire tool and the first to fourth blades. The top surface image sequence stores information on top surface images (top surface whole image and top surface individual image) captured by the image capture device 20 in association with the capture date and time and the capture target. Specifically, the top surface image sequence includes an original image sequence, a label image sequence, and a judgment result sequence. The original image sequence stores top surface images (top surface whole image and top surface individual image) that are original images captured by the image capture device 20, and the label image sequence stores label images obtained by a process of judging a deteriorated area for the original image (top surface image). The image data of the original image and the label image may be stored in a predetermined area of the storage unit 12 or an external storage device connected to the server 10 in addition to being stored in the tool DB 12a. In this case, the original image sequence and the label image sequence store information for reading out the image data of the original image and the label image (for example, a folder name and a file name indicating a storage location of the data). Note that the label image sequence may store information indicating the judgment result of a deteriorated area (context that is output data of the models M1 to M4) instead of a label image to which information indicating the judgment result of a deteriorated area is added to the original image. The judgment result column stores the judgment result of judging the presence or absence of a deteriorated area based on the label image. In this embodiment, the deterioration state of the blade is judged as the presence or absence of a worn area and a chipped area, so that "worn", "chipped", or "normal" is stored in the judgment result column.
[0025] The oblique image sequence, the outer periphery image sequence, and the rake face image sequence each include an original image sequence, a label image sequence, and a judgment result sequence, similar to the top surface image sequence, and store information on the oblique image, outer periphery image, and rake face image captured by the imaging device 20 in association with the imaging date and time and the imaging target. Note that the imaging targets of the top surface image are the entire blade portion and the first to fourth blades, so the top surface image sequence stores information on the top surface images (original images, label images, and judgment results) of the entire blade portion and the first to fourth blades. Note that the imaging targets of the oblique image, outer periphery image, and rake face image are the first to fourth blades, so the oblique image sequence, the outer periphery image sequence, and the rake face image sequence store information on the oblique image, outer periphery image, and rake face image of the first to fourth blades. The overall judgment result column stores the judgment result of the deterioration state of each blade of the tool, which is judged overall from the judgment result based on each captured image of each blade of the tool, in association with the shooting date and time and the captured object. The tool DB 12a is not limited to the configuration shown in Fig. 4, and various information related to the tool and various information related to the captured image of the tool may be stored.
[0026] FIG. 5A is an explanatory diagram showing an example of the configuration of the top surface model M1, FIG. 5B is an explanatory diagram showing an example of the configuration of the oblique model M2, FIG. 5C is an explanatory diagram showing an example of the configuration of the outer circumference model M3, and FIG. 5D is an explanatory diagram showing an example of the configuration of the rake face model M4. The top surface model M1 is trained to input a top surface image (top surface whole image and top surface individual image) of a tool, perform a calculation to recognize a deteriorated area of the blade in the top surface image as an object to be detected based on the input top surface image, and output the recognition result. The deteriorated area of the blade, which is an object to be detected, includes a worn area and a chipped (missing) area in each blade, and the top surface model M1 is a model that realizes multi-label classification that recognizes multiple types of objects (worn areas and chipped areas) in the top surface image. The top surface model M1 is configured using, for example, EfficientDet. The upper surface model M1 is not limited to EfficientDet, and can be configured using object detection algorithms such as YOLO (You Only Look Once), R-CNN (Regions with Convolution Neural Network), SSD (Single Shot Multibox Detector), etc., and may be configured by combining multiple algorithms. The upper surface model M1 may also be a model that recognizes objects (worn areas, missing areas) in the input upper surface image on a pixel-by-pixel basis by semantic segmentation. In this case, the upper surface model M1 can be configured using algorithms such as U-Net, FCN (Fully Convolutional Network), SegNet, etc.
[0027] The top surface model M1 has an input layer to which a top surface image is input, an intermediate layer to extract features from the input top surface image, and an output layer to output a context indicating a deteriorated area of the blade detected in the top surface image based on the calculation result of the intermediate layer. The intermediate layer calculates an output value based on the top surface image input from the input layer using various functions and thresholds. The output layer outputs a context including a discrimination label indicating the type of the detected object (worn area or chipped area), coordinate information indicating the area of the detected object (deteriorated area), and a confidence level for the discrimination label. With this configuration, when a top surface image is input, the top surface model M1 outputs a context (information regarding the deteriorated state of the blade) including coordinate information indicating the area of the object, the object name (worn or chipped), and a confidence level for the object (deteriorated area of the blade) detected in the top surface image. Therefore, by using the upper surface model M1, it is possible to obtain an image (label image) in which a bounding box (rectangle shown by a dashed line in FIG. 5A) indicating the area of the object, the name of the object in the recognition result (worn or chipped), and the confidence level for the recognition result are added to the object (deteriorated area of the blade) detected in the upper surface image. In FIG. 5A to FIG. 5D, the label image is shown as output data of the models M1 to M4.
[0028] The top surface model M1 can be generated by machine learning using training data including a training top surface image (plain top surface image) and a correct context including coordinate information indicating the area of an object (degraded area) in the top surface image and an object name (correct label). The training data is generated by associating a context (correct context) generated from the result of an annotation person adding a bounding box and an object name (correct label) to an object (degraded area) in the top surface image with the top surface image.
[0029] The top model M1 learns to output the correct context included in the training data when a top image included in the training data is input. In the learning process, the top model M1 performs calculations in the intermediate layer and the output layer based on the input top image, and calculates the context to be output from the output layer. The top model M1 compares the calculated context with the correct context, and optimizes parameters used in the calculation process in the intermediate layer and the output layer so that the two are close to each other. Specifically, the top model M1 compares the area (deteriorated area), discrimination label (type of deteriorated area), and confidence level of the detected object indicated by the calculated context with the area of the object indicated by the correct context and a value corresponding to the correct label (1 for the correct label, 0 for other labels), and optimizes the two so that the two are close to each other. The parameters are weights (coupling coefficients) between nodes in the intermediate layer and the output layer, and the parameter optimization method can be an error backpropagation method, a steepest descent method, or the like. This results in an upper surface model M1 that, when a top surface image is input, detects degraded areas and the type of degraded area in the top surface image and outputs a context indicating the detection results (coordinate information indicating the degraded area, a discrimination label indicating the type of degraded area, and the confidence level for the discrimination label).
[0030] The top surface model M1 can be further optimized by repeatedly performing the learning process using the training data as described above. In addition, there are multiple types of tools and tool cutting parts, and the top surface model M1 that has been trained using training data of top surface images of one or multiple types of tools can be fine-tuned by the above-mentioned learning process using training data of top surface images of other types of tools to generate the top surface model M1 for each type of tool.
[0031] The oblique model M2, the outer circumference model M3, and the rake face model M4 have the same configuration as the top surface model M1, and are generated by the same learning process. The oblique model M2 is trained to receive an oblique image of a tool, perform a calculation to recognize a deteriorated area of the blade (object to be detected) in the oblique image based on the input oblique image, and output the recognition result. The outer circumference model M3 is trained to receive an outer circumference image of a tool, perform a calculation to recognize a deteriorated area of the blade (object to be detected) in the outer circumference image based on the input outer circumference image, and output the recognition result. The rake face model M4 is trained to receive an image of a tool taken from a cutting surface, perform a calculation to recognize a deteriorated area of the blade (object to be detected) in the cutting surface image based on the input rake face image, and output the recognition result. The oblique model M2, the outer circumference model M3, and the rake face model M4 are also models that realize multi-label classification that recognizes multiple types of objects (worn areas and chipped areas).
[0032] The models M1 to M4 having the above-mentioned configuration may be trained by another training device. The trained models M1 to M4 generated by training on the other training device are downloaded from the training device to the server 10 via a network or a portable storage medium 10a, for example, and stored in the storage unit 12.
[0033] The following describes the process of determining the deterioration state of the cutting edge of a tool based on an image of the tool captured by the photography device 20 in the tool management system of this embodiment. Figs. 6 and 7 are flowcharts showing an example of the deterioration state determination process procedure, and Fig. 8 is an explanatory diagram showing an example screen. The following process is executed by the control unit 11 of the server 10 in accordance with the program P stored in the storage unit 12. The deterioration area estimation process shown in Fig. 7 is the process of step S16 in Fig. 6.
[0034] The control unit 11 of the server 10 acquires images of one tool taken from at least one of the image capturing directions shown in FIG. 3A to FIG. 3C (S11). For example, the control unit 11 acquires from the photographing device 20 one top surface whole image, top surface individual images, oblique images, outer periphery images, and rake face images taken for each blade for one tool. The control unit 11 acquires images taken from the image capturing direction according to the tool to be photographed. The photographing device 20 may start photographing the tool according to an instruction from a user via the input unit 24, may start photographing the tool according to an instruction from the server 10 via the communication unit 23, or may automatically start photographing the tool when the tool is attached to a predetermined position of the photographing device 20. The images photographed by the photographing device 20 may be temporarily stored in another server or storage device, and in this case, the control unit 11 acquires the photographed images from the other server or storage device. The control unit 11 stores the acquired photographed images in the storage unit 12.
[0035] The control unit 11 acquires the photographed date and time of the photographed image together with the photographed image, and stores the acquired photographed date and time in the tool DB 12a in association with the tool ID of the photographed tool (S12). Note that, when the control unit 11 cannot acquire the photographed date and time together with the photographed image, it may store the date and time at this time as the photographed date and time in the tool DB 12a.
[0036] The control unit 11 reads out one of the acquired captured images from the storage unit 12 (S13). The control unit 11 specifies the blade of the captured object and the capturing direction for the read captured image (S14). Here, if the file name of the captured image includes information on the blade of the captured object and the capturing direction for the blade, the control unit 11 specifies the blade of the captured object and the capturing direction in the captured image from the file name. If the file name of the captured image does not include information on the blade of the captured object and the capturing direction for the blade, the control unit 11 may specify the blade of the captured object and the capturing direction for the captured image based on the captured image. For example, if there are four upper surface individual images, the control unit 11 may specify the captured object of each image as the first blade to the fourth blade. The same applies to the oblique image, the outer periphery image, and the rake face image. In this case, the blades of the captured object, which are set as the first blade to the fourth blade, may not be the same in each of the upper surface individual images, the oblique image, the outer periphery image, and the rake face image. When the corresponding relationship of the blades of the photographed objects in each image can be specified based on the shape of the blade portion or each blade, it is possible to match each of the blades of the photographed objects, which are designated as the first blade to the fourth blade, in each image.
[0037] The shooting direction in the captured image may be specified by template matching using a template image prepared in advance, which indicates image features of the captured image of the tool captured from each shooting direction. In addition to template matching, the process of specifying the shooting direction in the captured image may be performed by image recognition using a learning model. For example, when a captured image of a tool is input, a learned learning model can be used to perform a calculation to determine the shooting direction of the tool in the input captured image and output the calculation result. In this case, the control unit 11 inputs the captured image to the learning model, and can specify the shooting direction of the tool in the captured image based on the output information from the learning model. With this configuration, even if the blade of the captured object and the shooting direction of the blade cannot be specified from the file name of the captured image, etc., the blade of the captured object and the shooting direction of the blade can be specified in the captured image.
[0038] The control unit 11 identifies (selects) a model according to the identified photographing direction from among the models M1 to M4 (S15). Here, the control unit 11 identifies the top surface model M1 when the identified photographing direction is the direction of the top surface image, and identifies the oblique model M2 when the identified photographing direction is the direction of the oblique image. The control unit 11 also identifies the outer periphery model M3 when the identified photographing direction is the direction of the rake face image, and identifies the rake face model M4 when the identified photographing direction is the direction of the rake face image. The control unit 11 reads out the identified model from the storage unit 12, and executes an estimation process of a deteriorated area on the blade of the subject photographed in the photographed image based on the read out model and the photographed image read in step S13 (S16).
[0039] In the estimation process of the deteriorated area shown in FIG. 7, TTA (Test Time Augmentation) is used. Specifically, the control unit 11 generates a horizontally inverted image by horizontally inverting (left-right inverting) each pixel of the photographed image (S31). The control unit 11 inputs the photographed image, which is the original image, to the model specified in step S15, and acquires a context (information on the deterioration state of the blade or each blade) output from the model (S32). The context includes coordinate information indicating the position of the deteriorated area detected in the original image (photographed image), a discrimination label indicating the type of the deteriorated area (wear or chipping), and a confidence level for the discrimination label. Similarly, the control unit 11 inputs the generated horizontally inverted image to the model specified in step S15, and acquires a context output from the model (S33).
[0040] The control unit 11 generates a label image that aggregates the estimation results for the two images (original image and horizontally inverted image) based on the context generated from the original image and the context generated from the horizontally inverted image (S34). For example, the control unit 11 generates a label image (hereinafter referred to as a first label image) to which a bounding box surrounding a degraded area detected in the original image, a discrimination label indicating the type of the degraded area (wear or chipping), and a confidence level for the discrimination label are assigned to the original image based on the context generated from the original image. The control unit 11 also generates a label image to which a bounding box surrounding a degraded area detected in the horizontally inverted image, a discrimination label indicating the type of the degraded area, and a confidence level for the discrimination label are assigned to the horizontally inverted image based on the context generated from the horizontally inverted image. Then, the control unit 11 horizontally inverts (left and right inverts) the label image generated from the horizontally inverted image to generate a second label image, and generates a label image that aggregates the first label image and the second label image using WBF (Weighted Boxes Fusion). For example, the control unit 11 calculates the average value of the confidence level for the discrimination label in each of the bounding boxes in the first and second label images. In this case, a confidence level of 0 is assigned to each discrimination label in the areas other than the bounding box areas in the label images, so that a small confidence level is calculated for the bounding boxes (degraded areas) that are detected in only a small number of label images.
[0041] In addition, when the size of a deteriorated area (bounding box) detected in each label image is equal to or larger than a predetermined value, the control unit 11 may exclude the bounding box from the detection result as a false positive. For example, when the horizontal length (number of pixels) of the bounding box is equal to or larger than 1 / 3 of the horizontal length (number of pixels) of the image, or when the vertical length (number of pixels) of the bounding box is equal to or larger than 1 / 3 of the vertical length (number of pixels) of the image, the control unit 11 may exclude the bounding box from the detection result as a false positive. In addition, as shown in FIG. 3C, the rake face of each blade is often photographed in the left half of the photographed image (rake face image). Therefore, when a deteriorated area (bounding box) detected in the rake face image is in the right half of the image, the control unit 11 may exclude the bounding box from the detection result as a false positive. For example, in a coordinate system in which the upper left corner of the scooping face image is the origin, the right direction is the X axis, and the lower direction is the Y axis, the control unit 11 may specify the coordinate value (x, y) of the upper left pixel of the bounding box, and if the specified x coordinate value is to the right (larger) than the x coordinate value of the center of the image, the bounding box may be excluded from the detection result. In addition, for captured images other than the scooping face image, if the area in which the subject is captured is limited to within a predetermined area, the bounding box detected in the captured image may be excluded from the detection result if it is outside the predetermined area. These processes may be configured to be executed appropriately taking into account the shooting conditions such as the shooting direction of the subject.
[0042] The control unit 11 judges whether or not the captured image has a degraded area based on the label image generated in step S34 (S35). For example, the control unit 11 judges whether or not the captured image has a degraded area based on whether or not the generated label image has a bounding box whose confidence level for the discrimination label is equal to or higher than a predetermined threshold. Then, the control unit 11 returns to the process of FIG. 6. In the process shown in FIG. 7, the control unit 11 uses TTA to perform estimation processing on two images, the captured image of the original image and a horizontally inverted image generated from the original screen, and can obtain a highly accurate label image by aggregating the estimation results of each.
[0043] The control unit 11 stores the captured image (original image) read out in step S13, the label image generated in step S34, and the determination result of whether or not there is a deteriorated area in step S35 in the tool DB 12a in association with the capture date and time and the captured object identified in step S14 (S17). The control unit 11 determines whether or not there is an unprocessed captured image that has not been subjected to the above-mentioned estimation process of the deteriorated area among the captured images acquired in step S11 (S18). If it is determined that there is an unprocessed captured image (S18: YES), the control unit 11 returns to step S13 and performs the processes of steps S13 to S17 on the unprocessed captured image. The control unit 11 performs processing of steps S13 to S17 for each of the top surface overall image, top surface individual image, oblique image, outer circumference image, and cutting face image, and stores in the tool DB 12a the original image for each image, the label image generated using models M1 to M4, and the judgment result based on the label image, corresponding to the tool ID, shooting date and time, and shooting subject.
[0044] When it is determined that there are no unprocessed photographed images (S18: NO), the control unit 11 specifies an overall judgment result for each blade to be photographed from the judgment results based on each photographed image stored in the tool DB 12a (S19). For example, the control unit 11 calculates the ratio of photographed images (individual upper surface images, oblique images, outer peripheral images, and rake face images) that are determined to have a deteriorated area (worn area or chipped area) for each blade to be photographed, and if the calculated ratio is equal to or greater than a predetermined value, it determines that the blade to be photographed is in a deteriorated state. Note that the control unit 11 may determine that the blade to be photographed is in a deteriorated state if it determines that at least one image has a deteriorated area.
[0045] The control unit 11 stores the overall judgment result determined for each blade to be photographed in the tool DB 12a, and displays it on the display unit 15 (S20). The control unit 11 displays, for example, a judgment result screen as shown in FIG. 8. The screen shown in FIG. 8 displays a photographed image in which a deteriorated area of the blade to be photographed is detected in association with the tool ID of the tool to be photographed, the photographing date and time, and the information of the blade to be photographed, and indicates the deteriorated area (worn area and chipped area) detected in the photographed image by a rectangle. In the screen of FIG. 8, a rake face image is displayed as the photographed image, but all photographed images in which a deteriorated area of the blade to be photographed is detected may be displayed. In addition, in the screen of FIG. 8, the chipped area is indicated by a dashed rectangle, and the worn area is indicated by a dashed rectangle, and the detected deteriorated area may be displayed in a manner corresponding to the type of deteriorated state (worn or chipped). This makes it easy to understand which deteriorated state the deteriorated area detected in the photographed image is in. 8 displays an enlarged view of the detected deteriorated area, and in the enlarged view, the chipped area and the worn area are indicated by a dashed rectangle and a dashed rectangle, respectively. Note that step S19 does not necessarily have to be executed in the above-mentioned process, and in step S20, the control unit 11 may display a determination result screen showing the deteriorated area (worn area and chipped area) detected in the captured image for the captured image determined to have a deteriorated area based on the determination result stored in the tool DB 12a in step S17.
[0046] According to the above-mentioned process, in this embodiment, the deteriorated area of each blade is detected by image recognition using the models M1 to M4 for the captured image of the blade of the tool captured by the imaging device 20, so that the workload of determining the deteriorated state of each blade can be reduced. Conventionally, the deteriorated state of the blade of the tool is often visually confirmed by a skilled worker, who judges the deteriorated state by looking at the tool from various angles, so that the confirmation work takes time and there may be individual differences in the judgment results of the worn area and the chipped area. In addition, some tools have a blade diameter of about 6 mm, which is difficult to confirm visually. However, in this embodiment, the deteriorated state of the blade to be photographed is automatically detected by image recognition using the models M1 to M4, so that the deterioration state can be accurately determined in a short time. In addition, the tool and each blade that are determined to be in a deteriorated state by image recognition using the models M1 to M4 may be confirmed by a skilled worker, in which case the workload of the skilled worker can be reduced and the deterioration state determination process can be performed by the skilled worker. By accurately determining the deteriorated state of the blade of the tool, the tool can be replaced at an appropriate time, and cutting processing can be performed well.
[0047] In this embodiment, the deterioration state of each blade of the tool is determined based on the captured images (top surface whole image, top surface individual image, oblique image, outer periphery image, and rake face image) obtained by capturing each blade of the tool from a plurality of capturing directions. Therefore, depending on the position of the deteriorated area, it may not be captured in any of the images, but by making a determination based on such a plurality of captured images, it is possible to suppress oversight of the determination of the deteriorated area. In addition, in this embodiment, models M1 to M4 (top surface model M1, oblique model M2, outer periphery model M3, and rake face model M4) are prepared for each capturing direction, so that the deteriorated area in the captured image in each capturing direction can be detected with high accuracy. In addition to the configuration in which the captured images in each capturing direction are trained to generate the models M1 to M4, a configuration in which the captured images in all capturing directions are trained as training data to generate one model may be used. In this case, it is possible to detect the deteriorated area in the image for all capturing directions using one model.
[0048] In this embodiment, the process of detecting a deteriorated area in a captured image by image recognition using the models M1 to M4 is not limited to a configuration in which it is performed by the server 10, and may be executed locally by, for example, the photographing device 20. In this case, the models M1 to M4 may be downloaded to the storage unit 22 of the photographing device 20, and after the photographing device 20 photographs a tool, the photographing device 20 may perform a process of detecting a deteriorated area in the photographed image using the models M1 to M4 for the obtained photographed image. Even with such a configuration, it is possible to execute the same process as in the above-mentioned embodiment, and the same effect can be obtained.
[0049] (Embodiment 2) A specific example of the photographing unit 26 and the photographing direction control unit 27 of the photographing device 20 in the tool management system of the above-mentioned embodiment 1 will be described. Fig. 9 is an explanatory diagram showing an example of the appearance of the photographing device 20, and Figs. 10A and 10B are explanatory diagrams showing an example of the configuration of the photographing device 20. Fig. 9 shows an example of the appearance of the photographing unit 26 and the photographing direction control unit 27, Fig. 10A shows the photographing unit 26 and the photographing unit holding member 27c in a state seen from the left side in the Y-axis direction in Fig. 9, and Fig. 10B shows the photographing unit 26 in a state seen from the right side in the X-axis direction in Fig. 9. The photographing device 20 of this embodiment shown in Figs. 9 to 10B is Tool Collage (registered trademark), which is an imaging unit manufactured by Mori Machinery Corporation.
[0050] The photographing direction control unit 27 has a base 27a that holds the tool 30 to be photographed with the blade portion 31 facing upward on a top plate 27g of the housing of the photographing device 20, and a base holding member 27b that holds the base 27a. The base holding member 27b has a mechanism for moving the base 27a in the Y-axis direction indicated by the arrow B on a horizontal plane (on the top plate 27g) and a mechanism for rotating or turning the base 27a in the θ-axis direction indicated by the arrow E. The θ-axis direction is a rotation direction centered on the central axis direction (vertical direction) of the tool 30 fixed to the base 27a. The photographing direction control unit 27 also has a photographing unit holding member 27c that holds the photographing unit 26, a vertical member 27d that extends vertically from the upper surface of the top plate 27g, and a connecting member 27e that connects the vertical member 27d and the photographing unit holding member 27c. One end of the connecting member 27e is engaged with the vertical member 27d, and the vertical member 27d has a mechanism for moving the connecting member 27e in the engaged state up and down in the Z-axis direction (vertical direction) indicated by the arrow C. The photographing unit holding member 27c is an L-shaped plate member, and the photographing unit 26 is attached to one end thereof, and the other end thereof is connected to the other end of the connecting member 27e. The other end of the connecting member 27e has a rotating member 27f for rotating the photographing unit holding member 27c in the connected state about an axis (rotation axis) parallel to the Y-axis. The connecting member 27e also has a mechanism for expanding and contracting from the engagement position with the vertical member 27d in the X-axis direction indicated by the arrow A, which is perpendicular to the Y-axis direction on the horizontal plane, and this expanding and contracting mechanism allows the rotating member 27f to move away from and toward the engagement position with the vertical member 27d.
[0051] With the above-mentioned configuration, the imaging direction control unit 27 can move the imaging unit holding member 27c connected to the rotating member 27f along the X-axis direction by the extension mechanism of the connecting member 27e, and accordingly, the imaging unit 26 held by the imaging unit holding member 27c can also be moved along the X-axis direction. The imaging direction control unit 27 can also move the connecting member 27e in the Z-axis direction by the vertical member 27d, and accordingly, the imaging unit 26 held by the imaging unit holding member 27c can also be moved along the Z-axis direction. The imaging direction control unit 27 can also move the base 27a in the Y-axis direction by the base holding member 27b, and can rotate or rotate in the θ-axis direction. The imaging direction control unit 27 can also rotate the imaging unit holding member 27c around the rotation axis by the rotating member 27f, as shown in FIG. 10A. Accordingly, the photographing unit 26 held by the photographing unit holding member 27c can be moved (rotated) along the α-axis direction indicated by the arrow D to any position from a position where the photographing direction is parallel to the X-axis direction (X-axis rightward) to a position where the photographing direction is parallel to the Z-axis direction (vertical downward direction). Fig. 10A shows a state where the photographing unit 26 is disposed at a photographing position where the photographing direction is to the right, a photographing position where the photographing direction is diagonally downward to the right, and a photographing position where the photographing direction is directly downward.
[0052] As described above, the photographing direction control unit 27 can move the photographing unit 26 along the X-axis direction, the Z-axis direction, and the α-axis direction, and can move the tool 30 fixed to the pedestal 27a along the Y-axis direction and rotate or turn along the θ-axis direction. Since the positions of the photographing unit 26 and the tool 30 can be moved in this manner, the photographing direction control unit 27 can appropriately change the photographing position and photographing direction of the photographing unit 26 with respect to the photographed object (tool 30).
[0053] 10B, the photographing unit 26 has a camera 26a and eight light source units 26b arranged to surround the periphery of the camera 26a. The light source units 26b are, for example, LEDs, and the eight light source units 26b are configured to be able to control the turning on and off and the amount of light (brightness) separately. The number of light source units 26b is not limited to eight.
[0054] In the photographing device 20 of this embodiment, the camera conditions for the camera 26a and the lighting conditions for the light source unit 26b can be set in advance as photographing conditions when photographing each of the top surface image, the oblique image, the outer peripheral image, and the rake face image for each type of tool 30 to be photographed (e.g., the model number of the tool 30). Once the photographing conditions are set, photographing processing is performed for the same type of tool (tool of the same model number) under the set camera conditions and lighting conditions, and the set images (top surface image, oblique image, outer peripheral image, and rake face image) can be photographed. The camera conditions include information on the photographing position, photographing direction, and focus adjustment of the camera 26a with respect to the tool 30 fixed to the pedestal 27a, and the lighting conditions include the light amount of each light source unit 26b.
[0055] 11 is an explanatory diagram showing an example of a record layout of the photographing condition DB. The photographing condition DB is a database that stores the camera conditions and lighting conditions when the tools managed by the tool management system are photographed by the photographing device 20, and is stored in, for example, the storage unit 22. The photographing condition DB includes, for example, a model number column, a manufacturer name column, a type column, a top surface image column, a diagonal image column, a peripheral image column, and a rake face image column, and stores information on each tool and the camera conditions and lighting conditions when photographed by the photographing device 20 in association with the model number uniquely assigned to each tool. The top surface image column stores the photographing conditions when the top surface images (top surface overall image and top surface individual image) are photographed by the photographing device 20. Specifically, the top surface image column includes a photographing subject column, a camera condition column, and a lighting condition column, and stores the camera conditions and lighting conditions when photographing each photographing subject. The camera conditions include the position and shooting direction of the camera 26a with respect to the blade 31 of the tool 30 fixed to the pedestal 27a, and specifically, the positions of the X-axis direction, Z-axis direction, and α-axis direction of the shooting unit 26 (camera 26a) with respect to a preset initial position as a reference, and the positions of the Y-axis direction and θ-axis direction with respect to the pedestal 27a with respect to a preset initial position as a reference. The camera conditions also include information on focus adjustment of the camera 26a. The lighting conditions include the amount of light (including on and off) for each of the eight light source units 26b. Each of the oblique image sequence, the outer periphery image sequence, and the rake face image sequence includes a shooting target sequence, a camera condition sequence, and a lighting condition sequence, similar to the top surface image sequence, and stores the camera conditions and lighting conditions when shooting each shooting target. The shooting condition DB is not limited to the configuration shown in FIG. 11, and various information related to the tool and various information related to shooting the tool may be stored.
[0056] Each piece of information registered in the photographing condition DB is stored according to, for example, an operation input via the input unit 24. Specifically, when registering photographing conditions (camera conditions and lighting conditions) for an unregistered tool 30, the person in charge of registration controls the photographing direction control unit 27 and the photographing unit 26 by operating via the input unit 24, photographs the tool 30 fixed to the pedestal 27a by the photographing unit 26, checks the photographed image, and stores the conditions under which the tool 30 can be photographed in an appropriate photographing state in the photographing condition DB. The appropriate photographing state means an image quality that suppresses halation occurring in the photographed image and allows the deterioration state of the blade part 31 in the photographed image (specifically, the presence or absence of a worn area and a chipped area) to be appropriately determined using the models M1 to M4. By registering the photographing conditions when photographing each tool 30 in such a photographing condition DB, when photographing a tool 30 of a registered model number, the person in charge of photographing can photograph various images simply by fixing the tool 30 to the pedestal 27a.
[0057] FIG. 12 is a flowchart showing an example of the photographing process procedure by the photographing device 20. The control unit 21 of the photographing device 20 executes the following process according to a program stored in the storage unit 22. After the tool 30 to be photographed is fixed to the pedestal 27a, the photographing device 20 executes the following process when, for example, an instruction to start the photographing process is received via the input unit 24. The control unit 21 of the photographing device 20 first identifies the model number of the tool 30 to be photographed (S41). The model number of the tool 30 may be received, for example, by input by a person in charge of photographing via the input unit 24, may be identified by template matching using a template image showing the image feature amount of the photographed image of each tool, or may be identified by image recognition using a learning model. When identifying by image recognition, for example, when a photographed image of a tool is input, a learned learning model can be used to perform a calculation to distinguish the model number of the tool in the input photographed image and output the calculation result. In this case, the control unit 11 inputs the photographed image into the learning model, and can identify the model number of the tool in the photographed image based on the output information from the learning model. Also, a code (e.g., a barcode or QR code (registered trademark)) encoding the model number of the tool 30 may be printed or affixed to the tool 30 or a holder that holds the tool 30, and the model number of the tool 30 being photographed may be obtained by reading the code using a code reading unit. Note that the information on the tool identified in step S41 is not limited to the model number as long as it is information that can identify the shape, size, etc. of the tool.
[0058] The control unit 21 of the photographing device 20 reads out the camera conditions and lighting conditions corresponding to the identified model number from the photographing condition DB (S42). Specifically, the control unit 21 reads out the camera conditions and lighting conditions of each image (top image, oblique image, outer periphery image, and rake face image) registered in the photographing condition DB in association with the model number. The control unit 21 judges whether the photographing conditions corresponding to the model number have been registered or not based on whether the camera conditions and lighting conditions have been read out (S43). Specifically, if the control unit 21 has been able to read out the camera conditions and lighting conditions, it judges that the photographing conditions corresponding to the model number have been registered, and if it has not been able to read out the camera conditions and lighting conditions, it judges that the photographing conditions corresponding to the model number have not been registered. If it is judged that the photographing conditions corresponding to the model number have not been registered (S43: NO), the control unit 21 executes a registration process of the photographing conditions corresponding to the model number identified in step S41 (S44). For example, the control unit 21 displays a message on the display unit 25 notifying that the photographing conditions for the tool to be photographed have not been registered, and prompts the execution of the registration process of the photographing conditions. Then, the control unit 21 controls the operation of the photographing direction control unit 27 and the photographing unit 26 in accordance with the operation via the input unit 24 to photograph the tool to be photographed while changing the camera conditions and the lighting conditions, and when the photographing conditions under which the tool can be photographed in an appropriate photographing condition are specified, the control unit 21 registers the specified photographing conditions in the photographing condition DB. When it is determined that the photographing conditions corresponding to the model number have been registered (S43: YES), the control unit 21 skips step S44.
[0059] The control unit 21 extracts one of the photographing conditions read out in step S42 or the photographing conditions registered in the photographing condition DB in step S44 (S45). For example, the control unit 21 extracts the camera conditions and the lighting conditions corresponding to one photographing target in one photographed image (top image, oblique image, outer peripheral image, or scooping face image). The control unit 21 executes the photographing process under the extracted camera conditions and lighting conditions (S46). The control unit 21 adds the photographing date and time (e.g., the date and time at this point in time) and the photographed image information to the photographed photographed image and stores it in the storage unit 22 (S47). For example, the control unit 21 stores the photographing date and time and the photographed image information as tag information (additional information) of the photographed image. The photographed image information includes, for example, the blade information of the photographed target and the photographing direction information.
[0060] The control unit 21 judges whether or not the photographing has been completed under all the photographing conditions included in the photographing conditions read out in step S42 or the photographing conditions registered in the photographing condition DB in step S44 (S48). If it is judged that the photographing has not been completed (S48: NO), the control unit 21 returns to step S45 and repeats the processing of steps S45 to S47 for the photographing conditions for which the photographing has not been completed. As a result, the control unit 21 can execute the photographing processing under the camera conditions and lighting conditions registered in association with the model number of the tool 30, and can obtain the photographed image from the photographing direction corresponding to each tool. If it is judged that the photographing has been completed under all the photographing conditions (S48: YES), the control unit 21 generates a folder for the tool 30 to be photographed in the storage unit 22, and stores each photographed image stored in the storage unit 22 in step S47 for the tool 30 to be photographed in the generated folder (S49). As a result, the photographed image can be stored in a folder for each tool 30 to be photographed, so that the photographed images in the folder can be treated as photographed images of the same tool 30. It should be noted that the control unit 21 does not need to perform step S49.
[0061] According to the above-mentioned process, in this embodiment, the positional relationship between the tool 30 to be photographed fixed to the pedestal 27a and the photographing unit 26 (camera 26a) can be controlled in the X-axis, Y-axis, Z-axis, α-axis, and θ-axis directions, so that each blade to be photographed can be photographed from an optimal photographing position. In addition, in this embodiment, the focus of the camera 26a can be adjusted and the light amount of each of the eight light source units 26b can be controlled separately, so that highly accurate photographed images (top surface image, oblique image, outer periphery image, rake face image) can be obtained under optimal camera conditions and lighting conditions for each blade. By detecting the deteriorated area of each blade by image recognition using the models M1 to M4 for such photographed images, it becomes possible to more accurately determine the deterioration state of each blade.
[0062] In this embodiment, the same effects as those of the above-mentioned embodiment 1 can be obtained. In addition, in this embodiment, since a highly accurate photographed image can be obtained using the photographing device 20, the deterioration state of each blade of the photographed subject can be determined more accurately based on the photographed image. In this embodiment, the modified examples described as appropriate in the above-mentioned embodiment 1 can also be applied.
[0063] (Embodiment 3) In the photographing device 20 of the first and second embodiments described above, the shank 32 of the tool 30 is held by a holder (not shown) and fixed to the base 27a, photographing is performed, and the tool 30 is stored and managed. Meanwhile, the tool 30 is removed from the holder and rearranged to attach the same type of tool 30. The mounting position of the tool 30 relative to the holder may shift before and after rearrangement, and when the mounting position of the tool 30 shifts, even if the tool 30 is the same type, a situation occurs in which the tool is not photographed in the same state when photographed under the same photographing conditions. Therefore, in this embodiment, when photographing the tool 30, if the photographing conditions for the tool 30 have been registered, it is determined whether it is possible to photograph the tool in the same state as when the photographing conditions were registered when photographing the tool under the registered photographing conditions, and if not possible, the photographing conditions are updated (reregistered). The tool management system of this embodiment can be realized by the same devices as those in the tool management systems of the first and second embodiments, so the configuration of each device will not be described. In this embodiment, for example, an image of each tool 30 photographed under stored photographing conditions (camera conditions and lighting conditions) is stored as a reference image in a photographing condition DB (see FIG. 11) stored in the storage unit 22 of the photographing device 20. The reference image may be photographed and stored for each photographing direction, or only reference images photographed in some photographing directions such as an outer periphery image or a rake face image may be stored.
[0064] Fig. 13 is a flowchart showing an example of the photographing process procedure by the photographing device 20 of the third embodiment, and Fig. 14 is an explanatory diagram of the photographing condition adjustment process. The process shown in Fig. 13 is obtained by adding steps S51 to S56 between YES in step S43 and step S45 in the process shown in Fig. 12. Explanations of the same steps as in Fig. 12 will be omitted.
[0065] When the control unit 21 of the photographing device 20 of this embodiment determines that the photographing conditions corresponding to the model number of the tool 30 to be photographed have been registered (S43: YES), it acquires a reference image of the outer periphery image of this tool 30 (the tool 30 of the same model number) from the photographing condition DB (S51). An example of the reference image of the outer periphery image is shown on the left side of FIG. 14. Then, the control unit 21 specifies the photographing conditions of the outer periphery image among the photographing conditions read out in step S42, and photographs the tool 30 to be photographed under the specified photographing conditions to acquire the outer periphery image of the tool 30 to be photographed (S52). An example of the photographed outer periphery image is shown on the right side of FIG. 14. From the two outer periphery images shown in FIG. 14, it can be seen that the mounting position of the tool 30 to be photographed relative to the holder is shifted compared to when the photographing conditions were registered (when the reference image was photographed). Specifically, the tool 30 in the photographed image is shifted downward in the Z-axis direction (see FIG. 9) and shifted leftward in the θ-axis direction (see FIG. 9) with respect to the tool in the reference image.
[0066] Therefore, the photographing device 20 of this embodiment automatically registers appropriate photographing conditions for the tool 30 held in the holder at this time by adjusting the photographing conditions when there is a deviation between the position of the tool in the reference image and the position of the tool 30 in the photographed image. First, the control unit 21 specifies the position of the blade 31a of the tool 30 to be photographed in the reference image acquired in step S51 and the outer periphery image (photographed image) acquired in step S52 (S53). Specifically, the control unit 21 performs binarization processing on each of the reference image and the outer periphery image, and performs contour detection of an area in which pixels with a pixel value of 1 are continuous in the binary image, for example, to specify the edge position of the cutting edge of the blade 31a photographed in the center of each image. In the reference image of FIG. 14, the edge position E1 is specified, and in the outer periphery image, the edge position E2 is specified. In addition, the control unit 21 specifies the top end surface of the tool in each image based on the reference image and the outer periphery image (photographed image). In FIG. 14, the top end surface indicated by the dashed line is specified. The edge position E1 and the position of the upper end surface of the tool in the reference image may be detected when the photographing conditions are registered and stored in the photographing condition DB.
[0067] The control unit 21 judges whether the detected edge positions E1, E2 and the top end face position of the tool match in the reference image and the outer periphery image (photographed image) (S54). Specifically, the control unit 21 judges whether the deviation between the edge position E1 in the reference image and the edge position E2 in the outer periphery image is less than a predetermined amount, and whether the deviation between the top end face of the tool in the reference image and the top end face of the tool in the outer periphery image is less than a predetermined amount. Then, the control unit 21 judges that they match when both are less than the predetermined amount, and judges that they do not match when at least one is equal to or greater than the predetermined amount. If it is judged that they match (S54: YES), there is no deviation between the position of the tool in the reference image and the position of the tool 30 in the photographed image, so that photographing is possible under the registered photographing conditions, and the control unit 21 proceeds to step S45. Then, the control unit 21 extracts one photographing condition from the photographing conditions read out in step S42 (S45), and executes photographing processing under the extracted photographing condition (camera condition and lighting condition) (S46).
[0068] When it is determined that the edge positions E1, E2 or the upper end face position of the tool do not match (S54: NO), there is a deviation between the position of the tool in the reference image and the position of the tool 30 in the captured image, so that adjustment (update) is required for the registered capturing conditions. Therefore, the control unit 21 calculates an adjustment amount for the capturing conditions read out in step S42 (S55). Here, the control unit 21 calculates an adjustment amount such that the position of the blade in the captured image approaches the position of the blade in the reference image. Specifically, the control unit 21 calculates a rotation amount (adjustment amount) for rotating the base 27a holding the tool 30 in the θ-axis direction based on the edge position E1 in the reference image and the edge position E2 in the captured image. The adjustment amount in the θ-axis direction is a rotation amount for matching (approaching) the edge position E2 to the edge position E1 in the captured image of FIG. 14, and can be specified by, for example, registering a rotation amount corresponding to a distance in the captured image in advance. The adjustment amount in the θ-axis direction may be determined by, for example, rotating pedestal 27a by a predetermined rotation amount in the θ-axis direction, repeatedly photographing tool 30 and comparing it with edge position E1 in the reference image, and summing the rotation amounts until edge position E2 in the photographed image matches edge position E1 in the reference image. Also, control unit 21 calculates an adjustment amount for lowering photographing unit 26 a predetermined amount downward in the Z-axis direction based on the top end surface position in the reference image and the top end surface position in the photographed image. The adjustment amount here is a movement amount for matching (bringing closer) the top end surface position of the photographed image to the top end surface position of the reference image in the image in FIG. 14.
[0069] The control unit 21 executes an update process of the photographing conditions stored in the photographing condition DB corresponding to the model number specified in step S41 based on the calculated adjustment amount for the photographing conditions (S56). For example, when the adjustment amount of the photographing conditions of the outer periphery image is calculated as the adjustment amount of rotating the base 27a in the θ-axis direction and the adjustment amount of lowering the photographing unit 26 downward in the Z-axis direction as described above, the control unit 21 updates (changes) the camera conditions of the outer periphery image stored in the photographing condition DB based on the calculated adjustment amount. Note that the adjustment amount of lowering the photographing unit 26 downward in the Z-axis direction affects not only the photographing conditions of the outer periphery image but also the photographing conditions of other images, so the camera conditions of the other images (the position of the photographing unit 26 in the Z-axis direction) may also be updated based on the calculated adjustment amount. As a result, even if the mounting position of the tool 30 relative to the holder is shifted from the mounting position at the time of registering the photographing conditions, the photographing conditions stored in the photographing condition DB can be updated to photographing conditions that allow the tool 30 to be photographed in the photographing state of the reference image.
[0070] Thereafter, the control unit 21 proceeds to step S45, extracts one of the updated photographing conditions (S45), and executes photographing processing under the extracted photographing condition (camera condition and lighting condition) (S46). By the above-mentioned processing, even if the mounting position of the tool 30 relative to the holder is shifted from the mounting position at the time of registering the photographing condition, a photographed image of the tool 30 to be photographed in the same state as the photographing state of the reference image can be obtained by updating the photographing condition. In this embodiment, based on the reference image and the photographed image of the outer periphery image, it is determined whether the mounting position of the tool 30 is shifted, and the adjustment amount of the photographing condition is calculated. In addition, based on the reference image and the photographed image of each image (top image, oblique image, outer periphery image, rake face image), it may be determined whether the mounting position of the tool 30 is shifted, and the adjustment amount of the photographing condition is calculated. In the above-mentioned processing, if the control unit 21 determines that the shooting conditions have been registered (S43: YES), it may be configured to determine whether the tool 30 has been replaced in the holder, and if a replacement has been made, to perform the processing of steps S51 to S56.
[0071] In this embodiment, the process of calculating the adjustment amount for the registered photographing conditions based on the reference image and the photographed image of the outer peripheral image is not limited to a configuration in which the photographing device 20 performs it locally, and may be a configuration in which the process is performed by, for example, the server 10. In this case, the photographing device 20 can be configured to transmit a photographed image photographed under the registered photographing conditions, for example, a photographed image of the outer peripheral image, to the server 10, and receive the result determined by the server 10 (information indicating whether or not there is a deviation between the position of the tool in the reference image and the position of the tool 30 in the photographed image) and the adjustment amount calculated by the server 10 (adjustment amount for the registered photographing conditions). In such a configuration, the server 10 identifies the position of the blade in the photographed image (edge position E2 of the blade tip) based on the photographed image (outer peripheral image) photographed by the photographing device 20. Then, the server 10 determines whether or not the edge position E2 of the blade tip matches a position registered in advance (edge position E1 in the reference image), and if it is determined that they do not match, calculates an adjustment amount for matching the edge position E2 in the photographed image with the edge position E1 in the reference image. The server 10 instructs the photographing device 20 to adjust the photographing conditions set in the photographing device 20 based on the calculated adjustment amount. The photographing device 20 updates (adjusts) the registered photographing conditions in accordance with the instruction from the server 10, thereby registering in the photographing condition DB the photographing conditions under which the tool 30 to be photographed can be photographed in the photographing state of the reference image.
[0072] The configuration of this embodiment is applicable to the tool management systems of the above-mentioned embodiments 1 and 2, and similar processes can be executed and similar effects can be obtained even when applied to the tool management systems of the above-mentioned embodiments 1 and 2. Furthermore, the modified examples described in the above-mentioned embodiments 1 and 2 as appropriate can be applied to this embodiment as well.
[0073] (Embodiment 4) In the tool management systems of the above-mentioned embodiments 1 to 3, a tool management system is described which automatically determines whether an image is in focus when photographed under each photographing condition and registers the photographing condition under which the image is in focus when photographing conditions for each model number of tool 30 are registered. The tool management system of this embodiment can be realized by the same devices as those of the tool management systems of embodiments 1 to 3, and therefore the configuration of each device will not be described.
[0074] FIG. 15 is a flow chart showing an example of the registration process of the photographing conditions, and FIG. 16 is an explanatory diagram of the registration process of the photographing conditions. The process shown in FIG. 15 is the process of step S44 in FIG. 12. In this embodiment, the photographing device 20 photographs each image (top image, oblique image, outer peripheral image, and scooping face image) under the photographing conditions set as defaults, judges whether the obtained photographed image is in focus or not, and if it judges that it is not in focus, adjusts the photographing conditions and photographs again, repeating this process until it is in focus. Then, the photographing device 20 registers the photographing conditions for the photographed image that is judged to be in focus as the photographing conditions for that image (photographing direction). The judgment of whether it is in focus is performed for a focus area set in advance for each image. This makes it possible to efficiently judge whether each image is in focus or not.
[0075] In the registration process of the photographing conditions, the control unit 21 of the photographing device 20 first photographs, for example, an entire top surface image (S61). The control unit 21 extracts a focus area in the entire top surface image (S62). The focus area of the entire top surface image is the entire image as shown by the dashed line in FIG. 16. Then, the control unit 21 judges whether or not the focus is correct in the extracted focus area (S63). Here, the control unit 21 performs a filter process using a Laplacian filter on the focus area to extract edges in the image, calculates a variance based on each pixel value in the image (focus area) after the filter process, and judges whether or not the focus is correct based on the calculated variance value. The variance can be calculated, for example, as the average value of the squares of the deviations (the difference between each pixel value and its average value). Furthermore, the control unit 21 judges that the focus is not correct when the variance value is equal to or greater than a predetermined value, and judges that the focus is correct when the variance value is less than the predetermined value. Note that the filter process is not limited to the Laplacian filter, and various differential filters for extracting contours (edges) from an image may be used.
[0076] When the control unit 21 determines that the image is not in focus (S63: NO), it adjusts (changes) the photographing conditions used in step S61 (S64). For example, the control unit 21 adjusts the photographing unit 26 by moving it a predetermined amount upward in the Z-axis direction, adjusts the focus of the camera 26a, and the like. This allows the photographing unit 26 to be moved a predetermined distance upward in the Z-axis direction relative to the tool 30 to be photographed, or allows the focus adjustment (adjustment of the focal point) of the camera 26a to be performed. Then, the control unit 21 returns to step S61, photographs an entire top surface image under the adjusted photographing conditions (S61), and repeats the processes of steps S61 to S64 until it is determined that the image is in focus. When the control unit 21 determines that the image is in focus (S63: YES), it registers the photographing conditions used in photographing the most recent entire top surface image in the photographing condition DB (S65).
[0077] Next, the control unit 21 executes the same processing as steps S61 to S65 for the oblique image (S66 to S70). The focus area of the oblique image is the center of the image in the left-right direction as shown by the dashed line in FIG. 16. The adjustment for focusing on the oblique image can be, for example, an adjustment to move the photographing unit 26 away from the tool 30 in the X-axis direction by a predetermined amount, or an adjustment to the focus of the camera 26a. As a result, the photographing conditions for focusing on the focus area for the oblique image are also registered in the photographing condition DB. Next, the control unit 21 executes the same processing as steps S61 to S65 for the outer periphery image (S71 to S75). The focus area of the outer periphery image is the center of the image in the left-right direction as shown by the dashed line in FIG. 16. Again, the adjustment for focusing can be, for example, an adjustment to move the photographing unit 26 away from the tool 30 in the X-axis direction by a predetermined amount, or an adjustment to the focus of the camera 26a. As a result, the photographing conditions for focusing on the focus area for the outer periphery image are also registered in the photographing condition DB. Finally, the control unit 21 executes the same processing as steps S61 to S65 for the rake face image (S76 to S80). In the case of shooting conditions where the tool 30 is shot in the center of the image as shown in the upper part of FIG. 16, the focus area of the rake face image can be the area on the left side of the image as shown by the dashed line. In addition, in the case of shooting conditions where the rake face of the blade 31a is shot in the center of the image as shown in the lower part of FIG. 16, such as a tool with a large diameter of the blade, the focus area may be the center in the left-right direction of the image as shown by the dashed line. Thereby, the shooting conditions where the focus is on the focus area for the rake face image are also registered in the shooting condition DB. The control unit 21 may also perform similar processing for the individual upper surface images.
[0078] After the process of step S80, the control unit 21 returns to the process of Fig. 12 and photographs the tool 30 to be photographed from multiple photographing directions according to the photographing conditions registered by the above-mentioned process. By the above-mentioned process, the photographing conditions in which the focus area is focused on for each image are registered. The focus area is the area in which the blade portion 31 (blade 31a) is photographed, and by photographing under photographing conditions in which this area is focused on, it is possible to obtain a high-quality photographed image that can appropriately detect the deteriorated area of each blade by image recognition using the learning models M1 to M4, and it becomes possible to more accurately determine the deterioration state of each blade.
[0079] In this embodiment, the registration process of the photographing conditions of each image is not limited to a configuration in which the photographing device 20 performs it locally, and may be a configuration in which it is performed by, for example, the server 10. In this case, the photographing device 20 can be configured to transmit each photographed image to the server 10 and receive the result determined by the server 10 (the photographing conditions determined to be in focus). In this configuration, the server 10 determines whether or not a predetermined area (focus area) of each photographed image is in focus based on each photographed image photographed by the photographing device 20. If the server 10 determines that the image is not in focus, it calculates an adjustment amount for focusing on the predetermined area and instructs the photographing device 20 to adjust the photographing conditions set in the photographing device 20 based on the calculated adjustment amount. The photographing device 20 updates (adjusts) the registered photographing conditions according to the instructions from the server 10, thereby making the photographing conditions closer to the photographing conditions in which the focus area is in focus. The photographing device 20 and the server 10 repeat the above-described process until the focus area of each image is brought into focus, thereby registering in the photographing condition DB the photographing conditions under which the focus area of each image is brought into focus.
[0080] The configuration of this embodiment is applicable to the tool management systems of the above-mentioned embodiments 1 to 3, and similar processes can be executed and similar effects can be obtained even when applied to the tool management systems of the above-mentioned embodiments 1 to 3. Furthermore, the modified examples described in the above-mentioned embodiments 1 to 3 can also be applied to this embodiment as appropriate.
[0081] (Embodiment 5) In the tool management systems of the above-mentioned embodiments 1 to 4, when it is determined that a deteriorated area is present in the blade (blade) in the captured image by the deteriorated area estimation process, a tool management system that estimates the size of the deteriorated area will be described. The tool management system of this embodiment can be realized by the same devices as those of the tool management systems of the embodiments 1 to 4, so the configuration of each device will not be described.
[0082] Fig. 17 is a flowchart showing an example of a procedure for estimating a degraded region, and Fig. 18A and Fig. 18B are explanatory diagrams of a process for estimating the size of a degraded region. The process shown in Fig. 17 is obtained by adding steps S91 and S92 after step S35 in the process shown in Fig. 7. Explanations of the same steps as in Fig. 7 will be omitted.
[0083] In the process of estimating the deteriorated area, the control unit 11 of the server 10 of this embodiment performs the same process as steps S31 to S35 in FIG. 7. This allows the control unit 11 to determine whether or not the blade in the captured image read out in step S13 in FIG. 6 has a deteriorated area. Thereafter, the control unit 11 determines whether or not the deteriorated area exists based on the determination result of step S35 (S91), and if it is determined that the deteriorated area exists (S91: YES), calculates the size of the deteriorated area (S92). For example, as shown in FIG. 18A, the control unit 11 acquires a label image in which a bounding box surrounding the deteriorated area detected in the captured image of the outer peripheral image, a discrimination label indicating the type of the deteriorated area (wear or chipping), and a confidence level for the discrimination label are assigned. When such a label image is acquired, the control unit 11 extracts the area of the bounding box from the captured image and performs a binarization process on the extracted area (see FIG. 18A(1)). The control unit 11 adds one diagonal line L1 (a diagonal line rising to the right) in the bounding box to the binary image, and identifies the midpoint P1 of the diagonal line L1 (see FIG. 18A(2)). Note that the tool 30 shown in FIG. 18A has a blade edge that forms a line segment rising to the right, so the diagonal line L1 rising to the right is added here, but a diagonal line falling to the right may be added depending on the state of the blade of the tool 30. Next, the control unit 11 identifies a perpendicular line L2 that passes through the midpoint P1 and is perpendicular to (intersects with) the diagonal line L1 (see FIG. 18A(3)), and identifies two intersection points P2 and P3 between the perpendicular line L2 and the contour line of the deteriorated area (see FIG. 18A(4)). The control unit 11 then calculates the distance between the intersection points P2 and P3, and sets the calculated distance as the size (width) of the deteriorated area.
[0084] The bounding box is often added with the detected deteriorated area located at the center, and the midpoint P1 of the diagonal line L1 of the bounding box is likely to be near the center of the deteriorated area. Therefore, the size of the deteriorated area can be calculated with high accuracy by calculating the size of the deteriorated area based on the perpendicular line L2 passing through the midpoint P1 of the diagonal line L1 of the bounding box. The size of the deteriorated area can also be calculated by similar processing for the oblique image and the rake face image. Note that, if the edge position of the blade 31a can be detected with high accuracy, the size of the deteriorated area in the direction perpendicular (intersecting) to the edge position may be calculated.
[0085] In addition, the control unit 11 acquires a bounding box surrounding the deteriorated area, a discrimination label indicating the type of the deteriorated area (wear or chipping), and a label image with a degree of confidence for the discrimination label for the captured image of the top surface image (top surface overall image and top surface individual image) as shown in FIG. 18B. When such a label image is acquired, the control unit 11 extracts the bounding box area from the captured image and performs a binarization process on the extracted area (see FIG. 18B(1)). The control unit 11 estimates the shape of the blade before deterioration (original shape) based on the binary image. For example, assuming that the shape of the blade before deterioration is a rectangle, the control unit 11 estimates a rectangle R1 including the deteriorated blade area (white pixel area) as the shape of the blade before deterioration as shown in FIG. 18B(2), and adds the estimated rectangle R1 to the binary image. Next, the control unit 11 identifies the area on the tip side of the blade (the upper gray area in FIG. 18B(3)), which is the difference area between the blade area (area of white pixels) in the binary image and the area of the rectangle R1. The tip side of the blade is the upper side in the bounding box in FIG. 18B, but by setting the tip side of the blade in advance in each of the upper surface whole image and the upper surface individual image, the difference area on the tip side of the blade can be identified. The difference area on the tip side of the blade indicates the deteriorated area of the blade. Therefore, the control unit 11 calculates the size (area) of the identified difference area, and sets the calculated size as the size of the deteriorated area. Note that the control unit 11 may count the number of pixels in the deteriorated area as the size of the deteriorated area, or may calculate the length in the left-right direction of the image (the circumferential direction of the blade portion 31 in the tool 30) and the length in the up-down direction of the image (the radial direction of the blade portion 31) as the size of the deteriorated area represented by a right-angled triangle, as shown in FIG. 18B(4). In the top view image, each blade is captured in a substantially rectangular shape, so by comparing the area of the blade 31a in the image with the area of the rectangle R1, the size of the deteriorated area of the blade 31a can be estimated with high accuracy.
[0086] When the control unit 11 determines that there is no deteriorated area (S91: NO), it skips step S92 and returns to the process of FIG. 6. In the process of FIG. 6, in step S17, the control unit 11 stores the photographed image (original image) read out in step S13, the label image generated in step S34, the determination result of determining the presence or absence of a deteriorated area in step S35, and the size of the deteriorated area calculated in step S92 when it is determined that there is a deteriorated area, in the tool DB 12a, in association with the photographing date and time and the photographed object specified in step S14. In addition, in step S20, the control unit 11 may display the size of each deteriorated area in association with the deteriorated area (worn area and chipped area) detected in the photographed image on the determination result screen shown in FIG. 8. In this configuration, it is possible to display, on the determination result screen, which part of the tool 30 is estimated to be a deteriorated area, the confidence level of the estimation of the deteriorated area, and the size of the deteriorated area all together.
[0087] In this embodiment, when it is estimated that the tool 30 has a deteriorated area based on the captured image of the tool 30, the size of the estimated deteriorated area can be automatically detected. Therefore, not only can the size of the automatically detected deteriorated area be notified, but also the size of the deteriorated area to be notified can be switched depending on the size of the deteriorated area. That is, a threshold value for whether or not to output an alert can be set for the size (deterioration width) of the deteriorated area. For example, a tool 30 in which a deteriorated area with a size (width) of 0.5 mm or more is detected can be set as an alert target, and in this case, a tool 30 having a deteriorated area with a size of less than 0.5 mm is not set as an alert target, thereby suppressing unnecessary alert output. The configuration of this embodiment is applicable to the tool management systems of the above-mentioned embodiments 1 to 4, and even when applied to the tool management systems of the above-mentioned embodiments 1 to 4, similar processing can be executed and similar effects can be obtained. In addition, the modified examples described appropriately in the above-mentioned embodiments 1 to 4 can be applied to this embodiment as well.
[0088] (Embodiment 6) A tool management system is described in which the threshold for the confidence level for a discrimination label, which is used when determining whether or not a deteriorated area exists on a tool being photographed based on a label image, can be switched according to the tool (tool model number). The tool management system of this embodiment can be realized by devices similar to those of the tool management systems of embodiments 1 to 5, so the configuration of each device will not be described. In this embodiment, in addition to the configuration shown in Fig. 1, the server 10 stores a discrimination threshold DB in the memory unit 12.
[0089] Fig. 19 is an explanatory diagram showing an example of a record layout of the discrimination threshold DB. The discrimination threshold DB is a database that stores thresholds used for determining the presence or absence of a deteriorated area using models M1 to M4, in association with each tool managed by the tool management system. The discrimination threshold DB includes, for example, a model number column, a manufacturer name column, a type column, and a discrimination threshold column, and stores information on each tool and a discrimination threshold for each tool in association with the model number of each tool. The discrimination threshold may be a preset numerical value, or may be an arbitrarily changed numerical value. The discrimination threshold DB is not limited to the configuration shown in Fig. 19.
[0090] Fig. 20 is a flowchart showing an example of a procedure for estimating a deteriorated area according to embodiment 6. The process shown in Fig. 20 is obtained by adding step S101 between steps S34 and S35 in the process shown in Fig. 7. Explanations of the same steps as in Fig. 7 will be omitted.
[0091] In the estimation process of the deteriorated area in this embodiment, the control unit 11 of the server 10 performs the same process as steps S31 to S34 in FIG. 7. As a result, the control unit 11 can generate a label image in which a bounding box surrounding the deteriorated area, a discrimination label indicating the type of the deteriorated area, and a confidence level for the discrimination label (probability that the deterioration state indicated by the discrimination label) are added to the original image (photographed image). The control unit 11 acquires a discrimination threshold for the tool 30 to be processed here from the discrimination threshold DB (S101). Specifically, the control unit 11 identifies the model number of the tool 30 to be processed, and reads out the discrimination threshold corresponding to the identified model number from the discrimination threshold DB. For example, a code (e.g., a barcode, a QR code) in which information such as the model number of the tool 30 is encoded is printed or attached to a holder that holds the tool 30, and the control unit 11 can acquire the model number of the tool 30 to be processed by reading the code using the code reading unit. The model number of the tool 30 may be received, for example, via the input unit 14, or may be identified by template matching using a template image that indicates the image features of the captured image of each tool, or may be identified by image recognition using a learning model.
[0092] The control unit 11 uses the discrimination threshold value acquired in step S101 to determine whether or not the tool in the captured image has a deteriorated area based on the label image generated in step S34 (S35). For example, the control unit 11 determines whether or not the deteriorated area exists depending on whether or not there is a bounding box in which the confidence level for the discrimination label in the label image is equal to or higher than the discrimination threshold value acquired in step S101. Specifically, when the control unit 11 determines that there is a bounding box equal to or higher than the discrimination threshold value, it determines that the area indicated by this bounding box is a deteriorated area. Then, the control unit 11 returns to the process of FIG. 6. Note that, in step S14 in FIG. 6, if the control unit 11 specifies the model number of the tool 30 to be captured in addition to the blade information of the captured image, the control unit 11 may acquire the discrimination threshold value of the tool 30 based on the model number of the tool 30 specified in step S14 in step S101.
[0093] By the above-mentioned process, in this embodiment, the threshold value for determining whether or not a region should be determined as a deteriorated region with respect to the certainty of the discrimination label in the label image can be switched for each tool (each model number of the tool). For example, the smaller the diameter of the blade 31, the more the deteriorated region is affected by deterioration, so that a small discrimination threshold value is set for a tool with a small diameter of the blade 31, thereby enabling a configuration in which even a deteriorated region with a small certainty is accurately detected. On the other hand, a large discrimination threshold value is set for a tool with a large diameter of the blade 31, thereby enabling a configuration in which a deteriorated region with a certainty below the discrimination threshold is excluded from the detection target, and a deteriorated region with a certainty equal to or greater than the discrimination threshold is reliably detected.
[0094] The configuration of this embodiment is applicable to the tool management systems of the above-mentioned embodiments 1 to 5, and similar processes can be executed and similar effects can be obtained even when applied to the tool management systems of the above-mentioned embodiments 1 to 5. Furthermore, the modified examples described in the above-mentioned embodiments 1 to 5 can also be applied to this embodiment as appropriate.
[0095] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. The claims may also be written in a format in which a multiple claim cites at least one other multiple claim (multi-multi claim).
[0096] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]
[0097] 10 Server 11 Control section 12 Storage section 13. Communications Department 20 Imaging Equipment 21 Control section 22 Memory section 23 Communications Department 26 Photography Department 27 Shooting direction control unit 30 Tools 31 Blade part 31a blade M1 Top model M2 Diagonal Model M3 Outer circumference model M4 rake face model 12a Tool DB
Claims
1. A plurality of photographed images of a cutting edge provided on a cutting tool are obtained from a plurality of photographing directions; Identifying a shooting direction in the acquired photographed image; Selecting a learning model corresponding to the specified shooting direction from learning models that have been trained to output information regarding a deterioration state of the cutting tool blade when a photographed image of the cutting tool blade is input, the learning model being prepared for each shooting direction; inputting the acquired photographed image into a selected learning model to acquire information regarding a deterioration state of the cutting edge of the cutting tool in the photographed image; A process for determining a deterioration state of the blade portion based on information on the deterioration state acquired for each of a plurality of shooting directions, The blade portion has a plurality of blades, The photographed images include a plurality of photographed images of each blade taken from the plurality of photographing directions, the plurality of photographed images include photographed images of a side surface of each blade of the blade portion, the photographed images of the side surface include an image photographed with a focus on a rake face of the blade and an image photographed with a focus on a flank face of the blade, The learning model includes a learning model for a photographed image focused on a rake face of the blade, and a learning model for a photographed image focused on a flank face of the blade, and the information on the deterioration state output by the learning model includes information indicating an area of the blade in a deteriorated state in the input photographed image, The image photographed with the focus on the rake face of the blade and the image photographed with the focus on the flank of the blade are inputted into a learning model for the image photographed with the focus on the rake face of the blade, or into a learning model for the image photographed with the focus on the flank of the blade, respectively, to obtain information regarding the deterioration state of the blade; When a photographed image in which the focus is set on the rake face of the blade is input to a learning model for photographed images in which the focus is set on the rake face of the blade and a region of the blade in a deteriorated state in the photographed image is acquired outside a predetermined region in the photographed image, the blade is not determined to be a deteriorated blade. A program that causes a computer to carry out processing.
2. Acquire an image of a cutting edge provided on a cutting tool; inputting the acquired photographed image into a learning model that has been trained to output information regarding a deterioration state of the blade portion when a photographed image of a blade portion of a cutting tool is input, and acquiring information regarding a deterioration state of the blade portion of the cutting tool in the photographed image; The information regarding the deterioration state includes a bounding box indicating a region of the blade in the deteriorated state in the input photographed image, Identifying a line segment that passes through a midpoint of one diagonal of the bounding box and intersects the diagonal; The distance between two intersections between the identified line segment and the contour of the deteriorated blade region is calculated as the size of the deteriorated blade region. A program that causes a computer to carry out processing.
3. Acquire an image of a cutting edge provided on a cutting tool; inputting the acquired photographed image into a learning model that has been trained to output information regarding a deterioration state of the blade portion when a photographed image of a blade portion of a cutting tool is input, and acquiring information regarding a deterioration state of the blade portion of the cutting tool in the photographed image; The information regarding the deterioration state includes a bounding box indicating a region of the blade in the deteriorated state in the input photographed image, estimating an undegraded shape of the blade based on an area of the deteriorated blade within the bounding box; A difference between the estimated shape before deterioration and the region of the blade in the deteriorated state is calculated as a size of the region of the blade in the deteriorated state. A program that causes a computer to carry out processing.
4. Identifying the position of the blade in the captured image based on the acquired captured image; determining whether the identified blade position matches a pre-registered position for the cutting tool; If it is determined that there is no match, the photographing position of the cutting tool is changed so that the position of the blade in the photographed image approaches the preregistered position.
4. The program according to claim 1, which causes the computer to execute a process.
5. The information regarding the deterioration state includes an area of the blade portion that is in a deteriorated state in the input photographed image and a possibility that the area of the blade portion is in a deteriorated state, acquiring a threshold value for determining that the cutting edge is in a deteriorated state based on the possibility included in the information on the deterioration state acquired from the learning model according to the type of the cutting tool; determining whether the blade portion is in a deteriorated state based on whether the possibility included in the information on the deteriorated state obtained from the learning model is equal to or greater than the obtained threshold value; 4. The program according to claim 1, which causes the computer to execute a process.
6. The captured image includes a plurality of captured images of the blade portion captured from a plurality of shooting directions, The learning model includes a plurality of learning models that have been trained for each shooting direction, Identifying a shooting direction in the acquired photographed image; Select a learning model according to the identified shooting direction, The acquired photographed image is input into a selected learning model to acquire information regarding a deterioration state of the blade portion in the photographed image; The deterioration state of the blade portion is determined based on the information on the deterioration state acquired for each of a plurality of shooting directions.
4. The program according to claim 2 or 3, which causes the computer to execute a process.
7. The blade portion has a plurality of blades, The captured images include a plurality of captured images of each blade captured from the plurality of capturing directions, The captured images include a captured image of a tip surface of the blade portion and a captured image of a tip surface of each blade of the blade portion, The captured image of the tip surface of the blade portion and the captured image of the tip surface of each blade are input into a learning model corresponding to the photographing direction of the tip surface, and information regarding the deterioration state of the blade portion or each blade is obtained. The program according to claim 6, which causes the computer to execute a process.
8. The information regarding the deterioration state includes an area of the blade portion in the input captured image that is in a deterioration state, and information indicating which of a plurality of types of deterioration states the blade portion is in; A screen is output that displays the deteriorated area of the blade portion in the input photographed image in a manner corresponding to the type of deterioration state.
4. The program according to claim 1, which causes the computer to execute a process.
9. determining whether a predetermined area of the captured image is in focus based on the captured image; If it is determined that the image is not in focus, the photographing position relative to the cutting tool is changed or the focus is adjusted.
4. The program according to claim 1, which causes the computer to execute a process.
10. Acquire an image of a cutting edge provided on a cutting tool; A process of inputting an acquired photographed image of a cutting tool blade into a learning model that has been trained to output information regarding a deterioration state of the blade when a photographed image of the cutting tool blade is input, and acquiring information regarding a deterioration state of the cutting tool blade in the photographed image, The captured image includes a plurality of captured images of the blade portion captured from a plurality of shooting directions, The learning model includes a plurality of learning models that have been trained for each shooting direction, For each of the plurality of captured images obtained Identifying a shooting direction in the captured image; Select a learning model according to the identified shooting direction, The acquired photographed image is input into a selected learning model to acquire information regarding a deterioration state of the blade portion in the photographed image; A comprehensive assessment result of the deterioration state of the blade portion is obtained based on the information on the deterioration state obtained for each of the plurality of photographed images. A program that causes a computer to carry out processing.
11. A plurality of photographed images of a cutting edge provided on a cutting tool are obtained from a plurality of photographing directions; Identifying a shooting direction in the acquired photographed image; Selecting a learning model corresponding to the specified shooting direction from learning models that have been trained to output information regarding a deterioration state of the cutting tool blade when a photographed image of the cutting tool blade is input, the learning model being prepared for each shooting direction; inputting the acquired photographed image into a selected learning model to acquire information regarding a deterioration state of the cutting edge of the cutting tool in the photographed image; A process for determining a deterioration state of the blade portion based on information on the deterioration state acquired for each of a plurality of shooting directions, The blade portion has a plurality of blades, The photographed images include a plurality of photographed images of each blade taken from the plurality of photographing directions, the plurality of photographed images include photographed images of a side surface of each blade of the blade portion, the photographed images of the side surface include an image photographed with a focus on a rake face of the blade and an image photographed with a focus on a flank face of the blade, The learning model includes a learning model for a photographed image focused on a rake face of the blade, and a learning model for a photographed image focused on a flank face of the blade, and the information on the deterioration state output by the learning model includes information indicating an area of the blade in a deteriorated state in the input photographed image, The image photographed with the focus on the rake face of the blade and the image photographed with the focus on the flank of the blade are inputted into a learning model for the image photographed with the focus on the rake face of the blade, or into a learning model for the image photographed with the focus on the flank of the blade, respectively, to obtain information regarding the deterioration state of the blade; When a photographed image in which the focus is set on the rake face of the blade is input to a learning model for photographed images in which the focus is set on the rake face of the blade and a region of the blade in a deteriorated state in the photographed image is acquired outside a predetermined region in the photographed image, the blade is not determined to be a deteriorated blade. An information processing method in which processing is performed by a computer.
12. Acquire an image of a cutting edge provided on a cutting tool; inputting the acquired photographed image into a learning model that has been trained to output information regarding a deterioration state of the blade portion when a photographed image of a blade portion of a cutting tool is input, and acquiring information regarding a deterioration state of the blade portion of the cutting tool in the photographed image; The information regarding the deterioration state includes a bounding box indicating a region of the blade in the deteriorated state in the input photographed image, Identifying a line segment that passes through a midpoint of one diagonal of the bounding box and intersects the diagonal; The distance between two intersections between the identified line segment and the contour of the deteriorated blade region is calculated as the size of the deteriorated blade region. An information processing method in which processing is performed by a computer.
13. Acquire an image of a cutting edge provided on a cutting tool; inputting the acquired photographed image into a learning model that has been trained to output information regarding a deterioration state of the blade portion when a photographed image of a blade portion of a cutting tool is input, and acquiring information regarding a deterioration state of the blade portion of the cutting tool in the photographed image; The information regarding the deterioration state includes a bounding box indicating a region of the blade in the deteriorated state in the input photographed image, estimating an undegraded shape of the blade based on an area of the deteriorated blade within the bounding box; A difference between the estimated shape before deterioration and the region of the blade in the deteriorated state is calculated as a size of the region of the blade in the deteriorated state. An information processing method in which processing is performed by a computer.
14. Identifying a position of a blade in the captured image based on the acquired captured image; determining whether the identified blade position matches a pre-registered position for the cutting tool; If it is determined that there is no match, the photographing position of the cutting tool is changed so that the position of the blade in the photographed image approaches the preregistered position.
14. The information processing method according to claim 11, wherein the processing is executed by the computer.
15. The information regarding the deterioration state includes an area of the blade portion that is in a deteriorated state in the input photographed image and a possibility that the area of the blade portion is in a deteriorated state, acquiring a threshold value for determining that the cutting edge is in a deteriorated state based on the possibility included in the information on the deterioration state acquired from the learning model according to the type of the cutting tool; determining whether the blade portion is in a deteriorated state based on whether the possibility included in the information on the deteriorated state obtained from the learning model is equal to or greater than the obtained threshold value; 14. The information processing method according to claim 11, wherein the processing is executed by the computer.
16. An information processing device having a control unit, The control unit is A plurality of photographed images of a cutting edge provided on a cutting tool are obtained from a plurality of photographing directions; Identifying a shooting direction in the acquired photographed image; Selecting a learning model corresponding to the specified shooting direction from learning models that have been trained to output information regarding a deterioration state of the cutting tool blade when a photographed image of the cutting tool blade is input, the learning model being prepared for each shooting direction; inputting the acquired photographed image into a selected learning model to acquire information regarding a deterioration state of the cutting edge of the cutting tool in the photographed image; A process for determining a deterioration state of the blade portion based on information on the deterioration state acquired for each of a plurality of shooting directions, The blade portion has a plurality of blades, The photographed images include a plurality of photographed images of each blade taken from the plurality of photographing directions, the plurality of photographed images include photographed images of a side surface of each blade of the blade portion, the photographed images of the side surface include an image photographed with a focus on a rake face of the blade and an image photographed with a focus on a flank face of the blade, The learning model includes a learning model for a photographed image focused on a rake face of the blade, and a learning model for a photographed image focused on a flank face of the blade, and the information on the deterioration state output by the learning model includes information indicating an area of the blade in a deteriorated state in the input photographed image, The image photographed with the focus on the rake face of the blade and the image photographed with the focus on the flank of the blade are inputted into a learning model for the image photographed with the focus on the rake face of the blade, or into a learning model for the image photographed with the focus on the flank of the blade, respectively, to obtain information regarding the deterioration state of the blade; A process of not determining that a blade is a deteriorated blade in a photographed image obtained by inputting the photographed image focused on the rake face of the blade into a learning model for photographed images focused on the rake face of the blade and, when the region of the deteriorated blade in the photographed image is outside a predetermined region in the photographed image, the blade is not determined to be a deteriorated blade. An information processing device that executes the above.
