Feature quantity extraction device and feature quantity extraction method

Abstract

To provide a technique for extracting feature quantities regarding a time series of sparks using multiple spark images.SOLUTION: A feature quantity extraction device is provided, comprising an image acquisition unit, a three-dimensional plotting unit for generating a three-dimensional plot map, a spark size determination unit, and a parent-child relationship determination unit configured to relax conditions for parent-child conditions to be satisfied in a specific case where the spark size is determined to be larger than a predetermined spark threshold level by the spark size determination unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a feature extraction device and a feature extraction method. [Background technology]

[0002] Spark observation has been known as a means of identifying the quality of steel and tool materials and the deterioration of steel and tools. Patent Document 1 discloses a technology in which sparks generated when steel is rubbed are photographed with a camera exposed for a fixed period of time, and the feature quantities of the sparks are determined from the obtained image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-134204 A Summary of the Invention [Problem to be solved by the invention]

[0004] When determining the feature amount of a spark from a single spark image, it is not possible to calculate information about the time series from the generation, explosion, and extinction of the spark. For this reason, it is possible to calculate information about the time series of the spark from multiple spark images arranged in chronological order. However, when identifying an explosion using multiple spark images arranged in chronological order, if an explosion occurs in a spark, the movement of the spark after the explosion differs from the movement of the spark before the explosion, which can cause a problem: it is difficult to distinguish multiple sparks caused by explosions from the same spark from other sparks. This problem is common to technologies that extract feature amounts of sparks generated by friction. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first aspect of the present disclosure, there is provided a feature extraction device that extracts feature amounts of sparks generated by friction. This feature extraction device includes an image acquisition unit that acquires multiple spark images obtained by capturing images of the sparks in chronological order using an imaging device; a three-dimensional plot unit that selects multiple chronologically consecutive images from the multiple spark images acquired by the image acquisition unit as multiple selected images and creates a three-dimensional plot map for the multiple selected images, which is a map with three axes: a first axis corresponding to the horizontal direction of the multiple selected images, a second axis corresponding to the vertical direction of the multiple selected images, and a time axis; a spark size determination unit that determines the size of the spark; and a parent-child relationship determination unit that calculates an index value related to the position of the spark on the three-dimensional plot map and, if the calculated index value satisfies a predetermined parent-child condition, determines that the multiple sparks that were the source of the calculation of the index value are in a parent-child relationship indicating that they were generated from the same spark, and that this parent-child relationship constitutes the feature. In a specific case where the spark size determination unit determines that the size of the spark is greater than a predetermined spark threshold, the parent-child relationship determination unit relaxes the condition for the parent-child condition to be met. The inventors have discovered that the size of a spark increases immediately before the spark explodes. According to this aspect, the feature extraction device relaxes the condition for the parent-child condition to be met when the size of the spark is greater than the spark threshold, thereby making it possible to easily determine the parent-child relationship of sparks in time-series spark images. (2) In the above embodiment, the system may further include an explosion identification unit that identifies that an explosion has occurred in a specific spark when the parent-child relationship determination unit determines that there is a parent-child relationship between a specific spark that is one of the sparks at a first point on the time axis of the three-dimensional plot map and each of the multiple sparks at a second point that is a point later on the time axis than the first point. According to this embodiment, an explosion can be easily identified using the determination result of the parent-child relationship determination unit. (3) In the above aspect, the parent-child relationship determination unit may calculate the index value related to the position of the spark using plots of two or more consecutive sparks on the time axis, and the parent-child condition may be a condition related to the position of the spark. According to this aspect, the parent-child relationship can be easily determined using the index value related to the position of the spark. (4) In the above-described embodiment, a clustering unit may be further provided that performs clustering to distinguish the sparks using the positions of the sparks in the three-dimensional plot map, and the parent-child relationship determination unit may determine whether the plurality of sparks classified into the same cluster by the clustering unit satisfy the parent-child condition. According to this embodiment, since it is determined whether the plurality of sparks classified into the same cluster satisfy the parent-child condition, it is possible to reduce the possibility that a plurality of sparks classified into different clusters will be determined to satisfy the parent-child condition. (5) In the above embodiment, the clustering unit may classify the plurality of sparks represented by the first axis and the second axis at each point on the time axis into the same cluster if there is an overlapping area between the plurality of sparks. According to this embodiment, clustering of the sparks can be easily performed based on whether or not there is an overlapping area. (6) In the above aspect, the clustering unit may classify the plurality of sparks into the same cluster when the distance between the plane coordinate positions of the plurality of sparks represented by the first axis and the second axis at each point on the time axis is equal to or less than a predetermined cluster threshold. According to this aspect, clustering of the sparks can be easily performed using the distance between the plane coordinate positions of the sparks. (7) In the above embodiment, the clustering unit may further include a line detection unit that detects, as a line, the trajectories of the sparks classified into the same cluster across the selected images, the line detection unit detecting a first line that is a line detected in a plurality of first selected images that are the selected images included in a first predetermined time period, and a second line that is detected in a plurality of second selected images that are the selected images included in a second predetermined time period that includes spark images captured after the plurality of first selected images, the parent-child relationship determination unit may calculate, as the index value, a relationship value that indicates a positional relationship between the first line and the second line, the parent-child condition being a condition that the relationship value is equal to or less than a predetermined parent-child relationship threshold, and the parent-child relationship determination unit may relax the condition for the parent-child condition to be met by setting the parent-child relationship threshold higher in the specific case than in other cases. According to this embodiment, the parent-child relationship can be easily determined using the positional relationship between the first line and the second line. (8) In the above aspect, the image processing device may further include a binarization processing unit, which performs binarization on the plurality of spark images acquired by the image acquisition unit using a predetermined binarization threshold. According to this aspect, the binarization processing unit can perform binarization. (9) In the above aspect, the tool estimation unit may further include a tool estimation unit that estimates the state of the tool by using the feature amount of the spark. According to this aspect, the tool estimation unit can estimate the state of the tool by using the feature amount of the spark. (10) In the above aspect, a condition adjustment unit may be provided, and the condition adjustment unit may adjust at least one of the tool replacement timing and the machining conditions of the tool using the state of the tool estimated by the tool estimation unit. According to this aspect, the condition adjustment unit can adjust the tool replacement timing and the machining conditions based on the state of the tool. (11) In the above aspect, the method may further include a processing object estimation unit, wherein the processing object estimation unit estimates the state of the processing object by using the feature amount of the sparks. According to this aspect, the processing object estimation unit can estimate the state of the steel material that is the processing object. (12) In the above aspect, the machining apparatus may further include a condition adjustment unit, and the condition adjustment unit may adjust at least one of the tool replacement timing and the machining conditions of the tool using the state of the workpiece estimated by the workpiece estimation unit. According to this aspect, the condition adjustment unit can adjust the tool replacement timing and the machining conditions based on the state of the workpiece. (13) According to a second aspect of the present disclosure, there is provided a feature extraction method for extracting feature quantities of sparks generated by friction, the feature extraction method comprising: an image acquisition step of acquiring a plurality of spark images obtained by capturing images of the sparks in time series using an imaging device; a three-dimensional plotting step of selecting a plurality of chronologically consecutive images from the plurality of spark images acquired by the image acquisition step as a plurality of selected images, and creating a three-dimensional plot map for the plurality of selected images, the three-axis map being a map of a first axis corresponding to the horizontal direction of the plurality of selected images, a second axis corresponding to the vertical direction of the plurality of selected images, and a time axis; a spark size determination step of determining a size of the spark on the three-dimensional plot map; and a parent-child relationship determination step of calculating an index value related to the position of the spark, and determining, if the calculated index value satisfies a predetermined parent-child condition, that the plurality of sparks plotted at different points on the time axis have a parent-child relationship constituting the feature quantities indicating that the sparks were generated from the same spark, The parent-child relationship determination step includes a step of relaxing the condition for the parent-child condition to be satisfied in a specific case where the spark size determination step determines that the size of the spark is larger than a predetermined spark size. According to this aspect, the feature extraction device can easily determine the parent-child relationship of sparks in time-series spark images by relaxing the condition for the parent-child condition to be satisfied when the size of the spark is larger than a spark threshold. The present disclosure can be realized in various forms, and in addition to the forms described above, can be realized in the form of a computer program that extracts features of sparks, a non-transitory recording medium on which the computer program is recorded, etc. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram showing the configuration of a feature amount determination system according to an embodiment of the present invention. [Figure 2] FIG. 2 is an explanatory diagram showing the configuration of a CPU. [Figure 3] 10 is a flowchart showing a feature extraction process performed by the feature extraction device. [Figure 4] FIG. 10 is a diagram for explaining a three-dimensional plotting process. [Figure 5] A diagram showing one plot map. [Figure 6] FIG. 10 is a diagram for explaining clustering processing. [Figure 7] FIG. 4 is a diagram for explaining a first straight line. [Figure 8] FIG. 10 is a diagram for explaining a second straight line. [Figure 9] 10A and 10B are diagrams for explaining a spark size determination process. [Figure 10] FIG. 10 is a diagram for explaining one routine of steps S4 to S8 that are repeated. [Figure 11] FIG. 10 is a diagram for explaining a parent-child relationship determination process. [Figure 12] FIG. 10 is a diagram for explaining step S11. [Figure 13] FIG. 1 is a diagram illustrating a process for determining a parent-child relationship in another embodiment. [Figure 14] FIG. 2 is a second diagram for explaining the process of determining a parent-child relationship in another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: 1 is an explanatory diagram showing the configuration of a feature extraction system 10 according to this embodiment. The feature extraction system 10 includes an imaging device 12 and a feature extraction device 14.

[0009] A feature extraction system 10 of this embodiment captures an image of a spark C generated by friction. Specifically, as shown in FIG. 1 , the feature extraction system 10 captures an image of the spark C generated when a steel material A comes into contact with a tool B. The tool B is a tool of a machining device currently performing machining, and the steel material A is an object of the machining device to be machined by the tool B. A feature extraction device 14 of the feature extraction system 10 determines a feature of the spark C based on an image of the spark C captured by the imaging device 12. The feature extraction device 14 also identifies an explosion D contained in the spark C. The feature determination system 10 is used, for example, in a grinding machine or machining center when machining a steel material A with a tool B. In a grinding machine, for example, the steel material A is cut with a rotating grinding wheel, which is the tool B. In a machining center, for example, the steel material A is cut with a tool B attached to a rotating spindle of a spindle head.

[0010] The imaging device 12 generates a plurality of spark images by continuously capturing images of the spark C in a time series. The imaging device 12 is, for example, a camera. In this embodiment, the imaging device 12 continuously captures images of the spark C at very short time intervals. Here, the very short time interval refers to a time interval in which, when focusing on one of the streamlines that are the lines of the spark C, the process from the appearance to the disappearance of that streamline can be captured by multiple images. In this embodiment, the time interval is an imaging speed of 10,000 frames per second.

[0011] The feature extraction device 14 is, for example, a server. The feature extraction device 14 is not limited to a single server, and may be configured with multiple servers. Furthermore, the feature extraction device 14 may be a personal computer or the like.

[0012] The feature extraction device 14 includes a CPU 15 that performs arithmetic processing, a ROM 17 that stores a control program for the CPU 15, and a RAM 19 that serves as a data work area. The CPU 15 reads and executes a program stored in an auxiliary storage device 18 such as an HDD or SSD, thereby cooperating with the above-mentioned hardware and performing various functions in accordance with the control program.

[0013] 2 is an explanatory diagram showing the configuration of the CPU 15. The CPU 15 includes an image acquisition unit 21, a brightness conversion processing unit 22, a binarization processing unit 23, a three-dimensional plotting unit 24, a clustering unit 25, a line detection unit 26, a parent-child relationship determination unit 27, a spark size determination unit 28, a rupture identification unit 29, a tool estimation unit 30, a processing object estimation unit 31, and a condition adjustment unit 32.

[0014] The image acquisition unit 21 acquires a plurality of spark images generated by the imaging device 12 and stores them in the RAM 19. In this embodiment, each of the plurality of spark images is a color image.

[0015] The brightness conversion processing unit 22 converts the brightness of each pixel (picture element) of the multiple images captured by the image acquisition unit 21, converting the images into grayscale images consisting of brightness values ​​only, and stores the converted multiple spark images in RAM 19.

[0016] The binarization processing unit 23 binarizes the plurality of grayscale images converted by the brightness conversion processing unit 22 using a predetermined threshold value, and stores the obtained plurality of binarized images in the RAM 19 as spark images.

[0017] The three-dimensional plot unit 24 selects a plurality of chronologically consecutive images as a plurality of selected images from the plurality of spark images acquired by the image acquisition unit 21, and creates a three-dimensional plot map for the selected plurality of selected images. The three-dimensional plot map is a map with three axes: a first axis corresponding to the horizontal direction of each of the plurality of selected images, a second axis corresponding to the vertical direction of each of the plurality of selected images, and a time axis corresponding to the frame numbers of the plurality of selected images and the time. In other words, the first axis and the second axis identify a planar position within the selected images. Details of the three-dimensional plot map will be described later.

[0018] The clustering unit 25 performs clustering to distinguish between sparks using the positions of the sparks in the three-dimensional plot map. The clustering method will be described in detail later.

[0019] The line detection unit 26 detects, as lines, the trajectories of sparks classified into the same cluster across each point on the time axis in the three-dimensional plot map, i.e., across multiple selected images, in the clustering unit 25. More specifically, the line detection unit 26 detects a first line, which is a line detected in multiple first selected images as multiple selected images included in a first predetermined period, and a second line, which is detected in multiple second selected images as multiple selected images included in a second predetermined period that includes spark images captured after the multiple first selected images. The line detection method will be described in detail later.

[0020] The spark size determination unit 28 determines the size of the spark using the positional relationship between the spark and the straight line detected by the straight line detection unit 26. The method for determining the size of the spark will be described in detail later.

[0021] The parent-child relationship determination unit 27 calculates an index value relating to the position of the spark on the 3D plot map. Furthermore, if the calculated index value satisfies a predetermined parent-child condition, the parent-child relationship determination unit 27 determines that the multiple sparks on the 3D plot map from which the index values ​​were calculated are related to the same spark. The parent-child relationship constitutes a feature of the spark. In a specific case where the spark size determination unit 28 determines that the size of the spark is greater than a predetermined spark threshold, the parent-child relationship determination unit 27 relaxes the condition for the parent-child condition to be satisfied compared to other cases where the size is determined to be equal to or less than the spark threshold. The degree to which the condition is relaxed may be determined experimentally, for example. The relaxation of the condition for the parent-child condition is based on the inventors' finding that the size of a spark increases immediately before the spark explodes. Relaxing the condition for the parent-child condition makes it easy to determine whether a spark after the explosion and a spark before the explosion have a parent-child relationship.

[0022] In this embodiment, the index value calculated by the parent-child relationship determination unit 27 is an index value related to the position of the spark, derived using plots of two or more consecutive sparks on the time axis of the three-dimensional plot map. In addition, in this embodiment, the parent-child condition is a condition related to the position of the spark. As will be described in detail later, the parent-child relationship determination unit 27 calculates a relationship value indicating the positional relationship between the first line and the second line as an index value. In this embodiment, the parent-child condition is a condition that the relationship value is equal to or less than a predetermined parent-child relationship threshold. In the above-described specific case, the parent-child relationship determination unit 27 relaxes the condition for the parent-child condition to be met by setting a higher parent-child relationship threshold in comparison with the above-described other cases. Details of the method for determining a parent-child relationship by the parent-child relationship determination unit 27 will be described later.

[0023] The rupture identification unit 29 identifies that an explosion has occurred in a specific spark when the parent-child relationship determination unit 27 determines that there is a parent-child relationship between a specific spark, which is one spark at a first point on the time axis of the three-dimensional plot map, and each of multiple sparks at a second point that is a point later on the time axis than the first point. In other words, the rupture identification unit 29 identifies that an explosion has occurred when it determines that multiple sparks have been generated from one spark. The method of identifying an explosion will be described in detail below.

[0024] The tool estimation unit 30 estimates the state of the tool using the feature amount of the spark. The method of estimating the tool state will be described in detail later.

[0025] The processing object estimation unit 31 estimates the state of the processing object using the feature amount of the spark. The method of estimating the state of the processing object will be described in detail later.

[0026] The condition adjustment unit 32 adjusts at least one of the tool replacement timing and the tool machining conditions, using the state of the tool estimated by the tool estimation unit 30 or the state of the workpiece estimated by the workpiece estimation unit 31. The method of adjusting the tool conditions will be described in detail later.

[0027] Fig. 3 is a flowchart showing the feature extraction process performed by the feature extraction device 14. In step S1 of Fig. 3, the feature extraction device 14 acquires a plurality of spark images generated by the imaging device 12 and stores them in the auxiliary storage device 18. Here, the feature extraction device 14 may receive the spark images via a wired or wireless connection. In this embodiment, each of the plurality of spark images is a color image, and has a resolution of 1280 x 720 pixels.

[0028] 3, the brightness conversion processing unit 22 converts the brightness of each pixel of the plurality of spark images acquired by the image acquisition unit 21, thereby converting the images into grayscale images consisting of brightness values ​​only, and stores the plurality of converted images in the RAM 19. Note that one method of converting a color image into a grayscale image is to convert the RGB values ​​of each pixel in the spark image into a brightness value Y in the YIQ notation system using a predetermined conversion formula, for example. Note that the method of converting a color image into a grayscale image is not limited to the above method.

[0029] In step S3 of Fig. 3, the binarization processing unit 23 performs binarization processing on the multiple grayscale images converted by the brightness conversion processing unit 22 using a predetermined binarization threshold, and stores the multiple images obtained in the RAM 19. This makes it possible to make the shape of the sparks clearer. In this embodiment, for the grayscale images, if the brightness value Y is greater than a predetermined value, it is determined to be a white pixel, and if it is equal to or less than the predetermined value, it is determined to be a black pixel. Therefore, the sparks are recognized as white. Alternatively, if the brightness value Y is greater than a predetermined value, it is determined to be a black pixel, and if it is equal to or less than the predetermined value, it is determined to be a white pixel.

[0030] In step S4 of Fig. 3, the three-dimensional plotting unit 24 selects a plurality of selected images that are consecutive in time series from the plurality of spark images acquired by the image acquiring unit 21. The plurality of selected images are images acquired during two predetermined periods that partially overlap. The three-dimensional plotting unit 24 performs a three-dimensional plotting process for the selected plurality of selected images to create a three-dimensional plot map that is a map of three axes: a first axis corresponding to the horizontal direction of the plurality of selected images, a second axis corresponding to the vertical direction of the plurality of selected images, and a time axis that is the frame number of the plurality of selected images and corresponds to time.

[0031] FIG. 4 is a diagram for explaining the three-dimensional plotting process. The diagram on the left side of FIG. 4 shows f spark images Im1 to Imf acquired continuously in time series. FIG. 4 shows a first axis Ax1, a second axis Ax2, and a time axis Axt that are orthogonal to each other. The first axis Ax1 corresponds to the horizontal direction of the spark images Im1 to Imf, and the second axis Ax2 corresponds to the vertical direction of the spark images Im1 to Imf. As described above, the resolution of the spark images Im1 to Imf is 1280 × 720 pixels. In this embodiment, the coordinates represented by the first axis Ax1 and the second axis Ax2 correspond to the coordinates of pixels in the spark images Im1 to Imf. For example, as shown in spark image Im1, the coordinate g1 (Ax1, Ax2) of the pixel located in the upper left corner is (1, 1). Furthermore, the coordinates g2 (Ax1, Ax2) of the pixel located at the bottom right are (1280, 720). Each spark image Im1 to Imf is indicated with a corresponding frame number. The frame numbers correspond to points on the time axis Axt. The spark images Im1 to Imf shown in FIG. 4 are images obtained by converting a color image acquired by the imaging device 12 into a grayscale image by the brightness conversion processing unit 22 and then binarizing it by the binarization processing unit 23. In this embodiment, sparks are captured at very short time intervals, so the trajectory of the spark is acquired by multiple spark images Im1 to Imf. The spark images Im1 to Im4 show two different sparks, spark a and spark b. Spark a progresses in the -X direction from frame 1 to frame 4. Spark b progresses in the +X direction from frame 1 to frame 4. That is, for spark a, the +X direction is the root side of the spark and the -X direction is the tip side of the spark. On the other hand, for spark b, the -X direction is the root side of the spark and the +X direction is the tip side of the spark. Spark image Imf shows spark g, which is different from sparks a and b.

[0032] The three-dimensional plotting unit 24 selects a plurality of first selected images CH1 that are consecutive in time series and included in a first predetermined period from the spark images Im1 to Imf. In this embodiment, the three-dimensional plotting unit 24 determines an arbitrary frame of interest N from the spark images Im1 to Imf, and selects 2R+1 spark images from frame NR to frame N+R as the plurality of first selected images CH1. In this embodiment, the value of R is "1," and the frame of interest is the middle frame. That is, in the first routine, the three-dimensional plotting unit 24 selects spark images Im1 to Im3 corresponding to frames 1 to 3 as the plurality of first selected images CH1. Note that the value of R may be any value smaller than N, and is not limited to "1." In addition, in the first routine, the three-dimensional plotting unit 24 selects a plurality of second selected images CH2 as a plurality of selected images included in a second predetermined period, including a spark image Im4 that was captured after the plurality of first selected images CH1. In this embodiment, the plurality of second selected images CH2 are a plurality of consecutive spark images obtained by adding "1" to the frame numbers of the plurality of first selected images CH1.

[0033] The three-dimensional plot unit 24 creates a three-dimensional plot map PL1 shown in the lower right of FIG. 4 from the spark images Im1 to Im3, which are the multiple first selected images CH1, and the frame numbers of each of the spark images Im1 to Im3. As described above, the three-dimensional plot map PL1 is represented by three orthogonal axes: the first axis Ax1, the second axis Ax2, and the time axis Axt. Note that frame numbers are assigned to the spark images in chronological order, so each point on the time axis Axt corresponds to a frame number. In the three-dimensional plot map PL1, to distinguish between the same sparks in different frames, the spark a in spark image Im1 is written as spark 1a, the spark a in spark image Im2 is written as spark 2a, and the spark a in spark image Im3 is written as spark 3a. Similarly, for spark b, spark b in spark image Im1 will be described as spark 1b, spark b in spark image Im2 will be described as spark 2b, and spark b in spark image Im3 will be described as spark 3b. Furthermore, the three-dimensional plot unit 24 creates a three-dimensional plot map PL2 (not shown) from spark images Im2 to Im4 in the plurality of second selected images CH2 and the respective frame numbers of the spark images Im2 to Im4. Note that the two three-dimensional plot maps PL1 and PL2 may be represented by a single plot map PLA shown in FIG. 5.

[0034] In step S5 of FIG. 3, the clustering unit 25 performs a clustering process using the sparks in the three-dimensional plot map PL1 to perform clustering for distinguishing the sparks.

[0035] FIG. 6 is a diagram for explaining the clustering process. The two-dimensional graph G1 shown in the upper diagram of FIG. 5 is a diagram of the three-dimensional plot map PL1 shown in FIG. 4 viewed from the direction of the time axis Axt. The three-dimensional plot map PL2 shown in the lower diagram of FIG. 6 is a diagram showing the state after clustering by the clustering unit 25. The two-dimensional graph G1 plots the coordinate positions of sparks a and b on the first axis Ax1 and the second axis Ax2 in spark images Im1 to Im3, which are the multiple first selected images CH1 shown in FIG. 4. For ease of understanding, the two-dimensional graph G1 represents sparks 1a and 1b with solid lines, sparks 2a and 2b with dashed lines, and sparks 3a and 3b with thick lines. As described above, the two-dimensional graph G1 represents the planar coordinate positions of sparks 1a to 3b represented by the first axis Ax1 and the second axis Ax2 at each point on the time axis Axt.

[0036] In this embodiment, the clustering unit 25 detects an overlapping region M between sparks a and b in the two-dimensional graph G1. The overlapping region M refers to an overlapping region between sparks in two chronologically consecutive spark images in the two-dimensional graph G1. In the two-dimensional graph G1, there is an overlapping region M1 between sparks 1a and 2a, which are sparks in two chronologically consecutive spark images. In addition, there is an overlapping region M2 between sparks 2a and 3a, which are sparks in two chronologically consecutive spark images. Similarly, there is an overlapping region M3 between sparks 1b and 2b, which are sparks in two chronologically consecutive spark images, for spark b. In addition, there is an overlapping region M4 between sparks 2b and 3b, which are sparks in two chronologically consecutive spark images. In the two-dimensional graph G1 shown in FIG. 5, the overlapping regions M1 to M4 are hatched. The clustering unit 25 clusters the sparks 1a and 2a into the same cluster when there is an overlapping region M1 between the sparks 1a and 2a. Similarly, the clustering unit 25 clusters the sparks 2a and 3a into the same cluster when there is an overlapping region M2 between the sparks 2a and 3a. As a result, the clustering unit 25 can cluster the sparks 1a to 3a into the same cluster Ca, as shown in the three-dimensional plot map PL2. Similarly, the clustering unit 25 can cluster the sparks 1b to 3b into the same cluster Cb.

[0037] The clustering unit 25 may not necessarily use the overlapping region M to perform clustering, but may instead perform clustering based on the shapes of sparks in two chronologically consecutive spark images. For example, the clustering unit 25 may cluster two chronologically consecutive spark images into the same cluster if the image similarity of the shapes of the sparks in those images is equal to or greater than a certain level. The clustering unit 25 may also perform clustering based on the distance between the sparks, instead of whether or not there is an overlapping region between the sparks. For example, the clustering unit 25 may calculate the shortest distance between the sparks at each point on the time axis Axt, at the planar coordinate positions of the sparks represented by the first axis Ax1 and the second axis Ax2, i.e., the positions of the sparks on the two-dimensional graph G1, and classify the sparks into the same cluster if the shortest distance is equal to or less than a predetermined cluster threshold. This facilitates clustering of sparks using the distance between the planar coordinate positions of the sparks. The clustering process is similarly performed on the three-dimensional plot map PL2.

[0038] 3, the line detection unit 26 detects, as lines, trajectories of sparks classified into the same cluster across the multiple selected images in the clustering unit 25. More specifically, the line detection unit 26 detects first lines, which are lines detected in the multiple first selected images CH1 as the multiple selected images included in a first predetermined period, and second lines, which are detected in the multiple second selected images CH2 as the multiple selected images included in a second predetermined period that includes spark images captured after the multiple first selected images.

[0039] 7 is a diagram for explaining the first straight line. The straight line detector 26 detects the trajectories of sparks as straight lines for each of the clusters Ca and Cb clustered in the three-dimensional plot maps PL1 and PL2. In this embodiment, the straight line detector 26 determines a representative point Ra1 in spark 1a, a representative point Ra2 in spark 2a, and a representative point Ra3 in spark 3a. The straight line detector 26 also determines a representative point Rb1 in spark 1b, a representative point Rb2 in spark 2b, and a representative point Rb3 in spark 3b. Here, the representative points Ra1 to Ra3 are, for example, the central coordinates of the sparks 1a to 3a, respectively. The representative points Rb1 to Rb3 are the central coordinates of the sparks 1b to 3b, respectively. The representative points Ra1 to Ra3 and the representative points Rb1 to Rb3 are not limited to the central coordinates of the sparks 1a to 3a and the sparks 1b to 3b, respectively, and may be any points of the sparks 1a to 3a and 1b to 3b. The line detection unit 26 detects the line T1a in the cluster Ca by approximating the representative points Ra1 to Ra3 with a line. The line detection unit 26 also detects the line T1b in the cluster Cb by approximating the representative points Rb1 to Rb3 with a line. The line approximation method uses, for example, the least squares method. In this embodiment, the lines T1a and T1b are lines detected in the plurality of first selected images CH1 shown in FIG. 4. Therefore, in the following description, the lines T1a and T1b will be referred to as first lines T1a and T1b. When the first lines T1a and T1b are not particularly distinguished from each other, they will simply be referred to as the first line T1. The first line T1 has direction information. The first straight line T1 is represented by parameters such as the direction and inclination, for example.

[0040] FIG. 8 is a diagram for explaining the second line. FIG. 8 shows a state in which the line detector 26 has detected a line T2 in the three-dimensional plot map PL2 using a plurality of second selected images CH2. The line detector 26 detects the second line T2 detected in the plurality of second selected images CH2 in addition to the first line T1 detected in the plurality of first selected images CH1. The plurality of second selected images CH2 are a plurality of selected images included in a second predetermined period that includes spark images captured after the plurality of first selected images. In this embodiment, the plurality of second selected images CH2 are a plurality of consecutive spark images obtained by adding "1" to the frame numbers of the plurality of first selected images CH1. In the first processing routine, the plurality of first selected images CH1 are frames 1 to 3 shown in FIG. 4, i.e., spark images Im1 to Im3 corresponding to points 1 to 3 on the time axis Axt. Therefore, the plurality of second selected images CH2 in the first processing routine are frames 2 to 4, that is, spark images Im2 to Im4 corresponding to points 2 to 4 on the time axis Axt.

[0041] In the following description, the straight line T2 detected using the multiple second selected images CH2 will be referred to as the second straight line T2. The second straight line T2a shown in the three-dimensional plot map PL2 is obtained by approximating, with a straight line, the representative points R2a to R4a of the sparks 2a to 4a clustered into cluster Ca by the above-mentioned clustering process. Similarly, the second straight line T2b is obtained by approximating, with a straight line, the representative points R2b to R4b of the sparks 2b to 4b clustered into cluster Cb. Like the first straight line T1, the second straight line T2 has directional information. The second straight line T2 is represented by parameters such as direction and inclination, for example.

[0042] In step S7 shown in FIG. 3, the spark size determination unit 28 performs a spark size determination process to determine the size of the spark using the positional relationship between the line detected by the line detection unit 26 and the spark.

[0043] FIG. 9 is a diagram illustrating the spark size determination process. The three-dimensional plot map PL1 shown in FIG. 9 is a diagram illustrating only the spark image of spark a and the first line T1a in the three-dimensional plot map PL1 shown in FIG. 7. The two-dimensional graph G2 shown at the bottom of FIG. 9 is a diagram illustrating the spark 2a in the three-dimensional plot map PL1 as viewed from the time axis Axt side. In the three-dimensional plot map PL6 and the two-dimensional graph G2, the representative point R is hatched. FIG. 9 explains how the spark size determination unit 28 determines the size of the spark 2a. The spark size determination unit 28 determines the size of the spark 2a using the positional relationship between the first line T1a and the spark 2a. Note that the three-dimensional plot map PL1 is created based on the multiple first selected images CH1 shown in FIG. 4, but the frame focused on when selecting the multiple first selected images CH1 is "frame 2," as described above. Therefore, when determining the size of the spark 2a in the spark image Im2 corresponding to frame 2, the first straight line T1a detected from the plurality of first selected images CH1 is used.

[0044] As shown in the two-dimensional graph G2 in FIG. 9, in this embodiment, the spark 2a is composed of nine pixels, g3 to g11. The pixel g8 corresponds to the representative point R2a. In this embodiment, the spark size determination unit 28 determines the size of the spark by calculating the sum of squares error using the first line T1a and the spark 2a. The spark size determination unit 28 first calculates the distance between each of the pixels g3 to g11 and pixel g8, which is the representative point R2a. For example, the distance from pixel g3 to pixel g8 is distance d3 shown in the two-dimensional graph G2. Here, if the distance from an arbitrary pixel gn to pixel g8 is dn, the spark size determination unit 28 calculates the mean square error MSE using the following formula (1). In this embodiment, the spark size determination unit 28 regards the value of the mean square error MSE as the size of the spark.

number

[0045] The spark size determination unit 28 determines the size of the sparks included in the target frames of the plurality of first selected images and the target frames of the plurality of second selected images using the above-described method. Specifically, the spark size determination unit 28 determines whether the size of the spark included in the target frame is larger than a predetermined spark threshold. Note that the determination of the size of the spark is not limited to the calculation method using the above-described formula (1). For example, the spark size determination unit 28 may obtain the number of pixels constituting the spark in the spark image or the area of ​​the spark in the spark image as the size of the spark.

[0046] Next, in step S8 shown in FIG. 3 , the parent-child relationship determination unit 27 determines whether the spark images classified into the same cluster by the clustering unit 25 satisfy a parent-child condition in the multiple first selected images CH1 and the multiple second selected images CH2, thereby determining a parent-child relationship. Specifically, the parent-child relationship determination unit 27 calculates a relationship value indicating the positional relationship between a first line T1 and a second line T2 acquired based on the spark images classified into the same cluster as an index value, and determines that the multiple sparks have a parent-child relationship indicating a relationship resulting from the same spark if the relationship value satisfies a predetermined parent-child condition. In this embodiment, the parent-child condition is a condition in which the relationship value is equal to or less than a predetermined parent-child relationship threshold. The parent-child relationship determination unit 27 determines that the parent-child relationship exists if the relationship value indicating the positional relationship between the first line T1 and the second line T2 is equal to or less than a predetermined parent-child relationship threshold. For example, the relationship value may be an index that can compare the degree of similarity between lines, such as the Hausdorff distance.

[0047] In step S8, in a specific case where there is a specific size spark, the size of which is larger than a predetermined spark threshold, among the frames of interest of the multiple sparks that were the subject of judgment in step S7, the parent-child relationship judgment unit sets the parent-child relationship threshold to a value higher than in other cases where there is no specific size spark in the parent-child relationship judgment process in the same cluster as the specific size spark.

[0048] The parent-child relationship determination process of step S8 will be described with reference to Fig. 5. In the three-dimensional plot map PLA shown in Fig. 5, the first lines T1a and T1b are represented by solid lines, and the second lines T2a and T2b are represented by dashed lines. The parent-child relationship determination unit 27 determines whether the first line T1 and the second line T2 are in a parent-child relationship. In this embodiment, the "parent-child relationship" between the first line T1 and the second line T2 means that the first line T1 and the second line T2 are trajectories generated by the same spark.

[0049] In the three-dimensional plot map PLA shown in FIG. 5, the parent-child relationship determination unit 27 acquires a relationship value indicating the positional relationship between a first line T1a and a second line T2a generated based on sparks in the same cluster. The parent-child relationship determination unit 27 acquires a relationship value indicating the positional relationship between a first line T1b and a second line T2b generated based on sparks in another cluster. In this embodiment, the relationship values ​​indicating the positional relationship between the first line T1a and the second line T2a and between the first line T1b and the second line T2b indicate the distance between the first line T1a and the second line T2a and between the first line T1b and the second line T2b. Therefore, for example, the greater the difference in orientation between the first line T1a and the second line T2a, the greater the relationship value indicating the positional relationship between the first line T1b and the second line T2b. The parent-child relationship determination unit 27 determines that the first line T1a and the second line T2a have a parent-child relationship when a relationship value indicating the positional relationship between the first line T1a and the second line T2a is equal to or less than a predetermined parent-child relationship threshold. That is, the parent-child relationship determination unit 27 determines that the first line T1a and the second line T2a have a parent-child relationship when the orientation of the first line T1a and the orientation of the second line T2a are similar. Similarly, the parent-child relationship determination unit 27 determines that the first line T1b and the second line T2b have a parent-child relationship when a relationship value indicating the positional relationship between the first line T1b and the second line T2b is equal to or less than a predetermined parent-child relationship threshold. In the example shown in FIG. 5, the parent-child relationship threshold is set to a value before the condition for the parent-child condition to be met was relaxed.

[0050] In step S9 of Fig. 3, the feature extraction device 14 determines whether steps S4 to S8 have been executed for the plurality of spark images to be processed for feature extraction processing out of all spark images acquired in step S4. If the determination in step S9 is "Yes," the process of step S10 is executed. On the other hand, if the determination in step S9 is "No," the feature extraction device 14 executes the processes of steps S4 to S8 again. In steps S4 to S8 in the next routine, the selected images to be processed are those obtained by adding "1" to the frame numbers of the plurality of first selected images and the plurality of second selected images in the previous routine.

[0051] Fig. 10 is a diagram illustrating one routine of steps S4 to S8 that are repeated. In Fig. 10, the plurality of first selected images CH1 are made up of spark images ImtN-2 to ImtN corresponding to frame numbers tN-2 to tN. Also, in Fig. 10, the plurality of second selected images CH2 are made up of spark images Imt-1 to Imt+1 corresponding to frame numbers tN-1 to tN+1. In Fig. 10, the frame of interest among the plurality of first selected images CH1 is the frame with frame number tN-1, which is located in the center on the time axis Axt. Also, in Fig. 10, the frame of interest among the plurality of second selected images CH2 is the frame with frame number tN, which is located in the center on the time axis Axt.

[0052] The three-dimensional plot map PL7 shown on the right side of Fig. 10 is created using the plurality of spark images ImtN-3 to tN+2 shown on the left side of Fig. 10 and the frame numbers of each of the plurality of spark images ImtN-3 to tN+2. The three-dimensional plot map PL7 may be created by combining each of the three-dimensional plot maps for a predetermined period created by the three-dimensional plotting unit 24. In the three-dimensional plot map PL7, in order to distinguish between the same spark in different frames, the spark c in the spark image ImtN-3 is written as spark 1c, the spark c in the spark image ImtN-2 is written as spark 2c, the spark c in the spark image ImtN-1 is written as spark 3c, and the spark c in the spark image ImtN is written as spark 4c. Similarly, for spark d, spark d in spark image ImtN+1 will be described as spark 1d, and spark d in spark image ImtN+2 will be described as spark 2d. Similarly, for spark e, spark e in spark image ImtN+1 will be described as spark 1e, and spark e in spark image ImtN+2 will be described as spark 2e. Note that in the three-dimensional plot map PL7, the sizes S1 to S8 of each spark determined by the spark size determination unit 28 are described.

[0053] In step S7 of Fig. 3, it is assumed that the spark size determination unit 28 determines that the size S4 of the spark 4c in the spark image ImtN, which is the frame of interest among the plurality of second selected images CH2 shown in Fig. 10, is larger than the spark threshold (specific case). Also in Fig. 10, an explosion D1 has occurred in the spark c in the spark image ImtN of frame number tN, and the two sparks d and e in the spark image ImtN+1 of frame number tN+1 have been created by the splitting of the spark c.

[0054] 3 is executed for the plurality of selected images CH1 and CH2 in Fig. 10, the parent-child relationship determination unit 27 determines the parent-child relationship by relaxing the conditions for the parent-child condition to be met. Specifically, the parent-child relationship determination unit 27 relaxes the conditions for the parent-child condition to be met by increasing the parent-child relationship threshold.

[0055] FIG. 11 is a diagram illustrating the parent-child relationship determination process. In step S5 of FIG. 3, the clustering unit 25 clusters the sparks 2c to 4c, 1d, and 1e into the same cluster Cc. The three-dimensional plot maps PL8 and PL9 shown in FIG. 11 show a first line T1 and a second line T2 before and after the explosion D1. The first line Tc1 shown in the three-dimensional plot maps PL8 and PL9 is a line detected by the line detection unit 26 when the plurality of spark images ImtN-2 to ImtN shown in FIG. 10 are set as the first selected image CH1. The second lines Td2 and Te2 shown in the three-dimensional plot maps PL8 and PL9 are lines detected by the line detection unit 26 when the plurality of spark images ImtN-1 to ImtN+1 shown in FIG. 10 are set as the second selected image CH2. The first line T1c is detected by approximating representative points (not shown) of each of the sparks 2c to 4c with a straight line. The second line Td2 is detected by approximating representative points (not shown) of each of the sparks 3c to 4c and 1d with a straight line. The second line Te2 is detected by approximating representative points (not shown) of each of the sparks 3c to 4c and 1e with a straight line. In the three-dimensional plot maps PL8 and PL9, the first line Tc1 is represented by a solid line, and the second lines Td2 and Te2 are represented by dashed lines.

[0056] As shown in the spark images ImtN-3 to ImtN+1 in FIG. 10, the trajectories of the sparks 1d and 1e generated by the explosion D1 of the spark c are significantly different from the trajectory of the spark c until the spark c explodes. Therefore, the positional relationship between the first line Tc1 and the second line Td2 shown in the three-dimensional plot maps PL8 and PL9 in FIG. 11 is also significantly different. Similarly, the positional relationship between the first line Tc1 and the second line Te2 is also significantly different. Therefore, when the parent-child relationship threshold before the above-described parent-child condition establishment condition is relaxed is used, the parent-child relationship determination unit 27 determines that the first line Tc1 and the second line Td2 do not have a parent-child relationship, as shown in the three-dimensional plot map PL8 in FIG. 11. The parent-child relationship determination unit 27 also determines that the first line Tc1 and the second line Te2 do not have a parent-child relationship. In other words, the parent-child relationship determination unit 27 determines that sparks 1d, 1e, and 4c are not in a parent-child relationship, and therefore cannot determine that sparks 1d and 1e were generated from spark 4c. In the three-dimensional plot map PL8, to indicate that sparks 1d and 1e are recognized as sparks different from spark 4c, the spark plots for sparks 1d and 1e are shown in white, and the spark plot for spark 4c is shown in black.

[0057] In this embodiment, the parent-child relationship determination unit 27 increases the parent-child relationship threshold in certain cases as described above. This allows the parent-child relationship determination unit 27 to determine that the sparks that make up the first line Tc1 and the sparks that make up the second line Td2 have a parent-child relationship, as shown in the three-dimensional plot map PL9 in FIG. 11 . The parent-child relationship determination unit 27 can also determine that the sparks that make up the first line Tc1 and the sparks that make up the second line Te2 have a parent-child relationship. From the above, the parent-child relationship determination unit 27 can determine that the sparks 1d and 1e, which are located at the same point on the time axis Axt, were generated from the spark 4c.

[0058] 3 have been performed on a plurality of spark images that are the target of the feature extraction process, the rupture identification unit 29 executes the rupture identification process in step S10. In the example shown in the three-dimensional plot map PL9 in Fig. 11, the rupture identification unit 29 identifies that a rupture has occurred in the specific spark 4c because the parent-child relationship determination unit 29 has determined that there is a parent-child relationship between a specific spark 4c, which is one spark at a first point corresponding to frame number tN, and two sparks 1d and 1e at a second point corresponding to frame number tN+1, which is a point later on the time axis Axt than the first point.

[0059] Next, in step S11 of Fig. 3, the parent-child relationship determination unit 27 executes feature amount processing to extract features of the spark and store them in the auxiliary storage device 18. The three-dimensional graph G5 shown in Fig. 12 is created using the processing results from steps S1 to S10.

[0060] FIG. 12 is a diagram for explaining step S11. The parent-child relationship determination unit 27 acquires spark streamlines that constitute the feature quantities of sparks by connecting representative points of sparks that have been determined to have a parent-child relationship in multiple spark images. The parent-child relationship determination unit 27 also acquires the number of specific sparks identified by the explosion identification unit 29 as the number of explosions. The number of explosions constitutes the feature quantities of sparks. In the example shown in FIG. 12, the number of explosions is "1." The parent-child relationship determination unit 27 also determines the number of sparks that constitute the feature quantities of sparks using the processing results of steps S5 to S8 in FIG. 3. For example, the parent-child relationship determination unit 27 may determine the number of sparks by counting one parent-child relationship as one spark. In the example shown in FIG. 12, the parent-child relationship determination unit 27 determines the number of sparks to be "1."

[0061] In step S12 of FIG. 3, the tool estimation unit 30 estimates the state of the tool using the feature amounts of the sparks. The number of sparks and the number of explosions described above can be used to estimate the state of the grinding wheel as tool B shown in FIG. 1. For example, it is estimated that an increase in the number of sparks per hour is related to the state of tool B falling off or being worn out. It is also estimated that an increase in the number of spark explosions is related to the state of tool B falling off or being worn out. Therefore, the tool estimation unit 30 can estimate the state of tool B by using the feature amounts of the sparks.

[0062] In step S13 of FIG. 3, the processing object estimation unit 31 estimates the state of the processing object using the feature amounts of the sparks. The number of sparks and the number of explosions described above can be used to estimate the state of the processing object as steel material A shown in FIG. 1. For example, an increase in the number of sparks per hour is presumed to be related to the deterioration of steel material A. Also, an increase in the number of spark explosions is presumed to be related to the deterioration of steel material A. Note that the deterioration of steel material A is caused, for example, by the inside of steel material A becoming hot. As a result, the processing object estimation unit 31 can estimate the state of steel material A by using the feature amounts of the sparks.

[0063] In step S14 of FIG. 3 , the condition adjustment unit 32 adjusts at least one of the replacement time of the tool B and the machining conditions of the tool B using the state of the tool B estimated by the tool estimation unit 30 or the state of the steel material A, which is the machining target, estimated by the workpiece estimation unit 31. The condition adjustment unit 32 may advance the replacement time of the tool B when the tool estimation unit 30 estimates the wear state of the tool B or when the workpiece estimation unit 31 estimates the deterioration of the steel material A. The condition adjustment unit 32 may suppress tool deterioration by changing the machining conditions without changing the replacement time of the tool B when the tool estimation unit 30 estimates the wear state of the tool B or when the workpiece estimation unit 31 estimates the deterioration of the steel material A. Note that changing the machining conditions means, for example, automatically stopping the tool B after it has been operating for a certain period of time. The machining conditions can also be changed in response to a user's request, such as a desire to use the tool B for a longer period of time. This embodiment allows the tool B to be used in an optimal state.

[0064] According to the first embodiment described above, based on the finding by the inventors that the size of a spark increases immediately before the spark explodes, the feature extraction device relaxes the condition for the parent-child condition to be met when the size of the spark is greater than the spark threshold, thereby making it possible to easily determine the parent-child relationship of sparks in time-series spark images. Furthermore, according to the first embodiment described above, it is possible to easily identify that a spark has exploded using the determination result of the parent-child relationship determination unit, making it possible to easily distinguish between multiple sparks that have exploded from the same spark and other sparks in time-series spark images acquired by the imaging device 12.

[0065] Furthermore, according to the first embodiment, the parent-child relationship determination unit 27 determines whether or not a plurality of sparks classified into the same cluster by the clustering unit 25 satisfy the parent-child condition. This reduces the possibility that a plurality of sparks classified into different clusters will be determined to satisfy the parent-child condition.

[0066] B. Other Embodiments: (B1) The processing of steps S6 to S8 in Fig. 3 is not limited to the above. That is, in the above embodiment, the parent-child relationship determination process determines the parent-child relationship of multiple sparks based on the positional relationship between the first line T1 and the second line T2 generated in the same cluster as shown in Figs. 7, 8, and 11, but is not limited to this. For example, the parent-child relationship determination unit 27 may predict the future position of a spark from the position of the spark, which is an object in a past frame, and determine the parent-child relationship based on the positional relationship between the predicted position and the actual position. Below, specific examples of the parent-child relationship determination process in other embodiments will be described.

[0067] FIG. 13 is a first diagram illustrating a parent-child relationship determination process in another embodiment. The parent-child relationship determination unit 27 predicts the position of a spark at a future point in time (a later point on the time axis Axt) corresponding to frame number tN+1, based on sparks 1c, 1d, 2c, and 2d of spark images corresponding to frame numbers tN-1 and tN, which are multiple past frames, in the 3D plot map. Specifically, the parent-child relationship determination unit 27 predicts the position of the spark at a point in time corresponding to frame number tN+1, assuming that sparks 1c and 2c, which are in the same cluster in past frames, are moving at a constant velocity in a straight line. The spark predicted using sparks 1c and 2c is designated as predicted spark 3cr. Furthermore, the parent-child relationship determination unit 27 predicts the position of the spark at a point in time corresponding to frame number tN+1, assuming that sparks 1d and 2d, which are in the same cluster in past frames, are moving at a constant velocity in a straight line. The spark predicted using sparks 1d and 2d is designated as predicted spark 3dr. When predicting the position of a spark, the position of the representative point of each spark at the time point corresponding to frame number tN+1 may be predicted using the positions of the representative points of each spark in past frames.

[0068] The parent-child relationship determination unit 27 acquires the positions (e.g., representative points) of the actual sparks 3c and 3d from the actual spark image at the time of frame number tN+1. Then, the parent-child relationship determination unit 27 calculates the distance between the positions of the actual sparks 3c and 3d and the positions of the predicted sparks 3cr and 3dr as an index value, and if the index value satisfies a parent-child condition, determines that the multiple sparks from which the index value was calculated have a parent-child relationship resulting from the same spark. In another embodiment, the parent-child condition is that the index value is equal to or less than a distance threshold. In FIG. 13, the parent-child relationship determination unit 27 determines that sparks 1c, 2c, and 3c have a parent-child relationship, and sparks 1d, 2d, and 3d have a parent-child relationship. Note that the parent-child relationship determination unit 27 may determine that sparks 3c and 3d are newly generated sparks if the distance is not equal to or less than the distance threshold.

[0069] 14 is a second diagram illustrating the parent-child relationship determination process in another embodiment. In a specific case where the size of at least one of the sparks 1c and 2c in past frames used to predict a spark is larger than the spark threshold, the parent-child relationship determination unit 27 relaxes the condition for the parent-child condition to be met by setting a larger distance threshold than in other cases. The parent-child relationship determination unit 27 determines that, at the time point corresponding to frame number tN+1, the index value representing the distance between the position of the predicted spark 3cr and the actual spark 3c is equal to or smaller than the distance threshold, and the index value representing the distance between the position of the predicted spark 3cr and the actual spark 3d is equal to or smaller than the distance threshold. As a result, the explosion identification unit 29 identifies that an explosion has occurred in the spark 2c at the time point of frame number tN.

[0070] (B2) In the first embodiment described above, the feature extraction device 14 includes the brightness conversion processing unit 22, the binarization processing unit 23, the tool estimation unit 30, the machining object estimation unit 31, and the condition adjustment unit 32. In contrast, the feature extraction device 14 does not necessarily include the brightness conversion processing unit 22, the binarization processing unit 23, the tool estimation unit 30, the machining object estimation unit 31, and the condition adjustment unit 32.

[0071] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features of the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0072] 10...feature extraction system, 12...imaging device, 14...feature extraction device, 15...CPU, 17...ROM, 18...auxiliary storage device, 19...RAM, 21...image capture processing unit, 22...luminance conversion processing unit, 23...binarization processing unit, 24...3D plotting unit, 25...clustering unit, 26...straight line detection unit, 27...parent-child relationship determination unit, 28...spark size determination unit, 29...rupture identification unit, 30...tool estimation unit, 31...machining object estimation unit, 32...condition adjustment unit

Claims

1. A feature extraction device that extracts feature quantities of sparks generated by friction, an image acquisition unit that acquires a plurality of spark images obtained by capturing images of the sparks in time series using an imaging device; a three-dimensional plotting unit that selects a plurality of images that are consecutive in time series from the plurality of spark images acquired by the image acquiring unit as a plurality of selected images, and creates a three-dimensional plot map for the plurality of selected images, which is a map of three axes: a first axis corresponding to a horizontal direction of the plurality of selected images, a second axis corresponding to a vertical direction of the plurality of selected images, and a time axis; a spark size determination unit that determines the size of the spark; a parent-child relationship determination unit that calculates an index value relating to the position of the spark on the three-dimensional plot map, and when the calculated index value satisfies a predetermined parent-child condition, determines that the plurality of sparks that are the source of calculation of the index value are in a parent-child relationship that indicates that they are caused by the same spark, and that this parent-child relationship constitutes the feature amount; The parent-child relationship determination unit relaxes the condition for the parent-child condition to be satisfied in a specific case where the spark size determination unit determines that the size of the spark is greater than a predetermined spark threshold.

2. 2. The feature extraction device according to claim 1, further comprising: A feature extraction device having an explosion identification unit that identifies that an explosion has occurred in a specific spark when the parent-child relationship determination unit determines that there is a parent-child relationship between a specific spark, which is one of the sparks at a first point on the time axis of the three-dimensional plot map, and each of the multiple sparks at a second point that is a point later on the time axis than the first point.

3. 2. The feature extraction device according to claim 1, the parent-child relationship determination unit calculates the index value relating to the position of the spark using plots of two or more consecutive sparks on the time axis; The parent-child condition is a condition related to the position of the spark.

4. 2. The feature extraction device according to claim 1, further comprising: a clustering unit that performs clustering to distinguish the sparks using positions of the sparks in the three-dimensional plot map; The parent-child relationship determination unit determines whether the plurality of sparks classified into the same cluster by the clustering unit satisfy the parent-child condition.

5. 5. The feature extraction device according to claim 4, The clustering unit classifies the plurality of sparks represented by the first axis and the second axis at each point on the time axis into the same cluster if there is an overlapping area between the plurality of sparks.

6. 5. The feature extraction device according to claim 4, The clustering unit classifies the plurality of sparks into the same cluster when the distance between the planar coordinate positions of the plurality of sparks represented by the first axis and the second axis at each point on the time axis is equal to or less than a predetermined cluster threshold.

7. 5. The feature extraction device according to claim 4, further comprising: a straight line detection unit configured to detect trajectories of the sparks classified into the same cluster across the plurality of selected images as straight lines, a first straight line that is a straight line detected in a plurality of first selected images that are the plurality of selected images included in a first predetermined period; a line detection unit that detects a second line detected in a plurality of second selected images as the plurality of selected images included in a second predetermined period that includes the spark image captured after the plurality of first selected images, the parent-child relationship determination unit calculates, as the index value, a relationship value indicating a positional relationship between the first line and the second line; the parent-child condition is a condition that the relationship value is equal to or less than a predetermined parent-child relationship threshold value, The parent-child relationship determination unit relaxes the condition for the parent-child condition to be met by setting the parent-child relationship threshold higher in the specific case than in other cases.

8. 2. The feature extraction device according to claim 1, Further, a binarization processing unit is provided, The binarization processing unit binarizes the plurality of spark images acquired by the image acquisition unit using a predetermined binarization threshold.

9. 9. The feature extraction device according to claim 1, Further, a tool estimation unit is provided, The tool estimation unit estimates the state of a tool using at least the determination result of the parent-child relationship determination unit.

10. 10. The feature extraction device according to claim 9, Further, a condition adjusting unit is provided, The condition adjustment unit adjusts at least one of a replacement timing of the tool and a machining condition of the tool using the state of the tool estimated by the tool estimation unit.

11. 9. The feature extraction device according to claim 1, Further, a processing object estimation unit is provided, The processing object estimation unit estimates a state of the processing object using the feature amount of the sparks.

12. The feature extraction device according to claim 11, Further, a condition adjusting unit is provided, The condition adjustment unit adjusts at least one of a replacement timing of the tool and a machining condition of the tool, using the state of the workpiece estimated by the machining object estimation unit.

13. A feature extraction method for extracting feature amounts of sparks generated by friction, comprising: an image acquisition step of acquiring a plurality of spark images obtained by capturing images of the sparks in time series using an imaging device; a three-dimensional plotting step of selecting a plurality of images that are consecutive in time series from the plurality of spark images acquired by the image acquiring step as a plurality of selected images, and creating a three-dimensional plot map for the plurality of selected images, the three-axis map being a first axis corresponding to a horizontal direction of the plurality of selected images, a second axis corresponding to a vertical direction of the plurality of selected images, and a time axis; a spark size determination step of determining the size of the spark on the three-dimensional plot map; a parent-child relationship determination step of calculating an index value relating to the position of the spark, and determining, when the calculated index value satisfies a predetermined parent-child condition, that the plurality of sparks plotted at different points on the time axis have a parent-child relationship that constitutes the feature amount indicating that the sparks were generated from the same spark, The feature extraction method, wherein the parent-child relationship determination step includes a step of relaxing the condition for the parent-child condition to be satisfied in a specific case where the spark size determination step determines that the size of the spark is larger than a predetermined spark size.

Images