Determination device, learning device, determination method, and determination program

Abstract

To provide a determination device, a learning device, a determination method, and a determination program for enhancing determination accuracy in object determination processing.SOLUTION: In a determination system, a determination device includes an acquisition unit 111, a storage unit 15, and a determination unit 113. The acquisition unit 111 acquires first data and second data. The first data has a plurality of pieces of height information indicating respective heights of a plurality of points of an object. The second data has complementary position information indicating a position of complementary height information generated based on other height information among the pieces of height information. The storage unit 15 stores a trained identification model M1. The trained identification model M1 outputs identification data for identifying a category of the object when a data set including the first data and the second data is inputted. The determination unit 113 determines whether the object is normal or abnormal based on the identification data obtained by inputting the data set into the trained identification model M1.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a determination device, a learning device, a determination method, and a determination program. [Background technology]

[0002] The image processing device of Patent Document 1 performs calculations for three-dimensional measurement using a first image and a second image based on images taken by a first camera and a second camera, which are positioned to photograph an object from different directions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-191312 Summary of the Invention [Problem to be solved by the invention]

[0004] The image that has undergone arithmetic processing for three-dimensional measurement is used, for example, to determine whether an object is good or bad. However, depending on the angles at which the first and second cameras capture the object, it may not be possible to perform arithmetic processing for three-dimensional measurement, which reduces the accuracy of determining whether the object is good or bad.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide a determination device, a learning device, a determination method, and a determination program that can improve the determination accuracy in the object determination process. [Means for solving the problem]

[0006] An exemplary determination device of the present disclosure includes an acquisition unit, a storage unit, and a determination unit. The acquisition unit acquires first data and second data. The first data includes multiple pieces of height information indicating the heights of multiple locations on an object. The second data includes complementary position information indicating the position of complementary height information generated based on other height information among the height information. The storage unit stores a trained discrimination model. When a dataset including the first data and the second data is input, the trained discrimination model outputs discrimination data that identifies a category to which the object belongs. The determination unit inputs the dataset to the trained discrimination model and determines whether the object is normal or abnormal based on the discrimination data output by the trained discrimination model.

[0007] An exemplary learning device of the present disclosure includes an acquisition unit and a discriminative model generation unit. The acquisition unit acquires a training dataset including first training data, second training data, and training discrimination data. The first training data includes a plurality of pieces of height information indicating the heights of a plurality of locations on an object. The second training data includes complementary position information indicating the position of complementary height information generated based on other height information among the height information. The training discrimination data indicates a category to which the object belongs. The discriminative model generation unit generates a trained discriminative model by performing machine learning on the first training data, the second training data, and the training classification data. The trained discriminative model outputs discrimination data that identifies the category to which the object belongs.

[0008] An exemplary determination method of the present disclosure includes the steps of acquiring first data having multiple pieces of height information each indicating the height of multiple locations on an object, and second data having complementary position information indicating the position of complementary height information generated based on other height information among the height information; inputting a dataset including the first data and the second data into a trained discrimination model that outputs discrimination data that identifies a category to which the object belongs; and determining whether the object is normal or abnormal based on the discrimination data output by the trained discrimination model.

[0009] An exemplary determination program of the present disclosure causes a computer to execute the steps of acquiring first data having multiple pieces of height information each indicating the height of multiple locations on an object, and second data having complementary position information indicating the position of complementary height information generated based on other height information among the height information; inputting a dataset including the first data and the second data into a trained identification model that outputs identification data that identifies the category to which the object belongs; and determining whether the object is normal or abnormal based on the identification data output by the trained identification model. [Effects of the Invention]

[0010] According to the exemplary embodiment of the present invention, it is possible to improve the accuracy of determination in the process of determining an object. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a determination system having a determination device according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of the determination device according to this embodiment. [Figure 3] FIG. 3 is a diagram showing image data after image processing. [Figure 4] FIG. 4 is a diagram showing another example of image data used in the determination process by the determination device. [Figure 5] FIG. 5 is a diagram showing image data after image processing. [Figure 6] FIG. 6 is a schematic diagram showing the complementation process. [Figure 7] FIG. 7 is a diagram schematically illustrating the determination process performed by the determination device. [Figure 8] FIG. 8 is a flowchart showing the determination method according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals, and description thereof will not be repeated.

[0013] First, the configurations of a determination device and a determination system according to this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing a determination system having a determination device according to this embodiment. Figure 2 is a block diagram showing the configuration of the determination device according to this embodiment.

[0014] As shown in FIG. 1, the determination system 100 includes a determination device 1, an imaging device C1, and an imaging device C2. The determination device 1 is, for example, a notebook personal computer (PC). The determination device 1 may be a terminal used by a user, such as a notebook PC, a desktop PC, a tablet terminal, or a smartphone. The imaging devices C1 and C2 are, for example, cameras. The imaging devices C11 and C2 capture images of the same object T1 and generate image data representing the captured image or video data representing the captured video, respectively.

[0015] The determination device 1 and the imaging devices C1 and C2 can communicate with each other via a network (not shown). The network may be, for example, a local area network (LAN), a wireless LAN, a mobile phone communication network, infrared communication, Bluetooth (registered trademark), etc. The imaging devices C1 and C2 transmit the generated image data or video data to the determination device 1 via the network. In the example shown in FIG. 1, the imaging devices C1 and C2 generate image data.

[0016] The determination device 1 performs a determination process to determine whether an object T1 indicated by image data or video data transmitted from the imaging devices C1 and C2 is normal or abnormal.

[0017] 2, the determination device 1 includes a control unit 11, a display unit 12, an operation unit 13, a communication unit 14, and a storage unit 15. The control unit 11 includes a processor such as a CPU (Central Processing Unit). The control unit 11 controls the display unit 12, the operation unit 13, the communication unit 14, and the storage unit 15.

[0018] The display unit 12 includes, for example, a liquid crystal display, an organic electroluminescence display, etc. The display unit 12 displays various screens.

[0019] The operation unit 13 includes a keyboard, a mouse, a trackpad, etc. The operation unit 13 accepts operations from the user.

[0020] The communication unit 14 communicates with devices external to the determination device 1 via a network. Examples of networks include the Internet, a local area network (LAN), and a public telephone network. Specifically, the communication unit 14 includes, for example, a network interface controller (NIC) that communicates according to a predetermined communication protocol, or a communication device conforming to a WiFi communication standard. The predetermined communication protocol is, for example, the TCP / IP (Transmission Control Protocol / Internet Protocol) protocol suite (i.e., the Internet Protocol Suite). The communication unit 14 may also be a wireless communication module that performs wireless communication. Specifically, the wireless communication module performs, for example, short-range wireless communication. Short-range wireless communication is, for example, wireless communication with a communication distance of several meters to several tens of meters. Short-range wireless communication is, for example, infrared communication, communication conforming to a communication standard such as Bluetooth (registered trademark), or ZigBee (registered trademark). Communication conforming to a Bluetooth (registered trademark) communication standard is, for example, Bluetooth (registered trademark) Low Energy (BLE).

[0021] The storage unit 15 includes a storage device such as a semiconductor memory and a hard disk drive (HDD), etc. The storage unit 15 stores data, computer programs, etc. The computer programs include, for example, a determination program.

[0022] The control unit 11 executes a judgment program stored in the storage unit 15, thereby functioning as an acquisition unit 111, an image processing unit 112, and a judgment unit 113. The acquisition unit 111 acquires data used in the judgment process. The image processing unit 112 performs image processing on image data or video data. The judgment unit 113 judges whether the object T1 indicated by the image data or video data is normal or abnormal.

[0023] Next, image data used in the determination process by the determination device 1 will be described with reference to Figures 1 and 2. For example, the imaging device C1 captures an image of the object T1 to generate image data GD1. The image data GD1 is data representing the captured image G1 of the object T1. The imaging device C2 captures an image of the object T1 to generate image data GD2. The image data GD2 is data representing the captured image G2 of the object T1.

[0024] The imaging device C1 transmits image data GD1 via the network to the determination device 1. The imaging device C2 transmits image data GD2 via the network to the determination device 1. The communication unit 14 of the determination device 1 receives the image data GD1 and GD2 transmitted from the imaging devices C1 and C2, respectively, via the network.

[0025] The image processing unit 112 acquires the image data GD1 and GD2 received by the communication unit 14 and performs image processing on the image data GD1 and GD2. As shown in Fig. 2, the image processing unit 112 has a first data generation unit 121. For example, based on the image data GD1 and image data GD2 obtained by capturing images of the object T1 from two angles, the first data generation unit 121 calculates height information for each pixel included in the image data GD1 and image data GD2, which indicates the height of the object represented by the pixel from a reference position.

[0026] For example, the first data generation unit 121 compares the image data GD1 and GD2, and for each pixel included in the image data GD1, identifies the corresponding pixel in the image data GD2. For example, for each pixel included in the image data GD1, the first data generation unit 121 compares the parameters of the pixel, such as color information, with the parameters of each pixel included in the image data GD2.

[0027] Specifically, the first data generation unit 121 extracts one comparison source pixel P1 from among the pixels included in the image data GD1, and identifies a pixel P2 in the image data GD2 that has parameters corresponding to the parameters of the pixel P1.

[0028] Alternatively, the first data generation unit 121 calculates relative parameters of pixel P1 relative to the surrounding pixels based on the extracted parameters of pixel P1 and parameters of one or more pixels surrounding pixel P1. Similar to the image data GD1, the first data generation unit 121 calculates relative parameters for each pixel of image data GD2. The first data generation unit 121 compares the relative parameters of pixel P1 with the relative parameters of each pixel included in the image data GD2. For example, the first data generation unit 121 identifies a pixel having a relative parameter substantially the same as the relative parameter of pixel P1 as pixel P2.

[0029] When the first data generation unit 121 identifies the pixel P2, it calculates the parallax d indicating the distance between the pixel P1 and the pixel P2 based on the pixel coordinates indicating the coordinates of the pixel P1 and the pixel coordinates indicating the coordinates of the pixel P2.

[0030] The first data generation unit 121 calculates the distance L from the image capturing devices C1 and C2 to the objects indicated by the comparison source pixel and comparison target pixel based on the calculated parallax d, the focal length f of the image capturing devices C1 and C2, and the base length b. The base length b indicates the distance between the image capturing devices C1 and C2 at the position of the focal length f.

[0031] Specifically, the first data generation unit 121 calculates the distance L using the following equation (1): Note that the focal length f and the base length b are stored in the storage unit 15 as parameters indicating the characteristics of the image capture devices C1 and C2, for example.

[0032] Distance L=(focal length f×baseline length b) / disparity d (1)

[0033] The first data generation unit 121 includes the calculated distance L in the parameters of pixel P1. The first data generation unit 121 performs the same process on pixels other than pixel P1 included in the image data GD1 to generate image data GD12.

[0034] Next, the image data GD12 after image processing by the first data generation unit 121 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the image data GD12 after image processing. For ease of explanation, the image data GD12 shown in Fig. 3 is image data representing an image of 16 pixels, 4 pixels vertically by 4 pixels horizontally.

[0035] In the image data GD12, the position of each pixel is expressed in pixel coordinates. For example, the pixel coordinates are expressed as coordinates with the upper left pixel in the image data GD12 as the origin (0,0), with the right direction being the positive direction of the X axis and the downward direction being the positive direction of the Y axis. In other words, the pixel coordinates of the upper right pixel in the image data GD12 are expressed as (3,0). The pixel coordinates of the lower left pixel in the image data GD12 are expressed as (0,3). The pixel coordinates of the lower right pixel in the image data GD12 are expressed as (3,3).

[0036] For example, the first data generation unit 121 converts the distance L of each pixel included in the image data GD12 into a plurality of levels of brightness K corresponding to the distance L. In the example shown in FIG. 3, the first data generation unit 121 converts each pixel into five levels of brightness K, from "1" to "5," according to the distance L. The converted brightness K is an example of height information. In this embodiment, the closer the brightness K is to "1," the greater the distance L, and the closer the brightness K is to "5," the smaller the distance L. In other words, the closer the brightness K is to "1," the lower the height from the reference position, and the closer the brightness K is to "5," the higher the height from the reference position. Therefore, the image data GD12 has a plurality of height information pieces that respectively indicate the heights of a plurality of locations on the object T1. The image data GD12 having a plurality of height information pieces is an example of first data.

[0037] Next, another example of image data used in the determination process by the determination device 1 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram showing another example of image data used in the determination process by the determination device 1. Fig. 5 is a diagram showing image data GD34 after image processing. As with the image data GD12, the position of each pixel in the image data GD34 is expressed by pixel coordinates.

[0038] The determination system 100 shown in Fig. 4 is the same as the determination system 100 shown in Fig. 1, except that the object imaged by the imaging devices C1 and C2 is object T2 instead of object T1. Compared to object T1, object T2 has a region whose height from a reference position is higher than that of object T1, and the slope of the surface is steeper than that of object T1.

[0039] In the determination system 100, the imaging device C1 captures an image of the object T2 and generates image data GD3. The image data GD3 is data representing a captured image G3 of the object T2. The imaging device C2 captures an image of the object T2 and generates image data GD4. The image data GD4 is data representing a captured image G4 of the object T2.

[0040] The imaging device C1 transmits image data GD3 via the network to the determination device 1. The imaging device C2 transmits image data GD4 via the network to the determination device 1. The communication unit 14 of the determination device 1 receives the image data GD3 and GD4 transmitted from the imaging devices C1 and C2, respectively, via the network.

[0041] The first data generation unit 121 acquires the image data GD3 and GD4 received by the communication unit 14, and performs the above-described image processing on the image data GD3 and GD4 to generate image data GD34.

[0042] For example, the first data generation unit 121 extracts one comparison source pixel P3 from among the pixels included in the image data GD3. The first data generation unit 121 identifies a pixel P4 in the image data GD4 that has parameters corresponding to the parameters of the pixel P3.

[0043] However, if the distance between the object T2 and the imaging devices C1 and C2 is closer than a predetermined distance, or if the object T2 has protrusions or holes on its surface, the first data generation unit 121 may not be able to identify the parameters corresponding to the parameters of pixel P3, and may not be able to identify pixel P4.

[0044] If pixel P4 cannot be identified, the first data generation unit 121 cannot calculate the disparity d and the distance L. Therefore, the first data generation unit 121 cannot convert the distance L to the luminance K. In this case, the first data generation unit 121 identifies, among the multiple pixels included in the image data GD34, pixels for which the distance L cannot be calculated as pixels to be complemented. In other words, the pixels to be complemented are pixels for which the luminance K has disappeared.

[0045] 5, the first data generation unit 121 sets the luminance K of the pixel to be complemented to "0." In the example shown in Fig. 5, the first data generation unit 121 sets the luminance K of the pixel whose pixel coordinates are (1,0), the pixel whose pixel coordinates are (1,1), the pixel whose pixel coordinates are (1,2), the pixel whose pixel coordinates are (1,3), the pixel whose pixel coordinates are (3,0), the pixel whose pixel coordinates are (3,1), the pixel whose pixel coordinates are (3,2), and the pixel whose pixel coordinates are (3,3) to "0."

[0046] In the determination process, if image data including a pixel to be complemented is used, an accurate determination result cannot be obtained, so the first data generation unit 121 performs a complementation process in which the luminance K of the pixel to be complemented is set to any one of "1" to "5." The luminance K set by the complementation process is an example of complemented height information. In other words, the first data generation unit 121 generates complemented height information.

[0047] Next, the details of the interpolation process will be described with reference to Fig. 2 and Fig. 6. Fig. 6 is a schematic diagram showing the interpolation process. Fig. 6 shows a pixel PA to be interpolated and pixels located around the pixel PA to be interpolated. The brightness K of the pixels located around the pixel PA to be interpolated is one of values ​​"1" to "5" depending on the distance L.

[0048] The first data generating unit 121 generates a luminance K, which is interpolated height information, based on height information of one or more pixels surrounding the interpolation target pixel PA.

[0049] For example, the first data generation unit 121 calculates the gradient of the object between the plurality of pixels based on the luminance K of the plurality of pixels surrounding the pixel PA to be complemented. Therefore, the height information can be complemented in a simple manner, and gradient information to be used as a label in the determination process can be obtained.

[0050] Specifically, the first data generation unit 121 approximates the relationship between the luminance K of pixels PB1 to PB4 adjacent to the pixel to be complemented PA, the luminance K of pixels PC1 to PC8 located two pixels away from the pixel to be complemented PA, and the luminance K of pixels PD1 to PD12 located three pixels away from the pixel to be complemented PA, and the distance from the pixel to be complemented PA, using a linear function.

[0051] For example, the first data generation unit 121 calculates the average value or median value of the luminance K of pixels PB1 to PB4, the average value or median value of the luminance K of pixels PC1 to PC8, and the average value or median value of the luminance K of pixels PD1 to PD12. The first data generation unit 121 calculates a linear equation F that indicates the relationship between the calculated three average values ​​or medians and the distance from the interpolation target pixel PA. The slope of the linear equation F indicates the gradient of the object between the pixels surrounding the interpolation target pixel PA. The first data generation unit 121 calculates the luminance K of the interpolation target pixel PA based on the linear equation F.

[0052] The first data generating unit 121 may extract some of the pixels PB1 to PB4, the pixels PC1 to PC8, and the pixels PD1 to PD12, and use the luminance K of the extracted pixels in the calculation of the linear equation F.

[0053] Furthermore, when multiple pixels to be complemented are adjacent to each other, the first data generation unit 121 sets the multiple pixels to be complemented in a region to be complemented. In this case, the linear equation F represents the relationship between the region to be complemented and the distance from the region to be complemented.

[0054] 2, the image processing unit 112 further includes a second data generating unit 122. The second data generating unit 122 generates interpolated position information based on the interpolated height information. Specifically, the second data generating unit 122 acquires pixel coordinates of the pixel to be interpolated identified by the first data generating unit 121. The interpolated position information indicates the position of the interpolated height information. For example, the pixel coordinates of the pixel to be interpolated PA are an example of the interpolated height information.

[0055] [Determination process] Next, the determination process will be described with reference to Fig. 2 and Fig. 7. Fig. 7 is a diagram schematically showing the determination process performed by the determination device 1.

[0056] As shown in FIG. 2, the memory unit 15 stores a trained classification model M1. The trained classification model M1 classifies, for example, objects T1 and T2 into multiple categories as a result of machine learning. The trained classification model M1 is an example of a trained discrimination model. In other words, the trained discrimination model outputs discrimination data that identifies the category to which the objects T1 and T2 belong.

[0057] In the determination process, the acquisition unit 111 acquires, as first data, the image data GD12 shown in FIGS. 1 and 3 and the image data GD34 after the interpolation process shown in FIGS. 4 and 5. The image data GD12 includes a plurality of pieces of height information indicating the heights of a plurality of locations on the object T1. The image data GD34 includes a plurality of pieces of height information indicating the heights of a plurality of locations on the object T2. Some of the pieces of height information included in the image data GD34 include the interpolated height information generated by the first data generation unit 121.

[0058] In the example shown in Fig. 7, three image data GD1a, GD1b, and GD1c are shown as the image data GD12 or GD34. The objects in the image data GD1a, GD1b, and GD1c are solder points that indicate the locations where soldering has been performed. Fig. 7 visualizes the height information included in the image data GD1a, GD1b, and GD1c. Note that the objects are not limited to the solder points shown in Fig. 7.

[0059] Image data GD1a is image data showing "good solder." "Good solder" indicates good soldering. Image data GD1b is image data showing "horny solder." "Horned solder" has protrusions on the surface and is an example of poor soldering. Image data GD1c is image data showing "bad solder." "Bad solder" has an excessive amount of solder and is an example of poor soldering.

[0060] Furthermore, the acquiring unit 111 acquires, as second data, pixel coordinates of the pixels to be interpolated, which are interpolation position information generated by the second data generating unit 122. In the example shown in Fig. 7, three pieces of second data GD2a, GD2b, and GD2c are shown. Fig. 7 visualizes the pixel coordinates of the pixels to be interpolated. Specifically, the second data GD2a and GD2c do not include the pixel coordinates of the pixels to be interpolated, while the second data GD2b includes the pixel coordinates of the pixels to be interpolated.

[0061] The acquisition unit 111 generates a data set DS including the acquired first data and second data. The determination unit 113 inputs the data set DS generated by the acquisition unit 111 to the trained classification model M1.

[0062] In addition to the first data and the second data, the acquisition unit 111 may acquire image data before the generation of the interpolated height information as third data, and generate a dataset DS including the first data, the second data, and the third data. By including the third data in the dataset DS, the accuracy of category identification by the trained classification model M1, which will be described later, is further improved. In the examples shown in FIGS. 1 and 3, the acquisition unit 111 acquires image data GD12, which is the same as the first data, as the third data. In the examples shown in FIGS. 4 and 5, the acquisition unit 111 acquires image data GD34 before the interpolation process.

[0063] 7, the acquisition unit 111 acquires image data GD3a, GD3b, and GD3c before generation of complementary height information as third data. Then, the acquisition unit 111 generates a data set DSa including image data GD1a, second data GD2a, and image data GD3a. The acquisition unit 111 generates a data set DSb including image data GD1b, second data GD2b, and image data GD3b. The acquisition unit 111 generates a data set DSc including image data GD1c, second data GD2c, and image data GD3c.

[0064] The determination unit 113 inputs the data sets DSa, DSb, and DSc generated by the acquisition unit 111 into the trained classification model M1.

[0065] When the data set DSa is input, the trained classification model M1 classifies the objects shown in the data set DSa into category A, which indicates "good solder." The trained classification model M1 outputs classification data DDa as a classification result indicating that the data set DSa has been classified into category A. The classification result is an example of identification data indicating the category to which the object belongs.

[0066] When the data set DSb is input, the trained classification model M1 classifies the objects shown in the data set DSb into category B, which indicates "horn solder." The trained classification model M1 outputs classification data DDb as a classification result indicating that the data set DSb has been classified into category B.

[0067] When the data set DSc is input, the trained classification model M1 classifies the objects shown in the data set DSc into category C, which indicates "potato solder." The trained classification model M1 outputs classification data DDc as a classification result indicating that the data set DSc has been classified into category C.

[0068] The determination unit 113 determines whether the object is normal or abnormal based on the classification data DDa, DDb, and DDc output by the trained classification model M1. Specifically, when the trained classification model M1 outputs classification data DDa, the determination unit 113 determines that the object is normal. On the other hand, when the trained classification model M1 outputs classification data DDb or classification data DDc, the determination unit 113 determines that the object is abnormal. For example, the display unit 12 acquires the determination result by the determination unit 113 and displays a screen showing the determination result.

[0069] By including complementary position information in each dataset input to the trained classification model M1, missing height information can be included in each dataset. As a result, the accuracy of category identification by the trained classification model M1 can be improved. Therefore, the accuracy of determining whether an object is normal or abnormal is improved.

[0070] As described above, the determination device 1 has the first data generation unit 121 and the second data generation unit 122. The first data generation unit 121 generates the complementary height information, and the second data generation unit 122 generates the complementary position information, so that the determination process for the object can be performed without adding any equipment other than the determination device 1.

[0071] For example, when generating the data set DSb, the determination unit 113 labels the second data GD2b with the gradient of the linear expression F calculated by the first data generation unit 121 as gradient information indicating the gradient of the object. By labeling the gradient information, the classification accuracy of the trained classification model M1 can be further improved.

[0072] Specifically, the determination unit 113 acquires the slope of the linear equation F calculated by the first data generation unit 121. For example, the determination unit 113 determines whether the slope of the acquired linear equation F is positive or negative. If the determination unit 113 determines that the slope of the linear equation F is positive, the determination unit 113 generates gradient information indicating that the object to be complemented is a "protrusion." On the other hand, if the determination unit 113 determines that the slope of the linear equation F is negative, the determination unit 113 generates gradient information indicating that the object to be complemented is a "hole." The determination unit 113 assigns the generated gradient information to the second data GD2b as a label. Note that the gradient information may be information indicating the gradient of the slope of the linear equation F.

[0073] As described above, the first data and the third data are image data. In this embodiment, the data set DS can be easily generated by using data in a general format, such as image data. Note that the first data and the third data are not limited to image data. For example, the first data and the third data may be data containing only height information, such as radar detection results.

[0074] In this embodiment, the dataset DS is classified into one of three categories, but the number of categories into which the dataset DS is classified is not limited to three. The number of categories into which the dataset DS is classified may be two, or may be four or more. Furthermore, the category that the determination unit 113 determines to be normal and the category that the determination unit 113 determines to be abnormal may each be one or more.

[0075] In this embodiment, the data set DS includes the first data, the second data, and the third data, but is not limited to this, and the data set DS may include only the first data and the second data.

[0076] In the present embodiment, the first data, the second data, and the third data may be generated outside the determination device 1. In this case, the acquisition unit 111 acquires the first data, the second data, and the third data generated outside the determination device 1.

[0077] In this embodiment, the first data generation unit 121 generates the interpolated height information based on the height information of the surrounding area of ​​the pixel PA to be interpolated, but this is not limited to this, and the first data generation unit 121 may set an appropriate value for the interpolated height information.

[0078] Next, the determination method according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the determination method according to this embodiment.

[0079] First, the acquisition unit 111 starts the determination process and acquires third data such as image data captured from two angles of the object or image data in which height information is generated based on the two image data (step S11).

[0080] The image processing unit 112 performs image processing on the third data to generate first data and second data (step S12). For example, if the image data of the third data includes a pixel to be interpolated, the image processing unit 112 performs interpolation processing to generate the first data and second data.

[0081] The determination unit 113 acquires the generated first data and second data and inputs them to the trained classification model M1 (step S13). The trained classification model M1 outputs the identification data (step S14).

[0082] The determination unit 113 determines whether the object is normal or abnormal based on the identification data output from the trained classification model M1 (step S15).

[0083] If the identification data indicates that the object is classified into category A (Yes in step S15), the determination unit 113 determines that the object is normal (step S16). The acquisition unit 111 and the determination unit 113 perform a new determination process (step S11).

[0084] On the other hand, if the identification data indicates that the object is classified into a category other than Category A (No in step S15), the determination unit 113 determines that the object is abnormal (step S17). The acquisition unit 111 and the determination unit 113 perform a new determination process (step S11).

[0085] Next, generation of the trained classification model M1 will be described with reference to Fig. 2. As shown in Fig. 2, the determination device 1 has a discrimination model generation unit 101. For example, the control unit 11 functions as the discrimination model generation unit 101 by executing a computer program stored in the storage unit 15.

[0086] [Machine Learning] The discrimination model generation unit 101 generates a trained classification model M1 by performing machine learning on a training dataset TD. The training dataset includes a plurality of training first data that is first data used for machine learning, a plurality of training second data that is second data used for machine learning, and a plurality of training discrimination data that is discrimination data used for machine learning.

[0087] In this embodiment, the learning first data is image data in which complementary height information is included as part of the height information. The learning second data is complementary position information of the complementary height information of the learning first data. The learning identification data is classification data indicating the category to which the object shown in the learning first data belongs.

[0088] The machine learning algorithm for generating the trained classification model M1 is not particularly limited as long as it is supervised learning, and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Therefore, the trained classification model M1 includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. In the machine learning for generating the trained classification model M1, backpropagation may be used.

[0089] In this embodiment, the machine learning algorithm that generates the trained classification model M1 is a neural network. That is, the trained classification model M1 includes a neural network. The neural network includes an input layer, one or more intermediate layers, and an output layer. Preferably, the neural network is a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN), and performs deep learning.

[0090] A deep neural network includes an input layer, multiple hidden layers, and an output layer. A convolutional neural network includes an input layer, multiple convolutional layers, multiple pooling layers, a fully connected layer, and an output layer. In a convolutional neural network, convolutional layers and pooling layers alternate between the input layer and the fully connected layer.

[0091] It is particularly preferable that the machine learning algorithm that generates the trained classification model M1 is a convolutional neural network, that is, it is particularly preferable that the trained classification model M1 includes a convolutional neural network.

[0092] The discrimination model generation unit 101 performs machine learning by inputting the training data set TD into a neural network, and generates a trained classification model M1. Specifically, the discrimination model generation unit 101 performs machine learning by associating an input data pair of the first training data and the second training data with the pre-training discrimination data.

[0093] The training dataset TD may be a dataset prepared in advance, or may be a dataset input to the determination device 1 before the discriminant model generation unit 101 performs machine learning.

[0094] In this embodiment, the discriminant model generating unit 101 performs machine learning so that when an input data pair is input to a neural network, classification data indicating a category to which the object indicated in the input data pair belongs is output.

[0095] As a result of this machine learning, the discriminant model generation unit 101 generates a trained classification model M1 that can classify objects indicated by the input data pair into multiple categories. For example, when a data set including first data in which image data indicating "horned solder" is interpolated with image data indicating "good solder" is input, the trained classification model M1 generated as a result of machine learning outputs classification data indicating that the objects included in the data set have been classified into category B indicating "horned solder." The discriminant model generation unit 101 stores the generated trained classification model M1 in the storage unit 15, for example.

[0096] The training data set TD may further include third training data, which is the first training data before the interpolation process is performed.

[0097] In this embodiment, the discriminative model generation unit 101 may generate the trained classification model M1 in a state where the trained classification model M1 is already stored in the storage unit 15. In this case, the discriminative model generation unit 101 updates the trained classification model M1 stored in the storage unit 15 to the newly generated trained classification model M1.

[0098] As described above, the determination device 1 includes the discrimination model generation unit 101, which enables machine learning in the determination device 1 and enables updating of the trained discrimination model. Therefore, the classification accuracy of the trained discrimination model can be further improved.

[0099] In this embodiment, a learning device 200 that generates a trained discrimination model may generate the trained discrimination model separately from the determination device 1. For example, the learning device 200 includes an acquisition unit 111 and a discrimination model generation unit 101. The learning device 200 may be provided inside the determination device 1 or outside the determination device 1. When the learning device 200 is provided outside the determination device 1, the learning device 200 transmits the generated trained discrimination model to the determination device 1. The determination device 1 receives the trained discrimination model transmitted from the learning device 200 and stores it in the storage unit 15.

[0100] In this embodiment, the trained classification model M1 is used as an example of a trained identification model, but the trained identification model may be a trained object detection model M2, an area extraction model M3, etc., in addition to the trained classification model M1.

[0101] For example, when the first data and the third data include multiple objects, the trained object detection model M2 or the trained region extraction model M3 is used as the trained identification model.

[0102] The trained object detection model M2 detects multiple objects contained in the first data and the third data, and outputs object detection data for each detected object to which information indicating the category to which the object belongs is added.

[0103] When the discriminative model generation unit 101 generates the trained object detection model M2, the training dataset TD includes pixel coordinates indicating each object.

[0104] The trained area extraction model M3 detects areas of multiple objects contained in the first data and the third data, and outputs area extraction data for each detected area to which information indicating the category to which the object belongs is added.

[0105] When the discrimination model generation unit 101 generates the trained region extraction model M3, the training data set TD includes pixel coordinates indicating each object and pixel parameters for each pixel coordinate.

[0106] The embodiments of the present disclosure have been described above with reference to the drawings (FIGS. 1 to 8). However, the present disclosure is not limited to the above embodiments and can be implemented in various forms without departing from the spirit and scope of the present disclosure. Furthermore, the components disclosed in the above embodiments can be modified as appropriate. For example, some of the components shown in one embodiment may be added to the components of another embodiment, or some of the components shown in one embodiment may be deleted from the embodiment.

[0107] Furthermore, the drawings mainly show each component in a schematic manner to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown in the drawings may differ from the actual ones due to the convenience of creating the drawings. Furthermore, the configurations of each component shown in the above embodiment are merely examples and are not particularly limited, and it goes without saying that various modifications are possible within a scope that does not substantially deviate from the effects of the present disclosure. [Industrial Applicability]

[0108] The present disclosure is applicable to the field of determination methods using machine learning. [Explanation of symbols]

[0109] 1: Judgment device 15: Storage section 101: Discrimination model generation unit 111: Acquisition Department 112: Image processing unit 113: Judgment section 121: First data generation unit 122: Second data generation unit 200: Learning device DDa, DDb, DDc: Classification data DS, DSa, DSb, DSc: Datasets G1, G2, G3, G4, G12, G34: Captured images GD1, GD2, GD3, GD4, GD12, GD34: Image data GD1a, GD1b, GD1c: Image data GD2a, GD2b, GD2c: Second data GD3a, GD3b, GD3c: Image data M1: Classification model M2: Object detection model M3: Region extraction model P1 to P4, PB1 to PB4, PC1 to PC8, PD1 to PD12: pixels PA: Pixel to be complemented T1, T2: Object TD: Training dataset

Claims

1. an acquisition unit that acquires first data having a plurality of pieces of height information indicating the heights of a plurality of locations on the object, and second data having complementary position information indicating the position of complementary height information generated based on other height information among the height information; a storage unit that stores a trained discrimination model that outputs discrimination data that identifies a category to which the object belongs when a data set including the first data and the second data is input; a determination unit that inputs the data set into the trained discrimination model and determines whether the object is normal or abnormal based on the discrimination data output by the trained discrimination model; A determination device having the above.

2. The determination device according to claim 1 , wherein the data set further includes third data that is data before the interpolated height information is generated.

3. The determination device according to claim 2 , wherein the third data is image data representing an image of the object.

4. further comprising an image processing unit that performs image processing on the image data; The image processing unit a first data generating unit that generates the interpolated height information based on the image data obtained by capturing images of the object from two angles; a second data generating unit that generates the complementary position information based on the complementary height information; The determination device according to claim 3 , further comprising:

5. 5. The determination device according to claim 4, wherein the first data generation unit identifies a pixel to be complemented, the height information of which has been lost, from among the plurality of pixels included in the third data, and calculates a gradient of the object between the plurality of pixels based on the height information of a plurality of pixels surrounding the pixel to be complemented.

6. The determination device according to claim 5 , wherein the determination unit labels the second data with gradient information indicating the gradient calculated by the first data generation unit.

7. The method further includes a discriminative model generation unit that generates the trained discriminative model by machine learning a training dataset, the training data set includes training first data, which is the first data used in the machine learning, training second data, which is the second data used in the machine learning, and training identification data, which is the identification data used in the machine learning; The determination device according to claim 1 , wherein the discrimination model generation unit performs machine learning by associating the first training data and the second training data with pre-trained discrimination data.

8. an acquisition unit that acquires a learning data set including: first learning data having a plurality of pieces of height information that respectively indicate the heights of a plurality of locations on an object; second learning data having complementary position information that indicates the position of complementary height information generated based on other height information among the height information; and learning identification data that indicates a category to which the object belongs; a discrimination model generation unit that generates a trained discrimination model by performing machine learning on the first training data, the second training data, and the training discrimination data in association with each other; and A learning device in which the trained discrimination model outputs discrimination data that identifies the category to which the object belongs.

9. acquiring first data having a plurality of pieces of height information indicating the heights of a plurality of locations on the object, and second data having complementary position information indicating the position of complementary height information generated based on other height information among the height information; inputting a dataset including the first data and the second data into a trained discrimination model that outputs discrimination data that identifies a category to which the object belongs; determining whether the object is normal or abnormal based on the identification data output by the trained identification model; A determination method comprising:

10. acquiring first data having a plurality of pieces of height information indicating the heights of a plurality of locations on the object, and second data having complementary position information indicating the position of complementary height information generated based on other height information among the height information; inputting a dataset including the first data and the second data into a trained discrimination model that outputs discrimination data that identifies a category to which the object belongs; determining whether the object is normal or abnormal based on the discrimination data output by the trained discrimination model; A judgment program that causes a computer to execute the above.

Images