Image processing device, machine learning device, and inference device

Abstract

[Problem] To provide an image-processing device, or the like, of high working ability and versatility, allowing the states of any number of yarn cones disposed on or around a knitting machine to be detected, through a simple structure. [Solution] An image-processing device 4A provided with an acquisition unit 40 for acquiring captured images, and a yarn cone state detection unit 41 performing image-processing on a captured image acquired by the acquisition unit 40 thereby detecting the states of any number of yarn cones that are included in the captured image. The yarn cone state detection unit 41 enters the captured image acquired by the acquisition unit 40 into a learned model 5A resulting from machine-learning of the correlation between a captured image and the amount of yarn remaining on any number of yarn cones included in the captured image, thereby detecting the amount of remaining yarn on any number of yarn cones that are included in the captured image.

Description

[Technical field]

[0001] The present invention relates to an image processing device, a machine learning device, and an inference device. [Background technology]

[0002] In a knitting machine, a number of yarn cones around which knitting yarns are wound are arranged on or around the knitting machine, and various kinds of yarn cones are used in the knitting operation for knitting a fabric (for example, a knitted product, etc.). The knitting machine operator replaces the yarn cone with a yarn cone of a different type, such as a different color or thickness, depending on the fabric to be knitted, and when the knitting yarn of the yarn cone runs out, the knitting operation of the knitting machine stops and productivity decreases, so it is necessary to visually monitor the amount of yarn remaining in the yarn cone. In addition, the knitting yarn that is paid out from the yarn cone is connected to one of a number of tensioning devices arranged in a row on the upper part of the knitting machine, for example, and is supplied to the knitting needle with a predetermined tension applied via the tensioning device. In order to knit a desired fabric, the user of the knitting machine is required to correctly connect the knitting yarn that is paid out from the yarn cone to a predetermined tensioning device, but there have been cases where an operational error occurs in the operation of connecting the yarn cone to the tensioning device via the knitting yarn.

[0003] As a device for detecting the amount of remaining thread in a thread cone, thread remaining amount detection devices employing various detection methods have been proposed. For example, Patent Document 1 discloses a thread remaining amount detection device that detects the amount of remaining thread by irradiating the side of the thread cone with light and receiving the reflected light. Patent Document 2 discloses a thread remaining amount detection device that detects the amount of remaining thread by irradiating light toward the bottom surface of the thread cone with light that passes through the side of the thread cone when the thread diameter of the thread cone becomes small, and receiving the reflected light reflected by a reflecting mirror at the back with a light receiving sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 8-277069 [Patent Document 2] Patent No. 2871318 Summary of the Invention [Problem to be solved by the invention]

[0005] In the yarn remaining amount detection devices disclosed in Patent Document 1 and Patent Document 2, optical components such as optical sensors and reflecting mirrors must be arranged near each yarn cone, resulting in a large number of optical components. In an environment where yarn cones are arranged densely, the arrangement space for the optical components is limited, and it is difficult to adjust the light path during assembly. In addition, in order to detect the remaining yarn amount, the relative arrangement relationship of the yarn cone to the optical components is uniquely determined, so if the thread cone is arranged in an arbitrary position by the worker, it is natural that the remaining yarn amount cannot be detected even if the arrangement of the thread cone is even slightly out of the detection range of the optical components, resulting in poor workability and lack of versatility. In addition, the yarn remaining amount detection devices disclosed in Patent Document 1 and Patent Document 2 cannot detect which tension applying device the knitting yarn unwound from the yarn cone is connected to, i.e., cannot detect the connection relationship between the thread cone and the tension applying device.

[0006] In view of the above-mentioned problems, the present invention aims to provide an image processing device, a machine learning device, and an inference device that are highly user-friendly and versatile, and that can detect the state of any number of yarn cones arranged on or around a knitting machine, such as the remaining yarn amount, the connection relationship with a tensioning device, and the arranged position of the yarn cones, with a simple configuration. [Means for solving the problem]

[0007] In order to achieve the above object, an image processing device (4A, 4B, 4C) according to a first aspect of the present invention comprises: An acquisition unit (40) for acquiring an image including a number of yarn cones disposed on or around the knitting machine; a thread cone state detection unit (41) that detects a state of the arbitrary number of thread cones included in the captured image by performing image processing on the captured image acquired by the acquisition unit (40),

[0008] In order to achieve the above object, a machine learning device (6A) according to a second aspect of the present invention comprises: A machine learning device (6A) for use in detecting remaining yarn amounts for any number of yarn cones positioned on or around a knitting machine, comprising: a learning dataset storage unit (61) configured to store a plurality of learning datasets each including input data of a captured image including an arbitrary number of yarn cones and output data associated with the input data and including remaining yarn amounts of the arbitrary number of yarn cones included in the captured image; a machine learning unit that learns a learning model that infers a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and a trained model memory unit that stores the learning model trained by the machine learning unit.

[0009] In order to achieve the above object, an inference device (5A) according to a third aspect of the present invention comprises: A reasoning device (5A) for use in detecting the amount of yarn remaining in any number of yarn cones arranged on or around a knitting machine, comprising: The inference device comprises a memory and at least one processor; The at least one processor: A process of inputting a captured image including a number of thread cones; When the captured image is input, a process of inferring the remaining yarn amounts of the arbitrary number of yarn cones included in the captured image is executed.

[0010] In order to achieve the above object, a machine learning device (6B1) according to a fourth aspect of the present invention includes: A machine learning device (6B1) used to detect the condition of any number of yarn cones placed on or around a knitting machine, comprising: a learning dataset storage unit (61) configured to store a plurality of learning datasets each including input data of a captured image including an arbitrary number of thread cones and output data corresponding to the input data and including area information defining an arbitrary number of areas surrounding each of the arbitrary number of thread cones included in the captured image; a machine learning unit (62) that learns a learning model that infers a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and a trained model storage unit (63) that stores the training model trained by the machine learning unit (62).

[0011] In order to achieve the above object, an inference device (5B1) according to a fifth aspect of the present invention comprises: A reasoning device (5B1) for use in detecting the state of any number of yarn cones arranged on or around a knitting machine, comprising: The inference device comprises a memory and at least one processor; The at least one processor: A process of inputting a captured image including a number of thread cones; When the captured image is input, the method is configured to execute a process of inferring area information that defines an arbitrary number of areas surrounding each of the arbitrary number of thread cones included in the captured image.

[0012] In order to achieve the above object, a machine learning device according to a sixth aspect of the present invention (6B2) includes: A machine learning device (6B2) used to detect remaining yarn amounts of any number of yarn cones located on or around a knitting machine, comprising: a learning dataset storage unit (61) configured to store a plurality of learning datasets each including input data of a captured image including one yarn cone and output data associated with the input data and including a remaining amount of yarn of the one yarn cone included in the captured image; a machine learning unit (62) that learns a learning model that infers a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and a trained model storage unit (63) that stores the training model trained by the machine learning unit (62).

[0013] In order to achieve the above object, an inference device (5B2) according to a seventh aspect of the present invention comprises: A reasoning device (5B2) for use in detecting the amount of yarn remaining in any number of yarn cones arranged on or around a knitting machine, comprising: The inference device comprises a memory and at least one processor; The at least one processor: A process of inputting a captured image including one yarn cone; and when the captured image is input, a process of inferring a remaining amount of yarn of the one yarn cone included in the captured image is executed. Reasoning device (5B2).

[0014] In order to achieve the above object, a machine learning device (6C) according to an eighth aspect of the present invention comprises: A machine learning device (6C) used to detect a connection relationship between a number of yarn cones and a number of tensioning devices arranged on or around a knitting machine, comprising: a learning dataset storage unit (61) configured to store a plurality of learning datasets each including input data consisting of a captured image including an arbitrary number of thread cones and output data corresponding to the input data and consisting of connection relationships between the arbitrary number of thread cones and a plurality of tension applying devices included in the captured image; a machine learning unit (62) that learns a learning model that infers a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and a trained model storage unit (63) that stores the training model trained by the machine learning unit (62).

[0015] In order to achieve the above object, an inference device (5C) according to a ninth aspect of the present invention comprises: A reasoning device (5C) for use in detecting the connection relationship between any number of yarn cones and a plurality of tensioning devices arranged on or around a knitting machine, said reasoning device comprising: The inference device comprises a memory and at least one processor; The at least one processor: A process of inputting a captured image including a number of thread cones; When the captured image is input, a process of inferring a connection relationship between the arbitrary number of yarn cones included in the captured image and a plurality of tension applying devices is executed. Effect of the Invention

[0016] According to the image processing devices (4A to 4C) and inference devices (5A, 5B1, 5B2, 5C) of the present invention, it is possible to detect the state of any number of yarn cones included in a captured image (e.g., the remaining amount of yarn in the yarn cone, the connection relationship between the yarn cone and the tensioning device, the position where the yarn cone is placed, etc.). Therefore, it is not necessary to install optical components for each yarn cone, and it is possible to detect the state of the yarn cone without being restricted by conditions such as the number and position of the yarn cones when the yarn cones are placed on or around the knitting machine. Therefore, it is possible to detect the state of the yarn cone with a simple configuration, and it is possible to improve workability and versatility.

[0017] In addition, according to the machine learning device (6A, 6B1, 6B2, 6C) of the present invention, it is possible to generate a trained model (5A, 5B1, 5B2, 5C) that can accurately detect (estimate) the state of any number of thread cones contained in a camera-captured image (e.g., the remaining amount of thread in the thread cone, the connection relationship between the thread cone and the tensioning device, the position at which the thread cone is placed, etc.) from the camera-captured image including the any number of thread cones. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a perspective view showing an example of a flat knitting machine 1 according to a first embodiment. [Diagram 2] 1 is a diagram showing an example of a camera-captured image 30 captured by a camera 3. FIG. [Diagram 3] 1 is a schematic block diagram showing an example of a flat knitting machine 1 equipped with an image processing device 4A according to a first embodiment. [Figure 4] 10 is a flowchart showing an example of an image processing method by the image processing device 4A according to the first embodiment. [Diagram 5] FIG. 2 is a schematic block diagram showing an example of a machine learning device 6A according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of an inference model 50A based on CNN used in a machine learning device 6A according to the first embodiment. [Figure 7] 4 is a flowchart illustrating an example of a machine learning method according to the first embodiment. [Figure 8] FIG. 11 is a schematic block diagram showing an example of a flatbed knitting machine 1 equipped with an image processing device 4B according to a second embodiment. [Figure 9] 13 is a flowchart showing an example of an image processing method by an image processing device 4B according to a second embodiment. [Figure 10] FIG. 11 is a schematic block diagram showing an example of a first machine learning device 6B1 according to the second embodiment. [Figure 11] A figure showing an example of a first inference model 50B1 based on CNN used in a first machine learning device 6B1 according to the second embodiment. [Figure 12] FIG. 11 is a schematic block diagram showing an example of a second machine learning device 6B2 according to the second embodiment. [Figure 13] A figure showing an example of a second inference model 50B2 based on CNN used in a second machine learning device 6B2 according to the second embodiment. [Figure 14] FIG. 11 is a schematic block diagram showing an example of a flat knitting machine 1 equipped with an image processing device 4C according to a third embodiment. [Figure 15] 13 is a flowchart showing an example of an image processing method by an image processing device 4C according to the third embodiment. [Figure 16] FIG. 13 is a schematic block diagram showing an example of a machine learning device 6C according to the third embodiment. [Figure 17]A figure showing an example of an inference model 50C based on CNN used in a machine learning device 6C according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, each embodiment for carrying out the present invention will be described with reference to the drawings. Note that, in the following, the scope necessary for the explanation to achieve the object of the present invention is shown in a schematic manner, and the scope necessary for the explanation of the relevant part of the present invention will be mainly explained, and the parts that are omitted from the explanation will be referred to as publicly known techniques.

[0020] (First embodiment) FIG. 1 is a perspective view showing an example of a flat knitting machine 1 according to a first embodiment.

[0021] The flat knitting machine 1 comprises a lower frame 11A placed on the floor surface, a pair of needle beds 13 that support a plurality of knitting needles 12 so that they can be advanced and retreated and that are arranged on the lower frame 11A in a state facing each other from the front to the rear, a carriage 14 that moves along the needle beds 13 and controls the knitting needles 12 to advance and retreat, and a top plate 15 that houses the needle beds 13 and the carriage 14 and on which any number of yarn cones 2 are placed. The flat knitting machine 1 may also be provided with a creel stand around it on which any number of yarn cones 2 are placed.

[0022] The flat knitting machine 1 also includes a mechanism for supplying knitting yarn 20 from any number of yarn cones 2 placed on the top plate 15 to the knitting needles 12, the mechanism comprising a plurality of tensioning devices 16 attached to the upper frame 11B and arranged above the top plate 15, side tension units 17 provided on the left and right sides of the lower frame 11A, and a plurality of yarn feeders 18 selectably connected to the carriage 14 when the carriage 14 moves. The knitting yarn 20 from the yarn cones 2 is supplied in the order of the tensioning devices 16, the side tension units 17, the yarn feeders 18, and the knitting needles 12 to knit a fabric.

[0023] Furthermore, the flat knitting machine 1 is equipped with two cameras 3 attached to the left and right sides of the upper frame 11B and an image processing device 4A housed in the lower frame 11A as a system for detecting the state of any number of yarn cones 2 placed on the top plate 15.

[0024] The camera 3 is composed of an image sensor such as a CMOS sensor or a CCD sensor. The camera 3 captures an image of the top plate 15 from above the top plate 15, and its position and orientation are adjusted so that the yarn cone 2 placed on the top plate 15 falls within the angle of view of the camera 3. The camera 3 may have pan / tilt / zoom functions. The number of cameras 3 is not limited to two, and may be one, or three or more. The mounting position of the camera 3 may be changed as appropriate, and for example, the camera may be mounted on the ceiling above the flat knitting machine 1.

[0025] FIG. 2 is a diagram showing an example of a camera-captured image 30 captured by the camera 3. As shown in FIG.

[0026] The camera-captured image 30 includes any number of yarn cones 2 placed on the top plate 15. In the example of FIG. 2, five yarn cones 2A to 2E are included. The yarn cone 2 has a truncated cone-shaped core 21 around which knitting yarn 20 is wound, and can be placed at any position on the upper surface of the top plate 15. Therefore, the number of yarn cones 2 included in the camera-captured image 30 is not a fixed number, but an arbitrary number. Note that the camera-captured image 30 may include not only the yarn cones 2 placed on the top plate 15, but also any number of yarn cones 2 placed on a creel stand or the like provided around the flat knitting machine 1 around the flat knitting machine 1.

[0027] Furthermore, the captured state (e.g., outer shape and outer size) of the yarn cone 2 included in the camera-captured image 30 differs not only depending on the outer diameter dimension of the core 21, the thickness and number of turns of the knitting yarn 20, and the state of use of the knitting yarn 20, but also depending on the relative positional relationship between the yarn cone 2 and the camera 3. Furthermore, the camera-captured image 30 may be captured in a state in which a part of the yarn cone 2 on the rear side is hidden by the yarn cone 2 on the front side as viewed from the camera 3. Note that the camera-captured image 30 includes, in addition to any number of yarn cones 2, the knitting yarn 20 that is unwound from each yarn cone 2 and connected to the tensioning device 16, and may further include objects other than the yarn cones 2, for example, the tensioning device 16.

[0028] In the five yarn cones 2A to 2E included in the camera-captured image 30 shown in FIG. 2, the yarn cone 2A is in a state immediately after being placed on the top plate 15, that is, in an unused state. Then, as the knitting yarn 20 is gradually used from the unused state, the outer diameter becomes smaller and the remaining amount of yarn decreases, as in the yarn cones 2B and 2C. Then, after a part of the core 21 is visible through the gaps in the knitting yarn 20, as in the yarn cone 2D, it finally becomes a "no yarn" state, as in the yarn cone 2E, and the entire core 21 is exposed. Note that the four yarn cones 2A to 2D are in a "yarn present" state, and the more detailed remaining amount of yarn is expressed as, for example, "100% remaining," "70% remaining," "40% remaining," and "10% remaining."

[0029] FIG. 3 is a schematic block diagram showing an example of a flatbed knitting machine 1 equipped with an image processing device 4A according to the first embodiment.

[0030] The electric configuration of the flat knitting machine 1 includes, in addition to the camera 3 and image processing device 4A, a control panel 100 configured with a microcontroller or the like, and an operation display panel 101 configured with a touch panel, switches, etc. The control panel 100 is connected to actuators such as motors and sensors (none of which are shown), and controls each part of the flat knitting machine 1, thereby controlling the knitting operation of the flat knitting machine 1. The operation display panel 101 accepts operations by an operator, and outputs various types of information by display and sound.

[0031] (Image processing device 4A) The image processing device 4A is composed of a general-purpose or dedicated computer, and includes, for example, a calculation means such as a CPU, a GPU, a storage means such as a ROM, a RAM, a communication unit for communicating with a network or other devices by wire or wirelessly, and a bus connecting these units. The image processing device 4A may be incorporated in the control panel 100 as a part of the function of the control panel 100. The image processing device 4A includes not only a device provided as a standalone device, but also a device provided in the form of a non-transitory computer-readable medium storing a program for causing an arbitrary processor to execute the operations described below or one or more instructions for executing the operations.

[0032] 3, the image processing device 4A includes an acquisition unit 40, a thread cone state detection unit 41, a trained model storage unit 42, and an output processing unit 43. The acquisition unit 40, the thread cone state detection unit 41, and the output processing unit 43 are configured by the above-mentioned calculation means, and the trained model storage unit 42 is configured by the above-mentioned storage means.

[0033] The acquisition unit 40 is connected to the camera 3 and functions as an interface for acquiring an image captured by the camera 3. The acquisition unit 40 acquires a camera-captured image 30 including an arbitrary number of yarn cones 2 arranged on or around the flat knitting machine 1 (in this embodiment, the upper surface of the top plate 15) as the camera-captured image 30 captured by the camera 3. Note that any connection method or communication method may be used when the image processing device 4A is connected to the camera 3 via the acquisition unit 40.

[0034] The thread cone state detection unit 41 detects the state of any number of thread cones 2 included in the camera-captured image 30 by performing image processing on the camera-captured image 30 acquired by the acquisition unit 40. Examples of the state of the thread cone 2 include the remaining amount of thread in the thread cone 2, the position, external shape, and external size of the thread cone 2, the type (color and thickness) of the knitting yarn 20, and the connection relationship between the thread cone 2 and the tensioning device 16. The thread cone state detection unit 41 may detect one or more of the various states of the thread cone 2 described above.

[0035] In this embodiment, a case will be described in which the thread cone state detection unit 41 detects the remaining amount of thread in the thread cone 2 as the state of an arbitrary number of thread cones 2. Also, a case will be described in which the thread cone state detection unit 41 performs a series of processes as image processing for the camera-captured image 30, in which the camera-captured image 30 is input as input data to the trained model 5A, and an inference result (the remaining amount of thread in the thread cone 2) is output from the trained model 5A by an inference process using the trained model 5A.

[0036] The trained model 5A has been trained by machine learning using a machine learning device 6A and a machine learning method described below to learn the correlation between a camera-captured image 30 containing an arbitrary number of thread cones and the remaining thread amount of an arbitrary number of thread cones 2 contained in the camera-captured image 30.

[0037] The thread cone state detection unit 41 not only performs an inference process using the trained model 5A, but also includes a pre-processing function of applying a predetermined image adjustment (e.g., image format, image size, image filter, image mask, etc.) to the camera-captured image 30 as input data acquired by the acquisition unit 40 as pre-processing of the inference process and inputting the image to the trained model 5A, and a post-processing function of applying a predetermined arithmetic expression or logical expression to the output data (inference result) output by the trained model 5A as post-processing of the inference process to finally determine the remaining amount of thread in the thread cone 2. The inference result of the thread cone state detection unit 41 may be stored in a storage means (not shown), and the stored past inference results may be used as a learning data set used for online learning or re-learning to further improve the inference accuracy of the trained model 5A in the trained model storage unit 42, for example.

[0038] The output processing unit 43 is a process for outputting the detection result detected by the yarn cone state detection unit 41, i.e., the state of the yarn cone 2. As a specific output means, various means can be adopted, and for example, the detection result may be notified to an operator by display or sound via the operation display panel 101, may be stored in a storage means of the control panel 100 as an operation history of the flat knitting machine 1, or may be transmitted to a production management system higher than the flat knitting machine 1. At that time, the output processing unit 43 may acquire necessary data (e.g., knitting data, etc.) from the flat knitting machine 1.

[0039] An image processing method by the image processing device 4A having the above configuration will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of an image processing method by the image processing device 4A according to the first embodiment. The series of image processing steps shown in Fig. 4 are repeatedly executed at a predetermined timing by the image processing device 4A. The predetermined timing may be any timing, and may be, for example, at a predetermined time interval or for each production number, or when a predetermined event occurs (when an operator operates the device, when setting up, when the device is stopped, etc.). Below, a case will be described in which the image processing steps are executed while the flatbed knitting machine 1 is performing a knitting operation.

[0040] When the knitting operation is performed by the flatbed knitting machine 1, the knitting yarn 20 is supplied from the yarn cone 2 placed on the top plate 15, and a knitted fabric is knitted. Then, at the above-mentioned predetermined timing, the acquisition unit 40 acquires a camera-captured image 30 in which the yarn cone 2 on the top plate 15 is captured by the camera 3 (step S100).

[0041] Next, the yarn cone state detection unit 41 refers to the trained model 5A stored in the trained model storage unit 42 and performs an inference process using the trained model 5A (step S110). At this time, it is preferable that the trained model 5A used in the inference process is specified in advance based on a predetermined program or an operation by an operator.

[0042] Specifically, the thread cone state detection unit 41 performs preprocessing on the camera-captured image 30 acquired by the acquisition unit 40, inputs the preprocessed image to the trained model 5A, and performs post-processing on the output data from the trained model 5A to detect the remaining thread amounts of an arbitrary number of thread cones 2 included in the camera-captured image 30 as an inference result of the trained model 5A. The inference result of step S110 illustrated in FIG. 4 is that five thread cones surrounded by five thread cone regions A to E (lower left coordinates and upper right coordinates) are detected from the camera-captured image 30, and the remaining thread amounts of the five thread cones 2 are detected in multi-values. Specifically, the remaining thread amounts of the four thread cones 2 surrounded by each of the four thread cone regions A to D are "100% remaining," "70% remaining," "40% remaining," and "10% remaining," respectively, and the remaining thread amount of the one thread cone 2 surrounded by thread cone region E is "no thread."

[0043] Next, the output processing unit 43 judges whether the detection result of the remaining yarn amount for any of the yarn cones 2 indicates "no yarn" based on the result of the inference process performed by the yarn cone state detection unit 41 in step S110 (step S120). If the output processing unit 43 judges affirmatively in step S120 (Yes in step S120), the output processing unit 43 notifies the operator, for example, by the operation display panel 101, that the inference result is "no yarn", that is, that the knitting operation of the flat knitting machine 1 will be stopped (step S130). In the inference result illustrated in FIG. 4, the output processing unit 43 notifies the operator that the yarn cone 2 present at the position defined by the yarn cone area E is "no yarn". At that time, the output processing unit 43 may stop the knitting operation of the flat knitting machine 1 by notifying the control panel 100 that the detection result is "no yarn".

[0044] On the other hand, if the detection result of the remaining yarn amount for all the yarn cones 2 is not "no yarn" and the output processing unit 43 makes a negative determination in step S120 (No in step S120), it further determines whether the detection result of the remaining yarn amount for any of the yarn cones 2 indicates a predetermined remaining yarn amount forecast value (e.g., 10%) or less (step S140). If the output processing unit 43 makes a positive determination in step S140 (Yes in step S140), it notifies the operator that the inference result is "no yarn forecast", that is, that the yarn cone 2 will soon become "no yarn" and the knitting operation of the flatbed knitting machine 1 will soon stop (step S150). At that time, the output processing unit 43 may notify the control panel 100 that the detection result is "no yarn forecast".

[0045] On the other hand, if the detection results of the remaining yarn amounts for all the yarn cones 2 are not equal to or less than the remaining yarn amount warning value and the output processing unit 43 makes a negative determination in step S140 (No in step S140), the image processing step is terminated without notifying the operator. Note that the output processing unit 43 may output that the detection result is not "no yarn" or "no yarn warning", i.e., "yarn present", and may notify the control panel 100, for example.

[0046] Although the output processing unit 43 is configured to output "no thread" and "no thread notice" as the detection result, it may output either one of them, in which case, either steps S120, S130 or steps S140, S150 may be omitted. If the tension applying device 16 has a function for detecting "no thread", the tension applying device 16 may detect "no thread" and the output processing unit 43 may output "no thread notice" as the detection result, so that steps S120 and S130 may be omitted. Furthermore, if the remaining thread amount is detected as a binary value, for example, the detection result of the remaining thread amount of four thread cones 2 surrounded by thread cone areas A to D is detected as "thread present" and the detection result of the remaining thread amount of one thread cone 2 surrounded by thread cone area E is detected as "no thread", steps S140 and S150 may be omitted.

[0047] As described above, the image processing device 4A and image processing method according to the present embodiment can detect the state of any number of yarn cones 2 included in the camera-captured image 30 (the remaining yarn amount is described as an example in the present embodiment). Therefore, there is no need to install optical components for each yarn cone 2, and the state of the yarn cones 2 can be detected without being restricted by conditions such as the number and positions of the yarn cones 2 when the yarn cones 2 are placed on or around the knitting machine. Therefore, the state of the yarn cones 2 can be detected with a simple configuration, and workability and versatility can be improved.

[0048] (Machine Learning Device 6A) Next, a machine learning device 6A used to detect the remaining yarn amounts of any number of yarn cones 2 arranged on or around the flat knitting machine 1 will be described with reference to Fig. 5. Fig. 5 is a schematic block diagram showing an example of the machine learning device 6A according to the first embodiment.

[0049] The machine learning device 6A generates a learning model (trained model 5A) capable of detecting (estimating) the remaining amount of yarn in the yarn cone 2 based on an arbitrary number of yarn cone captured images 31 including an arbitrary number of yarn cones 2. The trained model 5A generated by the machine learning device 6A is provided to the image processing device 4A and stored in a trained model storage unit 42, and is used for inference processing by the yarn cone state detection unit 41. Although the image processing device 4A and the machine learning device 6A are shown here as separate devices, the image processing device 4A and the machine learning device 6A may be configured as a single device.

[0050] The machine learning device 6A is composed of a general-purpose or dedicated computer, and includes, for example, a computing means such as a CPU, a GPU, a storage unit composed of a ROM, a RAM, etc., a communication unit for communicating with a network or other devices by wire or wirelessly, and a bus connecting these units. Note that the machine learning device 6A includes not only devices provided as devices that operate independently, but also devices provided in the form of a non-transitory computer-readable medium that stores a program for causing an arbitrary processor to perform the operations described below or one or more instructions for performing the operations.

[0051] 5, the machine learning device 6A includes a learning dataset acquisition unit 60, a learning dataset storage unit 61, a machine learning unit 62, and a trained model storage unit 63. The learning dataset acquisition unit 60 and the machine learning unit 62 are configured by the above-mentioned calculation means, and the learning dataset storage unit 61 and the trained model storage unit 63 are configured by the above-mentioned storage means.

[0052] The learning dataset acquisition unit 60 is, for example, an interface unit for acquiring a plurality of data constituting a learning (training) dataset from various devices connected via a communication unit. Here, examples of the various devices connected to the learning dataset acquisition unit 60 include an external device 7 and a terminal device 8 used by an operator. Note that, although the machine learning device 6A, the external device 7, and the terminal device 8 are shown as separate devices in FIG. 5, these devices may be appropriately combined. For example, the external device 7 and the terminal device 8 may be configured as a single device, or the machine learning device 6A, the external device 7, and the terminal device 8 may be configured as a single device.

[0053] The learning dataset acquisition unit 60 acquires the arbitrary number of thread cone captured images 31 as input data from, for example, the external device 7, and acquires the remaining yarn amounts of the arbitrary number of thread cones 2 included in the arbitrary number of thread cone captured images 31 as output data from, for example, the terminal device 8. Then, by associating these input data with the output data, one learning dataset is constructed.

[0054] 2, the arbitrary number of yarn cones captured image 31 is an image including an arbitrary number of yarn cones 2. The arbitrary number of yarn cones captured image 31 may be an image of an arbitrary number of yarn cones 2 arranged on or around the flat knitting machine 1 (in this embodiment, the upper surface of the top plate 15) captured by the camera 3, or may be an image of an arbitrary number of yarn cones 2 arranged on a simulation table simulating the top plate 15 captured by a camera for data creation, for example.

[0055] The remaining amount of yarn in the yarn cone 2 is information indicating the remaining amount of the knitting yarn 20 wound around the core 21 of the yarn cone 2, and various data formats can be adopted. For example, the remaining amount of yarn in the yarn cone 2 can be configured as information indicating either "no yarn" or "yarn present". In this case, the remaining amount of yarn is classified into two values, for example, a value indicating "no yarn" is "0" and a value indicating "yarn present" is "1", and the corresponding value can be input by the operator using the terminal device 8 in a form corresponding to the input data. In addition, the remaining amount of yarn in the yarn cone 2 can be configured as step-by-step information. In this case, the remaining amount of yarn is classified into multiple values, for example, when expressing the remaining amount of yarn in 11 steps, a value indicating "no yarn" is "0", and values ​​indicating the remaining amount of yarn of "10% remaining", "20% remaining", ..., "100% remaining" (unused state) are "0.1", "0.2", ..., "1", and the corresponding value can be input by the operator using the terminal device 8 in a form corresponding to the input data.

[0056] The training dataset storage unit 61 is a database for storing the training dataset acquired by the training dataset acquisition unit 60. The specific configuration of the database constituting the training dataset storage unit 61 may be changed as appropriate. For example, while the training dataset storage unit 61 and the trained model storage unit 63 are shown as separate storage units in FIG. 5, they may be configured as a single storage unit (database).

[0057] The machine learning unit 62 generates a learning model (trained model 5A) that infers a correlation between input data and output data included in the multiple learning datasets by performing machine learning using the multiple learning datasets stored in the learning dataset storage unit 61. Note that in the embodiment, a convolutional neural network (CNN) (described later) is adopted as a specific learning method for machine learning.

[0058] The trained model storage unit 63 is a database for storing the trained model 5A generated by the machine learning unit 62. The trained model 5A stored in the trained model storage unit 63 is applied to an actual system (image processing device 4A) upon request via a communication line including the Internet or a storage medium.

[0059] Next, a learning method in the machine learning unit 62 using a plurality of learning data sets acquired as described above will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of an inference model 50A based on CNN used in the machine learning device 6A according to the first embodiment.

[0060] Inference model 50A includes an input layer 51, an intermediate layer 52, and an output layer 53. Inference model 50A detects objects included in an image (object detection) and detects the state of the detected objects (state detection), and when an image includes multiple objects, performs object detection and state detection for each object.

[0061] The input layer 51 has neurons whose number corresponds to the number of images serving as input data, and the pixel value of each pixel is input to each neuron.

[0062] The intermediate layer 52 is composed of a convolutional layer 520, a pooling layer 521, and a fully connected layer 522. The convolutional layer 520 and the pooling layer 521 are, for example, provided in a plurality of alternating layers. The convolutional layer 520 and the pooling layer 521 extract features from an image input via the input layer 51. The fully connected layer 522 converts the features extracted from the image by the convolutional layer 520 and the pooling layer 521, for example, by an activation function, and outputs them as a feature vector. Note that the fully connected layer 522 may be provided in a plurality of layers.

[0063] The output layer 53 outputs output data including an object detection result and a state detection result as an inference result based on the feature vector output from the fully connected layer 522. The object detection result is, for example, information that defines an area (e.g., a rectangle, a circle, etc.) surrounding the detected object, and in the case of a rectangle, it may be the lower left coordinate and the upper right coordinate, or the center coordinate and the width and height. The state detection result is, for example, information indicating the state of the detected object, and may be a discrete value or a continuous value. In addition to these, the output data may also include, for example, a score indicating the likelihood of the estimation result.

[0064] Between each layer of the inference model 50A, synapses are laid that connect the neurons between the layers, and each synapse in the convolutional layer 520 and the fully connected layer 522 of the intermediate layer 52 is assigned a weight.

[0065] The machine learning unit 62 according to the present embodiment inputs a learning dataset to the inference model 50A, and learns a correlation between the arbitrary number of yarn cone captured images 31 and the remaining yarn amounts of the arbitrary number of yarn cones 2 included in the arbitrary number of yarn cone captured images 31. Specifically, the machine learning unit 62 inputs the arbitrary number of yarn cone captured images 31 constituting the learning dataset as input data to the input layer 51 of the inference model 50A. Note that the machine learning unit 62 may perform predetermined image adjustments (e.g., image format, image size, image filter, image mask, etc.) on the arbitrary number of yarn cone captured images 31 as preprocessing when inputting the arbitrary number of yarn cone captured images 31 to the input layer 51.

[0066] Then, the machine learning unit 62 uses an error function that compares the remaining thread amount (state detection result) indicated by the output data output from the output layer 53 as an inference result with the remaining thread amount (teacher data) that constitutes the learning dataset, and iteratively adjusts (backpropagation) the weights associated with each synapse so that the evaluation value of the error function becomes smaller.

[0067] Next, when the machine learning unit 62 determines that a specified learning termination condition has been met, such as each of the above-mentioned steps being repeated a specified number of times or the evaluation value of the error function becoming smaller than an allowable value, it terminates the learning and stores the inference model 50A (all weights associated with each synapse) in the learned model memory unit 63 as the learned model 5A.

[0068] (Machine learning methods) Next, in relation to the above-mentioned machine learning device 6A, a machine learning method will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the machine learning method according to the first embodiment.

[0069] In the machine learning method described below, a case where it is executed by the above-mentioned machine learning device 6A will be described. First, as a preparation before starting machine learning, a desired number of training datasets are prepared, and the prepared multiple training datasets are stored in the training dataset storage unit 61 (step S200). The number of training datasets to be prepared here should be set in consideration of the inference accuracy required for the finally obtained trained model 5A.

[0070] A number of methods can be adopted as a method for preparing the learning data set. For example, during the knitting operation of the flatbed knitting machine 1, the camera 3 is used to capture images of the yarn cones 2 in various states, and the operator uses the terminal device 8 or the like to specify and input the remaining yarn amount of the yarn cone 2 included in the arbitrary number of yarn cone captured images 31 in association with the captured arbitrary number of yarn cone captured images 31, thereby preparing input data and output data constituting the learning data set. Then, a method for preparing a desired number of learning data sets by repeating such operations may be adopted. In addition to such a method, various methods may be adopted, such as acquiring the learning data set by capturing an image of the yarn cone 2 placed on a simulation table simulating the top plate 15 with a data creation camera. In this case, it is preferable to prepare, as the learning data set, not only the arbitrary number of yarn cone captured images 31 including the yarn cones 2 in various states with and without yarn as the remaining yarn amount, but also the arbitrary number of yarn cone captured images 31 including various numbers of yarn cones 2 arranged at various positions and having various external shapes and sizes.

[0071] Next, to start learning in the machine learning unit 62, the pre-learning inference model 50A is prepared (S210). The pre-learning inference model 50A prepared here has, for example, the structure illustrated in FIG. 6, and the weights of each synapse are set to an initial value. Then, for example, one learning dataset is selected randomly from the multiple learning datasets stored in the learning dataset storage unit 61 (step S220), and input data in the one learning dataset is input to the prepared pre-learning inference model 50A (step S230). In the learning dataset illustrated in FIG. 6, the input data is an arbitrary number of yarn cone captured images 31 including five yarn cones 2 in an area defined by each of five yarn cone regions A to E (lower left coordinates and upper right coordinates), and the output data is information indicating that the remaining yarn amounts of the five yarn cones 2 surrounded by each of the five yarn cone regions A to E are "100% remaining", "70% remaining", "40% remaining", "10% remaining", and "no yarn", respectively.

[0072] The remaining thread amount (state detection result) output as an inference result from the output layer 53 in step S230 above is generated by a pre-learning neural network model, and therefore in most cases indicates a value different from a desired result, i.e., information different from the correct remaining thread amount of the yarn cone 2. Therefore, next, machine learning is performed using the remaining thread amount as teacher data in one learning dataset acquired in step S220 and the remaining thread amount (state detection result) output from the output layer 53 in step S230 (step S240).

[0073] Here, the machine learning is, for example, a process (backpropagation) of comparing the remaining thread amount constituting the teacher data with the remaining thread amount (state detection result) output from the output layer 53, and adjusting the weights associated with each synapse in the pre-learned inference model 50A so as to obtain a preferable inference model 50A. Note that the format of the output data output from the output layer 53 of the pre-learned inference model 50A is the same as that of the teacher data in the learning dataset to be learned.

[0074] 6 shows that five yarn cones 2 included in each of five yarn cone regions A to E (lower left coordinate and upper right coordinate) are detected from the arbitrary number of yarn cone captured image 31, and that the remaining yarn amounts of the five yarn cones 2 surrounded by each of the five yarn cone regions A to E are "100% remaining," "60% remaining," "40% remaining," "10% remaining," and "no yarn," respectively. Therefore, since an error occurs between the teacher data and the remaining yarn amount of the thread cone 2 surrounded by the thread cone region B, the weight of the inference model 50A is adjusted based on the error.

[0075] When machine learning is performed in step S240, whether or not machine learning needs to be continued is determined based on, for example, the remaining number of unlearned learning datasets stored in the learning dataset storage unit 61 (step S250). If the learning end condition is not met and machine learning is to be continued (No in step S250), the process returns to step S220, and if the learning end condition is met and machine learning is to be ended (Yes in step S250), the process proceeds to step S260. If the machine learning is to be continued, the process of steps S220 to S240 is performed multiple times on the inference model 50A being trained using unlearned learning datasets. The accuracy of the trained model 5A that is finally generated generally increases in proportion to the number of times.

[0076] If the machine learning is to be terminated (Yes in step S250), the inference model 50A generated by adjusting the weights associated with each synapse is stored as a trained model 5A in the trained model memory unit 63 (step S260), and the series of machine learning processes is terminated.

[0077] In the above-mentioned learning technique and machine learning method of the machine learning device 6A, in order to generate one trained model 5A, a machine learning process is repeatedly performed multiple times on one (before learning) inference model 50A to improve its accuracy and acquire a trained model 5A that is sufficient for application to the image processing device 4A. However, the present invention is not limited to this acquisition method. For example, a trained model 5A that has undergone machine learning a predetermined number of times may be stored as one candidate in a plurality of trained model storage units 63, a validity judgment dataset may be input to the plurality of trained models to generate an output layer (neuron value), and the accuracy of the values ​​identified in the output layer may be compared to select one best trained model 5A to be applied to the image processing device 4A. The validity judgment dataset may be composed of a dataset similar to the training dataset used for learning, and may be one that has not been used for learning.

[0078] As described above, according to the machine learning device 6A and machine learning method of this embodiment, it is possible to generate a trained model 5A that can accurately detect (estimate) the state of any number of thread cones 2 contained in a camera-captured image 30 (in this embodiment, the remaining thread amount is used as an example) from the camera-captured image 30 containing the any number of thread cones 2.

[0079] Second Embodiment In the first embodiment, the machine learning device 6A generates one trained model 5A, and the image processing device 4A inputs a camera-captured image 30 to the one trained model 5A to detect the remaining amount of thread of the thread cone 2 included in the camera-captured image 30. In contrast, in the second embodiment, the first and second machine learning devices 6B1 and 6B2 generate first and second trained models 5B1 and 5B2, respectively, and the image processing device 4B has a model structure in which the first trained model 5B1 is connected in series to the image processing device 4B, with the first trained model 5B1 as the front stage and the second trained model 5B2 as the rear stage, and the image processing device 4B inputs a camera-captured image 30 to the first trained model 5B1 to detect the remaining amount of thread of the thread cone 2 included in the camera-captured image 30. Since other basic configurations and operations are the same as those of the first embodiment, the following description will focus on the characteristic parts of the second embodiment. Although the first and second machine learning devices 6B1, 6B2 are described below as separate devices, the first and second machine learning devices 6B1, 6B2 may be configured as a single device.

[0080] (Image processing device 4B) 8 is a schematic block diagram showing an example of a flat knitting machine 1 equipped with an image processing device 4B according to the second embodiment. A first trained model 5B1 and a second trained model 5B2 are stored in a trained model storage unit 42 of the image processing device 4B.

[0081] The first trained model 5B1 is a model that has been trained by machine learning using the first machine learning device 6B1 and machine learning method described below to determine the correlation between an arbitrary number of thread cones image 31 containing an arbitrary number of thread cones 2 and an arbitrary number of areas surrounding each of the arbitrary number of thread cones 2 contained in the arbitrary number of thread cones image 31.

[0082] The second trained model 5B2 is a model that has been trained by machine learning using the second machine learning device 6B2 and machine learning method described below to determine the correlation between a single yarn cone image 33 including one yarn cone 2 and the remaining amount of yarn of one yarn cone 2 included in the single yarn cone image 33.

[0083] FIG. 9 is a flowchart showing an example of an image processing method by the image processing device 4B according to the second embodiment.

[0084] At a predetermined timing, the acquisition unit 40 acquires the camera-captured image 30 in which the thread cone 2 on the top board 15 is captured by the camera 3 (step S100).

[0085] Next, the thread cone state detection unit 41 refers to the first trained model 5B1 stored in the trained model storage unit 42 and performs an inference process using the first trained model 5B1 (step S111). Specifically, the thread cone state detection unit 41 inputs the camera-captured image 30 acquired by the acquisition unit 40 to the first trained model 5B1 to specify an arbitrary number of regions surrounding each of the arbitrary number of thread cones 2 included in the camera-captured image 30. The inference result of step S111 illustrated in FIG. 9 is that five thread cones 2 are detected from the camera-captured image 30, and thread cone regions A to E surrounding each of the five thread cones 2 are specified by two points, the lower left coordinate and the upper right coordinate.

[0086] Next, the thread cone state detection unit 41 refers to the second trained model 5B2 stored in the trained model storage unit 42 and performs an inference process using the second trained model 5B2 (step S112). Specifically, the thread cone state detection unit 41 inputs area images 32 including one thread cone 2, which are defined in each of an arbitrary number of areas in the camera-captured image 30 acquired by the acquisition unit 40, to the second trained model 5B2, and detects the remaining amount of thread of each of the thread cones 2 included in the area images 32. The inference result of step S112 illustrated in FIG. 9 is that the remaining thread amounts of five thread cones are detected in multi-value form from the five area images 32 identified in each of the five thread cone areas A to E (lower left coordinates and upper right coordinates). The remaining thread amounts of four thread cones 2 surrounded by thread cone areas A to D (lower left coordinates and upper right coordinates) are "100% remaining," "70% remaining," "40% remaining," and "10% remaining," respectively, and the remaining thread amount of one thread cone 2 surrounded by thread cone area E is "no thread."

[0087] Next, in steps S111 and S112, if the result of the inference process performed by the thread cone state detection unit 41 indicates that the thread remaining amount detection result for any of the thread cones 2 is "no thread" (Yes in step S120), the output processing unit 43 notifies "no thread" (step S130), and if the thread remaining amount detection result for any of the thread cones 2 is equal to or less than the thread remaining amount warning value (Yes in step S140), the output processing unit 43 notifies "no thread warning" (step S150). On the other hand, if the thread remaining amount detection result for any of the thread cones 2 does not indicate "no thread" or "no thread warning", that is, if the thread remaining amount detection results for all of the thread cones 2 indicate "thread present" (No in step S120 and No in step S140), the image processing process ends.

[0088] As described above, according to the image processing device 4B and image processing method of this embodiment, like the image processing device 4A and image processing method of the first embodiment, it is possible to detect the state of any number of yarn cones 2 included in the camera-captured image 30 (in this embodiment, the remaining yarn amount is described as an example). Therefore, it is not necessary to install optical components for each yarn cone 2, and the state of the yarn cones 2 can be detected without being restricted by conditions such as the number and positions of the yarn cones 2 when the yarn cones 2 are placed on or around the knitting machine. Therefore, the state of the yarn cones 2 can be detected with a simple configuration, and workability and versatility can be improved.

[0089] (First machine learning device 6B1) Next, a first machine learning device 6B1 used to detect the state of an arbitrary number of yarn cones 2 arranged on or around the flat knitting machine 1 will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is a schematic block diagram showing an example of the first machine learning device 6B1 according to the second embodiment. Fig. 11 is a diagram showing an example of a first inference model 50B1 based on CNN used in the first machine learning device 6B1 according to the second embodiment.

[0090] The first machine learning device 6B1 generates a learning model (first trained model 5B1) that is capable of identifying (estimating) an arbitrary number of areas surrounding each of the arbitrary number of thread cones 2 contained in an arbitrary number of thread cone captured image 31, based on the arbitrary number of thread cones 2.

[0091] The learning dataset acquisition unit 60 acquires the arbitrary number of thread cone captured images 31 as input data from, for example, the external device 7, and acquires an arbitrary number of regions surrounding each of the arbitrary number of thread cones 2 included in the arbitrary number of thread cone captured images 31 as output data from, for example, the terminal device 8. Then, by associating these input data with the output data, one learning dataset is configured and stored in the learning dataset storage unit 61. In the learning dataset illustrated in FIG. 11, the input data is the arbitrary number of thread cone captured images 31 in which five thread cones 2 are included in an area defined by each of five thread cone areas A to E (lower left coordinates and upper right coordinates), and the output data is information specifying the thread cone areas A to E (lower left coordinates and upper right coordinates) surrounding each of the five thread cones 2. Note that the input data constituting the learning dataset may be a single thread cone captured image, part or all of which includes one thread cone 2.

[0092] The machine learning unit 62 performs machine learning using multiple learning datasets stored in the learning dataset memory unit 61, to generate a learning model (first trained model 5B1) that infers the correlation between input data and output data contained in the multiple learning datasets.

[0093] Specifically, the machine learning unit 62 performs machine learning by inputting the arbitrary number of thread cone captured images 31 constituting the learning data set to the input layer 51 of the first inference model 50B1 shown in FIG. 11 and comparing the area (object detection result) output as an inference result from the output layer 53 with the area (teaching data) constituting the learning data set. The inference result illustrated in FIG. 11 is that four thread cones 2 included in each of five thread cone areas A to D (lower left coordinates and upper right coordinates) are detected from the arbitrary number of thread cone captured images 31, and the thread cone areas A to D surrounding each of the five thread cones 2 are identified by two points, the lower left coordinates and the upper right coordinates. If there is an error between the inference result and the teaching data for the thread cone areas A to E, the weight of the first inference model 50B1 is adjusted based on the error.

[0094] Then, when the learning end condition is satisfied and the machine learning is terminated, the machine learning unit 62 stores the first inference model 50B1 in the trained model memory unit 63 as the first trained model 5B1.

[0095] (Second Machine Learning Device 6B2) Next, a second machine learning device 6B2 used to detect the remaining yarn amount of an arbitrary number of yarn cones 2 arranged on or around the flat knitting machine 1 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a schematic block diagram showing an example of the second machine learning device 6B2 according to the second embodiment. Fig. 13 is a diagram showing an example of a second inference model 50B2 based on CNN used in the second machine learning device 6B2 according to the second embodiment.

[0096] The second machine learning device 6B2 generates a learning model (second trained model 5B2) capable of detecting (estimating) the remaining amount of yarn of one yarn cone 2 contained in a single yarn cone image 33 based on the single yarn cone image 33 including the single yarn cone 2.

[0097] The training dataset acquisition unit 60 acquires the single yarn cone captured image 33 as input data from, for example, the external device 7, and acquires the remaining yarn amount of one yarn cone 2 included in the single yarn cone captured image 33 as output data from, for example, the terminal device 8. Then, by associating these input data with the output data, one training dataset is constructed and stored in the training dataset storage unit 61. In the training dataset illustrated in FIG. 13, the input data is the single yarn cone captured image 33 including one yarn cone 2, and the output data is information indicating that the remaining yarn amount of one yarn cone 2 is "70% remaining."

[0098] The machine learning unit 62 performs machine learning using multiple learning datasets stored in the learning dataset memory unit 61, to generate a learning model (second trained model 5B2) that infers the correlation between input data and output data contained in the multiple learning datasets.

[0099] Specifically, the machine learning unit 62 performs machine learning by inputting the single yarn cone captured image 33 constituting the learning data set to the input layer 51 of the second inference model 50B2 shown in FIG. 12, and comparing the remaining yarn amount (state detection result) output as an inference result from the output layer 53 with the remaining yarn amount (teaching data) constituting the learning data set. The inference result illustrated in FIG. 13 indicates that the remaining yarn amount of one yarn cone 2 included in the single yarn cone captured image 33 is "60% remaining." Therefore, since an error occurs between the remaining yarn amount of the thread cone 2 and the teaching data, the weight of the second inference model 50B2 is adjusted based on the error.

[0100] Then, when the learning end condition is satisfied and the machine learning is terminated, the machine learning unit 62 stores the second inference model 50B2 in the trained model memory unit 63 as the second trained model 5B2.

[0101] As described above, according to the first and second machine learning devices 6B1, 6B2 and machine learning method of this embodiment, similar to the machine learning device 6A and machine learning method of the first embodiment, it is possible to generate first and second trained models 5B1, 5B2 that can accurately detect (estimate) the state of any number of thread cones 2 contained in a camera-captured image 30 (in this embodiment, the remaining thread amount is used as an example) from a camera-captured image 30 that contains the any number of thread cones 2.

[0102] In particular, the first machine learning device 6B1 generates a first trained model 5B1 specialized in object detection of the thread cone 2, which identifies an area surrounding each of an arbitrary number of thread cones 2 included in the camera-captured image 30, and the second machine learning device 6B2 generates a second trained model 5B2 specialized in state detection of the thread cone 2, which detects the state of one thread cone 2. Therefore, it is not necessary to prepare a large number of captured images combining various conditions as a training dataset, and it is only necessary to prepare a training dataset suitable for each of the first and second trained models 5B1 and 5B, which reduces the effort of preparing the training dataset and improves the inference accuracy of the first and second trained models 5B1 and 5B2.

[0103] (Third embodiment) In the first embodiment, a machine learning device 6A generates a trained model 5A, and an image processing device 4A inputs a camera-captured image 30 to the trained model 5A, thereby detecting the remaining amount of thread in the thread cone 2 included in the camera-captured image 30. In contrast, in the third embodiment, a machine learning device 6C generates a trained model 5C, and an image processing device 4C inputs a camera-captured image 30 to the trained model 5C, thereby detecting the connection relationship between the thread cone 2 included in the camera-captured image 30 and the tension applying device 16. Other basic configurations and operations are similar to those of the first embodiment, so the following description will focus on the characteristic features of the third embodiment.

[0104] (Image processing device 4C) 14 is a schematic block diagram showing an example of a flat knitting machine 1 equipped with an image processing device 4C according to the third embodiment. A trained model 5C is stored in a trained model storage unit 42 of the image processing device 4C.

[0105] The trained model 5C is a model that has been trained by machine learning using a machine learning device 6C and a machine learning method described later to learn a correlation between an arbitrary number of yarn cones captured image 31 including an arbitrary number of yarn cones 2 and a connection relationship between the arbitrary number of yarn cones 2 and a plurality of tensioning devices 16 included in the arbitrary number of yarn cones captured image 31. Specifically, as shown in FIG. 2, the arbitrary number of yarn cones captured image 31 includes an arbitrary number of yarn cones 2 and a knitting yarn 20 that is unwound from each yarn cone 2 and connected to a tensioning device 16. Therefore, the trained model 5C extracts, from the arbitrary number of yarn cones captured image 31, feature amounts between the positions where the arbitrary number of yarn cones 2 are arranged, the directions in which the knitting yarns 20 unwound from each yarn cone 2 extend, and the positions of the tensioning devices 16 that exist in the directions in which the knitting yarns 20 extend, and trains the feature amounts as the connection relationship between the yarn cones 2 and the tensioning devices 16.

[0106] 15 is a flowchart showing an example of an image processing method by the image processing device 4C according to the third embodiment. In the following description, the tension applying devices 16 are assumed to be n in number and are numbered "1, 2, 3, ..., n" in order from the left side of the device.

[0107] At a predetermined timing, the acquisition unit 40 acquires the camera-captured image 30 in which the thread cone 2 on the top board 15 is captured by the camera 3 (step S100).

[0108] Next, the yarn cone state detection unit 41 refers to the trained model 5C stored in the trained model storage unit 42 and performs an inference process using the trained model 5C (step S113). Specifically, the yarn cone state detection unit 41 inputs the camera-captured image 30 acquired by the acquisition unit 40 to the trained model 5C, and detects the connection relationship between the arbitrary number of yarn cones 2 and the multiple tensioning devices 16 included in the camera-captured image 30 by extracting feature amounts between the positions where the arbitrary number of yarn cones 2 are arranged, the directions in which the knitting yarns 20 unwound from each yarn cone 2 extend, and the positions of the tensioning devices 16 in the directions in which the knitting yarns 20 extend.

[0109] 15 shows that five yarn cones 2 included in each of the five yarn cone regions A to E (lower left coordinate and upper right coordinate) are detected from the camera-captured image 30, and that the knitting yarn 20 extending from the yarn cone 2 surrounded by the yarn cone region A is connected to the "1" tensioning device 16, the knitting yarn 20 extending from the yarn cone 2 surrounded by the yarn cone region C is connected to the "4" tensioning device 16, the knitting yarn 20 extending from the yarn cone 2 surrounded by the yarn cone region D is connected to the "10" tensioning device 16, the knitting yarn 20 extending from the yarn cone 2 surrounded by the yarn cone region B is connected to the "17" tensioning device 16, and no yarn cone 2 is connected to the other tensioning devices. The yarn cone 2 in the "no yarn" state surrounded by the yarn cone region E is detected as not being connected to the tensioning device 16.

[0110] Next, the output processing unit 43 determines whether the connection relationship between the yarn cone 2 and the tension applying device 16 detected by the yarn cone state detection unit 41 in step S113 matches the connection relationship set in the knitting data of the flat knitting machine 1, and if the connection relationship between the two does not match (Yes in step S121), it notifies the worker that an "operational error" has occurred (step S131). On the other hand, if the output processing unit 43 determines that the connection relationship between the two matches (No in step S121), it ends the image processing process.

[0111] As described above, the image processing device 4C and image processing method according to the present embodiment can detect the state of any number of yarn cones 2 included in the camera-captured image 30, and in particular, in the present embodiment, the connection relationship between the yarn cones 2 and the tensioning device 16. Therefore, there is no need to install optical components for each yarn cone 2, and the state of the yarn cones 2 can be detected without being restricted by conditions such as the number and positions of the yarn cones 2 when the yarn cones 2 are placed on or around the knitting machine. Therefore, the state of the yarn cones 2 can be detected with a simple configuration, and workability and versatility can be improved.

[0112] (Machine Learning Device 6C) Next, a machine learning device 6C used for detecting the connection relationship between an arbitrary number of yarn cones arranged on or around the flat knitting machine 1 and a plurality of tension applying devices will be described with reference to Fig. 16 and Fig. 17. Fig. 16 is a schematic block diagram showing an example of the machine learning device 6C according to the third embodiment. Fig. 17 is a diagram showing an example of an inference model 50C based on CNN used in the machine learning device 6C according to the third embodiment.

[0113] The machine learning device 6C generates a learning model (trained model 5C) that is capable of detecting (estimating) the correlation between the arbitrary number of thread cones 2 contained in the arbitrary number of thread cones image 31 containing an arbitrary number of thread cones 2 and the connection relationship of the multiple tensioning devices 16.

[0114] The learning dataset acquisition unit 60 acquires the arbitrary number of thread cone captured images 31 as input data, for example, from the external device 7, and acquires the connection relationship of the arbitrary number of thread cones 2 and the plurality of tensioning devices 16 included in the arbitrary number of thread cone captured images 31 as output data, for example, from the terminal device 8. Then, by associating the input data with the output data, one learning dataset is constructed and stored in the learning dataset storage unit 61. In the learning dataset illustrated in FIG. 17, the input data is the arbitrary number of thread cone captured images 31 including three thread cones 2 in areas defined by each of three thread cone areas A to C (lower left coordinates and upper right coordinates), and the output data is information indicating that the thread cone 2 surrounded by the thread cone area A is connected to the "2nd" tensioning device 16, the thread cone 2 surrounded by the thread cone area B is connected to the "4th" tensioning device 16, and the thread cone 2 surrounded by the thread cone area C is connected to the "15th" tensioning device 16.

[0115] The machine learning unit 62 performs machine learning using multiple learning datasets stored in the learning dataset memory unit 61, to generate a learning model (trained model 5C) that infers the correlation between input data and output data contained in the multiple learning datasets.

[0116] Specifically, the machine learning unit 62 performs machine learning by inputting the arbitrary number of thread cone captured images 31 constituting the learning data set to the input layer 51 of the inference model 50C shown in FIG. 17, and comparing the connection relationship (state detection result) output as an inference result from the output layer 53 with the connection relationship (teaching data) constituting the learning data set. The inference result illustrated in FIG. 17 shows that five thread cones 2 included in each of the three thread cone regions A to C (lower left coordinate and upper right coordinate) are detected from the arbitrary number of thread cone captured images 31, and that the thread cone 2 surrounded by the thread cone region C is connected to the tensioning device "2", the thread cone 2 surrounded by the thread cone region B is connected to the tensioning device "3", and the thread cone 2 surrounded by the thread cone region A is connected to the tensioning device "15". Therefore, an error occurs between the teaching data and the connection relationship of the thread cone 2 surrounded by the thread cone region B, and the weight of the inference model 50C is adjusted based on the error.

[0117] Then, when the learning end condition is satisfied and the machine learning is terminated, the machine learning unit 62 stores the inference model 50C in the trained model memory unit 63 as the trained model 5C.

[0118] In this embodiment, the output layer 53 of the inference model 50C is described as being associated with n tensioning devices 16, and is assigned a combination of the serial number (identification information) of the tensioning device 16 and the area of ​​the thread cone 2 connected to each tensioning device 16. In contrast, the output layer 53 may be associated with any number of areas of thread cones 2, and is assigned a combination of the area of ​​the thread cone 2 and the serial number of the tensioning device 16 connected to each thread cone 2. In that case, the format of the output data constituting the learning dataset can be changed to match the data format of the output layer 53.

[0119] As described above, the machine learning device 6C and machine learning method of this embodiment can generate a trained model 5C that can accurately detect (estimate) the state of any number of thread cones 2 contained in a camera-captured image 30 that includes the any number of thread cones 2, particularly, in this embodiment, the connection relationship between the thread cones 2 and the tensioning device 16.

[0120] (Other embodiments) The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention, all of which are included in the technical concept of the present invention.

[0121] (reasoning device) For example, the present invention can be provided not only in the form of the image processing devices 4A to 4C described above, but also in the form of inference devices 5A, 5B1, 5B2, and 5C for performing inference. In this case, the inference devices 5A, 5B1, 5B2, and 5C can include a memory and at least one processor, and the processor executes a series of processes. The series of processes includes a process of inputting a camera-captured image 30 including an arbitrary number of thread cones, and a process of inferring the state of an arbitrary number of thread cones 2 included in the camera-captured image 30 when the camera-captured image 30 is input (for example, the remaining amount of thread in the thread cone, the connection relationship between the thread cone and the tensioning device, the position where the thread cone is arranged, etc.). By providing the present invention in the form of the inference devices 5A, 5B1, 5B2, and 5C described above, it becomes possible to apply the present invention to various devices more easily than when the image processing devices 4A to 4C are implemented. At this time, when the inference device performs the process of inferring the state of the thread cone 2, as described above, it is possible to apply an inference method implemented by the thread cone state detection unit 41 of the image processing device 4A to 4C using the machine learning devices 6A, 6B1, 6B2, 6A and the trained models 5A, 5B1, 5B2, 5C trained by the machine learning method of the present invention, as will be naturally understood by those skilled in the art.

[0122] In the above embodiment, the image processing devices 4A-4C have been described as being applied to the flat knitting machines 1A-1C. However, the image processing devices 4A-4C may be applied to any type of knitting machine, such as a circular knitting machine or a warp knitting machine. The image processing devices 4A-4C may be attached to the knitting machine at a manufacturing factory for the knitting machine, or may be retrofitted to the knitting machine after shipment from the factory, or the functions of the parts provided in the image processing devices 4A-4C may be added to the control panel 100 of the knitting machine after shipment from the factory.

[0123] In the above embodiment, the image processing devices 4A to 4C have been described as performing inference processing using a trained model as image processing for the camera-captured image 30. Alternatively, the image processing devices 4A to 4C may perform pattern recognition processing, such as pattern matching, for the camera-captured image 30 using recognition parameters that represent the characteristics of the thread cone 2.

[0124] In the above embodiment, the inference models 50A, 50B1, 50B2, and 50C based on CNN are used as the learning method of machine learning, but the present invention is not limited to this, and other learning methods may be adopted as long as the correlation between input and output in the above embodiment can be learned from a learning data set. For example, learning methods such as neural network, ensemble learning (random forest, boosting, etc.), R-CNN (Region-based Convolutional Neural Network) using CNN, Fast-R-CNN, Faster-R-CNN, YOLO (You Only Look Once), and SSD (Single Shot multibox Detector) may be used. In this case, the data format of the input data for the input layer 51 and the output data for the output layer 53 may be appropriately selected. [Explanation of symbols]

[0125] Reference Signs List 1...flat knitting machine, 2, 2A to 2E...yarn cones, 3...camera, 4A to 4C...image processing device, 5A...trained model, 5B1...first trained model, 5B2...second trained model, 5C...trained model, 6A...machine learning device, 6B1...first machine learning device, 6B2...second machine learning device, 6C...machine learning device, 7...external device, 8...terminal device, 11A...lower frame, 11B...upper frame, 12...knitting needle, 13...needle bed, 14...carriage, 15...top plate, 16...tensioning device, 17...side tension section, 18...yarn feeder, 20...knitting yarn, 21... Core section, 30...camera captured image, 31...arbitrary number of yarn cone captured images, 33...single yarn cone captured image, 40...acquisition section, 41...yarn cone state detection section, 42...model storage section, 43...output processing section, 50A...inference model, 50B1...first inference model, 50B2...second inference model, 50C...inference model, 51...input layer, 52...intermediate layer, 53...output layer, 60...learning dataset acquisition section, 61...learning dataset storage section, 62...machine learning section, 63...model storage section, 100...control panel, 101...operation display panel, 520...convolution layer, 521...pooling layer, 522...fully connected layer

Claims

1. An acquisition unit that acquires an image including a number of yarn cones disposed on or around the knitting machine; a thread cone state detection unit that detects states of the arbitrary number of thread cones included in the captured image by performing image processing on the captured image acquired by the acquisition unit, The yarn cone state detection unit is inputting the captured image acquired by the acquisition unit into a first trained model that has been machine-learned to determine a correlation between a captured image including an arbitrary number of thread cones and an arbitrary number of regions surrounding each of the arbitrary number of thread cones included in the captured image; and identifying an arbitrary number of regions surrounding each of the arbitrary number of thread cones included in the captured image; a second trained model that has been machine-learned to learn a correlation between an image including one thread cone and a remaining amount of thread in the one thread cone included in the captured image, the second trained model inputting the image including one thread cone, the remaining amount of thread in the one thread cone included in the captured image, the second trained model being machine-learned to learn a correlation between an image including one thread cone and a remaining amount of thread in the one thread cone included in the captured image, the second trained model inputting the image including one thread cone, the remaining amount of thread in the one thread cone included in the captured image ... Image processing device.

2. An acquisition unit that acquires an image including a number of yarn cones disposed on or around the knitting machine; a thread cone state detection unit that detects states of the arbitrary number of thread cones included in the captured image by performing image processing on the captured image acquired by the acquisition unit, The yarn cone state detection unit is by performing the image processing, a connection relationship between the arbitrary number of yarn cones and a plurality of tension applying devices included in the captured image is detected. Image processing device.

3. The yarn cone state detection unit is The captured image acquired by the acquisition unit is input to a trained model that has been machine-learned to learn a correlation between a captured image including an arbitrary number of thread cones and a connection relationship between the arbitrary number of thread cones and a plurality of tension applying devices included in the captured image, thereby obtaining a correlation between the arbitrary number of thread cones and a connection relationship between the arbitrary number of thread cones and a plurality of tension applying devices included in the captured image. Detecting a connection relationship between the arbitrary number of yarn cones and the plurality of tension applying devices. The image processing device according to claim 2 .

4. 1. A machine learning device used to detect connections between a number of yarn cones and a number of tensioning devices disposed on or around a knitting machine, comprising: a learning dataset storage unit configured to store a plurality of learning datasets each including input data of a captured image including an arbitrary number of thread cones and output data associated with the input data and including a connection relationship between the arbitrary number of thread cones and a plurality of tension applying devices included in the captured image; a machine learning unit that learns a learning model that infers a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; A trained model storage unit that stores the learning model trained by the machine learning unit. Machine learning device.

5. 1. An inference device for use in detecting connections between a number of yarn cones and a number of tensioning devices disposed on or around a knitting machine, comprising: The inference device comprises a memory and at least one processor; The at least one processor A process of inputting a captured image including a number of thread cones; and when the captured image is input, a process of inferring a connection relationship between the arbitrary number of yarn cones included in the captured image and a plurality of tension applying devices. Reasoning device.

Images