Image processing device, machine learning device, and inference device

Abstract

The present invention provides an image processing device etc. which can detect the status of any number of yarn tubes arranged above or around a knitting machine with a simple structure and has high operability and versatility. The image processing device (4A) comprises: an acquisition unit (40) which acquires a captured image; and a yarn tube status detection unit (41) which detects the status of any number of yarn tubes included in the captured image by performing image processing on the captured image acquired by the acquisition unit (40). The yarn tube status detection unit (41) detects the remaining amount of yarn of any number of yarn tubes included in the captured image by inputting the captured image acquired by the acquisition unit (40) into a learned model (5A) obtained by machine learning the correlation between the captured image and the remaining amount of yarn of any number of yarn tubes included in the captured image.

Description

Technical Field

[0001] The present invention relates to an image processing device, a machine learning device and an inference device. Background Art

[0002] In a knitting machine, a plurality of yarn tubes wound with knitting yarns are arranged above or around the knitting machine, and various yarn tubes are used in the knitting operation of knitting knitted fabrics (e.g., knitted products, etc.). The operator of the knitting machine changes the yarn tubes with different types of knitting yarns such as colors and thicknesses according to the knitted fabrics to be knitted. When the knitting yarns in the yarn tubes are exhausted, the knitting operation of the knitting machine stops, and the productivity decreases. Therefore, it is necessary to visually monitor the remaining amount of yarn in the yarn tubes. In addition, the knitting yarn released from the yarn tubes is connected to any one of a plurality of tension applying devices arranged in parallel on the upper part of the knitting machine, and is supplied to the knitting needles in a state where a predetermined tension is applied by the tension applying devices. In order to knit the desired knitted fabric, the user of the knitting machine is required to correctly connect the knitting yarn released from the yarn tubes to the predetermined tension applying devices, but sometimes an operation error occurs in the operation of connecting the yarn tubes to the tension applying devices via the knitting yarns.

[0003] As a device for detecting the remaining amount of yarn on a bobbin, a yarn remaining amount detection device using various detection methods has been proposed. For example, Patent Document 1 discloses a yarn remaining amount detection device that irradiates light to the side of a bobbin and receives the reflected light to detect the remaining amount of yarn. In addition, Patent Document 2 discloses the following yarn remaining amount detection device: when the yarn diameter of the bobbin is reduced, the light irradiated toward the bottom surface of the bobbin passes through the side of the bobbin, and the light receiving sensor receives the reflected light reflected by the reflection mirror on the back side, thereby detecting the remaining amount of yarn.

[0004] [Prior art literature]

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 8-277069

[0007] Patent Document 2: Japanese Patent No. 2871318 Summary of the invention

[0008] Problems to be solved by the invention

[0009] In the yarn remaining amount detection devices disclosed in Patent Documents 1 and 2, optical system components such as optical sensors and reflection mirrors need to be arranged near each yarn bobbin, so the number of optical system components increases. In an environment where yarn bobbins are densely arranged, the arrangement space of the optical system components is limited, and the optical path adjustment during assembly is also difficult. In addition, in order to detect the remaining amount of yarn, the relative arrangement relationship of the yarn bobbin with respect to the optical system components is uniquely determined. Therefore, when the operator arranges the yarn bobbin at an arbitrary position, the arrangement of the yarn bobbin will inevitably deviate slightly from the detection range of the optical system components, and the remaining amount of yarn cannot be detected, which leads to poor operability and lack of versatility. In addition, in the yarn remaining amount detection devices disclosed in Patent Documents 1 and 2, it is impossible to detect which tension applying device the knitting yarn released from the yarn bobbin is connected to, that is, it is impossible to detect the connection relationship between the yarn bobbin and the tension applying device.

[0010] In view of the above-mentioned problems, the present invention aims to provide an image processing device, a machine learning device and an inference device, which can detect the status of yarn bobbins such as the remaining amount of yarn of any number of yarn bobbins arranged above or around a knitting machine, the connection relationship between the yarn bobbins and the tension applying device, and the arranged position with a simple structure, and have high operability and versatility.

[0011] Means for solving problems

[0012] In order to achieve the above-mentioned object, an image processing device (4A, 4B, 4C) according to a first aspect of the present invention comprises:

[0013] an acquisition unit (40) for acquiring a captured image including any number of yarn tubes arranged above or around the knitting machine; and

[0014] A bobbin state detecting section (41) detects the states of the arbitrary number of bobbins included in the captured image by performing image processing on the captured image acquired by the acquiring section (40).

[0015] In order to achieve the above object, a second aspect of the present invention provides a machine learning device (6A) for detecting the remaining amount of yarn of an arbitrary number of yarn bobbins arranged above or around a knitting machine, comprising:

[0016] A learning data set storage unit (61) stores a plurality of learning data sets, wherein the learning data sets are composed of input data and output data, wherein the input data are composed of captured images containing an arbitrary number of yarn bobbins, and the output data are associated with the input data and are composed of the remaining amount of yarn of the arbitrary number of yarn bobbins contained in the captured images;

[0017] a machine learning unit that learns a learning model for inferring a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and

[0018] The learned model storage unit stores the learned model learned by the machine learning unit.

[0019] In order to achieve the above object, the third aspect of the present invention is an inference device (5A) for detecting the remaining amount of yarn of any number of yarn tubes arranged above or around the knitting machine.

[0020] The inference device comprises a memory and at least one processor.

[0021] The at least one processor is configured to perform the following processing:

[0022] Input captured images containing any number of yarn tubes; and

[0023] When the captured image is input, the remaining amount of yarn of the arbitrary number of yarn bobbins included in the captured image is estimated.

[0024] In order to achieve the above object, a fourth aspect of the present invention provides a machine learning device (6B1) for detecting the states of any number of yarn bobbins arranged above or around a knitting machine, comprising:

[0025] A learning data set storage unit (61) stores a plurality of learning data sets, wherein the learning data sets are composed of input data and output data, wherein the input data are composed of captured images containing an arbitrary number of yarn tubes, and the output data are associated with the input data and are composed of area information that specifies an arbitrary number of areas surrounding each of the arbitrary number of yarn tubes contained in the captured images;

[0026] A machine learning unit (62) learns a learning model for inferring the correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and

[0027] A learned model storage unit (63) stores the learned model learned by the machine learning unit (62).

[0028] In order to achieve the above object, a fifth aspect of the present invention is an inference device (5B1) for detecting the states of any number of yarn bobbins arranged above or around a knitting machine.

[0029] The inference device comprises a memory and at least one processor.

[0030] The at least one processor is configured to perform the following processing:

[0031] Input captured images containing any number of yarn tubes; and

[0032] When the captured image is input, region information defining an arbitrary number of regions surrounding each of the arbitrary number of yarn bobbins included in the captured image is estimated.

[0033] In order to achieve the above object, a sixth aspect of the present invention is a machine learning device (6B2) for detecting the remaining amount of yarn of an arbitrary number of yarn bobbins arranged above or around a knitting machine, comprising:

[0034] A learning data set storage unit (61) stores a plurality of learning data sets, wherein the learning data sets are composed of input data and output data, wherein the input data is composed of a captured image including a yarn bobbin, and the output data is associated with the input data and is composed of the remaining amount of yarn of the yarn bobbin included in the captured image;

[0035] A machine learning unit (62) learns a learning model for inferring the correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and

[0036] A learned model storage unit (63) stores the learned model learned by the machine learning unit (62).

[0037] In order to achieve the above object, a seventh aspect of the present invention is an inference device (5B2) for detecting the remaining amount of yarn of any number of yarn bobbins arranged above or around a knitting machine.

[0038] The inference device comprises a memory and at least one processor.

[0039] The at least one processor is configured to perform the following processing:

[0040] Input a captured image containing a yarn bobbin; and

[0041] When the captured image is input, the remaining amount of the yarn of the one yarn bobbin included in the captured image is estimated.

[0042] In addition, in order to achieve the above-mentioned object, an eighth aspect of the present invention is a machine learning device (6C) for detecting the connection relationship between an arbitrary number of yarn bobbins arranged above or around a knitting machine and a plurality of tension applying devices, comprising:

[0043] A learning data set storage unit (61) stores a plurality of learning data sets, wherein the learning data sets are composed of input data and output data, wherein the input data are composed of captured images containing an arbitrary number of yarn bobbins, and the output data are corresponding to the input data and are composed of a connection relationship between the arbitrary number of yarn bobbins and a plurality of tension applying devices contained in the captured images;

[0044] A machine learning unit (62) learns a learning model for inferring the correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and

[0045] A learned model storage unit (63) stores the learned model learned by the machine learning unit (62).

[0046] In order to achieve the above object, a ninth aspect of the present invention provides an inference device (5C) for detecting a connection relationship between an arbitrary number of yarn bobbins arranged above or around a knitting machine and a plurality of tension applying devices.

[0047] The inference device comprises a memory and at least one processor.

[0048] The at least one processor is configured to perform the following processing:

[0049] Inputting a captured image containing any number of yarn tubes; and

[0050] When the captured image is input, the connection relationship between the arbitrary number of yarn bobbins and the plurality of tension applying devices included in the captured image is inferred.

[0051] Effects of the Invention

[0052] According to the image processing device (4A to 4C) and the inference device (5A, 5B1, 5B2, 5C) of the present invention, the states of any number of bobbins included in the captured image (for example, the remaining amount of yarn in the bobbins, the connection relationship between the bobbins and the tension applying device, the positions where the bobbins are arranged, etc.) can be detected. Therefore, it is not necessary to provide optical components for each bobbin, and the states of the bobbins can be detected without being restricted by conditions such as the number and position of the bobbins when the bobbins are arranged above or around the knitting machine. Therefore, the states of the bobbins can be detected with a simple structure, and the workability and versatility can be improved.

[0053] Furthermore, according to the machine learning device (6A, 6B1, 6B2, 6C) of the present invention, it is possible to generate a learned model (5A, 5B1, 5B2, 5C) that can accurately detect (infer) the states of any number of yarn bobbins contained in an image captured by a camera (for example, the remaining amount of yarn on the yarn bobbin, the connection relationship between the yarn bobbin and the tension applying device, the position at which the yarn bobbin is configured, etc.) based on an image captured by a camera including any number of yarn bobbins. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a perspective view showing an example of the flat knitting machine 1 according to the first embodiment.

[0055] Figure 2 1 is a diagram showing an example of a camera image 30 captured by the camera 3 .

[0056] Figure 3 This is a schematic block diagram showing an example of a flat knitting machine 1 including the image processing device 4A according to the first embodiment.

[0057] Figure 4 This is a flowchart showing an example of an image processing method by the image processing device 4A of the first embodiment.

[0058] Figure 5 This is a schematic block diagram showing an example of the machine learning device 6A according to the first embodiment.

[0059] Figure 6 This is a diagram showing an example of an estimation model 50A based on CNN used in the machine learning device 6A of the first embodiment.

[0060] Figure 7 : is a flowchart showing an example of the machine learning method according to the first embodiment.

[0061] Figure 8 This is a schematic block diagram showing an example of a flat knitting machine 1 including an image processing device 4B according to the second embodiment.

[0062] Fig. 9 This is a flowchart showing an example of an image processing method by the image processing device 4B of the second embodiment.

[0063] Fig.10 This is a schematic block diagram showing an example of the first machine learning device 6B1 according to the second embodiment.

[0064] Fig.11 This is a diagram showing an example of a first estimation model 50B1 based on CNN used in the first machine learning device 6B1 of the second embodiment.

[0065] Fig.12This is a schematic block diagram showing an example of the second machine learning device 6B2 according to the second embodiment.

[0066] Fig.13 This is a diagram showing an example of a second estimation model 50B2 based on CNN used in the second machine learning device 6B2 of the second embodiment.

[0067] Fig.14 This is a schematic block diagram showing an example of a flat knitting machine 1 including an image processing device 4C according to the third embodiment.

[0068] Fig.15 This is a flowchart showing an example of an image processing method of the image processing device 4C according to the third embodiment.

[0069] Fig.16 This is a schematic block diagram showing an example of a machine learning device 6C according to the third embodiment.

[0070] Fig.17 This is a diagram showing an example of an inference model 50C based on CNN used in a machine learning device 6C according to the third embodiment. DETAILED DESCRIPTION

[0071] Hereinafter, various embodiments for implementing the present invention will be described with reference to the accompanying drawings. It should be noted that the following schematically shows the scope required for the description of the purpose of the present invention, mainly describing the scope required for the description of the corresponding parts of the present invention, and the parts omitted for description are based on known technologies.

[0072] (First Embodiment)

[0073] Figure 1 It is a perspective view showing an example of the flat knitting machine 1 according to the first embodiment.

[0074] The flat knitting machine 1 includes: a lower frame 11A, which is installed on the ground; a pair of needle beds 13, which support a plurality of knitting needles 12 so as to be able to move forward and backward, and are arranged on the lower frame 11A in a front-rear facing state; a cam carriage 14, which moves along the needle beds 13 to move the knitting needles 12 forward and backward; and a top plate 15, which accommodates the needle beds 13 and the cam carriage 14, and on which an arbitrary number of bobbins 2 are placed. In addition, the flat knitting machine 1 may also include a creel frame on which an arbitrary number of bobbins 2 are placed.

[0075] In addition, the flat knitting machine 1 is provided with a plurality of tension applying devices 16 mounted on the upper frame 11B and arranged on the upper part of the upper plate 15, a side tensioning unit 17 provided on the left and right of the lower frame 11A, and a plurality of yarn feeders 18 which can be selectively connected to the carriage 14 when the carriage 14 moves as a mechanism for supplying knitting yarn 20 from an arbitrary number of yarn bobbins 2 placed on the top plate 15 to the knitting needles 12. The knitting yarn 20 from the yarn bobbins 2 is supplied in the order of the tension applying devices 16, the side tensioning units 17, the yarn feeders 18, and the knitting needles 12, thereby knitting a knitted fabric.

[0076] Furthermore, as a system for detecting the states of an arbitrary number of bobbins 2 placed on the top plate 15, the flat knitting machine 1 includes two left and right cameras 3 mounted on the upper frame 11B and an image processing device 4A housed in the lower frame 11A.

[0077] The camera 3 is composed of an image sensor such as a CMOS sensor or a CCD sensor. The camera 3 shoots the top plate 15 from above the top plate 15, and adjusts the position and direction so that the bobbin 2 placed on the top plate 15 is within the field of view of the camera 3. In addition, the camera 3 may also have the functions of panning, tilting, and zooming. In addition, the number of cameras 3 is not limited to two, and may be one or more than three. The installation position of the camera 3 may also be appropriately changed, for example, it may be installed on the ceiling above the flat knitting machine 1.

[0078] Figure 2 1 is a diagram showing an example of a camera image 30 captured by the camera 3 .

[0079] The camera image 30 includes an arbitrary number of yarn bobbins 2 placed on the top plate 15. Figure 2 In the example of , five bobbins 2A to 2E are included. The bobbins 2 are wound with knitting yarns 20 on the truncated cone-shaped core 21, and can be placed at any position on the upper surface of the top plate 15. Therefore, the number of bobbins 2 included in the camera-captured image 30 is not a fixed number, but an arbitrary number. In addition, the camera-captured image 30 includes not only the bobbins 2 placed on the top plate 15, but also any number of bobbins 2 placed on the creel provided around the flat knitting machine 1, etc.

[0080] In addition, the photographed state (e.g., outer shape, outer dimensions) of the yarn bobbin 2 included in the camera-photographed image 30 varies not only according to the outer diameter of the core 21, the thickness of the knitting yarn 20, the number of turns, and the use state of the knitting yarn 20, but also according to the relative positional relationship between the yarn bobbin 2 and the camera 3. In addition, in the camera-photographed image 30, the yarn bobbin 2 on the back side is sometimes photographed in a state where a part of the yarn bobbin 2 is blocked by the yarn bobbin 2 on the front side when viewed from the camera 3. In addition, in addition to an arbitrary number of yarn bobbins 2, the camera-photographed image 30 also includes the knitting yarn 20 released from each yarn bobbin 2 and connected to the tension applying device 16, and may also include objects other than the yarn bobbin 2, for example, the tension applying device 16.

[0081] exist Figure 2 Of the five yarn bobbins 2A to 2E included in the camera-photographed image 30 shown, the yarn bobbins 2A are in a state immediately after being placed on the top plate 15, i.e., in an unused state. Moreover, as the knitting yarn 20 is gradually used from the unused state, the outer diameter becomes smaller, and the remaining amount of yarn becomes less, as in the case of the yarn bobbins 2B and 2C. Moreover, after a portion of the core 21 is visually confirmed from the gap of the knitting yarn 20, as in the case of the yarn bobbins 2D, the yarn bobbins 2E finally become in a "yarn-free" state, and the core 21 is exposed as a whole. In addition, the four yarn bobbins 2A to 2D are in a "yarn-containing" state, and the remaining amount of yarn is indicated as "100% remaining", "70% remaining", "40% remaining", and "10% remaining" in more detail.

[0082] Figure 3 This is a schematic block diagram showing an example of a flat knitting machine 1 including the image processing device 4A according to the first embodiment.

[0083] As an electrical structure, the flat knitting machine 1 is provided with a control panel 100 composed of a microcontroller, etc., and an operation display panel 101 composed of a touch panel, switches, etc., in addition to the camera 3 and the image processing device 4A. The control panel 100 is connected to actuators such as motors and sensors (none of which are shown in the figure), and controls the knitting action of the flat knitting machine 1 by controlling various parts of the flat knitting machine 1. The operation display panel 101 receives the operation of the operator and outputs various information in the form of display and sound.

[0084] (Image Processing Device 4A)

[0085] The image processing device 4A is composed of a general-purpose or special-purpose computer, and includes, for example, a computing unit such as a CPU and a GPU; a storage unit composed of a ROM and a RAM; a communication unit for communicating with a network or other devices by wire or wireless; and a bus connecting these units. In addition, the image processing device 4A can also be assembled in the control panel 100 as a function of a part of the control panel 100. In addition, the image processing device 4A is not only provided as a device that acts alone, but also includes a device provided in the form of a non-volatile computer-readable medium storing a program for causing an arbitrary processor to perform the action described below or one or more commands for performing the action.

[0086] like Figure 3 As shown, the image processing device 4A includes an acquisition unit 40, a bobbin state detection unit 41, a learned model storage unit 42, and an output processing unit 43. The acquisition unit 40, the bobbin state detection unit 41, and the output processing unit 43 are composed of the above-mentioned calculation means, and the learned model storage unit 42 is composed of the above-mentioned storage means.

[0087] The acquisition unit 40 is connected to the camera 3 and functions as an interface for acquiring the image captured by the camera 3. The acquisition unit 40 acquires the camera-captured images 30 including any number of bobbins 2 arranged above or around the flat knitting machine 1 (the upper surface of the top plate 15 in this embodiment) as the camera-captured images 30 captured by the camera 3. In addition, the connection method and communication method when the image processing device 4A is connected to the camera 3 via the acquisition unit 40 may be any method.

[0088] The bobbin state detection unit 41 detects the states of any number of bobbins 2 included in the camera image 30 by performing image processing on the camera image 30 acquired by the acquisition unit 40. Examples of the states of the bobbins 2 include the remaining amount of yarn in the bobbins 2, the position, shape and size of the bobbins 2, the type (color, thickness) of the knitting yarn 20, and the connection relationship between the bobbins 2 and the tension applying device 16. The bobbin state detection unit 41 may detect one of the various states of the bobbins 2 described above, or may detect a plurality of states.

[0089] In the present embodiment, a case where the bobbin state detection unit 41 detects the remaining amount of yarn on the bobbin 2 as the state of an arbitrary number of bobbins 2 is described. Furthermore, a case where the bobbin state detection unit 41 performs a series of processing, as image processing on the camera-photographed image 30, of inputting the camera-photographed image 30 as input data to the learned model 5A and outputting an inference result (the remaining amount of yarn on the bobbin 2) from the learned model 5A through inference processing using the learned model 5A, is described.

[0090] The learned model 5A performs machine learning on the correlation between the camera image 30 including an arbitrary number of bobbins and the remaining yarn amounts of the arbitrary number of bobbins 2 included in the camera image 30 by machine learning using a machine learning device 6A and a machine learning method described later.

[0091] The bobbin state detection unit 41 includes not only a function of performing an inference process using the learned model 5A, but also a function of applying a predetermined image adjustment (e.g., image format, image size, image filter, image mask, etc.) to the camera image 30 as input data acquired by the acquisition unit 40 and inputting the image to the learned model 5A as a pre-processing of the inference process, and a function of applying, e.g., a predetermined calculation expression or logical expression to the output data (inference result) output by the learned model 5A as a post-processing of the inference process, thereby finally determining the remaining amount of the yarn of the bobbin 2. The inference result of the bobbin state detection unit 41 may be stored in a storage unit (not shown), and the stored past inference results may be used as a learning data set used for online learning or re-learning for further improving the inference accuracy of the learned model 5A in the learned model storage unit 42.

[0092] The output processing unit 43 is used to output the detection result detected by the bobbin state detection unit 41, that is, the state of the bobbin 2. Various units can be used as specific output units. For example, the detection result can be reported to the operator through the operation display panel 101 by display or sound, or stored in the storage unit of the control panel 100 as the operation history of the flat knitting machine 1, or sent to the upper production management system of the flat knitting machine 1. At this time, the output processing unit 43 can also obtain necessary data (such as knitting data, etc.) from the flat knitting machine 1.

[0093] Reference Figure 4 An image processing method of the image processing device 4A having the above-described configuration will be described. Figure 4 This is a flowchart showing an example of an image processing method of the image processing device 4A of the first embodiment. Figure 4 The series of image processing steps shown are repeatedly executed by the image processing device 4A at a predetermined timing. The predetermined timing may be any timing, for example, every predetermined time interval, every production quantity, or when a predetermined event occurs (operator operation, preparation, stop, etc.). The following describes the execution of the image processing step during the knitting operation of the flat knitting machine 1.

[0094] When the knitting operation by the flat knitting machine 1 is performed, the knitting yarn 20 is supplied from the yarn tube 2 placed on the top plate 15 to knit the knitted fabric. Then, when the above-mentioned predetermined timing is reached, the acquisition unit 40 acquires the camera image 30 obtained by the camera 3 capturing the yarn tube 2 on the top plate 15 (step S100).

[0095] Next, the bobbin state detection section 41 performs an estimation process using the learned model 5A by referring to the learned model 5A stored in the learned model storage section 42 (step S110). At this time, the learned model 5A used for the estimation process is preferably determined in advance based on a predetermined program or an operator's operation.

[0096] Specifically, the yarn bobbin state detection unit 41 performs pre-processing on the camera image 30 acquired by the acquisition unit 40, and inputs it into the learned model 5A, and performs post-processing on the output data from the learned model 5A, thereby detecting the remaining yarn amount of any number of yarn bobbins 2 contained in the camera image 30 as the inference result of the learned model 5A. Figure 4 The inference result of step S110 shown as an example is a result of detecting five bobbins surrounded by five bobbin regions A to E (lower left coordinates and upper right coordinates) from the camera captured image 30, and detecting the remaining yarn amounts of the five bobbins 2 using multiple values. Specifically, the remaining yarn amounts of the four bobbins 2 surrounded by the four bobbin regions A to D are "100% remaining", "70% remaining", "40% remaining", and "10% remaining", respectively, and the remaining yarn amount of the one bobbin 2 surrounded by the bobbin region E is "no yarn".

[0097] Next, the output processing unit 43 determines whether the detection result of the remaining yarn amount for any one of the yarn bobbins 2 indicates "no yarn" based on the result of the inference processing performed by the yarn bobbin state detection unit 41 in step S110 (step S120). Then, in the case where the output processing unit 43 makes an affirmative determination in step S120 ("yes" in step S120), for example, the operator is informed through the operation display panel 101 that the inference result is "no yarn", that is, the weaving action of the flat knitting machine 1 is stopped (step S130). Figure 4 In the illustrated estimation result, “no yarn” is reported for the bobbin 2 located at the position specified by the bobbin area E. At this time, the output processing section 43 may stop the knitting operation of the flat knitting machine 1 by reporting the detection result of “no yarn” to the control panel 100 .

[0098] On the other hand, when the detection result of the remaining yarn amount for all the bobbins 2 is not "no yarn", and the output processing unit 43 makes a negative determination in step S120 ("No" in step S120), it further determines whether the detection result of the remaining yarn amount for any one bobbin 2 indicates that it is less than a predetermined yarn remaining amount forecast value (for example, 10%) (step S140). Then, when the output processing unit 43 makes an affirmative determination in step S140 ("Yes" in step S140), it reports to the operator that the inferred result is "no yarn forecast", that is, the bobbin 2 is about to become "no yarn", and the knitting operation of the flat knitting machine 1 is about to stop (step S150). At this time, the output processing unit 43 may also notify the control panel 100 that the detection result is "no yarn forecast".

[0099] On the other hand, if the detection result of the remaining yarn amount for all the bobbins 2 is not less than the remaining yarn amount prediction value, and the output processing unit 43 makes a negative determination in step S140 ("No" in step S140), the image processing step is terminated without reporting to the operator. In addition, the output processing unit 43 may output that the detection result is not "no yarn" and "no yarn prediction", that is, the detection result is "yarn present", and may notify the control panel 100, for example.

[0100] The output processing unit 43 outputs "no yarn" and "no yarn notice" as the detection result, but may output either one. In this case, any one of steps S120, S130, S140, and S150 may be omitted. In addition, steps S120 and S130 may be omitted so that when the tension applying device 16 has the function of detecting "no yarn", the tension applying device 16 detects "no yarn" and the output processing unit 43 outputs "no yarn notice" as the detection result. Furthermore, when the remaining yarn amount is detected as two values, for example, the detection result of the remaining yarn amount of the four bobbins 2 surrounded by the bobbin areas A to D is "yarn present", and the detection result of the remaining yarn amount of the one bobbin 2 surrounded by the bobbin area E is "yarn absent", steps S140 and S150 may be omitted.

[0101] As described above, according to the image processing device 4A and the image processing method of the present embodiment, the states of any number of yarn bobbins 2 included in the camera-photographed image 30 can be detected (the remaining amount of yarn is described as an example in the present embodiment). Therefore, it is not necessary to provide optical components for each yarn bobbin 2, and the states of the yarn bobbins 2 can be detected without being restricted by the number and position of the yarn bobbins 2 when the yarn bobbins 2 are arranged above or around the knitting machine. Therefore, the states of the yarn bobbins 2 can be detected with a simple structure, and the workability and versatility can be improved.

[0102] (Machine Learning Device 6A)

[0103] Next, refer to Figure 5 A machine learning device 6A used to detect the remaining amounts of yarn of an arbitrary number of bobbins 2 arranged above or around the flat knitting machine 1 will be described. Figure 5 This is a schematic block diagram showing an example of the machine learning device 6A according to the first embodiment.

[0104] The machine learning device 6A generates a learning model (learned model 5A) that can detect (estimate) the remaining amount of yarn of the yarn bobbin 2 based on an arbitrary number of yarn bobbin captured images 31 including an arbitrary number of yarn bobbins 2. In addition, the learned model 5A generated by the machine learning device 6A is provided to the image processing device 4A and stored in the learned model storage unit 42, and is used for the inference processing performed by the yarn bobbin state detection unit 41. In addition, here, the image processing device 4A and the machine learning device 6A are shown as different devices, but the image processing device 4A and the machine learning device 6A may be configured as a single device.

[0105] The machine learning device 6A is composed of a general-purpose or special-purpose computer, and includes, for example: a computing unit such as a CPU and a GPU; a storage unit composed of a ROM and a RAM; a communication unit for communicating with a network or other devices by wire or wireless; and a bus connecting these units. In addition, the machine learning device 6A is provided not only as a device that acts alone, but also in the form of a non-volatile computer-readable medium storing a program for causing an arbitrary processor to perform the actions described below or one or more instructions for performing the actions.

[0106] like Figure 5 As shown, the machine learning device 6A includes a learning data set acquisition unit 60, a learning data set storage unit 61, a machine learning unit 62, and a learned model storage unit 63. The learning data set acquisition unit 60 and the machine learning unit 62 are composed of the above-mentioned computing units, and the learning data set storage unit 61 and the learned model storage unit 63 are composed of the above-mentioned storage units.

[0107] The learning data set acquisition unit 60 is, for example, an interface unit for acquiring a plurality of data constituting a learning (training) data set from various devices connected via a communication unit. Here, various devices connected to the learning data set acquisition unit 60 include, for example, an external device 7 and a terminal device 8 used by an operator. Figure 5 In the figure, the machine learning device 6A, the external device 7 and the terminal device 8 are represented as different devices, but these devices can also be appropriately combined. For example, the external device 7 and the terminal device 8 can also be composed of a single device, and the machine learning device 6A, the external device 7 and the terminal device 8 can also be composed of a single device.

[0108] The learning data set acquisition unit 60 acquires, for example, an arbitrary number of yarn bobbin images 31 as input data from the external device 7, and acquires, for example, the remaining amount of yarn of an arbitrary number of yarn bobbins 2 included in the arbitrary number of yarn bobbin images 31 as output data from the terminal device 8. Then, by establishing correspondence between these input data and output data, one learning data set is constructed.

[0109] like Figure 2 As shown, the arbitrary number of yarn bobbins captured image 31 is an image including an arbitrary number of yarn bobbins 2. The arbitrary number of yarn bobbins captured image 31 may be an image obtained by photographing an arbitrary number of yarn bobbins 2 arranged above or around the flat knitting machine 1 (in this embodiment, the upper surface of the top plate 15) by the camera 3. For example, it may be an image obtained by photographing an arbitrary number of yarn bobbins 2 arranged on a simulation table simulating the top plate 15 by a camera for data generation.

[0110] The remaining amount of yarn of the bobbin 2 is information indicating the remaining amount of the knitting yarn 20 wound on the core 21 of the bobbin 2, and various data formats can be used. For example, the remaining amount of yarn of the bobbin 2 can be composed of information indicating either "no yarn" or "yarn". In this case, the remaining amount of yarn is classified into two values, for example, the value indicating "no yarn" is set to "0" and the value indicating "yarn" is set to "1", and the operator can input the corresponding value using the terminal device 8 in a form corresponding to the input data. In addition, the remaining amount of yarn of the bobbin 2 can be composed of staged information. In this case, the remaining yarn amount is classified into multiple values. For example, when expressed in 11 stages, the value indicating "no yarn" is set to "0", and the value indicating the remaining yarn amount of "10% remaining", "20% remaining", ..., "100% remaining" (unused state) is set to "0.1", "0.2", ..., "1", and the operator can use the terminal device 8 to input the corresponding value in a form corresponding to the input data.

[0111] The learning data set storage unit 61 is a database for storing the learning data set acquired by the learning data set acquisition unit 60. The specific structure of the database constituting the learning data set storage unit 61 can also be changed appropriately. Figure 5 In FIG. 1 , the learning data set storage unit 61 and the learned model storage unit 63 are shown as different storage units, but they may be constituted by a single storage unit (database).

[0112] The machine learning unit 62 implements machine learning using a plurality of learning data sets stored in the learning data set storage unit 61, thereby generating a learning model (learned model 5A) that infers the correlation between the input data and output data contained in the plurality of learning data sets. In addition, in the embodiment, as a specific learning method for machine learning, a convolutional neural network (CNN (Convolutional Neural Network)) described later is adopted.

[0113] The learned model storage unit 63 is a database for storing the learned model 5A generated by the machine learning unit 62. The learned model 5A stored in the learned model storage unit 63 is applied to the actual system (image processing device 4A) via a communication line including the Internet and a storage medium in response to a request.

[0114] Next, refer to Figure 6 The following describes a learning method in the machine learning unit 62 using the plurality of learning data sets acquired as described above. Figure 6 This is a diagram showing an example of an estimation model 50A based on CNN used in the machine learning device 6A of the first embodiment.

[0115] The inference model 50A includes an input layer 51, an intermediate layer 52, and an output layer 53. The inference model 50A detects an object included in an image (object detection) and detects the state of the detected object (state detection). When the image contains multiple objects, object detection and state detection are performed on each object.

[0116] The input layer 51 has neurons whose number corresponds to the image as input data, and the pixel value of each pixel is input to each neuron.

[0117] The middle layer 52 is composed of a convolution layer 520, a pooling layer 521, and a fully connected layer 522. The convolution layer 520 and the pooling layer 521 are alternately provided with multiple layers, for example. The convolution layer 520 and the pooling layer 521 extract feature quantities from the image input via the input layer 51. The fully connected layer 522 converts the feature quantities extracted from the image by the convolution layer 520 and the pooling layer 521, for example, through an activation function, and outputs them as feature vectors. In addition, the fully connected layer 522 can also be provided with multiple layers.

[0118] The output layer 53 outputs output data including object detection results and state detection results as inference results based on the feature vector output from the fully connected layer 522. The object detection result is, for example, information that specifies the area (e.g., rectangle, circle, etc.) surrounding the detected object. In the case of a rectangle, it can be the lower left coordinate and the upper right coordinate, or it can be the center coordinate and the width and height. The state detection result is, for example, information indicating the state of the detected object, which can be a discrete value or a continuous value. In addition to these, the output data may also include, for example, a score indicating the certainty of the estimation result.

[0119] Synapses that connect neurons between the layers are extended between the layers of the inference model 50A, and weights are corresponding to the synapses of the convolutional layer 520 and the fully connected layer 522 of the intermediate layer 52.

[0120] The machine learning unit 62 of the present embodiment inputs the learning data set into the inference model 50A, and learns the correlation between the arbitrary number of yarn bobbin images 31 and the yarn remaining amount of the arbitrary number of yarn bobbins 2 included in the arbitrary number of yarn bobbin images 31. Specifically, the machine learning unit 62 inputs the arbitrary number of yarn bobbin images 31 constituting the learning data set as input data into the input layer 51 of the inference model 50A. In addition, as a pre-processing when the arbitrary number of yarn bobbin images 31 are input into the input layer 51, the machine learning unit 62 may also perform a predetermined image adjustment (for example, image format, image size, image filter, image mask, etc.) on the arbitrary number of yarn bobbin images 31.

[0121] Then, the machine learning unit 62 uses an error function that compares the remaining yarn amount (state detection result) represented by the output data output from the output layer 53 as an inference result and the remaining yarn amount (teacher data) constituting the learning data set, and repeatedly adjusts the weights corresponding to each synapse (back propagation algorithm) to reduce the evaluation value of the error function.

[0122] Next, when the machine learning unit 62 determines that the specified learning termination conditions are met, such as the series of steps described above are repeated a specified number of times and the evaluation value of the error function is less than the allowable value, the learning is terminated and the inference model 50A (all weights corresponding to each synapse are established) is saved as the learned model 5A in the learned model storage unit 63.

[0123] (Machine Learning Methods)

[0124] Next, in association with the above-mentioned machine learning device 6A, the machine learning method is referred to Figure 7 Provide explanation. Figure 7 This is a flowchart showing an example of the machine learning method according to the first embodiment.

[0125] In the machine learning method shown below, the case where it is executed by the above-mentioned machine learning device 6A is described. First, as a preliminary preparation for starting machine learning, a desired number of learning data sets are prepared, and the prepared plurality of learning data sets are stored in the learning data set storage unit 61 (step S200). The number of learning data sets prepared here can be set in consideration of the inference accuracy required for the final learned model 5A.

[0126] There are several methods for preparing the learning data set. For example, during the knitting operation of the flat knitting machine 1, the camera 3 is used to capture the yarn bobbins 2 in various states, and the operator uses the terminal device 8 or the like to determine and input the yarn remaining amount of the yarn bobbins 2 included in the arbitrary number of yarn bobbins captured images 31 in a manner associated with the captured arbitrary number of yarn bobbins, thereby preparing input data and output data constituting the learning data set. In addition, a method of preparing a desired number of learning data sets by repeatedly performing such operations may also be adopted. In addition, in addition to such a method, various methods may be adopted, such as obtaining the learning data set by capturing the yarn bobbins 2 arranged on a simulation table simulating the top plate 15 using a data generation camera. In this case, it is preferable to prepare, as the learning data set, not only the arbitrary number of yarn bobbins captured images 31 including the yarn bobbins 2 in various states of yarn remaining amount with and without yarn, but also the arbitrary number of yarn bobbins captured images 31 including various numbers of yarn bobbins 2 arranged at various positions and having various external shapes and external dimensions.

[0127] Next, in order to start learning in the machine learning unit 62, the inference model 50A before learning is prepared (S210). The inference model 50A before learning prepared here has, for example, Figure 6 The structure shown in the example is constructed, and the weight of each synapse is set to an initial value. Then, a learning data set is randomly selected from a plurality of learning data sets stored in the learning data set storage unit 61 (step S220), and the input data in the one learning data set is input to the prepared pre-learning inference model 50A (step S230). Figure 6 In the illustrated learning data set, the input data is an arbitrary number of yarn tube images 31 including five yarn tubes 2 in the areas respectively defined by five yarn tube areas A to E (lower left coordinates and upper right coordinates), and the output data is information indicating that the remaining amounts of yarn of the five yarn tubes 2 respectively surrounded by the five yarn tube areas A to E are "100% remaining", "70% remaining", "40% remaining", "10% remaining", and "no yarn", respectively.

[0128] The remaining amount of yarn (status detection result) outputted from the output layer 53 as the inference result as a result of the above-mentioned step S230 is generated by the neural network model before learning, and therefore represents a value different from the result preferred in almost all cases, that is, information different from the remaining amount of yarn of the correct bobbin 2. Therefore, next, machine learning is performed using the remaining amount of yarn as teacher data in one learning data set acquired in step S220 and the remaining amount of yarn (status detection result) outputted from the output layer 53 in step S230 (step S240).

[0129] The machine learning here is, for example, a process (back propagation algorithm) of comparing the remaining amount of yarn constituting the teacher data with the remaining amount of yarn output from the output layer 53 (state detection result) and adjusting the weights corresponding to the synapses in the inference model 50A before learning to obtain a preferred inference model 50A. In addition, the format of the output data output from the output layer 53 of the inference model 50A before learning is the same as the format of the teacher data in the learning data set that is the learning object.

[0130] Figure 6 The illustrated inference result indicates that five bobbins 2 included in five bobbin regions A to E (lower left coordinates and upper right coordinates) are detected from the arbitrary number of bobbin captured images 31, and the remaining yarn amounts of the five bobbins 2 surrounded by the five bobbin regions A to E are respectively "100% remaining", "60% remaining", "40% remaining", "10% remaining", and "no yarn". Therefore, the remaining yarn amount of the bobbins 2 surrounded by the bobbin region B has an error with the teacher data, and the weight of the inference model 50A is adjusted based on the error.

[0131] When machine learning is implemented in step S240, it is determined whether the machine learning needs to be continued further (step S250), for example, based on the remaining number of unlearned learning data sets stored in the learning data set storage unit 61. Then, in the case where the machine learning continues without satisfying the learning end condition ("No" in step S250), the process returns to step S220, and in the case where the machine learning ends when the learning end condition is satisfied ("Yes" in step S250), the process transfers to step S260. In the case of continuing the above-mentioned machine learning, the process of steps S220 to S240 is performed multiple times on the inference model 50A being learned using the unlearned learning data sets. The accuracy of the finally generated learned model 5A generally increases in proportion to the number of times.

[0132] When the machine learning is completed (YES in step S250 ), the inference model 50A generated by adjusting the weights corresponding to each synapse is stored in the learned model storage unit 63 as the learned model 5A (step S260 ), and a series of machine learning steps are completed.

[0133] In the learning method and machine learning method of the machine learning device 6A described above, it is explained that in order to generate a learned model 5A, a machine learning process is repeatedly performed multiple times on an inference model 50A (before learning) to improve its accuracy, and a learned model 5A sufficient for application to the image processing device 4A is obtained. However, the present invention is not limited to this acquisition method. For example, a learned model 5A that has been subjected to a predetermined number of machine learning processes may be stored as a candidate in a plurality of learned model storage units 63, a data set for appropriateness judgment may be input into the plurality of learned model groups to generate an output layer (the value of the neurons), the accuracy of the values ​​determined by the output layer may be compared and studied, and an optimal learned model 5A applied to the image processing device 4A may be selected. In addition, the data set for appropriateness judgment is composed of the same data set as the learning data set used in learning, and need not be used in learning.

[0134] As described above, according to the machine learning device 6A and the machine learning method of the present embodiment, it is possible to generate a learned model 5A that can accurately detect (infer) the state of any number of yarn tubes 2 contained in the camera-captured image 30 (in the present embodiment, the remaining yarn amount is used as an example) based on the camera-captured image 30 containing any number of yarn tubes 2.

[0135] (Second Embodiment)

[0136] In the first embodiment, the machine learning device 6A generates one learned model 5A, and the image processing device 4A inputs the camera-photographed image 30 to the one learned model 5A, thereby detecting the remaining amount of yarn of the bobbin 2 included in the camera-photographed image 30. In contrast, in the second embodiment, the first and second machine learning devices 6B1 and 6B2 generate the first and second learned models 5B1 and 5B2, respectively, and the image processing device 4B has a model structure in which the first and second learned models 5B1 and 5B2 are connected in series with the first learned model 5B1 as the front stage and the second learned model 5B2 as the rear stage, and the camera-photographed image 30 is input to the first learned model 5B1 to detect the remaining amount of yarn of the bobbin 2 included in the camera-photographed image 30. The other basic structures and operations are the same as those of the first embodiment, so the following description will focus on the characteristic parts of the second embodiment. In addition, the first and second machine learning devices 6B1 and 6B2 will be described as different devices below, but the first and second machine learning devices 6B1 and 6B2 may also be composed of a single device.

[0137] (Image Processing Device 4B)

[0138] Figure 8 1 is a schematic block diagram showing an example of a flat knitting machine 1 including an image processing device 4B according to the second embodiment. A first learned model 5B1 and a second learned model 5B2 are stored in a learned model storage unit 42 of the image processing device 4B.

[0139] The first learned model 5B1 performs machine learning on the correlation between any number of yarn tube captured images 31 including any number of yarn tubes 2 and any number of regions surrounding each of the any number of yarn tubes 2 included in the any number of yarn tube captured images 31 by machine learning using the first machine learning device 6B1 and the machine learning method described later.

[0140] The second learned model 5B2 performs machine learning on the correlation between a single bobbin captured image 33 including one bobbin 2 and the remaining amount of yarn of the one bobbin 2 included in the single bobbin captured image 33 by machine learning using a second machine learning device 6B2 and a machine learning method described later.

[0141] Fig. 9 This is a flowchart showing an example of an image processing method by the image processing device 4B of the second embodiment.

[0142] At a predetermined timing, the acquisition unit 40 acquires the camera image 30 obtained by imaging the yarn bobbin 2 on the top plate 15 by the camera 3 (step S100).

[0143] Next, the bobbin state detection unit 41 performs an inference process using the first learned model 5B1 with reference to the first learned model 5B1 stored in the learned model storage unit 42 (step S111). Specifically, the bobbin state detection unit 41 inputs the camera image 30 acquired by the acquisition unit 40 to the first learned model 5B1, thereby determining an arbitrary number of regions surrounding each of an arbitrary number of bobbins 2 included in the camera image 30. Fig. 9 The illustrated estimation result of step S111 is a result of detecting five yarn bobbins 2 from the camera image 30 and specifying yarn bobbin regions A to E surrounding each of the five yarn bobbins 2 using two points of the lower left coordinate and the upper right coordinate.

[0144] Next, the bobbin state detection unit 41 refers to the second learned model 5B2 stored in the learned model storage unit 42 and performs an inference process using the second learned model 5B2 (step S112). Specifically, the bobbin state detection unit 41 inputs the area images 32 including the bobbin 2 defined by the arbitrary number of areas in the camera image 30 acquired by the acquisition unit 40 to the second learned model 5B2, thereby detecting the remaining amount of yarn of the bobbin 2 included in the area images 32. Fig. 9 The inference result of the illustrated step S112 is the result of multi-value detection of the remaining yarn amounts of the five yarn bobbins from the five area images 32 respectively determined by the five yarn bobbin areas A to E (lower left coordinates and upper right coordinates). The remaining yarn amounts of the four yarn bobbins 2 surrounded by the yarn bobbin areas A to D (lower left coordinates and upper right coordinates) are "100% remaining", "70% remaining", "40% remaining", and "10% remaining", respectively, and the remaining yarn amount of the one yarn bobbin 2 surrounded by the yarn bobbin area E is "no yarn".

[0145] Next, in steps S111 and S112, when the result of the inference processing by the bobbin state detection unit 41 is that the detection result of the remaining yarn amount for any bobbin 2 is "no yarn" ("Yes" in step S120), the output processing unit 43 reports "no yarn" (step S130), and when the detection result of the remaining yarn amount for any bobbin 2 is less than the remaining yarn amount prediction value ("Yes" in step S140), the output processing unit 43 reports "no yarn prediction" (step S150). On the other hand, when the detection result of the remaining yarn amount for any bobbin 2 does not indicate "no yarn" or "no yarn prediction", that is, when the detection result of the remaining yarn amount for all bobbins 2 indicates "yarn presence" ("No" in step S120 and "No" in step S140), the image processing step is directly terminated.

[0146] As described above, according to the image processing device 4B and the image processing method of the present embodiment, the states of any number of bobbins 2 included in the camera-photographed image 30 can be detected (the remaining amount of yarn is described as an example in the present embodiment). Therefore, it is not necessary to provide an optical component for each bobbin 2, and the state of the bobbin 2 can be detected without being restricted by the number and position of the bobbins 2 when the bobbins 2 are arranged above or around the knitting machine. Therefore, the state of the bobbin 2 can be detected with a simple structure, and the workability and versatility can be improved.

[0147] (First Machine Learning Device 6B1)

[0148] Next, refer to Fig.10 , Fig.11 The first machine learning device 6B1 used to detect the states of an arbitrary number of yarn bobbins 2 arranged above or around the flat knitting machine 1 will be described. Fig.10 This is a schematic block diagram showing an example of the first machine learning device 6B1 according to the second embodiment. Fig.11 This is a diagram showing an example of a first estimation model 50B1 based on CNN used in the first machine learning device 6B1 of the second embodiment.

[0149] The first machine learning device 6B1 generates a learning model (first learned model 5B1) capable of specifying (estimating) an arbitrary number of regions surrounding each of the arbitrary number of bobbins 2 included in the arbitrary number of bobbins 2 captured images 31 based on the arbitrary number of bobbins 2 captured images 31.

[0150] The learning data set acquisition unit 60 acquires, for example, an arbitrary number of yarn bobbin images 31 as input data from the external device 7, and acquires, for example, an arbitrary number of regions surrounding each of an arbitrary number of yarn bobbins 2 included in the arbitrary number of yarn bobbins images 31 from the terminal device 8 as output data. Then, by establishing correspondence between these input data and output data, a learning data set is constructed and stored in the learning data set storage unit 61. Fig.11 In the illustrated learning data set, the input data is an arbitrary number of bobbin images 31 including five bobbins 2 in an area defined by each of five bobbin areas A to E (lower left coordinates and upper right coordinates), and the output data is information identifying the bobbin areas A to E (lower left coordinates and upper right coordinates) surrounding each of the five bobbins 2. Alternatively, the input data constituting the learning data set may be a single bobbin image including one bobbin 2 in part or in its entirety.

[0151] The machine learning unit 62 performs machine learning using the plurality of learning data sets stored in the learning data set storage unit 61 , thereby generating a learning model (first learned model 5B1 ) that infers the correlation between input data and output data included in the plurality of learning data sets.

[0152] Specifically, the machine learning unit 62 inputs an arbitrary number of yarn bobbin photographed images 31 constituting a learning data set into the Fig.11 The input layer 51 of the first inference model 50B1 shown and the area (object detection result) output as the inference result from the output layer 53 are compared with the area (teacher data) constituting the learning data set, thereby implementing machine learning. Fig.11 The illustrated inference result is a result of detecting four bobbins 2 included in each of five bobbin regions A to D (lower left coordinates and upper right coordinates) from an arbitrary number of bobbin captured images 31, and specifying the bobbin regions A to D surrounding each of the five bobbins 2 using the two points of the lower left coordinates and the upper right coordinates. When an error occurs between the inference result and the teacher data for the bobbin regions A to E, the weight of the first inference model 50B1 is adjusted based on the error.

[0153] Then, when the learning end condition is satisfied and the machine learning is ended, the machine learning unit 62 stores the first estimation model 50B1 as the first learned model 5B1 in the learned model storage unit 63 .

[0154] (Second Machine Learning Device 6B2)

[0155] Next, refer to Fig.12 , Fig.13 The second machine learning device 6B2 used to detect the remaining amount of yarn of an arbitrary number of bobbins 2 arranged above or around the flat knitting machine 1 will be described. Fig.12 This is a schematic block diagram showing an example of the second machine learning device 6B2 according to the second embodiment. Fig.13 This is a diagram showing an example of a second estimation model 50B2 based on CNN used in the second machine learning device 6B2 of the second embodiment.

[0156] The second machine learning device 6B2 generates a learning model (second learned model 5B2) capable of detecting (estimating) the remaining amount of yarn of one bobbin 2 included in the single bobbin image 33 based on the single bobbin image 33 including the one bobbin 2.

[0157] The learning data set acquisition unit 60 acquires a single bobbin image 33 as input data, for example, from an external device 7, and acquires the remaining amount of yarn of one bobbin 2 included in the single bobbin image 33 as output data, for example, from a terminal device 8. Then, by establishing correspondence between these input data and output data, a learning data set is constructed and stored in the learning data set storage unit 61. Fig.13 In the illustrated learning data set, the input data is a single yarn bobbin captured image 33 including one yarn bobbin 2, and the output data is information indicating that the remaining amount of yarn of the one yarn bobbin 2 is "70% remaining".

[0158] The machine learning unit 62 performs machine learning using the plurality of learning data sets stored in the learning data set storage unit 61 to generate a learning model (second learned model 5B2) that infers the correlation between input data and output data included in the plurality of learning data sets.

[0159] Specifically, the machine learning unit 62 inputs a single bobbin image 33 constituting a learning data set into the Fig.12 The input layer 51 of the second estimation model 50B2 shown is used to compare the remaining yarn amount (state detection result) output as the estimation result from the output layer 53 with the remaining yarn amount (teacher data) constituting the learning data set, thereby performing machine learning. Fig.13 The illustrated estimation result indicates that the remaining yarn amount of one bobbin 2 included in the single bobbin captured image 33 is "60% remaining". Therefore, the remaining yarn amount of the bobbin 2 has an error from the teacher data, and the weight of the second estimation model 50B2 is adjusted based on the error.

[0160] Then, when the learning end condition is satisfied and the machine learning is ended, the machine learning unit 62 stores the second inference model 50B2 as the second learned model 5B2 in the learned model storage unit 63 .

[0161] As described above, according to the first and second machine learning devices 6B1, 6B2 and machine learning methods of the present embodiment, similar to the machine learning device 6A and machine learning method of the first embodiment, it is possible to generate the first and second learned models 5B1, 5B2 that can accurately detect (infer) the states of any number of yarn tubes 2 contained in the camera-captured image 30 (in the present embodiment, the remaining yarn amount is used as an example) from the camera-captured image 30 containing any number of yarn tubes 2.

[0162] In particular, the first machine learning device 6B1 generates a first learned model 5B1 specialized for object detection of the yarn bobbin 2, which determines an area surrounding each of an arbitrary number of yarn bobbins 2 included in the camera-captured image 30, and the second machine learning device 6B2 generates a second learned model 5B2 specialized for state detection of the yarn bobbin 2, which detects the state of one yarn bobbin 2. Therefore, as a learning data set, it is not necessary to prepare a plurality of captured images combining various conditions, and it is sufficient to prepare learning data sets suitable for the first and second learned models 5B1 and 5B, respectively, so that the effort for preparing the learning data sets can be reduced, and the inference accuracy of the first and second learned models 5B1 and 5B2 can be improved.

[0163] (Third Embodiment)

[0164] In the first embodiment, the machine learning device 6A generates the learned model 5A, and the image processing device 4A inputs the camera-photographed image 30 to the learned model 5A, thereby detecting the remaining amount of yarn of the bobbin 2 included in the camera-photographed image 30. In contrast, in the third embodiment, the machine learning device 6C generates the learned model 5C, and the image processing device 4C inputs the camera-photographed image 30 to the learned model 5C, thereby detecting the connection relationship between the bobbin 2 included in the camera-photographed image 30 and the tension applying device 16. The other basic structures and operations are the same as those of the first embodiment, so the following description will focus on the characteristic parts of the third embodiment.

[0165] (Image Processing Device 4C)

[0166] Fig.14 1 is a schematic block diagram showing an example of a flat knitting machine 1 including an image processing device 4C according to the third embodiment. A learned model 5C is stored in a learned model storage unit 42 of the image processing device 4C.

[0167] The learned model 5C performs machine learning on the correlation between the arbitrary number of yarn tube images 31 including the arbitrary number of yarn tubes 2 and the connection relationship between the arbitrary number of yarn tubes 2 and the plurality of tension applying devices 16 included in the arbitrary number of yarn tube images 31 by machine learning using the machine learning device 6C and the machine learning method described later. Specifically, Figure 2As shown, the arbitrary number of yarn tube shooting images 31 include arbitrary number of yarn tubes 2 and knitting yarns 20 released from each yarn tube 2 and connected to the tension applying device 16. Therefore, the learned model 5C extracts the feature quantities between the positions where the arbitrary number of yarn tubes 2 are arranged, the directions in which the knitting yarns 20 released from each yarn tube 2 extend respectively, and the positions of each tension applying device 16 existing in the directions in which the knitting yarn 20 extends respectively, from the arbitrary number of yarn tube shooting images 31, and performs machine learning as the connection relationship between the yarn tube 2 and the tension applying device 16.

[0168] Fig.15 4C is a flowchart showing an example of an image processing method of the image processing device 4C of the third embodiment. Hereinafter, a case where the plurality of tension applying devices 16 are composed of n devices and are numbered "1, 2, 3, ..., n" in order from the left side of the device is described.

[0169] At a predetermined timing, the acquisition unit 40 acquires the camera image 30 obtained by imaging the yarn bobbin 2 on the top plate 15 by the camera 3 (step S100).

[0170] Next, the bobbin state detection unit 41 refers to the learned model 5C stored in the learned model storage unit 42 and performs an inference process using the learned model 5C (step S113). Specifically, the bobbin state detection unit 41 inputs the camera image 30 acquired by the acquisition unit 40 to the learned model 5C, extracts feature quantities between the positions where the arbitrary number of bobbins 2 are arranged, the directions in which the knitting yarns 20 released from the respective bobbins 2 extend, and the positions of the respective tension applying devices 16 existing in the directions in which the knitting yarns 20 extend, thereby detecting the connection relationship between the arbitrary number of bobbins 2 and the plurality of tension applying devices 16 included in the camera image 30.

[0171] Fig.15 The inference result of the illustrated step S113 indicates the following situation: the five bobbins 2 included in the five bobbin regions A to E (lower left coordinates and upper right coordinates) are detected from the camera captured image 30, and the knitted yarn 20 extending from the bobbin 2 surrounded by the bobbin region A is connected to the "1st" tension applying device 16, the knitted yarn 20 extending from the bobbin 2 surrounded by the bobbin region C is connected to the "4th" tension applying device 16, the knitted yarn 20 extending from the bobbin 2 surrounded by the bobbin region D is connected to the "10th" tension applying device 16, the knitted yarn 20 extending from the bobbin 2 surrounded by the bobbin region B is connected to the "17th" tension applying device 16, and the other tension applying devices are not connected to the bobbins 2. In addition, the bobbins 2 in the "no yarn" state surrounded by the bobbin region E are detected as not being connected to the tension applying device 16.

[0172] Next, the output processing unit 43 determines whether the connection relationship between the yarn bobbin 2 and the tension applying device 16 detected by the yarn bobbin state detection unit 41 in step S113 is consistent with the connection relationship set in the knitting data of the flat knitting machine 1. If the connection relationship between the two is inconsistent ("Yes" in step S121), the output processing unit 43 reports to the operator that "operation error" has occurred (step S131). On the other hand, if the output processing unit 43 determines that the connection relationship between the two is consistent ("No" in step S121), the image processing step is directly terminated.

[0173] As described above, according to the image processing device 4C and the image processing method of the present embodiment, the states of any number of yarn bobbins 2 included in the camera-photographed image 30 can be detected, and in particular, in the present embodiment, the connection relationship between the yarn bobbins 2 and the tension applying device 16 can be detected. Therefore, it is not necessary to provide optical components for each yarn bobbin 2, and the state of the yarn bobbin 2 can be detected without being restricted by conditions such as the number and position of the yarn bobbins 2 when the yarn bobbins 2 are arranged above or around the knitting machine. Therefore, the state of the yarn bobbins 2 can be detected with a simple structure, and the workability and versatility can be improved.

[0174] (Machine Learning Device 6C)

[0175] Next, refer to Fig.16 , Fig.17 A machine learning device 6C used to detect the connection relationship between an arbitrary number of bobbins arranged above or around the flat knitting machine 1 and a plurality of tension applying devices will be described. Fig.16 This is a schematic block diagram showing an example of a machine learning device 6C according to the third embodiment. Fig.17 This is a diagram showing an example of an inference model 50C based on CNN used in a machine learning device 6C according to the third embodiment.

[0176] The machine learning device 6C generates a learning model (learned model 5C) that can detect (estimate) the correlation between the connection relationship between the arbitrary number of yarn bobbins 2 included in the arbitrary number of yarn bobbins 2 and the plurality of tension applying devices 16 based on the arbitrary number of yarn bobbins captured images 31 including the arbitrary number of yarn bobbins 2.

[0177] The learning data set acquisition unit 60 acquires, for example, an arbitrary number of yarn bobbin images 31 as input data from the external device 7, and acquires, for example, the connection relationship between an arbitrary number of yarn bobbins 2 and a plurality of tension applying devices 16 included in the arbitrary number of yarn bobbins images 31 from the terminal device 8 as output data. Then, by establishing correspondence between these input data and output data, a learning data set is constructed and stored in the learning data set storage unit 61. Fig.17In the illustrated learning data set, the input data is an arbitrary number of yarn bobbin images 31 including three yarn bobbins 2 in the areas respectively defined by three yarn bobbin areas A to C (lower left coordinates and upper right coordinates), and the output data is information indicating that the yarn bobbin 2 surrounded by the yarn bobbin area A is connected to the "2nd" tension applying device 16, the yarn bobbin 2 surrounded by the yarn bobbin area B is connected to the "4th" tension applying device 16, and the yarn bobbin 2 surrounded by the yarn bobbin area C is connected to the "15th" tension applying device 16.

[0178] The machine learning unit 62 performs machine learning using the plurality of learning data sets stored in the learning data set storage unit 61 , thereby generating a learning model (learned model 5C) that infers the correlation between input data and output data included in the plurality of learning data sets.

[0179] Specifically, the machine learning unit 62 inputs an arbitrary number of yarn bobbin photographed images 31 constituting a learning data set into the Fig.17 The connection relationship (state detection result) output as the inference result from the output layer 53 of the inference model 50C shown is compared with the connection relationship (teacher data) constituting the learning data set, thereby implementing machine learning. Fig.17 The illustrated inference result shows that five bobbins 2 included in three bobbin regions A to C (lower left coordinates and upper right coordinates) are detected from the arbitrary number of bobbin captured images 31, and the bobbins 2 surrounded by the bobbin region C are connected to the tension applying device "2", the bobbins 2 surrounded by the bobbin region B are connected to the tension applying device "3", and the bobbins 2 surrounded by the bobbin region A are connected to the tension applying device "15". Therefore, an error occurs between the connection relationship of the bobbins 2 surrounded by the bobbin region B and the teacher data, and the weight of the inference model 50C is adjusted based on the error.

[0180] Then, when the learning end condition is satisfied and the machine learning is ended, the machine learning unit 62 stores the inference model 50C as the learned model 5C in the learned model storage unit 63 .

[0181] In the present embodiment, the case where n tension applying devices 16 are made to correspond to the output layer 53 of the estimation model 50C, and the sequential numbers (identification information) of the tension applying devices 16 and the regions of the yarn bobbins 2 connected to the respective tension applying devices 16 are combined and allocated. On the other hand, the regions of the yarn bobbins 2 of any number may be made to correspond to the output layer 53, and the regions of the yarn bobbins 2 and the sequential numbers of the tension applying devices 16 connected to the respective yarn bobbins 2 may be combined and allocated. In this case, the format of the output data constituting the learning data set may be changed in accordance with the data format of the output layer 53.

[0182] As described above, according to the machine learning device 6C and the machine learning method of the present embodiment, it is possible to generate a learned model 5C based on a camera-captured image 30 containing any number of yarn bobbins 2, which can accurately detect (infer) the state of any number of yarn bobbins 2 contained in the camera-captured image 30, particularly in the present embodiment, the connection relationship between the yarn bobbins 2 and the tension applying device 16.

[0183] (Other embodiments)

[0184] The present invention is not limited to the above-mentioned embodiments, and can be implemented with various modifications within the scope not departing from the gist of the present invention. Moreover, all of these are included in the technical concept of the present invention.

[0185] (Inference Device)

[0186] For example, the present invention can be provided not only in the form of the above-mentioned image processing devices 4A to 4C, but also in the form of inference devices 5A, 5B1, 5B2, and 5C for inference. In this case, the inference devices 5A, 5B1, 5B2, and 5C include a memory and at least one processor, and the processor can execute a series of processes. The series of processes includes: a process of inputting a camera-photographed image 30 including an arbitrary number of yarn bobbins; and a process of inferring the states of the arbitrary number of yarn bobbins 2 included in the camera-photographed image 30 (for example, the remaining amount of yarn in the yarn bobbins, the connection relationship between the yarn bobbins and the tension applying device, the position where the yarn bobbins are arranged, etc.). By providing the present invention in the form of the above-mentioned inference devices 5A, 5B1, 5B2, and 5C, it can be applied to various devices more easily than in the case of installing the image processing devices 4A to 4C. At this time, those skilled in the art will certainly understand that when the inference device performs processing to infer the state of the yarn bobbin 2, as described above, the yarn bobbin state detection unit 41 of the image processing device 4A~4C can also be applied to implement the inference method using the learned models 5A, 5B1, 5B2, 5C learned by the machine learning device 6A, 6B1, 6B2, 6A and the machine learning method in the present invention.

[0187] In the above-mentioned embodiment, the case where the image processing devices 4A to 4C are applied to the flat knitting machines 1A to 1C is described. In contrast, the image processing devices 4A to 4C may be applied to any type of knitting machine, such as a circular knitting machine or a warp knitting machine. In addition, the image processing devices 4A to 4C may be installed in the knitting machine at the knitting machine manufacturing factory, or may be installed in the knitting machine after the knitting machine is shipped from the factory. The functions of the various parts of the image processing devices 4A to 4C may be added to the control panel 100 of the knitting machine after the knitting machine is shipped from the factory.

[0188] In the above embodiment, the image processing devices 4A to 4C perform inference processing using the learned model as image processing for the camera-photographed image 30. Alternatively, the image processing devices 4A to 4C may perform pattern recognition processing such as pattern matching using recognition parameters indicating the characteristics of the yarn bobbin 2 on the camera-photographed image 30.

[0189] In addition, in the above-mentioned embodiment, as a learning method of machine learning, a method using CNN-based inference models 50A, 50B1, 50B2, and 50C is described, but it is not limited to this. As long as the correlation between the input and output in the above-mentioned embodiment can be learned based on the learning data set, other learning methods can also be used. For example, neural networks, ensemble learning (random forest, boosting, etc.), R-CNN (Region-based Convolutional Neural Network) applied to CNN, Fast-R-CNN, Faster-R-CNN, YOLO (You Only Look Once), SSD (Single Shot multibox Detector) and other learning methods can also be used. At this time, the data format of the input data for the input layer 51 and the output data for the output layer 53 can also be appropriately adjusted.

[0190] Description of Reference Numerals

[0191] 1… flat knitting machine, 2, 2A to 2E… yarn bobbins, 3… cameras, 4A to 4C… image processing devices, 5A… learned models, 5B1… first learned models, 5B2… second learned models, 5C… learned models, 6A… machine learning devices, 6B1… first machine learning devices, 6B2… second machine learning devices, 6C… machine learning devices, 7… external devices, 8… terminal devices, 11A… lower frame, 11B… upper frame, 12… knitting needles, 13… needle beds, 14… cam carriages, 15… top plates, 16… tension applying devices, 17… side tensioning units, 18… yarn feeders, 20… knitting yarns, 21… core units, 3 0…Image captured by camera, 31…Image captured by any number of bobbins, 33…Image captured by a single bobbin, 40…Acquisition unit, 41…Bobbin state detection unit, 42…Model storage unit, 43…Output processing unit, 50A…Inference model, 50B1…First inference model, 50B2…Second inference model, 50C…Inference model, 51…Input layer, 52…Intermediate layer, 53…Output layer, 60…Learning data set acquisition unit, 61…Learning data set storage unit, 62…Machine learning unit, 63…Model storage unit, 100…Control panel, 101…Operation display panel, 520…Convolution layer, 521…Pooling layer, 522…Fully connected layer

Claims

1. An image processing device, wherein: The image processing device comprises: an acquisition unit that acquires a captured image including an arbitrary number of yarn tubes arranged above or around the knitting machine; and a bobbin state detecting section for detecting states of the arbitrary number of bobbins included in the captured image by performing image processing on the captured image acquired by the acquiring section; The bobbin state detection unit inputs the captured image acquired by the acquisition unit into a first learned model obtained by machine learning the correlation between a captured image including an arbitrary number of bobbins and an arbitrary number of regions surrounding each of the arbitrary number of bobbins included in the captured image, thereby determining an arbitrary number of regions surrounding each of the arbitrary number of bobbins included in the captured image. The yarn bobbin status detection unit detects the remaining amount of yarn of the yarn bobbin contained in the captured image respectively by inputting the captured image including the yarn bobbin defined by each of the arbitrary number of areas in the captured image acquired by the acquisition unit into a second learned model obtained by machine learning the correlation between the captured image including the yarn bobbin and the remaining amount of yarn of the yarn bobbin contained in the captured image, thereby respectively detecting the remaining amount of yarn of the yarn bobbin contained in the captured image.

2. An image processing device, wherein: The image processing device comprises: an acquisition unit that acquires a captured image including an arbitrary number of yarn tubes arranged above or around the knitting machine; and a bobbin state detecting section for detecting states of the arbitrary number of bobbins included in the captured image by performing image processing on the captured image acquired by the acquiring section; The bobbin state detection section detects the connection relationship between the arbitrary number of bobbins included in the captured image and the plurality of tension applying devices by performing the image processing.

3. The image processing device according to claim 2, wherein: The yarn bobbin state detection unit detects the connection relationship between the arbitrary number of yarn bobbins contained in the captured image and the plurality of tension applying devices by inputting the captured image acquired by the acquisition unit into a learned model obtained by machine learning the correlation between a captured image including an arbitrary number of yarn bobbins and the connection relationship between the arbitrary number of yarn bobbins contained in the captured image and the plurality of tension applying devices.

4. A machine learning device for detecting the connection relationship between any number of yarn tubes and a plurality of tension applying devices arranged above or around a knitting machine, wherein: The machine learning device comprises: A learning data set storage unit stores a plurality of learning data sets, wherein the learning data sets are composed of input data and output data, wherein the input data are composed of captured images including an arbitrary number of yarn bobbins, and the output data are corresponding to the input data and are composed of a connection relationship between the arbitrary number of yarn bobbins and a plurality of tension applying devices included in the captured images; a machine learning unit that learns a learning model for inferring a correlation between the input data and the output data by inputting a plurality of sets of the learning data sets; and The learned model storage unit stores the learned model learned by the machine learning unit.

5. An inference device for detecting the connection relationship between any number of yarn tubes arranged above or around a knitting machine and a plurality of tension applying devices, wherein: The inference device comprises a memory and at least one processor. The at least one processor is configured to perform the following processing: Input captured images containing any number of yarn tubes; and When the captured image is input, the connection relationship between the arbitrary number of yarn bobbins and the plurality of tension applying devices included in the captured image is inferred.

Images