Paper jam prediction device, paper jam prediction method, and program

Abstract

A paper jam sign estimation device (100) estimates signs of a paper jam in a paper feeder (200) and comprises: a sound-collecting unit (112) that collects the sound of friction between paper sheets (10) generated when the paper sheets (10) are fed into the interior of the paper feeder (200) from a holding unit (270) that holds the plurality of paper sheets (10); an estimation unit (114) that estimates the presence or absence of signs of a paper jam in the paper feeder (200) on the basis of the output result obtained by inputting information relating to the friction sound into a learned model that is a learned machine learning model; and an output unit (116) that outputs, to the paper feeder (200), a signal for stopping the paper sheets (10) being fed into the interior of the paper feeder (200) when the estimation unit (114) estimates that there are signs of a paper jam.

Description

Technical Field

[0001] The present disclosure relates to a paper jam预兆estimation device, a paper jam预兆estimation method, and a program.

Background Art

[0002] For example, in a paper feeding device that supplies paper to an image reading device such as a printer or an image copying device (so-called scanner), paper jams may occur due to, for example, double feeding of paper, skew feeding, or stapling. Depending on the degree of paper jam, not only the work may be delayed, but the paper may also be damaged and become unusable. Therefore, a technique for early detection of the occurrence of paper jams is required.

[0003] For example, Patent Document 1 discloses a technique for receiving ultrasonic waves transmitted from an ultrasonic transmitter provided in a part of a medium support unit by an ultrasonic receiver provided in another part of the medium support unit, and determining the presence or absence of floating of the medium with respect to the placement surface (hereinafter referred to as paper floating) based on the sound pressure of the received ultrasonic waves.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique described in Patent Document 1, it is necessary to provide an ultrasonic transmitter in order to determine the presence or absence of the occurrence of paper floating, and it is difficult to simply estimate the presence or absence of a paper jam预兆(that is, the occurrence of paper floating).

[0006] The present disclosure provides a paper jam预兆estimation device, a paper jam预兆estimation method, and a program that can simply estimate the presence or absence of a paper jam预兆 [Means for solving the problem]

[0007] A paper jam prediction device according to one aspect of the present disclosure is a paper jam prediction device for estimating signs of a paper jam in a paper feed device, wherein when paper is supplied into the inside of the paper feed device from a holding unit that holds a plurality of sheets of paper... In the vicinity of the separation roller that separates the paper in the paper feeding device Occur paper on paper A sound collection unit that collects friction sounds, and a trained model which is a trained machine learning model that inputs information about the friction sounds, and based on the output results obtained, the paper feed device Paper lifting An estimation unit that estimates the presence or absence of the above The paper floated. The device includes an output unit that outputs a signal to the paper feeder to stop the paper from being supplied into the paper feeder when it is estimated that the paper is not being supplied to the inside of the paper feeder. [Effects of the Invention]

[0008] According to this disclosure, a paper jam prediction device, a paper jam prediction method, and a program can be provided that can easily estimate whether or not there are signs of a paper jam. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of a paper feed device to which the paper jam prediction device according to Embodiment 1 is applied. [Figure 2] Figure 2 shows an example of the transport section of the paper feed device in Embodiment 1. [Figure 3] Figure 3 shows an example of the configuration of a paper jam prediction device and paper feeding device according to Embodiment 1. [Figure 4] Figure 4 shows an example of a paper jam warning sign in Embodiment 1. [Figure 5] Figure 5 is a flowchart showing the operation of the paper jam prediction device according to Embodiment 1. [Figure 6] Figure 6 shows an example of the configuration of a paper jam prediction device according to Modification 1 of Embodiment 1. [Figure 7]Figure 7 shows an example of the configuration of a paper jam prediction device and paper feeding device according to Embodiment 2. [Figure 8] Figure 8 is a flowchart showing the operation of the paper jam prediction device according to Embodiment 2. [Figure 9] Figure 9 shows an example of the configuration of a paper jam prediction device and paper feeding device according to Modification 1 of Embodiment 2. [Figure 10] Figure 10 is a flowchart showing the operation of the paper jam prediction device according to Modification 1 of Embodiment 2. [Figure 11] Figure 11 is a diagram illustrating the machine learning models used in Example 1 and Example 2. [Figure 12] Figure 12 shows the results of Example 1. [Figure 13] Figure 13 shows the results of Example 2. [Figure 14] Figure 14 shows a comparison of the estimation accuracy of Example 1 and Example 2 for the four types of paper used in Example 2. [Modes for carrying out the invention]

[0010] (Summary of this disclosure) A paper jam prediction device according to one aspect of the present disclosure is a paper jam prediction device for estimating signs of a paper jam in a paper feed device, comprising: a sound collection unit that collects friction sounds generated when paper is supplied into the paper feed device from a holding unit that holds a plurality of sheets of paper; an estimation unit that estimates whether or not there are signs of a paper jam in the paper feed device based on the output result obtained by inputting information regarding the friction sounds into a trained model, which is a trained machine learning model; and an output unit that outputs a signal to the paper feed device to stop the supply of paper into the paper feed device when the estimation unit estimates that there are signs of a paper jam.

[0011] As a result, the paper jam prediction estimation device can collect the frictional sound when the paper is supplied from the holding unit into the inside of the paper feeding device, and based on the output result obtained by inputting the information regarding the collected frictional sound into the learned model, it can estimate the presence or absence of a paper jam prediction such as the lifting of the paper that occurs before the paper jam. Therefore, unlike the prior art, for example, in order to estimate the presence or absence of the occurrence of paper lifting, it is not necessary to provide an ultrasonic transmission unit, and it is sufficient to provide a sound collection unit that collects the frictional sound. Accordingly, the paper jam prediction estimation device can simply estimate the presence or absence of a paper jam prediction with a configuration fewer than that of a configuration including an ultrasonic irradiation unit.

[0012] Further, the paper jam prediction estimation device can not only prevent the occurrence of a paper jam but also suppress paper breakage in order to estimate the presence or absence of the occurrence of a paper jam prediction such as the lifting of the supplied paper.

[0013] In the paper jam prediction estimation device according to one aspect of the present disclosure, the information regarding the frictional sound input into the learned model may be an image of the spectrogram of the frictional sound or an image of the frequency characteristics.

[0014] As a result, the paper jam prediction estimation device can more easily extract the regularity (so-called feature amount) of the image by using a machine learning model. Therefore, the paper jam prediction estimation device can more simply estimate the presence or absence of a paper jam prediction.

[0015] In the paper jam prediction estimation device according to one aspect of the present disclosure, the frictional sound may be an inaudible sound generated by the friction between the paper supplied from the holding unit and the paper held by the holding unit. For example, the inaudible sound may be a sound having a frequency in the ultrasonic band.

[0016] As a result, the paper jam prediction device estimates whether or not paper is floating based on inaudible sounds (e.g., sounds in the ultrasonic frequency range) in the friction noise generated when paper is supplied from the holding unit. Therefore, it is less susceptible to the influence of various audible sounds, i.e., noise, generated around the paper jam prediction device, and the sound collection accuracy is improved. Consequently, the paper jam prediction device can accurately estimate whether or not there is a paper jam.

[0017] In a paper jam prediction device according to one aspect of the present disclosure, the training data used to train the machine learning model may include first data consisting of information about the friction sound and annotations indicating the occurrence of a paper jam, and second data consisting of information about the friction sound and annotations indicating that no paper jam has occurred.

[0018] As a result, the paper jam prediction device can accurately estimate the presence or absence of signs of a paper jam, due to the improved learning accuracy of its learning unit.

[0019] In a paper jam prediction device according to one aspect of the present disclosure, the trained model includes a plurality of trained models corresponding to each of a plurality of types of paper, the paper jam prediction device further includes an identification unit that identifies the type of paper supplied from the holding unit into the paper feed device, and the estimation unit may input information regarding the friction sound to the trained model corresponding to the identified type of paper based on the type of paper identified by the identification unit.

[0020] This allows the paper jam prediction device to switch the trained model used depending on the type of paper being supplied from the paper holding unit into the paper feeding device. Therefore, the paper jam prediction device can accurately estimate the presence or absence of a paper jam depending on the type of paper.

[0021] In a paper jam prediction device according to one aspect of the present disclosure, the identification unit may identify the type of paper based on data acquired by at least one of an image sensor, an ultrasonic sensor, an optical sensor, a weight sensor, and machine learning.

[0022] As a result, the paper jam prediction device can identify the type of paper using at least one of the following: a database that associates data indicating paper characteristics with paper types, and a trained model that takes data indicating paper characteristics as input and outputs the type of paper 10 supplied. Therefore, the paper jam prediction device can identify the type of paper with high accuracy.

[0023] In a paper jam prediction device according to one aspect of this disclosure, the machine learning model may be a convolutional neural network model.

[0024] As a result, the paper jam prediction device can more easily extract image regularities (so-called features) by using a convolutional neural network model.

[0025] Furthermore, a paper jam prediction method according to one aspect of the present disclosure is a paper jam prediction method for estimating signs of a paper jam in a paper feeder, comprising: a sound collection step of collecting friction sounds generated when paper is supplied into the paper feeder from a holding section that holds a plurality of sheets of paper; an estimation step of estimating whether or not there are signs of a paper jam in the paper feeder based on the output results obtained by inputting information regarding the friction sounds into a trained model, which is a trained machine learning model; and an output step of outputting a signal to the paper feeder to stop the supply of paper into the paper feeder when it is estimated that there are signs of a paper jam.

[0026] As a result, the paper jam prediction method can estimate the presence or absence of signs of a paper jam, such as paper lifting, which occurs before a paper jam, based on the output result obtained by inputting information about the friction sound when paper is supplied from the holding unit into a trained model. Therefore, unlike conventional technology, it is not necessary to have an ultrasonic transmitting unit to estimate, for example, whether or not paper lifting occurs, and it is sufficient to simply collect friction sound. Consequently, the paper jam prediction device can estimate the presence or absence of signs of a paper jam with fewer components than a configuration that includes an ultrasonic irradiation unit.

[0027] Furthermore, the paper jam prediction method estimates whether or not signs of a paper jam, such as the paper being fed floating up, are occurring. This not only prevents paper jams from occurring but also suppresses paper damage.

[0028] Furthermore, a program relating to one aspect of this disclosure is a program for causing a computer to execute the above-described paper jam prediction method.

[0029] This allows a computer to be used to achieve the same effect as the paper jam prediction method described above.

[0030] These comprehensive or specific embodiments may be implemented as systems, methods, apparatus, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs (Compact Disc Read Only memory), or as any combination of systems, methods, apparatus, integrated circuits, computer programs, and recording media.

[0031] The embodiments of this disclosure will be described in detail below with reference to the drawings. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the scope of the claims. Furthermore, among the components in the following embodiments, components that are not described in the independent claim representing the highest-level concept will be described as optional components. In addition, the figures are not necessarily strictly illustrative. In each figure, substantially identical components are denoted by the same reference numerals, and redundant explanations may be omitted or simplified.

[0032] Furthermore, in this disclosure, terms indicating relationships between elements such as parallel and perpendicular, terms indicating the shape of elements such as rectangles, and numerical values ​​do not represent only strict meanings, but also include substantially equivalent ranges, such as differences of a few percent.

[0033] (Embodiment 1) Embodiment 1 will be described in detail below with reference to the drawings.

[0034] [Paper feeder] First, the paper feeding device will be described with reference to Figures 1, 2, and 3. Figure 1 is a diagram showing an example of a paper feeding device 200 to which the paper jam prediction device 100 according to Embodiment 1 is applied. Figure 2 is a diagram showing an example of the transport section 210 of the paper feeding device 200 in Embodiment 1. Figure 3 is a diagram showing an example of the configuration of the paper jam prediction device 100 and the paper feeding device 200 according to Embodiment 1.

[0035] The paper feeder 200 supplies paper to, for example, a paper processing device (not shown) that processes paper. The processing device may be a processing device that processes the supplied paper itself or processes the paper, a copying device that copies information such as characters, symbols, and diagrams printed on the supplied paper to another recording medium, or an output device that reads the information and outputs it as an analog image signal.

[0036] As shown in Figure 1, the paper feeder 200 includes, for example, a supply port 260 for supplying paper 10 from a holding unit 270 that holds multiple sheets of paper 10 (see Figure 2), a paper feed roller 212 for supplying paper 10 from the supply port 260, a separation roller 214 for separating the paper 10 supplied from the supply port 260 one sheet at a time, and a retard roller 216 that rotates in the opposite direction to the rotation direction of the separation roller 214. In Figure 1, the multiple sheets of paper 10 shown in Figure 2 are shown as a paper stack 20 and are shaded for clarity. The dashed circles indicate points where a portion of the supplied paper 10 lifts when it is supplied from the holding unit 270. Hereafter, these points will also be referred to as paper lifting points 30. Paper lifting during supply will be described later.

[0037] Next, the transport unit 210 will be described with reference to Figure 2. In Figure 2, for the sake of clarity, the supply port 260 and the holding unit 270 are not shown, but multiple sheets of paper 10 are held in the holding unit 270 as a paper bundle 20, and the paper 10 is supplied from the supply port 260.

[0038] As shown in Figure 2, the paper feed roller 212, the separation roller 214, and the retard roller 216 are each components of the transport unit 210. The transport unit 210 separates and transports the paper 10 supplied from the holding unit 270 one sheet at a time. Hereinafter, the paper feed rollers 212a and 212b may be collectively referred to as the paper feed roller 212, the separation rollers 214a and 214b may be collectively referred to as the separation roller 214, and the retard rollers 216a and 216b may be collectively referred to as the retard roller 216.

[0039] The paper feed rollers 212a and 212b are installed to move up and down so as to contact the uppermost sheet of paper 10 among the multiple sheets of paper 10 in the stack of paper 20 held in the holding section 270, and pick up the uppermost sheet of paper 10 and supply it from the supply port 260. The paper feed roller 212 is configured to easily change its position in accordance with the change in the thickness of the stack of paper 20 in the holding section 270 as paper 10 is supplied. Alternatively, the paper feed roller 212 may be installed so as to contact the lowest sheet of paper 10 among the multiple sheets of paper 10 in the stack of paper 20. In this case, the supply port 260 is located below the stack of paper 20.

[0040] Separation rollers 214a and 214b separate the paper 10 supplied by the paper feed roller 212 one sheet at a time. Here, separation rollers 214a and 214b function as a separation unit that separates the paper 10 one sheet at a time, together with retard rollers 216a and 216b which are positioned opposite to separation rollers 214a and 214b. The retard roller 216 returns the paper 10 that has been supplied overlapping with the paper 10 that is in contact with the separation roller 214 to the holding unit 270 side.

[0041] Next, the specific operation of the transport unit 210 will be explained. First, the paper feed roller 212 rotates in the direction of arrow A, picking up the topmost sheet of paper 10 from among the multiple sheets of paper 10 held in the holding unit 270 and supplying it from the supply port 260 in the direction of arrow D. Subsequently, the separation roller 214 rotates in the direction of arrow B, supplying the paper 10 that is in contact with the separation roller 214 in the direction of arrow D. At this time, the retard roller 216 rotates in the direction of arrow C, returning the paper 10 that is in contact with the retard roller 216 in the opposite direction to arrow D. Because the torque of the retard roller 216 is limited, if only one sheet of paper 10 is supplied, the movement of the separation roller 214 will supply the paper 10 in the direction of arrow D. Furthermore, for example, if two sheets of paper 10 are supplied from the paper feed roller 212 in a stacked state, the paper 10 that comes into contact with the separation roller 214 is supplied in the direction of arrow D, and the paper 10 that comes into contact with the retard roller 216 is returned in the direction opposite to arrow D.

[0042] Through the above operation, the transport unit 210 can separate the paper 10 supplied from the supply port 260 one sheet at a time by the paper feed roller 212 and supply it to the processing device. As a result, the transport unit 210 can reduce double feeding of the paper 10 supplied from the holding unit 270, and thus reduce paper jams in the paper feed device 200.

[0043] Next, the functional configuration of the paper feeder 200 will be described with reference to Figure 3. Here, the configurations described with reference to Figures 1 and 2 will be omitted or simplified in the explanation.

[0044] As shown in Figure 3, the paper feeder 200 includes, for example, a transport unit 210, a drive unit 220, a control unit 230 that controls the movement of the drive unit 220, a storage unit 240, and a communication unit 250.

[0045] The drive unit 220 drives the paper feed roller 212, the separation roller 214, and the retard roller 216 of the transport unit 210, respectively. For example, the drive unit 220 includes one or more motors and rotates the paper feed roller 212, the separation roller 214, and the retard roller 216 according to control signals from the control unit 230.

[0046] As described above, the control unit 230 performs information processing to control the operation of the transport unit 210. The control unit 230 may be implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

[0047] The memory unit 240 is a storage device that stores control programs and other data executed by the control unit 230. The memory unit 240 is implemented, for example, by semiconductor memory.

[0048] The communication unit 250 is a communication module (communication line) for the paper feed device 200 to communicate with the paper jam prediction device 100 and the processing device (not shown) via a local communication network. The communication performed by the communication unit 250 may be, for example, wireless communication or wired communication. There are no particular limitations on the communication standard used for communication.

[0049] [Paper jam prediction device] [1. Overview, etc.] Next, an overview of the paper jam prediction device 100 according to Embodiment 1 will be described with reference to Figures 1 and 4. Figure 4 is a diagram showing an example of a paper jam prediction in Embodiment 1. As with Figure 2, the holding part 270 is omitted from the illustration in Figure 4 for the sake of clarity.

[0050] The paper jam prediction device 100 is a device that estimates whether or not there are signs of a paper jam in the paper feed device 200. Specifically, the paper jam prediction device 100 collects friction sounds generated when paper 10 is supplied from the holding unit 270, which holds multiple sheets of paper 10, into the inside of the paper feed device 200. Based on the output results obtained by inputting the information about the collected friction sounds into a trained model, the device estimates whether or not there are signs of a paper jam in the paper feed device 200. If the paper jam prediction device 100 estimates that there are signs of a paper jam, it outputs a signal to the paper feed device 200 to stop the supply of paper 10 from the holding unit 270 into the inside of the paper feed device 200.

[0051] The trained model is a pre-trained machine learning model. The trained model is obtained by training performed by the training unit 140. The trained model is constructed by learning the relationship between the friction sound generated when paper 10 is supplied from the holding unit 270 into the paper feed device 200 and the presence or absence of signs of a paper jam. The information about the friction sound input to the trained model is, for example, an image of the spectrogram of the friction sound or an image of its frequency characteristics.

[0052] Friction noise is, for example, the friction noise generated when paper 10 is supplied from the holding part 270, due to friction between the supplied paper 10 and the paper 10 held by the holding part 270. Paper 10 held by the holding part 270 includes paper 10 in which only a portion is held by the holding part 270. Friction noise can also be, for example, the friction noise generated when paper 10 supplied from the holding part 270 is supplied at an angle to the supply opening 260 instead of straight, due to friction between the supplied paper 10 and the paper 10 held by the holding part 270, or due to friction between the supplied paper 10 and the inner wall of the holding part 270 or the surrounding material of the supply opening 260. Furthermore, friction noise may occur when the condition of the paper 10 is different from normal, such as when a portion of the supplied paper 10 is bent, wrinkled, or has a sticky note or sticker attached to it. Fricative sounds may include audible sounds that can be heard by the human ear, as well as inaudible sounds that cannot be heard by the human ear, but they may also be inaudible. Inaudible sounds are, for example, sounds with frequencies in the ultrasonic range. If the fricative sound is a sound with frequencies in the ultrasonic range, the frequency range of the fricative sound may be 60 kHz or more and 95 kHz or less, more specifically 75 kHz or more and 95 kHz or less, and more particularly 85 kHz or more and 90 kHz or less.

[0053] The paper 10 supplied from the holding unit 270 may be one sheet or several sheets. The paper 10 is separated one sheet at a time by the separation roller 214 and retard roller 216, which will be described later, and supplied to the processing device. The paper 10 held in the holding unit 270 that generates friction with the supplied paper 10 may be the paper 10 located at the top of the plurality of papers 10 held in the holding unit 270, or it may be a plurality of papers 10 including the paper 10 located at the top. The paper jam prediction device 100 collects the friction sound generated when the paper 10 is supplied from the holding unit 270 into the paper feed device 200, and estimates whether or not there is a paper jam in the paper feed device 200 based on the output result obtained by inputting the information on the collected friction sound into a trained model.

[0054] The warning signs of a paper jam in the paper feed device 200 are precursors to a paper jam, occurring immediately before a paper jam occurs due to the cause of the jam. For example, as shown in Figure 4, let's consider the case where the cause of the paper jam is that the supplied paper 10 is stapled together with a stapler 15 (hereinafter also referred to as stapling). For example, when multiple stapled papers 10 are fed in the direction of arrow D by the paper feed roller 212, only the paper 10 that comes into contact with the separation roller 214 is fed in the direction of arrow D by the separation roller 214. At this time, the paper 10 lifts up around the stapled area. This phenomenon occurs at the paper lifting location 30 shown in Figures 1 and 4. Then, when the paper 10 is further fed in the direction of arrow D by the separation roller 214, the paper 10 rotates and distorts around the stapled area. Then, if the paper 10 continues to be fed in the direction of arrow D by the separation roller 214, a paper jam will occur. Thus, the phenomenon of a portion of the paper 10 supplied from the holding unit 270 floating is a sign of an impending paper jam. The paper jam prediction device 100 estimates whether or not there is an impending paper jam based on the output result obtained by inputting information about the friction sound generated by the friction between the paper 10 supplied from the holding unit 270 and the multiple pieces of paper 10 held by the holding unit 270 into a trained model.

[0055] Here, staples were used as an example to explain the cause of a paper jam, but the causes of a paper jam are not limited to this. Causes of a paper jam also include, for example, when a portion of the supplied paper 10 is bent, when sticky notes or the like are attached to the supplied paper 10, when a portion of the supplied paper 10 is glued to another piece of paper 10, or when the paper quality of the supplied paper 10 differs from that of the other pieces of paper 10, such as when the surface of the paper 10 is rough.

[0056] When paper 10 is supplied from the holding unit 270, a portion of the supplied paper 10 floats near the separation roller 214, and particularly between the separation roller 214 and the paper feed roller 212. For example, the portion of the supplied paper 10 that floats may be on the side in front of the point where the separation roller 214 contacts the topmost sheet of paper 10 when two or more sheets of paper 10 are fed together by the paper feed roller 212. Here, "front side" refers to the direction opposite to the paper supply direction (direction of arrow D in the figure). In other words, "front side" refers to the side of the holding unit 270 when looking at the supply opening 260 from the holding unit 270.

[0057] As described above, the paper jam prediction device 100 can estimate whether or not a portion of the supplied paper 10 will float (paper floating) based on the output result obtained by inputting information about the friction sound between the sheets of paper 10 into a trained model when the separation roller 214 separates only the sheet of paper 10 that is in contact with the separation roller 214 from the two or more sheets of paper 10 that have been double-fed by the paper feed roller 212 and sends it to the processing device.Therefore, the paper jam prediction device 100 can stop the supply of paper 10 before the paper 10 that has floated rotates or is supplied at an angle to the supply direction.In this way, the paper jam prediction device 100 can not only reduce the occurrence of paper jams, but also suppress damage to the supplied paper 10 such as bending, wrinkling, or tearing.

[0058] The friction noise between the papers 10 is caused, for example, by the friction between the supplied paper 10 and the paper 10 held in the holding section 270. The paper 10 held in the holding section 270 includes paper 10 in which only a part is held in the holding section 270. Therefore, the friction noise between the papers 10 is caused, for example, by the friction between the paper 10 that is in contact with the separation roller 214 and the other paper 10 that is not in contact with the separation roller 214 when two or more sheets of paper 10 are fed in a double feed towards the separation roller 214 by the paper feed roller 212 and paper floating occurs near the separation roller 214.

[0059] Furthermore, the paper jam prediction device 100 does not need to irradiate the multiple sheets of paper 10 held in the holding unit 270 with ultrasonic waves and detect the reflected waves of the irradiated ultrasonic waves in order to estimate the presence or absence of a paper jam (i.e., an indication of a paper jam). Instead, it picks up the friction sound between the sheets of paper 10, which is a sound in the ultrasonic frequency band. In other words, the paper jam prediction device 100 only needs to be equipped with a passive ultrasonic sensor instead of an active ultrasonic sensor, so it can estimate the presence or absence of an indication of a paper jam with a simpler configuration.

[0060] [2. Structure] Next, the configuration of the paper jam prediction device 100 will be explained with reference to Figure 3.

[0061] The paper jam prediction device 100 comprises an information processing unit 110, a storage unit 120, a communication unit 130, and a learning unit 140. The following describes each of these components.

[0062] [Information Processing Department] The information processing unit 110 performs information processing related to the estimation of signs of a paper jam. The information processing unit 110 is implemented, for example, by a microcomputer or processor. Specifically, the information processing unit 110 comprises a sound collection unit 112, an estimation unit 114, and an output unit 116.

[0063] [Sound recording section] The sound-collecting unit 112 collects the frictional sound generated when the paper 10 is supplied from the holding unit 270, which holds multiple sheets of paper 10. More specifically, the sound-collecting unit 112 collects the frictional sound generated by the friction between the paper 10 supplied from the holding unit 270 and the paper 10 held by the holding unit 270. The sound-collecting unit 112 is, for example, a microphone. In this case, the sound-collecting unit 112 converts the collected frictional sound into an electrical signal and outputs the electrical signal to the estimation unit 114.

[0064] Furthermore, if the sound-collecting unit 112 is a microphone, it is installed in a position where it can pick up the friction sound of the papers 10 together. For example, the sound-collecting unit 112 may be installed closer to the holding unit 270 than the position of the separation roller 214, that is, closer to the separation roller 214 when viewed from the holding unit 270. More specifically, the sound-collecting unit 112 may be installed above the holding unit 270. In particular, the sound-collecting unit 112 may be installed near the supply port 260. Near the supply port 260 means, for example, from the midpoint between the supply port 260 and the separation roller 214 to the top of the paper feed roller 212. In particular, the sound-collecting unit 112 may be installed above the supply port 260 and in a direction intersecting the direction in which the paper 10 is supplied from the holding unit 270, alongside the paper feed roller 212. More specifically, the sound-collecting unit 112 may be installed above the supply opening 260 and in a direction intersecting the direction in which the paper 10 is supplied from the holding unit 270, at the same height as the paper feed roller 212. The sound-collecting unit 112 only needs to be installed at a height that does not come into contact with the supplied paper 10, and for example, it may be installed at the same height as the rotation axis of the paper feed roller 212, and installed alongside the paper feed roller 212.

[0065] Furthermore, the sound-collecting unit 112 only needs to be installed in a position where it can pick up the friction sound between the papers 10, and is not limited to being located above the supply port 260. For example, the sound-collecting unit 112 may be installed below the supply port 260, or on the side of the supply port 260.

[0066] In Figure 3, the paper jam prediction device 100 is shown as having one sound-collecting unit 112, but it may also have two or more sound-collecting units 112. For example, multiple (i.e., two or more) sound-collecting units 112 may be installed around the paper feed roller 212 in a direction intersecting the direction in which the paper 10 is supplied from the holding unit 270.

[0067] [Estimation part] The estimation unit 114 uses a pre-trained machine learning model (a so-called pre-trained model) stored in the memory unit 120 to input information about the friction sound picked up by the sound pickup unit 112 into the pre-trained model. Based on the output result obtained from this input, the estimation unit 114 estimates whether or not there is a paper jam. The specific operation of the estimation unit 114 will be described later.

[0068] The information about the fricative sound input to the trained model is, for example, an image of the spectrogram or frequency response of the fricative sound. This information may be image data in a format such as JPEG (Joint Photographic Experts Group) or BMP (Basic Multilingual Plane), but it does not have to be image data. In this case, the information may also be numerical data (more specifically, time-series numerical data) in a format such as WAV (Waveform Audio File Format). This information may include, for example, the frequency band of the fricative sound, the duration of the fricative sound, the sound pressure, and at least one of the waveform.

[0069] Furthermore, the output result may be, for example, whether or not there is a sign of a paper jam, whether or not there is a decrease in friction, or the absolute value of the friction sound or its relative value to a predetermined value. Whether or not there is a decrease in friction between the papers 10 may be information indicating whether or not the friction sound (more specifically, the sound pressure of the friction sound) has decreased below a predetermined value (for example, if it is the absolute value of the difference in sound pressure, whether or not it has increased above the predetermined value).

[0070] [Output section] If the estimation unit 114 estimates that there is a sign of a paper jam, the output unit 116 outputs a signal to the paper feed device 200 to stop the supply of paper 10 from the holding unit 270 into the paper feed device 200.

[0071] [Storage] The memory unit 120 is a storage device that stores computer programs executed by the information processing unit 110. The memory unit 120 may temporarily store training data and data related to fricative sounds collected by the sound collection unit 112. The memory unit 120 updates the stored trained model with a machine learning model (so-called trained model) generated by the learning unit 140. The memory unit 120 is implemented by semiconductor memory or an HDD (Hard Disk Drive), etc.

[0072] [g section] The communication unit 130 is a communication path for the paper jam prediction device 100 to communicate with the paper feed device 200. Communication between the communication unit 130 and the paper feed device 200 may be performed directly or via a relay device such as a wireless router (not shown). The communication unit 130 may be, for example, a wireless communication circuit that performs wireless communication, or a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard used by the communication unit 130.

[0073] [Learning Department] The learning unit 140 performs machine learning using training data. For example, the learning unit 140 uses machine learning to create a machine learning model that takes information about friction sounds as input and outputs whether or not there are signs of a paper jam. The output may be whether or not there are signs of a paper jam, or it may be whether or not there is a decrease in friction between the papers 10. The trained model is constructed by learning the relationship between the friction sounds between the papers 10 and whether or not there are signs of a paper jam. As the signs of a paper jam have been described above, the explanation is omitted here.

[0074] The training data used to train a machine learning model includes, for example, a first set of data consisting of information about friction sounds and annotations indicating the occurrence of a paper jam (in other words, signs of a paper jam), and a second set of data consisting of information about friction sounds and annotations indicating the absence of a paper jam (in other words, no signs of a paper jam). More specifically, the training data includes, for example, a first set of data in which an image of a spectrogram or frequency response of a friction sound is labeled as indicating signs of a paper jam, and a second set of data in which an image of a spectrogram or frequency response of a friction sound is labeled as not indicating signs of a paper jam. More specifically, the training data is a dataset containing multiple sets of information about friction sounds recorded in the past and paper jam information indicating whether or not a paper jam has occurred.

[0075] The machine learning model is, for example, a neural network model, and more specifically, a convolutional neural network (CNN) model. The machine learning model does not have to be a CNN and is not particularly limited, but for example, if the information about the fricative sound is time-series numerical data (e.g., a spectrogram of the fricative sound or time-series numerical data of the frequency characteristics), it may be a recurrent neural network (RNN) model. In other words, the machine learning model may be appropriately selected depending on the format of the input data. The trained machine learning model (so-called trained model) generated by the learning unit 140 includes trained parameters that have been adjusted by machine learning. The learning unit 140 stores the generated trained model in the storage unit 120. The learning unit 140 is implemented, for example, by a processor executing a program stored in the storage unit 120.

[0076] [3. Operation] Next, the operation of the paper jam prediction device 100 will be explained. Figure 5 is a flowchart showing the operation of the paper jam prediction device 100 according to Embodiment 1.

[0077] As shown in Figure 5, the sound-collecting unit 112 collects the friction sound generated when the paper 10 is supplied from the holding unit 270 into the paper feeding device 200 (S101). Here, the sound-collecting unit 112 is, for example, a microphone, which converts the collected friction sound into an electrical signal and outputs the converted electrical signal to the estimation unit 114. The microphone includes a microphone device. For example, the sound-collecting unit 112 may be a microphone capable of collecting inaudible sounds, or a microphone capable of collecting both audible and inaudible sounds and extracting sounds in a specific frequency band. Furthermore, the sound-collecting unit 112 may be a directional microphone. The sound-collecting unit 112 may be, for example, a MEMS microphone. Inaudible sounds are, for example, sounds with frequencies in the ultrasonic band. For example, if the estimation unit 114 receives information about inaudible sounds from the collected frictional sounds as input, the sound collection unit 112 may extract the inaudible sounds (for example, sounds with frequencies in the ultrasonic range) from the collected frictional sounds, convert them into electrical signals, and output the converted electrical signals to the estimation unit 114.

[0078] Next, the estimation unit 114 inputs the information regarding the fricative sound picked up by the sound pickup unit 112 into the trained model and obtains an output result (S102). More specifically, in step S102, first, the estimation unit 114 acquires the electrical signal output from the sound pickup unit 112 and converts the acquired electrical signal into a digital signal using PCM (Pulse Code Modulation) or the like. At this time, for example, the estimation unit 114 may acquire the electrical signal of the fricative sound, including audible and inaudible sounds, picked up by the sound pickup unit 112, convert the electrical signal into a digital signal, and then extract the digital signal of the inaudible sound. Next, the estimation unit 114 generates a spectrogram image or a frequency characteristic image of the fricative sound based on the digital signal. The spectrogram image or frequency characteristic image of the fricative sound is information regarding the fricative sound input into the trained model, but a digital signal (i.e., time-series numerical data of the spectrogram or frequency characteristic of the fricative sound) may also be used as information regarding the fricative sound. Next, the estimation unit 114 inputs the generated friction sound information into the trained model and obtains an output result. As described above, the output result may be whether or not there is a sign of a paper jam, whether or not there is a decrease in friction between the papers 10, or the absolute value of the friction sound or a relative value to a predetermined value.

[0079] Next, the estimation unit 114 estimates whether there are signs of a paper jam based on the output result obtained in step S102 (S103). If the estimation unit 114 estimates in step S103 that there are signs of a paper jam (Yes in S104), the output unit 116 outputs a signal to the paper feeder 200 to stop the supply of paper 10 from the holding unit 270 into the paper feeder 200 (S105). More specifically, if the trained model provides an output result indicating "there are signs of a paper jam" in step S104, the estimation unit 114 estimates that there are signs of a paper jam based on this output result. Alternatively, if the trained model provides an output result indicating "there is a decrease in friction between the papers 10" in step S104, the estimation unit 114 may estimate that there are signs of a paper jam based on this output result.

[0080] On the other hand, if the estimation unit 114 estimates in step S103 that there are no signs of a paper jam (No in S104), the paper jam prediction estimation device 100 returns to the process of step S101. More specifically, in step S104, if the trained model outputs "no signs of a paper jam," the estimation unit 114 estimates that there are no signs of a paper jam based on that output. Also, in step S104, if the trained model outputs "no decrease in friction between the papers 10," the estimation unit 114 estimates that there are no signs of a paper jam based on that output.

[0081] The paper jam prediction device 100 repeatedly executes the above processing flow each time paper 10 is supplied from the holding unit 270.

[0082] [4. Effects, etc.] As described above, the paper jam prediction device 100 according to Embodiment 1 is a paper jam prediction device that estimates signs of a paper jam in a paper feed device 200, and comprises: a sound collection unit 112 that collects friction sounds generated when paper 10 is supplied into the inside of the paper feed device 200 from a holding unit 270 that holds a plurality of sheets of paper 10; an estimation unit 114 that estimates whether or not there are signs of a paper jam in the paper feed device 200 based on the output result obtained by inputting information on the friction sounds into a trained model, which is a trained machine learning model; and an output unit 116 that outputs a signal to the paper feed device 200 to stop the supply of paper 10 into the inside of the paper feed device 200 when the estimation unit 114 estimates that there are signs of a paper jam.

[0083] As a result, the paper jam prediction device 100 can collect friction sounds when the paper 10 is supplied from the holding unit 270 to the inside of the paper feeder 200, and based on the output results obtained by inputting information about the collected friction sounds into a trained model, it can estimate whether or not there are any signs of a paper jam, such as the paper 10 lifting up, which occurs before a paper jam.Therefore, unlike the conventional technology, it is not necessary to have an ultrasonic transmitting unit to estimate, for example, whether or not the paper 10 lifts up, and it is sufficient to have a sound collecting unit 112 that collects friction sounds.Indeed, the paper jam prediction device 100 can estimate the presence or absence of signs of a paper jam in a simple manner with fewer components than a configuration that includes an ultrasonic irradiation unit.

[0084] Furthermore, the paper jam prediction device 100 estimates whether or not signs of a paper jam, such as the paper 10 being lifted, are occurring. This not only prevents paper jams from occurring but also suppresses damage to the paper 10.

[0085] In the paper jam prediction device 100 according to Embodiment 1, the information regarding frictional sound input to the trained model may be an image of the spectrogram of the frictional sound or an image of its frequency characteristics.

[0086] As a result, the paper jam prediction device 100 can more easily extract regularity (so-called features) from images by using a machine learning model. Therefore, the paper jam prediction device 100 can more easily estimate whether or not there is a paper jam.

[0087] In the paper jam prediction device 100 according to Embodiment 1, the friction sound may be an inaudible sound generated by friction between the paper 10 supplied from the holding unit 270 and the paper 10 held in the holding unit 270. In this case, the inaudible sound may be a sound with a frequency in the ultrasonic band.

[0088] As a result, the paper jam prediction device 100 estimates whether or not the paper 10 is floating based on inaudible sounds (for example, sounds with ultrasonic frequencies) in the friction noise generated when the paper 10 is supplied from the holding unit 270. Therefore, it is less susceptible to the influence of various audible sounds, i.e., noise, generated around the paper jam prediction device, and the sound collection accuracy is improved. Consequently, the paper jam prediction device 100 can accurately estimate whether or not there is a paper jam.

[0089] In the paper jam prediction device 100 according to Embodiment 1, the training data used to train the machine learning model may include first data consisting of information about friction sounds and annotations indicating the occurrence of a paper jam, and second data consisting of information about friction sounds and annotations indicating that no paper jam has occurred.

[0090] As a result, the paper jam prediction device 100 can accurately estimate whether or not there are signs of a paper jam because the learning accuracy of the learning unit 140 is improved.

[0091] In the paper jam prediction device 100 according to Embodiment 1, the machine learning model may be a convolutional neural network model.

[0092] As a result, the paper jam prediction device can more easily extract image regularities (so-called features) by using a convolutional neural network model.

[0093] (Modification 1 of Embodiment 1) Next, a paper jam prediction device 100a according to Modification 1 of Embodiment 1 will be described with reference to Figure 6. Figure 6 is a diagram showing an example of the configuration of a paper jam prediction device 100a according to Modification 1 of Embodiment 1. In Embodiment 1, an example was described in which the sound collection unit 112 is a microphone, but in Modification 1 of Embodiment 1, the sound collection unit 112a differs from Embodiment 1 in that it acquires an electrical signal including friction sound output from the microphone 300. In the following, the differences from Embodiment 1 will be explained in detail, and redundant explanations will be simplified or omitted.

[0094] [1. Structure] As shown in Figure 6, in Modification 1 of Embodiment 1, the paper jam prediction device 100a is connected to the microphone 300 via the communication unit 130. The paper jam prediction device 100a comprises an information processing unit 110a, a storage unit 120, a communication unit 130, and a learning unit 140. The information processing unit 110a comprises a sound collection unit 112a, an estimation unit 114, and an output unit 116. The sound collection unit 112a will be described below.

[0095] The sound-collecting unit 112a acquires, for example, a frictional sound picked up by at least one microphone 300 as an electrical signal and outputs the acquired electrical signal to the estimation unit 114. At this time, the sound-collecting unit 112a may also acquire, for example, the electrical signal output from at least one microphone 300 and information indicating the microphone 300 that output the electrical signal, and output the acquired information and the electrical signal to the estimation unit 114. Furthermore, for example, if the frictional sound is an inaudible sound (for example, a sound with a frequency in the ultrasonic band), the sound-collecting unit 112a may extract an electrical signal indicating the sound pressure with a frequency in the ultrasonic band from the acquired electrical signal and output it to the estimation unit 114.

[0096] [2. Operation] In the first modification of Embodiment 1, the sound-collecting unit 112a acquires an electrical signal corresponding to the friction sound picked up by the microphone 300, so the process of step S101 in Figure 5, which was referenced in Embodiment 1, is different.

[0097] For example, in step S101 of Figure 5, the sound-collecting unit 112a acquires an electrical signal corresponding to the friction sound picked up by the microphone 300. The sound-collecting unit 112a then outputs the acquired electrical signal to the estimation unit 114. In this case, the sound-collecting unit 112a functions as a so-called acquisition unit.

[0098] Furthermore, for example, if friction sounds are picked up by multiple microphones 300, in step S101 of Figure 5, the sound pickup unit 112a acquires electrical signals corresponding to the friction sounds picked up by the multiple microphones 300. At this time, the sound pickup unit 112a may acquire electrical signals output from the multiple microphones 300 and information indicating which microphone 300 output the electrical signals. The sound pickup unit 112a then outputs the acquired electrical signals and information to the estimation unit 114.

[0099] As described above, in the modified example 1 of Embodiment 1, the paper jam prediction device 100a differs from Embodiment 1 in that it acquires an electrical signal including friction sound picked up by the microphone 300 and performs information processing related to prediction estimation.

[0100] [3. Effects, etc.] The paper jam prediction device 100a according to the modified example 1 of Embodiment 1 is configured separately from the microphone 300, so the installation position and number of microphones 300 can be appropriately changed according to the design, and the paper jam prediction device 100a can be mounted on a single integrated circuit.

[0101] (Embodiment 2) Next, a paper jam prediction device according to Embodiment 2 will be described. Figure 7 is a diagram showing an example of the configuration of the paper jam prediction device 100b and paper feed device 200 according to Embodiment 2. The paper jam prediction device 100b according to Embodiment 2 differs from Embodiment 1 and Modification 1 of Embodiment 1 in that, in addition to the configuration of Embodiment 1, it is equipped with an identification unit 113a that identifies the type of paper 10 supplied from the holding unit 270 to the inside of the paper feed device 200, and the learned model includes multiple learned models corresponding to each of multiple types of paper. In the following, the differences from Embodiment 1 and Modification 1 of Embodiment 1 will be explained in detail, and redundant explanations will be omitted or simplified.

[0102] [1. Structure] The paper jam prediction device 100b comprises an information processing unit 110b, a storage unit 120, a communication unit 130, and a learning unit 140a. The information processing unit 110b comprises a sound collection unit 112, an identification unit 113a, an estimation unit 114a, and an output unit 116. The identification unit 113a, the estimation unit 114a, and the learning unit 140a will be described below.

[0103] [Identification section] The identification unit 113a identifies the type of paper 10 supplied from the holding unit 270 to the inside of the paper feeder 200. More specifically, the identification unit 113a identifies the type of paper 10 based on the friction sound generated between the papers 10 when they are supplied from the holding unit 270 to the inside of the paper feeder 200. For example, the identification unit 113a uses the sound pickup unit 112's pickup of friction sound as a trigger to acquire the friction sound from the sound pickup unit 112. Here, the type of paper 10 may include not only the type of paper 10 (e.g., copy paper, typewriter paper, tracing paper, cardboard, etc.) but also the size of the paper 10 (e.g., A4 size, B5 size, A3 size, etc.).

[0104] For example, the identification unit 113a identifies the type of paper 10 based on the output result obtained by inputting the acquired friction sound (more specifically, information about the friction sound) into a trained model that shows the relationship between the friction sound and the type of paper 10 (hereinafter also referred to as the second trained model). The friction sound used by the identification unit 113a may be a friction sound from a different time than the friction sound used by the estimation unit 114a. More specifically, the friction sound used to identify the paper 10 is the friction sound between the paper 10 when the paper 10 begins to be supplied from the holding unit 270 to the supply port 260 by the paper feed roller 212 (see Figure 1), and is the friction sound between the paper 10 before paper lifting occurs near the separation roller 214.

[0105] [Estimation part] The estimation unit 114a inputs information about the friction sound based on the type of paper 10 identified by the identification unit 113a to a trained model (hereinafter also referred to as the first trained model) corresponding to the identified type of paper 10. The trained model created by the learning unit 140a includes multiple first trained models corresponding to each of the multiple types of paper 10. Based on the type of paper 10 identified by the identification unit 113a, the estimation unit 114a selects the first trained model corresponding to the identified type of paper 10 from among the multiple first trained models stored in the storage unit 120. Then, the estimation unit 114a acquires the friction sound between the paper 10 collected by the sound collection unit 112 and inputs the acquired information about the friction sound to the selected first trained model. Based on the output result of the first trained model, the estimation unit 114a estimates whether or not there is a sign of a paper jam.

[0106] [Learning Department] The learning unit 140a performs machine learning using training data. For example, the learning unit 140a uses machine learning to create multiple first trained models for each of the multiple types of paper 10, taking information about friction sounds as input and outputting whether or not there are signs of a paper jam, such as paper floating. In other words, the learning unit 140a creates a first trained model corresponding to each of the multiple types of paper 10. The training data includes, for each of the multiple types of paper 10 (in other words, for each type of paper 10), first data consisting of information about friction sounds and annotations indicating the occurrence of a paper jam, and second data consisting of information about friction sounds and annotations indicating the absence of a paper jam.

[0107] Furthermore, the learning unit 140a uses machine learning to create a second trained model that takes friction sound (i.e., information about friction sound) as input and outputs the type of paper 10. The training data includes data consisting of information about friction sound and annotations indicating the type of paper 10. The information about friction sound used in the training data may be generated using the friction sound between the papers 10 when the paper 10 begins to be supplied from the holding unit 270 to the supply port 260 by the paper feed roller 212, or it may be generated using the friction sound when no paper jam occurs. The information about friction sound may be, for example, a spectrogram image of the friction sound, or an image of the frequency characteristics of the friction sound. In this case, the machine learning model may be a CNN model. Alternatively, the information about friction sound may be time-series numerical data including an electrical signal (e.g., a digitally converted signal) corresponding to the friction sound. In this case, the machine learning model may be an RNN model.

[0108] [2. Operation] Next, the operation of the paper jam prediction device 100b will be explained. Figure 8 is a flowchart showing the operation of the paper jam prediction device 100b according to Embodiment 2.

[0109] As shown in Figure 8, the sound-collecting unit 112 collects the friction sound generated when the paper 10 is supplied from the holding unit 270 into the paper feeding device 200 (S201). Here, the sound-collecting unit 112 is, for example, a microphone, but as in the modified example 1 of Embodiment 1, it may also function as an acquisition unit that acquires the friction sound collected by the microphone 300 as an electrical signal.

[0110] Next, the identification unit 113a identifies the type of paper 10 supplied from the holding unit 270 to the inside of the paper feed device 200 based on the friction sound picked up by the sound pickup unit 112 in step S201 (S202). For example, the identification unit 113a acquires the friction sound from the sound pickup unit 112, triggered by the sound pickup of the friction sound by the sound pickup unit 112. At this time, the identification unit 113a identifies the type of paper 10 based on the data acquired by machine learning. For example, the identification unit 113a may identify the type of paper 10 based on the output result obtained by inputting the friction sound (more specifically, information about the friction sound) into a second trained model that shows the relationship between the friction sound of the paper 10 and the type of paper 10.

[0111] Next, the estimation unit 114a inputs the information regarding the friction sound picked up by the sound pickup unit 112 in step S201 to the first trained model corresponding to the type of paper 10 identified by the identification unit 113a in step S202, and obtains the output result (S203). More specifically, based on the type of paper 10 identified by the identification unit 113a, the estimation unit 114a selects the first trained model corresponding to the type of paper 10 from among a plurality of first trained models stored in the storage unit 120, and inputs the information regarding the friction sound to the selected first trained model. In other words, the estimation unit 114a switches the first trained model according to the type of paper 10 supplied from the holding unit 270 into the paper feed device 200.

[0112] Next, the estimation unit 114a estimates whether or not there are signs of a paper jam based on the output result obtained in step S203 (S204). If the estimation unit 114a estimates in step S204 that there are signs of a paper jam (Yes in S205), the output unit 116 outputs a signal to the paper feed device 200 to stop the supply of paper 10 from the holding unit 270 into the inside of the paper feed device 200 (S206). On the other hand, if the estimation unit 114a estimates in step S204 that there are no signs of a paper jam (No in S205), the paper jam prediction estimation device 100b returns to the process of step S201.

[0113] [3. Effects, etc.] As described above, in the paper jam prediction device 100b according to Embodiment 2, the learned model includes a plurality of learned models (so-called first learned models) corresponding to each of the plurality of types of paper 10, and the paper jam prediction device 100b further includes an identification unit 113a that identifies the type of paper 10 supplied from the holding unit 270 into the paper feeding device 200, and the estimation unit 114a inputs information regarding friction noise to the learned model (so-called first learned model) corresponding to the identified type of paper 10 based on the type of paper 10 identified by the identification unit 113a.

[0114] As a result, the paper jam prediction device 100b can switch the trained model used depending on the type of paper 10 supplied from the holding unit 270 into the paper feed device 200. Therefore, the paper jam prediction device 100b can accurately estimate whether or not there is a paper jam depending on the type of paper 10.

[0115] In the paper jam prediction device 100b according to Embodiment 2, the identification unit 113a may identify the type of paper 10 based on data acquired by machine learning at least.

[0116] As a result, the paper jam prediction device 100b can accurately identify the type of paper 10 being supplied based on data acquired through machine learning.

[0117] (Modification 1 of Embodiment 2) Next, a paper jam prediction device according to Modification 1 of Embodiment 2 will be described. Figure 9 is a diagram showing an example of the configuration of the paper jam prediction device 100c and paper feed device 200 according to Modification 1 of Embodiment 2. The paper jam prediction device 100b according to Embodiment 2 identifies the type of paper 10 based on friction sounds picked up by the sound pickup unit 112, but the paper jam prediction device 100c according to Modification 1 of Embodiment 2 differs from Embodiment 2 in that it identifies the type of paper 10 based on sensing data indicating the characteristics of the paper 10, such as the surface roughness of the paper 10.

[0118] [1. Structure] The paper jam prediction device 100c comprises an information processing unit 110c, a storage unit 120, a communication unit 130, a learning unit 140b, and a sensor unit 150. The information processing unit 110c comprises a sound collection unit 112, an identification unit 113b, an estimation unit 114a, and an output unit 116. The identification unit 113b, estimation unit 114a, learning unit 140b, and sensor unit 150 will be described below.

[0119] [Identification section] The identification unit 113b identifies the type of paper 10 supplied from the holding unit 270 to the inside of the paper feeder 200. More specifically, the identification unit 113b identifies the type of paper 10 based on data acquired by the sensor unit 150 (also called sensing data). Sensing data is data that indicates the characteristics of the paper 10. Characteristics of the paper 10 include, for example, the smoothness of the surface of the paper 10, the presence or absence of gloss on the surface, thickness, weight, or size. The identification unit 113b may identify the type of paper 10 using a database that associates sensing data with the type of paper 10, or it may identify the type of paper 10 using a trained model (hereinafter also called the third trained model) that takes sensing data as input and outputs the type of paper 10. The identification unit 113b may use both the database and the third trained model in combination.

[0120] [Estimation part] The estimation unit 114a inputs information about the friction sound to a first trained model corresponding to the identified paper 10, based on the type of paper 10 identified by the identification unit 113b. More specifically, the estimation unit 114a selects a first trained model corresponding to the identified paper 10 from among a plurality of first trained models stored in the storage unit 120, based on the type of paper 10 identified by the identification unit 113b. Then, the estimation unit 114a acquires the friction sound between the paper 10s picked up by the sound pickup unit 112 and inputs the acquired information about the friction sound to the selected first trained model. Based on the output result of the first trained model, the estimation unit 114a estimates whether or not there is a sign of a paper jam.

[0121] [Learning Department] The learning unit 140b performs machine learning using training data. Similar to Embodiment 2, the learning unit 140b creates a first trained model corresponding to each of the multiple types of paper 10.

[0122] Furthermore, the learning unit 140b may, by machine learning, create multiple third pre-trained models for each of several types of paper 10, taking at least one of the data representing the characteristics of the paper 10, such as the surface roughness of the paper 10, the surface reflectance of the paper 10, and the light transmittance of the paper 10, as input, and outputting the type of paper 10. The training data includes data consisting of information representing the characteristics of the paper 10 and annotations representing the type of paper 10. The information representing the characteristics of the paper 10 may, for example, be data representing at least one of the surface roughness of the paper 10, the surface reflectance of the paper 10, and the light transmittance of the paper 10. The form of such data may be, for example, an image or time-series numerical data.

[0123] [Sensor unit] The sensor unit 150 acquires data (sensing data) indicating the characteristics of the paper 10 supplied from the holding unit 270 to the inside of the paper feeding device 200. For example, the sensor unit 150 operates using the sound collection of frictional sound by the sound collection unit 112 as a trigger. The sensor unit 150 includes, for example, at least one of an image sensor, an ultrasonic sensor, an optical sensor, and a weight sensor. The image sensor acquires image data indicating the surface characteristics of the paper 10 by imaging the paper 10. The ultrasonic sensor acquires data indicating the thickness of the paper 10 by transmitting ultrasonic waves through the paper 10. The optical sensor acquires data indicating the smoothness and glossiness of the surface of the paper 10 by irradiating the surface of the paper 10 with light. The weight sensor acquires data indicating the weight of the paper 10.

[0124] [2. Operation] Next, the operation of the paper jam prediction device 100c will be explained. Figure 10 is a flowchart showing the operation of the paper jam prediction device according to Modification 1 of Embodiment 2.

[0125] As shown in Figure 10, the sound-collecting unit 112 collects the friction sound generated when the paper 10 is supplied from the holding unit 270 into the paper feeding device 200 (S301). Here, the sound-collecting unit 112 is, for example, a microphone, but as in the modified example 1 of Embodiment 1, it may also function as an acquisition unit that acquires the friction sound collected by the microphone 300 as an electrical signal.

[0126] Although not shown in the diagram, the sensor unit 150 operates using the sound pickup unit 112's pickup of frictional sound as a trigger to acquire data that indicates the characteristics of the paper 10.

[0127] Next, the identification unit 113b identifies the type of paper 10 supplied from the holding unit 270 to the inside of the paper feed device 200 based on the data acquired by the sensor unit 150 (S302). For example, the identification unit 113b may identify the type of paper 10 using a database that associates sensing data with the type of paper 10, or it may identify the type of paper 10 using a trained model (hereinafter also referred to as the third trained model) that takes sensing data as input and outputs the type of paper 10. The identification unit 113b may use both the database and the third trained model in combination.

[0128] Next, the estimation unit 114a inputs the information regarding the friction sound picked up by the sound pickup unit 112 in step S301 to a trained model corresponding to the type of paper 10 identified by the identification unit 113b in step S302, and obtains the output result (S303). More specifically, based on the type of paper 10 identified by the identification unit 113b, the estimation unit 114a selects a first trained model corresponding to the type of paper 10 from among a plurality of first trained models stored in the storage unit 120, and inputs the information regarding the friction sound to the selected first trained model. In other words, the estimation unit 114a switches the first trained model according to the type of paper 10 supplied from the holding unit 270 into the paper feed device 200.

[0129] Next, the estimation unit 114a estimates whether or not there are signs of a paper jam based on the output result obtained in step S303 (S304). If the estimation unit 114a estimates in step S304 that there are signs of a paper jam (Yes in S305), the output unit 116 outputs a signal to the paper feed device 200 to stop the supply of paper 10 from the holding unit 270 into the inside of the paper feed device 200 (S306). On the other hand, if the estimation unit 114a estimates in step S304 that there are no signs of a paper jam (No in S305), the paper jam prediction estimation device 100c returns to the process of step S301.

[0130] [3. Effects, etc.] As described above, in the paper jam prediction estimation device 100c according to the modified example 1 of Embodiment 2, the learned model includes a plurality of learned models (so-called first learned models) corresponding to each of a plurality of types of paper 10, and the paper jam prediction estimation device 100c further includes an identification unit 113b that identifies the type of paper 10 supplied from the holding unit 270 into the paper feeding device 200, and the estimation unit 114a inputs information regarding friction noise to the learned model (so-called first learned model) corresponding to the identified type of paper 10 based on the type of paper 10 identified by the identification unit 113b.

[0131] As a result, the paper jam prediction device 100c can switch the trained model used depending on the type of paper 10 supplied from the holding unit 270 into the paper feed device 200. Therefore, the paper jam prediction device 100c can accurately estimate whether or not there is a paper jam depending on the type of paper 10.

[0132] In the paper jam prediction device 100c according to the modified example 1 of Embodiment 2, the identification unit 113b may identify the type of paper 10 based on data acquired by at least one of an image sensor, an ultrasonic sensor, an optical sensor, a weight sensor, and machine learning.

[0133] As a result, the paper jam prediction device 100c can identify the type of paper 10 using at least one of the following: a database that associates data representing the characteristics of the paper 10 with the type of paper 10, and a trained model (also called a third trained model) that takes data representing the characteristics of the paper 10 as input and outputs the type of paper 10 supplied. Therefore, the paper jam prediction device 100c can identify the type of paper 10 with high accuracy. [Examples]

[0134] The paper jam prediction device and paper jam prediction method of this disclosure will be specifically described below in the examples provided. However, the following examples are just examples, and this disclosure is not limited in any way to these examples.

[0135] The following describes (1) the machine learning models used in Example 1 and Example 2, (2) the estimation accuracy for each type of paper when using one pre-trained model (the so-called first pre-trained model), and (3) the estimation accuracy for each type of paper when using a pre-trained model corresponding to each of the eight types of paper (the so-called first pre-trained model).

[0136] (1) Machine learning models used in Example 1 and Example 2 Figure 11 is a diagram illustrating the machine learning models used in Example 1 and Example 2. As shown in Figure 11, the machine learning models used in Example 1 and Example 2 are convolutional neural network (CNN) models. The machine learning model consists of an input layer, a classification layer including a convolutional layer (3x3), a ReLU (Normalized Linear Unit) layer, a pooling layer, a fully connected layer, and a Softmax layer, and an output layer.

[0137] (1-1) About the machine learning model used in Example 1 The machine learning model used in Example 1 was a single model, trained using the following training data. • Number of data points: 560 • Training data: A dataset containing both the first and second data sets. Data 1: Data consisting of a spectrogram image of the friction sound between sheets of paper and annotations indicating the occurrence of a paper jam. Second set of data: Data consisting of spectrogram images of the friction sound between sheets of paper and annotations indicating that no paper jams occurred.

[0138] The friction sounds between the papers were recorded when the following eight types of paper were fed into the paper feeder, and were captured within 30 msec from the start of paper feeding. The types of paper were: high-quality paper 1, high-quality paper 2, high-quality paper 3, high-quality paper (thin), gloss coated paper, pressure-sensitive paper base, tracing paper, and typewriter paper.

[0139] The training data used in Example 1 does not include information about the type of paper.

[0140] The output was either normal (indicating no paper jam) or abnormal (indicating a paper jam). Note that normal and abnormal may also be represented by binary values, 0 and 1.

[0141] (1-2) About the machine learning model used in Example 2 The machine learning models used in Example 2 consisted of eight models, and for each of the eight types of paper, training data corresponding to the paper type was selected from the above training data and trained individually.

[0142] (2) Estimation accuracy for each type of paper when using one pre-trained model (the so-called first pre-trained model) [Example 1] In Example 1, using the eight types of paper described above, the friction sounds between the papers as they were supplied from the holding unit were recorded. The spectrogram images of the friction sounds recorded within 30 msec from the start of paper supply were input into a pre-trained machine learning model trained under the conditions described in (1-1) above. This operation was performed 10 times for each of the eight types of paper. The supplied paper was stapled together, and the number of times it was estimated to be abnormal (i.e., the number of times it was correctly estimated) out of the 10 trials was counted. The results are shown in Figure 12. Figure 12 shows the results of Example 1.

[0143] The estimation accuracy (%) shown in Figure 12 represents the number of times an anomaly was estimated out of 10 trials. The eight types of paper used were selected from among the types of paper that the scanner could handle, differing in paper material, thickness, and surface roughness.

[0144] As shown in Figure 12, in Example 1, there was variation in estimation accuracy depending on the type of paper. This variation is thought to be due to the fact that some types of paper had a small amount of training data.

[0145] (3) Estimation accuracy for each of the eight types of paper 10 when using the pre-trained model corresponding to each of the eight types of paper 10 (the so-called first pre-trained model) [Example 2] In Example 2, the procedure was the same as in Example 1, except that four types of paper were used from the eight types of paper listed above: high-quality paper 1, high-quality paper (thin paper), tracing paper, and typewriter paper, and a pre-trained model corresponding to these four types of paper was used. The results are shown in Figure 13. Figure 13 shows the results of Example 2. Figure 14 shows a comparison of the estimation accuracy of Example 1 and Example 2 for the four types of paper used in Example 2.

[0146] As shown in Figure 13, the estimation accuracy was 70% or higher for all four types of paper.

[0147] Furthermore, as shown in Figure 14, it was confirmed that the estimation accuracy improved when four pre-trained models, each corresponding to one of the four types of paper, were switched and used for each type of paper.

[0148] (result) In Example 1, there was variability in estimation accuracy due to bias in the training data, but it was confirmed that the presence or absence of signs of a paper jam can be estimated by using a machine learning model.

[0149] Furthermore, the results from Examples 1 and 2 confirmed that by switching and using different machine learning models for each type of paper, it is possible to accurately estimate the presence or absence of signs of a paper jam regardless of the type of paper.

[0150] (Other embodiments) The above description has explained a paper jam prediction device and a paper jam prediction method relating to one or more embodiments of the present disclosure, based on the above embodiments. However, the present disclosure is not limited to these embodiments. Without departing from the spirit of the present disclosure, various modifications to the embodiments that a person skilled in the art could conceive, or forms that combine components from different embodiments, may also be included within the scope of one or more embodiments of the present disclosure.

[0151] For example, some or all of the components of the paper jam prediction device according to the above embodiment may be composed of a single system LSI (Large Scale Integration). For example, the paper jam prediction device may consist of a system LSI having a sound collection unit, an estimation unit, and an output unit. Note that the system LSI does not necessarily include a microphone.

[0152] A system LSI is a highly functional LSI manufactured by integrating multiple components onto a single chip. Specifically, it is a computer system consisting of a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), and other components. The ROM stores the computer program. The system LSI achieves its function by operating according to the computer program, with the microprocessor working accordingly.

[0153] Here, we refer to it as a system LSI, but depending on the degree of integration, it may also be called an IC, LSI, super LSI, or ultra LSI. Furthermore, the method of integrated circuit implementation is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. After LSI manufacturing, an FPGA (Field Programmable Gate Array) that can be programmed, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of the circuit cells inside the LSI, may also be used.

[0154] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possible possibility.

[0155] Furthermore, one aspect of this disclosure may be not only a paper jam prediction device, but also a paper jam prediction method in which characteristic components included in the device are used as steps. Also, one aspect of this disclosure may be a computer program that causes a computer to execute each characteristic step included in the paper jam prediction method. Also, one aspect of this disclosure may be a computer-readable non-temporary recording medium on which such a computer program is recorded. [Industrial applicability]

[0156] According to this disclosure, information regarding friction sounds when supplying paper can be input into a trained model, and based on the output results obtained, it is possible to easily estimate signs of paper jams, such as paper floating. The paper jam prediction device and paper jam prediction method of this disclosure are applicable to devices that supply paper to various paper processing devices, and can therefore be applied to various fields such as household, industrial, and research use. [Explanation of Symbols]

[0157] 10 Paper 15 staplers 20 stacks of paper 30. Locations where paper lifting occurs. 100, 100a, 100b, 100c Paper jam prediction device 110, 110a, 110b, 110c Information Processing Unit 112, 112a Sound collection section 113a, 113b identification part 114, 114a Estimation part 116 Output section 120 Storage section 130 Communications Department 140, 140a, 140b Learning Section 150 Sensor section 200 Paper feeder 210 Conveying section 212, 212a, 212b Paper feed roller 214, 214a, 214b Separation rollers 216, 216a, 216b Retard Rollers 220 Drive unit 230 Control Unit 240 Storage section 250 Communications Department 260 supply ports 270 Holding part 300 microphones

Claims

1. A paper jam prediction device for estimating signs of a paper jam in a paper feed device, A sound-collecting unit that collects the friction sound between sheets of paper generated near the separation roller that separates the paper in the paper feeder when paper is supplied into the paper feeder from a holding unit that holds multiple sheets of paper, An estimation unit that estimates whether or not paper floats occur in the paper feed device based on the output result obtained by inputting the information regarding the friction sound into a trained model, which is a trained machine learning model, If the estimation unit estimates that the paper has become airborne, the output unit outputs a signal to the paper feeder to stop the paper from being supplied into the paper feeder, Equipped with, Paper jam prediction device.

2. The information regarding the fricative sound input to the trained model is either a spectrogram image or a frequency characteristics image of the fricative sound. The paper jam prediction device according to claim 1.

3. The aforementioned friction sound is an inaudible sound caused by friction between the paper supplied from the holding part and the paper held in the holding part. The paper jam prediction device according to claim 1 or 2.

4. The aforementioned inaudible sound is a sound with a frequency in the ultrasonic range. The paper jam prediction device according to claim 3.

5. The training data used to train the aforementioned machine learning model is: The first data consists of information regarding the friction sound and annotations indicating the occurrence of paper floating, The second data consists of the aforementioned friction sound information and annotations indicating that no paper lifting occurred, including, A paper jam prediction device according to any one of claims 1 to 4.

6. The trained model includes multiple trained models corresponding to each of several types of paper, The aforementioned paper jam prediction device further includes: The paper feeder includes an identification unit that identifies the type of paper supplied from the holding unit to the inside of the paper feeder, The estimation unit inputs information regarding the friction sound based on the type of paper identified by the identification unit into the trained model corresponding to the identified type of paper. A paper jam prediction device according to any one of claims 1 to 5.

7. The friction sound is a sound with a frequency in the ultrasonic band, The identification unit identifies the type of paper supplied from the holding unit to the inside of the paper feed device based on the friction sound. The paper jam prediction device according to claim 6.

8. The identification unit identifies the type of paper based on data acquired by at least one of an image sensor, an ultrasonic sensor, an optical sensor, a weight sensor, and machine learning. The paper jam prediction device according to claim 6 or 7.

9. The aforementioned machine learning model is a convolutional neural network model. A paper jam prediction device according to any one of claims 1 to 8.

10. A method for estimating signs of a paper jam in a paper feed device, A sound collection step is performed to collect the friction sound between sheets of paper that occurs near the separation roller that separates the paper in the paper feeder when paper is supplied into the paper feeder from a holding section that holds multiple sheets of paper, An estimation step in which the presence or absence of paper floating in the paper feed device is estimated based on the output result obtained by inputting the information regarding the friction sound into a trained model, which is a trained machine learning model, An output step in which, when it is estimated that the aforementioned paper floating has occurred, a signal is output to the paper feed device to stop the supply of the paper into the paper feed device, including, Method for predicting paper jams.

11. To cause a computer to execute the paper jam prediction method described in claim 10, program.

Images