Sewing machine and using method thereof

Abstract

A method for calibrating one or more optical sensors on a sewing machine includes acquiring data of one or more features of one or more predefined areas associated with the sewing machine, processing the data via one or more neural networks, and calibrating one or more optical sensors on the sewing machine. Wherein the one or more neural networks detect and identify one or more features of the one or more predefined regions from the data, calculate one or more accuracy indicators of the one or more features from the data compared to the one or more training features from the one or more neural networks, the values of the one or more accuracy indicators are compared to one or more indicator thresholds, and one or more parameters of the one or more optical sensors are adjusted based on the comparison between the one or more accuracy indicators and the one or more indicator thresholds.

Description

[0001] This application is a divisional application of application number 202280074919.8, filed on November 10, 2022, entitled “SEWING MACHINE AND METHOD OF USING THE SAME” to the State Intellectual Property Office, with the filing date of November 10, 2022.

[0002] Cross Reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 278,286, filed on November 11, 2021, entitled “SEWING MACHINE AND METHOD OF USING THE SAME” (Attorney Docket No. 31982.04247), the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present invention relates generally to sewing machines, and in particular to control systems thereof. BACKGROUND

[0004] Sewing machines can be used to form stitches on a single piece of material and to join pieces of material together. Particular sewing machines can be used to form stitches in a workpiece having a particular shape, to cut and sew on the edges of a workpiece, to attach a decorative element to a workpiece, and to cut and hem the edges of a workpiece, to attach a decorative, sewn embroidery pattern on a workpiece mounted in an embroidery frame, or to cut a workpiece during a sewing operation. In addition to or separate from the sewing operation, a sewing machine can also cut, fold, roll, or otherwise manipulate a workpiece. The workpiece is moved under a needle in order to form stitches in the fabric. A user configures the sewing machine for each particular application by adjusting various parameters of the sewing machine and attaching various different tools or accessories to the sewing machine. SUMMARY

[0005] Exemplary embodiments of a sewing machine, a control system thereof, and a method of using the same are disclosed herein.

[0006] An example sewing machine includes a sewing head attached to an arm suspended above a sewing table by a post, a needle bar extending from the sewing head toward the sewing table, a needle secured by the needle bar, a motor connected to the needle bar for moving the needle bar in a reciprocating motion during a sewing operation to pass the needle and thread through a workpiece, and a user interface for receiving instructions from a user of the sewing machine and providing feedback information to the user. The example sewing machine also includes a data acquisition device, a data storage device, and a processor. The data acquisition device is for acquiring data related to at least one of the sewing machine, an environment surrounding the sewing machine, a sewing material, a sewing operation performed by the sewing machine, and one or more interactions of the user with the sewing machine. The data storage device is for storing the data acquired by the data acquisition device as acquired data and for storing data related to a neural network. The neural network is composed of a plurality of nodes. Each node of the neural network has an input connection for receiving input data, node parameters, a computation unit for calculating an activation function based on the input data and the node parameters, and an output connection for transmitting output data. The processor is configured to process the acquired data through the neural network to generate processed data and to control at least one of the user interface for interacting with the user, the data storage device for storing the processed data, and the motor for changing the sewing operation based on the processed data.

[0007] An example method of controlling a sewing machine includes the steps of acquiring data, storing the acquired data in a data storage device, processing the acquired data through a neural network, and controlling a user interface, the data storage device, and a motor based on the processed data. The step of acquiring data includes acquiring data related to at least one of the sewing machine, an environment surrounding the sewing machine, a sewing material, a sewing operation performed by the sewing machine, and one or more interactions of the user with the sewing machine. The neural network in the step of processing has a plurality of nodes, wherein each node includes an input connection for receiving input data, node parameters, a computation unit for calculating an activation function based on the input data and the node parameters, and an output connection for transmitting output data. During the step of controlling, the processor controls the user interface to interact with the user, the data storage device to store the processed data, and / or the motor to change the sewing operation.

[0008] An exemplary control system for a sewing machine includes a data acquisition device, a data storage device, and a processor. The data acquisition device is to acquire data related to at least one of the sewing machine, an environment surrounding the sewing machine, a sewing material, a sewing operation performed by the sewing machine, and one or more interactions of a user with the sewing machine. The data storage device is to store the data acquired by the data acquisition device as acquired data, and to store data related to a neural network. The neural network is comprised of a plurality of nodes. Each node of the neural network has an input connection to receive input data, a node parameter, a computation unit to compute an activation function based on the input data and the node parameter, and an output connection to transmit output data. The processor is configured to process the acquired data through the neural network to generate processed data, store the processed data in the data storage device, and control at least one of a user interface to interact with the user and a motor to change the sewing operation based on the processed data.

[0009] An exemplary method for calibrating one or more optical sensors on a sewing machine includes acquiring data of one or more features of one or more predefined areas related to the sewing machine, processing the data through one or more neural networks, wherein the one or more neural networks detect and identify the one or more features of the one or more predefined areas from the data, computing one or more accuracy indicators of the one or more features from the data compared to one or more training features from the one or more neural networks, comparing values of the one or more accuracy indicators to one or more indicator thresholds, and adjusting one or more parameters of the one or more optical sensors based on the comparison between the one or more accuracy indicators and the one or more indicator thresholds.

[0010] An exemplary sewing machine has a sewing head attached to an arm suspended above a sewing table by a column, a needle bar extending from the sewing head toward the sewing table, wherein the needle bar holds a needle, a presser bar having a presser foot extending from the sewing head toward the sewing table, one or more optical sensors arranged to acquire data from one or more features of one or more predefined areas related to the sewing machine, and one or more processors to process the data acquired by the one or more optical sensors through one or more neural networks. The one or more processors are configured to receive the data from the one or more optical sensors, process the data through the one or more neural networks, wherein the one or more neural networks detect and identify the one or more features of the one or more predefined areas from the data, compute one or more accuracy indicators of the one or more features from the data compared to training features from the one or more neural networks, compare values of the one or more accuracy indicators to one or more indicator thresholds, and adjust one or more parameters of the one or more optical sensors based on the comparison between the one or more accuracy indicators and the one or more indicator thresholds.

[0011] An example sewing machine having a sewing head attached to an arm suspended above a sewing table by a post, a needle bar extending from the sewing head toward the sewing table, wherein the needle bar holds a needle, a presser bar having a presser foot extending from the sewing head toward the sewing table, one or more data acquisition devices associated with the sewing machine arranged to acquire data from one or more features of one or more predefined areas associated with the sewing machine, and one or more processors for processing data acquired by the one or more data acquisition devices by one or more neural networks. The one or more processors are configured to receive data from the one or more data acquisition devices, process the data by the one or more neural networks, wherein the one or more neural networks detect and identify one or more features of the one or more predefined areas from the data, compute one or more accuracy indicators of the one or more features from the data compared to trained features from the one or more neural networks, compare values of the one or more accuracy indicators to one or more indicator thresholds, and adjust one or more parameters of the one or more data acquisition devices based on the comparison between the one or more accuracy indicators and the one or more indicator thresholds.

[0012] In the following description and claims, particularly when considered in conjunction with the accompanying drawings, further understanding of the nature and advantages of the present disclosure will be set forth. As such, it is to be understood that specific details need not be used to practice the present disclosure and that the present disclosure can be implemented in a variety of ways. BRIEF DESCRIPTION OF DRAWINGS

[0013] To further clarify various aspects of embodiments of the present disclosure, certain embodiments will be described in more detail below with reference made to the figures. It will be appreciated that these figures depict only typical embodiments of the present disclosure and are therefore not to be considered limiting of its scope. Further, while the figures can generally be drawn to scale, the figures are not necessarily to scale, as the emphasis generally lies in the relative positioning of the components. The embodiments of the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which: Figures 1-18 Various views and diagrams related to example sewing machines and systems thereof are shown; Figures 19-23B Schematic diagrams and flowcharts related to artificial intelligence and neural networks are shown; Figures 24-33 Various views related to stitch regulation of example sewing machines are shown; Figures 34-39 Schematic diagrams of various machine vision techniques are shown; Figures 40-42 Various views related to example sewing projection features of example sewing machines are shown; Figures 43-65Various views related to an example fabric and thread compatibility monitoring function of an example sewing machine are shown; Figures 66-70 Various views related to an example thread quality monitoring feature of an example sewing machine are shown; Figures 71-81 Various views related to an example object recognition feature of an example sewing machine are shown; Figures 82-84 Various views related to an example haptic feedback feature of an example sewing machine are shown; and Figures 85-88 Various views related to an example machine diagnostic feature of an example sewing machine are shown; Figure 89 A cross-sectional view of an example thread sensor is shown; Figure 90 A perspective view of an example sewing machine is shown; Figure 91 A front view of an example sewing machine is shown; Figure 90 A front view of an example sewing machine is shown; Figure 92 A front view of an example sewing machine is shown; Figure 90 A front view of an example sewing machine is shown; Figure 93 A front view of an example sewing machine is shown; Figure 90 A front view of an example sewing machine is shown; Figure 94 A front view of an example sewing machine is shown; Figure 92 A detailed view of an example area 92 is shown; Figure 95 A detailed view of an example area 93 is shown; Figure 93 A detailed view of an example area 93 is shown; Figure 96 An example process for controlling a sewing machine is shown; and Figures 97-109 A flowchart detailing an example sewing machine operation is shown. DETAILED DESCRIPTION

[0014] The following description refers to the accompanying drawings, which illustrate specific embodiments of the present disclosure. Other embodiments of the present disclosure are possible and contemplated without departing from the scope or spirit of the present disclosure. Exemplary embodiments of the present disclosure relate to sewing machines and accessories used therewith.

[0015] As described herein, when one or more components are described as connected, joined, affixed, coupled, attached, or otherwise interconnected, such interconnection can be direct as between the components or indirect, such as through the use of one or more intermediate components. Also as described herein, reference to a “member,” “component,” or “portion” shall not be limited to a single structural member, component, or element, but can include a collection of members, components, or elements. Also as described herein, the terms “substantially” and “approximately” are defined as at least close or nearby (and including) a given value or state, preferably within 10% of the given value or state, more preferably within 1% of the given value or state, and most preferably within 0.1% of the given value or state.

[0016] Reference is now made to Figures 1-18 and Figures 90-95 various views and diagrams of example sewing machines and portions thereof. An example sewing machine, such as the sewing machine 100 shown in Figure 1 includes a sewing table or bed 104 having a post 106 extending upwardly from one end to support an arm extending horizontally above the sewing table. A sewing head 102 is attached to an end of the arm and can include one or more needle bars 108 for moving one or more needles 110 up and down to sew a workpiece on the sewing table 104 below the sewing head 102. The sewing table includes a needle plate or presser foot disposed below the sewing head having openings for the one or more needles 110 to pass through when making or forming stitches in the workpiece. In some sewing machines, a bobbin disposed below the needle plate assists in forming the stitches and dispenses a lower thread that is sewn together with an upper thread delivered by the needle from above. In other sewing machines, such as a serger or overlocker, the lower thread is dispensed by a loopers. A user can interact with the sewing machine 100 through various buttons, knobs, switches, and other user interface elements. A touchscreen display 112 can also be used to present a software-based user interface to the user and receive inputs from the user. A projector 114 disposed in the sewing head 102 can be used to project one or more user interface elements onto the sewing table 104 or a workpiece placed thereon. One or more cameras 116 disposed in the sewing head 102 or elsewhere around the sewing machine 100 gather information from the workpiece and the surrounding environment of the sewing machine 100 that can be used to enhance the performance and user experience of the sewing machine 100.

[0017] As used herein, a “sewing machine” refers to a device that forms one or more stitches in a workpiece with a reciprocating needle and a length of thread. As used herein, a “sewing machine” includes, but is not limited to, a sewing machine configured to form a particular stitch (e.g., a sewing machine configured to form a lock stitch, a chain stitch, a buttonhole stitch), an embroidery machine, a quilting machine, an overlock machine, or a serger, among others. It should be noted that various embodiments of sewing machines and accessories are disclosed herein, and these options can be combined in any combination, unless specifically excluded. In other words, various components or portions of the disclosed devices can be combined, unless mutually exclusive or physically impossible.

[0018] A “stitch” refers to a loop of thread formed by one or more threads, where at least one thread passes through a hole formed in a workpiece. The mechanical components of a sewing machine, such as a needle, a hook, a looper, a thread tensioning device, a feed mechanism, and the like, cooperate to form a stitch in one or more workpieces. One repetition of this complex mechanical dance can form one stitch or a stitch pattern in a workpiece. The “stitch length” of a repetition or pattern refers to the distance the workpiece moves while the repetition is occurring. Stitch length measurements are different for different types of repetitions and patterns, and can include one or more stitches in a workpiece.

[0019] A presser bar with a presser foot also extends downward from the sewing head to press the workpiece against the sewing table and against the feed dog that moves the workpiece from back to front and optionally side to side. The feed dog cooperates with the presser foot and moves the workpiece at a speed that can be fixed or variably controlled by the user (e.g., with a foot pedal). A wide variety of presser feet and other types of accessories can be attached to the presser bar to assist in forming certain types of stitches or features in the workpiece, for example, a buttonhole presser foot. An accessory holder can also extend below the sewing head for holding specialized tools or accessories above or on the sewing table.

[0020] As described above, the speed or frequency of the needle bar moving up and down is controlled by the user. While the needle bar is typically moved up and down in a cyclical motion to form stitches in a workpiece, the needle bar can also be moved side to side simultaneously to form different stitches, such as Z-stitches or tapered stitches, or to vary the width of a stitch. The user can select the type and spacing of stitches performed by the sewing machine through a manual interface including buttons, knobs, joysticks, and the like, through a user interface presented on a touchscreen by a computer, or through a voice-controlled interface.

[0021] Different types of sewing machines can include additional components for forming stitches in a workpiece or otherwise manipulating the workpiece during the sewing process. For example, in a serger (a type of sewing machine that can be used to form edges of a workpiece), among other functions, a needle known as a loopers runs below the sewing table to deliver lower thread for forming various stitches. A serger can also include two, three, or more needles above the needle plate and a knife for cutting the edge of the workpiece. A sewing machine can also create embroidery patterns in a workpiece by including a holder for an embroidery hoop on the sewing table (e.g., Figure 6 ). The embroidery hoop holder can be actuated on at least two axes so that the controller of the sewing machine can move the embroidery frame to cause the needle to trace out an embroidery pattern on the workpiece.

[0022] Thread used in the sewing process is secured at different locations on the sewing machine, for example, inside a bobbin ( Figures 13-15 ) or on a spool held by a spool holder that is part of or extends above the sewing machine arm ( Figures 10-12 ). Thread is drawn from the thread source (e.g., bobbin or spool) and through various other elements of the sewing machine that are arranged to change the direction of the thread so that the thread is smoothly drawn out and delivered to the workpiece with as little damage to the thread as possible ( Figures 7-9 ). The tension of the thread can also be changed by various tensioning devices arranged along the thread path or within the thread source. Thread tensioning and dispensing devices ensure that only the desired amount of thread is dispensed and that the thread forming the stitches in the workpiece is properly taut. Loose thread can cause stitches to come undone and tight thread can cause stitches to be improperly formed. The tension of the thread on the upper and lower thread can also be adjusted to ensure that the tension is balanced from top to bottom so that stitches are properly formed along the desired sewing path in the workpiece.

[0023] Reference is now made to Figure 16FIG. 1 shows a block diagram of a computer-based control system for a sewing machine 100. The sewing machine includes one or more data acquisition devices 118, one or more data storage devices 120, a processor 122, a user interface 124, and motors and actuators 126. The sewing machine 100 can also include a network interface connected to the processor 122 for connecting the sewing machine 100 to a cloud system and / or other sewing machines or devices via a wireless network. The data acquisition devices 118 include a wide variety of digital sensors, analog sensors, active sensors, passive sensors, and software components, as described in more detail below. These sensors acquire data related to the sewing machine itself, the workspace or environment surrounding the sewing machine, the sewing operations performed by the sewing machine, user interaction with the sewing machine, and the sewing materials (e.g., fabric and thread) operated by the sewing machine. The data storage devices 120 include one or more computer storage chips for storing data acquired by the data acquisition devices 118 and the operating software of the sewing machine 100. The structure, various functions, and parameters of one or more neural networks 128 are also stored by the data storage devices 120. The processor 122 accesses the data stored on the data storage devices 120 and executes the operating software to impart functionality to the sewing machine 100. The user interface 124 is presented to the user through a touchscreen display 112 and physical controls (e.g., buttons, joysticks, dials, lights, speakers, actuators, etc.). The motors and actuators 126 include electromechanical actuators, motors, general mechanical components controlled by the control system to cause motion of various moving parts of the sewing machine (i.e., needle bar, feed dog, bobbin, curved needle, etc.). For example, the speed of a motor can be controlled directly through input from a foot pedal actuated by a user, or can be controlled through a computer that receives and interprets the foot pedal input before transmitting a signal to one or more motor controllers that control the motors of the sewing machine.

[0024] As used herein, a “computer” or “processor” includes, but is not limited to, any programmable or programmable electronic device or coordination device that can store, retrieve, and process data, and can be a processing unit or a distributed processing configuration. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), floating point units (FPUs), reduced instruction set computing (RISC) processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), etc. One or more cores of a single microprocessor and / or multiple microprocessors each having one or more cores can be used to perform the operations described herein as being performed by a processor. The processor can also be a processor dedicated to training neural networks and other artificial intelligence (AI) systems. The one or more processors can be installed locally on the sewing machine and can be located remotely that can be accessed through a network interface.

[0025] As used herein, a "network interface" or "data interface" includes, but is not limited to, any interface or protocol used to transmit and receive data between electronic devices. A network or data interface can refer to a connection to a computer over a local network or over the Internet, as well as a connection to a portable device (e.g., a mobile device or a USB thumb drive) over a wired or wireless connection. A network interface can be used to form a network of computers to facilitate distributed and / or remote computing (i.e., cloud-based computing). "Cloud-based computing" refers to computing that is implemented on a network of computing devices that are remotely connected to the sewing machine over a network interface.

[0026] As used herein, "logic" is synonymous with "circuitry" and includes, but is not limited to, hardware, firmware, software and / or combinations of each to perform one or more functions or actions. For example, logic can include a software-controlled processor, discrete logic such as an application specific integrated circuit (ASIC), programmed logic devices or other processors, based on the desired application or need. Logic can also be embodied entirely in software. As used herein, "software" includes, but is not limited to, one or more computer readable and / or executable instructions that cause a processor or other electronic device to perform functions, actions, processes and / or behave in a desired manner. The instructions can be embodied in various forms such as routines, algorithms, modules or programs including separate applications or code from dynamically linked libraries (DLLs). Software can also be implemented in various forms such as individual programs, web-based programs, function calls, subroutines, microcode, applications, application programs, applets, servlets, plugins, instructions stored in memory, parts of an operating system or other types of executable instructions or interpreted instructions created from other types of executable or interpreted languages.

[0027] As used herein, a "data storage device" refers to one or more devices for non-transitory storage of code or data, such as a device with a non-transitory computer readable medium. As used herein, a "non-transitory computer readable medium" refers to any suitable non-transitory computer readable medium for storing code or data, such as magnetic media, such as a fixed disk within a external hard drive, a fixed disk within an internal hard drive, and a floppy disk; optical media, such as a CD-ROM, a DVD, and other media, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory PROM, an external flash drive, and the like.

[0028] The user interface of the sewing machine can include a variety of input devices and ways of communicating with the user, such as buttons, knobs, switches, lights, displays, speakers, touch interfaces, and lights. The user interface of the sewing machine can be presented to the user graphically through one or more displays, including a touchscreen display 112 that includes a touch-sensitive overlay for detecting the position of a user’s finger touching the display. Thus, the user can interact with the user interface by directly touching the screen at a particular location with their hand 150 and performing touch gestures (such as Figure 2 touch 152, touch 152 and hold 154, touch 152 and pinch or expand 156, and touch 152 and move 158 gestures) shown. The presence, position, and movement of the user’s hand, fingers, or eyes can also be detected through perturbations in sound, light, infrared radiation, or electromagnetic fields through analysis of data from optical sensors (e.g., cameras) or proximity sensors (e.g., Figure 3 The graphical user interface can also be projected by one or more projectors of the sewing machine onto the sewing table 104, the workpiece, an adjacent surface such as a wall or table, or any other suitable surface. Alternatively, the sewing machine 100 can operate without a graphical user interface through voice commands and auditory feedback in the form of certain sounds and / or computerized speech. Haptic feedback can also be provided through actuators that vibrate various parts of the machine (e.g., feedback portion 160) when acting on or in response to various conditions of the workpiece, machine, etc. Auditory and haptic interaction with the sewing machine is particularly useful for users with impaired vision.

[0029] As shown in Figure 17 , the sewing machine 100 can provide notifications and feedback to the user through visual, auditory, and haptic means. For example, an indication that an incorrect accessory is installed on the machine can be displayed to the user through the user interface on the sewing machine display, while a notification sound, such as a beep or computerized voice, is sent to the user through a speaker in the sewing machine. The notification can also be sent to the user via haptic feedback through a vibration of the feedback portion 160 of the sewing machine 100 that the user touches. That is, the sewing machine can vibrate the sewing table 104 to provide a warning to the user that the machine is not properly configured for a particular sewing operation selected by the user. The user will feel the vibration under their fingers in contact with the workpiece and sewing table, prompting the user to look at the display for more information. The sewing machine’s illumination lights can also be controlled to alert the user, such as when an incorrect accessory is installed, by changing the color of the flashing, prompting the user to look at the display for additional information.

[0030] As shown in Figure 1 , Figure 4 , Figure 92 , and Figure 94As shown, a projector 114 can also be provided in the sewing head 102 and directed downward toward the sewing table 104 and the workpiece. The projector 114 is arranged to project useful information onto the workpiece to assist the user in using the sewing machine. For example, the projector 114 can project the drop point of the needle down onto the fabric so that the user can see where the needle is before making a stitch. Thread or other guides can also be projected onto the workpiece to assist the user in sewing in a straight line or along a desired path. Similar to the guides, the projector 114 can project a selected stitch pattern onto the workpiece to represent the planned stitches. The projector 114 can also be able to project images onto the workpiece to display a selected embroidery pattern on the workpiece so that the user can position the embroidery pattern at a desired location on the workpiece. The information projected by the projector 114 can also include feedback to the user regarding the status of the machine or a particular sewing operation. For example, the projector 114 can project a warning notice onto the workpiece that an incorrect needle has been installed for the type of material being used as the workpiece. The projector 114 can also provide visual instructions to assist the user, such as static or animated images instructing the user how to change a needle, thread the machine, or rotate the workpiece. In other words, the projector 114 can be used by the computer as another means of providing feedback and instructions to the user. It should also be noted that the images projected by the projector 114 can also be detected by the optical sensors so that the user's interaction with these images, such as by touching a projected image of a button or series of buttons, causes the sewing machine to respond to the user's interaction with the projected images.

[0031] The sewing machine 100 can use a variety of data acquisition devices 118, namely digital sensors, analog sensors, active sensors, passive sensors, and software components, all of which can be used by the sewing machine 100 to acquire data related to the sewing machine itself, the workspace or environment around the sewing machine, the sewing materials operated by the sewing machine (e.g., fabric workpieces and threads used to form stitches), the sewing operations performed by the sewing machine, and the user's interaction with the sewing machine. A non-exhaustive list of sensor types for the sewing machine 100 includes: acoustic, sound, vibration, chemical, biometric, sweat, respiration, fatigue detection, gas, smoke, retina, fingerprint, fluid velocity, speed, temperature, optical (e.g., camera), light, infrared, ambient light level, color, red-green-blue (RGB) color (or other color space sensors, such as sensors using four-color printing (CMYK) or grayscale color space), touch, tilt, motion, metal detector, magnetic field, humidity, moisture, imaging, photons, pressure, force, density, proximity, ultrasound, load cell, digital accelerometer, motion, translation, friction, compressibility, sound, microphone, voltage, current, impedance, barometer, gyroscope, Hall effect, magnetometer, GPS, resistance, tension, strain, etc. The software-based data acquisition device 118 may include various data logs entered during the use of the sewing machine 100. For example, the user activity log may record events involving input from the user via the user interface 124, and the system event log may record software events that occur during normal use of the sewing machine 100 and can be used for machine learning or diagnostic purposes.

[0032] The sensors can be placed in various locations on the machine and can be used by the sewing machine in various ways. For example, the sewing machine may include touch and proximity sensors 170 (e.g. Figure 3 The proximity sensor 170 shown is used to provide touch control of the user interface displayed on the sewing machine's monitor. Similar touch or proximity sensors may also be located in other locations on the sewing machine, such as the arm or sewing table. These other touch sensors may be used in conjunction with the user interface presented to the user, or for safety purposes during sewing, to monitor the position of the user's hand (or other foreign objects, such as the user's hair or a loose sewing needle) on the machine. The sewing machine may also include eye-tracking sensors, including optical sensors, such as cameras or other detection means, for tracking the user's eye position and / or line of sight. One or more optical sensors on the machine may be used not only to collect user-related data about the machine, but also to collect data related to the workpiece, other sewing materials (such as thread), and the sewing machine itself. Additional examples of the use of sensors and sensor data through sewing are provided throughout this disclosure.

[0033] Many of the sensors used in the sewing machine need to be calibrated after installation to ensure that the data collected by the sensors and provided to the neural network is accurate. Calibration of the sensors can also be updated periodically in the field or at the user’s discretion. The sensors can be calibrated in any suitable manner. For example, calibration of the cameras can be performed using the techniques described in U.S. Patent No. 8,606,390, which is incorporated by reference herein. The cameras and other sensors can also be calibrated using techniques that use neural networks, such as recognizing features of the sewing machine when calibrating the cameras.

[0034] One or more optical sensors of the sewing machine can be arranged in various locations around the sewing machine. As used herein, “optical sensor” refers to a sensor that is capable of collecting data from electromagnetic radiation (see Figure 18 ), which can include, but is not limited to, sensors for detecting ultraviolet, visible, infrared radiation, etc. Certain optical sensors can be tuned to a particular wavelength of electromagnetic radiation, such as a particular wavelength of laser light. One particular optical sensor that can be used in an example sewing machine is a camera. The camera can include a lens for focusing or otherwise redirecting light onto a sensor that receives the optical data and transmits the optical data to another device for processing.

[0035] One or more optical sensors can be arranged in the sewing machine to view the workpiece during the sewing process, such as the camera 116 shown in Figure 1 , 4 , 6, 93, and 95. The one or more optical sensors that view the workpiece can be used to determine the color of the workpiece, the material of the workpiece, the position of the workpiece, the orientation of the workpiece, the amplitude and direction of motion of the workpiece, etc. The same optical sensors can also be used to detect objects in the sewing area, such as a user’s hand, a hazardous object (e.g., the user’s hair, clothing, a sewing needle, etc.), the type of needle or needles installed in the sewing machine, the type of presser foot, etc. The optical sensors can also monitor whether the needle, presser foot, needle plate, or accessory are properly installed and remain properly installed during use. Additional optical sensors or similar sensing devices can be arranged on the machine facing the user to provide information to the computer of the sewing machine about the user, such as the position and line of sight of the user’s eyes, so that the sewing machine can determine where the user is looking on the machine. For example, tracking the user’s eyes and current line of sight can allow the sewing machine to determine the best location to illuminate the sewing table or provide useful information to the user so that important notifications or warnings are not missed.

[0036] Various security features can be included in a sewing machine to limit access and prevent the sewing machine from being stolen. For example, when the machine is powered on or awakens from a sleep mode, the user can be presented with a prompt that requires the user to prove their identity. The user can then enter a predetermined code to prove that they are a user with access to and use of the sewing machine. In addition to or in place of a predetermined code, the user can provide biometric information, such as through a fingerprint sensor or facial recognition, as proof of identity. The fingerprint sensor can be included on the sewing table or another location where the user typically places their hand to use the machine. One or more cameras facing the user enable the sewing machine to use facial recognition technology to identify the user to provide access to the machine.

[0037] The user can also associate another device with their account on the sewing machine and use the device to unlock the sewing machine. For example, an application on a smartphone or tablet can be associated with the user’s account so that the sewing machine can be unlocked through the application or by holding the smartphone or tablet within a predetermined range of the sewing machine. Any of these methods of authenticating a user can be used individually or together to provide two-factor authentication. A phone number that can receive text messages can also be associated with the user’s account so that a code can be sent for two-factor authentication. These other devices or phones can also receive alerts from the sewing machine when other attempts to access the machine have failed, such as after a predetermined number of attempts to access the sewing machine. If the sewing machine is believed to have been stolen, these other devices can be used to determine the location of the sewing machine through a GPS sensor in the sewing machine or through other means, such as a local network detected by the sewing machine. Further, the alert generated due to the machine being moved from its normal location or failed attempts to enter the machine can include the location of the sewing machine determined by an onboard GPS sensor of the sewing machine so that the sewing machine can be recovered in relevant circumstances.

[0038] To process and act on the various data provided by the above-described sensors to one or more computers located inside and / or outside the sewing machine, various artificial intelligence (“AI”) tools and techniques are used (see, e.g., Figure 19 ), enabling the analysis of very large structured or unstructured and changing data sets, deductive or inductive reasoning, complex problem solving, and computer learning based on historical patterns, expert input, and feedback loops. “Artificial intelligence” as used herein refers to the broad field of tools and techniques in the field of computer science that enable computers to learn and improve over time. Figure 19 A non-exhaustive overview of these tools is shown, such as symbolic artificial intelligence, machine learning, and evolutionary algorithms. As Figure 19As shown, artificial neural networks can be used in various machine learning applications and employ a variety of learning methods, including but not limited to statistical learning, deep learning, supervised learning, unsupervised learning, and reinforcement learning. Artificial intelligence enables sewing machines to adapt to situations that software programmers did not anticipate or accurately predict, and facilitates complex yet intuitive ways of interacting with the sewing machine to achieve desired results. That is, the AI ​​tools and techniques described herein are used by one or more computers integrated inside or outside the sewing machine to make decisions that support or benefit the user based on data provided to the computer via the sensors described herein. While specific artificial intelligence tools (such as neural networks) may be described below, other artificial intelligence tools can be used for the same tasks; therefore, unless otherwise stated herein, the description of a tool or technique should not be construed as limiting its application to that tool or technique.

[0039] The related neural network diagrams and processes are as follows: Figures 20-2 As shown in Figure 3. The "neural network" used in this paper includes, but is not limited to, multiple interconnected software nodes or neurons arranged in multiple layers, such as... Figure 20 The input layer, hidden layer, and output layer are shown. Figure 20 A schematic diagram of a neural network 128 is shown, which includes nodes 130 arranged in various layers. Like neurons in the human brain, each node 130 may have one or more input connections 132 and output connections 138 to establish many-to-many relationships with other nodes 130 in the network. That is, the output of a single node can be connected to the inputs of multiple different nodes, and a single node can receive the outputs of multiple different nodes as input.

[0040] Each node 130 in the network is configured to compute data from other nodes and, in conjunction with node parameters adjusted during neural network training, compute output data. Figure 21). That is, each node of the network is a computational unit having one or more input connections for receiving input data from nodes in a previous layer of the network, and one or more output connections for transmitting output data to nodes in a subsequent layer or next layer of the network. Each node 130 includes a computational unit 136 for computing a result of an activation function that can incorporate input data received via the input connections, input parameters associated with each input connection, and optional function parameters 134, to compute output data that can be further modified by output parameters. For example, input data from each input connection can be modified by an associated input parameter, such as a weight parameter for that input connection, to provide a relative weight for the input connection. The result of the activation function, which can be modified by optional function parameters, is transmitted as output data through the output connections to nodes in a subsequent layer of the neural network. The optional function parameters can be, for example, a threshold value such that the computed result of the activation function is transmitted as output data to other nodes only when the combined weighted input data exceeds a threshold set by the threshold value.

[0041] All forms of data available to the sewing machine, i.e., data from sensors, software, data storage devices, user input through software, etc., can be processed through the neural network. The information to be processed first encounters the input layer, which performs initial processing of the input data and outputs the results to one or more hidden layers to process the output values from the input layer. The information processed through the hidden layers is presented at the output layer as a confidence probability of a given result, e.g., the location of a detected object in an image and the classification of that object. Software in the computer of the sewing machine receives information from one of the multiple layers of the neural network and can take action accordingly to adjust parameters of the sewing machine and / or notify the user of the results of the neural network processing (e.g., to alert the user of a detected defect in a sewn seam). Figure 22 .

[0042] During training of the neural network 128, the node parameters (i.e., at least one of the input parameters, function parameters, and output parameters) of each node in the neural network are adjusted by a backpropagation algorithm until the output of the neural network corresponds to the desired output for a set of input data. Referring now to Figure 21, showing the process of training a neural network. The neural network begins the training process with node parameters that can be randomized or can be transferred from an existing neural network. Data from sensors is then presented to the neural network for processing. For example, an object can be presented to an optical sensor to provide visual data to the neural network. The data is processed by the neural network and the output is tested, whereby the node parameters of the individual nodes of the neural network can be updated to increase the confidence probability of the detection and classification performed by the neural network. For example, when a presser foot is displayed to an optical sensor to be recognized by the neural network, the neural network will present a confidence probability that the object displayed to the optical sensor is within the coordinate range of the image and can be classified as a particular presser foot. When the training process is performed, the node parameters of the nodes of the neural network are adjusted so that the neural network becomes more confident that a particular answer is correct. Thus, the neural network begins to“understand” that a particular answer is the most correct answer when certain visual data is presented, even if the data is not exactly the same as data that was“seen” previously.

[0043] When the decisions made by the neural network reach a desired level of accuracy, the neural network is considered to be“trained.” The trained neural network can be characterized by the set of node parameters that were adjusted during the training process. The set of node parameters can be transferred to other neural networks having the same node structure so that those other neural networks process data in the same way as the initially trained network. Thus, a neural network stored in a data storage device of a particular sewing machine can be updated by downloading new node parameters, as shown in Figure 23A It should be noted that the node parameters of a neural network (e.g., input weight parameters and thresholds) tend to occupy significantly less storage space than an image library used for comparison with images or visual data collected by an optical sensor. Thus, the neural network file and other critical files can be updated quickly and efficiently over a network. For example, the structure of the neural network, i.e., the connection map between nodes and the activation function calculated in each node, can also be updated in this manner.

[0044] The neural network can also be continuously trained so that the node parameters are periodically updated based on feedback provided from various data sources. For example, the node parameters of a locally or externally stored neural network can be periodically updated based on data collected from sensors that are consistent or inconsistent with the output of the neural network. These adjusted node parameters can also be uploaded to a cloud computing-based system and shared with other sewing machines so that the neural networks of all sewing machines improve over time. Input data for the neural network can also be shared with a server or cloud computing-based system to provide further training information for the neural network. The large amount of data from field sewing machines can improve the accuracy of the neural network predictions through training.

[0045] Reference is now made to Figure 96, an exemplary process 200 for controlling a sewing machine 100 is shown. The process 200 includes the steps of collecting data 202, storing the collected data in a data store 204, processing the collected data through a neural network 206, and controlling the sewing machine based on the processed data 208. The collected data is related to at least one of the sewing machine, a workspace or environment surrounding the sewing machine, sewing material (e.g., thread and workpieces), a sewing operation performed by the sewing machine, and an interaction with a user of the sewing machine (e.g., recorded by a user interface or other sensor). The neural network includes a plurality of nodes, each node including input and output connections, node parameters, and a computational unit. Based on the processed data, a user interface can be controlled to interact with a user (e.g., by presenting alerts and / or prompts), a data store can be controlled to store the processed data (e.g., as a separate record or by updating the neural network parameters), and an adjustable component can be adjusted to change a current state of the sewing machine (e.g., change a motor speed, illuminate a light, move a needle, or change any other action of the sewing machine or a sewing operation performed by the sewing machine). As Figure 22 shown, the data collected by the sensors of the sewing machine can be processed locally on the sewing machine or by an external processor in a cloud-based neural network. The locally stored neural network can be pre-trained or can be a continuously updated neural network. The software of the sewing machine then uses the data processed by the neural network, either locally or remotely, to make decisions that result in machine and / or user interactions.

[0046] Techniques employing one or more neural networks or other artificial intelligence tools can also be used to calibrate cameras and other data collection devices (e.g., sensors). Figure 23B An exemplary camera calibration method is described. Camera calibration can be run at any relevant time, such as during startup of the sewing machine, during use of the sewing machine, and at any user-determined point in time. Camera calibration can be run automatically, without any input from the user. Camera calibration can also be manually initiated by the user. The user can also fine-tune the location of the calibration points.

[0047] To perform camera calibration, the camera collects data from one or more predefined regions on one or more objects associated with the sewing machine. The data can be any relevant data that can be used for camera calibration, such as visual or image data related to the geometry, color, contrast, or reflections of the one or more predefined regions or portions thereof. The one or more predefined regions on the one or more objects used for calibration have known characteristics, such as known geometric characteristics (e.g., distances and angles) and / or known color and contrast characteristics (e.g., hue, saturation, and brightness). Color and contrast references are used to calibrate the image settings of the camera (i.e., saturation, white balance, temperature, etc.). For example, geometric references are used to calibrate the focus of the camera.

[0048] The cameras can capture data related to any suitable object, such as one or more of a needle bar, a presser foot, a presser foot ankle, a needle plate, a needle, a piece of paper or plastic, or other sewing machine features and / or accessories (e.g., fabric, projected images, and any movable object associated with the sewing machine). The calibration can use a single image or multiple images to capture data, including images that illustrate the two- or three-dimensional directional motion of the object.

[0049] The surface reference pattern can include, but is not limited to, any camera-detectable surface variation with defined geometry and location, and can include, for example, holes, edges, lines, and shapes engraved, stamped, embossed, intaglio, etched, cut, or painted on the sewing machine or sewing accessory. The surface reference pattern can be used with or without the area used for calibration. For example, in some cases, one or more objects used for calibration (e.g., a presser foot) can already have a unique topology that provides sufficient information, so no surface reference pattern is needed. If such an accessory is used, color and contrast references can be taken from another location simultaneously as needed, such as from the needle or needle plate. In this case, for robustness, multiple references can be used.

[0050] The data captured by the one or more cameras from the one or more predefined areas is sent to one or more computers on the sewing machine or to another one or more processing units associated with the sewing machine. The one or more computers process the data through one or more predictive value algorithms, such as in the hidden layers of one or more trained neural networks or in another keypoint method. The one or more neural networks are trained to detect and identify the one or more objects through known one or more features (e.g., geometry, color, contrast, topology, etc.) of the one or more predefined areas used.

[0051] The predictive value algorithm groups the object features, including the final reference pattern selected and assigned for camera calibration, and identifies the intersection of the features, including radial and tangential intersections; either already intersecting or inferred to intersect. The latent line areas of interest can be pre-trained in the neural network. For example, if the lines on the needle plate are to be used as calibration features, finding these lines and inferring them when necessary is done relatively quickly, as there is no need to compute the entire image; only the predefined area of interest is processed.

[0052] The one or more prediction algorithms provide a confidence probability of the accuracy of the known features. In other words, the one or more prediction algorithms compare the known features in the data provided by the one or more cameras to the previously learned known features of the one or more neural networks and calculate one or more accuracy prediction percentages to quantify the similarity between the one or more data features provided by the one or more cameras and the learned features.

[0053] The threshold level of acceptability can be set to any desired level. If the calculated accuracy prediction percentage (i.e. probability) is below the threshold level of acceptability, the one or more camera settings are tweaked and the process is repeated with the one or more cameras capturing more data and running the algorithms again on the new data. This process can be repeated multiple times until an acceptable probability is reached. If an acceptable probability is not reached, an alert signal or message can be sent to the user asking the user to ensure that the one or more objects to be identified are within the full field of view of the one or more cameras and that the lighting conditions are acceptable, there are no obstructions, and the relevant camera calibration surface is not unacceptably soiled. If an acceptable probability is still not reached after completing the above steps, a service alert message can be sent to the user, the plant, and the manufacturer (e.g. lens needs cleaning or other issue that can only be handled by a service technician).

[0054] If the algorithm determines that one or more features in the data are within the acceptable threshold of accuracy prediction percentage, the one or more features are used to estimate the parameters of the camera lens and image sensor calibration and adjustments can be made through software to correct the final lens and sensor image quality. For example, various parameters related to the one or more cameras can be adjusted such as, but not limited to, focus, image format, focal length, tilt, distortion, image center, color / intensity, exposure, temperature, and brightness.

[0055] For one example of a calibration procedure, at startup, the camera can acquire an image of the presser foot currently installed on the sewing machine. The image is sent to the neural network which identifies the currently installed presser foot based on its geometry and color coding. The color detected by the neural network on the presser foot (e.g. orange) is different than the color identified on the same presser foot (e.g. red) during multiple earlier sewing processes where the image setup and predicted presser foot ID were considered acceptable. The camera settings can then be modified so that the color of the image data appears as red.

[0056] Reference is now made to Figures 24-33 and Figures 104-105where various views and diagrams related to the use of artificial intelligence in a sewing machine are shown to control the position of stitches made in a fabric during a sewing process. During normal sewing operations, the position of the stitches is typically left to the user. That is, the user can be provided with various visual aids, i.e. guides on the needle plate, projected guides, markings on the workpiece, etc., and it is up to the user to keep the sewing path in the correct position. However, the visual aids cannot control the position of the workpiece, and thus the final position of any stitch depends on the skill of the user to keep and guide the fabric in the correct direction. However, the sewing machine described herein can use one or more optical sensors and a depth perception system (which will be described in more detail below, and which can include optical sensors, projectors, ultrasonic, and thermal vision systems) to find the desired path on the workpiece, and can manipulate the lateral position of the needle bar and workpiece by the feed dog, so that even if the user happens to move the workpiece out of line, the stitches can be placed along the desired path. The feed dog used to affect the feed direction of the workpiece can include a linear translation feed dog, a linear translation feed dog combined with a circular rotating feed dog, and a multi-component feed dog with two or more independent moving components, such as left and right portions that translate different distances and / or speeds (similar to a gas tank pedal) to turn the workpiece during feeding. The two or more components of the feed dog can also be arranged at different heights to accommodate sewing together fabrics with different thicknesses.

[0057] During sewing, it is often desirable to form a line of stitches along a seam that has already been formed between two or more pieces of fabric, such as Figure 24 shown. Sewing along a slit seam is colloquially referred to as “sewing in” because the two fabrics tend to rise up from the seam, so the seam appears to have a long, low groove with a relatively deep wedge-shaped cross-section between the two pieces of fabric. Forming stitches along a slit seam helps to hide the stitches in the finished quilt. Since the slit seam is a moving and very narrow target, it is very challenging to maintain a constant stitch position along the slit seam. Thread that is slightly matched in color to the surrounding quilt piece or even transparent thread can be used to try to hide the thread in the event of a missed stitch.

[0058] Referring now to Figure 25 , the field of view of the optical sensor or optical sensor and depth perception system of the sewing machine is overlaid on the image of Figure 24 . As Figure 104As shown, the user begins stitching and activates the "stitch into groove" function. As the workpiece is arranged on the sewing table, data is continuously collected by sensors and processed through a neural network. The collected data includes data collected from cameras directed at the sewing area upstream and / or downstream of the drop needle position, data related to the sewing operation (e.g., stitch type and parameters), thread data (e.g., thread tension), and collected data related to the sewing material (e.g., feed rate, motion vectors, and topology of the workpiece). The data is processed through a neural network that has been trained to detect and identify a groove formed between two or more pieces of fabric. That is, the neural network provides a confidence probability as to the location of the groove and the details of its appearance. When the user begins sewing, for example by depressing a foot pedal, pressing a button, issuing a voice command, etc., and moves the workpiece into position under the needle, the sewing machine detects and identifies the groove and controls the position of the stitch formed on the workpiece by oscillation of the needle bar and / or lateral feed of the workpiece to form a stitch along and in the groove. Figures 26-33 The actuator shown can be used to change the lateral position of the needle bar, thereby causing the needle to form a stitch through the groove. When moving the workpiece, the Figures 30-33 The actuator shown controls the feed dog so as to slightly move the workpiece laterally during the usual forward and backward feed of the workpiece. That is, the feed dog is capable of moving in two axes, thereby allowing the direction of the sewing path to be changed in addition to the feed rate of the workpiece, which is normally controlled by the feed dog.

[0059] Reference is now made to Figure 105A process similar to "grooving" can be used to form stitches at predetermined offset distances or tolerances of workpiece features. As the workpiece is placed on the sewing table, data continuously acquired by sensors is processed by a neural network. The acquired data includes data from cameras pointing upstream and / or downstream of the needle drop position towards the sewing area, data related to the sewing operation (e.g., stitch type and parameters), thread data (e.g., thread tension), and data related to the sewing material (e.g., feed rate, motion vector, and workpiece topology). The data is processed by a neural network trained to detect and recognize workpiece features such as edges, seams, "grooves" between two pieces of fabric, buttonholes, pockets, etc. In other words, the neural network provides confidence probabilities about the location of features and their appearance details. When the user begins sewing, for example by pressing a foot pedal, a button, or issuing a voice command, and moves the workpiece to the needle position, the sewing machine detects and recognizes the features and controls the position of the stitches formed on the workpiece by the oscillation of the needle bar and / or the lateral feed of the workpiece, thereby forming stitches at predetermined offset distances from the recognized features. For example, a user can specify a half-inch stitch tolerance and begin sewing with the workpiece if the edge is positioned within the lateral movement range of the needle bar. As the workpiece moves through the sewing machine, the edge is detected and a stitch is formed half an inch from the edge, without the user needing to precisely follow the edge guide.

[0060] Now for reference Figures 34-39 The diagram illustrates various computer vision systems. In addition to visual data provided by optical sensors, neural networks can also receive input from depth-sensing systems to provide more accurate calculations of the location of grooves and 3D topology. Depth-sensing systems can use any of, but are not limited to, stereo vision techniques and other computer vision techniques. Figures 34-39 The techniques shown use two optical sensors, two optical sensors and a projector, a projector and a single optical sensor, an optical sensor and a laser light source, a thermal vision system, or an ultrasonic vision system to calculate the distance to the workpiece. Now refer to... Figure 34 This paper illustrates a passive stereo depth sensing system that uses two cameras similar to human stereo vision to determine the distance to a target object based on a comparison between two images captured by the cameras. Figure 35 The active stereo vision system shown is similar, but includes a projector for projecting graphics onto a target object to enhance distance measurements between the two cameras. Figure 36 The structured light vision system also uses a projector to project lines or other visual patterns onto the target object. The camera can observe these lines or patterns to determine the distance from the camera to the target object. Another depth-sensing system is... Figure 37As shown, the system uses a camera to calculate the distance to the target object, which measures the time it takes for light from the laser to travel from the laser to the target object and then back to the camera. This distance information can also be provided to a neural network that processes visual data of the workpiece from the optical sensor to calculate the position of the stitch as the workpiece is moved under the sewing head to form a stitch in the workpiece. That is, the line connecting the points on the workpiece that are farthest from the sewing head can be identified as the stitch in the fabric.

[0061] It should also be noted that the optical sensor and depth perception system described above can have a variety of uses. That is, the one or more optical sensors and depth perception devices can be used to identify the topology of the fabric and thread in three dimensions to identify the type of fabric material and the type of thread that has been used in the workpiece. The density and type of fabric material can also be determined using an ultrasonic or thermal vision system, which can be part of the depth perception system. That is, denser materials respond differently to ultrasonic pulses than less dense materials. A laser, infrared radiation, or some other heat source can be used to heat a portion of the workpiece, which can be detected by a thermal vision system including, for example, an infrared sensor. Thus, the thermal conductivity of the fabric can be measured and compared to known values for different types of fabric. This functionality is particularly useful in the embroidery process when the workpiece is being worked on using existing stitches. The information provided by these systems can also be used to identify the type of fabric and thread used in the workpiece to automatically adjust the sewing machine for sewing that type of material and to recommend to the user a particular needle or other accessory that can be installed in the machine for that workpiece. Automatic lighting adjustments can be made to enable the user of the sewing machine and the sensors to view the workpiece material in a manner particularly suited for sewing (i.e., lower light levels improve the visibility of highly reflective fabrics). Furthermore, as described in further detail below, the sewing machine can provide suggestions or even warnings to the user based on the identified combination of thread and fabric types. The 3D topology of the workpiece can also be used to determine when to release the pre-tension on the presser foot to more easily crawl over multiple layers of fabric, such as when sewing a hem.

[0062] The processing of the distance information by the neural network along with the visual data further improves the accuracy of the information because the neural network can be trained to take into account the appearance and shape of the workpiece when determining the stitch position. The neural network used to process the visual and distance data can be trained elsewhere, and during the sewing process, the neural network can be updated with new data from the optical sensor and camera to improve the accuracy of the stitch position determination. Figure 23Aand / or the neural network. For example, the same or additional optical sensors can be used to observe the stitches being formed in the workpiece to identify stitches that miss the channel. When the computer knows the stitch pitch, it can determine the control data for the particular missed stitch and use it to adjust the node parameters of the neural network to reduce the chance of a missed stitch. The computer on the sewing machine can work in conjunction with a cloud-based neural network that can provide additional computing power for processing the data provided to the neural network and training the neural network during the operation of the sewing machine to make the neural network a continuously learning neural network.

[0063] The techniques described above for precisely forming a "stitch into channel" can be more broadly applied to sewing to form "perfect stitches" in a wide variety of situations. That is, data from one or more optical sensors and depth perception systems can be processed by a neural network to provide control data to one or more motors and actuators of a sewing machine to precisely and accurately form any type of stitch desired at any particular location on a workpiece. In addition to using visual data from optical sensors and depth perception data from depth perception systems, a perfect stitch control system can take into account data from thread tension sensors, needle position sensors, needle force sensors, fabric feed rate sensors, speed and frequency of needle bar motion, pressure applied by presser feet, feed rate of feed dogs, etc. Data from these sensors can be processed by a neural network to predict whether it is likely that an incorrect stitch can be made and can direct the control system to adjust various parameters accordingly to compensate for any factors that can cause errors. In processing the data provided by these sensors, the computer of the sewing machine can use decision information from the neural network to adjust various sewing parameters such as thread tension, needle position, force, speed and timing, stitch length and type, motor speed, and fabric feed settings to actively achieve ideal stitch precision and accuracy. All of these features can be combined to correlate machine performance with the skill level of the user. That is, the sewing machine can learn to work with novice, intermediate, and advanced users to adapt the speed of the machine, the presentation of corrections and alerts, recommendations for guidance or assistance provided to the user, etc.

[0064] As with the above example of sewing into a groove, the sewing machine can also check for errors in the stitches that have been formed. That is, for quality purposes, each completed stitch can be actively monitored. If the data collected by the sensors of the sewing machine indicates that an imperfect stitch has been formed (e.g., a stitch has been skipped or misaligned), the output data generated by the neural network can be used to make a decision about adjustments that can be made to the parameters of the sewing machine. These adjustments can be made and the resulting stitches monitored until a stitch that is perfect is formed. The sensors can also be used to detect a broken thread so that sewing can be stopped and the thread replaced. Thus, as the neural network is continually trained, the quality of the stitches can improve over time. For example, zigzag stitches can be controlled to maintain a particular width on either side of a seam in fabric, thereby creating a continuous stitch in the opposing piece of fabric. Or, when performing a simple straight stitch, the tension of the upper and lower thread can be controlled to avoid the stitch going through one side of the workpiece. The optical sensors can also identify a pre-existing pattern of thread that is part of a pattern on the workpiece (e.g., by weaving into fabric or by printing onto fabric), stretch, superimpose, stretch, or project onto the fabric, thereby forming stitches along or at a constant offset distance from the thread. That is, the optical sensors can be used to detect the edge of a piece of fabric and help the user sew along the edge of the fabric with a constant thread tolerance. Two or more pieces of material can have edges that the user attempts to align during the sewing process, and the sewing machine can detect misaligned workpieces and suggest corrections to the user.

[0065] Figure 106An example of a process to detect and adjust for sewing errors is shown. As a user sews, data is constantly collected from cameras pointing at the sewing area upstream and / or downstream from the drop point, the sewing operation (e.g., stitch type and parameters), thread data (e.g., thread tension), and the sewing material (e.g., feed rate, motion vectors, and the topology of the workpiece). Input data can also be provided from a database of known sewing errors and their causes, such as a puckered seam can be caused by an imbalance in the thread tension of the top and bottom thread. This data is processed through a neural network that has been trained to recognize sewing errors, and the recognized errors can be recorded with contextual information (e.g., the parameters of the sewing machine when the error occurred or the motion of the workpiece when the error occurred) and reported to the user. The recorded information can be used to update local and remote neural networks to improve error detection and prediction. A non-exhaustive list of sewing errors includes skipped stitches, stitch imbalance, stitch misalignment, seam puckering, stitch density variation, bobbin thread breakage, needle thread breakage, needle thread breakage, melted thread, needle breakage, stuck needle, needle hitting the needle plate, thread being cut by the needle, inconsistent thread tension, wavy seam, threadless stitch, needle holder loose, presser foot loose, presser foot misaligned, needle unable to move, workpiece unable to move, workpiece bunching, thread bunching, thread knotting, stitch loose, thread tangling, thread wear, thread tear, workpiece feed variation, needle bending, bent needle damage, needle finger damage, bent needle misalignment, needle finger misalignment, and dull fabric knife.

[0066] A continuously trained neural network, i.e., a neural network that is trained and can be adjusted during the sewing process, can ultimately adjust many parameters of the sewing process in unpredictable ways to compensate for unforeseen problems that are very difficult to predict and solve or impossible to predict and solve through traditional control software or by a user adjusting the settings of the sewing machine. For example, the sewing machine can adjust the feed rate and the stitch pitch in response to a user applying an external force to the sewing machine that would otherwise move the workpiece out of the thread. In doing so, the neural network can also determine that an adjustment to the thread tension or the presser foot pressure is useful. That is, the sewing machine can learn to compensate for and even resist the user’s incorrect movements to further guarantee that the stitches formed are accurate and precise.

[0067] The projector of the sewing machine can be used in conjunction with the artificial intelligence technology described herein to improve the position of the image projected onto the workpiece. For example, as Figure 107As shown, a neural network can be used to identify features of the workpiece so that a sewing guide can be projected at the location of the feature or at a predetermined distance from the feature. As the workpiece is arranged on the sewing table, the data continuously collected by the sensors is processed through the neural network. The data collected includes data collected from cameras directed at the sewing area upstream and / or downstream of the drop needle position, data related to the sewing operation (e.g., stitch type and parameters), thread data (e.g., thread tension), and collected data related to the sewing material (e.g., feed rate, motion vectors, and topology of the workpiece). The data is processed through the neural network, which has been trained to detect and identify features of the workpiece such as edges, seams, “gutters” between two pieces of fabric, buttonholes, pockets, etc. That is, the neural network provides a confidence probability as to the location of the feature and details of its appearance. When the user begins sewing, e.g., by depressing a foot pedal, pressing a button, issuing a voice command, etc., and moves the workpiece into position under the needle, the sewing machine detects and identifies the feature and controls the projector to project a sewing guide, e.g., a straight line in the feed direction at the location of the feature or at a predetermined offset distance from the identified feature. For example, the user can specify a half-inch seam allowance and can activate the sewing guide that projects a line half an inch from the edge of the workpiece as the user moves the workpiece so that the user can correct the lateral position of the workpiece to form the stitches at the desired location.

[0068] Reference is now made to Figures 40-42 various views and diagrams related to the use of artificial intelligence in a sewing machine to predict the path of stitches being formed so that a predicted stitch image is projected onto the fabric ahead of the needle position to inform and guide the user. As with the stitch adjustment and control features described above, an optical sensor collects visual data from the fabric workpiece and provides that data to a computer. The computer processes the data through a neural network that has been trained to predict a sewing path based on the visual data related to the stitches that have been formed and the parameters of the sewing machine (e.g., needle position and speed, fabric position and speed, feed dog rate, force applied by the presser foot, tension of the upper and lower thread, user selected speed, etc.). The neural network processes the data and provides the predicted sewing path to the computer of the sewing machine, which then projects a series of stitches ahead of the needle along the predicted path. The user selects a stitch type through a user interface (e.g., a touchscreen, a button, a voice command, etc.) and the computer of the sewing machine projects the selected stitch type ahead of the needle. Figure 40 ) of the predicted path. The user selects a stitch type through a user interface (e.g., a touchscreen, a button, a voice command, etc.) and the computer of the sewing machine projects the selected stitch type ahead of the needle. Figure 41so that the user can see the shape of the stitches being formed along the predicted sewing path. As the user or the fabric translation portion of the sewing machine moves the workpiece, the projected path of the stitches also moves, so that the path appears in a constant location on the workpiece. (In an embroidery machine, the projected embroidery pattern can move with the workpiece during movement of the embroidery frame.) The predicted path can also be adjusted to suggest that the user can follow the path back to a pattern that has been deviated from. The projected path of the stitches can also begin at the drop point and extend in the feed direction in a straight line or a curve that does not move when the workpiece is rotated or translated.

[0069] Reference is now made to Figure 42 The projected stitches appear to be swallowed by the actual stitches being formed in the workpiece. The user can also set a prediction distance so that only a few predicted stitches are shown or a line showing the stitches extending to the limits of the projector's range. The embroidery pattern can also be projected in a similar manner so that the projected stitches disappear as the pattern is formed in the workpiece. As the workpiece held by the embroidery frame moves, the projected image also moves to track the workpiece so that the stitches follow the projection of the predicted embroidery patch.

[0070] There are many benefits to projecting the predicted stitch path along the workpiece in front of the needle. In some cases, the user can wish to place a smooth stitch curve that terminates near or a distance away from an existing feature of the workpiece. Or, the user can wish to avoid contacting or overlapping an existing feature of the workpiece. In these cases, the predicted sewing path that moves with the workpiece will help to create the desired seam in a single pass. In addition to the predicted sewing path, additional information can also be provided. For example, if the predicted projected path is going to encounter or come too close to a feature of the workpiece that the user has designated as an object to be avoided, or the sewing machine recognizes and predicts a feature that the user wishes to avoid (e.g. a needle, a button, another seam, a buttonhole, a decorative element, an edge of the fabric, etc.), the projected stitches can change color. In these scenarios, the projected stitches can also blink and can be combined with other notifications such as the audible or haptic feedback discussed in this disclosure. Alternatively, the projected path can be automatically changed by the sewing machine to guide the user around the obstacle, with the original path and the new, changed path projected in different colors, and / or with a motion cue that clearly indicates that the path has changed, such as by blinking or an animated arrow near the path.

[0071] Such warning signals and alerts can also be sent if the user’s fingers move into the projected sewing path or the path of other components of the sewing machine, such as the presser foot or connected accessories. It should be noted that the projector is not limited to projecting only the projected sewing path, but can also project many other symbols and / or words in the vicinity of the projected path to notify and remind the user of changes in the path or obstacles. For example, the neural network can recognize a button on the fabric and provide the computer system with the location and size data of the button so that the computer can instruct the projector to project the outline of the button around the button on the workpiece, thereby drawing the user’s attention to that feature.

[0072] As a last resort, if an obstacle is about to be hit by the needle and the user has not responded (e.g., through a touchscreen interface or through a voice control system) to avoid the obstacle, the sewing machine can stop completely. Figure 108 A flowchart is shown illustrating the use of a neural network to avoid injury to the user or damage to the sewing machine during a sewing operation. When the user begins sewing on the sewing machine, data is collected from a camera pointed at the sewing area and can also be collected from other sensors, for example, one or more microphones listening to the environment to capture vocal cues or other expressions made by the user. The collected data is processed through a neural network that is trained to detect foreign objects that can be damaged by the sewing machine or that can cause damage to the sewing machine. For example, the neural network can recognize a user’s finger under the presser foot or in the path of the needle. Audio data can also help determine if the user is engaged in a conversation and can be distracted, thereby increasing the likelihood of inadvertently placing a finger or hand. Once an object is identified, the sewing machine can alert the user and stop the sewing operation or lower the presser foot to avoid injury to the user and the sewing machine. The foreign object can not be directly in the sewing path, but can be in the vicinity of the path, causing the sewing machine to generate an alert. For example, as described above, the sewing machine can alert the user with a sound or project a warning on the workpiece. If no foreign objects are detected, the sewing operation proceeds in a normal manner.

[0073] In addition to compensating for deviations from the expected sewing path, the data collected by the sewing machine during a user’s sewing can also be analyzed by the neural network to detect the user’s level of expertise. For example, frequent deviations from the expected sewing path can indicate that the user is a novice, while a small number of deviations can indicate that the user is an expert. The sewing machine can then suggest guidance and training exercises to the user to improve. The feedback can be shared in any individual or combination of ways, including audio, text, video, image projection, and augmented reality configurations from the sewing machine or connected devices. Adjustments to the settings of the sewing machine can also be suggested to improve the sewing of novice sewers and increase the efficiency of expert sewers. The sewing machine can also provide new opportunities and challenges for advanced users to help them further improve and expand their skills.

[0074] Figure 109 An exemplary flowchart showing the problem of detecting a line using a neural network is shown. As the user uses the sewing machine in any manner, data is collected from user-facing sensors such as cameras, real-time interactions of the user with the user interface, a log of historical interactions with the sewing machine, and information related to the current sewing operation, if any. As described above, the collected data is processed through a neural network that has been trained to detect the skill level of the user. If the neural network has assessed the skill level of the user, the sewing machine can continue to alert the user that the task is beyond the detected skill level or can appropriately provide helpful tips or hints. As described above, the sewing machine can also provide recommended training exercises based on the detection of the skill level of the user.

[0075] Analysis of the skill level of the user can also be applied to the interaction between the user and the sewing machine. That is, the sewing machine can analyze through a neural network that the user is struggling to use a feature of the sewing machine correctly and can suggest a tutorial video or instructions and can provide a prompt on the screen to help the user know which user interface control to interact with next. The user interaction data can include the user-facing camera data described above and can also include temporal information from the user interface that indicates the speed at which the user interacts with the settings of the sewing machine. The time it takes for the user to interact with the sewing machine can be an indicator of the skill level of the user; that is, a user that selects menu items more quickly in the user interface can be more familiar with the sewing machine and, in combination with other data, can help the sewing machine identify an estimated skill level of the user. For example, after activating a feature, the sewing machine can highlight the button and display a pop-up message prompting the user to take the next step to use the activated feature. Input from the user-facing camera and facial recognition technology provides further input about the emotional state of the user as they interact with the sewing machine. That is, when the user appears frustrated or confused, graphical and sound prompts can be provided. Alternatively, the sewing machine can refrain from providing further prompts that can be perceived as annoying and unhelpful in order to best support and guide the user in resolving any issues they are trying to solve.

[0076] Based on data collected from monitoring the usage of the sewing machine, the sewing machine can also provide useful recommendations for other products or accessories. Advertising for products can be done through any single or combination of means, including audio, text, video, image projection, and augmented reality configurations from the sewing machine or connected devices. In recommending products, the sewing machine or external processor collects and monitors data through real-time or retrospective data analysis, particularly frequency and preferences of, for example, user-selected sewing accessories, programs, and machines. For example, the sewing machine can keep track of the amount of each type of thread used and, knowing the typical amount of thread purchased, can recommend purchasing more thread when there is an estimated shortage. Another example is when a user uses a certain presser foot for certain purposes and there is a more suitable presser foot, the sewing machine can recommend purchasing the more suitable option if the user has not inputted it into the list of sewing accessories currently owned. The list of sewing accessories can be stored on one or both of the sewing machine and the application on the connected device. This data can be sent back to the manufacturer to enable the engineering, marketing, and customer service teams to improve the quality of the sewing machine and other products.

[0077] Reference is now made to Figures 43-65 , which shows various views and diagrams related to the use of artificial intelligence in a sewing machine to identify the thread and fabric materials used in the sewing machine to adjust the sewing parameters and provide information to the user about the combination of thread and fabric identified by the sewing machine. Reference is now made to Figures 43-49 , which shows parts of a sewing machine that illustrate the path that thread can take from bobbins mounted on top of the sewing machine ( Figures 43-45 ) and from bobbins mounted below the needle plate (see Figures 46-49 ) to the sewing needle.

[0078] The sewing machine can include various sensors along these thread paths to detect the type of thread that the user has installed in the machine. These sensors can include, but are not limited to, RGB sensors, light sensors, optical sensors, such as cameras, and the like. Illumination sources and magnifying lenses can also be provided with specific sensors. For example, as shown in Figure 50 , optical thread sensors can be included at the top of the sewing machine arm and behind the bobbin mounting location. In Figure 87An exemplary thread sensor 140 is shown in FIG. 1, which includes a tubular housing 142 through which a thread 141 passes. The tubular housing 142 blocks ambient light from shining on the thread 141, so a light source 144 is provided to illuminate the thread 141 for detection with an optical sensor 146, such as a camera or RGB sensor, which is used to collect thread data. The sewing machine can also include sensors for detecting parameters of the thread and mechanisms for adjusting that parameter. The tubular geometry of the sensor assembly provides a known background for the illumination of the thread, which improves the accuracy and precision of the thread information collected by the RGB or other sensor. The sensor disposed in the tubular housing can detect the light, sound, or other parameters of the thread to determine the color, density or weight, surface quality, material or fiber type, and overall quality of the thread. That is, the RGB or other sensor can be used to detect the inherent characteristics of the thread as it passes through the sensor housing.

[0079] The data collected by the thread sensor is transmitted to the computer of the sewing machine and can be compared to a database of thread information containing information about various thread types and colors. Thus, the sewing machine can identify the thread and present information to the user that the user can not know. If a particular thread can be identified from the information on the spool (either manually entered by the user or detected by the machine), then the detected thread characteristics of the thread can be compared to the thread characteristics stored in the thread information database. Thus, the sewing machine can detect a thread that is significantly different from the stored thread characteristics, which can indicate a defective spool of thread, and thus a reminder can be presented to the user indicating the same. The information on the spool of thread can be collected by an optical sensor or other sensor disposed near the spool pin on which the spool is mounted during the sewing process. The spool information can also be collected from the spool when the user holds the spool in front of an optical sensor or other sensor disposed in the sewing head or another location, such as a camera in the sewing head or one or more cameras facing the user. The time and date that the thread is identified can be stored and associated with the project, type of stitch, etc. to establish a history of thread usage in the sewing machine.

[0080] The sewing machine can also include sensors for detecting the current state of the thread as it is manipulated by the machine and can include mechanisms for adjusting the sewing machine. For example, the sewing machine can include a thread tension sensor Figures 51-54 , a thread dispensing unit Figures 55-58 , and a thread tension unit Figures 59-62 . The sensors for detecting the inherent characteristics of the thread and the current state of the thread are disposed so that data about the quality and state of the thread is collected as the thread passes from the spool or bobbin, through the thread tensioner, around the hook or other elements of the sewing machine, and ultimately through the needle. As described in more detail below, the optical sensor can also be used in conjunction with a neural network to detect the type of presser foot and / or needle that is assembled to the machine.

[0081] Sewing machines also include optical or other sensors that can be used in conjunction with neural networks to detect material or fiber type, color, reflectivity, pattern, weave direction, orientation (i.e., right side and wrong side), and the topology of the fabric used in the workpiece. Exemplary sensors for acquiring data about the workpiece fabric include a radiation source (e.g., a light source or infrared light source) positioned on the sewing head and pointing downwards at the workpiece. A radiation detector, such as a light sensor or infrared light sensor, is positioned on the sewing table, i.e., below the workpiece. It should be noted that the positions of the transmitter and receiver can be reversed, i.e., by providing the transmitter in the sewing table and the receiver in the sewing head. Therefore, the amount or portion of emitted radiation (e.g., visible or infrared light) passing through the workpiece, and thus the amount of radiation reflected by the top surface of the workpiece, can be detected and measured. Ultrasonic transmitters and receivers can be arranged in a similar manner, i.e., the transmitter on the sewing head and the receiver on the sewing table, to provide a method for determining fabric density more accurately than other techniques. These transmitters and detectors—that is, those for light (infrared radiation (IR), cameras), color (RGB), ultrasound, etc.)—can be used individually or together to determine the material or fiber type, density, and reflectivity of the workpiece material. The additional depth sensing techniques described in this paper can also be used to detect the topology of a workpiece.

[0082] Data collected by fabric sensors is transmitted to the sewing machine's computer or any connected external processor and compared with a fabric information database containing information about various fabric types with different colors and patterns. Therefore, the sewing machine can identify the fabric of a workpiece and present information the user might not be aware of. If a specific fabric can be identified from information on a roll of fabric (either manually entered by the user or detected by the machine), the detected fabric characteristics can be compared with stored fabric characteristics from the fabric information database. Thus, the sewing machine can detect fabrics with significantly different characteristics from those stored, potentially indicating defective pieces, and can then present the user with a warning indicating the same issue. Workpiece identification data can be combined with stitch data to train a neural network to associate workpiece features with different stitches. Therefore, the sewing machine can alert the user if a stitch is being formed on the wrong side of a workpiece facing the wrong direction.

[0083] like Figure 63 As shown, the sewing machine's computer processes data collected by the various sensors and other devices mentioned above using a neural network. The neural network is trained to provide suggestions, reminders, and warnings to the user based on the input data. That is, the neural network is trained to recognize compatible and incompatible combinations of thread, fabric, presser foot, and needle types. For example, Figure 64A table of fabrics and threads is shown that indicates whether a heavy or light fabric is compatible with a heavy or light thread. If the sewing machine detects that a combination of thread and fabric can have issues, the user can be provided with a suggestion on the display accompanied by an audible or haptic notification. If the potential compatibility issue is more serious, the user can be alerted with a reminder or even a warning. In some cases, the sewing machine can stop completely and provide a combination of audible, haptic, and visual warnings. In addition to notifying the user of potential compatibility issues, the sewing machine can also make adjustments such as thread tension, presser foot pressure, type and speed of stitches, etc. to improve sewing performance when using heavy or light threads and / or heavy or light fabrics. Even when the correct type of thread is selected for a given fabric, the color of the thread can not be aesthetically pleasing given the selected fabric color and / or pattern. Thus, the neural network can also be trained to suggest compatibility of various thread and fabric colors and patterns to the user as Figure 65 shown.

[0084] Figure 102 An exemplary flowchart is shown for identifying a workpiece and potential issues with the workpiece using a neural network. When a user begins sewing on a sewing machine, data is collected from a camera pointed at the sewing area, the sewing operation, optical thread sensors, feed rate sensors, a database of known workpiece or fabric materials, and a log of previously identified workpiece materials. The collected data is processed through a neural network that is trained to detect workpiece compatibility issues, damage, and other thread quality issues. If the neural network identifies a workpiece and the workpiece is not compatible with the current sewing operation and other sewing materials (e.g., a lightweight thread can break when used with a thicker or heavier workpiece fabric), the sewing machine alerts the user that sewing can continue if the user chooses to override or ignore the notification. If workpiece damage or other quality issues are identified, the user is also alerted. When damage has progressed sufficiently that user intervention (e.g., replacing or repairing the workpiece) is required, the sewing machine can selectively disable further sewing.

[0085] Reference is now made to Figures 66-71 and Figure 101 show various views and diagrams related to using artificial intelligence in a sewing machine to identify reduced thread quality, adjust sewing parameters, and provide information to the user about the quality of thread being used. As described above, the sewing machine can include various sensors along one or more paths of thread in the sewing machine from a thread source to a sewing head as shown in Figures 43-49 These sensors can include, but are not limited to, RGB sensors, light sensors, optical sensors such as cameras, etc. The sensors are arranged to collect data about the quality and state of the thread as it passes from a spool or bobbin through a thread tensioner, around a hook or other elements of the sewing machine, and ultimately through a needle. Additional sensors can be included, for example, thermal sensors, to monitor the temperature of various components that interface and can cause damage to the thread.

[0086] Now for reference Figure 66 and 67 Examples of the appearance and characteristics of high-quality and low-quality wires are shown. Wires considered high-quality or in good condition have the following characteristics: tight and strong fibers, consistent diameter, consistent color, consistent reflectivity, and consistent friction. Wires considered low-quality or in poor condition have the following characteristics: loose and worn fibers, inconsistent diameter, inconsistent color, inconsistent reflectivity, inconsistent friction, and poor splicing. Additional light can be provided in or near a sensor, such as the tubular sensor housing 142 described above, to provide a consistent light source when observing the wire, so that the wire is not misdiagnosed based on color changes in varying lighting conditions (e.g., daylight, cool white light, horizon, and incandescent lamps). Figure 68 As shown, when using low-quality thread, one or more optical sensors in the sewing machine can also detect debris buildup in areas of the sewing machine known to accumulate in the sewing machine area.

[0087] Now for reference Figure 69 This document illustrates a flowchart of an exemplary scenario regarding the quality of thread used in a sewing machine. In the illustrated scenario, sensors collect data related to the state of the thread used in the sewing machine. The data is processed by a previously trained or continuously trained neural network to determine if the thread exhibits any signs of low-quality thread. When poor-quality thread is detected, the user is notified via the notification methods disclosed herein. Figure 70 This can be achieved through various means, such as a user interface, computer-generated voice, indicator lights, and haptic feedback. The user can then view the sewing machine's display for an auditory description of the thread quality, or request further details. The user can choose to reject a warning or take action, and then continue sewing. In sewing machines with multiple spools, the machine can also track the thread parameters of each spool and notify the user which spool (if any) contains low-quality thread.

[0088] Figure 101Another flowchart is shown using a neural network to detect thread issues. As the user begins to sew on the sewing machine, data is collected from the camera pointing at the sewing area, the sewing operation, optical thread sensors, other thread sensors for measuring thread tension, feed rate, and dispensing, a database of known thread materials, and a log of previously identified thread materials. The collected data is processed through a neural network trained to detect thread compatibility issues, damage, and other thread quality issues. If the neural network identifies a thread and that thread is not compatible with the current sewing operation (e.g., the thread can break when used in a particular stitch), the sewing machine alerts the user, and if the user chooses to override or ignore the notification, sewing can continue. If thread damage or other quality issues are identified, the user is also alerted. The sewing machine can optionally disable further sewing when damage has progressed sufficiently that user intervention (e.g., to replace the thread) is required.

[0089] The sewing machine can also include multiple thread quality sensors, such as one or more sensors 140 disposed in the tubular housing 142 as described above, along the thread path to determine if the quality of the thread changes along the path. For example, if a drop in thread quality is found after a particular feature of the thread path, the sewing machine can suggest changing a sewing parameter to reduce the likelihood that the sewing machine is causing damage to the thread. Monitoring of thread quality at multiple locations along the thread path also provides the sewing machine with an opportunity to suggest checking various components that can need repair or replacement, such as a guide that can have a sharp edge that is causing thread wear. Such monitoring can also allow the sewing machine to identify improper threading of the sewing machine based on where the thread appears to deviate from the intended thread path through the sewing machine.

[0090] Referring now to Figures 71-81 various views and diagrams related to using artificial intelligence in a sewing machine to recognize and identify objects placed in the field of view of optical sensors of the sewing machine and to provide information to the user about the characteristics of the object and the relationship between the object and the sewing machine. The optical sensors (e.g., cameras) can be sensors pointing at the sewing area or can be front-facing or user-facing sensors that allow the user to hold up an object in front of the sensor facing the user to detect components. A library or database of recognized objects is stored to enable the sewing machine to build an inventory of known objects, such as components of the sewing machine or accessories used with the sewing machine. For example, the sewing machine is able to identify the type of needle installed on the sewing machine and whether the needle is properly installed ( Figures 71-72 ), the type of presser foot installed on the sewing machine and whether the presser foot is properly installed ( Figure 73 ), the type and characteristics of an embroidery frame installed on the sewing machine and whether the workpiece is properly installed within the embroidery hoop ( Figure 74 ), the user’s fingers and hands, and whether there is a safety risk to the user during the current operation ( Figure 75). With respect to embroidery hoops, the sewing machine can identify, for example, whether the clamping mechanism used to secure the embroidery hoop has been secured, whether the workpiece is flat in the hoop, and whether all of the fabric edges are outside of the hoop. The quality of the assembly can also be identified, that is, the sewing machine can also detect whether the assembly is damaged, rusted, bent, worn, incorrectly threaded (in the case of needles and bobbins), or otherwise altered from acceptable quality standards for the assembly. In each of these examples, a neural network is used to process visual data captured by one or more optical sensors of the sewing machine.

[0091] Referring now to Figure 76 Data captured by various sensors of the sewing machine is processed by a computer of the sewing machine through a neural network to determine whether the sewing machine detects a particular object and whether that object should be present there. For example, as shown in Figure 77 optical data within the range of the presser foot of the sewing machine can be captured. The visual data of the image is processed through a neural network to determine whether a presser foot is present, which presser foot is present, and whether the presser foot is properly installed. Similar determinations can be made for the needles installed on the sewing machine. Once the presser foot and needles are identified, the range of translation for the respective needle for the presser foot is stored, and the user can be notified if the combination of needle and presser foot is not recommended. The user can then choose to override the warning, for example, by selecting an “expert mode” that includes a warning about the safety risks that can be involved in selecting the “expert mode.” The selected stitch is also compared to the installed presser foot and needles to determine whether the installed presser foot and needles are appropriate for and compatible with the selected stitch or series of stitches in the project. If no presser foot or needle is installed, a presser foot and needle can be recommended by the sewing machine. Upon installation of the presser foot and / or needle, the sewing machine can again check the presser foot and needle to confirm that the appropriate presser foot and / or needle has been installed and that the needle and / or presser foot has been properly installed. The sewing machine can also identify conflicts, for example, between the needle and the sewing bed, the presser foot and the selected stitch pattern, and one or more needles and the selected stitch pattern. Incompatibilities between stitch types and presser feet can be provided, for example, in a table or database of incompatibilities, or can be learned over time by monitoring sewing errors related to the identification of various assemblies and sewing operations performed.

[0092] Figure 100Another flowchart is shown for detecting objects, identifying compatibility, and installing problems using a neural network. As the user begins to stitch on the sewing machine, data is collected from the camera pointing at the sewing area, the sewing operation, a database of known components and accessories previously used with the sewing machine, and a database of known components and accessories that are compatible with the sewing machine. The collected data is processed through a neural network that is trained to detect components and accessories, classify those components and accessories, determine if the components and accessories are properly installed, and determine if the combination of components and accessories and the selected sewing operation creates any conflicts or other problems. If the neural network identifies that a component is not compatible with the sewing operation or can cause problems, the user is alerted and given the opportunity to override the alert (e.g., similar to the “expert mode” described above). The neural network identifies if the components and accessories are properly installed. If not, the user is alerted and the sewing machine can be prohibited from operating until the components are removed or properly installed.

[0093] Similar determinations can be made for embroidery hoops that can be installed above the sewing table. Once the type and size of the embroidery hoop is determined, the sewing machine can notify the user if the selected embroidery pattern will exceed the limits of the embroidery frame. The sewing machine can also check the edges of the fabric held in the embroidery frame to detect incorrect installation of the fabric in the hoop. In the event that a problem with the fabric installation or embroidery hoop size is detected, the user can be notified through any of the notification means described herein, such as a visual display of information on the sewing machine display, an audible notification, or haptic feedback.

[0094] When the embroidery hoop is identified and checked, or when specified by the user, one or more cameras pointing at the sewing table can be used to capture images of the workpiece installed in the embroidery hoop. The entire workpiece can be captured in a single image, or the embroidery hoop can be moved to capture multiple images of the workpiece that are stitched together to form a single image of the entire workpiece. The data collected during the scanning process can be used as input to a neural network that is trained to identify and predict colors. This pre-learned color calibration helps make more accurate color predictions over time as the neural network learns from correct color identifications. The scanning data can also be used as input to a neural network that is trained to detect translational jumps or other motion anomalies, so that the actuation system of the embroidery hoop can be controlled to correct the anomalies.

[0095] When attached to a sewing machine, other accessories can also be recognized, and the sewing machine can provide feedback about whether the accessory is properly installed and whether the machine is configured to operate properly with that accessory. For example, when a user attaches an accessory for attaching a strap to a workpiece to the machine, the sewing machine can display information on the screen related to the accessory to help the user properly use the accessory. The functionality of the sewing machine can also be limited to functions that are compatible with the accessory, unless the user overrides those limitations. The sewing machine can also display information on the screen related to materials that can be used with the accessory and can recommend other accessories to the user.

[0096] Reference is now made to Figures 78-81 , which shows various views and diagrams related to example presser feet, sewing needles, and other components, including features designed to make the presser feet and needles more easily recognized by object recognition techniques, such as by using neural networks or by sensors that include magnetic sensors configured to detect component features. The presser feet, sewing needles, needle plate, or other sewing machine components can include various markings to improve the robustness of optical or other sensor-based object recognition systems. For example, the markings or indicia can include a pattern of two or more geometric shapes Figures 78-81 , colored lines in specific locations Figure 80 , the overall shape of the component (including identifiable protrusions or indentations), paint color and other color treatments, reflective finishes, barcodes, QR codes, and other surface treatments that enable UV, IR, or other optical sensing technologies. The markings can include a unique pattern of etched rings or lines, shapes or patterns etched, recessed, or embossed on the surface of the sewing machine component. The markings can also be composed of different regions of the surface of the sewing machine component that have different reflectivity, i.e., the markings can include a first region with a first surface finish and a second region with a second surface finish. The markings can use electronic recognition technologies, such as near-field communication (NFC) devices and radio-frequency identification (RFID) devices.

[0097] Additional alternative identification can be based on markings with magnetic field line profiles or polarity profiles for each component that can be detected by sensors when the components are installed in the sewing machine. For example, a needle can include a magnet that forms a particular magnetic field that is only detected when the needle is inserted into the needle bar. Similar techniques are applied to embroidery hoops to improve recognition of such hoops by neural networks or other object recognition techniques.

[0098] Reference is now made to Figures 82-84 , which shows various views and diagrams related to providing haptic feedback to a user about the use of a sewing machine. Haptic feedback is feedback that is provided to a user in a way that can be felt. For example, when a particular location is reached, a control component (e.g., a knob, a button, a pedal, a joystick, a slider, etc.) that the user interacts with can be moved in a way that the user can feel, such as by being moved up or down, or by being moved in a circular motion.Figure 83 The components shown) can vibrate slightly, or even create resistance to further movement of the control object. Small vibrations can also be provided through the surface of the sewing machine, on which the user's hands and fingers can rest during use, such as the sewing table. Haptic feedback can be used to alert the user to a particular state of the sewing machine or workpiece, or can be used as further reinforcement to the user that an action taken by the user has been received. For example, the sewing table under the workpiece and the user's hands can vibrate when the user deviates from the desired sewing path. Or, a knob or button can vibrate to indicate that the button is pressed or the knob has reached a particular position. Haptic feedback can also replace mechanical features that provide similar feedback, such as detents in a knob, indicating that a particular position around the knob has been reached. Haptic feedback can be used on any sewing machine surface. Vibratory haptic feedback through piezoelectric sensors can be used on any surface of the sewing machine and can be used in place of mechanical user interfaces. Piezoelectric and capacitive sensors can be arranged in an array under an organic light emitting diode (OLED) or similar type screen that is formed to replace a traditional plastic sewing machine cover. The presence of a user's fingers on or near the OLED interface will engage menus activated based on user requests or current sewing actions and related user interface needs, such as, for example, finger tapping, swiping, and scrolling motions for threading, adjusting thread tension, and activating or deactivating sewing accessories. Other forms of haptic feedback can include force, electrohaptics, ultrasonic, air vortex ring, and thermal haptic feedback.

[0099] Referring now to Figure 83 , a flowchart showing an example case in which haptic feedback can be used is shown. In the scenario shown, a user attaches a presser foot to the machine, which is recognized through neural network processing of visual data received from optical sensors or other sensors of the machine. The user then selects a particular pattern or stitch to be performed. The sewing machine then determines whether the combination of the particular presser foot and the selected operation is a valid combination, i.e., whether the attached presser foot can be used with the particular stitch selected, and provides haptic and other feedback to the user if the combination is not valid. This haptic feedback can be provided at the location of the last action, such as through a touchscreen when the user selects the stitch or other operation to be performed. At the same time, the user can be provided with visual and audible reminders that the presser foot and the selected operation are not compatible, and prompted to install the correct presser foot or select a compatible operation Figure 84 The interface can also provide the user with the option to override the warning.

[0100] Referring now to Figures 85-88, various views and schematics related to using artificial intelligence in a sewing machine to monitor the mechanical and electrical health of the sewing machine are shown. Operating a sewing machine produces a variety of sounds and mechanical vibrations, as well as changes in electrical signals of the motors and actuators driving the sewing machine. An example sewing machine includes sensors to monitor the sounds and noises, mechanical vibrations, and electrical signals to identify patterns related to the performance of the relevant components. Sensors can also be provided on or near various components to measure component temperatures, which can indicate excessive wear if the temperatures rise. As shown in Figure 85 the sensors are arranged at various locations on the sewing machine. The sensors can be continuously active, or can be turned on for a specific time to collect data, for example, during a start-up process, an idle state, an active state, and a shut-down process.

[0101] The collected data can be processed through a neural network that is trained to detect performance issues with the components of the particular sewing machine in question. For example, certain sounds can be related to the friction of two components, which in turn indicates that a bushing or bearing needs to be replaced. Or, when a motor performance is degraded, the voltage required to run the motor at a particular speed can be higher compared to a motor running under nominal conditions. Motor performance can be monitored to determine when issues arise, for example, when performing a certain task or when using a certain fabric or thread material. These situations can also be identified by an increase in heat generated by the machine components and a corresponding rise in temperature of certain components. More importantly, the sensors used by the sewing machine can be significantly more sensitive to changes in the sound or other parameters generated by the sewing machine components, and thus can make predictions earlier than other possible predictions, such as those made by an experienced repair technician. Moreover, these performance issues can be correlated with other information from the sewing machine, for example, the sewing operation being performed when the performance issue is detected and identified. In this way, particular performance issues can be correlated with particular uses of the sewing machine, and information about this relationship can be provided to engineers and repair technicians to better identify the cause of the repair and improve future designs. As with other data collected by the sewing machine and generated by the neural network, the data can be sent to the cloud to be shared with other sewing machines to improve the training of the neural network for all sewing machines in the network.

[0102] Reference is now made to Figure 86, the flowchart shows various ways that diagnostic information can be generated by the sewing machine and used by the user. When the neural network of the sewing machine computer identifies that a correction needs to be made to the machine, an event is logged and the user is notified. The user can then be instructed to perform a specific task to correct the problem, such as removing thread or moving the sewing machine to a harder table. After the corrective action is taken, the sewing machine can be used normally while periodic diagnostics are performed to see if additional corrective action is needed. If the user does not take the corrective action, the motor or other actuator can be calibrated in an attempt to correct the problem. If the calibration does not correct the problem, the user can be notified, the event logged, and a service request sent to a service provider. As a preventative maintenance, calibration of the motor or other components can also be set for every certain number of cycles. Calibration can also be performed when the thread used and the fabric used with it or any job performed by the sewing machine is changed.

[0103] Once a potential problem is diagnosed, the sewing machine can notify the user of the problem in a variety of ways, as described in this disclosure. In particular, the sewing machine can present the user with a reminder via a user interface, a sound, speaking to the user via computerized speech, and / or sending an email to the user via a network connection. For example, as shown in Figure 87 the sewing machine can present the user with an indication that maintenance is needed and prompt the user to schedule a service request with a service dealer. Alternatively, as shown in Figure 88 the sewing machine can suggest a change in the operating environment to improve the performance of the sewing machine, as described in more detail below. When a change to the operating environment or a repair action is deemed necessary or recommended, a reminder and instructional illustrations, animations, and videos can be presented to the user on a touchscreen or through local directed lighting or 2D or 3D static or dynamic light projections, which can be used to guide the user to change the user’s work environment by placing the sewing machine on a harder table to reduce vibration, or to guide the user to perform a simple repair, or to guide a technician to perform a complex machine repair.

[0104] For software problems, an update can be installed automatically so that the user is not aware of the update. Alternatively, the user can be guided through a software update process and a customer service representative can be contacted through the user interface to provide support and corrections for the software problem. Referring now to Figure 99FIG. 6 shows an exemplary flowchart of using a neural network with automatic software updates as described above. When the sewing machine has not been used for a predetermined period of time, i.e., the user is inactive, data can be gathered from the user interaction or activity log, a user-facing camera and microphone, a network interface connected to a software update server, and a clock providing the current date and time. If a software update is available, the neural network provides an indication as to whether the time the user is typically away from the sewing machine during the day is long enough to install the software update before the user returns to the machine. If there is enough time, the software update is allowed to be installed if the user has automatic updates enabled. Similar procedures can be followed to calibrate various sensors, motors, actuators, etc.

[0105] The sewing machine can also include light sources, such as light emitting diode (LED) lights, disposed near various components known to wear out during use, so particular components can be illuminated with a light (e.g., a yellow, orange, or red light) to indicate that the component is degrading in performance and can need repair or replacement. These lights can be activated when the machine is in a maintenance or repair mode and can quickly provide a picture of the overall health of the machine.

[0106] Information related to the health of the sewing machine can be stored in a health log and can be transmitted to a remote customer service representative or repair technician to help the remote worker determine what maintenance, if any, can be needed on the machine and whether the sewing machine needs to be sent to a service center for repair. With the user’s permission, the health data of the sewing machine can also be automatically sent to the distributor, service center, and / or manufacturer so that the data recipients can take proactive measures to order replacement components and notify the customer that particular components of the sewing machine can need to be replaced soon. In a commercial setting, the owner of the sewing machine can opt into a maintenance plan in which such replacement parts are delivered or service calls are automatically scheduled so that the sewing machine maintains a particular uptime.

[0107] Historical data recorded in the health log is particularly useful when diagnosing the cause of a sewing machine failure. For example, historical temperature data can include ambient temperature readings and temperature readings at various points of the machine. The ambient temperature history can show that the sewing machine has been exposed to an overheated environment that damaged the sewing machine. Point temperature readings, i.e., temperature readings at particular locations within the sewing machine, can help a technician determine the root cause of the damage to the sewing machine, such as wear between damaged components. Historical vibration or acceleration data can be similarly used. Acceleration data can also indicate whether the machine experienced a drop or fall, which can be the cause of the damage.

[0108] As described above, the optical sensors can be used in conjunction with a neural network to detect when a user’s finger or other foreign object is in the way of the sewing head and can cause injury to the user or damage to the machine. Similarly, a neural network can be trained to recognize when a user’s finger or other foreign object is in the way of the presser foot, cutting accessory, or any other moving component of the sewing machine that can cause injury to the user during machine use. When a finger or other foreign object is detected, the sewing machine can control the needle and other components to avoid the object, or further sewing can be prohibited if avoidance is not possible or the potential injury is sufficient to warrant prohibiting further operation of the sewing machine. For example, when a finger is detected under the presser foot, the sewing machine can prohibit the presser foot from descending. Or when a finger or the user’s hand is detected in the sewing path, the sewing machine can prohibit further sewing. If the foreign object is detected to be a pin inserted into the seam, the sewing machine can adjust the feed rate or other sewing parameters to avoid the needle hitting the pin.

[0109] The neural network can also take into account the orientation of the sewing machine (through an accelerometer and / or pressure sensors on the base) so that the sewing machine is turned off or prevented from starting if the sewing machine is tilted enough to be turned upside down and potentially injure the user. The accelerometer can also be activated when the sewing machine is in a sleep or standby mode to detect movement of the machine and prohibit power to the machine if the sewing machine is moved, picked up, or knocked over. Heat data from temperature sensors can be input into the neural network so that the machine can automatically shut down to prevent components from overheating or be a sign of an electrical anomaly because of heat buildup.

[0110] User-facing proximity sensors (e.g., infrared sensors) and / or cameras can be used to monitor the presence of a user of the sewing machine so that the sewing machine is automatically turned off after the user is away for a predetermined time to conserve energy. These user-facing sensors can also prevent the activation of the sewing machine after it is determined through a neural network or other means that an unauthorized person is attempting to access the sewing machine. For example, a neural network can be trained to recognize when a child is attempting to access the sewing machine. In response, the computer can prevent the activation of the sewing machine and notify an authorized user of the attempted access by generating an audible sound or sending a notification to the user through an internet connection, text message, or smartphone app. An example flowchart of a child safety feature is shown in FIG. 9. Figure 98The child safety analysis can be triggered for a variety of reasons, such as after a failed attempt to access the sewing machine, or during a long embroidery stitching process when the user can want to leave the sewing machine. Data is then collected from the user-facing camera, microphone, and various user interface elements such as touchscreens, buttons, and knobs. If the neural network determines that a child is attempting to access the sewing machine or is approaching the operational components of the sewing machine, the sewing machine can issue an audible reminder and send a reminder to a mobile device assigned to an authorized user. If the child does not respond to the reminder, the sewing machine can repeat the reminder and stop the sewing operation to prevent injury. If the neural network determines that the unsuccessful attempt was not made by a child, the sewing machine can still issue an audible reminder and send a message to the authorized user. As another example, the sewing machine can periodically monitor the environment around the sewing machine to identify the presence of a person such as a user or a child. This periodic monitoring can be performed, for example, when a long embroidery stitch is being performed, if a child gets too close to the ongoing sewing operation, it can cause damage to the embroidery workpiece or injury to the person. If a child is detected, the stitching operation can be stopped, and a reminder can be sent to the user to inform the user that the sewing operation has stopped and the reason for the stoppage.

[0111] When using the sewing machine, user-set profile settings, user preferences, graphical user interface settings, feedback settings, object recognition preferences, tutorial preferences, and the like are monitored and stored. In addition to machine settings, every interaction between the user and the sewing machine can be recorded and stored. The collection of data related to the interaction between the user and the sewing machine is processed by a neural network such that the sewing machine can learn the user’s preferences to interact with the sewing machine and can predict the user’s preferences in new situations. That is, setting changes can be related to the project, stitch type, thread type, material type, and the like detected by the sewing machine or provided by the user. This data collection enables the sewing machine to assist the user, for example, by suggesting a feed rate setting for a stitch that the user has never sewn before based on the characteristics of the new stitch and the feed rate settings that the user has set for other stitch patterns. As another example, the sewing machine can remind the user of settings that the user typically sets given the current context, that is, by suggesting a particular feed rate or stitch length for a thinner material, and a different feed rate or stitch length for a thicker material. An example workflow for recommending setting changes using a neural network is illustrated in FIG. 9. Figure 97As shown. When settings are changed, data is collected from user interaction logs, current real-time interactions with the user, other sensors, neural networks about sewing materials, and data related to the current sewing operation. If the neural network identifies that the user typically makes the same changes in similar situations, the sewing machine prompts the user to decide whether the default settings should be changed. If the setting changes are not typically made in similar situations, the sewing machine can prompt the user to confirm that the change was intentional. Furthermore, the neural network can identify other settings that might typically be changed in similar environments and suggest these other changes to the user.

[0112] The sewing machine can also suggest that users take breaks or exercise periodically while using the machine to improve their ergonomic health. The suggested times, types of exercise, and rest periods are based on analysis of machine usage by a neural network trained to monitor user health. User posture can also be detected through neural network analysis of data from one or more user-facing cameras, allowing for further customization of exercise suggestions to benefit the user.

[0113] Sewing machines can also detect the conditions of the user's workspace and analyze them using neural networks. Ambient light sensors allow the neural network to consider the lighting conditions of the sewing machine's room and workspace to reduce or minimize contrast between the work area and the room. For example, the sewing machine can suggest brightening the room lights to reduce eye strain caused by the contrast between the sewing machine's bright work surface and the dark room. The sewing machine can also connect to the workspace and room's lighting systems, for example, via a wireless (Wi-Fi) network, to automatically manage brightness adjustments. A user-facing camera can be used to determine the height of the work surface, the position of the user's chair, and other environmental conditions. When the sewing machine can approach an actively controlled surface, such as a height-adjustable workbench, it can suggest and adjust to improve the ergonomics of the work environment. Figure 103 An exemplary flowchart is shown, demonstrating how a sewing machine can reduce user stress by monitoring the environment around the machine. During sewing machine use, data collected from accelerometers, photoelectric sensors, user-facing cameras, and historical logs from previous sessions can be processed by a neural network to identify user health issues. For example, the neural network can identify if a user tends to sit in poor posture and can suggest changes, such as adjusting the user's chair. The neural network can also identify whether the work surface is unstable by monitoring vibration and acceleration data and suggest adjustments to the work surface to make it level and less prone to movement during sewing machine use.

[0114] As mentioned above, data collected by various sensors on the sewing machine and data generated by monitoring the machine's usage can be stored in the machine's database and transmitted to remote servers. Data transmitted to these remote servers can be collected into a central database and used to analyze sewing machine performance and user sewing behavior across a larger dataset. This so-called "big data" analytics can reveal patterns that are undetectable in smaller datasets. The results of this analysis can be fed back into the sewing machine's neural network or a remote neural network, which operates to support the machine's operation, thereby improving the quality of the results determined by the neural network. Big data analytics can also help R&D teams improve factory quality control processes and testing of various components in a laboratory environment. For example, failure modes that might not be predictable during the initial development of the machine can be identified through big data analytics, and these modes can be adapted to change parts and processes in future generations.

[0115] While various inventive aspects, concepts, and features of this disclosure may be described and illustrated in combination in exemplary embodiments, these different aspects, concepts, and features may be used individually or in various combinations and sub-combinations in many alternative embodiments. Unless expressly excluded herein, all such combinations and sub-combinations are within the scope of this application. Furthermore, although various alternative embodiments relating to various aspects, concepts, and features of this disclosure may be described herein, such as alternative materials, structures, configurations, methods, apparatuses and components, alternatives regarding shape, fit, and function, such description is not intended to be a complete or exhaustive list of available alternative embodiments, whether currently known or developed hereafter. Those skilled in the art will readily adopt one or more aspects, concepts, or features of the invention into other embodiments and uses within the scope of this application, even if such embodiments are not expressly disclosed herein.

[0116] Furthermore, although certain features, concepts, or aspects of this disclosure may be described herein as preferred arrangements or methods, such descriptions are not intended to imply that such features are necessary or essential unless explicitly stated otherwise. Additionally, exemplary or representative values ​​and ranges may be included to aid in understanding this application; however, these values ​​and ranges should not be construed as limiting and are only critical values ​​or ranges where such explicit statements are made.

[0117] Furthermore, while various aspects, features, and concepts may be explicitly identified herein as having inventive step or forming part of the disclosure, such identification is not intended to be exclusive, but rather may exist in aspects, concepts, and features that are fully described herein but are not explicitly identified as such or as part of a particular disclosure, which is instead set forth in the appended claims. The description of exemplary methods or processes is not limited to including all steps necessary in all cases, and the order in which steps are presented is not construed as necessary or essential unless explicitly stated otherwise. The words used in the claims have their full ordinary meaning and are not limited in any way by the description of the embodiments in the specification.

Claims

1. A sewing machine, comprising: A sewing table, the sewing table including a feed mechanism and a needle plate, wherein a workpiece can be placed on the sewing table and moved thereon; A sewing head, disposed above a sewing table, comprising: a needle bar extending distally toward the sewing table; a needle attached to the distal end of the needle bar, wherein thread is threaded through the needle; an auxiliary rod extending distally toward the sewing table; and an accessory attached to the distal end of the auxiliary rod; a camera having a field of view including at least a portion of any one of the sewing table, needle plate, workpiece, needle, and accessory, wherein the camera generates camera data signals associated with portions of the sewing table, needle plate, workpiece, needle, and accessory within the camera's field of view; a user interface configured to display information to a user of the sewing machine and receive input from the user of the sewing machine; and an object recognition neural network trained to detect and classify identified objects from the camera data signals as at least one of a needle plate, at least one needle, at least one accessory, workpiece, thread, embroidery hoop, and foreign object, wherein the object recognition neural network generates object detection data signals associated with at least one of the position, orientation, and velocity of the identified object, and the identified object... The system includes an object classification data signal related to the identity of an object, wherein the identified object comprises a first identified object and a second identified object; and a processor configured to: receive from an object recognition neural network an indication of at least one of the position and orientation of the identified object from the object recognition data signal; receive from the object recognition neural network an indication of the identity of the identified object from the object classification data signal; control a user interface to display to a user at least one of the position, orientation, and identity of the identified object; receive from the object recognition neural network a first indication of at least one of the first position, first orientation, and first velocity of the first identified object from the object detection data signal; receive from the object recognition neural network a second indication of at least one of the second position, second orientation, and second velocity of the second identified object from the object detection data signal; and determine, based on the first and second indications, whether a collision will occur or has already occurred between the first identified object and the second identified object.

2. The sewing machine of claim 1, wherein the processor is configured to: determine, based on at least one of the position and orientation of the identified object, whether the identified object is incorrectly installed and whether it is damaged; and control the user interface to display an indication to the user of incorrect installation or damage of the identified object.

3. The sewing machine of claim 2, wherein the processor is configured to alter the operation of the sewing machine while the identified object remains improperly installed or damaged at least once.

4. The sewing machine of claim 1, wherein the processor is configured to determine, based on a first position, a first orientation, a second position, and a second orientation, whether there is incompatibility between a first identified object and a second identified object.

5. The sewing machine of claim 4, wherein the processor is configured to control the user interface based on the determination of existing incompatibilities to display the existing incompatibilities to the user.

6. The sewing machine of claim 4, wherein the processor is configured to modify the operation of the sewing machine based on the determination of existing incompatibilities.

7. The sewing machine of claim 1, wherein the processor is configured to: receive an indication of a first identity of a first identified object from an object recognition neural network based on object classification data signals; The second identification of a second identified object is received from an object classification data signal from an object recognition neural network. And based on the first identity and the second identity, determine whether there is any incompatibility between the first identified object and the second identified object.

8. The sewing machine of claim 7, wherein the processor is configured to control the user interface based on the determination of existing incompatibilities to display the existing incompatibilities to the user.

9. The sewing machine of claim 7, wherein the processor is configured to modify the operation of the sewing machine based on the determination of existing incompatibilities.

10. The sewing machine of claim 7, wherein the processor is configured to: determine an alternative component for at least one of a first identified object and a second identified object based on a first identity, a second identity, and existing incompatibilities, wherein installing the alternative component resolves the existing incompatibilities; and control the user interface to display the alternative component to the user.

11. The sewing machine of claim 1, wherein the processor is configured to change the operation of the sewing machine based on the determination that the first identified object and the second identified object will collide or have already collided.

12. A sewing machine, comprising: A sewing table, the sewing table including a feed mechanism and a needle plate, wherein a workpiece can be placed on the sewing table and moved thereon; A sewing head, disposed above a sewing table, comprising: a needle bar extending distally toward the sewing table; a needle attached to the distal end of the needle bar, wherein thread is threaded through the needle; an auxiliary rod extending distally toward the sewing table; and an accessory attached to the distal end of the auxiliary rod; a camera having a field of view including at least a portion of any one of the sewing table, needle plate, workpiece, needle, and accessory, wherein the camera generates camera data signals relating to portions of the sewing table, needle plate, workpiece, needle, and accessory within the camera's field of view; a user interface configured to display information to a user of the sewing machine and receive input from the user of the sewing machine; and an object recognition neural network trained to detect and classify a first identified object and a second identified object from the camera data signals, wherein object recognition... A neural network generates object detection data signals relating to at least one of a first position, a first orientation, and a first velocity of a first identified object and at least one of a second position, a second orientation, and a second velocity of a second identified object; and a processor configured to: receive from the object recognition neural network a first indication of at least one of the first position, a first orientation, and a first velocity of a first identified object from the object detection data signals; receive from the object recognition neural network a second indication of at least one of the second position, a second orientation, and a second velocity of a second identified object from the object detection data signals; and determine, based on the first indication and the second indication, whether a collision will occur or has already occurred between the first identified object and the second identified object.

13. The sewing machine of claim 12, wherein the processor is configured to change the operation of the sewing machine based on a determination that the first identified object and the second identified object will collide or have already collided.

14. The sewing machine of claim 12, wherein the processor is configured to control the user interface to display the determination to the user based on a determination that the first identified object and the second identified object will collide or have already collided.

15. The sewing machine of claim 12, wherein the processor is configured to determine, based on at least one of a first position, a first orientation, a second position, and a second orientation, whether at least one of a first identified object and a second identified object is incorrectly installed and whether it is damaged.

16. The sewing machine of claim 15, wherein the processor is configured to control a user interface to display to the user an indication of incorrect installation or damage to at least one of the first identified object and the second identified object.

17. The sewing machine of claim 15, wherein the processor is configured to alter the operation of the sewing machine while at least one of the first identified object and the second identified object remains improperly installed and damaged.

18. The sewing machine of claim 12, wherein the object recognition neural network generates object classification data signals related to a first identity of a first identified object and a second identity of a second identified object.

19. The sewing machine of claim 18, wherein the processor is configured to control a user interface to display at least one of a first identity and a second identity to a user.

20. The sewing machine of claim 18, wherein the processor is configured to determine, based on a first identity and a second identity, whether there is incompatibility between the first identified object and the second identified object.

21. The sewing machine of claim 20, wherein the processor is configured to modify the operation of the sewing machine based on the determination of existing incompatibilities.

22. The sewing machine of claim 20, wherein the processor is configured to control the user interface to display the existing incompatibilities to the user based on the determination of existing incompatibilities.

23. The sewing machine of claim 22, wherein the processor is configured to change the operation of the sewing machine based on input received from the user of the sewing machine after an existing incompatibility is displayed via a user interface.

24. The sewing machine of claim 20, wherein the processor is configured to determine an alternative component for at least one of a first identified object and a second identified object based on a first identity, a second identity, and a determination of existing incompatibilities, wherein installing the alternative component resolves the existing incompatibilities.

25. The sewing machine of claim 24, wherein the processor is configured to control the user interface to display alternative components to the user.

26. The sewing machine of claim 12, wherein at least one of the first identified object and the second identified object is classified by an object recognition neural network as at least one of a needle plate, at least one needle, at least one accessory, a workpiece, thread, embroidery hoop, and foreign object.

Images