Interactive Augmented Reality Sewing Machine

Abstract

A system and method for interactive guidance of sewing actions are described herein. Many sewing actions, such as threading media, performing general machine maintenance, and sewing using a pattern, can be difficult and require multiple steps to perform correctly. Through the use of augmented reality (AR), the technology disclosed herein provides interactive guidance to users to perform one or more sewing actions. The execution of these actions further enhances the use of sensor feedback, which can reduce common errors associated with each action.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to and any benefit of U.S. Provisional Application No. 63 / 175,194, filed April 15, 2021, the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure generally relates to a sewing machine configured for use with interactive augmented reality (AR) features. [Background technology]

[0003] Sewing machines may be used to create seams in a single material and to sew various materials together. Certain sewing machines may be used to create seams in a shaped workpiece, cut and sew over the edge of a workpiece, attach decorative elements to a workpiece, cut and sew around the edge of a workpiece, attach decoration or sew an embroidery pattern onto a workpiece mounted in an embroidery frame, or cut a workpiece during the sewing operation. Sewing machines may also cut, fold, roll, or otherwise manipulate a workpiece in addition to or separate from the sewing procedure. The workpiece is moved under a needle so that a seam can be formed in the fabric. The user configures the workpiece and sewing machine for each particular application by adjusting various parameters of the machine and by attaching a variety of different tools or accessories to the machine. This procedure requires the user to configure the workpiece and sewing machine, is often complicated, and if done incorrectly, may cause damage to the workpiece and / or sewing machine. Relying on a manual or similar printed instructions can be tedious for the user and can still result in inaccurate configurations.

[0004] It is therefore appreciated that there is a need for improved guidance regarding workpiece and sewing machine configurations. Summary of the Invention

[0005] In an exemplary embodiment, a method for interactive sewing machine guidance is provided, the method comprising: receiving image data from a camera, the image data including a reference image of at least a portion of a sewing machine; displaying the reference image on a user device, the user device including a display; overlaying a guidance image over the reference image, the guidance image illustrating at least one step of an activity associated with the sewing machine; responding to an indication that the activity has been initiated, receiving sensor feedback from at least one sensor at the sewing machine, the sensor feedback generated based on performance of the activity; and modifying the guidance image in response to the sensor feedback.

[0006] In another exemplary embodiment, a method for interactive sewing machine guidance is provided that includes the steps of initiating an action at a sewing machine in response to an action request, the action including manipulating a workpiece with the sewing machine; receiving image data from a camera, the image data including a reference image of the workpiece; projecting a guidance image onto the workpiece, the guidance image based on the reference image; receiving sensor feedback from at least one sensor at the sewing machine, the sensor feedback generated based on a comparison of the guidance image and the reference image; and determining completeness of the action based on the sensor feedback.

[0007] In another exemplary embodiment, a method for interactive sewing machine guidance is provided, comprising the steps of: initiating an action at a sewing machine in response to an action request, the action including manipulating a workpiece with the sewing machine; receiving image data from a camera, the image data including a reference image of at least a portion of the sewing machine; displaying the reference image at a user device, the user device including a display; overlaying a first guidance image over the reference image, the guidance image illustrating at least one step of an action associated with the sewing machine; receiving sensor feedback from at least one sensor at the sewing machine in response to an indication that the action has been initiated, the sensor feedback generated based on performance of the action, the sensor feedback operable to generate physical feedback; and modifying the guidance image at the user device in response to the sensor feedback.

[0008] In yet another exemplary embodiment, a system for interactive sewing machine guidance is provided, the system comprising a sewing machine and a user device configured to receive image data from a camera, the image data including a reference image of at least a portion of the sewing machine, display the image data on the user device, the user device including a display, overlaying a guidance image on the reference image of the sewing machine, the guidance image illustrating at least one stage of an action, receiving sensor feedback from at least one sensor at the sewing machine in response to initiation of the action at the sewing machine, the sensor feedback generated based on performance of the action, and modifying the guidance image in response to the sensor feedback.

[0009] These and other objects, features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0010] These and other features of the present disclosure will become better understood with regard to the following description and the accompanying drawings.

[0011] [Figure 1] FIG. 1 illustrates an exemplary system for interactive sewing machine guidance.

[0012] [Diagram 2] FIG. 13 illustrates an exemplary flow chart relating to sensor feedback associated with a sewing activity.

[0013] [Diagram 3] FIG. 13 illustrates an exemplary flow chart relating to sensor feedback associated with a sewing activity.

[0014] [Figure 4] FIG. 13 illustrates a flow chart relating to an exemplary method of performing a surger threading action.

[0015] [Diagram 5] FIG. 13 illustrates a flow chart relating to an exemplary method for performing a bobbin changing action.

[0016] [Figure 6] FIG. 1 illustrates a flowchart relating to an exemplary method for performing an embroidery hoop loading activity.

[0017] [Figure 7] FIG. 1 shows a flowchart illustrating an exemplary method for sewing machine guidance.

[0018] [Figure 8] FIG. 1 illustrates an exemplary computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Aspects and implementations of the present disclosure will become more fully understood from the detailed description given below and the accompanying drawings of various aspects and implementations of the present disclosure, which should not be taken as limiting the present disclosure to any particular aspect or implementation, but should be taken as illustrative and understanding only.

[0020] FIG. 1 illustrates an exemplary system 100 for interactive sewing machine guidance according to various embodiments of the present disclosure. The system 100 may include at least a sewing machine 102 and a user device 112. The sewing machine 102 and the user device 112 may communicate data, for example, via a network 120 (e.g., the Internet). In certain embodiments, the sewing machine 102 and the user device 112 may communicate data using one or more data communication protocols (e.g., Bluetooth, RFID, near field communication, etc.). It is understood that the data communication between the sewing machine 102 and the user device 112 may be real-time or near real-time. It is further understood that such data communication allows the sewing machine 102 and the user device 112 to read and / or write data from and / or to local and / or remote memory. In certain embodiments, the sewing machine and the user device 112 are configured to transmit data over the network 120 to be stored on one or more remote servers (e.g., the cloud).

[0021] An exemplary sewing machine (e.g., sewing machine 102) may include a sewing bed or base having a pillar extending upward from one end to support an arm that extends horizontally above the sewing bed. A sewing head is attached to the end of the arm and may include one or more needle bars for moving needles up and down to sew a workpiece on the sewing bed below the sewing head. The sewing bed includes a needle or sewing plate located below the sewing head, which has an opening for one or more needles to pass through when creating or forming a seam in the workpiece. In some sewing machines, a bobbin located below the needle plate assists in seam formation and provides a lower thread that is sewn together with a top thread that is fed through the workpiece from above by a needle. In other sewing machines, such as overlock or serger machines, the lower thread is provided by a looper.

[0022] As used herein, "sewing machine" refers to a device that forms one or more stitches in a workpiece with a reciprocating needle and a length of sewing thread. As used herein, "sewing machine" includes, but is not limited to, sewing machines for forming specific stitches (e.g., sewing machines configured to form lock stitches, chain stitches, buttonhole stitches), embroidery machines, quilting machines, overlock or serger machines, and the like. It should be noted that various embodiments of sewing machines and accessories are disclosed herein, and any combination of these options may be made unless specifically excluded. In other words, individual components or portions of the disclosed devices may be combined unless they are mutually exclusive or otherwise physically impossible.

[0023] "Stitch" means a loop formed with one or more sewing threads, at least one of which passes through a hole formed in the workpiece. The mechanical components of the sewing machine, such as needle, hook, looper, thread tensioning device, feed mechanism, etc., cooperate to form a stitch in one or more pieces of the workpiece. One iteration of this complex machine movement may form one stitch or a pattern of stitches in the workpiece. The "stitch length" of a repeat or pattern refers to the distance the workpiece is moved as the repeat is performed. Stitch length measurements vary for different types of repeats and patterns and may encompass one or more stitches in the workpiece.

[0024] A pressure bar having a presser foot also extends downward from the sewing head to press the workpiece against the sewing bed and against feed dogs that move back to front and optionally side to side to move the workpiece. The feed dogs move the workpiece in coordination with the presser foot and at a speed that may be fixed or variably controlled by the user, such as with a foot pedal. A wide variety of presser feet and other types of accessories, such as a buttonhole foot, may be attached to the pressure bar to assist in forming certain types of stitches or features on the workpiece. Accessory mounts may also extend below the sewing head to hold special tools or accessories on or above the sewing bed. For example, as illustrated in FIG. 1, the sewing machine 102 includes an operable embroidery frame 118 operable to enable manipulation of a workpiece attached to the embroidery frame 118 using an X-unit 120 and a Y-unit 122. The Y-unit 122 also includes a Y-slide 124.

[0025] The speed or frequency at which the needle bar is moved up and down is controlled by the user as described above. The needle bar typically moves up and down in a circular motion to form a seam in the workpiece, but the needle bar may also move side to side to form different seams, such as zigzag seams or tapered seams, or to vary the width of the seam. The type and pitch of stitches performed by the machine may be selected by the user via a manual interface including buttons, knobs, levers, and the like. In some embodiments, the type and pitch of stitches performed (or similar settings) may be controllable by the user via input to a user interface, such as, for example, a user interface provided on the display 114 and / or display 108 of the user device 112. In other embodiments, the settings may be controllable via voice command input to a voice control interface.

[0026] Different types of sewing machines may include additional components for forming stitches on or otherwise manipulating the workpiece during the sewing process. For example, in a serger, one type of sewing machine that may be used to form the edges of a workpiece, among other functions, a needle called a looper operates under the sewing bed to feed a bobbin thread to form various stitches. Sergers may also include one, two, or more needles above the needle plate and may include a knife for cutting the edges of the workpiece. Sewing machines may also be used to generate embroidery patterns on a workpiece by including a holder for an embroidery frame (e.g., embroidery frame 118) on the sewing bed. The embroidery frame holder may be operated in at least two axes so that the sewing machine's controller can move the embroidery frame so that the needle can trace the embroidery pattern on the workpiece. Sergers generally have multiple complex threading paths. The effectiveness of written or verbal instructions is often deemed inadequate. Thus, threading a serger is commonly viewed by users as a difficult task, and many users will pay a technician to thread their machine, a time-consuming and costly task, rather than performing the task themselves.

[0027] The thread used during sewing is held at a location on the sewing machine, such as within a bobbin or on a spool held by a spool holder that is part of or extends above the arm of the sewing machine. The thread is directed from a sewing thread source (e.g., a bobbin or spool) to one or more needles of the sewing machine through various different elements of the sewing machine arranged to change the direction of the thread so that the thread is smoothly drawn out and fed to the workpiece with as little damage as possible to the thread. The tension of the sewing thread may also be changed by various tensioning devices located along the sewing thread path or within the sewing thread source. These tensioning devices ensure that only the desired amount of thread is let out and that the thread that will form the seam in the workpiece is properly tensioned. A loose sewing thread may allow the seam to unravel and a tight sewing thread may cause the seam to form incorrectly. The sewing thread tension for the top and bottom threads may also be adjusted to ensure that the top and bottom tensions are balanced so that the seam is properly formed along the desired sewing path in the workpiece. It will be appreciated that the aforementioned features and functions of the exemplary sewing machine may be operatively connected to one or more sensors (e.g., sensor 110) to generate sensor feedback. For example, the tension of the sewing thread may be recorded by a tension sensor. If the thread is too loose or too tight, the tension sensor may generate sensor feedback indicating that the tension in the sewing thread is outside of an acceptable threshold.

[0028] The sewing machine 102 may further include at least one camera 104, projector 106, display 108, and / or sensor 110. Such features as previously described may be provided on a device separate from the sewing machine 102 and may be applicable for use on the sewing machine. For example, the projector, display, and other user interfaces such as speech recognition and their hardware components may be included on the sewing machine or on a separate user device, such as a smartphone, tablet, goggles or glasses. The hardware may include microphones, speakers, piezoelectric tactile sensors, or the like. In certain embodiments, mirrors may be used to reflect and direct the projected image onto the workpiece. The camera 104 may be configured to capture image data associated with the workpiece and / or the sewing machine itself. In certain embodiments, the camera 104 may be used in conjunction with the sensor 110 to generate status information regarding the sewing machine 102 or the workpiece. In certain other embodiments, the camera 104 may capture images, e.g., of the workpiece, of one or more components of the sewing machine 102, and / or of the user. In certain embodiments, the projector 106 may be used to project a guidance image onto the workpiece. In certain other embodiments, the projector 106 may be used to project a guidance image onto a portion of and / or in the immediate vicinity of the sewing machine 102 (e.g., on the workpiece) to facilitate completing maintenance and / or media threading related activities. It is understood that the sewing machine 102 may further include one or more processors and memories to facilitate communication between its described components, the user device 112, and the user. More specifically, the sewing machine 102 has at least a processor and a memory that stores instructions that, when executed by the processor, cause the processor to perform functions set forth by the instructions (e.g., to control various aspects of the sewing machine 102, as described in detail herein).Processing may also occur separately or in combination with a processor separate from the sewing machine, for example via a smartphone or cloud computing.

[0029] In some embodiments, the display 108 associated with the sewing machine 102 is configured to utilize an exemplary user interface that may be visually presented to the user via one or more displays (e.g., display 108), including a touchscreen display that includes a touch-sensitive overlay that detects the location of a user's fingers touching the display. The user may then interact with the user interface by directly touching the screen at a particular location, or by performing touch gestures, such as touch, touch-and-hold, pinch or spread, and touch-and-move gestures. The presence, location, and movement of the user's hands, fingers, or eyes or facial expressions may also be detected via analysis of data from optical sensors (e.g., camera 106) or near-field sensors via sound, light, infrared radiation, or electromagnetic field disturbances. The graphical user interface may also be projected by one or more projectors (e.g., projector 104) of the sewing machine onto an adjacent surface, such as a sewing bed, a workpiece, a wall or table, or any other suitable surface. Alternatively, the sewing machine may operate without a graphical user interface, via voice commands and audible feedback in the form of certain sounds and / or computerized voices. Tactile feedback is also provided via actuators that, when activated, otherwise vibrate various parts of the machine via phototactile substitution laws, where a digital micromirror device projector emitting time-modulated spatial light, coupled with a finger or tool fitted with phototransistors and actuators, generates special tactile feedback in response to a wide variety of conditions of the workpiece, machine, etc. Audible and tactile interaction with the sewing machine is particularly useful for users who are visually impaired.

[0030] In an exemplary embodiment, the sewing machine 102 includes one or more sensors 110. The sensors may measure various aspects associated with the sewing machine 102, the user device 112, and / or the user, including, but not limited to, light, tension, pressure, hearing, touch, acoustic, sound, vibration, chemical, biometrics, sweat, respiration, fatigue detection, gas, smoke, retina, fingerprint, fluid velocity, speed, temperature, infrared light, ambient light, color, RGB color (or another color space sensor such as one using CMYK or grayscale color spaces), contact, tilt, motion, metal detection, magnetic field, humidity, moisture, imagination, photons, force density, proximity, ultrasound, load cell, digital accelerometer, motion, translation, friction, compressibility, voice, microphone, voltage, barometer, gyroscope, Hall effect, magnetometer, voltage, barometer, GPS, electrical resistance, tension, strain, and many others.

[0031] In certain embodiments, the sensor 110 may provide real-time or near real-time feedback to the user and / or user device, for example, by tactile sensors / feedback, and it is understood that incorrect movement in accordance with a desired action may trigger a tactile response in the sewing machine 102 so that the user is immediately aware that the action has been performed incorrectly. This immediate feedback may include feedback via projected images and / or in conjunction with vibrations and sounds representing negative confirmation. Positive reinforcement scenarios may also be applied in a similar manner. In some embodiments, the sensor feedback may trigger a response in one or more user devices. For example, when performing a sewing action using a pattern, misalignment of the pattern and workpiece may trigger a sensor feedback response in a smartwatch associated with the user. This immediate feedback may alert the user before the misalignment results in an undesired output. In an example, tactile feedback through vibration is emitted to the user's finger as the user's finger is positioned on the fabric or workpiece to be sewn, thereby informing the user of the appropriate point in time at which the user must move the fabric to achieve the desired sewing result. This instruction may be accompanied by or in lieu of auditory instructions and instruction arrows projected onto the fabric.

[0032] The user device 112 may be a handheld computing device such as a smartphone, tablet computer, etc. The user device 112 includes at least a display 114 and a camera 116. It is understood that additional user devices having similar components are contemplated for use with the described systems and methods, including, but not limited to, smart glasses, head-up displays (HUDs), smart watches, etc. It is understood that in certain embodiments, the user device 112 may be combined with aspects of multiple devices, such as, for example, a smartphone and a smart watch. It is understood that the user device 112 may be configured to provide feedback to a user of the user device 112 based on its interaction with the sewing machine 102 and / or sensor feedback therefrom. The camera 116 is configured to capture image data associated with the sewing machine 102 and the surrounding environment. As used herein, such data may refer to reference image data. Reference image data is a real-time or near real-time capture of the sewing machine 102 or its surrounding environment, including, for example, a workpiece. It is understood that the image data includes an image or a series of images captured by the camera 116 according to a frame rate associated with the camera 116. The reference image data is displayed to the user via the display 112. In accordance with numerous embodiments disclosed herein, the reference image may be used in conjunction with a guidance image to facilitate interactive sewing machine guidance. In some embodiments, the user device 112 may be attached to or otherwise operatively connected to the sewing machine 102. This configuration may be useful when a user requires two hands to perform an action.

[0033] FIG. 2 illustrates an exemplary method 200 for interactive sewing machine guidance according to the present disclosure. The method begins at step 202, where a user requests information about an action related to a sewing machine (e.g., sewing machine 102). An action can be an action for locating a feature or for performing an operation on the sewing machine. Exemplary actions are described in further detail below. In some embodiments, the interactive guidance methodology described herein facilitates guidance of an action by displaying a digital image overlay on a reference image of the sewing machine, e.g., external and / or internal components. For example, a user device may generate and display a digital image (and / or a series of images / videos) that may animate the movement of several components of the sewing machine to perform the action. The images are then overlaid on a reference image of the actual sewing machine. In certain embodiments, this reference image is captured by a camera associated with the user device. An exemplary action that may utilize image overlay interactive guidance is a threading action, e.g., threading a sewing machine or serger.

[0034] More exemplary actions that may utilize image overlay interactive guidance are actions associated with locating and identifying sewing machine components and / or accessories. As a further example, image overlay may be used to show the location and names of critical parts of the sewing machine, the location and function of the user interface associated with the sewing machine, and the location and function of all other components of the sewing machine. Image overlay may also be used to identify unknown components or accessories. For example, using a reference image, the user device 112 may recognize a component or accessory based on its size, shape, or location on the sewing machine. In another example, image overlay may provide interactive guidance for changing accessories, such as, but not limited to, a needle, a presser foot, a sewing plate, or a bobbin. Image overlay may also be used for interactive guidance for using and fixing large accessories. For example, stabilizing a fabric on an embroidery frame, fixing an embroidery frame on a sewing machine. Image overlay may further assist in interactive guidance for certain maintenance tasks, such as cleaning or lubricating some parts (simple), or repairing some components (complex).

[0035] In some embodiments, the interactive guidance methodology as described herein facilitates guidance of actions by projection of images onto the sewing bed. Such actions may be difficult to use with a user device (such as actions requiring two hands) or with those actions requiring special attention to the workpiece on the sewing bed area. One or more short throw projectors (e.g., projector 104) may generate 2D or 3D projections onto the sewing bed to guide the performance of the actions. In some embodiments, aerial visual projections may be generated via projection or hologram technology. In certain embodiments, additional accessories such as smart glasses may be used to interpret the visual projections. Projections onto the sewing bed may facilitate viewing and selection of different stitch and embroidery options on the fabric, different decoration options on the fabric, or available sewing techniques appropriate for the fabric. Projections may also be used to perform complex stitch or embroidery actions, such as understanding when and how to rotate and / or translate the fabric, when and how to add textile decorations, and when and how to use special sewing accessories. Further still, the projections may assist in performing basic operations in the sewing domain, such as, for example, when and how to change needles, how to thread one or more needles, how to attach accessories, when and how to change presser feet, and when and how to lift presser feet.

[0036] It is understood that the exemplary acts as described above are for example purposes only, and it is further understood that each act may be adapted for use with image overlay and / or image projection within the scope of the present disclosure.

[0037] 2, in some embodiments, a method 200 for interactive sewing machine guidance may begin at stage 202 when an action is automatically recognized by the sewing machine 102 (e.g., via AI, neural networks, and / or algorithms coupled with sensor data input triggers), or the user device 112 (e.g., via recognition algorithms coupled with mapping, tracking, and optical sensor data), or otherwise specifically requested by a user. In certain embodiments, the action may be automatically recognized by the sewing machine 102 and / or the user device 112 based on operational feedback of the sewing machine (e.g., pedal activation, engagement with threading components, etc.), or may be recognized based on observed optical data from a camera 104 associated with the sewing machine 102 and / or from a camera 116 associated with the user device 112.

[0038] After an action is selected, interactive guidance may be provided, for example, using augmented reality (AR). As used herein, AR encompasses a virtual image overlay for the purpose of guiding a user to complete an action. In certain embodiments, the AR image overlay may be accomplished by displaying a guidance image on a reference image in a display (e.g., display 114 and / or display 108). For example, the user device 112 may display an image of the sewing machine 102 with the overlaid guidance image. In certain other embodiments, the guidance image may be physically projected, for example, by projector 104, onto a surface such as a sewing bed. The guidance image may inform the user how to perform a certain task associated with their selected action. Various types of image overlays are contemplated herein and are illustrated by steps 206a-e. In step 206a, "markerless AR" includes a virtual image overlaid on a reference image. The overlaid image may be fully or partially augmented, meaning, for example, that in certain embodiments, the overlaid image may reconstruct aspects of the reference image rather than displaying the reference image itself. For example, in some embodiments, rather than overlaying an arrow pointing to the bobbin to be replaced (e.g., a partial overlay), the overlaid guidance image may generate a digital version of the bobbin and animate the bobbin to show some of the actions previously accomplished by the arrow. Another example may be a digital version of a button or embellishment to be sewn onto the fabric to help the user place the object to show how the sewing process should occur. Yet another example may be a digital version of a finger holding a thread and showing how a person should place the thread into various paths on the serger. This facilitates an interactive experience, where the user's own hand attempts to follow the guidance of the virtual hand and thread to achieve the correct threading of the machine. In a similar manner, other actions facilitate interactive experiences.It is further understood that while the guidance image is displayed, it may be modified based on feedback from the sewing machine 102 and / or the user device 112. In embodiments in which the user device provides feedback, such feedback data may include additional camera data, accelerometer data, digital compass data, GPS assisted calculations, etc. For example, if a user selects to rotate or translate the 3D overlaid image of the applique over the sewing workpiece and selects an impossible position for sewing the applique on the workpiece, the feedback vectors may modify the guidance image, which may be modified to guide the user to an acceptable positioning of the applique for sewing onto the workpiece.

[0039] In step 206b, a "location-based AR" may be selected. The "location-based AR" includes a guidance image that is fully or partially overlaid on a reference image. In some embodiments, the "location-based AR" may utilize GPS, digital compass, speedometer, or accelerometer data from the user device 112 to calculate a precise position and aid in the image overlay.

[0040] In stage 206c, "Simultaneous Localization and Mapping (SLAM)" may be selected, which involves overlaying a guidance image on top of the reference image and modifying the guidance image based on a set of algorithms configured to solve the task of simultaneous localization and mapping by locating the sensor relative to the surroundings. For example, a user may rotate the workpiece fabric while sewing, and the simultaneous localization and mapping of the reference image will identify or follow the pattern or topography of the fabric, ensuring that any associated overlaid guidance image for a newly occurring stitch on the fabric correctly follows the fabric as translation occurs.

[0041] In certain embodiments, sewing machine 102 may have one or more identifying features that enable user device 112 and camera 116 to calibrate the position of the user device so that improved guidance images can be generated by the user device. Such features may be used in stage 206d, where “recognition-based AR” may be selected. By “recognition-based AR,” image data from camera 106 and / or camera 116 may be used to identify visual markers, such as, for example, unique features, QR / 2D codes, natural feature tracking (NFT) markers, objects, directions, etc., and use the visual markers to affect the guidance image.

[0042] In step 206e, "projection-based AR" may be selected, which includes images projected onto a surface and / or onto the environment surrounding the sewing machine 102. With this type of image guidance, visual mappings, including 2D and / or 3D static and / or dynamic / animated images, may be projected onto the sewing bed area or other relevant area using light (e.g., projector 104) in figures and shapes that instruct or guide the user in performing an action. In certain embodiments, the projected light may come from the user device 112 or another external light source. In some embodiments, "projection-based AR" may be used to draw contours along a workpiece to guide the user to sew along a line, thread a serger, or other action.

[0043] It is understood that the types of image overlays discussed with reference to steps 206a-e are presented for illustrative purposes only, and that additional types of image overlays may be apparent from the detailed description herein. In addition, it is contemplated that the exemplary image overlay types may be combined together during the exemplary interactive sewing machine guidance.

[0044] At step 208, feedback is collected and may affect the guidance image generated using one or more of the exemplary guidance image types listed above. In certain embodiments, the type of guidance image used may change based on the feedback. When the feedback is taken into account, method 200 returns to step 204 using feedback loop 210. Feedback loop 210 may be utilized an unlimited number of times or may be utilized until the action is perfected. The continuous collection of feedback and the continuous modification of the guidance image according to the feedback allows the user to diagnose and correct potential problems in their action, for example, whether they have threaded the serger incorrectly. With real-world mechanisms, if a user threads the sewing machine incorrectly, the problem may not be realized until a sewing-related action has begun, thus breaking the thread, causing a jam in the sewing machine, and making correct sewing impossible until removal and rethreading of the damaged thread, a time-consuming task, is properly completed. It is further understood that it is an aspect of the presently described method and system that the improved guidance image may serve to inform the user that an action is not being performed correctly according to the feedback.

[0045] FIG. 3 illustrates an exemplary method 300 relating to sensor feedback in relation to a sewing action. In step 302, a user selects an action. In step 304, the connected device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or related environment. In step 306, the user follows the guidance provided by the guidance image and proceeds with the execution of the desired action. In step 308a, data collection associated with the execution of the action (e.g., image data collected by the camera 116) is collected at the user device. In step 308b, data collection associated with the execution of the action (e.g., data collected from the sensors 110, such as light, tension, and pressure sensors) is collected at the sewing machine. In step 308c, data collection associated with the execution of the action is collected at the user device and the sewing machine. In step 310, local or external data processing is performed. In certain embodiments, data processing may be performed at the user device 112, the sewing machine 102, and / or an external source (e.g., the cloud). In step 312a, the data (e.g., light, tension, and / or pressure data) is analyzed to ascertain whether the measurable data attributes constitute acceptable values. The analysis may result in a reference image comparison via a simple input image, or may be combined with object detection and tracking sensors (i.e., optical, capacitive, photoelectric, laser, proximity, infrared, optical time-of-flight, mechanical, etc.). Alternatively, the analysis may be performed via more sophisticated artificial intelligence methods, where data is processed via one or more neural networks and multi-sensor data inputs. For example, data may be received and analyzed in a neural network layer, and a prediction of the recognition of the object, its location, and whether the movement is correct or incorrect may be analyzed in a pre-trained or in-training network designed to recognize the situation and user actions. A response may then be triggered based on the aforementioned steps. The accurate prediction may be based on a scoring system.For example, an accuracy prediction of whether a user action is correct or incorrect that is greater than a certain percentage or threshold may be considered acceptable based on the final form collected in-data. In step 312b, the data enters an input layer in the neural network, and the output layer provides an assessment of the completeness of the action. It is understood that all forms of data available to the sewing machine may be processed through the neural network, i.e., data from sensors, data from the software, data from a data storage device, data from user input from the software, and the like. An exemplary neural network for use with the interactive sewing machine guidance technology described herein is described in further detail in U.S. Patent Application No. 17 / 352,035, entitled Sewing Machine and Method of Use Thereof, the subject matter of which is incorporated herein by reference in its entirety. The information to be processed first encounters an input layer of the neural network, which performs initial processing of the input data and outputs the results to one or more hidden layers that process the output values ​​from the input layer. The information processed through the hidden layers is presented in the output layer as a probability of confidence in a given result, e.g., the location of a detected object in an image and the classification of that object. Software in the sewing machine's computer may receive information from one of the layers of the neural network and take action accordingly, such as to adjust the sewing machine's parameters and / or to notify a user based on the results of the neural network processing.

[0046] During training of the neural network, the node parameters (i.e., at least one of the input parameters, function parameters, and output parameters) for each node of the neural network are adjusted via a back-propagation algorithm until the output of the neural network corresponds to the desired output for a set of input data. During neural network processing, the neural network begins the training process with node parameters that may be randomized or transferred from an existing neural network. The neural network is then presented with data from a sensor to process. For example, an object may be presented to an optical sensor to provide visual data to the neural network. The data is processed by the neural network and the output is tested such that the node parameters of the various nodes of the neural network may be updated to increase the confidence probability of the detection and classification performed by the neural network. For example, when a presser foot is presented to the optical sensor for identification by the neural network, the neural network presents a confidence probability that the object presented to the optical sensor is located within the coordinates of the image and can be classified as a particular presser foot. During the training process, the node parameters of the nodes of the neural network are adjusted such that the neural network's confidence that a particular answer is correct increases. Then, when presented with certain visual data, the neural network "understands" that a certain answer is the most correct answer, even if that data is not exactly the same as what it "saw" before.

[0047] A neural network is considered "trained" when the decisions made by the network reach a desired level of accuracy. A trained neural network may be characterized by a set of node parameters that were adjusted during the training process. The set of node parameters may be transmitted to other neural networks with the same node structure, so that those other neural networks process data in the same manner as the originally trained network. Thus, the neural network stored in the data storage device of a particular sewing machine may be updated by downloading new node parameters. It should be noted that the node parameters of a neural network, such as input weight parameters and thresholds, tend to occupy significantly less storage space than the image library used for comparison with images or visual data collected by an optical sensor. As a result, the neural network's files and other important files may be updated quickly and efficiently over the network. For example, the structure of the neural network, i.e., the map of connections between the nodes and the activation functions calculated at each node, may also be updated in this manner.

[0048] The neural network may also be trained continuously such that the node parameters are periodically updated based on feedback provided from various data sources. For example, the locally or externally stored neural network node parameters may be periodically updated based on data collected from sensors that may or may not match the neural network output. These adjusted node parameters may also be uploaded to a cloud-based system and shared with other sewing machines such that all of the sewing machine's neural networks are improved over time.

[0049] In certain embodiments, data collected by the sewing machine's sensors may be processed locally at the sewing machine or by an external processor in a cloud-based neural network. The locally stored neural network may be pre-trained or may be a continuously updating neural network. The data processed by the neural network (locally or remotely) is then used by the sewing machine's software to make decisions that result in machine and / or user interaction.

[0050] It is understood that various types of feedback mechanisms are contemplated herein. It is understood that the feedback can trigger a feedback response in the user device and / or the sewing machine. For example, the feedback response can be relayed to the user via visual, audible, and tactile means. For example, an indication that an incorrect accessory is installed on the machine can be presented to the user via a user interface on the sewing machine's display, while a notification sound, such as a beep or computerized sound, is transmitted to the user via a speaker in the sewing machine and / or the user device. Notification can also be transmitted to the user via haptic or tactile feedback, via vibration of a part of the machine that the user is in contact with. That is, the sewing bed can be vibrated by the sewing machine to provide an alert to the user that the machine is not properly configured for the particular sewing operation selected by the user. The user feels the vibration under his fingers in contact with the workpiece and the sewing bed, which prompts the user to look at the display for further information. The illumination lights of the sewing machine may also be controlled to alert the user when an incorrect accessory is installed, such as by illuminating an area on the machine, changing color or flashing to capture the user's attention, etc., to prompt the user to look at the display for additional information. It is understood that the neural networks described above can be readily adapted to make predictive assessments of the completeness and / or accuracy of a given action based on sensor data. It is further understood that the exemplary neural networks may be used throughout the methods and systems described herein to analyze, interpret, and utilize feedback data (e.g., from the sewing machine 102, the user device 112, and / or from external sources).

[0051] Returning to Figure 3, at step 314, it is determined whether the action is complete (e.g., using the metrics provided by steps 312a and / or 312b). If the action is determined to be incomplete, method 300 returns to 304 and repeats all or part of the action until the action is determined to be complete at step 314. Once the action is determined to be complete, the method ends at step 318. In certain embodiments, certain validation procedures may be performed to evaluate the results of the action, which may require the action to be repeated or, alternatively, additional actions to be performed in such embodiments.

[0052] FIG. 4 illustrates an exemplary method 400 relating to sensor feedback in connection with the sewing activity of threading a serger. In step 402, a user selects an activity related to threading a serger. In step 404, the connected device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or related environment that instructs the user how to thread the serger. In step 406, the user follows the guidance provided by the guidance image and proceeds with the performance of the desired activity. In step 408a, data collection associated with the performance of the activity (e.g., image data collected by camera 116) is collected at the user device. In step 408b, data collection associated with the performance of the activity (e.g., data collected from sensors 110 such as light, tension, and final pressure sensors) is collected at the sewing machine. In step 408c, data collection associated with the performance of the activity is collected at the user device and the sewing machine. In step 410, local or external data processing is performed. In certain embodiments, data processing may be performed at the user device 412, the sewing machine 102, and / or an external source (e.g., the cloud). In step 412a, the data is analyzed to determine whether measurable data attributes (e.g., light, tension, and / or pressure values) constitute acceptable values. In step 412b, the data enters an input layer in a neural network, and an output layer provides a predictive assessment of the accuracy of the completeness of the action.

[0053] At step 414, it is determined whether the serger threading action is complete (e.g., using the metrics provided by steps 412a and / or 412b). If the action is determined to be not complete, method 400 returns to 404 and repeats all or part of the action until the action is determined to be complete at step 414. Once the action is determined to be complete, the method ends at step 418. In certain embodiments, certain verification procedures may be performed to evaluate the results of the action, which in such embodiments may require the action to be repeated or, alternatively, additional actions to be performed. For example, in a serger threading action, multiple threads and thread paths may be present on the serger requiring ordered verification of the correct threads and paths, which may be performed after the action is complete.

[0054] FIG. 5 illustrates an exemplary method 500 relating to sensor feedback in connection with a sewing action related to changing a bobbin. In step 502, a user selects an action related to changing a bobbin. In step 504, the connected device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or related environment. In step 506, the user follows the guidance provided by the guidance image and proceeds with the execution of the desired action. In step 508a, data collection associated with the execution of the action (e.g., image data collected by the camera 116) is collected at the user device. In step 508b, data collection associated with the execution of the action (e.g., data collected from the sensors 110, such as light, tension, and final pressure sensors) is collected at the sewing machine. In step 508c, data collection associated with the execution of the action is collected at the user device and the sewing machine. In step 510, local or external data processing is performed. In certain embodiments, data processing may be performed at the user device 112, the sewing machine 102, and / or an external source (e.g., the cloud). In step 512a, the data (e.g., light, tension, and / or pressure values) are analyzed to see if measurable data attributes constitute acceptable values. In step 512b, the data enters an input layer in the neural network, and the output layer provides an assessment of the completeness of the action. In step 514, it is determined whether the action is complete (e.g., using metrics provided by steps 512a and / or 512b). If the action is determined to be incomplete, the method 500 returns to 504 and repeats all or part of the action until the action is determined to be complete in step 514. Once the action is determined to be complete, the method ends in step 518. In certain embodiments, certain validation procedures may be performed to evaluate the results of the action, which in such embodiments may require the action to be repeated or, alternatively, additional actions to be performed.For example, in the above example act, the user would need to ensure that the thread unrolls from the bobbin in the correct direction, otherwise the needle may break or the thread tension would be incorrect.

[0055] FIG. 6 illustrates an exemplary method 600 relating to sensor feedback in relation to a sewing action of how to attach and connect an embroidery frame. In step 602, a user selects the action of attaching and connecting an embroidery frame. In step 604, the connection device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or related environment. In step 606, the user follows the guidance provided by the guidance image and proceeds with the execution of the desired action. In step 608a, data collection associated with the execution of the action (e.g., image data collected by the camera 116) is collected at the user device. In step 608b, data collection associated with the execution of the action (e.g., data collected from the sensors 110, such as light, tension, and final pressure sensors) is collected at the sewing machine. In step 608c, data collection associated with the execution of the action is collected at the user device and the sewing machine. In step 610, local or external data processing is performed. In certain embodiments, data processing may be performed at the user device 112, the sewing machine 102, and / or an external source (e.g., the cloud). In step 612a, the data is processed by an algorithm to see if measurable data attributes constitute acceptable values, such as light, tension, and / or pressure values. In step 612b, the data enters an input layer in the neural network, and the output layer provides an assessment of the completeness of the action. In step 614, it is determined whether the action is complete (e.g., using the metrics provided by steps 612a and / or 612b). If the action is determined to be incomplete, the method 600 returns to 604 and repeats all or part of the action until the action is determined to be complete in step 614. Once the action is determined to be complete, the method ends in step 618. In certain embodiments, certain validation procedures may be performed to evaluate the results of the action, which in such embodiments may require the action to be repeated or, alternatively, additional actions to be performed.

[0056] 7 illustrates a flow chart of an exemplary method for interactive sewing machine guidance. It will be understood that the illustrated method and associated steps may be performed in a different order, by omitting illustrated steps, by adding additional steps, or by combinations of reordered, combined, omitted, or additional steps.

[0057] The method begins at step 702 where a reference image is received. A guidance image is overlaid on the reference image at step 704. At step 706, an action is initiated. As described herein, the action may be any action associated with operation and / or maintenance of the sewing machine. At step 708, action feedback is analyzed, for example, by the user device 112. It is understood that in certain embodiments, the action feedback may also be analyzed by the sewing machine 102, an external source, and / or some combination of the user device 112, the sewing machine 102, and an external source. At step 710, the guidance image is modified based on the action feedback. It is understood that modifications to the guidance image may be directed to facilitating the completion of the action. For example, an exemplary modification may be to alert the user that the sewing machine has been incorrectly loaded with thread.

[0058] Referring now to FIG. 8, a high level diagram of an example computing device 800 that may be used in accordance with the systems and methodologies disclosed herein is illustrated. It is understood that the user device 112 is one exemplary implementation of the computing device 800. It is further understood that the sewing machine 102 may further include elements of the computing device 800. The computing device 800 may be used in a system for interactive sewing machine guidance (e.g., the system 100). The computing device 800 includes at least one processor 802 that executes instructions stored in a memory 804. The instructions may be, for example, instructions for implementing a function described as being performed by one or more components discussed above, or instructions for implementing one or more of the methods described above. The processor 802 may access the memory 804 by a system bus 806.

[0059] Computing device 800 additionally includes a data storage 808 accessible by processor 802 via system bus 806. Data storage 808 may include executable instructions related to performing the systems and methods disclosed herein. Computing device 800 also includes an input interface 810 that allows external devices to communicate with computing device 800. For example, input interface 810 may be used to receive instructions from a computer device, from a user, etc. Computing device 800 also includes an output interface 812 that interfaces computing device 800 with one or more external devices. For example, computing device 800 may display text, images, etc. via output interface 812.

[0060] It is contemplated that external devices communicating with computing device 800 via input interface 810 and output interface 812 are included in an environment that ultimately provides any type of user interface with which a user may interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface may accept input from a user using an input device such as a keyboard, mouse, remote control, and the like, and may provide output on an output device such as a display. Furthermore, a natural user interface may allow a user to interact with computing device 800 in a manner free of the constraints imposed by input devices such as a keyboard, mouse, remote control, and the like. Rather, a natural user interface may rely on speech recognition, touch, and stylus recognition, gesture recognition both on-screen and near-screen, in-air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and the like.

[0061] In addition, although illustrated as a single system, it should be understood that computing device 800 may be a distributed system such that, for example, several devices may be in communication over network connections and collectively perform the tasks described as being performed by computing device 800.

[0062] It should be understood that the exemplary systems described herein (e.g., sewing machine 102, user device 112, etc.) may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. A computer-readable medium includes a computer-readable storage medium. A computer-readable storage medium may be any available storage medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to hold or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disk usually reproduces data magnetically and disc usually reproduces data optically by laser. Furthermore, propagated signals are not included within the scope of computer-readable storage media. Computer-readable media also includes communication media, including any medium that facilitates the transfer of a computer program from one place to another. A connection may be, for example, a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.

[0063] Alternatively, or in addition, the functionality described herein may be performed, at least in part, by one or more hardware logic components. By way of example, and not by way of limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), composite programmable logic devices (CPLDs), and the like.

[0064] It should be understood that the detailed description is intended to be illustrative and not limiting of the described embodiments. Other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. In addition, in some instances, elements described with one embodiment may be readily adapted for use with other embodiments. Thus, the methods and systems described herein are not limited to the specific details, exemplary embodiments, or illustrative examples shown and described. Thus, departures may be made from such details without departing from the spirit and scope of the general aspects of the present disclosure. (Other possible items) (Item 1) receiving image data from a camera, said image data including a reference image of at least a portion of a sewing machine; displaying the reference image at a user device, the user device including a display; overlaying a guidance image over said reference image, said guidance image illustrating at least one step of an action associated with said sewing machine; receiving sensor feedback from at least one sensor in the sewing machine in response to an indication that the action has been initiated, the sensor feedback being generated based on the performance of the action; and modifying the guidance image in response to the sensor feedback; A method for interactive sewing machine guidance comprising: (Item 2) 2. The method of claim 1, wherein the sensor feedback indicates that a user has completed the at least one stage of the activity. (Item 3) 3. The method of claim 1 or 2, wherein modifying the guidance image includes illustrating a second step of the action. (Item 4) 4. The method of any one of claims 1 to 3, wherein the step of modifying the guidance image is repeated until the user completes the action. (Item 5) 5. The method of any one of claims 1 to 4, wherein the completeness of the action is based on a comparison of the guidance image and the reference image. (Item 6) 2. The method of claim 1, wherein the sensor feedback indicates that at least one step of the action was performed in error. (Item 7) 7. The method of claim 6, wherein the sensor feedback is haptic feedback operable due to vibration in the user device. (Item 8) 8. The method of any one of claims 1 to 7, wherein the act comprises threading the sewing machine. (Item 9) 8. The method of any one of claims 1 to 7, wherein the act comprises threading a serger. (Item 10) 8. The method of any one of claims 1 to 7, wherein the act comprises sewing the workpiece. (Item 11) initiating an action at the sewing machine in response to an action request, the action including manipulating a workpiece with the sewing machine; receiving image data from a camera, said image data including a reference image of said workpiece; projecting a guidance image onto the workpiece, the guidance image being based on the reference image; receiving sensor feedback from at least one sensor of the sewing machine, the sensor feedback being generated based on a comparison of the guidance image and the reference image; and determining the completeness of the action based on the sensor feedback; A method for interactive sewing machine guidance comprising: (Item 12) Item 12. The method of item 11, further comprising detecting an error in the workpiece based on the sensor feedback. (Item 13) 13. The method of claim 11 or 12, wherein the act comprises threading the sewing machine. (Item 14) 13. The method of claim 11 or 12, wherein the act comprises threading a serger. (Item 15) 15. The method of any one of claims 11 to 14, further comprising modifying the guidance image based on the completeness of the action. (Item 16) 16. The method of any one of items 11 to 15, wherein the step of projecting the guidance image onto the workpiece is performed by a user device. (Item 17) 16. The method of any one of claims 11 to 15, wherein the step of projecting the guidance image onto the workpiece is performed by the sewing machine. (Item 18) Sewing machines, and A user device, the user device comprising: receiving image data from a camera, said image data including a reference image of at least a portion of said sewing machine; displaying the image data at the user device, the user device including a display; overlaying a guidance image over the reference image of the sewing machine, the guidance image illustrating at least one stage of an action; receiving sensor feedback from at least one sensor in the sewing machine in response to initiation of the action on the sewing machine, the sensor feedback being generated based on performance of the action; and Modifying the guidance image in response to the sensor feedback. It is configured as follows: A system for interactive sewing machine guidance, comprising: (Item 19) 20. The system of claim 18, wherein the sensor feedback indicates that a user has completed the at least one stage of the activity. (Item 20) 20. The system of claim 18 or 19, wherein modifying the guidance image includes illustrating a second stage of the action. (Item 21) 21. The system of any one of items 18 to 20, wherein modifying the guidance image is repeated until the user completes the action. (Item 22) 22. The system of any one of claims 18 to 21, wherein the completeness of the action is based on a comparison of the guidance image and the reference image. (Item 23) initiating an action at the sewing machine in response to an action request, the action including manipulating a workpiece with the sewing machine; receiving image data from a camera, said image data including a reference image of at least a portion of said sewing machine; displaying the reference image at a user device, the user device including a display; overlaying a first guidance image over the reference image, the first guidance image illustrating at least one stage of an action associated with the sewing machine; receiving sensor feedback from at least one sensor at the sewing machine in response to an indication that the action has been initiated, the sensor feedback being generated based on performance of the action, the sensor feedback being operable to generate physical feedback; and modifying the first guidance image at the user device in response to the sensor feedback. A method for interactive sewing machine guidance comprising: (Item 24) 24. The method of claim 23, wherein the physical feedback is a vibration on the user device. (Item 25) 24. The method of claim 23, wherein the physical feedback is vibration in the sewing machine.

Claims

1. receiving image data from a camera of a user device, the image data including a reference image of at least a portion of a sewing machine; displaying the reference image at the user device, the user device including a display; overlaying a guidance image on the reference image at the user device, the guidance image illustrating at least one step of an activity associated with the sewing machine; receiving sensor feedback from at least one sensor in the sewing machine in response to an indication that the action has been initiated, the sensor feedback being generated based on the performance of the action; and modifying the guidance image at the user device in response to the sensor feedback; A method for interactive sewing machine guidance comprising:

2. The method of claim 1 , wherein the sensory feedback indicates that a user has completed the at least one stage of the activity.

3. The method of claim 2 , wherein modifying the guidance image includes illustrating a second step of an action.

4. The method of claim 3 , wherein the step of modifying the guidance image is repeated until the user completes the action.

5. The method of claim 4 , wherein the completion of the action is based on a comparison of the guidance image and the reference image.

6. The method of claim 1 , wherein the sensor feedback indicates that at least one step of the action was performed in error.

7. The method of claim 6 , wherein the sensory feedback is haptic feedback operable due to vibration in the user device.

8. The method of claim 1 , wherein the act includes threading the sewing machine.

9. The method of claim 1 , wherein the act includes threading a serger.

10. The method of claim 1 , wherein the act includes sewing the workpiece.

11. initiating an action at the sewing machine in response to an action request, the action including manipulating a workpiece with the sewing machine; receiving image data from a camera, the image data including a reference image of the workpiece; projecting a guidance image onto the workpiece, the guidance image being based on the reference image; receiving sensor feedback from at least one sensor of the sewing machine, the sensor feedback being generated based on a comparison of the guidance image and the reference image; and determining the completeness of the action based on the sensor feedback; A method for interactive sewing machine guidance comprising:

12. The method of claim 11 , further comprising detecting an error in the workpiece based on the sensor feedback.

13. The method of claim 11 , wherein the act includes threading the sewing machine.

14. The method of claim 11 , wherein the act includes threading a serger.

15. The method of claim 11 , further comprising modifying the guidance image based on the completeness of the action.

16. The method of claim 11 , wherein projecting the guidance image onto the workpiece is performed by a user device.

17. 16. The method of any one of claims 11 to 15, wherein projecting the guidance image onto the workpiece is performed by the sewing machine.

18. Sewing machines, and A user device, the user device comprising: receiving image data from a camera, the image data including a reference image of at least a portion of the sewing machine; displaying the image data at the user device, the user device including a display; overlaying a guidance image over the reference image of the sewing machine, the guidance image illustrating at least one step of an action; receiving sensor feedback from at least one sensor in the sewing machine in response to initiation of the action on the sewing machine, the sensor feedback being generated based on performance of the action; and Modifying the guidance image in response to the sensor feedback. It is configured as follows: A system for interactive sewing machine guidance, comprising:

19. The system of claim 18 , wherein the sensory feedback indicates that a user has completed the at least one stage of the activity.

20. The system of claim 18 , wherein modifying the guidance image includes illustrating a second step of an action.

21. The system of claim 18 , wherein modifying the guidance image is repeated until a user completes an action.

22. 22. The system of claim 18, wherein the completion of the action is based on a comparison of the guidance image and the reference image.

23. initiating an action at the sewing machine in response to an action request, the action including manipulating a workpiece with the sewing machine; receiving image data from a camera, said image data including a reference image of at least a portion of said sewing machine; displaying the reference image at a user device, the user device including a display; overlaying a first guidance image over the reference image, the first guidance image illustrating at least one stage of an action associated with the sewing machine; receiving sensor feedback from at least one sensor at the sewing machine in response to an indication that the action has been initiated, the sensor feedback being generated based on the performance of the action, the sensor feedback being operable to generate physical feedback; and modifying the first guidance image at the user device in response to the sensor feedback. A method for interactive sewing machine guidance comprising:

24. The method of claim 23 , wherein the physical feedback is a vibration on the user device.

25. 24. The method of claim 23, wherein the physical feedback is a vibration in the sewing machine.

Images