Interactive Augmented Reality Sewing Machine

Abstract

A system and method for interactive guidance of sewing actions are described herein. Many sewing actions, such as threading media, performing general machine maintenance, and sewing using a pattern, can be difficult and require multiple steps to perform correctly. Through the use of augmented reality (AR), the technology disclosed herein provides interactive guidance to users to perform one or more sewing actions. The execution of these actions further enhances the use of sensor feedback, which can reduce common errors associated with each action.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 175,194, filed on April 15, 2021, the content of which is hereby incorporated by reference in its entirety.

[0002] The present disclosure generally relates to sewing machines configured to be used with interactive augmented reality (AR) features.

Background Art

[0003] Sewing machines can be used to form stitches in a single material and to sew together various materials. Certain sewing machines form stitches in a workpiece having a certain shape, cut the edge of the workpiece and sew over the edge, attach decorative elements to the workpiece, cut the edge of the workpiece and sew to encircle the edge, attach a decoration or sew an embroidery pattern onto a workpiece mounted within an embroidery frame, or cut the workpiece during the sewing operation. Sewing machines can also, in addition to or separately from the sewing procedure, cut, fold, roll, or otherwise manipulate the workpiece, and the workpiece is moved under the needle so as to be able to form stitches in the fabric. The user configures the workpiece and the sewing machine by adjusting various parameters of the machine and by attaching a variety of different tools or accessories to the machine for each particular application. This procedure requires the user to configure the workpiece and the sewing machine, is often complex, and can damage the workpiece and / or the sewing machine if done incorrectly. Relying on a manual or similar printed instructions can be cumbersome for the user and still result in an incorrect configuration.

[0004] Therefore, it is understood that there is a need for improved guidance regarding workpiece and sewing machine configuration. [Overview of the project]

[0005] In an exemplary embodiment, a method for interactive sewing machine guidance is provided. The method comprises the steps of: receiving image data from a camera, the image data including a reference image of at least a portion of the sewing machine; displaying the reference image on a user device, the user device including a display; overlaying a guidance image on the reference image, the guidance image illustrating at least one step of an action associated with the sewing machine; responding to an instruction that the action has been initiated; receiving sensor feedback from at least one sensor in the sewing machine, the sensor feedback generated based on the performance of the action; and modifying the guidance image in response to the sensor feedback.

[0006] In another exemplary embodiment, a method for interactive sewing machine guidance is provided. The method includes the steps of: initiating an action in a sewing machine in response to an action request, the action comprising operating a workpiece using the sewing machine; receiving image data from a camera, the image data comprising a reference image of the workpiece; projecting a guidance image onto the workpiece, the guidance image being based on the reference image; receiving sensor feedback from at least one sensor in the sewing machine, the sensor feedback being generated based on a comparison of the guidance image and the reference image; and determining the completeness of the action based on the sensor feedback.

[0007] In another exemplary embodiment, a method for interactive sewing machine guidance is provided. The method comprises the steps of: initiating an action in the sewing machine in response to an action request, the action including operating a workpiece using the sewing machine; receiving image data from a camera, the image data including a reference image of at least a portion of the sewing machine; displaying the reference image on a user device, the user device including a display; overlaying a first guidance image on the reference image, the guidance image illustrating at least one stage of an action associated with the sewing machine; receiving sensor feedback from at least one sensor in the sewing machine in response to an instruction that the action has been initiated, the sensor feedback being generated based on the performance of the action, the sensor feedback being operable to generate physical feedback; and modifying the guidance image on the user device in response to the sensor feedback.

[0008] In yet another exemplary embodiment, a system for interactive sewing machine guidance is provided. The system comprises a sewing machine and a user device, the user device being configured to perform the steps of: receiving image data from a camera, the image data including a reference image of at least a portion of the sewing machine; displaying the image data on the user device, the user device including a display; overlaying a guidance image on the reference image of the sewing machine, the guidance image illustrating at least one step of an action; receiving sensor feedback from at least one sensor on the sewing machine in response to the commencement of the action on the sewing machine, the sensor feedback being generated based on the performance of the action; and modifying the guidance image in response to the sensor feedback.

[0009] These and other purposes, features, and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which should be read in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0010] These and other features of this disclosure will be better understood with reference to the following description and accompanying drawings.

[0011] [Figure 1] This figure shows an exemplary system for interactive sewing machine guidance.

[0012] [Figure 2] This diagram shows an illustrative flowchart related to sensor feedback associated with the sewing process.

[0013] [Figure 3] This diagram shows an illustrative flowchart related to sensor feedback associated with the sewing process.

[0014] [Figure 4] This diagram shows a flowchart illustrating an exemplary method for performing surger threading.

[0015] [Figure 5] This diagram shows a flowchart illustrating an exemplary method for performing a bobbin change operation.

[0016] [Figure 6] This diagram shows a flowchart relating to an exemplary method for performing the act of attaching an embroidery hoop.

[0017] [Figure 7] This is a flowchart illustrating an example method for sewing machine guidance.

[0018] [Figure 8] This is a diagram illustrating an exemplary computing device. [Modes for carrying out the invention]

[0019] The aspects and implementations of this disclosure will be better understood from the detailed descriptions and accompanying drawings provided below of the various aspects and implementations of this disclosure. This should not be considered as limiting the disclosure to any particular aspect or implementation, but rather as a description and understanding only.

[0020] Figure 1 shows an exemplary system 100 for interactive sewing machine guidance according to various embodiments of the present disclosure. System 100 may include at least a sewing machine 102 and a user device 112. The sewing machine 102 and the user device 112 may communicate data, for example, over a network 120 (e.g., the Internet). In certain embodiments, the sewing machine 102 and the user device 112 may communicate data using one or more data communication protocols (e.g., Bluetooth®, RFID, near-field communication, etc.). It is understood that the data communication between the sewing machine 102 and the user device 112 may be in real time or near real time. Furthermore, it is understood that such data communication may enable the sewing machine 102 and the user device 112 to read data from and / or write data to local and / or remote memory. In certain embodiments, the sewing machine and the user device 112 are configured to transmit data over the network 120 to be stored on one or more remote servers (e.g., the cloud).

[0021] An exemplary sewing machine (e.g., sewing machine 102) may include a sewing bed or base having a pillar extending upward from one end to support an arm that extends horizontally above the sewing bed. A sewing head is attached to the end of the arm and may include one or more needle bars for moving a needle up and down to sew a workpiece on the sewing bed below the sewing head. The sewing bed may include a needle or a sewing plate disposed below the sewing head, and the sewing head has an opening for one or more needles to pass through when creating or forming stitches in the workpiece. In some sewing machines, a bobbin disposed below the needle plate assists in stitch formation and supplies a lower thread that is sewn together with an upper thread supplied through the workpiece from above by the needle. In other sewing machines, such as an overlock or serger machine, the lower thread is supplied by a looper.

[0022] As used herein, "sewing machine" means a device that forms one or more stitches in a workpiece with a reciprocating needle and a length of sewing thread. As used herein, "sewing machine" includes, but is not limited to, a sewing machine configured to form a particular stitch (e.g., a lock stitch, a chain stitch, a buttonhole stitch), an embroidery machine, a quilting machine, an overlock or serger machine, etc. Note that various embodiments of sewing machines and accessories are disclosed herein, and any combination of these options may be made unless specifically excluded. In other words, the individual components or parts of the disclosed devices may be combined unless they are mutually exclusive or physically impossible otherwise.

[0023] A "seam" means a loop formed by one or more sewing threads, with at least one of the sewing threads passing through a hole formed in the workpiece. The mechanical components of a sewing machine, such as needles, hooks, loopers, thread tension devices, feed mechanisms, etc., cooperate to form a seam in one or more pieces of the workpiece. One repetition of the movement of this complex machinery can form one seam or a pattern of seams in the workpiece. The "seam length" of a repetition or pattern refers to the distance the workpiece is moved when the repetition is executed. The measurement of seam length varies for different types of repetitions and patterns and can include one or more seams in the workpiece.

[0024] A presser bar with a presser foot also extends downward from the sewing head and presses the workpiece against the sewing bed and against the feed teeth that move the workpiece from back to front and optionally side to side. The feed teeth cooperate with the presser foot and move the workpiece at a speed that can be fixed or variably controlled by the user, such as with a foot pedal. A variety of presser feet and other types of accessories, such as buttonhole presser feet, can be attached to the presser bar to assist in forming certain types of seams or features in the workpiece. An accessory mount can also extend below the sewing head to hold special tools or accessories above or on the sewing bed. For example, as shown in FIG. 1, sewing machine 102 includes an operable embroidery frame 118 that is operable to use X unit 120 and Y unit 122 to enable manipulation of a workpiece attached to embroidery frame 118. Y unit 122 also includes Y slide 124.

[0025] The speed or frequency at which the needle bar moves up and down is controlled by the user as described above. While the needle bar generally moves up and down in a cyclical motion to form a stitch in the workpiece, it can also move left and right to form different stitches, such as zigzag or tapered stitches, or to vary the stitch width. The type and pitch of stitches performed by the machine can be selected by the user via a manual interface, including buttons, knobs, levers, etc. In some embodiments, the type and pitch of stitches performed (or similar settings) may be controllable by the user via input to a user interface, such as a user interface provided on the display 114 and / or display 108 of the user device 112. In other embodiments, the settings may be controllable via voice command input to a voice control interface.

[0026] Different types of sewing machines may include additional components for forming stitches on a workpiece or manipulating the workpiece in other ways during the sewing process. For example, in a surger, one type of sewing machine that can be used to form the edges of a workpiece, among other features, a needle called a looper operates beneath the sewing bed to supply the bobbin thread for forming various stitches. A surger may also include one, two, or more needles above the needle plate and a knife for cutting the edges of the workpiece. A sewing machine may also be used to generate embroidery patterns on a workpiece by including a holder for an embroidery hoop (e.g., embroidery hoop 118) on the sewing bed. The embroidery hoop holder may be actuated on at least two axes so that the machine's controller moves the embroidery frame so that the needle can trace the embroidery pattern on the workpiece. Surgers generally have multiple complex threading paths. The effectiveness of written or verbal instructions is often considered insufficient. Therefore, threading surgers is generally considered a difficult task by users, and many users pay technicians to do the threading on their machines, which is a time-consuming and costly task, rather than performing the task themselves.

[0027] The thread used during sewing is held at a location on the sewing machine, for example, inside the bobbin, or on a spool held by a spool holder that is part of the sewing machine's arm or extends upward. The thread is led from the thread source (e.g., bobbin or spool) to one or more needles of the sewing machine via various different elements of the sewing machine that are arranged to change the direction of the thread so that it is smoothly drawn out and supplied to the workpiece with as little damage to the thread as possible. The tension of the thread can also be changed by various tension devices located along the thread path or within the thread source. These tension devices ensure that only the desired amount of thread is released and that the thread is properly tensioned to form a stitch in the workpiece. A loose thread can allow the stitch to unravel, and a tight thread can cause the stitch to form incorrectly. The tension of the thread for the upper and lower threads can also be adjusted to ensure that the upper and lower tensions are balanced so that the stitch is properly formed along the desired sewing path in the workpiece. It is understood that the aforementioned features and functions of the exemplary sewing machine may be operably connected to one or more sensors (e.g., sensor 110) to generate sensor feedback. For example, the tension of the sewing thread may be recorded by a tension sensor. If the thread is too loose or too tight, the tension sensor may generate sensor feedback indicating that the tension of the sewing thread is outside an acceptable threshold.

[0028] The sewing machine 102 may further include at least one camera 104, a projector 106, a display 108, and / or a sensor 110. Such features, as previously described, may be provided on a device separate from the sewing machine 102 and may be adapted for use on the sewing machine. For example, a projector, a display, and other user interfaces and their hardware components, such as speech recognition, may be included on the sewing machine or on a separate user device, such as a smartphone, tablet, goggles, or glasses. Hardware may include a microphone, a speaker, a piezoelectric tactile sensor, or the like. In certain embodiments, a mirror may be used to reflect and direct a projected image onto a workpiece. The camera 104 may be configured to capture image data associated with the workpiece and / or the sewing machine itself. In certain embodiments, the camera 104 may be used in conjunction with the sensor 110 to generate status information about the sewing machine 102 or the workpiece. In several other specific embodiments, the camera 104 may capture multiple images, e.g., of a workpiece, of one or more components of the sewing machine 102, and / or of the user. In certain embodiments, the projector 106 may be used to project guidance images onto the workpiece. In several other specific embodiments, the projector 106 may be used to project guidance images onto the sewing machine 102 and / or a portion immediately adjacent to it (e.g., on the workpiece) to facilitate maintenance and / or media threading-related activities. It is understood that the sewing machine 102 may further include one or more processors and memory to facilitate communication between its described components, the user device 112, and the user. More specifically, the sewing machine 102 has at least a processor and memory for storing instructions, which, when executed by the processor, causes the processor to perform functions described by instructions (e.g., controlling various aspects of the sewing machine 102, as described in detail herein).Processing can also occur individually or in combination with a processor separate from the sewing machine, for example, via a smartphone or cloud computing.

[0029] In some embodiments, the display 108 associated with the sewing machine 102 is configured to utilize exemplary user interfaces that can be visually presented to the user via one or more displays (e.g., display 108) including a touchscreen display with a touch-sensitive overlay that detects the position of the user's fingers in contact with the display. The user can then interact with the user interface by directly touching the screen at specific locations or by performing touch gestures such as touch, touch-and-hold, pinch or spread, and touch-and-move gestures. The presence, location, and movement of the user's hands, fingers, or eyes or facial expressions can also be detected via analysis of data from optical sensors (e.g., camera 106) or near-field sensors via sound, light, infrared radiation, or electromagnetic field disturbances. A graphical user interface can also be projected by one or more projectors (e.g., projector 104) of the sewing machine onto an adjacent surface such as a sewing bed, workpiece, wall or table, or any other suitable surface. Alternatively, the sewing machine may operate without a graphical user interface, via voice commands and audible feedback in the form of sound and / or computerized voice. Tactile feedback is also provided via actuators that vibrate various parts of the machine when activated, or otherwise via the optical tactile substitution law, where a digital micromirror device projector, emitting time-modulated spatial light and coupled with a finger or tool to which an optical transistor and actuator are attached, generates specific tactile feedback in response to a wide variety of states of the workpiece, machine, etc. Audible and tactile interaction with the sewing machine is particularly useful for visually impaired users.

[0030] In an exemplary embodiment, the sewing machine 102 includes one or more sensors 110. The sensors may measure a variety of aspects associated with the sewing machine 102, the user device 112, and / or the user, including but not limited to light, tension, pressure, hearing, touch, acoustics, sound, vibration, chemistry, bytemetrics, sweat, respiration, fatigue detection, gas, smoke, retina, fingerprints, fluid velocity, speed, temperature, infrared light, ambient light, color, RGB color (or another color space sensor such as one using the CMYK or grayscale color space), touch, tilt, motion, metal detection, magnetic field, humidity, moisture, imagination, photons, force density, proximity, ultrasound, load cell, digital accelerometer, motion, translation, friction, compressibility, voice, microphone, voltage, barometer, gyroscope, Hall effect, magnetometer, GPS, electrical resistance, tension, strain, and many others.

[0031] In certain embodiments, the sensor 110 may provide real-time or near-real-time feedback to the user and / or user device, for example, by tactile sensor / feedback, which may trigger a tactile response in the sewing machine 102 so that incorrect actions in accordance with a desired action immediately alert the user that they have performed the action incorrectly. This immediate feedback may include feedback via projected images and / or in conjunction with vibrations and sounds representing negative confirmation. Positive reinforcement scenarios may also be applied in a similar manner. In some embodiments, sensor feedback may trigger responses in one or more user devices. For example, when performing a sewing action using a pattern, a misalignment of the pattern and workpiece may trigger a sensor feedback response in a smartwatch associated with the user. This immediate feedback may warn the user before the misalignment results in an undesirable output. In an example, when the user's finger is positioned over the fabric or workpiece to be sewn, tactile feedback by vibration is emitted to the user's finger, thereby informing the user of the appropriate time when the user must move the fabric to achieve the desired sewing result. This instruction may be accompanied by, or replaced by, an auditory instruction and an arrow indicating the instruction projected onto the fabric.

[0032] The user device 112 may be a handheld computing device such as a smartphone or tablet computer. The user device 112 includes at least a display 114 and a camera 116. It is understood that additional user devices having similar components, including but not limited to smart glasses, head-up displays (HUDs), and smartwatches, are intended for use with the described system and method. It is understood that in certain embodiments, the user device 112 may be combined with embodiments of multiple devices, such as a smartphone and a smartwatch. It is understood that the user device 112 may be configured to provide feedback to the user of the user device 112 based on its interaction with the sewing machine 102 and / or sensor feedback from there. The camera 116 is configured to capture image data associated with the sewing machine 102 and the surrounding environment. When used herein, such data may refer to reference image data. Reference image data is a real-time or near-real-time capture of the sewing machine 102 or its surrounding environment, including, for example, a workpiece. Image data is understood to include an image or a series of images captured by camera 116 according to a frame rate associated with camera 116. Reference image data is displayed to the user via display 112. According to many embodiments disclosed herein, the reference image may be used in conjunction with guidance images to facilitate interactive sewing machine guidance. In some embodiments, the user device 112 may be attached to the sewing machine 102, or otherwise operablely connected. This configuration may be useful when the user requires both hands to perform an action.

[0033] Figure 2 shows a method 200 for exemplary interactive sewing machine guidance according to the present disclosure. The method begins in step 202, in which the user requests information regarding an action related to a sewing machine (e.g., sewing machine 102). The action may be an action to position a feature on the sewing machine or to perform an operation. Exemplary actions are described in more detail below. In some embodiments, the interactive guidance methodology described herein facilitates guidance of an action by displaying a digital image overlay on a reference image of the sewing machine, e.g., external and / or internal components. For example, a user device may generate and display a digital image (and / or a series of images / videos) that can animate the movement of several components of the sewing machine in order to perform an action. The image is then overlaid on a reference image of the actual sewing machine. In certain embodiments, this reference image is captured by a camera associated with the user device. Exemplary actions for which image overlay interactive guidance may be utilized include threading actions, e.g., threading a sewing machine or surger.

[0034] More exemplary actions in which image overlay interactive guidance may be utilized are those associated with the positioning and identification of sewing machine components and / or accessories. Further examples include using image overlays to indicate the location and names of critical parts of the sewing machine, the location and function of user interfaces associated with the machine, and the location and function of all other components of the machine. Image overlays may also be used to identify unknown components or accessories. For example, using a reference image, user device 112 may recognize a component or accessory based on its size, shape, or location on the sewing machine. In another example, image overlays may provide interactive guidance for changing accessories, such as, but not limited to, needles, presser feet, sewing plates, or bobbins. Image overlays may also be used for interactive guidance on using and securing larger accessories, such as stabilizing fabric in an embroidery hoop or securing an embroidery hoop on the sewing machine. Image overlays may further assist in interactive guidance for certain maintenance tasks, such as cleaning or lubricating some parts (simple) or repairing some components (complex).

[0035] In some embodiments, interactive guidance methodologies, such as those described herein, facilitate guidance of actions by projecting images onto a sewing bed. Such actions may be difficult to perform with user devices (such as actions requiring both hands) or with those actions that require special attention to the workpiece on the sewing bed area. One or more short-throw projectors (e.g., projector 104) may generate 2D or 3D projections onto the sewing bed to guide the performance of actions. In some embodiments, aerial visual projections may be generated via projection or hologram technology. In certain embodiments, additional accessories such as smart glasses may be used to interpret the visual projections. Projections onto a sewing bed can facilitate the visualization and selection of different stitch and embroidery options on the fabric, different decorative options on the fabric, or available sewing techniques appropriate for the fabric. Projections may also be used to perform complex stitch or embroidery actions, such as understanding when and how to rotate and / or translate the fabric, when and how to add textile decorations, and when and how to use special sewing accessories. Furthermore, projection can still help in performing basic actions in the sewing domain, such as when and how to change needles, how to thread one or more needles, how to attach accessories, when and how to change presser feet, and when and how to lift presser feet.

[0036] It is understood that the exemplary actions described above are for illustrative purposes only, and that each action may be adapted for use with image overlays and / or image projections within the scope of this disclosure.

[0037] Referring to Figure 2, in some embodiments, the method 200 for interactive sewing machine guidance may be initiated in step 202 when an action is automatically recognized by the sewing machine 102 (e.g., via an algorithm coupled with AI, a neural network, and / or sensor data input triggers), or by the user device 112 (e.g., via a recognition algorithm coupled with mapping, tracking, and optical sensor data), or by any other specifically requested by the user. In certain embodiments, an action may be automatically recognized by the sewing machine 102 and / or the user device 112 based on operational feedback of the sewing machine (e.g., pedal activation, engagement with threading components), or based on observed optical data from a camera 104 associated with the sewing machine 102 and / or a camera 116 associated with the user device 112.

[0038] After an action is selected, interactive guidance may be provided, for example, using augmented reality (AR). As used herein, AR includes a virtual image overlay intended to guide the user to complete the action. In certain embodiments, the AR image overlay may be achieved by displaying a guidance image on a reference image on a display (e.g., display 114 and / or display 108). For example, a user device 112 may display an image of a sewing machine 102 by overlaying a guidance image. In several other specific embodiments, the guidance image may be physically projected onto a surface such as a sewing bed, for example, by a projector 104. The guidance image may inform the user how to perform a certain task associated with those selected actions. Various types of image overlays are contemplated herein and illustrated by steps 206a-e. In step 206a, “markerless AR” includes a virtual image overlaid on a reference image. The overlaid image may be fully or partially enlarged, meaning, for example, in certain embodiments, the overlaid image may reconstruct an aspect of the reference image rather than displaying the reference image itself. For example, in some embodiments, rather than overlaying an arrow pointing to the bobbin to be replaced (e.g., a partial overlay), the overlaid guidance image may generate a digital version of the bobbin and animate the bobbin to show part of the action previously accomplished by the arrow. Another example could be a digital version of a button or decoration to be sewn onto the fabric to help the user position the object and show how the sewing process should unfold. Yet another example could be a digital version of a finger holding the thread and showing how the person should position the thread along various paths on the surger. This facilitates an interactive experience, and the user's own hand attempts to follow the guidance of the virtual hand and thread in order to achieve the machine's correct threading. In a similar manner, other actions may facilitate an interactive experience.It is further understood that while the guidance image is displayed, it may be modified based on feedback from the sewing machine 102 and / or the user device 112. In embodiments where the user device provides feedback, such feedback data may include additional camera data, accelerometer data, digital compass data, GPS-assisted calculations, etc. For example, if the user chooses to rotate or translate a 3D overlaid image of an appliqué on a sewing workpiece and selects a position on the workpiece where it is impossible to sew the appliqué, the feedback vector may modify the guidance image so that it guides the user to an acceptable position for sewing the appliqué onto the workpiece.

[0039] In step 206b, “location-based AR” may be selected. “Location-based AR” includes a guidance image that is fully or partially overlaid on a reference image. In some embodiments, “location-based AR” may utilize GPS, digital compass, speedometer, or accelerometer data from the user device 112 to calculate precise location and assist in image overlay.

[0040] In step 206c, "Simultaneous Localization and Mapping (SLAM)" may be selected, which involves overlaying a guidance image on a reference image and modifying the guidance image, based on a set of algorithms configured to solve the task of simultaneous localization and mapping by localizing a sensor relative to its surroundings. For example, a user may rotate the fabric of a workpiece while sewing, and the simultaneous localization and mapping of the reference image may identify or follow the pattern or topography of the fabric, ensuring that any relevant overlaid guidance image correctly follows the fabric as translation occurs with respect to newly formed seams on the fabric.

[0041] In certain embodiments, the sewing machine 102 may have one or more identifying features that enable the user device 112 and camera 116 to calibrate the user device's position so that an improved guidance image can be generated by the user device. Such features may be used in step 206d, where “recognition-based AR” may be selected. With “recognition-based AR”, image data from camera 106 and / or camera 116 may be used to identify visual markers such as unique features, QR / 2D codes, natural feature tracking (NFT) markers, objects, or directions, and to use the visual markers to influence the guidance image.

[0042] In step 206e, “projection-based AR” may be selected, which includes images projected onto a surface and / or onto the environment surrounding the sewing machine 102. This type of image guidance allows visual mapping, including 2D and / or 3D static and / or dynamic / animated images, to be projected onto the sewing bed area or other relevant areas using light (e.g., projector 104) in the form of shapes and forms that command or guide the user in performing an action. In certain embodiments, the projected light may come from the user device 112 or another external light source. In some embodiments, “projection-based AR” may be used to contour along a workpiece to guide the user in performing a stitch along a line, threading a surger, or other actions.

[0043] The types of image overlays discussed with reference to steps 206a-e are presented for illustrative purposes only, and it is understood that additional types of image overlays may become apparent from the detailed description herein. In addition, it is intended that the exemplary types of image overlays may be combined during exemplary interactive sewing machine guidance.

[0044] In step 208, feedback is collected and may influence the guidance image generated using one or more of the exemplary guidance image types described above. In a particular embodiment, the type of guidance image used may change based on the feedback. When feedback is considered, method 200 returns to step 204 using the feedback loop 210. The feedback loop 210 may be used an unlimited number of times, or may be used until the action is complete. Through the continuous collection of feedback and the continuous modification of the guidance image in accordance with the feedback, the user can diagnose and correct potential problems in those actions, such as whether the surger has been threaded incorrectly. In a real mechanism, if the user has threaded the sewing machine incorrectly, the problem may not be realized until the sewing-related action has begun, thus making correct sewing impossible until the thread is broken, causing a jam in the sewing machine and making the time-consuming task of removing and rethreading the damaged thread correct and complete. Furthermore, it is understood that in the embodiments of the methods and systems described now, the improved guidance image may serve to inform the user that the action has not been performed correctly in accordance with the feedback.

[0045] Figure 3 shows an exemplary method 300 relating to sensor feedback in relation to a sewing action. In step 302, the user selects an action. In step 304, the connected device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or the associated environment. In step 306, the user proceeds to perform the desired action, following the guidance provided by the guidance image. In step 308a, data collection associated with the performance of the action (e.g., image data collected by camera 116) is collected on the user device. In step 308b, data collection associated with the performance of the action (e.g., data collected from sensors 110 such as light, tension, and pressure sensors) is collected on the sewing machine. In step 308c, data collection associated with the performance of the action is collected on the user device and the sewing machine. In step 310, local or external data processing is performed. In certain embodiments, data processing may be performed on the user device 112, the sewing machine 102, and / or an external source (e.g., the cloud). In step 312a, data (e.g., light, tension, and / or pressure data) is analyzed to determine whether the measurable data attributes constitute acceptable values. The analysis may involve a reference image comparison via a simple input image and may be combined with object detection and tracking sensors (i.e., optical sensors, capacitive sensors, photoelectric sensors, lasers, proximity sensors, infrared sensors, optical time-of-flight sensors, mechanical sensors, etc.). Alternatively, the analysis may be performed via more sophisticated artificial intelligence methods in which the data is processed via one or more neural networks and multi-sensor data inputs. For example, data may be received and analyzed in a neural network layer, and predictions of whether the recognition of an object, its position, and motion is correct or incorrect may be analyzed in a pre-trained or training network designed to recognize the situation and user actions. A response may then be triggered based on the steps described above. Accurate predictions may be based on a scoring system.For example, an accuracy prediction of whether a user action is correct or incorrect, greater than a certain percentage or threshold, may be considered acceptable based on the final format collected in-data. In step 312b, the data enters the input layer of the neural network, and the output layer provides an assessment of the integrity of the action. It is understood that all forms of data available to the sewing machine, namely data from sensors, data from software, data from data storage devices, data from user input from software, and similar, can be processed via the neural network. An exemplary neural network for use with the interactive sewing machine guidance technology described herein is described in further detail in U.S. Patent Application No. 17 / 352,035, entitled "Sewing Machine and Method of Use thereof," the subject matter of which is incorporated herein by reference in whole. The information to be processed first faces the input layer of the neural network, which performs initial processing of the input data and outputs the results to one or more hidden layers that process the output values ​​from the input layer. The information processed through the hidden layer is presented in the output layer as a probability of confidence in a given result, such as the location of a detected object in an image and the classification of that object. The sewing machine's computer software may receive information from one of the neural network layers and act accordingly to adjust the sewing machine's parameters and / or to notify the user based on the results of the neural network processing.

[0046] During the training of a neural network, the node parameters for each node of the neural network (i.e., at least one of the input parameters, function parameters, and output parameters) are adjusted via a backpropagation algorithm until the output of the neural network corresponds to a desired output for a set of input data. During the neural network processing, the neural network begins the training process with node parameters that can be randomized or transferred from an existing neural network. Next, the neural network is presented with data from a sensor to be processed. For example, an object may be presented to an optical sensor to provide visual data to the neural network. The data is processed by the neural network, and the output is tested so that the node parameters of various nodes of the neural network can be updated to increase the confidence probabilities of the detection and classification performed by the neural network. For example, if a clamp is presented to an optical sensor for identification by the neural network, the neural network presents a confidence probability that the object presented to the optical sensor is located within the coordinate range of the image and can be classified as a specific clamp. As the training process is performed, the node parameters of the nodes of the neural network are adjusted to increase the neural network's confidence that a particular answer is correct. Thus, when presented with specific visual data, the neural network learns to "understand" that a particular answer is the most correct answer, even if that data is not exactly the same as what it has "seen" before.

[0047] A neural network is considered "trained" when the decisions made by the network reach a desired level of accuracy. A trained neural network can be characterized by a set of node parameters that have been tuned during the training process. This set of node parameters can be transmitted to other neural networks with the same node structure, thereby allowing those other neural networks to process data in the same way as the initially trained network. Thus, a neural network stored in a particular sewing machine's data storage device can be updated by downloading new node parameters. It should be noted that the node parameters of a neural network, such as input weight parameters and thresholds, tend to occupy significantly less storage space than the image library used for comparison with image or visual data collected by optical sensors. Consequently, the neural network file and other important files can be updated quickly and efficiently via the network. For example, the structure of the neural network, i.e., the map of connections between nodes and the activation function computed at each node, can also be updated in this way.

[0048] Neural networks can also be continuously trained so that their node parameters are periodically updated based on feedback from various data sources. For example, the node parameters of a neural network, stored locally or externally, can be periodically updated based on data collected from sensors, which may or may not match the neural network's output. These tuned node parameters can also be uploaded to a cloud-based system and shared with other sewing machines so that all neural networks in a sewing machine improve over time.

[0049] In certain embodiments, data collected by the sewing machine's sensors may be processed locally within the sewing machine or by an external processor in a cloud-based neural network. The locally stored neural network may be pre-trained or a continuously updated neural network. The data processed by the neural network (locally or remotely) is then used by the sewing machine's software to make decisions that result in machine and / or user interactions.

[0050] It is understood that various types of feedback mechanisms are contemplated herein. It is understood that feedback can trigger a feedback response in a user device and / or sewing machine. For example, a feedback response may be relayed to the user via visual, audible, and tactile means. For example, while a notification sound, such as a beep or computerized sound, is transmitted to the user via a speaker in the sewing machine and / or user device, an instruction that an incorrect accessory is installed in the machine may be presented to the user via the user interface on the sewing machine's display. The notification may also be transmitted to the user via tactile or haptic feedback through vibration of a part of the machine in contact with the user. That is, the sewing bed may be vibrated by the sewing machine to provide the user with an alert that the machine is not properly configured for a particular sewing operation selected by the user. The user feels the vibration under their fingers in contact with the workpiece and sewing bed, thereby prompting the user to look at the display for further information. The sewing machine's illumination can also be controlled to warn the user, for example, by illuminating a certain area on the machine, changing color or flashing, to prompt the user to view additional information on the display when an incorrect accessory is installed. It is understood that the neural networks described above can be readily adapted to make predictive assessments of the completeness and / or accuracy of a given action based on sensor data. It is further understood that exemplary neural networks can be used through the methods and systems described herein to analyze, interpret, and utilize feedback data (e.g., from the sewing machine 102, user device 112, and / or external sources).

[0051] Returning to Figure 3, in step 314, it is determined whether the act is complete (for example, using the metrics provided by steps 312a and / or 312b). If the act is determined to be incomplete, method 300 returns to 304 and repeats all or part of the act until the act is determined to be complete in step 314. Once the act is determined to be complete, the method terminates in step 318. In certain embodiments, certain verification steps may be performed to evaluate the outcome of the act, and in such embodiments, it may be required that the act be repeated or, as an alternative, that additional acts be performed.

[0052] Figure 4 shows an exemplary method 400 relating to sensor feedback in relation to the sewing action of threading a surger. In step 402, the user selects an action related to threading the surger. In step 404, the connected device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or related environment, instructing the user how to thread the surger. In step 406, the user proceeds to perform the desired action following the guidance provided by the guidance image. In step 408a, data collection associated with the performance of the action (e.g., image data collected by camera 116) is collected on the user device. In step 408b, data collection associated with the performance of the action (e.g., data collected from sensors 110 such as light, tension, and final pressure sensors) is collected on the sewing machine. In step 408c, data collection associated with the performance of the action is collected on the user device and the sewing machine. In step 410, local or external data processing is performed. In certain embodiments, data processing may be performed on the user device 412, the sewing machine 102, and / or an external source (e.g., the cloud). In step 412a, the data is analyzed to determine whether measurable data attributes (e.g., light, tension, and / or pressure values) constitute acceptable values. In step 412b, the data enters the input layer of a neural network, and the output layer provides a predictive assessment of the accuracy of the integrity of the action.

[0053] In step 414, it is determined whether the surger threading act is complete (for example, using the metrics provided by steps 412a and / or 412b). If the act is determined to be incomplete, method 400 returns to 404 and repeats all or part of the act until the act is determined to be complete in step 414. Once the act is determined to be complete, the method terminates in step 418. In certain embodiments, certain verification steps may be performed to evaluate the outcome of the act, and in such embodiments, the act may be required to be repeated or, as an alternative, to perform an additional act. For example, in a surger threading act, multiple threads and thread paths may exist on the surger that require ordered verification of the correct threads and paths, which may be performed after the act has been determined to be complete.

[0054] Figure 5 shows an exemplary method 500 relating to sensor feedback in relation to a sewing action related to changing a bobbin. In step 502, the user selects an action related to changing a bobbin. In step 504, the connected device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or the associated environment. In step 506, the user proceeds to perform the desired action following the guidance provided by the guidance image. In step 508a, data collection associated with the performance of the action (e.g., image data collected by camera 116) is collected at the user device. In step 508b, data collection associated with the performance of the action (e.g., data collected from sensors 110 such as light, tension, and final pressure sensors) is collected at the sewing machine. In step 508c, data collection associated with the performance of the action is collected at the user device and the sewing machine. In step 510, local or external data processing is performed. In certain embodiments, data processing may be performed at the user device 112, the sewing machine 102, and / or an external source (e.g., the cloud). In step 512a, data (e.g., light, tension, and / or pressure values) are analyzed to determine whether the measurable data attributes constitute acceptable values. In step 512b, the data enters the input layer of a neural network, and the output layer provides an assessment of the completeness of the action. In step 514, it is determined whether the action is complete (for example, using the metrics provided by steps 512a and / or 512b). If the action is determined to be incomplete, method 500 returns to 504 and repeats all or part of the action until the action is determined to be complete in step 514. Once the action is determined to be complete, the method terminates in step 518. In certain embodiments, certain verification steps may be performed to evaluate the outcome of the action, and in such embodiments, it may be required that the action be repeated or, as an alternative, that additional actions be performed.For example, in the exemplary action described above, the user would need to ensure that the thread is properly unrolled from the bobbin in the correct direction; otherwise, the needle may break or the thread tension may be inaccurate.

[0055] Figure 6 shows an exemplary method 600 relating to sensor feedback in relation to the sewing action of how to attach and connect the embroidery hoop. In step 602, the user selects the action of attaching and connecting the embroidery hoop. In step 604, the connecting device or sewing machine presents an AR guidance image overlaid on a reference image of the sewing machine or associated environment. In step 606, the user proceeds to perform the desired action following the guidance provided by the guidance image. In step 608a, data collection associated with the performance of the action (e.g., image data collected by camera 116) is collected at the user device. In step 608b, data collection associated with the performance of the action (e.g., data collected from sensors 110 such as light, tension, and final pressure sensors) is collected at the sewing machine. In step 608c, data collection associated with the performance of the action is collected at the user device and the sewing machine. In step 610, local or external data processing is performed. In certain embodiments, data processing may be performed on a user device 112, a sewing machine 102, and / or an external source (e.g., the cloud). In step 612a, the data is processed by an algorithm to determine whether measurable data attributes constitute acceptable values, such as light, tension, and / or pressure values. In step 612b, the data enters the input layer of a neural network, and the output layer provides an assessment of the completeness of the action. In step 614, it is determined whether the action is complete (e.g., using the metrics provided by steps 612a and / or 612b). If the action is determined to be incomplete, method 600 returns to 604 and repeats all or part of the action until the action is determined to be complete in step 614. Once the action is determined to be complete, the method terminates in step 618. In certain embodiments, certain verification steps may be performed to evaluate the outcome of the action, and in such embodiments, it may be required that the action be repeated or, as an alternative, that additional actions be performed.

[0056] Figure 7 illustrates a flowchart of an exemplary method for interactive sewing machine guidance. It will be understood that the illustrated method and associated steps can be performed in different orders, by omitting illustrated steps, by adding additional steps, or by rearranged, combined, omitted, or additional steps.

[0057] The process begins in step 702, when a reference image is received. A guidance image is overlaid on the reference image in step 704. In step 706, an action is initiated. As described herein, the action may be any action associated with the operation and / or maintenance of the sewing machine. In step 708, action feedback is analyzed by, for example, a user device 112. In certain embodiments, it is understood that action feedback may also be analyzed by the sewing machine 102, an external source, and / or any combination of the user device 112, the sewing machine 102, and the external source. In step 710, the guidance image is modified based on the action feedback. It is understood that modifications to the guidance image may be directed to facilitate the completion of the action. For example, an exemplary modification may be to warn a user who has incorrectly threaded the sewing machine.

[0058] Now, referring to Figure 8, a high-level diagram of the computing device 800 is shown as an example of a system and methodology that may be used according to the system and methodology disclosed herein. It is understood that the user device 112 is one exemplary implementation of the computing device 800. It is further understood that the sewing machine 102 may further include elements of the computing device 800. The computing device 800 may be used in a system for interactive sewing machine guidance (e.g., system 100). The computing device 800 includes at least one processor 802 that executes instructions stored in memory 804. The instructions may be, for example, instructions for implementing a function described above as being performed by one or more components discussed above, or instructions for implementing one or more of the methods described above. The processor 802 may access memory 804 by system bus 806.

[0059] The computing device 800 also includes a data storage 808 accessible by the processor 802 via a system bus 806. The data storage 808 may contain executable instructions related to the execution of the systems and methods disclosed herein. The computing device 800 also includes an input interface 810 that allows external devices to communicate with the computing device 800. For example, the input interface 810 may be used to receive instructions from a computer device, from a user, etc. The computing device 800 also includes an output interface 812 that interfaces the computing device 800 with one or more external devices. For example, the computing device 800 may display text, images, etc., via the output interface 812.

[0060] It is intended that external devices communicating with the computing device 800 via the input interface 810 and output interface 812 are included in an environment that ultimately provides any type of user interface with which the user can interact. Examples of user interface types include graphical user interfaces and natural user interfaces. For example, a graphical user interface may accept input from a user using input devices such as a keyboard, mouse, or remote control, and provide output on an output device such as a display. Furthermore, a natural user interface may enable the user to interact with the computing device 800 in a manner that is not constrained by input devices such as a keyboard, mouse, or remote control. Rather, a natural user interface may rely on speech recognition, touch and stylus recognition, gesture recognition both on and near the screen, air gestures, head and eye tracking, voice and conversation, vision, touch, gestures, machine intelligence, etc.

[0061] Furthermore, although illustrated as a single system, it should be understood that computing device 800 can be a distributed system. For example, several devices may be in a communication state via a network connection and collectively perform tasks described as being performed by computing device 800.

[0062] It should be understood that the exemplary systems described herein (e.g., sewing machine 102, user device 112, etc.) may be implemented in hardware, software, or any combination thereof. If implemented in software, the functionality may be stored on a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable medium includes computer-readable storage media. Computer-readable storage media may be any available storage medium accessible by a computer. For example, and not limited to, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to hold or store desired program code in the form of instructions or data structures and is accessible by a computer. As used herein, the terms "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital multipurpose discs (DVDs), floppy disks, and Blu-ray discs (BDs), where a "disk" typically reproduces data magnetically, and a "disc" typically reproduces data optically by a laser. Furthermore, transmitted signals are not included in the scope of computer-readable storage media. Computer-readable media also include communication media, which include any medium that facilitates the transfer of a computer program from one location to another. A connection can be a communication medium, for example. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of communication media. Any combination of the above should also be included in the scope of computer-readable media.

[0063] Alternatively, or in addition, functionally, what is described herein can be performed, at least in part, by one or more hardware logic components. For the sake of example, and not limited to, illustrated types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), programmable integrated circuits (ASICs), programmable standard products (ASSPs), system-on-chip systems (SOCs), coupled programmable logic devices (CPLDs), and the like.

[0064] It should be understood that the detailed description is intended to be illustrative and not to limit the embodiments described. Other embodiments will become apparent to those skilled in the art when reading and understanding the above description. In addition, in some examples, elements described in one embodiment may readily be adapted for use in other embodiments. Thus, the methods and systems described herein are not limited to specific details, exemplary embodiments, or exemplary examples shown and described. Accordingly, deviations from such details may be made without departing from the spirit and scope of the general aspects of this disclosure. (Other possible items) (Item 1) In the step of receiving image data from the camera, the image data includes a reference image of at least a part of the sewing machine; The step of displaying the reference image on the user device includes a display; In the step of overlaying a guidance image on the reference image, the guidance image illustrates at least one step of an action associated with the sewing machine; In response to an instruction that the aforementioned action has been initiated, the step of receiving sensor feedback from at least one sensor in the sewing machine, the sensor feedback being generated based on the performance of the aforementioned action; and, Steps to modify the guidance image in response to the sensor feedback, A method for interactive sewing machine guidance, including... (Item 2) The method according to item 1, wherein the sensor feedback indicates that the user has completed at least one stage of the action. (Item 3) The method according to item 1 or 2, wherein the step of modifying the guidance image includes a step of illustrating the second step of the action. (Item 4) The step of modifying the guidance image is repeated until the user completes the action, as described in any one of items 1 to 3. (Item 5) The integrity of the said action is determined by the method described in any one of items 1 to 4, based on a comparison of the guidance image and the reference image. (Item 6) The method according to item 1, wherein the sensor feedback indicates that at least one step of the action was performed incorrectly. (Item 7) The method according to item 6, wherein the sensor feedback is haptic feedback that can operate for vibration in the user device. (Item 8) The aforementioned action is the method according to any one of items 1 to 7, which includes the step of threading the sewing machine. (Item 9) The aforementioned action is the method described in any one of items 1 to 7, which includes the step of threading the surger. (Item 10) The aforementioned act is the method described in any one of items 1 to 7, including sewing the workpiece together. (Item 11) In response to a request for action, the sewing machine initiates an action, the action including the step of using the sewing machine to manipulate a workpiece; In the step of receiving image data from the camera, the image data includes a reference image of the workpiece; In the step of projecting a guidance image onto the workpiece, the guidance image is based on the reference image; The step of receiving sensor feedback from at least one sensor of the sewing machine, the sensor feedback is generated based on a comparison of the guidance image and the reference image; and, A step of determining the completeness of the action based on the sensor feedback, A method for interactive sewing machine guidance, including... (Item 12) The method according to item 11, further comprising the step of detecting an error in the workpiece based on the sensor feedback. (Item 13) The aforementioned action is the method of item 11 or 12, which includes the step of threading the sewing machine. (Item 14) The aforementioned action is the method described in item 11 or 12, which includes the step of threading the surger. (Item 15) The method according to any one of items 11 to 14, further comprising the step of modifying the guidance image based on the integrity of the aforementioned action. (Item 16) The method according to any one of items 11 to 15, wherein the step of projecting the guidance image onto the workpiece is performed by a user device. (Item 17) The method according to any one of items 11 to 15, wherein the step of projecting the guidance image onto the workpiece is performed by the sewing machine. (Item 18) Sewing machine, and User device, the user device is Image data is received from the camera, and the image data includes a reference image of at least a portion of the sewing machine; The image data is displayed on the user device, and the user device includes a display; A guidance image is overlaid on the reference image of the sewing machine, and the guidance image illustrates at least one step of the process; In response to the commencement of the action in the sewing machine, sensor feedback is received from at least one sensor in the sewing machine, and the sensor feedback is generated based on the performance of the action; and, The guidance image is modified in response to the sensor feedback. It is structured in such a way. A system for interactive sewing machine guidance, including [specific feature / feature]. (Item 19) The system according to item 18, wherein the sensor feedback indicates that the user has completed at least one stage of the action. (Item 20) Modifying the aforementioned guidance image includes illustrating the second stage of the action, as described in item 18 or 19 of the system. (Item 21) The system described in any one of items 18 to 20 repeats the modification of the aforementioned guidance image until the user completes the action. (Item 22) A system according to any one of items 18 to 21, wherein the integrity of the aforementioned action is based on a comparison of the guidance image and the reference image. (Item 23) In response to a request for action, the sewing machine initiates an action, the action including the step of using the sewing machine to manipulate a workpiece; In the step of receiving image data from the camera, the image data includes a reference image of at least a portion of the sewing machine; In the step of displaying a reference image on a user device, the user device includes a display; In the step of overlaying a first guidance image on the aforementioned reference image, the first guidance image illustrates at least one step of an action associated with the sewing machine; In response to an instruction that the aforementioned action has been initiated, the sewing machine receives sensor feedback from at least one sensor, the sensor feedback is generated based on the performance of the aforementioned action, and the sensor feedback is operable to generate physical feedback; and Steps to modify the first guidance image in the user device in response to the sensor feedback. A method for interactive sewing machine guidance, including... (Item 24) The method according to item 23, wherein the physical feedback is vibration in the user device. (Item 25) The method according to item 23, wherein the physical feedback is vibration in the sewing machine.

Claims

1. In the step of receiving image data from the user device's camera, the image data includes a reference image of at least a portion of the sewing machine; The step of displaying the reference image on the user device includes a display; In the user device, the guidance image is overlaid on the reference image, and the guidance image illustrates at least one step of an action associated with the sewing machine; In response to an instruction that the aforementioned action has been initiated, the user device receives sensor feedback from at least one sensor in the sewing machine, the sensor feedback being generated based on the execution of the aforementioned action; and, In response to the sensor feedback, the user device modifies the guidance image. A method for interactive sewing machine guidance, including...

2. The method according to claim 1, wherein the sensor feedback indicates that the user has completed at least one step of the action.

3. The method according to claim 2, wherein the step of modifying the guidance image includes the step of illustrating a second stage of the action.

4. The method according to claim 3, wherein the step of modifying the guidance image is repeated until the user completes the action.

5. The method according to claim 4, wherein the completion of the aforementioned action is based on a comparison of the guidance image and the reference image.

6. The method according to claim 1, wherein the sensor feedback indicates that at least one step of the action was performed incorrectly.

7. The method according to claim 6, wherein the sensor feedback is haptic feedback that can operate for vibrations in the user device.

8. The method according to any one of claims 1 to 6, wherein the act includes the step of threading the sewing machine.

9. The method according to any one of claims 1 to 6, wherein the aforementioned action includes the step of threading a surger.

10. The method according to any one of claims 1 to 6, wherein the act includes sewing the workpiece together.

11. Sewing machine, and User device, the user device is The user device receives image data from its camera, and the image data includes a reference image of at least a portion of the sewing machine; The user device displays the image data, and the user device includes a display; On the user device, a guidance image is overlaid on the reference image of the sewing machine, and the guidance image illustrates at least one step of an action associated with the sewing machine; In response to the commencement of the action in the sewing machine, sensor feedback is received from at least one sensor in the sewing machine, and the sensor feedback is generated based on the performance of the action; and, In response to the sensor feedback, the user device modifies the guidance image. It is structured in such a way. A system for interactive sewing machine guidance, including [specific feature / feature].

12. The system according to claim 11, wherein the sensor feedback indicates that the user has completed at least one step of the action.

13. The system according to claim 12, wherein modifying the guidance image includes illustrating a second stage of the action.

14. The system according to claim 13, wherein modifying the guidance image is repeated until the user completes the action.

15. The system according to claim 14, wherein the completion of the aforementioned action is based on a comparison of the guidance image and the reference image.

16. In response to a request for action, the sewing machine initiates an action, the action including the step of using the sewing machine to operate a workpiece; In the step of receiving image data from the user device's camera, the image data includes a reference image of at least a portion of the sewing machine; The step of displaying a reference image on the user device includes a display; In the user device, the first guidance image is overlaid on the reference image, the first guidance image illustrates at least one step of an action associated with the sewing machine; In response to an instruction that the aforementioned action has been initiated, the user device receives sensor feedback from at least one sensor in the sewing machine, the sensor feedback is generated based on the performance of the aforementioned action, and the sensor feedback is operable to generate physical feedback; and Steps to modify the first guidance image in the user device in response to the sensor feedback. A method for interactive sewing machine guidance, including...

17. The method according to claim 16, wherein the physical feedback is vibration in the user device.

18. The method according to claim 16, wherein the physical feedback is vibration in the sewing machine.

Images