Device, system and method for automatically delaying video presentation

Abstract

A computer-implemented method for automatically presenting delayed video. During normal operation, a real-time video feed is sent to a digital screen for presentation and is also sent to a memory for storage. When a delay trigger is issued, the system freezes the frame presented in this case or stores a time indicator, the system also monitors the delay time, and when the delay time is reached, the system starts to fetch frames from the memory, starting with the frame immediately following the frozen frame or starting with the frame stored at the time indicator, and sends these frames to the digital screen for presentation. The system also monitors the delay period, and when the delay period is reached, the system reverts to normal mode, where the real-time video feed is sent to the digital screen for presentation and is also sent to the memory for storage.

Description

[0001] This application is a divisional application of the invention patent application with application number 201680005388.1, application date January 8, 2016, and invention name “Device, system and method for automatic delayed video presentation”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 101,397, filed on January 9, 2015, entitled “Provisional of Computerizedvirtual digital mirror control features and UI (user interface), as well as product and application design flow to improve the usability of the device / service the user experience and the engagements between sales associate / stylists with customers in the store and on the go,” which is incorporated herein by reference in its entirety. Technical Field

[0004] The present invention relates generally to imaging and display systems, and more particularly to monitors and interactive displays, e.g., in retail and / or service environments, in medical or home situations, video conferencing, gaming, and the like. A particular embodiment relates to a virtual mirror in situations where a user desires to see a mirror, such as when trying on clothing. Another particular embodiment relates to situations where a natural appearance is preferred over a standard video image, such as in video conferencing. Background Art

[0005] Conventional mirrors (i.e., reflective surfaces) are a common and most reliable tool for individuals to obtain their true self-appearance in real time. Some alternatives have been proposed around the combination of cameras and screens to replace traditional mirrors. However, these technologies are not convincing and have not yet been accepted as reliable personal images of oneself as if looking at oneself in a conventional mirror. This is mainly because the image generated by a camera is very different from the image produced by a mirror.

[0006] In U.S. Application No. 13 / 843,001, we proposed a novel technique (computerized method) to address the challenge of converting and transforming still images or 2D / 3D videos generated by one or more cameras (with or without other sensors) into a mirrored or video conferencing experience. Summary of the invention

[0007] In this disclosure, we describe computerized techniques that take real-time video and implement automatic presentation of delayed video. Such a feature is beneficial in various situations, such as when the system is implemented as a digital mirror and a user wishes to have a fitting session to try on an item of clothing and see how it looks from various directions, including from the back. Using a standard mirror, a user cannot see how an item of clothing looks from behind. Using this technique in a digital mirror enables a person to see the item from 360 degrees.

[0008] In another implementation, the system can be used for training purposes. For example, when a dancer wants to see how a particular move is performed, or when a golfer wants to see how to execute a swing. With a standard video recorder, the user must start recording, start performing the movement, and then return to the recorder, rewind and review the recorded video, which disrupts the flow of training. In contrast, using the disclosed embodiments, the user is able to focus on the movement, and when the movement is completed, the user only needs to look at the screen and watch an automatic "instant replay" of the movement, which does not disrupt the flow of training. In addition, after the replay is completed, the system automatically reverses to normal mode, so the user does not have to interact with the camera at all.

[0009] The trigger to enter the delay mode can be done in many ways, for example, by simply providing the user with a button to press or tap, or by analyzing the video stream and issuing a trigger based on the analysis. The analysis can look for specific movements of the user, such as the user starting to turn, or can look for specific items in the image. For example, if the system is used for golf training, the analysis might look for a golf club being raised above a preset height, indicating that the user is performing a golf swing.

[0010] In one aspect, a computer-implemented method for operating a system having a monitor, a camera, and a processor so as to sequentially display a real-time video stream and a delayed video stream on the monitor to enable a user to enhance the experience of a viewing session is described as comprising: storing a program for execution by the processor on a device having the processor and a memory, the program comprising instructions for the following operations: performing the following steps in each session: acquiring a live video feed from the camera; simultaneously sending the video feed to the monitor to be displayed as a real-time video stream, and storing the video feed as frames in the memory; upon receiving an indication to present a delayed video stream, acquiring the frames from the memory and sending them to the monitor to be displayed as a delayed video stream, starting with a frame stored N seconds before the indication was received, and after acquiring a series of frames extending for a preset M seconds, stopping the acquisition and resuming sending the video feed to the monitor to be displayed as a real-time video stream.

[0011] According to another aspect, a computer-implemented method is provided for operating a system having a monitor, a camera, and a processor so as to sequentially display a real-time video stream and a delayed video stream on the monitor to enable a user to enhance the experience of a viewing session, comprising: setting a delay period of N seconds; setting a recording length of M seconds; acquiring a live video feed from the camera; simultaneously sending the video feed to the monitor to be displayed as a real-time video stream, and saving the video feed as frames in the memory; upon receiving a trigger, performing: indicating the frame stored at the time of the trigger as the first frame; counting N seconds, and then, starting from the first frame, acquiring the frame from the memory and sending it to the monitor to be displayed as a delayed video stream; and after acquiring a series of frames for M seconds, stopping the acquisition and resuming sending the video feed to the monitor to be displayed as a real-time video stream.

[0012] According to another aspect, a computer-implemented method for automatically presenting delayed video is provided, comprising: during a normal operating mode, sending real-time video from a camera to a digital screen for presentation and simultaneously sending the real-time video to a memory for storage; when a delay trigger is issued, entering a delay mode and performing one of the following operations: freezing the frame presented when the delay trigger is issued, but continuing to send the real-time video to the memory for storage, or storing a time indicator representing the time when the trigger is issued, but continuing to send the real-time video from the camera to the digital screen for presentation and simultaneously sending the real-time video to the memory for storage; monitoring a delay time, and when the delay time is reached, starting to acquire frames from the memory, starting with the frame immediately following the frozen frame or with the frame stored at the time indicator, and sending these frames to the digital screen for presentation; monitoring a delay period, and when the delay period is reached, reverting to a normal mode, wherein the real-time video feed is sent to the digital screen for presentation and also to the memory for storage.

[0013] In the method, the delayed trigger may be generated by continuously analyzing the video feed and issuing the delayed trigger when it is determined that the user is performing a triggering motion. The delayed trigger may also be generated by continuously analyzing the video feed and issuing the delayed trigger when it is determined that an object in the image has moved a specified distance. The delayed trigger may also be generated by continuously analyzing the video feed and issuing the indication when it is determined that the user has entered the field of view of the camera. The delayed trigger may be generated by the user pressing or tapping a trigger button. The method may also include: flipping each frame of the video feed about a vertical axis so that the right and left sides of the image are reversed; and applying a transformation map to each frame of the video feed to modify the frames so that they appear to mimic the reflection of a mirror, thereby obtaining a transformed frame. The method may also include enabling the user to pause the replay of the delayed video. The length of the delay period may be configured to have sufficient time to enable the user to turn around. The method may also include overlaying a graphic on the real-time video during the period of the delay time. The program may also include instructions for generating a model mask to enable the color of the clothing item to be changed. The model mask may include multiple layers including at least a color layer and a texture layer.The program may also include instructions for generating a split screen on the monitor and displaying the real-time video on one side of the screen and playing the delayed video on the other half.

[0014] In other embodiments, the mechanical design and appearance of the product are disclosed to enhance the overall usability and functionality and user experience. The system may include: a housing configured to house a processor therein, the housing having an extension; an optical sensor housing coupled to the extension; a processor housed in the housing, the processor including a memory and a video engine, and having an output configured to send a digital image to a digital screen; an optical sensor housed in the optical sensor housing and having a means to send an image to the processor; and a connection mechanism configured to connect the housing to the back of the digital screen.

[0015] According to a further aspect, a program may be stored in the system that, when executed by a processor, causes the system to display a mirrored simulated image on a monitor and / or perform automatic delayed mode rendering as described in various embodiments herein.

[0016] Also described herein is a non-transitory computer-readable storage medium for operating a monitor, a camera, and a processor to display a clip of delayed video on a monitor. In some embodiments, a program executed by the processor is stored in a device having a processor and a memory, the program including instructions for operating in a normal mode and a delayed mode, wherein in the normal mode, the program includes instructions for receiving a real-time video stream from a camera and sending the real-time video stream for display on a screen and storing in a memory; and when entering the delayed mode, performing: indicating a trigger time when the trigger is received; counting N seconds from the trigger time, then acquiring the frame from the memory and sending it to the monitor for display as a delayed video stream, starting with the frame stored at the trigger time, and simultaneously continuing to store the video stream to the memory; after acquiring a series of frames for M seconds, stopping the acquisition and resuming sending the video feed to the monitor for display as a real-time video stream. In some embodiments, the program includes instructions for a mirror simulation mode for displaying a mirror simulation image on a monitor.

[0017] A computer system for operating a monitor, a camera, and a processor to display a mirrored simulated image on a monitor is also described herein. The computer system includes a program stored in a device having a processor and a memory and executed by the processor, the program including instructions for the following operations: sensing for a user, starting a delay mode to automatically display a clip of a delayed video on a monitor and restoring to real-time video when the presentation of the delayed clip is terminated. The program may also include instructions for performing frame grabbing of an image captured by a camera, performing element selection to obtain a selected element from the frame, generating a mask corresponding to the selected element, selecting a new color, applying the mask to the new color to generate a new image, and displaying the new image on a monitor. That is, the delay mode described herein can be implemented with or without other features described herein (e.g., mirroring video, applying masks and changing colors, determining BMI, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings incorporated into this specification illustrate one or more exemplary embodiments of the invention disclosed herein, and together with the detailed description, are used to explain the principles and exemplary implementations of these inventions. Those skilled in the art will appreciate that the accompanying drawings are illustrative only, and that the contents described herein may be adjusted based on the textual contents of this specification and the spirit and scope of this invention's teachings.

[0019] In the drawings, like reference numerals refer to like elements in the specification:

[0020] Figure 1 A system block diagram of an augmented reality platform for supporting real-time or recorded video / images is shown.

[0021] Figure 2 shows an example of an augmented reality module that may correspond to the augmented reality modules described herein;

[0022] Figure 3 An augmented reality module is shown that can replace body parts, colors, orientations, and textures of items or objects in the foreground or background of an image, for example, the module can add hair to a user, change the user's eyes, skin, and hair color, can change the expression of the eyes, etc.; and

[0023] Figure 4 A computational method for creating a model for color and texture exchange, which may be referred to as Colograma, is shown;

[0024] Figure 5 Describes a computational method for creating a model for color and texture exchange, which may be referred to as Colograma;

[0025] Figure 6is an example of a security system application for multi-level user learning and authentication;

[0026] Figure 7 Depicted are examples of security system applications for parallel or in-band camera manipulation;

[0027] Figure 8a Describes the shared mirroring experience in video / voice conferencing;

[0028] Figure 8b depicts video conferencing that shares the entire body;

[0029] Figure 8c Multiple reflectors / full body stations with split or multiple screens are depicted;

[0030] Fig. 9 An example involving a 2D input to a model generator where we wish to create a model of a user's shirt;

[0031] Fig.10 is an example of a 2D model of an upper shirt;

[0032] Fig.11 is a schematic diagram showing the classification of male body types;

[0033] Fig.12 is a schematic diagram showing the classification of female body types;

[0034] Fig.13 An example of the results of additional processing that eliminates the hands from the body and points to different intervals of where to find specific body measurements is depicted.

[0035] Fig.14 , 15 and 16 depict examples of virtual mirror user interface (UI) flows that may be sequenced together to simulate a mirroring experience;

[0036] Fig.17 An implementation of a proposed usage process is described;

[0037] Fig.18 One embodiment of a proposed usage flow for delay mode is depicted.

[0038] Fig.19 One embodiment of a GUI for delay mode is depicted.

[0039] Fig. 20 An embodiment of a digital mirror system for implementing automatic switching of idle, real-time, and delayed modes is described.

[0040] Fig.21An embodiment of a fully contained unit is shown that can be attached to the back of any television screen and convert the TV screen into a digital mirror.

[0041] Fig. 22 yes Fig.21 A side view of the unit, showing the attachment to the back of a television.

[0042] Fig.23 and 24 A flow chart illustrating a process that may be implemented by any of the embodiments disclosed herein to enable automatic display of delayed video. DETAILED DESCRIPTION

[0043] The following examples illustrate some embodiments and aspects of the present invention. It will be apparent to those skilled in the art that various modifications, additions, substitutions, etc. may be made without changing the spirit or scope of the present invention, and that these modifications and variations are included within the scope of the present invention as defined by the following claims. The following examples do not limit the present invention in any way.

[0044] Embodiments of the present invention address issues related to color variation, multi-level user identification, and body measurement. A virtually calibrated camera is provided that can provide augmented reality features such as color, texture, shape manipulation of body items or backgrounds in virtual mirroring or video conferencing applications. It can also perform body measurement, body monitoring for commercial, security, and healthcare applications.

[0045] Specifically, accurate segmentation of objects and / or elements in an image or video is one of the greatest challenges in the art. Known techniques for solving this problem involve a clear tradeoff between accuracy and speed. Prior to the present invention, there was no satisfactory solution to this problem. Using infrared (IR) cameras that utilize three-dimensional (3D) depth measurement is an industrial approach; however, the quality of this approach is affected. For example, edges, image distortions, and / or low-resolution misalignments between infrared cameras and conventional cameras hinder successful and convincing segmentation. Both color separation and edge detection techniques are demanding from a computer resource perspective and do not produce consistent performance. Some techniques for extracting users or objects use, for example, a green background that is easily recognizable, but are not applicable in applications where a green background is not available.

[0046] Once a model of the item or items being manipulated is acquired, the second problem is to create a smooth rendering stream that will update the model and render it smoothly back to the original video or into another augmented reality video with the new information. The embodiments disclosed below achieve a relatively fast smooth rendering stream with high quality so that manipulation can be performed in real time, semi-real time, or faster, depending on the requirements of the specific application. In addition, the embodiments are suitable for manipulating items (color, texture, shape) in real time on a camera mainstream feed or offline on a user recorded video.

[0047] Based on the model, mask, and / or alpha channel of each object from the video, knowing the user's motion, different objects that fit or don't fit the shape and form of the object can be installed in a convincing manner. The disclosed embodiments can be distinguished from the prior art, which attempts to do everything in a virtual way. Embodiments can include manipulation with masks to cover other changes, such as different cloths, changes in the user's face, changes in the user's body shape, eliminating the user from the image, and the like.

[0048] Some embodiments may include multi-level user authentication. Specifically, embodiments may include improvements, user learning, and adaptation of face recognition. Known face recognition methods may utilize algorithms that may be very sensitive to face poses. In the present technology, a smooth experience of face poses may be created to adapt to different camera positions and angles.

[0049] Various embodiments may include code recognition, such as quick recognition (QR) codes, one-dimensional codes, hidden codes, etc. The embodiments may be suitable for discovering codes with relatively small images compared to images in projection situations and in other optical distortions from a relatively long distance. In addition, wireless recognition may also be included, such as NFC, wireless broadband, microwave 3D, body access network (BAN) chips, etc. Wireless recognition can be from a mobile device, watch glasses, microchips, or other devices carried on a device or microchip. Other body measurement technologies may include fingerprints, body recognition, body shape, eyes, palm recognition, X-ray correlation, body temperature, body pulse, blood pressure, etc.

[0050] Another disclosed feature is fabric recognition, which can identify items and clothes that users are wearing. For example, it can include hidden codes or information in the fabric.

[0051] Other features and embodiments may include: a microwave scanner in which a microwave sensor is integrated into the screen; anthropometric features including, for example, 2D and 3D body line detection or 2D / 3D model / mask generation from video; analytical conclusions from measurement results, such as body shape, origin, gender, age, recommended clothing size, recommended diet or treatment, etc.; body line tracking over a period of time for diagnosis, change identification, user identification, and the like.

[0052] Regarding hardware, various embodiments may include single or multiple cameras with all types of sensors (including but not limited to CCD, CMOS, IRCMOS, etc.) and format protocols (including HDMI, FireWire, USB, IP, HDBaseT wireless, etc.) and any available resolution (including HD, conventional SIF, 4k, 8k, etc.). Displays may include single or multiple screens or virtual screens and conventional screens of all sizes, shapes and proportions. Displays may include mirror-like screens, smartphones or tablets, projectors, holograms, three-dimensional screens, Google Glass, projector head-mounted displays (HMDs), glasses (with switchable blur layers), etc.

[0053] In the disclosed embodiments, the camera can be located in any position. It is best practice to place the camera above the screen facing the user. Additional positions may include at the bottom of the screen, to the side of the screen, or behind the screen (if the screen is a two-way screen). In the case of 3D, two cameras may be placed, for example, at the top and side by side, which may provide better results and eliminate computerized methods. In addition, once the user moves and / or rotates, the 3D can be operated by a single camera.

[0054] Embodiments may include computerized methods to include full or partial body and different scaling according to each user selection (eg, correction of head and / or eyes, correction of viewing direction / viewing point).

[0055] Each of the above embodiments may be provided on a still hard copy image 2D / 3D. Each of the above embodiments may be provided on a still digital image 2D / 3D. Each of the above embodiments may be provided on an analog / digital video recorded by an analog / digital camera. Each of the above embodiments may be provided on an analog / digital video recorded by a frame grabber HDMI (or equivalent), IP, USB, FireWire (wired or wireless link), etc. Each of the above embodiments may be provided on a digital streaming video that may be transmitted to a computing device via any suitable communication method, such as USB, HDMI, IP CAT-5 / fiber, wireless LAN, FireWire, HDBaseT, any combination of the above on a single cable or multiple cables. Each of the above embodiments may be provided in a case where a computing device is contained in a camera, in a screen, in the cloud, in a computer including a workstation, server, etc.

[0056] Each of the above embodiments may be provided with the computing device distributed among the system elements, for example the computerized method may be contained partially in a camera, partially in a video acquisition element, partially in a screen graphics processing unit (GPU), partially in the cloud, partially in a user smartphone device, or any suitable combination of these. Each of the above embodiments may be provided with the computing device contained in a smartphone, tablet, laptop, display, camera, HMD, Google Glass, etc.

[0057] Each of the above embodiments can be provided with the video pre-processed to extract 3D models of selected items, where the pre-selection can be done manually by the user or automatically by applying rules for selection. The model extraction can be performed locally by the DSP / CPU next to the camera, in the cloud, or in a user application on the user's device. For single objects or still images it is more efficient to extract the model in the GPU (Graphics Processing Unit); in this case a conventional CPU is sufficient.

[0058] Each of the above embodiments can be provided in the case where the rendering is being performed locally in the augmented reality station, in the cloud, or at the user's mobile device. For a single image or very short video, the rendering can be done in the GPU; in these cases, the CPU can provide sufficient performance.

[0059] Each of the above embodiments can be provided in the case where enhanced rendering techniques are performed to improve the video quality, for example, interpolating objects and images to a higher resolution and sampling after combination to smooth edges and eliminate aliasing effects. The present invention can, for example, eliminate flickering edges by smoothing the model frame by frame, by eliminating the variation allowed per frame on the edges, by smoothing the edges of each frame, by applying an average filter on a single frame or multiple frames, etc. Additional improvements can be achieved by increasing the frame rate and applying additional smoothing techniques on the increased frames to smooth the effects that would occur at the original frame rate.

[0060] Each of the above embodiments may be provided whether or not the screen resides near the camera. For example, the screen may be located in a control room. The camera stream or recorded information may be accessed via remote control, via other locations that directly utilize physical infrastructure, or through the cloud.

[0061] In one embodiment, control of the station functions can be achieved through gestures, eye movements, palm movements, finger movements or brain control. The present invention can be controlled with additional accessories such as laser beam pointers, voice control, wireless or ultrasonic tracking, etc.

[0062] Provided below are several examples of other applications that can utilize the techniques of the present invention. Specifically, basic video transformations can be used to create reliable virtual try-ons.

[0063] Basic video transformations can be used to create a reliable video conference where the user / users on the other side of the line can see the participant as if the participant is looking directly at them rather than seeing the image from the camera's point of view. Alternatively, the transformation can be used to create an image as if a user is looking into the eyes of another user to provide a convincing, real-time experience.

[0064] Basic video conversion for augmented reality can be provided for a variety of uses including, but not limited to: body manipulation and resizing, body measurement, medical diagnosis / monitoring / training and / or tracking, gaming, fittings, dating, meetings, security, full body simulators, dance training, hairdressing, makeup, general training, tourism, etc.

[0065] Real-time and off-line appearance registration and comparison can be provided so that the user can track appearance changes over time and over years. These processes can be provided for a variety of purposes including medical registration, which can use multiple / different sensors, such as for body temperature, blood pressure, etc., between multiple examinations in a given period.

[0066] Embodiments may be applicable to video conferencing or gaming where a user wants to change their appearance (e.g., skin color, dress, face, etc.) According to the invention, the model may allow for a smooth change to the desired appearance, or even create an image where the user is eliminated from the video.

[0067] Embodiments may be used for security applications with accurate facial recognition on calibrated eye match (eye match) when the camera is at the projected view of the user's eye (see, U.S. Application No. 13 / 843001), with dynamic QR (Quick Response) code recognition (e.g., via user authorization and management of a mobile app, or badges, QR codes may be generated for a mobile app from the cloud and may be verified by user authentication in the app), with full body recognition, scanning and learning, with item recognition (e.g., clothing, uniforms, badges, etc.), with wireless sniffers (e.g., wireless, cellular, ultrasonic, etc.), with APIs to security databases, with positioning systems, with public information, and the like. Embodiments may be used with threat prevention actions based on recognition of specified actions (e.g., lock door, turn light on / off, release fog, target machine, such as electric shock, bullet, paper, etc.). Due to the geometry of the calibrated image, the location of the target in the image may be measured very accurately and may be used to perform precision attacks on the target without posing a threat to others.

[0068] In some embodiments, the main camera stream can be manipulated in real time to change the user appearance, or completely eliminate it from the video or image. In this case, the camera DSP or a separate DSP or GPU function device needs to acquire the camera stream and process the camera stream via a remote control center and package it again as the camera main stream. In one embodiment, parallel computing (sniffer) of the camera main stream is also an option.

[0069] In some embodiments, the calibrated camera / cameras with / without supplemental information from other sensors can be used for patient diagnostics and patient monitoring. For example, but not limited to, the present invention can be used to monitor and alert skin problems such as skin cancer melanoma, breast changes that may suggest further testing for breast cancer, heart rate and blood pressure measurements, patient temperature, user halos, skin and nail record monitoring changes, eye diseases, rotation, bone, muscle and body flexibility trends and changes.

[0070] We now turn to a detailed description and conceptual block diagrams related to embodiments of the present invention.

[0071] Figure 11 is a system block diagram of an augmented reality platform for supporting live or recorded video / images. The system may include one or more (1:n) input devices 101, including video cameras, still cameras, infrared cameras, 2D cameras, or 3D cameras. The input device 101 may be adapted to send information to one or more machine vision augmented reality modules 102, 103, 104, 105, 107, 108, and 109. The one or more machine vision augmented reality modules 102, 103, 104, 105, 107, 108, and 109 may be adapted to send information to one or more (1:m) screens 106. The one or more machine vision augmented reality modules 102, 103, 104, 105, 107, 108, and 109 may be adapted to send / receive information to and from an interface or user interface module 110. The interface 110 may be adapted to send / receive information to and from one or more of a cloud, a network / store, or a user device (e.g., a smart phone or tablet).

[0072] The one or more machine vision augmented reality modules 102 , 103 , 104 , 105 , 107 , 108 and 109 may include an image capture module 102 , an eye matching transformation module 103 , an augmented reality module 104 , a video / still image recording module 105 , a trigger event module 107 , a control element module 108 and a factory calibration module 109 .

[0073] The image capture module 102 may include one or more of the following features: enhancement filters, format conversion, video frame segmentation, image cropping, image resizing, image stitching, etc. The image capture module 102 may be adapted to send information to the eye matching transformation module 103. The image capture module 102 may be adapted to send / receive information to / from the trigger event module 107.

[0074] The eye match transformation module 103 may be adapted to apply the correct mapping on the image to match the camera viewpoint with the theoretical mirror viewpoint (user eye reflection) and fill in the blank pixels (if present after mapping). The eye match transformation module 103 may be adapted to send information to the augmented reality module 104 and / or the video / still image recording module 105. In addition, the eye match transformation module 103 may be adapted to send / receive information to and from the control element module 108. In addition, the eye match transformation module 103 may be adapted to send information to one or more screens 106.

[0075] The augmented reality module 104 may be adapted to, for example, provide virtual color and texture replacement, virtual fitting, object insertion, etc. The augmented reality module 104 may be adapted to send / receive information to / from the control element module 108 and / or the video / still image recording module 105. Furthermore, the augmented reality module 104 may be adapted to send information to one or more screens 106.

[0076] The video / still image recording module 105 may be adapted to record a single image or a short-term capture based on software control. The video / still image recording module 105 may be adapted to send / receive information to and from the control element module 108. In addition, the video / still image recording module 105 may be adapted to send information to one or more screens 106.

[0077] The trigger event module 107 may include one or more of the following features: identifying the user in front of the mirror, face recognition, user gesture command recognition, item recognition, distance measurement, user body measurement / assessment (including, for example, height, age, weight, ethnicity, gender, etc.), and calculation of the user's theoretical viewpoint in the theoretical mirror. The trigger event module 107 may be adapted to send / receive information to and from the control element module 108.

[0078] The control element module 108 may include one or more of the following features: control and management for setting up the camera to optimize quality, control and management for setting up other hardware elements, control and management of the interface between the algorithm module and higher-level code / application / user interface, and control and management of pushing calibrated data from the factory to the algorithm element. The control element module may be adapted to send / receive information to and from the factory calibration module 109.

[0079] The factory calibration module 109 may be adapted to define a mapping transformation between the camera and the user's viewpoint in front of the screen. Furthermore, the factory calibration module 109 may be adapted to calibrate the image based on distance, special position, user height, or any combination thereof.

[0080] Figure 1 The following description merely represents an example of one embodiment of the present invention; other processes or functions may be distributed between modules, representing additional embodiments as part of the present invention. The present invention proposes two methods to enable augmented reality functions (real-time and offline). Both methods load real image or video data, either in real-time or after processing via, for example, eye matching transformation module 103, into augmented reality module 104.

[0081] One feature is that the user can manually or automatically define (via, for example, interface 110) rules regarding which items the user wants to process and manipulate and what the expected end result is, for example, automatic rules can be some rules like finding a user's shirt that can be changed into a different color, or the user selects a shirt via manual selection and manual color selection using a touch screen. The selected objects can then be processed and extracted / segmented out and recorded to a database linked to the original recorded video or image. The augmented reality module 104 can then process the model / mask in real time at a given frame rate, which can be at a lower or higher frame rate than the original and at the same or different size than the original. Some applications do not require (but can still include) real-time augmented reality, such as trying on clothes when the user wants to see himself change (one or more options). Once the objects extracted from the real-time scene are saved, it is easier to render multiple variations (color, texture, size, etc.). In addition, it is easier to perform longer processes more accurately with higher quality and use processes that can produce more information, such as user movement, body measurements, and quality based on frame integration, etc. For other applications that require real-time processing, such as video conferencing, the second option is preferred. With the second option, the performance should be adapted to be as close as possible to the actual frame rate of the camera to avoid introducing delays / lags that could produce frame drops and poor performance. If delays / lags are encountered, they can be managed in a way that produces smooth video without frame lags. This management can be achieved using a buffer, image timestamps plus some delay, or rendering using timestamps and frame rescheduling.

[0082] For video input, it is highly recommended that the rendering process is done within the DSP or GPU device to avoid introducing latency into the video. The creation of the mask / model can also be done in the CPU for the first option (non-real time). For the second option, the mask / model can also be executed and calculated in the GPU / DSP.

[0083] In the trigger event module 107, some trigger functions may be fully automatic, for example, if a face detection or presence detection is made, processing may be initiated. Some triggers may be performed in a semi-automatic manner from the user interface module 110, which may include any means for controlling a computerized device. Some functions of the trigger events are to calculate image transformations based on geometric information, calibration and / or real-time user tracking, such as the user's position, eyes, head, hands, position, movement, etc. Tracking may be accomplished using one or more techniques, such as background substations, pattern recognition, color segmentation, etc. Transform tracking calculation functions may also be implemented in other modules.

[0084] The control element module 108 can be adapted to configure system settings, camera device authentication, etc., and can also provide information from the tracking transformation function to the actual geometric transformation module or augmented reality module, etc.

[0085] With the factory calibration module 109, some of the information needed to calculate the transformations to be applied to the image / video may also be generated during the factory calibration or may be calculated based on additional information on the actual orientation of the camera in the field, such as height above the floor or table, etc., 3D viewpoint, lens field of view (FOV), etc. The factory information plus the actual geometry of the implementation may be processed and passed to the correct elements in the system that will use it for better calibration and accuracy.

[0086] In addition to being applied to a local screen for mirror applications, the present invention can also be used to send video and devices to transform it or transform the actual manipulated and transformed video to another address in the cloud or locally such as one-way or two-way video conferencing.

[0087] Figure 2 An example of an augmented reality module is depicted, which corresponds to the augmented reality module 104 described above. Specifically, the augmented reality module may have the function of enabling users to virtually dress themselves. In this embodiment, the system obtains input images or videos from, for example, eye matching computerized methods 201 or from any other image / video source (e.g., user smartphones, security cameras, Google glasses, mobile cameras, or stationary cameras). Additional embodiments may include additional geometric information that can help calculate additional information such as user height, line of sight, etc. If the user video or image comes from the eye matching module (calibrated image / video), a more complex model that can be used for body measurements, object posture, size, high-precision orientation, etc. can be generated. The additional information that can be calculated by the calibrated object or video can be used for object adaptation, object replacement, and inserting new objects into frames / videos.

[0088] The selection module 202 can obtain selection information from the interface 206 manually by a user (X, Y or object name) or automatically by a selection method (eg, a mechanism that can automatically detect predefined objects such as pants, skirts, etc.).

[0089] Module 203 can obtain the position and sample the color (or the average color of the object, which can be more than one color). Module 203 can use this information to generate a black and white mask, which is first used to generate a shadow and color mask for a 2D or 3D texture.

[0090] The technique used for extracting the modules is based on 3D color association or any other technique like the closest Euclidian distance between the object average color and the pixel color that distinguishes the object pixel from the whole image.

[0091] The decision as to whether a pixel is within an object cannot be performed at multiple levels and is not limited to the following examples:

[0092] 1. The color association and the first decision may be based on a Euclidean distance threshold, wherein the Euclidean distance threshold is in the RGB color space or the color space.

[0093] 2. Filtering noise by applying morphological operators such as thinning and erosion, which can improve the decision regarding pixels that are "misclassified" as being part of an object or not.

[0094] 3. Make a decision based on information from previous or next frames or from neighboring pixels in the pixel queue or near the pixel. This step represents the main decision in the process.

[0095] 4. The original selected object distance, which can be used as a threshold.

[0096] 5. Object surface continuity, where some noise can be filtered out if the object surface continuity is known.

[0097] 6. Object edges, where the decision about edges can be improved through edge detection which can be achieved by a high pass (HP) filter or other techniques.

[0098] 7. Decision based on color energy. One of the problems with color segmentation is that colors in low light conditions can be seen as black, and the dynamic range of the decision is significantly reduced. Dark / black pixels can be isolated, and other techniques can be used to determine whether the dark / black pixel belongs to an object, for example, the present invention can determine whether the pixel is within the boundary of an object, or the energy distance from the object STD color change.

[0099] 8. Use previous information on the desired object shape to get better results.

[0100] 9. In the case where the object is a combination of multiple colors or shapes or has a logo or other pattern, multiple color associations and combinations can be used. In addition, any of the multi-level methods specified above can be used to obtain a higher level decision about the object.

[0101] 10. The decision can also be based on majority or on a decision related to neighboring pixels / images as a weighting factor in the decision. In case the decision is processed on the image as a vector, it may be easier to see neighboring vectors in the same row or column, depending on how the image matrix is ​​reconstructed into vectors.

[0102] 11. Estimation of the color STD of an item's material / texture can also add salient information for object segmentation.

[0103] 12. Any combination of one or more of the above steps 1-11

[0104] The mask can be used for rendering as a simple black and white mask. However, in order to create a convincing feel for the object, additional information of the object's texture or appearance can be maintained. To obtain additional important information, the mask can be used on the original frame or video and the RGB or grayscale texture shading or brightness range of the object can be obtained. This information is more accurate and convincing for color changes because it omits the original object's wrinkle texture, shadows, light reflections, material identification marks, etc.

[0105] Model masks can be structured in layers for improved operation. An example of a usable layer structure may be as follows:

[0106] 1. Black and white mask (to segment out the object). Black and white mask is very important for distinguishing between the object and the background or between the object and another element around the object. A variety of techniques can be used to optimize the object mask / boundary determination.

[0107] 2. Object edge mask - represents the edge or outline of an object.

[0108] 3. Red Mask—Represents the red area of ​​the object.

[0109] 4. Green Mask—Represents the green area of ​​the object.

[0110] 5. Blue Mask—Represents the blue areas of the object.

[0111] 6. Texture applied to all color masks - represents the textured appearance of the object.

[0112] 7. Shadow or Luminance Mask - Indicates the shadow or bright areas of an object.

[0113] 8. Material Light Reflection Mask - represents the light reflection of the object.

[0114] 9. Material Absorption Mask - Indicates the light absorbing areas of the object.

[0115] 10. Masking from other sensors like IR, microwave, depth, ultrasound, ultra-band, etc.

[0116] 11. Layers similar to those described above.

[0117] Once the mask model has the required information, the rendering module 204 can be used to modify the specified layer or multiple layers and regenerate objects outside multiple layers in order to change the color or texture, resulting in an extremely realistic rendered video 205. For example, if a user is wearing a red shirt, the red mask can be replaced with a blue mask to render the user wearing a blue shirt. Since all other masks remain unchanged, the blue shirt will be rendered with all the shadows, brightness, reflections, textures, etc. of the red shirt, thus presenting a very realistic blue shirt with the same folds as the red shirt and matching the same body contours. The influence of some layers can be introduced by multiplication or by adding correction layers to the frame. Subtraction and division can also define the relationship between layers. Other techniques that allow the operation of more complex projects include registration techniques, which can expand / transform objects or materials based on several points to fit within the boundaries of the manipulated object.

[0118] The module takes the original video / image, the multi-channels of the modeled mask and the desired changes. The desired changes can be any combination of color, brightness, material, texture, logo / label, printing, fabric, etc.

[0119] In one embodiment, the desired changes may be outside or inside the original object boundary, and a modified mask for the new object boundary may be generated to replace the original mask model.

[0120] In one embodiment, the desired change is a fabric with a specific orientation, texture and color that can be inserted in a specific orientation. In this embodiment, the material orientation can be modified and applied accordingly.

[0121] In one embodiment, the desired change may be another user's try-on mask. Using registration techniques, the other user's try-on mask may be applied to fit the new item to a similar object of the user for the try-on. For example, one user is measured for a shirt and another user is measured for a jacket to show the first user what the second user would look like wearing the jacket. The method may use the registration techniques that fit the second user's jacket to capture and adopt the correct body line shape of the first user, which may produce a more convincing fit. One advantage of the method is that the jacket will look very real because it is rendered to maintain all of the second user's textures. To produce these results, the method may register the orientation, position, and size of the first user.

[0122] In one embodiment, the mask can be used as an indicator for virtual object registration. In one embodiment, the mask plus the background recording in the correct transformation of the user can be used to eliminate the user / item from the graphic / video.

[0123] In one embodiment, a selection of a single object or multiple (1:n) objects that need to be modeled may be obtained. A mask may be generated from each frame of the video. If the user turns, a 3D or partially 3D frame-by-frame model may be generated. Different perspectives may be obtained from the frame-by-frame model and used to generate a 3D model that includes some or all of the user's movements. This information may then be used to create a more convincing virtual try-on. That is, the method of the present invention may utilize the user's own movements during the formation of the model.

[0124] In one embodiment, rendering can be performed in GPU, CPU, cloud GPU or cloud CPU. Input elements to be rendered can come from CPU, from user database in cloud, or from active links with inventory / any other database / 3D printing, e-commerce database, social database, etc.

[0125] Item color changes can be based on actual try-ons on site or in the workshop and available stock or optional 3D printing for user customization. One advantage over regular virtual try-ons is that the item is shown wrapped around the body as it would in reality, as all wrinkles, shadows, etc. will be transferred with appropriate masks. This is a very important feature for many purposes. The user can feel how the item looks on him, how it will affect and change his body shape, etc.

[0126] In one embodiment, accessories or any other items can be added by understanding the dynamic motion and mask model of the related objects. In addition, the background can be enhanced to change or create different environments through the same technology. Once all the necessary objects are marked, the required objects can be masked and the combined mask can be used to change the background.

[0127] In one embodiment, the rendering module can render objects using enhanced rendering techniques that interpolate objects and frames to a higher resolution, can combine objects at a high resolution, can smooth edges, and can extract objects back to the desired resolution to integrate into the frame with better quality. Additional techniques include acting directly on object edges by averaging pixel values ​​with some weighting factors to better blend objects with the background color.

[0128] Figure 3 An augmented reality module is depicted that can replace the color, orientation, and texture of body parts, items, or objects in the foreground or background of an image. For example, the module can add hair to a user, change the color of the user's eyes, skin, and hair, change the posture of the eyes, and so on.

[0129] Modules 301, 302, 303 and 306 may work in a similar manner to modules 201, 202, 203 and 206 of the general augmented reality module described previously. Module 304 is able to directly calculate or obtain additional information (e.g., head pose or body movement direction) from the eye matching module or from module 307 using a dedicated detector of the 3D motion and orientation of the item, and may use this information to modify the desired body parts, for example, obtaining the head pose will allow the eyes to be corrected for turning by modifying the eyes in the mask / model to the desired direction. Additionally, head detection may allow the addition of hair, hats, etc. in the correct orientation.

[0130] In one embodiment where the size of the desired mask is smaller than the original mask, module 307 may be performed or delivered in real-time or offline in an eye-matched transform of a recorded background image or video. The background-transformed image or video can render body parts or items with foreground or background manipulation, for example, in the case where it is desired to eliminate a user's head or upper torso from a video. The transformed background image or video can be applied over the captured head and upper torso mask, and the result can be rendered using the original transformed or untransformed video. The result can be a rendered video 305 without the user's head and upper torso.

[0131] For example, in more complex cases, one may desire to depict a given garment of a shorter length. The mask manipulation in module 304 may require the creation of a shorter mask for the new garment, and the difference between the original mask and the manipulated mask may be the new mask used for the manipulation. In the new mask, some portions will be estimates of the user's legs, and some portions will represent the new visible background using the shorter garment length. The new mask may be divided into legs and background, and the new rendered object may take a combination of the background image and the predicted legs to create a new rendered item. After the modified item is rendered into the video, the result is that the user is wearing a shorter garment. A variety of techniques may be used to predict how the legs will look within the available area.

[0132] Figure 4 A computational method for creating a color and texture swap model, which may be referred to as Colograma, is described. This technique focuses on parallel computation that can support a large number of users and a large number of frames / videos, as opposed to the very high quality color swapping techniques that can be found in software programs such as Photoshop. Such methods can take a large amount of time and may not be practical to perform on an arbitrarily large number of user images or videos. Figure 4 The description is only an example, and any derivative examples of the described process are part of the present invention.

[0133] One challenge in changing the color of an object in a video or image is to accurately identify the relevant pixels of the object. In video files, speed is a limiting factor in the available transformations. Figure 4 In , a simple example of a method for segmenting / extracting objects from a video is described.

[0134] In 401, an image or video to be modified is received. In 402, frames of a color image or video are converted into linear vectors, which is optional, although image vectorization can greatly speed up processing time. In addition, in 403, brightness effects are removed. There are many techniques to remove brightness effects. In this example, energy is averaged to each pixel in an XYZ color space by dividing each pixel by the sum of XYZ. For example, using the chromaticity coordinate system of the RGB system (x r ,y r ),(x g ,y g ) and (x b ,y b ) and its reference white coordinate (X W ,Y W ,Z W ), a 3×3 matrix can be used to transform RGB into XYZ. In parallel, in 404, object selection is achieved by selecting all points K(x, y) that are considered to belong to the object to be transformed. K is the number of objects / regions with distinguished colors that can be separated from the background or other objects. Then, in 405, each point K(x, y) undergoes the same transformation as performed in module 403. In 406, k iterations are performed to find the color of each pixel and find the closest color of the object. In this technology, K is equal to or greater than 2 (K> / =2). For each K, a 2D or 3D Euclidean distance is calculated. The minimum distance and K value are saved. This operation can be completed at once on all pixels in a fairly fast process.

[0135] dist=sqrt((X-xi(k)).^2+(Y-yi(k)).^2+(Z-zi(k)).^2)

[0136] After K iterations, the labeled image can be obtained. The Euclidean distance "dist" is only one example of a method for calculating the distinction between colors; there are other methods for calculating the distance between colors, for example, color distance models based on human perception of color (hue, saturation and brightness), advanced calibration techniques that match the sensitivity and ability of the human eye to separate colors, such as in CIE76, CIE94, CIEDE2000, etc., or any combination of IR / 3D depth cameras with histogram stretching, color integration over a period of time, or any other method for improving color detection (module 411). Applying or canceling (crossing) additional information from module 411 can occur at the distance comparison level 406, can occur at the end of model creation 409, or any combination, depending on the characteristics of the additional information (deterministic, statistical, time-varying, etc.).

[0137] In addition to chromatic aberration, other techniques that can add information about the object can also be used to improve the decision, such as: area likelihood (a given pixel needs to have neighboring pixels or a certain number of pixels), area characteristics, boundary filters to isolate object boundaries before making a final decision, depth information (which usually requires matching the outline of the depth information with an end image of the object in 2D or 3D), temporal integration to determine whether a pixel is within an object area over multiple frames, and so on.

[0138] Module 407 is an example of one embodiment of how to distinguish the desired color from other color spaces. In module 407, all pixels with a distance greater than a threshold are set to zero as non-relevant (pixels of a color different from any of the colors 1 to k), and 1 is assigned to all relevant pixels, thereby generating a binary mask. In module 408, a black and white filter can be used to remove noise and smooth object shapes. Other techniques can be used to improve the determination of whether a pixel belongs to an object. As a result, for all relevant colors, the index starts from 2 up to K+1.

[0139] Block 407 is an example of where it is desirable to separate a particular color or colors. In this case, all indices except the desired one may be zeroed. The process is as follows: zero out all non-relevant indices, get background and non-relevant color values ​​equal to 0, and select the desired color object marked equal to 1. If there is more than one color in the object, 1 may be assigned to any selected index 2 to k+1, and 0 to the other indices.

[0140] In module 409, the resulting black and white mask is applied to the original color image to obtain a 3D model for color and texture changes. The model can be a grayscale 2D alpha channel or a 3D color space. Module 410 can obtain a two-dimensional or three-dimensional model of the object. In the case of video from a single camera, the 3D model can be obtained even if the user moves in front of the camera, for example, turns around in front of the camera. In this case, it is also possible to obtain object measurements in multiple cuts to estimate the user's 3D body curves.

[0141] The following is an example of a 2D input to the Model Builder where we wish to create Fig. 9 A model of a user's shirt is shown. Fig.10 Here is an example of a 2D model of an upper shirt. The texture of the shirt is preserved in this way, so it is relatively easy to manipulate the color or texture or even change the borders of the model to create different objects.

[0142] Models based only on color differences are not perfect in terms of quality, so additional information and techniques can be used to improve the quality of the object model (see, module 411). Additional techniques like interpolation and decimation or edge smoothing can be applied after processing via module 410 to improve the quality of the model.

[0143] Figure 5 Described is a computational method for creating a model for color and texture exchange, which may be referred to as a Colograma.

[0144] In module 501, the image is analyzed using body line and body mass index (BMI) analysis by first using a method similar to the eye matching method described in U.S. Patent Application No. 13 / 843,001. In this method, the image can be rearranged to compensate for distortion caused by the setup geometry and optical distortion of the camera. Examples of setup distortion factors include the distance of the user to the camera, the height of the camera, the projection angle of the camera, the optics of the camera's FOV, and other optical distortions. After the image is corrected, each pixel area represents an area (cm 2 ) is roughly uniform across the image. Processing can be done in the camera's CPU, GPU, DSP, in a local computer close to the camera, or in a remote server.

[0145] Module 502 is an optional element that can convert a 2D image into a vector conversion, which can speed up processing. The size of the vector is width times height times three pixels (for RGB). The image input to module 501 can also already be a vector. It can more easily perform some types of image processing on a 2D matrix, while other types of image processing are more suitable for vector images.

[0146] In module 503, it is easier to perform color operations to remove light and saturation effects on the vector image. This is done in a similar manner relative to 403 above.

[0147] In module 506, the image can be passed through an edge detection filter that can emphasize color changes and enhance the visibility of body lines. Edge detection can support the edges of body lines and therefore can support multiple edge directions. For example, an edge filter with vertical, horizontal, and + / -45° capabilities can result in good body line edge detection.

[0148] Modules 504 and 505 can provide additional information to the algorithm that can aid in image analysis such as the user's gender, age, race, etc. and statistical ratios associated with that information. This information can be used later to focus searches on body parts. Height measurements can be used with supplemental body proportion information to focus searches on specific body parts such as chest, navel, waist, hips, etc.

[0149] Module 507 can provide head detection. There are many techniques for head detection. For example, an edge-emphasized image can be taken, flipped along the length of the body, and the association between the images can be established. The peak of the association can indicate the center mass of the body. Another alternative is to perform centroid center mass calculation only at the edges. If the light on the user is not uniform enough, the centroid center mass calculation is not accurate enough; however, this technique can be faster. Other techniques can be based on pattern recognition, eye, nose and head shape detection. Once the central mass is obtained, the relevant window can be cut to identify the body from the image. In addition, the emphasized edge at the end of the head can obtain the height of the user in pixels. Since each pixel of the image has a uniform length weight, the total height can be calculated accordingly. It is assumed that the module 501 transformation will produce a consistent manipulated image on a configuration (e.g., 1-5 meters (m)), where the actual height of the user on the screen in pixels will be proportional to the actual user height. If this assumption is not completely true for all distances, additional factors can be added.

[0150] Module 508 may measure the size of the body part in pixels. The technique may be additional processing on the focus area to further emphasize the body part. Fig.13 An example of the results of additional processing that removes the hands from the body and points to different intervals of where to find specific body measurements is depicted.

[0151] Measurements in pixels can be converted to centimeters (cm) based on the predicted curve, for example, the neck can be modeled as a cylinder so that the neck width measured in pixels can represent 2x the radius in pixels, with the radius in pixels converted to each calibrated centimeter. The human chest has a more oval model, so the conversion will be a bit different. If additional side measurements are available, such measurements can be added to provide more accurate information to the model.

[0152] In block 510, measurements may be inferred from the user's body shape, for example, if the hips are wider than the chest and waist, then the user is "pear" shaped (see, Fig.11 and Fig.12 ). In addition, some analysis can be concluded as follows:

[0153] 1. Physical conditions, health risks, etc. of heart problems;

[0154] 2. The type of clothing that best suits the user’s body shape;

[0155] 3. Historical comparison of user body trends;

[0156] 4. Focused advertising based on user body type (e.g., mesomorphs / endomorphs can get focused coupons for optimal nutrition for their body type);

[0157] 5. Physical recognition; and

[0158] 6. Body diagnostics can monitor body changes over time (rotational condition, flexibility, potential masses, etc.).

[0159] Can from Figure 4 or directly from the object model described in Figure 5 The image BMI analysis described in the figure is used to obtain or estimate the user's BMI information.

[0160] Figure 6 This is an example of a security system application for multi-level user learning and authentication. This is just an example of a possible process implementation; any combination of processes or division of different functions between graphical modules is part of the present invention.

[0161] As shown in the previous figure, a still image or video from device 601 can be input to image capture module 602. Image capture module 602 can be controlled by trigger events such as those previously described and processed by trigger event module 607, which can provide tracking information and rules to initiate the image capture process. Specifically, image capture module 602 can collect images from input 601 and push the images into eye matching transformation module 603 together with additional tracking information (real-time geometric measurements). Based on the tracking information and event requirements, eye matching transformation module 603 can calculate the transformation to manipulate with camera viewpoints, perspectives, etc. to create a calibrated scene image. In addition, the eye matching transformation can be performed locally in GPU / CPU / DSP, in the cloud, etc. In one example, high-level information on the user or the geometric structure of the scene can be provided to the tracking algorithm and can play a part in the calibration. Additional information can be provided as a temporal geometric information, such as user height, distance between eyes, etc. In this case, the additional geometric information can be used to calculate the required transformation. The difference between the distorted image and the known geometric structure can be used to calculate and correct the transformation.

[0162] The calibrated output from the eye matching transformation module 603 can then fade into the security scanning module 604, which can also be referred to as a scene analysis module. The security scanning module 604 can be adapted to learn one or more of the following: body line or body curve scanning based on a variety of techniques, such as edge detection, statistical edge detection, infrared cameras, microwave sensors, 3D cameras, single cameras, and multiple clips when the user is turning. A full human 3D model can also be acquired as the user turns in front of the video camera, and the user's clothing can be extracted as shown. Figure 4 The Alpha Channel described, i.e. Colograma, etc.

[0163] Other learning or authentication methods include, for example, cloth detection, creating item identification marks in a database to be sorted. The database can be updated from the Internet, by scanning clothes from multiple databases and e-commerce stores or by actively scanning and updating information by the operator, for example, security personnel uniforms, fabric types, etc. can be entered.

[0164] Furthermore, the combination of eye matching and face recognition can allow the operator to mount the camera above the user's height, for example, on a door facing downwards at about 30-45 degrees; such a mounting allows the user to move freely under the camera. In this case, for example, the detected face recognition range is about 1-5 meters in front of the lens. This is an advantage, because known face recognition systems are not very accurate if the user's head is scanned at an angle of + / -15 degrees.

[0165] Additional body authentication sensors may include voice, smell, palm, fingerprint, eyes, DNA, bones with X-ray sensors or ultrasound, teeth, hair, impression, color, eyes, blood, aura, temperature, skin landmarks, ears, etc.

[0166] Video, still images or analyzed data can be recorded directly from the camera using the video / still image recording module 605, from the eye matching transformation module 604, from processed images or sensor output data.

[0167] Visual video / images or analyzed data can be displayed in front of the user's screen (full body or smaller), transmitted on the cloud or directly transmitted to the control center to be displayed on the screen 606 or transmitted to the module for further processing and analysis.

[0168] Modules 607, 608, 609 and 610 are similar to modules 107, 108, 109 and 110, respectively (see Figure 1 and its related description).

[0169] Figure 7 An example of a security system application for parallel or in-band camera operation is described.

[0170] Figure 7 The functions of the devices in Figure 7 Shown as included in a separate device or as part of the camera DSP functionality; control feature activation can be done wirelessly or via a wired infrastructure. In addition, some functions, such as measurement and tracking, can be supported at remote locations. In addition, the device can be placed in front of the camera and can project the manipulated image on a small screen to the camera.

[0171] The camera mainstream may be provided to the device or the device may be adapted to record and sniff the camera mainstream. After processing, the device may be adapted to send parallel data to an operator.

[0172] The device may have multi-image computing capabilities, eye-matching geometry capabilities such as in the eye-matching transformation module 703, anthropometric and user authentication such as in the security scanning module 704, and augmented reality functions such as in the augmented reality module 706, etc., where images can be manipulated on top of or in parallel with the mainstream, for example, manipulations may include changes in the user's body shape, color, items the user has, hairstyle, complete disappearance of the user, etc. These capabilities are very important for safe use.

[0173] Additionally, the video stream can be provided to another device such as a robot, a head mounted display, a user application, etc.

[0174] The interface that controls the device functions can be on a wired or wireless infrastructure or at a local or remote location. The measurement module can be contained on the device or in the cloud.

[0175] In one example, the function may be completed in real time or offline, for example. For example, the activation of the device may be completed periodically, in response to a triggering event, or manually, as needed or desired.

[0176] The device may support real-time control and triggering of other devices, for example, a projector to change the image a user sees (disguise), or triggering events for accurate threat mitigation devices that require accurate coordination before the camera.

[0177] Video can be shared with additional stations in real time. There are many ways to share video. Some exemplary scenarios are discussed in detail below.

[0178] Share a mirroring experience remotely with one or more other users, where the user sees himself in the local station, while the remote user sees the user in the mirror and shares the experience with him. The remote user can have any suitable device; the main experience is to see the user standing in front of the mirror and doing some actions to watch. Based on eye matching or any other technology, the user will be able to see his own mirror image. Figure 8a A scenario is depicted where the remote user can only see the user broadcasting, send messages to the user, talk to the user and / or be seen in a small window on the mirrored screen.

[0179] Figure 8a Describes the shared mirroring experience in a video / voice conference. Figure 8a As shown, the mirroring experience can be shared in real time with the user / users in the cloud or via another mirroring station. In the first case, the user can see himself in mirror mode, although the video stream forwarded to the cloud needs to be flipped back through an additional mirror to correct the direction of sight on the other side. So, when the user (in mirror mode) looks to the right or left, his eyes / gaze on the other side will move to the correct direction, as if he is looking in the right direction.

[0180] Figure 8b Depicted is a shared full body video conference. In contrast to conventional video conferencing where the camera can be streamed "as is" to the other side, at a mirrored station, the image being sent to the other side needs to be flipped left-right at the remote location. This can be done by flipping the image captured locally or at the remote side.

[0181] exist Figure 8b In , when two or more users communicate with each other at full body stations, the user / users at one station will see that the user / users in the other direction are also in the correct direction. Figure 8bAs shown, the user can see himself standing on the left side of the local screen, and the user can see another user on the right side of the local screen. In this case, the local video has been mirrored, and when the video is streamed to a remote location / multiple remote locations (broadcasting), additional flipping / mirroring will be required.

[0182] Any combination of mirroring, flipping, rotating, etc. of local or streamed video to match the local mirrored and sight-corrected experience of other remote locations is part of the present invention.

[0183] In one embodiment, both stations are eye matched / calibrated to the same geometry and size; each of the stations creates a full body mirror image and sends it to the remote location / locations. A full body experience and eye matched experience will also appear in the video conference, and users will feel like they are looking into each other's eyes.

[0184] In one embodiment where there is more than one user at one location, then the tracking function for each user can be utilized to perform eye matching or body distortion correction, or both, for each user. To eliminate the line effect, in this case, the augmented reality function can replace the background as described above. In augmented reality mode, Colograma or another technology can be used to replace the background with a simple background. Connecting lines can be placed between users to eliminate lines or discontinuities relative to the users themselves.

[0185] In an embodiment where there are more than two stations, the screen may be split to allow multiple users from multiple locations to be displayed at the same time, or multiple screens may be displayed as e.g. Figure 8c Specifically, Figure 8c Multiple mirrored / full body stations with split or multiple screens are depicted. Figure 8c As shown, the order in which the users appear in the mirror allows correct line of sight communication. When the user is looking to the right after flipping, for example, on the remote screen, he will see the user looking to the left at the remote user.

[0186] This is just one example, any number of users near the screen or at remote locations should be organized to allow the correct gaze pattern. In one embodiment, further gaze and eye contact improvements can be used to improve the experience. In one embodiment, the screen can include 3D capabilities, and 3D capabilities for each user on the screen.

[0187] In one embodiment, in addition to eye matching or any computerized image processing on one side, the remote location may have additional computerized functions to manipulate on the far end video based on local user tracking, for example, if the user is closer to the screen, the far end video or image will be processed to look a little bigger and projected to provide a better sense of liveness. Another example is that when the user moves closer to one end, he will see a wider FOV of the far end.

[0188] The present invention includes virtual image control and virtual system features to ease the availability of virtual images. The virtual image control and virtual system may include station operation modes and modes of operation including, for example, on, off, restart, mirror, display, idle, etc. The virtual image control and virtual system may include user authentication and registration, which may be automatic, manual, and a combination of automatic and manual. The virtual image control and virtual system may include optimal operation flows to ease and enhance user experience.

[0189] The virtual mirror control and virtual system may include one or more interfaces to control the mirror, such as automatic gestures, voice, eye pointing, mobile applications, remote experts, local services, etc. The virtual mirror control and virtual system may include a comprehensive combination of interfaces, including the following: playback of recorded video (automatic / manual); advanced effects (project color manipulation, augmented reality functions, game functions, background changes, 3D effects, lighting effects, etc.); screen modes (directional / full screen or split screen display); sharing technologies and methods to share experiences with friends / experts in real time or mid-stream; mobile user applications for controlling display and sharing, such as sharing of personal videos and images taken from the virtual mirror or user's mobile device; retail or enterprise remote for daily management (using advanced control and analysis functions to provide retailer information about the user in real time); and modules to teach users how to operate the mirror (animation, video, voice, visual prompts, etc.).

[0190] Fig.14 , 15 16 depict examples of virtual mirror user interface (UI) flows that can be sequenced together to simulate a mirror experience. For example, when a user stands in front of a mirror, the mirror can track the user once the user enters a designated area in front of the mirror. Based on this tracking, the video engine can calculate a transformation of the image to simulate the behavior of the mirror.

[0191] In idle mode, when there is no user in front of the mirror, the engine can take the default setting (e.g., 2m) and apply the 2m transformation to the camera stream to create a mirror effect on the environment, as shown in the figure (e.g. Fig.14 ). In another embodiment of the idle mode, the mirror can present other videos of models, commercials, etc.

[0192] Outside of runtime, the mirrored screen and / or computing device provided to the screen can be turned off or maintained by a screen saver to eliminate rapid aging of the display electronics. The mirroring station can be enabled to automatically power on. The mirroring station can be controlled locally by remote control or through the cloud to restart and enter normal working state.

[0193] In one embodiment, the trigger area can be specified with a carpet. The carpet can be designed with specific patterns and colors that improve user tracking and detection. In addition, the color of the carpet can improve the video quality. For example, lighter colors can be used to improve lighting conditions and improve the quality of the resulting video. In one embodiment, the pattern in color can change with lighter and darker colors at about 5-10 cm, so that when the user stands in dark or light shoes, the user's shoes will be easily detected and tracked, and the maximum distance error will not be higher than 5-10 / 2. In one embodiment, the carpet may include instructions to the user that inform the user how to operate the mirror. Alternatively, in one example, an operating device can be provided, wherein the user can step on the sensing pad with his or her feet to operate the mirror with legs and feet instead of hands or voice.

[0194] In one embodiment, the mirror can prompt the user to stand in the correct position, for example, when the user gets closer, the mirror can switch back to idle mode or the mirror can cut out the user's legs in the image, etc.

[0195] The mirror can be configured so that when a user steps into the tracking / registration area in front of the station, the video engine can react and begin tracking the object. Based on the object position, the video engine can adjust the video transform to mimic the mirror behavior. Additional inputs for tracking can include user height, spatial position, posture, etc., as described in the EyesMatch process. At this point, the user is still not registered to the system, so additional processing for user identification can be performed. For example, as described in U.S. application No. 61 / 862,025, authentication may include facial recognition, a special card that the user can submit to the system for scanning, the user's mobile device (audio, wireless, QR code), or other physiological registration features. If the user is recognized, their account can be opened and the last record can be displayed, for example, in one embodiment, a thumbnail can be displayed, as in Fig.15 Alternatively, any other image control bar may also be displayed. If the user is not recognized, the user registration process may begin, and then, after a few seconds, a new account may be opened and the mirror may be configured to automatically start recording.

[0196] In one embodiment, the mirror may record for N seconds, e.g., 10-13 seconds is sufficient time for a particular item to appear to make a good impression on the user, and may include, for example, the user turning around.

[0197] In one embodiment, although the mirror can be configured to automatically begin recording upon recognition of the user, if the user does not desire such automatic recognition, the mirror can be configured to save the video, the user can trigger one of the previous thumbnails, the mirror can be configured to switch to display mode and the mirror can be configured to play the video. That is, the user can be prompted to perform a process by which a particular scene is cut by the user and not saved in the system.

[0198] In one embodiment, thumbnails can be added to the mirror, which can be useful when the screen is a narrow configuration, such as 21:9, or any other widescreen configuration. In one embodiment, the thumbnails can be displayed to pop up next to the mirrored stream or in a separate bar below the simulated video stream. This feature can be useful when the screen is wider or proportional than necessary for the simulated video stream.

[0199] In one embodiment, for example, Fig.15 As can be seen in FIG. 1 , a record indicator may also be displayed. The record indicator may include a display phrase (such as “REC”), a red circle, etc.

[0200] In one embodiment, since facial recognition is not completely reliable 100% of the time, additional identification features can be displayed so as to pop up on the screen for an appropriate period of time, such as a few seconds. In the current example, the QR stand GUI is opened and the user can be shown to the mirror a QR badge or QR that has been sent to his special application such as a smartphone, and effectively communicate to the system that the current facial image is associated with the authenticated user. This technology can be used to allow semi-automatic user merging in the event that a particular user has multiple facial accounts. In addition, once the facial information is merged, additional facial measurements / refinements can be performed to improve user recognition over a period of time.

[0201] In one embodiment, which may be presented at the bottom of the screen, an additional password may be associated with the user and the user may employ such password-based authentication and apply it to the registration process. In one embodiment, the employee or the user themselves may enter a phone, email, NFC signal, or any other identifying information and may obtain a link to the cloud on the spot or later. In one embodiment, six (6) thumbnails are presented to the user, but any suitable number of thumbnails may be presented. The size of the thumbnails may be configurable to support a reasonable ratio for the display.

[0202] In one embodiment, additional control functions may also be added, such as starting recording, erasing video, adding video effects such as lighting, color changes, background selection, etc. In one embodiment, all videos may be uploaded to the cloud and may be deleted from the local station after a certain period. In one embodiment, the video may be processed after recording to improve the video or change the video to reflect augmented reality effects or additional effects for BMI / tailoring / body line measurement analysis. In one embodiment, the video recording may include an audio recording of the environment for further analysis. In one embodiment, the video recording may include a WLAN recording of the environment to record the user's MAC and further correlation later, which may be suitable for associating the MAC of the mobile device to the user.

[0203] The present invention includes a system for facilitating user control of the mirror. In one embodiment, if the user is still in a predetermined area and is recognized by the mirror once, as long as he is still standing there and no one else is being recognized, the user will be able to control the mirror, such as playing, starting and stopping images / videos, deleting images / videos, adding augmented reality functions, etc. The user can also control the mirror through gesture control or from his device (such as a smart phone), through a dedicated application or additional control functions as part of a retail application. In one embodiment, the gesture control can enable certain basic functions and the user application can allow more functions to be enabled relative to the gesture control. In one embodiment, an in-store assistant or an assistant available virtually / remotely in the cloud can help the user operate the mirror. In one embodiment, the user can set his own preferences for the UI from his application or from the web. All user-specific settings can be added to the user account, for example, the user can change the time of recording, the number of thumbnails, etc.

[0204] In default mode, users can see their last n videos. The videos may or may not have been acquired at a specific mirror location, but can also be accessed from a central storage location such as from the cloud. Users can set the appearance of local try-on video thumbnails. From their own application, users can see all thumbnails and can touch them or any other selection method to activate the video associated with a specific thumbnail to play.

[0205] In one embodiment, the thumbnails are constructed such as in Fig.15 The thumbnail can be sent to the background while the video is playing, or when, for example, Fig.16 When there is no gesture control recognition as shown in the figure, it is in split mode. Users can also set the screen to split mode, such as Fig.16 shown in . Specifically, Fig.16Depicted is a side-by-side setup where the user can still see themselves in real time on one side of the screen and can play a previously recorded video on the other half.

[0206] In one embodiment, the user's remote control can automatically obtain the identified user information so that a store associate can send the user a link to their account to register or help the user perform additional functions such as changing the color of the clothing or generally changing the clothing.

[0207] In one embodiment, a user can use their mobile device to initiate a video call in the mirror with a remote expert or friend who can give them advice in real time. The call can come from a dedicated application or any other third-party application such as Skype. The mirrored video stream can be fed to the local Skype and the remote user can obtain the mirrored stream in real time.

[0208] Fig.17 Describes one implementation of the proposed usage flow. Specifically, Fig.17 Depicted in Fig.14 , 15 and 16. In step 1701, the mirror is in an idle position, and in step 1702, as the user approaches the mirror, the user's presence is detected, for example, by a change in the image seen by the camera, and the system will start operating in mirror mode. In step 1703, the system will start an authentication procedure to identify and verify the user. For example, in one embodiment, the system uses facial recognition to identify and verify the user, while in another embodiment, the user can use a device such as a smart phone with functions such as WiFi, Bluetooth, NFC, etc. for authentication. If the user has not been identified, for example, is a new user, the user can be prompted to open an account. Otherwise, the user can continue as a visitor, but some functions, such as remote access to stored images, may not be accessible to the visitor.

[0209] At step 1704, user controls are presented to the user on the display screen. In one embodiment, specific control preferences are saved for each specific user and activated once the user has been identified. Otherwise, a general user interface is launched, such as a gesture-launched interface. In one embodiment, where a user has been identified or is attempting to try on several garments in the same session, previous attempts are presented as thumbnails on the side of the main display. When the user corresponds to the thumbnail, the user is prompted to try on the garment. Figure 1When the user raises his hand, the corresponding thumbnail is highlighted for selection, depending on the height of the hand. As the user raises or lowers the hand, another thumbnail corresponding to the height of the hand is highlighted for selection. Then, when the user closes the fist, the highlighted thumbnail is selected and the corresponding image or video is displayed on the main screen.

[0210] As shown at 1705, images and videos of each session can be stored on the cloud and can be accessed remotely using, for example, a smartphone, tablet, etc. Thus, a user can remotely converse with a friend and get their opinion on the clothes the user is trying on. For example, a user can send a friend a link to a recorded trial or a link to a live streaming video from the system. Thus, a user can share a shopping experience with someone who is remotely located.

[0211] Furthermore, as also shown at 1705, since the system is able to identify the user and also calculate the user's parameters, such as weight, height, etc., the system is able to access a database of available items that will be recommended to the user based on these parameters. More specifically, if the user has logged two trials of two different shirts within the same session, the system can interpret that the user is interested in purchasing the shirt and make either an alternative recommendation, i.e., a different shirt, or a supplemental recommendation, such as specific pants that go with the shirt being tried on. Additionally, since the system is able to identify the shirt and the brand of shirt, it can offer specific incentives from the manufacturer, as illustrated at 1706.

[0212] Additionally, at 1706, the user may be provided with the ability to change color without actually changing the clothing item. As described above, by using multiple masks or layers, the system is able to change the color of an item while maintaining a realistic visual of the fabric when it is worn by the user. Since the system can maintain shadow layers, texture layers, reflection layers, etc., the color layer can be changed while maintaining all other layers so that the rendered image retains all of the characteristics of the previous image, with only the color changing.

[0213] To avoid cluttering the experience flow in front of the mirror, additional applications can implement additional features and settings, for example, advanced users who are comfortable with gesture control can enable more gestures, which will help operate advanced features without using the touch interface application. For users who are not comfortable with gestures, such users can be provided with basic automation functions, and the mobile application can be used to operate other advanced features.

[0214] Fig.18is a block diagram illustrating the delay mode operation of the mirror. The delay mode can be implemented in any of the embodiments described herein. It is noteworthy that the delay mode can be implemented in software, hardware, or a combination of software and hardware. For example, the delay mode can be implemented in a computer readable medium storing a program that, when executed by a processor, causes the mirror to implement the delay mode of operation.

[0215] Delay mode can be best understood by referring to a solution to a well-known shortcoming of conventional mirrors. That is, when a person tries on a garment in front of a conventional mirror, it is difficult for the person to see how the garment looks from different angles, especially from the back. This problem is solved by various implementations of delay mode, as exemplified below.

[0216] In various implementations of delayed mode, the mirroring system solves the traditional mirroring problem by using delayed replies to fit the session so that the user can see the session in semi-real time. For example, a user can try on a dress in front of a mirror and rotate 360 ​​degrees. The recorded fitting session will be played in semi-real time by implementing a delay in the session replay. Therefore, when the user completes the rotation and faces the mirror, the session starts replaying so that the user can see the entire rotation and can see the dress from different angles, including from behind.

[0217] The delay mode can be implemented in a variety of ways. For example, the system can set two parameters: the delay time and the recording period. The delay time indicates how long the playback is delayed, and the recording period indicates how long the recording session is. For example, it can be configured that it takes an average of 20 seconds for a user to try on a piece of clothing and rotate 360 ​​degrees to check the clothing at all angles. In this case, the recording period can be set to 20 seconds. Then, the delay can be set, for example, from zero to five seconds. In this case, when the fitting session starts, the system records a session with a length of 20 seconds. If the delay is set to zero, the system will start replaying the session immediately after finishing recording the 20-second session. On the contrary, if the delay time is set to 5 seconds, the session will not be played until 5 seconds after the 20-second session recording is completed.

[0218] Various embodiments deal with projection on the screen during different recordings. For example, in one embodiment, before and / or during a recording session, the system projects instructions for the user on the screen, such as cuing the start of the session or instructing the user to rotate. Such examples are shown in FIG. Fig.19, a timer for starting a session is provided. After the user activates the delayed mode session, for example by selecting a start button, or automatically when the mirroring system recognizes the presence of the user in front of the mirror, a prompt can be initiated and the idle mode can be exited. In another embodiment, the mirroring system projects a real-time mirrored video of the session, but also records and replays the session according to the parameter settings of the delayed mode. One advantage of projecting the real-time mirrored video of the session is that it maintains the appearance of a standard mirror, making it easier for the user to interact with the mirroring system. Therefore, in such an embodiment, when the camera records the session, the processor applies various transformations as described in detail in the above embodiments to generate a mirrored video stream that is displayed on the screen and stored in the memory. When the session recording period ends and the delay time expires, the mirrored video is retrieved from the memory and replayed on the screen.

[0219] Now refer to Fig.18 Describe the implementation of delayed mode. Box 1800 represents the mirror being in idle mode or real-time mirror mode, as explained in various embodiments herein. In real-time mirror mode, the user can see him / herself in the digital mirror in real time. In this mode, the camera sends a video stream to the processor, which processes the video stream to be projected as a digital mirror stream (mirror video). Alternatively, the mirror may be in some idle mode, displaying commercials, demonstration mode, instructions for operating the mirror, etc.

[0220] Box 1801 indicates that a user has approached the mirror. This event can be automatically detected by the mirroring system by analyzing the video stream from the camera, for example using a motion sensor. The user's appearance in front of the screen can be used to automatically enter the delay mode. Alternatively, switching to the delay mode can be done manually by the user using, for example, a remote control, voice, gestures, etc.

[0221] In block 1802, the mirror is instructed to switch to delayed mode. As described above, the delayed mode may be configurable, i.e., the length of the delay and the length of the recording session. The delayed mode should be configured to provide enough time so that the user can see themselves in semi-real time and preview the motion while still having the perception of seeing themselves in the mirror.

[0222] The delay can be implemented in many ways, but one way is to direct the live stream from the camera to a storage device (such as RAM), solid state drive, hard drive, etc. for 2-5 seconds. Once the FIFO buffer is filled, the stream is pulled out to the graphics card frame by frame and presented to the screen. If the graphics card has enough storage space, the FIFO buffer can be defined in the graphics card, but if not, RAM is a good choice for short delays. Otherwise, for long delays, a solid state drive or hard drive is a better solution. The mirroring conversion can be performed in storage or after the frame is acquired.

[0223] In one embodiment, during the delay time, the image is frozen on the screen, or some graphics are displayed on the screen. In one embodiment, during the delayed start time, the GUI hides the frozen image, so the user cannot see what is happening in the mirror stream / camera stream.

[0224] In one embodiment, the stream from the camera is always split into a live stream and a storage buffer. In real-time mode, the live stream is being rendered to the screen and the buffer is cleared of the previous frame as soon as a new frame is obtained. In delayed mode, the live stream is cleared frame by frame in delayed mode while the frames exiting the buffer are being rendered to the screen. In this way, the transition between delayed and real-time is fast. In this embodiment, the mirroring transition is preferably done before the live stream is stored in memory and presented on the screen.

[0225] Box 1804 represents a delayed session. In one embodiment, the delayed mode is implemented by simply sending the video stream to a FIFO buffer, where each frame is acquired after a set time delay. The length of the delay can be pre-set. Therefore, in this embodiment, once the mirroring system enters the delayed mode, the user continuously sees a semi-real-time, i.e., delayed, mirrored video stream on the screen. In another embodiment, the duration of the delayed mode can also be configured to be, for example, 8-30 seconds, after which the mirroring system returns to the real-time mirroring mode. During the period of the delayed mode, the user will see a delayed (i.e., semi-real-time) mirrored stream. Therefore, when the user turns around, the user will get enough time to complete the 360 ​​rotation movement, and when the user is finished, the user can see himself turning around to preview the 360 ​​view, or any other action that the user likes to capture in semi-real time. In another embodiment, the delay period and the delay mode duration are set in advance. In this embodiment, once the system enters the delayed mode, the video stream is stored in the FIFO memory, however, the image is not taken out until the delay mode duration has passed and the delay time has passed. Only then is the image fetched and displayed as a delayed mirrored video.

[0226] In any of the delayed mode implementations described herein, a user may be given the ability to pause, rewind, or replay a delayed session.

[0227] Box 1805 shows an optional feature. According to this optional implementation, in parallel with the semi-real-time preview (delayed mode), the mirror stream can be captured to storage so that at the end of the session, the user can replay his session and see himself again. The user can also be given the ability to pause, rewind or replay the delayed session. If the user is logged in, he / she can choose to compare with a previous recorded full screen or side by side.

[0228] In addition, by storing the session in the memory, the user can download the session to a mobile device, share the session through, for example, a social network, send a link to family and friends, and the like. That is, once the session is over and the recorded file is created as an image or video and optionally compressed, the file can be shared. The sharing of the file can be accomplished by, for example, uploading the file to the cloud, giving a designated user a phone number, an email, a QR code, or automatically verifying a code from a short-range communication such as NFC, BT, etc. Once the file is stored in the cloud, the user can directly obtain a link through his social account to open the video and / or share the video from the response application. Alternatively or in addition, the user can also wirelessly make the file reach his device directly from the mirroring system. Moreover, the system can upload the video to the cloud, instead of sending a link to the video, and the user may be able to access his record from a dedicated application that has accessed his account from the API.

[0229] In block 1806, once the file is recorded, and the user wants to explore the session through image modification, the user can replay the video into the image processing engine, either in real time or post-processing, to, for example, change the scene, modify the color or fabric of their clothing, or add accessories such as glasses. Additionally, in 1807, the stored video can be analyzed in retiming or post-processing to add a specified UI and / or virtual reality to the video, or simply to generate more information that can be used by the application (e.g., calculating BMI as described above).

[0230] In one embodiment, recording starts at the end of the switching period, in one embodiment, recording starts after a few seconds in a delayed mode, and in one embodiment, recording starts and stops by detection of automatic events (such as the orientation of the user's face, the user starting to turn around, etc.), which will be referred to as Fig. 20 Further explanation.

[0231] In one embodiment, the user will manually or automatically start the delay mode for a few seconds, such as 5-30 seconds, and the engine will start the delay mode. At the end of the delay timer, the system will automatically return to the real-time mode. In one embodiment, the delay length in the delay mode is shorter than the delay mode period, for example, the delay length can be configured to be 2-5 seconds, and the delay period is 15-30 seconds. In such an arrangement, the mirrored video begins streaming to the screen after 2-5 seconds, and the entire session will last for 20-30 seconds, after which the system will revert to the real-time mode. In addition, if no user is detected in front of the mirror, the system can revert to the idle mode.

[0232] In one embodiment, when the delay mode is triggered manually or automatically, the real-time rendering is frozen to the duration of the actual delay, and optionally, a GUI is applied on the screen. The video stream during this time is directed and saved to a memory buffer such as RAM or HD. After the buffer is full, or the delay time is reached, the renderer will fetch the delayed stream from the buffer, and the user can see the delayed self. At the end of the delay mode, the video stream returns to real-time mode.

[0233] In one embodiment, during the time that the buffer is being fed with a stream and the renderer is frozen, the GUI or UI is being rendered on the screen and the freezing of the real-time stream can be completely or semi-transparently hidden. In this way, the transition between real-time mode and delayed mode is smoother. The UI essentially hides the period of time that the buffer is accumulated.

[0234] In one embodiment, the buffer is always filled in parallel to the real-time display, and the renderer simply switches between getting the feed from the delay buffer or getting the feed from the real-time feed. In such an embodiment, hiding the delay accumulation may not be required.

[0235] In one embodiment, during delay mode, the engine automatically records the user, and at the end of delay mode, the user can choose whether he likes to play the delay mode session again, or to himself, or to share it directly to his social network.

[0236] In one embodiment, the user can play the delayed session in a split screen to compare with real time or to other recordings.

[0237] In one embodiment, the user may choose to post-process the recorded video with a different background image or video or a different fabric or a different color or completely different clothing and accessories.

[0238] In one embodiment, during a delayed session, the user can invite a friend or a salesperson or a designer to assist in the session.

[0239] In one embodiment, the delayed session can be used for other applications in addition to or instead of trying on clothes. For example, the delayed mode can be used for weight loss programs, dancing, fitness exercises, body scans, and other sports.

[0240] In one embodiment, the system may automatically stop and freeze the image of the user's back for a set period, or until the user instructs the system to resume delayed or real-time mode.

[0241] In one embodiment, the system will automatically switch to delay mode when a user is identified as entering the field of view or when a user within the field of view is determined to perform a triggering motion. For example, when used to try on clothes, the triggering motion may be when the user begins to turn around. For other use cases, other motions may be pre-programmed as triggering motions, and the analysis of the feed from the camera will search for that pre-programmed motion. When the pre-programmed motion is detected, the system sends a signal to resume delay mode. In another example, the system may be used for golf or baseball, such that the triggering motion would be the swing of a club or bat.

[0242] Fig. 20 2000. In this example, the camera 2010 of the mirror system 2000 sends a captured video stream 2015 to a transformation module 2020. The transformation module 2020 operates on the video stream 2015 to generate a real-time mirror stream 2025. According to any embodiment described herein, according to any method and process described herein, the transformation module 2020 can be implemented to convert the video stream from the camera 2010 so as to generate an image that simulates the image of the mirror.

[0243] During the real-time mode or idle mode of operation, the real-time mirrored video stream 2025 is sent to the screen 2005. The real-time mirrored video stream 2025 is also sent to the buffer 2030 and the image analysis module 2035. The image analysis module 2035 continuously analyzes the real-time mirrored video stream 2025. When the image analysis module 2035 detects that the user appears in the real-time mirrored video stream 2025, the image analysis module 2035 temporarily mirrors the video stream 2025 and issues an instruction to the system controller to switch from the idle mode to the real-time mode. In addition, when the image analysis module 2035 detects that the user begins to turn around, the image analysis module 2035 issues an instruction 2040 to the controller to enter the delay mode. When the instruction to enter the delay mode is received, the time of the current frame in the buffer is recorded, and the counter starts counting the delay time. When the delay length is reached, the image is obtained from the buffer 2030 as a delay stream 2045 and sent to the screen 2005 for delayed presentation.

[0244] In one embodiment, it is assumed that it takes some time for the image analysis module 2035 to sense that the user starts turning. Therefore, in this embodiment, when an instruction to enter the delay mode is received, the time of the current frame in the buffer is marked, and a counter starts counting the delay length. When the delay length is reached, the image is obtained from the buffer 2030 as a delay stream 2045 and sent to the screen 2005 for delayed presentation. However, the acquisition of the image from the buffer starts with the image recorded in the advance period before the marked time. For example, the delay stream 2045 starts from the image recorded in the buffer 2-5 seconds before the time when the image analysis module 2035 issues the instruction to enter the delay mode.

[0245] In one embodiment, the user can select the system in delayed mode to record his session or select the system in delayed mode not to record his session.

[0246] In one embodiment, the user can set the system to automatically send the video to his cloud account / mobile device or dedicated application.

[0247] In one embodiment, the mirror will recognize the user through his smartwatch, mobile app, or other identifier.

[0248] In one embodiment, the mirror or remote kiosk will send a beacon that the user's mobile device or smartwatch can receive and use the information to gain access and control of the mirror. For example, the identified user with access can automatically re-encode the session into his device or cloud account.

[0249] In one embodiment, the mirroring station can be configured to recognize the user's behavior and adjust the gesture control to the way the user is trying to operate, for example, some users will click, some users will grab, and some users will push to operate the thumbnails. The system can be adapted to learn and update the contour control of each user.

[0250] In one embodiment, the mirroring station may support instructions to the user so that when the system recognizes that the user is not controlling the mirroring in the correct way, a brief explanation may pop up and guide the user through the process.

[0251] In one embodiment, the mirror may be adapted to display predetermined features based on information related to the user's familiarity with the system, for example, based on whether the user is a first-time user, based on the number of videos of users in the system, based on the last time the user requested system activation instructions, may be adapted to provide voice services with relatively simple functions, may be adapted to provide remote expert assistance, etc.

[0252] In one embodiment, if Fig.15 and 16As shown in , the idea of ​​a predefined number of thumbnails can be replaced by a sliding menu of thumbnails.

[0253] In normal operating mode, e.g. Fig.15 and 16 As depicted in , the user can move his hand up and down to select thumbnails, and once the user's hand stops, the most recent thumbnail can be selected and can be specified so that the user will get feedback for selection. For example, a blue shadow can be provided around the selected thumbnail to identify the selection. When the user grabs, pushes, or his hand / finger clicks on the thumbnail, the video can start playing based on the display mode setting, and the user can also stop playing the video and perform other operations, etc. Once, for example, the user stops playing the video, a hold bar can be displayed on the thumbnail to specify the state of the played video and confirm back to the user that its command has been accepted by the mirror station.

[0254] In one embodiment, to further simplify operation, gesture control with the hand can be configured to detect jumps between thumbnails even if the user moves the hand left to right instead of up and down or in any other direction. The system can be configured to adapt to the user's span and speed of hand movements so that the user can see the selection and the user can adjust the speed and span of their movement accordingly.

[0255] In one embodiment, a specific brand or partner or any other logo can be added to the recorded video, so that the user will get a video with a brand logo, for example, and the user can share the resulting video with his friends.

[0256] In one embodiment, the mirror can be configured to provide a convincing augmented reality capability by matching items previously tried on by another user with a similar body type. This matching technology creates a more convincing experience.

[0257] The mirror can be configured to perform a business. Since the mirror station according to the present invention is a new device on the market, the use of the mirror to generate revenue sources is also unique. The next few embodiments include details on how dynamic mirrors can be used to create new revenue sources.

[0258] The mirror can insert commercials, coupons and logos into the mirror in idle mode, within a designated section of the mirror, or via split screen mode. The mirror can also introduce commercials, coupons and logos into recorded videos so that users can view videos with commercials / coupons / logos. These displays can be sold as advertisements. Users can share commercials / coupons / logos. Users can be offered incentives in exchange for sharing commercials / coupons / logos.

[0259] The clothes that the user tries on can be analyzed for measurements / characteristics such as BMI, gender, origin, age, body measurements, facial expressions, voice expressions, suggested sizes, etc. This data can be incorporated into e-commerce applications. This data is of high value and can be shared with brands, users, third parties based on agreements between users and brands, etc. Using the present invention, a revenue stream can be generated from the analyzed data about the user, for example, by presenting focused and accurate advertising to the user.

[0260] The mirror can be configured as a service that allows the user to invite an expert, friend or consultant when the user is considering a specific purchase. Additional revenue can be generated using the present invention by enhancing sales opportunities associated with the advice of the remote expert. The expert can be specified by the user, and the user can choose an expert of his own liking or select an expert specified by the brand.

[0261] The user may be presented with a display including suggestions and / or incentives for a particular item being tried on by the user, such as using a different color. This approach provides further opportunities to boost sales. Augmented reality may be used to boost sales. Specifically, based on information acquired by the mirror and trials by other users, a more compelling augmented reality display may be generated based on a combination of information about the user and similar users with similar body types.

[0262] In one embodiment, the user's social network can be integrated in a way that helps the user and / or the user comment on the project's concerns. The system can be configured to share the user's video. This social networking function can be used to expand the database of addressable customers.

[0263] In one embodiment, users can upload photos of themselves and the video engine can be configured to process the image and provide similar analysis of BMI, face, origin, etc. The system can provide suggested sizes for e-commerce applications. The system can display purchased items in different colors. In this way, the mirror can be used to obtain market information from appropriate databases even when the user is not physically standing in front of the mirror itself.

[0264] In one embodiment, users can see the predicted size in the application and update measurements, which can be used to improve the size prediction model and include brand-specific adjustments.

[0265] The invention includes both the mechanical and cosmetic design of the device. The screen may be mounted vertically or horizontally on a wall, or may be switchable between vertical and horizontal (through a larger or similar mechanical solution, the screen may be tilted, swiveled, etc.). The screen may be mounted on a dedicated stand, mounted on the wall, or mounted behind a wall. When the screen is located within the wall, heat venting ducts should be provided to support the screen and computing device.

[0266] Fig.21 An embodiment of a fully contained unit that can be attached to the back of any television screen and transform the TV screen into a digital mirror is shown, while Fig. 22 yes Fig.21 A side view of the unit, shown attached to the back of a television. Fig.21 and Fig. 22 One advantage of the illustrated embodiment is that the system can operate on any digital television, allowing the purchaser of the unit to use an existing television or purchase a television of his / her choice. This can significantly reduce the cost of the system, especially if a television is already owned. Furthermore, the system can be installed on any television, allowing the television to be used regularly as a television and switched to a mirror when desired.

[0267] like Fig.21 and 22 As shown, the digital mirror system 2110 includes a housing 2112 that houses a processor. In one embodiment, the processor is a processor such as that available from Gigabyte Technology, Taiwan. TM The mini PCs such as the Brix series available from , and can be purchased by consumers at electronics stores such as Fry's Electronics and MicroCenter. The advantage of using a mini PC is that it is an ultra-compact PC design, measuring only 62 mm x 111.4 mm x 114.4 mm. This makes the entire system very compact and can be easily connected to the back of any TV using the connection mechanism 2115. The connection mechanism 2115 can be a VESA mount in the case where the TV is mounted to a wall, or the connection mechanism 2115 can be other connection mechanisms, such as specially designed mechanical clamps, double-sided tape, Etc. Fig. 22 As shown, when the housing 2112 is directly connected to the back of the TV 2005, a two-part connection mechanism can be used, represented by a 2115 part connected to the housing 2112 and a part 2115' connected to the back of the TV 2005, wherein the two parts 2115 and 2115' are connected using an adhesive, etc. are mechanically connected to each other.

[0268] The processor inside the housing includes a microprocessor such as an Intel Core, memory, a DSP to process video, a wireless transmitter (e.g., WiFi, Bluetooth), etc. For example, the Brix includes an Intel Core processor, a solid state drive or hard drive storage, a wireless transmitter, and a LAN connection. Importantly, for enhanced visualization, the Brix includes a 4K resolution HDMI output 2117, which connects to the HDMI input of the TV 2005. Fig.21 Also shown is the processor's power input, which connects to a standard 110V or 240V outlet, depending on the country. In addition, a high-quality digital signal processor can be used to obtain the best results for processing video signals. For example, Pro graphics cards can be used for video processing.

[0269] The housing 2112 includes an extension portion 2113 on which a light sensor housing 2114 is mounted. Fig. 22 In an embodiment, the photosensor housing is mounted to the extension 2113 via an adjustable coupler 2216 so that the height of the photosensor's field of view can be adjusted by rotating the housing 2114 about the coupler 2116, for example as indicated by the curved arrows. The photosensor (not shown) is housed within the housing 2114 and sends its signal to the processor for processing via a cable 2118, such as Fig.21 Shown for illustration purposes, but should be hidden from a visual perspective. Cable 2118 can be an electrical cable or an optical fiber.

[0270] According to one embodiment, a system for achieving real-time and delayed video playback is provided. The system can be used to simulate a mirror, thereby enabling a user to try on clothing items. The system can also be used for various training purposes, where the mirror simulation function can be omitted. The system includes: a housing configured to accommodate a processor therein, the housing having an extension; an optical sensor housing coupled to the extension; a processor housed in the housing, the processor including a memory and a video engine, and having an output configured to send a digital image to a digital screen; an optical sensor, which is housed in the optical sensor housing and has a device for sending an image to the processor; and a connecting mechanism, which is configured to connect the housing to the back of the digital screen.

[0271] Fig.23A flow chart showing a process that may be performed by any of the embodiments disclosed herein. Step 2300 shows a normal operating condition, where a live video stream from a photoelectric sensor is presented on the screen and also saved in memory. When the screen is operating as a digital mirror, the live video stream may be processed by applying flips and transformations before being presented on the screen and / or stored in memory so that it mimics a mirror image. Otherwise no flipping or transformation is required.

[0272] In step 2302, the system enters delay mode, for example, when a trigger is issued by a user clicking an icon, by analyzing motion in a live video stream, etc. In this embodiment, when the system enters delay mode, in step 2304, the frame presented on the screen in this case is frozen or paused. However, the live video stream from the light sensor continues to be stored in the memory. The delay time is being monitored, and when the time reaches the current delay time at 2306, the frames from the memory are sent to the screen at 2308, starting with the frame immediately following the frozen frame. During this time, the delay period is monitored, and when the delay period is reached at 2310, the system resumes normal operation 2300. Based on testing, it has been shown that the delay time is preferably set to be shorter than the delay period. In addition, based on experience, it has been shown that the delay time is preferably set to 1 to 5 seconds, while the delay period is 5 seconds to 30 seconds.

[0273] Fig.24 A flow chart showing a process that can be implemented by any embodiment disclosed herein. Step 2400 shows a normal operating condition, where a live video stream from a photoelectric sensor is presented on a screen and also saved in a memory. The live video stream can be processed by applying flips and transformations before being presented on a screen and / or stored in a memory so that it mimics a mirror image. Otherwise, no flipping or transformation is required.

[0274] In step 2402, the system enters a delay mode when a trigger is issued, such as by a user clicking an icon, by analyzing motion in a live video stream, etc. In this embodiment, when the system enters the delay mode, in step 2404, the time is marked, such as by storing the time when the trigger was issued or by marking a delayed frame, such as by storing the memory address of the current frame, which is the frame stored when the trigger was issued. At 2406, the system continues to present the live video stream on the screen, and the live video stream from the photoelectric sensor also continues to be stored in the memory. The delay time is being monitored, and when the time reaches the current delay time at 2408, the frame from the memory is sent to the screen at 2410 starting from the time marked in step 2404. During this time, the delay period is monitored, and when the delay period is reached at 2412, the system resumes normal operation 2400. According to testing, it has been shown that the delay time is preferably set to be shorter than the delay period. In addition, according to experience, it has been shown that the delay time is preferably set to 1 to 5 seconds, while the delay period is 5 seconds to 30 seconds.

[0275] Fig.23 and 24 The method shown is particularly beneficial because the user will most likely miss the transaction from the live stream to the delayed stream because the user will not be looking at the screen in this case. For example, a user using the system to improve a golf swing will look at the screen when taking a position, at which point the user will see the live feed. When the user is swinging, the user will not be looking at the screen, and it is at this point that the transition to delayed mode will occur and the user will not see it. After the user takes a swing, the user can look in the mirror, at which point the user will see the screen playing the delayed frame and can check the swing. Thereafter, the system will resume normal operation. Therefore, the unnatural jump from a freeze frame or live stream to a delayed video stream will most likely not be seen by the user.

[0276] In one embodiment, the screen may have a mirror ratio of, for example, 21:9.

[0277] In one embodiment, the screen may have a regular ratio, ie, 16:9, and the display setting may be a split screen or dark bars on both sides to create a mirror ratio, ie, 21:9.

[0278] In one embodiment, the screen may come in a matte finish to eliminate reflections that can be seen on dark colors. In other words, when the user is wearing black, and the screen is not matte finished, then the user may see a true mirror reflection of himself; this effect is undesirable and may reduce, distort, or completely destroy the virtual mirror effect.

[0279] Frames can be Fig.14 , 1516. A minimized frame may be used, or the frame may be hidden behind a wall.

[0280] In one embodiment, the camera or controlled camera can be located in the frame. The camera can be mounted above the screen frame, can be hidden, and can be mounted on the wall. The camera can be provided with a shade that can filter out the image. Since the camera is facing downward, a cover that will cover most of the camera body can block the distant view.

[0281] The screen can be LED, LCD, plasma, glass, projector, etc.

[0282] A background may be used that produces a better white balance resulting in higher quality video.

[0283] The lighting may include white, yellow, or a combination of any combination in a spot or projector combination and may be configured to improve video quality and color.

[0284] Specialized carpets can be used to make backgrounds varied, define user areas, improve user detection and tracking in front of the mirror, indicate where the user is standing, and instruct the user on how to operate the mirror.

[0285] Each module or program identified above corresponds to a set of instructions for performing the above functions. These modules and programs (i.e., instruction groups) do not have to be implemented as separate software programs, processes or modules, so various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, the memory can store a subset of the modules and data structures identified above. In addition, the memory can store additional modules and data structures not described above.

[0286] The illustrated aspects of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0287] In addition, it is understood that the various components described herein may include circuits (multiple circuits), which may include components and circuit elements with suitable values ​​for implementing the creativity (multiple creativity) of the present subject. In addition, it is understood that many of these various components can be implemented in one or more integrated circuit (IC) chips. For example, in one embodiment, the grouped components can be implemented in a single IC chip. In other embodiments, one or more corresponding components are manufactured or implemented on a separate IC chip.

[0288] What has been described above includes examples of embodiments of the present invention. It is certainly not possible to describe every conceivable combination of components or methods for describing the claimed subject matter, but it is understood that many further combinations and arrangements of the subject matter creativity are possible. Therefore, the claimed subject matter is intended to cover all such changes, modifications and variations that fall within the spirit and scope of the appended claims. In addition, the above description of the illustrated embodiments disclosed in the subject matter, including what is described in the abstract, is not intended to be exhaustive or limit the disclosed embodiments to the precise form disclosed. Although specific embodiments and examples are described herein for illustrative purposes, various modifications that are considered to be within the scope of such embodiments and examples are possible, as those skilled in the relevant art can recognize.

[0289] In particular and with respect to the various functions performed by the above-described components, devices, circuits, systems, etc., the terms used to describe these components are intended to correspond (unless otherwise specified) to any component that performs the specific function of the described component (e.g., a functional equivalent), even if not structurally equivalent to the disclosed structure, which performs the functions in the exemplary aspects of the claimed subject matter described herein. In this regard, it should also be recognized that the inventive concept includes systems and computer-readable storage media having computer-executable instructions for performing the actions and / or events in the various methods of the claimed subject matter.

[0290] The above-described systems / circuits / modules have been described with respect to the interactions between several components / blocks. It will be appreciated that such systems / circuits and components / modules may include those components or specified subcomponents, certain specified components or subcomponents, and / or additional components, and according to various permutations and combinations of the above. Subcomponents may also be implemented as components that are communicatively coupled to other components rather than being included in a parent component (layering). In addition, it should be noted that one or more components may be combined into a single component that provides aggregate functionality or divided into several separate subcomponents, and any one or more intermediate layers, such as a management layer, may be provided to communicatively couple to such subcomponents to provide integrated functionality. Any component described herein may also interact with one or more other components that are not specifically described herein but are known to those skilled in the art.

[0291] Furthermore, while a particular feature of the subject innovation may have been disclosed with respect to only one of several embodiments, such a feature may be combined with one or more other features in other embodiments as may be desired and advantageous for any given or particular application. In addition, to the extent that the terms "includes," "comprising," "having," "containing," and variations thereof, and other similar terms are used in the detailed description or claims, these terms are intended to be inclusive, in a manner similar to the term "comprising" as an open transition word without excluding any additional or other elements.

[0292] As used in this application, the terms "component", "module", "system", etc. are generally intended to refer to a computer-related entity, or hardware (e.g., circuitry), a combination of hardware and software, or software, or an entity related to an operational machine having one or more specific functions. For example, a component can be, but is not limited to, a program running on a processor (e.g., a digital signal processor), a processor, an object, an executable, an execution thread, a program, and / or a computer. By way of example, both an application running on a controller and the controller can be components. One or more components can be contained within a process and / or execution thread, and a component may be localized in one computer and / or distributed between two or more computers. In addition, a "device" can be included in specially designed hardware; generalized hardware specifically implemented by software thereon that causes the hardware to perform specific functions; software stored on a computer-readable medium; or a combination thereof.

[0293] Computing equipment typically includes various media, which may include computer-readable storage media and / or communication media. Computer-readable storage media can generally be any available storage media that can be accessed by a computer, which is generally non-transient, and may include volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media can be connected to any method or technology such as computer-readable instructions, program modules, structured data or unstructured data for storing information to achieve. Computer-readable storage media may include but are not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassette, tape, disk storage or other magnetic storage device, or other tangible and / or non-temporary media that can be used to store desired information. Computer-readable storage media can be accessed by one or more local or remote computing devices for various operations relative to the information stored by the medium, such as via access requests, inquiries or other data retrieval protocols.

[0294] Communication media typically embodies computer readable instructions, data structures, program modules, or other structured or unstructured data in a data signal that may be transitory, such as a carrier wave or other transport mechanism, and includes any information delivery or transmission media. The term "modulated data signal" or signal refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in one or more signals.

[0295] In view of the exemplary system described above, the method that can be implemented according to the described subject matter will be better understood with reference to the processes in the accompanying drawings. For simple explanation, the method is depicted and described as a series of actions. However, the actions according to the present disclosure can occur in various orders and / or simultaneously, and with other actions not presented and described herein. In addition, not all the actions shown must implement the method according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that these methods are replaceably represented as a series of interrelated states via state diagrams or events. In addition, it should be understood that the methods disclosed in this specification can be stored on manufactured articles to facilitate the delivery and transmission of such methods to computing devices. The term article of manufacture as used herein is intended to cover a computer program that can be accessed from any computer-readable device or storage medium, such as stored on a server coupled to a cloud.

[0296] For the purpose of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or limited to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments are selected and described to best explain the principles of the aspects and their practical application, thereby enabling those skilled in the art to best utilize the aspects and various embodiments with various modifications suitable for the intended specific use.

Claims

1. A computer-implemented method for automatically presenting delayed video in a digital mirror having a camera and a screen, comprising: operating the camera to generate a video stream, and sending the video stream to a transformation module of the digital mirror; operating a calibration module of the digital mirror to define a transformation mapping between the viewpoint of the camera and the viewpoint of the user in front of the screen, and providing the transformation mapping to the transformation module; operating the transformation module to apply the transformation map to the video stream and fill in blank pixels caused by the transformation map, thereby generating a mirrored video; During a normal operating mode, sending the mirrored video from the camera to the screen for presentation and simultaneously sending the mirrored video to a memory for storage; When a delay trigger is issued, enter delay mode and execute: freezing a frame that is rendered if the delay trigger is issued; Continue to send the mirrored video to the memory for storage; and storing a time indicator representing a time when the trigger is issued; monitoring a delay time, and when the delay time is reached, starting with frames recorded in the advance period before the time indicator, starting to obtain frames of the mirrored video from the memory and sending the frames to the screen for presentation; A delay period is monitored, and when the delay period is reached, a normal mode is restored, wherein the mirrored video is sent to the screen for presentation and also to the memory for storage.

2. The method according to claim 1, wherein: The delayed trigger is generated by continuously analyzing the video stream and issuing the delayed trigger when it is determined from the analysis that the user is performing a triggering motion.

3. The method according to claim 1, wherein: The delayed trigger is generated by continuously analyzing the video stream and is issued when it is determined from the analysis that an object in an image of a video frame has moved a specified distance.

4. The method according to claim 1, wherein: The delayed trigger is generated by continuously analyzing the video stream and is issued when it is determined from the analysis that a user has entered the field of view of the camera.

5. The method according to claim 1, wherein: The delayed trigger is generated by the user pressing a trigger button.

6. The method according to claim 1, further comprising: flipping each frame of the video stream about a vertical axis so that the right and left sides of the image in each frame are reversed; The transformation map is applied to each frame of the video stream to modify the frames so that they appear to mimic reflections of a mirror, thereby obtaining transformed frames.

7. The method of claim 1, further comprising enabling the user to pause replay of the delayed video.

8. The method of claim 1, wherein the length of the delay period is configured to have sufficient time to enable the user to turn around.

9. The method of claim 1, further comprising overlaying graphics on the mirrored video during the period of the delay time.

10. The method of claim 1, further comprising generating a model mask to enable changing the color of the clothing item.

11. The method according to claim 10, wherein: The model mask includes a plurality of layers including at least a color layer and a texture layer.

12. The method of claim 1 further comprising instructions to generate a split screen on the screen and display the mirrored video on one side of the screen and play the delayed video on the other half.

13. A computer-implemented method for operating a system having a monitor, a camera, and a processor to sequentially display a real-time video stream and a delayed video stream on the monitor to enable a user to enhance the experience of a viewing session, comprising: Set a delay period of N seconds; Set the recording length to M seconds; acquiring a live video feed from the camera and sending the live video feed to a transformation module of the system; operating a calibration module of the system to define a transformation mapping between the viewpoint of the camera and the viewpoint of a user in front of the monitor, and providing the transformation mapping to the transformation module; operating the transform module to apply the transform map to the live video feed and fill in blank pixels caused by the transform map, thereby generating a mirrored video; Simultaneously sending the mirrored video to the monitor to be displayed as a real-time video stream, and storing the mirrored video as frames in a memory; When a trigger is received, execute: indicating a trigger time when said trigger is received; Counting N seconds from the trigger time, then starting from the frame recorded in advance period before the trigger time, acquiring the frame from the memory and sending it to the monitor for display as a delayed video stream, and simultaneously continuing to store the mirrored video from the transformation module to the memory; After acquiring a series of frames for M seconds, the acquisition is stopped and the mirrored video is resumed to be sent to the monitor for display as a real-time video stream. The method of claim 13 , wherein N is shorter than M.

15. The method according to claim 14, characterized in that Set N from 1 to 5 seconds and M from 5 to 30 seconds.

16. The method according to claim 13, wherein: The trigger is generated by continuously analyzing the live video feed and issuing the trigger when it is determined from the analysis that the user is performing a triggering motion.

17. The method of claim 16, further comprising freezing the frame being displayed at the trigger time until the count of the N seconds is completed.

18. The method according to claim 13, wherein: The trigger is generated by continuously analyzing the live video feed and issued when it is determined from the analysis that a user has entered the field of view of the camera.

19. The method of claim 13, further comprising: flipping each frame of the live video feed about a vertical axis so that right and left sides of the image in each frame are reversed; The transformation map is applied to each frame of the live video feed to modify the frames so that they appear to mimic reflections of a mirror, thereby obtaining transformed frames.

20. The method of claim 19, wherein the flipping and applying the transform mapping to each frame is done before sending the live video feed to the monitor to display it as the real-time video stream and before saving the live video feed as a frame in the memory.

Images