Automatic video capture and synthesis system
By using an inertial measurement unit and a stereo camera in a portable electronic device, a stable output path is generated, which solves the problem of unstable recording of the device while capturing intense activities, and realizes the stability and simplicity of recording and sharing videos by portable devices.
Patent Information
- Application Number
- CN202411348429.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-27
- Filing Date
- 2020-08-26
- Publication Date
- 2024-11-12
AI Technical Summary
Existing portable electronic devices are difficult to record video stably when capturing intense user activities. High-resolution video files are large and difficult to process. The recorded video is unstable and dull, and users are busy operating the equipment, making it difficult to share videos.
The inertial measurement unit is used to combine a digital accelerometer, gyroscope and magnetometer to calculate the camera direction and generate a stable output path. The image is captured through the stereo camera to generate a relatively stable, short and easy to share video synthesis file.
It realizes stable video recording when capturing active users' activities, reducing the complexity of device operation, and the generated video is suitable for sharing with others and is suitable for portable devices.
Smart Images

Figure CN118945306A_ABST
Abstract
Description
This application is a divisional application for the invention patent with the invention name “Automatic video capture and synthesis system”, the application date of August 26, 2020, and the application number of 202080067646.5. CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Utility Application Serial No. 16 / 585,603, filed on September 27, 2019, entitled “Automatic Video Capture and Composition System,” the contents of which are incorporated herein by reference in their entirety. Technical Field
[0002] Examples described in this disclosure relate to portable electronic devices, including wearable devices such as eye-mounted devices. Specifically, but not by way of limitation, this disclosure describes systems and methods for capturing a series of raw video clips and combining the clips to generate a video composite file. Background Art
[0003] Many types of computers and electronic devices available today, including mobile devices (e.g., smartphones, tablets, and laptops) and wearable devices (e.g., smart glasses, digital eyewear, headgear, helmets, and head-mounted displays), include one or more cameras for capturing still photos and videos and internal sensors for collecting information about the device's position, orientation, motion, and direction of travel. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] The following detailed description (with reference to the accompanying drawings) can facilitate the reader to understand the features of the various embodiments disclosed. In the embodiments and several views of the accompanying drawings, each element uses a reference number. When there are multiple similar elements, a single reference number can be assigned to the similar elements, and a lowercase letter is added to indicate a specific element.
[0005] Unless otherwise noted, the various elements shown in the figures are not drawn to scale. The sizes of the various elements may be enlarged or reduced for clarity. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The following images are included in the drawings:
[0006] Figure 1A is a side view (right view) of an example hardware configuration of an eye-mounted device that can be used in a video compositing and sharing system;
[0007] Figure 1B yes Figure 1A A top partial cross-sectional view of the right side assembly of the eye-mounted device, showing the right side visible light camera and circuit board;
[0008] Figure 1C yes Figure 1AA side view (left) of an example hardware configuration of an eye-mounted device, showing a visible light camera on the left side;
[0009] Figure 1D yes Figure 1C A top partial cross-sectional view of the left side assembly of the eye-mounted device, showing the left visible light camera and circuit board;
[0010] Figure 2A and 2B is a rear view of an example hardware configuration of an eye-mounted device used in a video compositing and sharing system;
[0011] Figure 3 is a schematic diagram of a three-dimensional scene, a left original image captured by a left visible light camera, and a right original image captured by a right visible light camera;
[0012] Figure 4 is a functional block diagram of an example video compositing and sharing system including an eye-mounted device, a mobile device, and a server system connected via various networks; and
[0013] Figure 5 yes Figure 4 Schematic diagram of an example hardware configuration for a mobile device of the video composition and sharing system shown. Detailed description
[0014] Various embodiments and detailed diagrams are provided with reference to an example: a video composition and sharing system for capturing a series of raw video clips with a portable electronic device, calculating a camera orientation based on information from a device sensor, calculating a stabilized output path, and combining the clips to generate a video composite file. The portable device includes an inertial measurement unit to calculate the camera orientation during recording, the inertial measurement unit including a digital accelerometer, a gyroscope, and a magnetometer. The camera orientation can be used to calculate a stabilized output path, i.e., an imaginary route traveled by the camera that extends along and through the series of raw video clips. The clips are combined along the stabilized output path to produce a relatively stable, short, and easily shareable video composite file.
[0015] It can be difficult to capture vigorous user activities, such as skiing or biking, using a portable electronic device. In many cases, the user's hands are busy with the activity and cannot operate the controls on the device. High-resolution video files are often large and difficult to process on a portable device. Video recordings may be shaky, unremarkable, too long, or too boring to be shared with others. The examples described herein, for example, work well when capturing active user activities.
[0016] The following specific embodiments include systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples described in this disclosure. In order to facilitate a thorough understanding of the disclosed subject matter and its related teaching content, many detailed drawings and examples are provided herein. However, those skilled in the relevant art may understand how to apply the related teaching content without these detailed drawings. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, and the related teaching content can be applied or practiced in various ways. The terms and nomenclature used herein are intended only to describe specific aspects and are not restrictive. In general, well-known instruction instances, protocols, structures, and techniques will not necessarily be shown in detail.
[0017] As used herein, the terms "coupled" or "connected" refer to any logical, optical, physical, or electrical connection, including links, etc., through which an electrical or magnetic signal generated or provided by one system element is transmitted to another system element coupled or connected thereto. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or transmit electrical signals. The term "on..." refers to being supported directly by an element, or indirectly supported by another element integrated into the element, or by another element supported by the element.
[0018] The orientation of the eye-mounted device, related components, and any set of equipment incorporating a 3D camera shown in any of the figures is by way of example only and is used for purposes of illustration and discussion. In actual operation, the eye-mounted device may be oriented in any other direction suitable for the specific application of the eye-mounted device. For example, up, down, sideways, or any other direction. In addition, for purposes of use herein, any directional terms, such as front, back, inward, outward, direction, left, right, lateral, longitudinal, upper, lower, top, bottom, side, horizontal, vertical, and diagonal, are used only as examples and do not limit the orientation or direction of any 3D camera or 3D camera assembly constructed as described herein.
[0019] The following description will partially set forth other objects, advantages, and novel features of the examples. Such other objects, advantages, and novel features will be apparent to those skilled in the art after reviewing the following and accompanying drawings, or may be learned by making or operating the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means, and combinations particularly pointed out in the appended claims.
[0020] Various types of visible light cameras are integrated into portable electronic devices, including mobile phones and wearable devices such as smart glasses. Stereo cameras have two lenses or two cameras that work in concert, each with its own image sensor or film frame. A stereo camera captures two images of the same scene, but with slightly different perspectives because the lenses are separated by a large distance. The lenses can be calibrated to be close to the distance between human eyes to better simulate human binocular vision. A stereo camera captures two separate images of the same scene, taken from slightly different angles.
[0021] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below.
[0022] Figure 1A 1 is a side view (right side) of an example hardware configuration of an eye-mounted device 100 used in a video composition and sharing system as described herein, showing a right visible light camera 114B for collecting image information. As further described below, the two cameras 114A, 114B capture image information of the same scene from two separate viewpoints. The two captured images can be used to project a three-dimensional display onto a screen for viewing with 3D glasses.
[0023] The eye-mounted device 100 includes a right optical assembly 180B having an image display for presenting an image, such as a depth image. Figure 1A and 1B As shown, the eye-mounted device 100 includes a right visible light camera 114B. The eye-mounted device 100 may include a plurality of visible light cameras 114A and 114B, which may form a passive three-dimensional camera, such as a stereo camera, wherein the right visible light camera 114B is located on the right block 110B. Figure 1C As shown in FIG. 1A -D, the eye-mounted device 100 also includes a visible light camera 114A on the left side.
[0024] The left and right visible light cameras 114A, 114B are sensitive to wavelengths in the visible light range. The visible light cameras 114A, 114B each have different forward-facing fields of view that overlap to generate a three-dimensional depth image, for example, the right visible light camera 114B depicts the right field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a specific location and orientation in space. Fields of view 111A and 111B have overlapping fields of view 813. When the visible light cameras capture an image, objects or object features outside the fields of view 111A, 111B are not recorded in the original image (e.g., a photograph or picture). The field of view refers to the angular range or degree to which the image sensors of the visible light cameras 114A, 114B pick up electromagnetic radiation of a given scene in a captured image of the given scene. The field of view can be expressed as the angular size of the cone of view, i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.
[0025] In one example, visible light cameras 114A, 114B have a field of view between 15° and 30°, such as 24°, and a resolution of 480×480 pixels. “Coverage angle” refers to the angle of view of visible light camera 114A, 114B or infrared camera 220 (see FIG. Figure 2A ) is the range of angles that a lens can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, possibly including some vignetting toward the edges. If the angle of coverage of a camera lens does not fill the sensor, the image circle will be visible, usually with strong vignetting toward the edges, and the effective viewing angle is limited by the angle of coverage.
[0026] Examples of such visible light cameras 114A, 114B include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA (video graphics array) cameras supporting 640p (e.g., 640×480 pixels, 0.3 megapixels in total), 720p, or 1080p resolutions. Other examples of visible light cameras 114A, 114B may capture high-definition (HD) still images and store images at a resolution of 1642×1642 pixels (or higher); and / or record HD video at a high frame rate (e.g., 30 to 60 frames per second or higher) and store recorded content at a resolution of 1216×1216 pixels (or higher).
[0027] The eye-mounted device 100 can capture image sensor data and geographic location data from the visible light cameras 114A and 114B, digitize them through the image processor, and store them in the memory. The left and right original images captured by the corresponding visible light cameras 114A and 114B are stored in a two-dimensional spatial domain and include a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for representing a horizontal position and a Y-axis for representing a vertical position. Each pixel has a color attribute value (e.g., a red pixel light value, a green pixel light value, and / or a blue pixel light value), and a position attribute (e.g., an X-axis coordinate and a Y-axis coordinate).
[0028] To capture stereoscopic images for later three-dimensional projection display, the image processor 912 ( Figure 4114B) is coupled to visible light cameras 114A, 114B to receive and store visual image information. A timestamp for each image may be added by image processor 912 or another processor controlling the operation of visible light cameras 114A, 114B, which act as stereo cameras to simulate human binocular vision. The timestamps on each pair of images enable the images to be displayed together as part of a three-dimensional projection. A three-dimensional projection can provide an immersive, lifelike experience that is ideal for a variety of environments, including virtual reality (VR) and video games.
[0029] Figure 3 858A captured by the left visible light camera 114A and the right raw image 858B captured by the right visible light camera 114B. As shown, the left field of view 111A can overlap with the right field of view 111B. The overlapping field of view 813 represents the portion of the image captured by the two cameras 114A, 114B. The term "overlap" when referring to the field of view means that the pixel matrix in the generated raw image overlaps by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrix in the generated raw image or the pixel matrix in the infrared image of the scene overlaps by fifty percent (50%) or more. As described herein, the two raw images 858A, 858B can be processed into a form that includes a timestamp so that the images can be displayed together as part of a three-dimensional projection.
[0030] For stereoscopic image capture, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a real scene 715 are captured at a given moment, with the left raw image 858A being captured by the left camera 114A and the right raw image 858B being captured by the right camera 114B. When the pair of raw images 858A and 858B are processed (e.g., by the image processor 912), a depth image is generated. The generated depth image can be viewed through the optical components 180A and 180B of the eye-mounted device, another display (e.g., the image display 880 on the mobile device 890), or a screen.
[0031] The generated depth image is stored in a three-dimensional spatial domain, which may include a vertex matrix on a three-dimensional position coordinate system, which includes an X-axis representing horizontal position (e.g., length), a Y-axis representing vertical position (e.g., height), and a Z-axis representing depth (e.g., distance). Each vertex may have a color attribute (e.g., red pixel light value, green pixel light value, and / or blue pixel light value), and a position attribute (e.g., X-axis coordinate, Y-axis coordinate, and Z-coordinate), as well as a texture attribute and / or a reflection attribute. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in the vertex area of the depth image.
[0032] In one example, the video composition and sharing system 1000 includes an eye-mounted device 100, which includes a frame 105 and a left temple 125A extending from a left side 170A of the frame 105 and a right temple 125B extending from a right side 170B of the frame 105. The eye-mounted device 100 may also include at least two visible light cameras 114A, 114B, which may have overlapping fields of view. In one example, as Figure 3 As shown, the eye-mounted device 100 includes a left visible light camera 114A having a left field of view 111A. The left camera 114A is connected to the frame 105 or the left temple 125A to capture a left raw image 858A from the left side of the scene 715. The eye-mounted device 100 also includes a right visible light camera 114B having a right field of view 111B. The right camera 114B is connected to the frame 105 or the right temple 125B to capture a right raw image 858B from the right side of the scene 715.
[0033] Figure 1B yes Figure 1A The top cross-sectional view of the right side block 110B of the eye-mounted device 100 is shown, showing the right side visible light camera 114B and the circuit board of the camera system. Figure 1C yes Figure 1A A side view (left) of an example hardware configuration of the eye-mounted device 100 is shown, which shows the left visible light camera 114A of the camera system; Figure 1D yes Figure 1C The top cross-sectional view of the left side block 110A of the eye-mounted device is shown, which shows the left visible light camera 114A and the circuit board of the three-dimensional camera. The left hinge 126A connects the left side block 110A to the left temple 125A of the eye-mounted device 100. The configuration and installation position of the left visible light camera 114A are basically similar to those of the right visible light camera 114B, and the only difference is that it is connected and coupled to the left side surface 170A. Figure 1B As shown in the example of FIG. 1 , the eye-mounted device 100 includes a right visible light camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The right hinge 126B connects the right assembly block 110B to the right temple 125B of the eye-mounted device 100. In some examples, the right visible light camera 114B, the flexible printed circuit board 140B, or other electrical connectors or contacts may be located on the right temple 125B or the right hinge 126B.
[0034] The right block 110B includes a block body 211 and a block cover. Figure 1BThe right block 110B has various interconnected circuit boards arranged inside, such as a printed circuit board or a flexible printed circuit board, which include controller circuits for the right visible light camera 114B, a microphone, a low-power wireless circuit (e.g., for wireless short-range network communication via Bluetooth), and a high-speed wireless circuit (e.g., for wireless local area network communication via WiFi).
[0035] The right visible light camera 114B is coupled or disposed on the flexible printed circuit board 140B and is covered by a visible light camera cover lens, which is focused through an opening formed in the frame 105. For example, Figure 2A As shown, the right frame 107B of the frame 105 is connected to the right block 110B and includes an opening for a visible light camera cover lens. The frame 105 includes a front side, which is configured to ensure that it faces outward and away from the user's eyes. The opening for the visible light camera cover lens is formed on and through the front side or outward-facing side of the frame 105. In this example, the right visible light camera 114B has an outward-facing field of view 111B (e.g., Figure 3 ), whose line of sight or viewing angle is associated with the right eye of the user of the eye-mounted device 100. The visible light camera cover lens can also be adhered to the front or outward-facing surface of the right block 110B, wherein the outward-facing coverage angle, but in different outward directions, constitutes an opening. Indirect coupling via an intermediate component can also be used.
[0036] like Figure 1B As shown, the flexible printed circuit board 140B is disposed in the right block 110B and is coupled to one or more other components mounted in the right block 110B. Although it is shown as being formed on a circuit board of the right block 110B, the right visible light camera 114B may also be formed on a circuit board of the left block 110A, the temples 125A, 125B, or the frame 105.
[0037] Figure 2A and 2B A rear perspective view of an example hardware configuration of the eye-mounted device 100, which includes two different types of image displays. The eye-mounted device 100 is designed to be suitable for the user to wear; the shape of glasses is shown in the example. The eye-mounted device 100 can take other forms and can be combined with other types of frames, such as headgear, headphones, or helmets.
[0038] In the glasses example, the eyewear device 100 includes a frame 105, which includes a left frame 107A connected to a right frame 107B via a nose bridge 106 suitable for being mounted on the user's nose. The left and right frames 107A, 107B include respective openings 175A, 175B, which are used to fix respective optical elements 180A, 180B, such as lenses and display devices. As used herein, the term "lens" includes a transparent or translucent glass or plastic sheet having a curved and / or flat surface that can converge / diverge light or converge / diverge little or no light.
[0039] Although the figure shows that two optical elements 180A and 180B are provided, the eye-mounted device 100 may also adopt other arrangements, such as a single optical element (or may not include any optical elements 180A and 180B) depending on the specific application or expected user situation of the eye-mounted device 100. As further shown, the eye-mounted device 100 is provided with a left block 110A adjacent to the left side 170A of the frame 105 and a right block 110B adjacent to the right side 170B of the frame 105. The blocks 110A and 110B may be integrated in the frame 105 on the corresponding 170A and 170B sides (as shown in the figure), or may be implemented as separate components attached to the frame 105 on the corresponding 170A and 170B sides. In addition, the blocks 110A and 110B may also be integrated in the temples (not shown) attached to the frame 105.
[0040] In one example, the image display of the optical components 180A, 180B includes an integrated image display. Figure 2A As shown, each optical component 180A, 180B includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or any other such display. Each optical component 180A, 180B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and other optical components in any combination. Optical layers 176A, 176B, ... 176N (in Figure 2AThe optical layer 176A-N (shown herein as 176A-N) may include a prism of suitable size and configuration, and includes a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prism of the optical layer 176A-N extends along all or at least a portion of the corresponding openings 175A, 175B in the left and right frames 107A, 107B, thereby allowing the user to see the second surface of the prism when the user's eyes look through the corresponding left and right frames 107A, 107B. The first surface of the prism of the optical layer 176A-N is arranged to face upward along the frame 105, and the display matrix 177 covers the prism, ensuring that photons and light emitted from the display matrix 177 hit the first surface. The size and shape of the prism ensure that light is refracted within the prism and directed to the user's eyes by the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N may be convex to direct light to the center of the eye. The prisms are optionally sized and shaped to magnify the image projected from the display matrix 177, with light passing through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 177.
[0041] In one example, the optical layers 176A-N may include a transparent LCD layer (keeping the lens open) unless a voltage is applied that makes the layer opaque (closing or blocking the lens). The image processor 912 on the eye-mounted device 100 may be programmed to apply a voltage to the LCD layer to create an active shutter system so that the eye-mounted device 100 is suitable for viewing visual content when displayed as a three-dimensional projection. In active shutter mode, technologies other than LCD may be used, including other types of reactive layers that may respond to a voltage or another type of input.
[0042] In another example, the image display device of the optical assembly 180A, 180B includes Figure 2B The projected image display shown. Each optical assembly 180A, 180B includes a respective laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as a laser projector 150 is disposed in or on one of the temples 125A, 125B of the eye-mounted device 100. The optical assembly 180B in this example includes one or more light strips 155A, 155B, ... 155N (in Figure 2B 155A-N), which may be spaced apart and arranged across the width of each optical component 180A, 180B lens and / or across the depth of the lens between the front and rear surfaces of the lens.
[0043] When the photons projected from the laser projector 150 pass through the lenses of each optical component 180A, 180B, the photons encounter the light bands 155A-N. When a particular photon encounters a particular light band, the photon is either redirected to the user's eye or passed to the next light band. The specific photons or light beams can be controlled by combining the laser projector 150 modulation and the light band modulation. In one example, the processor controls the light bands 155A-N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical components 180A, 180B, the eye-mounted device 100 may also adopt other arrangements, such as a single or three optical components, or, different optical component 180A, 180B arrangements may be adopted according to the application of the eye-mounted device 100 or the specific circumstances of the intended user.
[0044] like Figure 2A and 2B As further shown, the eyewear device 100 is provided with a left side block 110A adjacent to the left side 170A of the frame 105 and a right side block 110B adjacent to the right side 170B of the frame 105. The blocks 110A and 110B may be integrated into the frame 105 on the respective 170A and 170B sides (as shown in the figure), or implemented as separate components attached to the respective 170A and 170B sides of the frame 105. In addition, the blocks 110A and 110B may also be integrated into the temples 125A and 125B attached to the frame 105.
[0045] In another example, Figure 2B The eye-mounted device 100 shown may include two projectors, a left projector 150A (not shown) and a right projector 150B (shown as projector 150). The left optical assembly 180A may include a left display matrix 177A (not shown) and / or a left light strip group 155'A, 155'B, ... 155'N (155', A to N, not shown), which is configured to interact with light from the left projector 150A. Similarly, the right optical assembly 180B may include a right display matrix 177B (not shown) and / or a right light strip group 155"A, 155"B, ... 155"N (155", A to N, not shown), which is configured to interact with light from the right projector 150B. In this example, the eye-mounted device 100 includes a left display and a right display.
[0046] Figure 4 1 is a functional block diagram of an example video composition and sharing system 1000, which includes an eye-mounted device 100, a mobile device 890, and a server system 998 connected via various networks 995, such as the Internet. As shown, the system 1000 includes a low-power wireless connection 925 and a high-speed wireless connection 937 between the eye-mounted device 100 and the mobile device 890.
[0047] The eye-mounted device 100 includes one or more visible light cameras 114A, 114B that are capable of capturing still images and / or videos, as described herein. The cameras 114A, 114B may have direct memory access (DMA) permissions to the high-speed circuit 930. As described herein, a pair of cameras 114A, 114B may be used as a stereo camera. The cameras 114A, 114B may be used to capture an initial depth image that can be rendered into a three-dimensional (3D) model that is a texture-mapped image of a red, green, and blue (RGB) imaging scene. The device 100 may also include a depth sensor 213 that uses infrared signals to estimate the position of an object relative to the device 100. In some examples, the depth sensor 213 includes one or more infrared emitters 215 and an infrared camera 220.
[0048] The eye-mounted device 100 also includes two image displays corresponding to each optical component 180A, 180B (one associated with the left side 170A and one associated with the right side 170B). The eye-mounted device 100 also includes an image display driver 942, an image processor 912, a low power circuit 920, and a high speed circuit 930. The image display of each optical component 180A, 180B is used to present images, including still images and videos. The image display driver 942 is coupled to the image display of each optical component 180A, 180B to control the displayed image. The eye-mounted device 100 also includes a user input device 991 (e.g., a touch sensor or a touch pad) to receive a two-dimensional input selection from a user.
[0049] Figure 4 The components of the eye-mounted device 100 shown are located on one or more circuit boards, such as a printed circuit board or a flexible printed circuit board in the frame or temple. Alternatively, or in addition, the components shown can be located in a block, frame, hinge, or nose bridge of the eye-mounted device 100. The left and right visible light cameras 114A, 114B can include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including still images or videos of scenes with unknown objects).
[0050] like Figure 4As shown, the high-speed circuit 930 includes a high-speed processor 932, a memory 934, and a high-speed wireless circuit 936. In this example, an image display driver 942 is coupled to the high-speed circuit 930 and is operated by the high-speed processor 932 to drive the left and right image displays of each optical assembly 180A, 180B. The high-speed processor 932 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required by the eye-mounted device 100. The high-speed processor 932 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over a high-speed wireless connection 937 using the high-speed wireless circuit 936. In some examples, the high-speed processor 932 executes an operating system of the eye-mounted device 100, such as a LINUX operating system or other such operating system, and the operating system is stored in the memory 934 for execution. In addition to any other tasks, the high-speed processor 932, which executes the software architecture of the eye-mounted device 100, is used to manage data transmission with the high-speed wireless circuit 936. In some examples, high-speed wireless circuit 936 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by high-speed wireless circuit 936.
[0051] The low power circuit 920 includes a low power processor 922 and a low power wireless circuit 924. The low power wireless circuit 924 and the high speed wireless circuit 936 of the eye-mounted device 100 may include a short range transceiver (Bluetooth TM ) and a wireless wide area, local area or wide area network transceiver (e.g., cellular or WiFi). The mobile device 890 includes a transceiver that communicates via a low power wireless connection 925 and a high speed wireless connection 937, and can be implemented using details of the architecture of the eye-mounted device 100, as can other elements of the network 995.
[0052] The memory 934 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 114A, 114B, the infrared camera 220, the image processor 912, and images generated by the image display driver 942 for display on the image display of each optical assembly 180A, 180B. Although the memory 934 is shown as being integrated with the high-speed circuit 930, in other examples, the memory 934 can be a separate, offline component of the eye-mounted device 100. In some such examples, the circuit line can provide a connection from the image processor 912 or the low-power processor 922 to the memory 934 through a chip including the high-speed processor 932. In other examples, the high-speed processor 932 can manage the addressing of the memory 934 so that the low-power processor 922 will start the high-speed processor 932 whenever a read or write operation involving the memory 934 is required.
[0053] like Figure 4 As shown, the high-speed processor 932 of the eye-mounted device 100 can be connected to the camera system (visible light cameras 114A, 114B), the image display driver 942, the user input device 991 and the memory 934. Figure 5 As shown, the CPU 830 of the mobile device 890 can be coupled to the camera system 870, the mobile display driver 882, the user input layer 891, and the memory 840A. The eye-mounted device 100 can perform all or a subset of any functions described herein generated by the eye-mounted device 100 processor 932 executing the video synthesis system 500 in the memory 934. The mobile device 890 can perform all or a subset of any functions described herein generated by the CPU 830 of the mobile device 890 executing the video synthesis system 500 in the flash memory 840A. The functions can be divided in the video synthesis system 500 so that the eye-mounted device 100 captures video, collects inertial measurement unit data, and collects sensor data, and the mobile device 890 performs calculations, operations, and combination functions.
[0054] The server system 998 can be one or more computing devices as part of a service or network computing system, for example, a network computing system including a processor, a memory, and a network communication interface to communicate with the eye-mounted device 100 and the mobile device 890 via the network 995.
[0055] The output component of the eye-mounted device 100 includes a visual element, such as Figure 2A and 2BThe left and right image displays associated with each lens or optical component 180A, 180B described in the figure (e.g., the following displays: liquid crystal display (LCD), plasma display panel (PDP), light emitting diode (LED) display, projector or waveguide). The eye-mounted device 100 may include indicators facing the user (e.g., LED, speaker or vibration actuator), and / or signals facing outward (e.g., LED, speaker). The image display of each optical component 180A, 180B is driven by the image display driver 942. In some example configurations, the output components of the eye-mounted device 100 also include additional indicators, such as auditory elements (e.g., speakers), tactile components (e.g., actuators, such as vibration motors that generate tactile feedback), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The configuration of the indicator group facing the user can ensure that it can be seen or otherwise perceived by the user of the device 100. For example, the device 100 may include an LED display arranged to be visible to a user, a speaker arranged to produce a sound audible to a user, or an actuator that provides tactile feedback that a user can feel. The outward-facing signal group configuration can ensure that it is seen or felt by an observer near the device 100. Similarly, the device 100 may include LEDs, speakers, or actuators configured and arranged to ensure that they can be felt by an observer.
[0056] The input components of the eye-mounted device 100 may include alphanumeric input components (e.g., a touch screen or touchpad configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configuration element), pointer-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointer tool), tactile input components (e.g., button switches, a touch screen or touchpad that senses touch or touch gesture location and / or force, or other tactile configuration elements), and audio input components (e.g., a microphone), and the like. The mobile device 890 and the server system 998 may include alphanumeric, pointer-based, tactile, audio, and other input components.
[0057] In some examples, the eye-mounted device 100 includes a sensor array 980, such as Figure 4 As shown. The elements of sensor array 980 include visible light cameras 114A, 114B, user input device 991 (e.g., touch screen or touchpad, push button switch), microphone 993 (e.g., an array of two or more microphones), depth sensor 213, and a collection of motion sensing components called inertial measurement unit 972. The motion sensing components can be micro-electromechanical systems (MEMS) with tiny moving parts, which are usually small enough to be part of a microchip.
[0058] In some example configurations, the inertial measurement unit (IMU) 972 includes an accelerometer 974, a gyroscope 976, and a magnetometer 978. The accelerometer 974 senses the linear acceleration (including the acceleration of gravity) of the device 100 relative to three orthogonal axes (x, y, z). The gyroscope 976 senses the angular velocity of the device 100 around three rotation axes (pitch, roll, yaw). The accelerometer 974 and the gyroscope 976 can work together to provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer 978 senses the direction of travel of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor, such as a GPS receiver, one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates can also be received from the mobile device 890 via a low-power wireless circuit 924 or a high-speed wireless circuit 936 via a wireless connection 925, 937.
[0059] The inertial measurement unit 972 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values about the position, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer 974 is integrated to obtain velocity data relative to each axis (x, y, z); and then integrated again to obtain the position (in linear coordinates x, y, and z) data of the device 100. The angular velocity data from the gyroscope 976 can be integrated to obtain the position (in spherical coordinates) data of the device 100. The program for calculating these useful values can be stored in the memory 934 and executed by the high-speed processor 932 of the eye-mounted device 100.
[0060] The eye-mounted device 100 may optionally include additional peripheral sensors, such as biometric sensors, specialized sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device elements may include any input / output components, including output components, motion components, position components, or any other such components described herein. For example, the biometric sensors may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or recognize humans (e.g., recognition based on voice, retina, facial features, fingerprints, or electrical biosignals such as EEG data), etc.
[0061] like Figure 4As shown, the video composition and sharing system 1000 includes a computing device, such as a mobile device 890, coupled to the eye-mounted device 100 via a network. As described herein, the sensor array 980 of the eye-mounted device 100 includes one or more visible light cameras 114A, 114B for capturing a series of raw video clips, and an inertial measurement unit 972 for collecting position, orientation, and motion data about the eye-mounted device 100.
[0062] The video composition and sharing system 1000 also includes a memory for storing instructions, including instructions in the video composition system 500, and a processor for executing instructions. The system 1000 can utilize the memory 934 of the eye-mounted device 100 and / or the storage elements 840A, 840B ( Figure 5 ). In addition, the system 1000 can utilize the processor elements 932, 922 of the eye-mounted device 100 and / or the central processing unit (CPU) 830 ( Figure 5 ). In addition, the system 1000 can also utilize memory and processor elements of the server system 998. In this regard, the memory and processing functions of the video composition and sharing system 1000 can be shared or distributed between the eye-mounted device 100 and the mobile device 890 and / or the server system 998.
[0063] The mobile device 890 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eye-mounted device 100 using both the low power wireless connection 925 and the high speed wireless connection 937. The mobile device 890 accesses the server system 998 and the network 995. The network 995 may include any combination of wired and wireless connections.
[0064] Figure 5 840A is a high-level functional block diagram of an example mobile device 890. The mobile device 890 includes a flash memory 840A including programs to perform all or a subset of the functions described herein. The mobile device 890 may include a camera 870 including at least two visible light cameras (a first and a second visible light camera with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 840A may also include a plurality of images or videos generated via the camera 870.
[0065] As shown in the figure, the mobile device 890 includes an image display 880, a mobile display driver 882 and a controller 884 for controlling the image display 880. Figure 4 In the example of FIG. 8 , the image display 880 includes a user input layer 891 (eg, a touch screen) that is layered on top of or integrated into a screen used by the image display 880 .
[0066] Examples of touch screen mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example, and the subject technology described herein is not intended to be limited thereto. For purposes of this discussion, Figure 4 A block diagram illustration of an example mobile device 890 is provided with a user interface including a touch screen input layer 891 for receiving input (via hand, stylus or other tool touch, multi-touch or gesture, etc.) and an image display 880 for displaying content.
[0067] like Figure 4 As shown, the mobile device 890 includes at least one digital transceiver (XCVR) 810, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 890 also includes additional digital or analog transceivers, such as a short-range XCVR 820 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 820 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.
[0068] The mobile device 890 may include a global positioning system (GPS) receiver to generate location coordinates for locating the mobile device 890. Alternatively, or in addition, the mobile device 890 may utilize one or both of the short-range XCVR 820 and the WWAN XCVR 810 to generate location coordinates for positioning. For example, a cellular network, WiFi, or Bluetooth-based TM The positioning system of the eyewear can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the eyewear device via the XCVR 810, 820 through one or more network connections.
[0069] The transceivers 810, 820 (i.e., network communication interfaces) conform to one or more digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 810 include, but are not limited to, transceivers configured to operate according to code division multiple access (CDMA) and third generation partnership project (3GPP) network technologies, such as, but not limited to, third generation partnership project 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 810, 820 provide two-way wireless information communication, including digitized audio signals, still images and video signals, web page information for display and input related to the web page, and various types of mobile message communication with mobile device 890.
[0070] The mobile device 890 also includes a microprocessor that functions as a central processing unit (CPU), such as Figure 4 830 in . A processor is a circuit whose elements are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, components that constitute a programmable CPU are used in the example. For example, a microprocessor includes one or more integrated circuit (IC) chips that contain electronic components that perform the functions of a CPU. For example, CPU 830 can be based on any known or available microprocessor architecture, such as reduced instruction set computing (RISC) using the ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits can be used to make up CPU 830 or processor hardware in smartphones, laptops, and tablets.
[0071] The CPU 830 functions as a programmable main controller for the mobile device 890 by configuring the mobile device 890 to perform various operations, for example, according to instructions or programs executable by the CPU 830. For example, such operations may include various general operations of the mobile device, as well as operations related to application programming on the mobile device. Although the processor may be configured by using hard-wired logic, a typical processor in a mobile device is a general processing circuit that is configured by executing programming.
[0072] The mobile device 890 includes a memory or storage system for storing programs and data. In this example, the memory system may include flash memory 840A, random access memory (RAM) 840B, and other memory components as needed. RAM 840B is used as short-term storage of instructions and data processed by CPU 830, such as working data processing memory. Flash memory 840A generally provides longer-term storage.
[0073] Thus, in the example of mobile device 890, flash memory 840A is used to store programming or instructions executed by CPU 830. Depending on the type of device, mobile device 890 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.
[0074] Reference again Figure 4 , the video composition and sharing system 1000 includes a goggled device 100 coupled to a mobile device 890 via a network 995. The memory 934 includes the video composition system 500, a capture profile 510, and a processing profile 520. The execution of the instructions of the video composition system 500 by the processor 932 can configure the goggled device 100 to - according to the capture profile 510 - capture a series 540 of raw video clips and collect a data set 572 associated with each video clip in the series 540. Each data set 572 includes camera orientation related information collected by the inertial measurement unit 972. In some embodiments, each data set 572 includes camera orientation related information associated with each frame in each video clip.
[0075] After capturing the series of video clips 540, continued execution of instructions of the video composition system 500 by the mobile CPU 830 configures the mobile device 890 to process the series of video clips 540 according to the processing profile 520, according to one embodiment. In some example configurations, the processing profile 520 includes a stabilization setting. Example stabilization settings include "No" (no stabilization is performed during processing), "Yes", and "Custom" (where a user selects a particular type of stabilization). The default stabilization setting may be set to "Yes" and may include instructions to: (1) calculate a camera orientation 585 associated with each original video clip based on the data set 572 associated with each particular video clip, (2) calculate a stabilized output path 590 based on the calculated camera orientation 585, and (3) combine the series of video clips 540 according to the stabilized output path 590 to generate a video composition file 600.
[0076] The capture profile 510 is a set of conditions or variables that determine how and when the camera 114A, 114B on the eye-mounted device 100 will capture the series of raw video clips 540. In one example, the capture profile 510 includes a start condition, a clip duration, a pause duration (between clips), and a number of clips. For example, the capture profile 510 may include a start condition equivalent to 9:18 pm, a clip duration of six seconds, a pause duration of twenty seconds, and a number of ten clips. According to the capture profile, the camera 114A, 114B on the eye-mounted device 100 will capture ten six-second video clips starting at 9:18 pm, with a twenty-second pause between each clip. Because the conditions are set before the video is captured and do not change, the example capture profile 510 is also called a fixed capture profile.
[0077] The original video clip series 540 includes at least first and second original video clips. The first original video clip includes a plurality of first frames. The second original video clip includes a plurality of second frames, and so on.
[0078] An inertial measurement unit (IMU) 972 of the eye-mounted device 100 collects position, orientation, and motion information about the eye-mounted device 100 during capture. The video synthesis system 500 configures the eye-mounted device 100 to collect data from the inertial measurement unit during the capture of each video clip. The eye-mounted device 100 will collect a first data set from the inertial measurement unit 972 during the capture of a first raw video clip; then collect a second data set from the inertial measurement unit 972 during the capture of a second raw video clip, and so on, until the number of clips specified in the specific capture profile 510 is captured. Each data set from the inertial measurement unit 972 is associated with at least one frame of the captured raw video clip. In some embodiments, the eye-mounted device 100 will collect a first data set from the inertial measurement unit 972 associated with each first frame of the first raw video clip. The data sets 572 collected from the inertial measurement unit 972 can be stored in the memory 934, such as Figure 4 shown.
[0079] For each data set 572 stored in memory 934, video composition system 500 configures mobile device 890 to calculate camera orientation 585. Camera orientation 585 calculated by mobile device 890 may be stored in memory 840A, such as Figure 5 shown.
[0080] The video synthesis system 500 configures the mobile device 890 to calculate a stabilized output path 590 based on the camera orientation 585 associated with each video clip. The stabilized output path 590 can be calculated based on any video stabilization algorithm and process, including algorithms and processes used in time-lapse photography and other video synthesis systems. In some embodiments, the stabilized output path 590 is a hypothetical three-dimensional route that the camera travels from the first original video clip to the last original video clip. The camera orientation 585 associated with each video clip includes a discrete position and direction (x, y, z, pitch, roll, yaw), which can be used to draw the path points along the hypothetical route that the camera travels. One or more frames in each video clip can be moved or adjusted to ensure that each path point is on the stabilized output path 590. In some algorithms, one or more frames in each video clip can be moved or otherwise adjusted to ensure that each frame remains stable relative to a reference plane (e.g., a wall, a table, or a horizon). Arranging one or more frames of each video segment along the stabilized output path 590 may generate a final video composite file 600 that is smoother and more stable than the original series of video segments 540 .
[0081] In some example embodiments, by executing instructions of the video composition system 500, the eye-mounted device 100 may be configured according to a capture profile 510 to collect sensor data 580, which includes information collected by a sensor array 980. The sensor array 980 includes cameras 114A, 114B, a user input device 991 such as a touchpad, a microphone 993, a depth sensor 213, and an inertial measurement unit (IMU) 972. In this example, the capture profile 510 is a reactive capture profile; rather than a fixed set of conditions, the conditions react to the sensor data 580 collected in real time from the sensor array 980. For example, the reactive capture profile may include a reactive start condition, which is configured to start recording when the sensor data 580 includes a start input. The start input may be an auditory start command received by a microphone 993 in the sensor array 980, a physical start command received by an input device 991 (e.g., a tap or tap pattern on a touchpad), a start gesture made by the wearer and captured by the cameras 114A, 114B, a selected orientation (vertical, horizontal, stationary for a period of time) of the eye-mounted device 100 collected by the inertial measurement unit 972 as indicated by the data set 572, or a selected lighting condition captured by the cameras 114A, 114B. The selected orientation of the eye-mounted device 100 may include a position component and a time component. For example, the selected orientation may include a substantially horizontal position (e.g., on the face) maintained for a minimum time period (e.g., maintained for a few seconds without moving beyond a threshold). Similarly, the selected orientation may include a substantially vertical position (e.g., in a pocket, hanging on a shirt collar) maintained for a minimum period of time.
[0082] In another example, a reactive capture profile may include a reactive segment duration configured to adjust (extend or shorten) the current segment duration when the sensor data 580 includes information from the microphone 993 indicating that speech is ongoing. In this regard, the current segment duration may be adjusted to avoid interrupting or cutting off active speech of the user, observer, or other user near the eye-mounted device 100. During the capture of the original video segment, the microphone 993 provides real-time sensor data 580. If active speech is detected and ends before the scheduled or default segment duration, the current segment duration may be adjusted (shortened) to end after the speech ends. If active speech is detected and continues at or near the end of the scheduled or default segment duration, the current segment duration may be adjusted (extended) to end after the speech ends. In one embodiment, the current segment duration is adjusted to include: (1) an active period that lasts as long as the microphone indicates that speech is ongoing, and (2) a buffer period (e.g., one or two seconds) after the microphone indicates that the speech has ended to avoid interrupting or abruptly ending the speech.
[0083] In a related aspect, a start input associated with active speech can be used to adjust when to (a) end the pause between segments, and (b) begin capturing the next raw video segment. A reactive capture profile may include a reactive pause duration that is configured to support adjusting (lengthening or shortening) the current pause when sensor data 580 includes information from microphone 993 indicating that speech is ongoing. This reactive setting can avoid starting a new raw video segment in the middle of active speech. If active speech is detected and continues at or near the end of a scheduled or default pause duration, the current pause duration can be adjusted (lengthened) to end after the speech ends. In one embodiment, the current pause duration is adjusted to include: (1) an active period that lasts as long as the microphone indicates that speech is ongoing, and (2) a buffer period (e.g., one or two seconds) after the microphone indicates that speech has ended to avoid capturing the end of the speech in the next raw video segment. The end of the pause duration or process coincides with the beginning of the next raw video segment.
[0084] The reactive capture profile may include a reactive quantity segment that ensures that the eye-mounted device 100 is configured to stop the current segment when the sensor data 580 collected in real time from the sensor array 980 includes a stop input. The stop input may be an auditory stop command received by the microphone 993, a physical stop command received by the input device 991 (e.g., a tap or tap pattern on a touchpad), a stop gesture made by the wearer and captured by the cameras 114A, 114B, a stop camera orientation collected by the inertial measurement unit 972 as indicated by the data set 572, a stop lighting condition captured by the cameras 114A, 114B, or a low battery or zero battery condition indicated by the eye-mounted device 100.
[0085] The reactive capture profile may include a reactive camera identifier, ensuring that the eye-mounted device 100 is configured to respond to sensor data 580 collected in real time from the sensor array 980 to activate a specific camera (left, right, left and right). For example, if the environmental conditions sensed by the left camera 114A indicate that the lighting level nearby is insufficient (e.g., completely dark, obscured, too dark, or too bright to capture usable video), the device 100 may deactivate the left camera 114A and activate the right camera 114B based on the reactive camera identifier. In another example, when the sensor data 580 collected in real time from the sensor array 980 indicates that the environmental conditions or device movement are particularly suitable for composition and later 3D display, the device may activate both cameras 114A, 114B for stereoscopic recording (for later 3D viewing).
[0086] In another aspect, the capture profile 510 may include one or more settings that control a user-facing indicator and / or an outward-facing signal. The size, shape, and position settings of the user-facing indicator may support event warnings or notifications to the wearer of the eye-mounted device 100. For example, a user-facing LED may be configured to turn on when recording begins, or a user-facing speaker may be configured to sound an audible prompt when recording ends. In some examples, the user-facing indicator is a tactile signal generated by a vibration actuator connected to the eye-mounted device 100, or an audible sound from a speaker connected to the eye-mounted device 100.
[0087] The size, shape, and location of the outward-facing signal may support event alerts or notifications to observers other than the wearer of the eye-mounted device 100. For example, the outward-facing LED may be configured to turn on when recording is in progress to notify an observer that video capture is in progress.
[0088] In some examples, the processing profile 520 includes a steady state setting (e.g., no, yes, a personalized selection of a particular steady state algorithm), spacing effects (visual effects applied between segments, if any; e.g., cut, fade, dissolve, wipe), visual layers (e.g., tint settings, color gradients, borders, frames, overlay elements), audio layers (e.g., silence, ambient sounds, music tracks, voice-over narration), and playback speed (relative to the recording speed). Each setting may include a default setting, as well as a user interface that enables a user to adjust each setting. For example, a default processing profile 520 may include "yes" for a steady state setting, "cut" for spacing effects (making the video composite file 600 seamless between segments), "none" for a visual layer (no tint setting or added elements), "silent" for an audio layer, and "2x" for playback speed.
[0089] In related aspects, the processing profile 520 can be one or more reactive settings; that is, settings that change based on the first or second data set and / or the first or second camera orientation and / or the stabilized output path. For example, information within the first data set can indicate that a particular stabilization algorithm is well suited for that data. Information contained in the stabilized output path can indicate that a particular playback speed is well suited for video composition during playback.
[0090] As described herein, any video synthesis and sharing functions described herein for the eye-mounted device 100, the mobile device 890, and the server system 998 can be implemented in one or more computer software applications or programming instruction sets. According to some examples, a "single function", "multiple functions", "single application", "multiple applications", "single instruction", "multiple instructions", or "programming" is a program for performing the functions defined in the program. Various programming languages can be used to create one or more applications structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) applications can be developed on IOS TM ANDROID TM , Mobile software running on a mobile operating system such as iOS or other mobile operating systems such as Android phones. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article.
[0091] Thus, machine-readable media may take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer device, etc., such as may be used to implement a client device, a media gateway, a code converter, etc. As shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including the wires that make up a bus within a computer system. Carrier transmission media may take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example: a floppy drive, a floppy disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, punched card tape, any other physical storage medium using a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or memory cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read program code and / or data. Many such forms of computer readable media may be involved in conveying one or more sequences of one or more instructions to a processor for execution.
[0092] Except as stated in the preceding paragraph, what is described and shown is not intended and should not be construed to cause any element, step, function, object, benefit, advantage, or equivalent to be exclusive to the public regardless of whether it is recited in the claims.
[0093] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meanings assigned to them by their respective fields of investigation and study. Relational terms such as "first" and "second" may be used only to distinguish one entity or action from another entity or action, without requiring or implying any actual such relationship or order between such entities or actions. The terms "comprise", "include", "contain", "included" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising or including a series of elements or steps includes not only these elements or steps, but also other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. Without further limitation, an element preceded by "a" does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0094] Unless otherwise stated, any and all measurements, values, ratings, positions, magnitudes, dimensions, etc., described in this specification (including the claims that follow) are approximate values, not exact values. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and consistent with the practices in the field to which they belong. For example, unless otherwise expressly stated, parameter values, etc. may vary from the specified amount by ±10%.
[0095] Furthermore, in the foregoing detailed description, it can be seen that various examples group various features together in order to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, and each claim stands on its own as a separately claimed subject matter.
[0096] Although the foregoing describes what are believed to be the best modes and other examples, it is understood that various modifications may be made in such modes and examples, that the subject matter disclosed herein may be implemented in various forms and examples, and that such modes and examples may be applied to a variety of applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.
Claims
1. A video synthesis method, comprising: capturing raw video frames using a camera coupled to an electronic eyewear device that moves along a series of locations in a physical environment; Collecting motion data using an inertial measurement unit coupled to the electronic eyewear device; dividing the raw video frame into a series of raw video segments according to a reactive capture profile, wherein the reactive capture profile includes a plurality of variables, the variables including a start condition, a segment duration, a pause duration, and a number of segments, wherein one or more of the plurality of variables are updated based on one or more of the raw video frame and the motion data; calculating a stabilized output path based on the motion data and the reactive capture profile such that the stabilized output path approximates the series of positions, and wherein the calculating comprises: (a) estimating camera directions associated with a plurality of selected segments of the series of original view segments, and (b) establishing one or more waypoints along the stabilized output path based on the camera directions; as well as A video composite is generated based on the stabilized output path, wherein generating the video composite includes adjusting a plurality of selected segments relative to one or more path points such that at least a portion of the video composite appears stabilized relative to a reference plane.
2. The video synthesis method according to claim 1, wherein adjusting the plurality of selected segments further comprises: identifying the reference plane in the original video frame; Calculating the distance between each of the camera directions and the one or more path points; as well as The plurality of selected segments are mathematically adjusted based on the distance.
3. The video synthesis method according to claim 1, wherein the inertial measurement unit comprises an accelerometer, a gyroscope and a magnetometer, and wherein collecting the motion data further comprises: Collecting linear accelerations relative to three orthogonal axes based on the accelerometer; Acquiring angular velocities relative to three rotation axes based on the gyroscope; as well as The direction of travel relative to magnetic north is acquired based on the magnetometer.
4. The video synthesis method according to claim 1, further comprising: The raw video frame is captured in response to the start condition, wherein the start condition includes a start input selected from the start group consisting of: (a) an auditory start command received by a microphone coupled to the electronic eyewear device, (b) a physical start command received by a touch pad coupled to the electronic eyewear device, (c) a start gesture captured by the camera, (d) a selected direction sensed by the inertial measurement unit, and (e) a selected lighting condition based on the camera.
5. The video synthesis method according to claim 1, wherein: Dividing the original video frame according to the segment duration further comprises: The current original video segment duration is adjusted to include an active period and a buffer period, wherein the active period is characterized by speech received by a microphone coupled to the electronic eye-mounted device, and wherein the buffer period includes a predetermined period after the active period.
6. The video synthesis method according to claim 1, wherein: Dividing the frames of the original video according to the segment duration also includes: The current raw video segment duration is terminated in response to a stop input, wherein the stop input is selected from a stop group, the stop group including (a) an auditory stop command received by a microphone coupled to the electronic eye-mounted device, (b) a physical stop command received by a touch pad coupled to the electronic eye-mounted device, (c) a stop gesture captured by the camera, (d) a stop direction sensed by the inertial measurement unit, (e) a stop lighting condition based on the camera, and (f) a low battery state sensed by the electronic eye-mounted device.
7. The video synthesis method according to claim 1, further comprising: dividing the raw video frame into a series of raw video segments according to a fixed capture profile, wherein the fixed capture profile includes a fixed start condition, a fixed segment duration, a fixed pause duration, and a fixed number of segments; and The stabilized output path is calculated based on the fixed capture profile.
8. The video synthesis method according to claim 1, further comprising: transmitting the video composition to a mobile device including a mobile memory and a mobile processor; as well as The video composition is generated using a mobile processor according to the stabilized output path and a processing profile stored in a mobile memory, wherein the processing profile includes steady-state settings, interval effects, visual layers, audio layers, and playback speed.
9. A video synthesis system, comprising: An electronic eyewear device including a memory, a processor, a camera, and an inertial measurement unit; A program stored in the memory, wherein the program is executed by the processor to configure the electronic eyewear device to perform functions, including the following functions: capturing raw video frames using the camera, wherein the electronic eyewear moves along a series of locations in a physical environment; Using the inertial measurement unit to collect motion data; dividing the raw video frame into a series of raw video segments according to a reactive capture profile, wherein the reactive capture profile includes a plurality of variables, the variables including a start condition, a segment duration, a pause duration, and a number of segments, wherein one or more of the plurality of variables are updated based on one or more of the raw video frame and the motion data; calculating a stabilized output path based on the motion data and the reactive capture profile, such that the stabilized output path approximates the series of positions, and wherein the calculating function further comprises the following functions: (a) estimating camera directions associated with a plurality of selected segments in a sequence of original view segments, and (b) establishing one or more waypoints along the stabilized output path based on the camera directions; as well as A video composite is generated based on the stabilized output path, wherein the generated functionality includes functionality to adjust a plurality of selected segments relative to the one or more path points such that at least a portion of the video composite appears stabilized relative to a reference plane.
10. The video composition system according to claim 9, wherein the function of adjusting the plurality of selected segments further comprises the following functions: identifying the reference plane in the original video frame; calculating the distance between each of the camera directions and the one or more path points; and The plurality of selected segments are mathematically adjusted based on the distance.
11. The video synthesis system of claim 9, wherein the inertial measurement unit comprises an accelerometer, a gyroscope, and a magnetometer, and wherein the function of collecting the motion data further comprises the following functions: Collecting linear accelerations relative to three orthogonal axes based on the accelerometer; Acquiring angular velocities relative to three rotation axes based on the gyroscope; and The direction of travel relative to magnetic north is acquired based on the magnetometer.
12. The video synthesis system according to claim 9, further comprising the following functions: The raw video frame is captured in response to the start condition, wherein the start condition includes a start input selected from the start group consisting of: (a) an auditory start command received by a microphone coupled to the electronic eyewear device, (b) a physical start command received by a touch pad coupled to the electronic eyewear device, (c) a start gesture captured by the camera, (d) a selected direction sensed by the inertial measurement unit, and (e) a selected lighting condition based on the camera.
13. The video composition system according to claim 9, wherein the function of dividing the original video frame according to the segment duration further comprises the following functions: The current original video segment duration is adjusted to include an active period and a buffer period, wherein the active period is characterized by speech received by a microphone coupled to the electronic eye-mounted device, and wherein the buffer period includes a predetermined period after the active period.
14. The video composition system according to claim 9, wherein the function of dividing the original video frame according to the segment duration further comprises the following functions: The current raw video segment duration is terminated in response to a stop input, wherein the stop input is selected from a stop group, the stop group including (a) an auditory stop command received by a microphone coupled to the electronic eye-mounted device, (b) a physical stop command received by a touch pad coupled to the electronic eye-mounted device, (c) a stop gesture captured by the camera, (d) a stop direction sensed by the inertial measurement unit, (e) a stop lighting condition based on the camera, and (f) a low battery state sensed by the electronic eye-mounted device.
15. The video synthesis system according to claim 9, further comprising the following functions: The original video frame is divided into a series of original video segments according to a fixed capture profile, wherein: The fixed capture profile includes a fixed start condition, a fixed segment duration, a fixed pause duration, and a fixed number of segments; and The stabilized output path is calculated based on the fixed capture profile.
16. The video synthesis system according to claim 9, further comprising the following functions: transmitting the video composition to a mobile device comprising a mobile memory and a mobile processor; and The video composition is generated using a mobile processor according to the stabilized output path and a processing profile stored in a mobile memory, wherein the processing profile includes steady-state settings, interval effects, visual layers, audio layers, and playback speed.
17. A non-transitory computer readable medium storing program code, which, when executed, causes an electronic processor coupled to an electronic eyewear device to perform the following steps: capturing raw video frames using a camera coupled to an electronic eyewear device that moves along a series of locations in a physical environment; Collecting motion data using an inertial measurement unit coupled to the electronic eyewear device; dividing the raw video frame into a series of raw video segments according to a reactive capture profile, wherein the reactive capture profile includes a plurality of variables, the variables including a start condition, a segment duration, a pause duration, and a number of segments, wherein one or more of the plurality of variables are updated based on one or more of the raw video frame and the motion data; calculating a stabilized output path based on the motion data and the reactive capture profile such that the stabilized output path approximates the series of positions, and wherein the calculating comprises: (a) estimating camera directions associated with a plurality of selected segments of the series of original view segments, and (b) establishing one or more waypoints along the stabilized output path based on the camera directions; as well as A video composite is generated based on the stabilized output path, wherein generating the video composite includes adjusting a plurality of selected segments relative to one or more path points such that at least a portion of the video composite appears stabilized relative to a reference plane.
18. The non-transitory computer readable medium of claim 17, wherein execution of the program code is operable to cause the electronic processor to perform the following further steps: identifying the reference plane in the original video frame; calculating the distance between each of the camera directions and the one or more path points; and The plurality of selected segments are mathematically adjusted based on the distance.
19. The non-transitory computer readable medium of claim 17, wherein execution of the program code is operable to cause the electronic processor to perform the following further steps: capturing a raw video frame in response to the start condition, wherein the start condition comprises a start input selected from a start group consisting of: (a) an auditory start command received by a microphone coupled to the electronic eyewear device, (b) a physical start command received by a touchpad coupled to the electronic eyewear device, (c) a start gesture captured by the camera, (d) a selected direction sensed by the inertial measurement unit, and (e) a selected lighting condition based on the camera; adjusting the current original video segment duration to include an active period and a buffer period, wherein the active period is characterized by speech received by a microphone coupled to the electronic eye-mounted device, and wherein the buffer period includes a predetermined period after the active period; as well as The current raw video segment duration is terminated in response to a stop input, wherein the stop input is selected from a stop group, the stop group including (a) an auditory stop command received by a microphone coupled to the electronic eye-mounted device, (b) a physical stop command received by a touch pad coupled to the electronic eye-mounted device, (c) a stop gesture captured by the camera, (d) a stop direction sensed by the inertial measurement unit, (e) a stop lighting condition based on the camera, and (f) a low battery state sensed by the electronic eye-mounted device.
20. The non-transitory computer readable medium of claim 17, wherein execution of the program code is operable to cause the electronic processor to perform the following further steps: transmitting the video composition to a mobile device comprising a mobile memory and a mobile processor; and The video composition is generated using a mobile processor according to the stabilized output path and a processing profile stored in a mobile memory, wherein the processing profile includes steady-state settings, interval effects, visual layers, audio layers, and playback speed.
