Coordination of slow action video and audio

Abstract

An apparatus includes a memory configured to store data corresponding to a media stream, the data including video and a plurality of audio segments recorded at a first frame rate, the plurality of audio segments includes one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. The device also includes one or more processors coupled to the memory and operatively configured to initiate playback of the media stream, the playback of the media stream including: playing back the video at a second frame rate different from the first frame rate, the one or more first audio segments are played back at a first playback speed corresponding to the first frame rate, and the one or more second audio segments are played back at a second playback speed corresponding to the second frame rate.

Description

[0001] I. Cross-Reference to Related Applications

[0002] This application claims the benefit of priority from commonly owned U.S. non-provisional patent application No. 17 / 932,361 filed on September 15, 2022, the entire contents of which are expressly incorporated herein by reference.

[0003] II. Technical field

[0004] The present disclosure generally relates to generating and coordinating slow motion video and audio.

[0005] III. Description of Related Art

[0006] Technological advances have led to smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones (such as mobile and smart phones, tablet computers and laptop computers) that are small, lightweight and easy for users to carry. These devices can communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional functions, such as digital still cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions, including software applications, such as web browser applications, which can be used to access the Internet. As such, these devices can include significant computing capabilities.

[0007] Slow motion recording is a widely used feature in such devices, but most slow motion recordings are hampered by the absence of audio or by misaligned, poor audio synchronization. For illustration, conventional techniques treat audio as a single data stream with an associated playback speed. By changing the playback speed of conventional video and audio files, slow motion audio behavior can be simulated.

[0008] However, slow motion video is not simply a regular video that is played back slowly. Instead, slow motion video is recorded at a high frame rate and played back at a slower frame rate, thereby producing a slow motion effect. Attempting to generate slow motion audio by reducing the playback speed of an audio stream (such as an audio stream recorded with a high frame rate video recording) results in misalignment and poor synchronization of the audio with the video during playback. This can result in a poor user experience and poor video quality, and is one reason why most conventional portable computing devices, such as smartphones, do not record audio in a slow motion recording mode.

[0009] Another reason why many conventional portable computing devices are unable to play back audio in slow motion video is the poor audio perception that accompanies slow motion audio playback. To illustrate, spoken words captured in an audio recording during a high frame rate video recording may be converted into a slow-paced warbling that is incomprehensible or difficult for a viewer to understand when played back in slow motion.

[0010] Solving the above problems (synchronization between slowed-down audio and slow-motion video during playback, improving the intelligibility of audio when played back in slow motion, or both) will enable improved audio playback accompanying slow-motion video playback and enhance the user experience.

[0011] IV. Summary of the Invention

[0012] According to a specific implementation of the present disclosure, a device includes: a memory configured to store data corresponding to a media stream, the data including a video and a plurality of audio segments recorded at a first frame rate. The plurality of audio segments include one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. The device also includes one or more processors coupled to the memory and operably configured to initiate playback of the media stream. The playback of the media stream includes: playing back the video at a second frame rate different from the first frame rate, playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate, and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0013] According to another specific implementation of the present disclosure, a method includes: obtaining data corresponding to a media stream at one or more processors, the data including a video and a plurality of audio segments recorded at a first frame rate. The plurality of audio segments include one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. The method also includes initiating playback of the media stream at the one or more processors. The playback of the media stream includes: playing the video at a second frame rate different from the first frame rate, playing the one or more first audio segments at a first playback speed corresponding to the first frame rate, and playing the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0014] According to another specific implementation of the present disclosure, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to: obtain data corresponding to a media stream, the data including: a video recorded at a first frame rate and a plurality of audio segments. The plurality of audio segments include one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. When executed by the one or more processors, the instructions also cause the one or more processors to initiate playback of the media stream. The playback of the media stream includes: playing back the video at a second frame rate different from the first frame rate, playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate, and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0015] According to another specific implementation of the present disclosure, a device includes: a component for storing data corresponding to a media stream, the data including a video and a plurality of audio segments recorded at a first frame rate. The plurality of audio segments include one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. The device also includes a component for initiating playback of the media stream, the playback of the media stream including: playing back the video at a second frame rate different from the first frame rate, playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate, and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0016] Other aspects, advantages, and features of the present disclosure will become apparent upon review of the entire application, including the Brief Description of the Drawings, Detailed Description, and Claims.

[0017] V. Description of the Figures

[0018] Figure 1 is a block diagram of certain illustrative aspects of a system configured to coordinate slow-motion video and audio according to some examples of the present disclosure.

[0019] Figure 2 is a graphical representation of audio segments associated with video frames for playback of coordinated slow-motion video and audio according to some examples of the present disclosure.

[0020] Figure 3 is a block diagram of certain illustrative aspects of components and operations that may be included in a system configured to coordinate slow-motion video and audio according to some examples of the present disclosure.

[0021] Figure 4is a block diagram of another illustrative aspect of components that may be included in a system configured to coordinate slow-motion video and audio according to some examples of the present disclosure.

[0022] Figure 5 is an illustration of another illustrative aspects of components and operations that may be included in a system configured to coordinate slow-motion video and audio according to some examples of the present disclosure.

[0023] Figure 6 is an illustration of a particular implementation of a method of coordinating slow-motion video and audio according to some examples of the present disclosure.

[0024] Figure 7 An example of an integrated circuit including a slow motion video with audio engine according to some examples of the present disclosure is illustrated.

[0025] Figure 8 is an illustration of a mobile device including a slow motion video and audio engine according to some examples of the present disclosure.

[0026] Fig. 9 is an illustration of a head mounted device including a slow motion video and audio engine according to some examples of the present disclosure.

[0027] Fig.10 is an illustration of a wearable electronic device including a slow-motion video and audio engine according to some examples of the present disclosure.

[0028] Fig.11 is an illustration of a sound-controlled speaker system including a slow-motion video and audio engine according to some examples of the present disclosure.

[0029] Fig.12 is an illustration of a camera including a slow motion video and audio engine according to some examples of the present disclosure.

[0030] Fig.13 is an illustration of an extended reality head-mounted device (such as a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device) including a slow-motion video and audio engine according to some examples of the present disclosure.

[0031] Fig.14 is an illustration of a mixed reality or augmented reality eyewear device including a slow motion video and audio engine according to some examples of the present disclosure.

[0032] Fig.15 is an illustration of a first example of a vehicle including a slow motion video and audio engine according to some examples of the present disclosure.

[0033] Fig.16is an illustration of a second example of a vehicle including a slow motion video and audio engine according to some examples of the present disclosure.

[0034] Fig.17 is an illustration of another particular implementation of a method of coordinating slow-motion video and audio according to some examples of the present disclosure.

[0035] Fig.18 is an illustration of another particular implementation of a method of coordinating slow-motion video and audio according to some examples of the present disclosure.

[0036] Fig.19 is an illustration of another particular implementation of a method of coordinating slow-motion video and audio according to some examples of the present disclosure.

[0037] Fig. 20 is an illustration of another particular implementation of a method of coordinating slow-motion video and audio according to some examples of the present disclosure.

[0038] Fig.21 is a block diagram of a specific illustrative example of a device operable to coordinate slow motion video and audio according to some examples of the present disclosure.

[0039] VI. Specific Implementation Methods

[0040] Slow motion recording is a widely used feature of many electronic devices (such as smart phones). However, most slow motion recordings are hampered by the absence of audio or by misaligned, poor audio synchronization. For example, reducing the playback speed of an audio stream recorded with a high frame rate video often results in misalignment or poor synchronization of the audio with the video during playback. In addition, reducing the playback speed of the audio for the slow motion video may cause the audio to be distorted to the extent that the viewer cannot understand the audio (such as when the voice is slowed down to an incomprehensible tremor).

[0041] Systems and techniques are described herein that provide solutions for coordinating slow motion video with audio. For example, these systems and techniques can process audio recorded with high frame rate (HFR) video to distinguish between perceptible audio portions and imperceptible audio portions. As used herein, audio that is intelligible to a viewer when played back in slow motion is referred to as "perceptible," and audio that is intelligible to a viewer when played back in slow motion is referred to as "imperceptible." In this context, the terms "perceptible" and "imperceptible" indicate whether the audio is predicted to be intelligible to a viewer when played at a reduced speed and accompanied by playback of the slow motion video, and do not indicate whether the audio is audible to a viewer (e.g., with a volume loud enough and an appropriate frequency range for human hearing); specifically, audio can be both audible and "imperceptible" at the same time.

[0042] While speech may be imperceptible, other sounds may be perceptible when played back in slow motion, such as sounds associated with actions or events in the slow motion video, such as the sound of a ball bouncing or a door closing in the video. The perceptible portions (such as audio) may be played back at a reduced speed corresponding to the slow motion video playback, while the imperceptible portions may be played back at regular speed as separate audio segments that are temporally aligned with corresponding video frames for each of the imperceptible sounds. Playing back the imperceptible audio (e.g., speech) at regular speed during slow motion video playback prevents the loss of intelligibility that would occur if the imperceptible audio were played back at a reduced speed.

[0043] According to some aspects, a neural network, such as a recurrent neural network (RNN), is configured (e.g., trained) to filter (e.g., distinguish) perceptible audio substreams and imperceptible audio substreams from an audio stream recorded when recording an HFR video. The neural network can be configured as a single-input multiple-output network, where the input is the recorded audio stream, and the output is the perceptible audio substream and the imperceptible audio substream. According to some aspects, if the audio is not relevant to an associated scene in the HFR video, an "ignore" parameter (e.g., which can be trained using the neural network, can be input from a user, etc.) can be used to ignore the audio.

[0044] Thus, the disclosed systems and techniques provide a solution for providing audio accompanying slow motion videos in a manner that enhances the user experience. This solution reduces or eliminates synchronization drift and sound perception issues typically associated with conventional techniques. Thus, the disclosed systems and techniques can be used to generate slow motion videos with high quality audio (low synchronization drift) and suitable for users (high perception). Other advantages and examples of applications in which the disclosed techniques can be used are described in more detail below with reference to the accompanying drawings.

[0045] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In this specification, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some implementations and plural in other implementations. For illustration, Figure 1 Depicts a system including one or more processors ( Figure 1 116 ), indicating that in some implementations, the device 102 includes a single processor 116 , while in other implementations, the device 102 includes multiple processors 116 .

[0046] It can be further understood that the term "including" can be used interchangeably with "comprising". In addition, it should be understood that the term "wherein" can be used interchangeably with "wherein". As used herein, "exemplary" can indicate examples, specific implementations and / or aspects, and should not be understood as restrictive or indicating preferences or preferred specific implementations. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) used to modify elements (such as, structures, components, operations, etc.) do not themselves indicate any priority or order of the element relative to another element, but only distinguish the element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements in a specific element, and the term "plurality" refers to multiple (e.g., two or more) specific elements.

[0047] As used herein, "coupling" may include "communicatively coupled", "electrically coupled" or "physically coupled", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. Two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some specific implementations, two devices (or components) that are communicatively coupled (such as electrically connected) may directly or indirectly transmit and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled or physically coupled) without intermediate components.

[0048] In the present disclosure, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that such terms should not be interpreted as limiting, and other technologies may be used to perform similar operations. In addition, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access", and "determine" may be used interchangeably. For example, "generating", "calculating", "estimating" or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating or determining the parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) that has already been generated (such as by another component or device).

[0049] refer to Figure 1, discloses certain illustrative aspects of a system 100 configured to coordinate slow motion video and audio. The system 100 includes a device 102, which includes a memory 112 and one or more processors 116. In some specific implementations, the device 102 corresponds to a computing device (such as a mobile phone, a laptop computer, a server, etc.), a head-mounted device or other device worn on the head, or a vehicle, as illustrative, non-limiting examples.

[0050] The memory 112 is configured to store data 132 corresponding to a media stream 130. In a particular example, the media stream 130 corresponds to HFR video and audio of a scene 160. The scene 160 includes a plurality of audio sources corresponding to visible objects, including a person speaking 162, a bouncing ball 164, a chirping bird 166, and a babbling brook 168. In some implementations, the media stream 130 is received from one or more sensors 104 that are optionally included in or coupled to the device 102. In other implementations, the media stream 130 is received from another device, such as via a wireless transmission from a remote device, as further described below.

[0051] As an illustrative, non-limiting example, data 132 includes video 134 recorded at a first frame rate 136, such as a frame rate associated with recording HFR video (e.g., 120 frames per second (FPS)). Data 132 also includes a plurality of audio segments, including one or more first audio segments 140 and one or more second audio segments 150. One or more first audio segments 140 are indicated as having a first playback type 142, and one or more second audio segments 150 are indicated as having a second playback type 152. As further explained below, the first playback type 142 is associated with imperceptible sounds that are to be played back at a normal speed, and the second playback type 152 is associated with perceptible sounds that may be played back at an adjusted (e.g., reduced) speed.

[0052] One or more processors 116 are coupled to the memory 112 and include a media stream playback engine 120, and optionally a media stream data structure generator 170. According to some implementations, one or more of the components of the one or more processors 116 may be implemented using dedicated circuits. As a non-limiting example, one or more of the components of the one or more processors 116 may be implemented using a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. According to another implementation, one or more of the components of the one or more processors 116 may be implemented by executing instructions stored in the memory 112. For example, the memory 112 may be a non-transitory computer-readable medium storing instructions that can be executed by the one or more processors 116 to perform the operations described herein.

[0053] The media stream playback engine 120 includes a video playback unit 122 and an audio playback unit 124. The video playback unit 122 is configured to perform video playback to generate a video output 172. For example, the video playback unit 122 performs slow motion playback of the video 134 at a second frame rate 138 that is different from the first frame rate 136 to generate the video output 172. For illustration, the video 134 may correspond to an HFR video recorded at 120 FPS (the first frame rate 136) and played back as a slow motion video at 30 FPS (the second frame rate 138).

[0054] The audio playback unit 124 is configured to perform audio playback of the audio segments to generate an audio output 174. During audio playback, audio segments having a first playback type 142 are played at a first playback speed 144 (e.g., non-perceptible audio is played at the original speed of the audio recording), and audio segments having a second playback type 152 are played at a second playback speed 154 (e.g., perceptible audio is played at a reduced speed). According to some aspects, particular audio segments are associated with particular video frames, and the audio playback unit 124 is configured to control the playout time of the audio segments to synchronize the audio output 174 with the video output 172, as described with reference to FIG. Figure 2 Further described.

[0055] The one or more processors 116 are operably configured to initiate playback of the media stream 130, such as in response to a user command received via a touch screen or a voice interface, as illustrative, non-limiting examples. According to some aspects, initiating playback of the media stream 130 includes transmitting one or more control signals that cause the video playback unit 122 to perform playback of the video 134, and transmitting one or more control signals that cause the audio playback unit 124 to perform playback of the one or more first audio segments 140 and playback of the one or more second audio segments 150. During playback of the media stream 130, the video playback unit 122 plays out the video 134 at the second frame rate 138, thereby generating a video output 172 representing a slow motion video of the scene 160.

[0056] Playback of the media stream 130 also includes playing back the one or more first audio segments 140 at a first playback speed 144 (e.g., the speed of the audio when recorded) corresponding to the first frame rate 136. For example, the audio playback unit 124 processes the one or more first audio segments 140 to generate a first substream of imperceptible sounds (e.g., speech of a person 162, chirping of a bird 166), each of which is played out at the first playback speed 144 (e.g., at a regular speed) and is timed to be synchronized with an event in the video output 172 corresponding to a respective source of each of the imperceptible sounds.

[0057] In a particular example, the start of person 162 speaking in audio output 174 is synchronized with the slow-motion video playback such that the audible speech of person 162 begins substantially simultaneously with the video frame showing person 162 beginning to speak in slow-motion video output 172. Because the person's speech is played out at regular speed and the video of person 162 speaking is played out in slow motion, the speech will remain intelligible when played out, although the movement of the person's mouth producing the speech in the slow-motion video may appear to lose synchronization with the audible speech over time due to the difference in playback speeds of the speech and video.

[0058] Playback of the media stream 130 also includes playing back the one or more second audio segments 150 at a second playback speed 154 (e.g., a slower audio speed than when recorded) corresponding to the second frame rate 138. For example, the audio playback unit 124 processes the one or more second audio segments 150 to generate a second substream of perceptible sounds (e.g., the bouncing sound of the ball 164, the sound of the stream 168) that are played out at the second playback speed 154 (e.g., at a slower speed). As described in reference Figure 2 As further described, one or more second audio segments 150 may be linked to regularly spaced "anchor" video frames so that any synchronization drift that may have accumulated between the second substream of perceptible sound and the video playout is corrected when the next anchor video frame is played out.

[0059] The resulting first substream of imperceptible sound of the one or more first audio segments 140 and the second substream of perceptible sound of the one or more second audio segments 150 are combined to generate an audio output 174, such as reference 174. Figure 3 Further described.

[0060] In some implementations, device 102 includes or is coupled to an optional display device 106 configured to display a video output 172 to a viewer (e.g., a user of device 102). In some implementations, device 102 includes or is coupled to an optional one or more speakers 108 configured to play an audio output 174 to the viewer. In some implementations, display device 106 and one or more speakers 108 are included in a second device, such as a television or a mixed reality head-mounted device, and video output 172 and audio output 174 are sent to the second device for playback.

[0061] Although the examples described above are directed to audio playback that accompanies slow motion playback of an HFR video such that the second frame rate 138 is slower than the first frame rate 136 and the second playback speed 154 is slower than the first playback speed 144, in other implementations, the media stream playback engine 120 may be used for other media playback applications. As an illustrative example, the video 134 may be played back at an increased speed, one or more first audio segments 140 may correspond to audio that is not perceptible when played at the increased speed, and one or more second audio segments 150 may correspond to audio that is perceptible at the increased speed. In this illustrative example, the second frame rate 138 is faster than the first frame rate 136, and the second playback speed 154 is faster than the first playback speed 144.

[0062] The optional media stream data structure generator 170 is configured to process the media stream 130 to generate data 132 including one or more first audio segments 140 indicated as a first playback type 142 and one or more second audio segments 150 indicated as a second playback type 152, as shown in reference Figures 4 to 6 The media stream data structure generator 170 thus enables the device 102 to process its own recordings (eg, media streams 130 from one or more sensors 104) to coordinate slow motion video with audio for playback. Alternatively or in addition, the device 102 may receive data 132 from another device, such as via a modem 190.

[0063] As indicated above, device 102 optionally includes a modem 190 coupled to one or more processors 116. Modem 190 is configured to enable wireless communication with another device, such as to receive media stream 130 from a second device or to transmit video output 172, audio output 174, or both to a second device. In some examples, device 102 may use a low-power protocol (e.g., Bluetooth ® In some examples, the wireless connection corresponds to sending and receiving signals according to an IEEE 802.11-type (e.g., WiFi) wireless local area network or one or more other wireless radio frequency (RF) communication protocols.

[0064] The functionality described with respect to device 102 may be implemented in a variety of different types of devices, examples of which are shown in FIG. Figures 7 to 16 For illustration, in some examples, one or more processors 116 are integrated into at least one of a mobile phone, a tablet device, a wearable electronic device, or a camera device, such as the reference Figure 8 , Fig.10 and Fig.12In some examples, one or more processors 116 are integrated into a virtual reality head mounted device, a mixed reality head mounted device, or an augmented reality head mounted device, such as the reference Fig.13 In some examples, one or more processors 116 are integrated into a vehicle, such as a vehicle described in reference Fig.15 and Fig.16 As described.

[0065] By playing out the audio of the one or more first audio segments 140 having the first playback type 142 using the first playback speed 144, sound perception issues caused by adjusting the playback speed of the audio can be avoided. By playing the one or more second audio segments 150 as a segment sequence associated with an anchor frame sequence in the video 134, synchronization drift associated with playing out the audio using a second playback speed 154 (e.g., slower than the original speed) can be reduced or eliminated. Thus, the disclosed systems and techniques can be used to provide speed-adjusted video playback with enhanced sound quality.

[0066] Figure 2 A graphical representation 200 of a plurality of audio segments associated with a video frame that may be used by a device 102 according to some implementations is depicted. The plurality of audio segments includes one or more first audio segments 140 and one or more second audio segments 150 associated with a video frame 210. Each audio segment in the plurality of audio segments is associated with a respective particular frame of the video representing a playback start point for the audio segment, as further explained below.

[0067] Video frames 210 correspond to video 134 and are illustrated as being arranged in a playback sequence, starting with an initial video frame (F1) that is played first in the playback sequence. Figure 2 The horizontal axis of 210 represents a time axis, wherein playback of an initial video frame (F1) occurs at a first time (e.g., t1), and playback of each successive video frame occurs at regular time intervals after F1 (e.g., frame 2 is played after frame 1, frame 3 is played after frame 2, and so on). It should be understood that for purposes of illustration, fewer video frames than indicated by the index of the video frame are depicted. For example, nine video frames between F1 and frame 31 (F31) are illustrated, nine video frames between F31 and frame 61 (F61) are illustrated, and so on. Therefore, the particular depiction of video frames 210 should be considered illustrative rather than limiting.

[0068] The one or more first audio segments 140 are indicated as having a speed preservation (SP) playback type 220 corresponding to the first playback type 142. The speed preservation playback type 220 indicates that the playback speed of the audio of the one or more first audio segments 140 is to be preserved and not adjusted to match the speed of the video playback.

[0069] The one or more first audio segments 140 include a first speed keeping segment (SP1) 241 and a second speed keeping segment (SP2) 242. Each of the one or more first audio segments 140 is associated with a corresponding original frame 260 in the video frames 210, which corresponds to an associated audio generation event in the video 134.

[0070] As a first example, SP1 corresponds to the chirping of the bird 166 that begins at frame 37 (F37) of the video frame 210. Therefore, F37 is the original frame 260 of SP1, and the audio playout of SP1 should begin substantially simultaneously with the video playout of F37. The association of SP1 with F37 (graphically depicted as an arrow) may be achieved via one or more pointers, lists, tables, etc., such as an index to F37 stored with SP1 (e.g., as metadata for SP1), an index to SP1 stored with F37 (e.g., as metadata for F37), or a linked table or linked list stored separately from the video frame 210 and separately from the one or more second audio segments 150, as illustrative, non-limiting examples.

[0071] As a second example, SP2 corresponds to speech by person 162 beginning at frame 104 (F104) of video frames 210. Thus, F104 is the original frame 260 of SP2, and the audio playout of SP2 should begin substantially simultaneously with the video playout of F104. The association of SP2 with F104 (graphically depicted as an arrow) can be accomplished in a manner similar to that described above.

[0072] The length of SP1 indicates the duration for playing out SP1, and the length of SP2 indicates the duration for playing out SP2. For example, the tweet represented by SP1 is shorter than the speech of person 162 represented by SP2. As illustrated, during playback, SP1 and SP2 do not overlap in time (e.g., playback of SP1 ends before playback of SP2 begins). However, in other implementations, two or more of the second audio segments 150 may overlap, such as described below with reference to Figure 5 Depicted.

[0073] The one or more second audio segments 150 are indicated as having an adjustable speed (SA) playback type 230 corresponding to the second playback type 152. The adjustable speed playback type 230 indicates that the playback speed of the audio of the one or more second audio segments 150 is to be adjusted to match the speed of the video playback.

[0074] The one or more first audio segments 140 include a series of adjustable speed segments, including a first segment (SA1) 251, a second segment (SA2) 252, a third segment (SA3) 253, a fourth segment (SA4) 254, a fifth segment (SA5) 255, and a sixth segment (SA6) 256. The one or more second audio segments 150 have substantially equal lengths and are associated with regularly spaced anchor frames 270 in the video 134. As illustrated, F1 is an anchor frame of SA1, frame 31 (F31) is an anchor frame of SA2, frame 61 (F61) is an anchor frame of SA3, frame 91 (F91) is an anchor frame of SA4, frame 121 (F121) is an anchor frame of SA5, and frame 151 (F151) is an anchor frame of SA6. Because F1 is the anchor frame of SA1, the audio broadcast of SA1 should start substantially simultaneously with the video broadcast of F1; because F31 is the anchor frame of SA2, the audio broadcast of SA2 should start substantially simultaneously with the video broadcast of F31 (and the audio broadcast of SA1 should end), and so on.

[0075] As reference Figure 4 and Figure 5 As further described, the audio segments SA1-SA6 may represent a continuous audio sound (e.g., the sound of the creek 168) that spans the time period from F1 to F180 during recording and is divided into segments of equal length and associated with corresponding anchor frames 270 during generation of the data 132. Starting playback of each of the segments SA1-SA6 at the corresponding anchor frame 270 enables reduction or elimination of user-perceivable synchronization drift by resynchronizing the second audio segment 150 with the video frame 210 at each anchor frame 270.

[0076] Figure 3 An illustrative example 300 of operations and components that may be implemented in the audio playback unit 124 is depicted. As illustrated, the one or more first audio segments 140 and the one or more second audio segments 150 may be provided to a stream generator 310.

[0077] The stream generator 310 is configured to process the one or more first audio segments 140 to generate a first audio stream 340 corresponding to a speed-preserving (e.g., imperceptible) sound played back at a regular speed. The stream generator 310 is also configured to process the one or more second audio segments 150 to generate a second audio stream 350 corresponding to a speed-adjustable (e.g., perceptible) sound played back at an adjusted speed (e.g., a slower speed for slow-motion video playback). In some implementations, the stream generator 310 is configured to process the one or more first audio segments 140 and the one or more second audio segments 150 in parallel, such that generation of the first audio stream 340 and the second audio stream 350 occurs substantially simultaneously.

[0078] Combiner 320 is configured to process first audio stream 340 and second audio stream 350 and generate output audio stream 360 corresponding to a superposition of first audio stream 340 and second audio stream 350. In certain implementations, second audio stream 360 corresponds to audio output 174.

[0079] Thus, playback of the media stream 130 may include generating a first audio stream 340 including one or more first audio segments 140 , a second audio stream 350 including one or more second audio segments 150 , and an output audio stream 360 corresponding to a superposition of the first audio stream 340 and the second audio stream 350 .

[0080] Figure 4 Describes what can be included in Figure 1 Component 400 includes one or more cameras 402 configured to generate video data 406 corresponding to a recorded video stream 414, one or more microphones 404 configured to generate audio data 408 corresponding to a recorded audio stream 416, and media stream data structure generator 170. In a specific example, one or more cameras 402, one or more microphones 404, or both are included in Figure 1 One or more sensors 104.

[0081] The media stream data structure generator 170 includes a recorder 410, a neural network 420, and an audio to video frame linker 430. The recorder 410 is configured to record the media stream 130 (e.g., video data 406 and audio data 408) to generate a recorded media stream 412, which includes a recorded video stream 414 and a recorded audio stream 416 corresponding to the video 134.

[0082] The media stream data structure generator 170 is configured to process the recorded audio stream 416 using the neural network 420 to generate a plurality of audio streams 421. The plurality of audio streams 421 includes a first audio stream 422 that includes audio classified as having a first playback type 142, such as a speed-preserving (e.g., non-perceptible) sound. The plurality of audio streams 421 also includes a second audio stream 424 that includes audio classified as having a second playback type 152, such as a speed-adjustable (e.g., perceptible) sound.

[0083] The audio to video frame linker 430 is configured to generate a playback structure 432 for the media stream 130 to facilitate playback of the recorded video stream 414 at the second frame rate 138, playback of the first audio stream 422 at the first playback speed 144, and playback of the second audio stream 424 at the second playback speed 154. For example, the playback structure 432 may include Figure 1The data 132 has Figure 2 One or more of the linking mechanisms (e.g., pointers, tables, lists, metadata, etc.) described in for associating audio segments with corresponding original frames and anchor frames.

[0084] In some implementations, the audio-to-video frame linker 430 is configured to process the first audio stream 422 differently than the second audio stream 424 when generating the playback structure 432. For example, the audio-to-video frame linker 430 is configured to process the recorded video stream 414 to identify specific segments in the recorded video stream 414 that are consistent with the first audio stream 422 (e.g., Figure 2 SP1) associated with the original frame (for example, Figure 2 F37 of the recorded audio stream 416). According to some aspects, the audio to video frame linker 430 is configured to identify the original frame based on the time alignment between the occurrence of the sound in the recorded audio stream 416 and the particular frame of the recorded video stream 414. According to some aspects, the audio to video frame linker 430 is configured to identify one or more objects (e.g., bird 166) in the recorded video stream 414 from which the sound associated with the beginning of the particular segment originates. The audio to video frame linker 430 may associate the particular segment of the first audio stream 422 with the original frame in the playback structure 432 so that during the playback of the media stream 130, the initiation of the playback of the particular segment of the first audio stream 422 is substantially simultaneous with the initiation of the playback of the original frame (e.g., the chirping begins to play out at regular speed when the slow motion playback of the video 134 reaches the original frame F37).

[0085] The audio-to-video frame linker 430 may also assign the first playback speed 144 to a particular segment of the first audio stream 422. For example, the audio-to-video frame linker 430 may set one or more values ​​in the playback structure 432 to assign the first playback speed 144 to the audio segment of the first audio stream 422. Alternatively or in addition, the audio-to-video frame linker 430 may designate the audio segment of the first audio stream 422 as having the first playback type 142 or as having the speed-preserving playback type 220, which may indicate that playback is to be performed using the first playback speed 144.

[0086] In some implementations, when processing the second audio stream 424 to generate the playback structure 432, the audio to video frame linker 430 is configured to identify an anchor frame in the recorded video stream 414 based on the length of the recorded video stream 414 and the anchor frequency 440. Figure 2In the example of , the anchor frequency 440 of the anchor frame 270 is 30, and every 30th frame of the video frame 210 can be identified as an anchor frame 270 over the duration of the recorded video stream. In some examples, the anchor frequency 440 is set to a value based on an expected, modeled, or observed rate of audio synchronization drift to prevent audio synchronization drift from accumulating to a perceptible amount over time (or to reduce the likelihood of such an occurrence). In other examples, the anchor frequency 440 is dynamic. To illustrate, one or more processors 116 (e.g., the media stream playback engine 120) can compare the amount of audio synchronization drift detected during playback to a drift threshold, and can reduce the anchor frequency 440 based on the detected drift exceeding the drift threshold.

[0087] The audio to video frame linker 430 may split the second audio stream 424 into a specific number of sequential segments having matching durations, wherein the specific number of sequential segments is based on the number of identified anchor frames. Figure 2 As an example, upon determining that six anchor frames 270 are identified, the portion of the second audio stream 424 spanning video frames 1-180 is split into six segments SA1-SA6 having matching durations.

[0088] The audio-to-video frame linker 430 may associate a first segment of the sequential segments (e.g., SA1) with a first identified anchor frame (e.g., F1) in the playback structure 432 such that during playback of the media stream 130, initiation of playback of the first segment of the sequential segments is substantially simultaneous with initiation of playback of the first identified anchor frame, such as Figure 2 The audio-to-video frame linker 430 may also associate a second segment (e.g., SA2) in the sequential segments with a second identified anchor frame (e.g., F31) in the playback structure 432, such that during playback of the media stream 130, initiation of playback of the second segment is substantially simultaneous with initiation of playback of the second identified anchor frame. Each of the remaining segments in the sequential segments may also be associated with a corresponding identified anchor frame in the playback structure 432 in a similar manner.

[0089] During operation, when the video 134 is played as a slow motion video, the media stream data structure generator 170 specifies the second audio stream 424 (e.g., a perceptible audio stream) for playback at a reduced speed. As discussed above, one of the disadvantages of playing back audio at a reduced speed is that the audio bytes tend to drift relative to the video. This drift occurs because the slow motion video is not a simple slow-paced playback, but a high frame rate recording played back at a different rate. In the present technology, such drift is corrected by anchoring the segments of the perceptible stream to corresponding anchor video frames. For illustration, the perceptible audio stream can be stored in the playback structure 432 as a series of several audio substreams rather than a single audio stream. Each substream in the substream starts at a corresponding anchor frame of the video (e.g., anchored to the corresponding anchor frame). For example, the substreams can be created so that each substream starts to play back 30 video frames (e.g., an anchor frequency of 30) after the previous substream starts to play back, so that any audio drift is corrected every 30 video frames. The anchor frequency may be set to an appropriate amount, such as being dynamically set or being trainable, so that the audio drift is not perceptible to the user.

[0090] In a specific example, a 120FPS video recording starts at timestamp 00:00 and ends at timestamp 00:01, i.e., a 1 second long video. The length of the full audio stream is 1 second, and the number of recorded frames is 120. Assuming the anchor frequency is 30, the number of anchor frames can be determined as 120 / 30, indicating that there are 4 anchor frames. Therefore, the audio is divided into 4 audio substreams, each of which is 0.25 seconds. Each audio substream is attached to its corresponding anchor frame (e.g., at timestamps 00:00, 00:25, 00:50, and 00:75, or at frame numbers 0, 30, 60, and 90). In slow motion playback, each audio substream acts as a simple stream with a reduced playback speed from one anchor frame to the next, thereby preventing synchronization drift from accumulating to a user-perceivable level.

[0091] The media stream data structure generator 170 also segments the first audio stream 422 (eg, the imperceptible stream) into sub-streams, and each sub-stream is attached to its corresponding original frame and played back at full speed. Figure 5 An example of the operation of the media stream data structure generator 170 is further illustrated.

[0092] In a particular implementation, ignore parameter 426 is used to specify that one or more specific sounds of recorded audio stream 416 are to be ignored and not included in any of the plurality of audio streams 421. For example, when a sound is detected in recorded audio stream 416 that has no corresponding source in recorded video stream 414 (e.g., an invisible airplane passing overhead, or the voice of a person passing nearby who is not in scene 160 or interacting with anything in scene 160), the sound may be selected to be ignored by neural network 420 and omitted during playback. Sounds may be ignored based on whether the sound has a certain amount of connection to the scene. For example, if something in scene 160 (e.g., person 162 or bird 166) reacts to the sound of a second bird that is not in scene 160, the sound of the second bird may be included in first audio stream 422; otherwise, the sound of the second bird may be ignored.

[0093] Ignore parameter 426 can be set as a trainable parameter. For example, a separate neural network (such as an RNN structure) can be trained to receive the recorded video stream 414 and the recorded audio stream 416 as inputs and generate an output indicating whether the audio signal has a connection with the scene. Alternatively or in addition, ignore parameter 426 can be a manual parameter by which the user can decide which audio should be kept in the final video. In other specific implementations, a fixed method can be used to ignore all sounds that do not have a source in the scene based on the basic expectation of an enhanced user experience.

[0094] Figure 5 An example 500 of components and operations that may be performed on a device 102, such as a media stream data structure generator 170, is depicted. Figure 5 , scene 160 includes chirping bird 166, talking person 162, bouncing ball 164, and babbling brook 168 captured in image 560 (e.g., frame of video data 406) by one or more cameras 402 during HFR recording. Video data 406 and audio data 408 associated with the HFR recording are provided to one or more processors 116.

[0095] The audio data 408 is processed at an audio feature extractor 502 (e.g., which includes a fast Fourier transform (FFT) unit 504) to extract audio features 506. The extracted audio features 506 are input to a neural network 420 including an RNN. The neural network 420 generates a second audio stream 424 as a "perceptible stream" determined by the neural network 420 as audio that is perceptible when played back at a reduced speed, and generates the first audio stream 422 as an "imperceptible stream" determined as audio that is not perceptible when played back at a reduced speed.

[0096] The audio data 408 is graphically illustrated as a signal 520 that is a combination of four signals: a signal 530 corresponding to the chirping of the bird 166, a signal 532 corresponding to the speaking of the person 162, a signal 534 corresponding to the sound of the stream 168, and a signal 536 corresponding to the bouncing of the ball 164. As illustrated, the audio corresponding to the stream 168 and the ball 164 is output to the second audio stream 424 (the perceptible stream), and the audio corresponding to the bird 166 and the person 162 is output to the first audio stream 422 (the imperceptible stream).

[0097] The second audio stream 424 is illustrated as signal 544 comprising a combination of signal 534 and signal 536, with signal 544 illustrated as having its time stamp extended to match the slower playback speed / increased playout time associated with the slow motion playout of video data 406. Signal 544 is split into sequential segments (e.g., Figure 2 SA1-SA6).

[0098] The first audio stream 422 is divided into a plurality of segments associated with original frames in the video data 406 and played out at a first playback speed 144. For example, signal 542 represents the audio of eight chirps of a bird. Signal 542 is segmented into eight substreams, which are graphically illustrated via rectangular boxes representing the start and end points of each segment (such as representative segment 552). Each of the eight substreams is "attached" (e.g., linked via a pointer, frame number, timestamp, etc.) to its corresponding original frame in the video, i.e., the video frame at which each particular chirp begins. Similarly, signal 540 illustrates that the audio of person 162 speaking is assigned to a single segment 550. Segment 550 is attached to the original frame in the video that coincides with the time at which person 162 begins speaking.

[0099] Combining the systems and techniques described above, the solution framework for processing audio data can be summarized in the following steps:

[0100] Receive Input: Receives HFR video and accompanying audio.

[0101] Perform audio stream filtering: filter the audio segments into perceptible audio streams and non-perceptible audio streams.

[0102] Generate audio playback structure: Attach all segments of the non-perceptible audio stream to their original frames (video frames at the beginning of each audio segment). On the remaining (perceptible) segments, apply anchor frame attachments (e.g., attach the perceptible substream to the anchor frame at the chosen anchor frequency).

[0103] Start playback: For each audio substream, i.e., from one anchor frame to another, the perceptible audio stream is played back at the playback speed ratio (high frame rate) / (playback frame rate). Each non-perceptible audio segment maintains its original playback speed (as experienced during recording) and is played back during video playback starting from the original frame of that audio segment.

[0104] Figure 6 An example of a method 600 including operations for performing coordination of slow motion video and audio is illustrated. The method 600 includes, at 602, starting recording. The method 600 includes, at 604, recording high frame rate video and audio, such as using one or more cameras 402 and one or more microphones 404.

[0105] The method 600 includes performing an audio recognition operation at 606. For example, the audio recognition operation may be performed by the neural network 420 to generate an imperceptible audio stream (eg, the first audio stream 422) and a perceptible audio stream (eg, the second audio stream 424).

[0106] The method 600 includes, at 608 , adding the imperceptible audio stream to the original frame and maintaining the original playback speed (which is not linked to the playback speed of the video), such as described with reference to processing the first audio stream 422 at the audio to video frame linker 430 .

[0107] The method 600 also includes, at 610, splitting the perceptible audio stream into substreams and attaching the start of each substream to its anchor frame; and at 612, linking the playback of each substream of the perceptible audio stream to the video playback speed, such as described with reference to processing the second audio stream 424 at the audio to video frame linker 430.

[0108] The method 600 includes, at 614, collating the captured video frames with the attached audio streams and associated speeds. For example, the collated video frames, audio frames, and associated speeds can be included in a playback structure 432. The method 600 includes, at 616, storing the video. For example, the playback structure 432 can be stored as data 132.

[0109] Figure 7 A specific implementation 700 of the device 102 is depicted as an integrated circuit 702 that includes a slow motion video and audio engine 790. The slow motion video and audio engine 790 includes the media stream playback engine 120, the media stream data structure generator 170, or both.

[0110] Integrated circuit 702 also includes an input section 704 (such as one or more bus interfaces) to enable receiving media data 710 for processing. For example, media data 710 may include media stream 130, data 132, or both. Integrated circuit 702 also includes a signal output section 706 (such as a bus interface) to enable transmitting video output 172 and audio output 174.

[0111] Integrated circuit 702 can be used as a component in a system to implement slow motion video and audio coordination, the system including one or more other components, such as Figure 8 The depicted mobile phone or tablet device, such as Fig. 9 The head mounted device depicted, such as Fig.10 The wearable electronic devices depicted, such as Fig.11 The described sound control speaker system, such as Fig.12 The camera depicted, Fig.13 The virtual reality head-mounted device, mixed reality head-mounted device or augmented reality head-mounted device depicted, such as Fig.14 Augmented reality glasses or mixed reality glasses as depicted, or Fig.15 or Fig.16 The vehicle depicted.

[0112] Figure 8 A specific implementation 800 is depicted in which the device 102 is a mobile device 802 (such as a phone or tablet device, as illustrative, non-limiting examples above). The mobile device 802 includes one or more cameras 402, one or more microphones 404, a display device 106 (e.g., a touch screen), and one or more speakers 108. The slow motion video and audio engine 790 is integrated into the mobile device 802 and is illustrated using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 802. In a specific example, the slow motion video and audio engine 790 can be used to generate a video output 172 and an audio output 174, which can be output to a viewer via the display device 106 and the one or more speakers 108, respectively.

[0113] Fig. 9 A specific implementation 900 is depicted in which the device 102 is a head mounted device 902. The head mounted device 902 includes one or more speakers 108, one or more cameras 402, and one or more microphones 404. A slow motion video and audio engine 790 is integrated into the head mounted device 902. In a specific example, the one or more cameras 402 and the one or more microphones 404 generate a media stream 130. The slow motion video and audio engine 790 can be used to generate an audio output 174 that can be played at the one or more speakers 108, and a video output 172 that the head mounted device 902 can send to a second device (not shown) for display.

[0114] Fig.10 A specific implementation 1000 is depicted in which the device 102 is a wearable electronic device 1002 (illustrated as a “smart watch”). The slow motion video and audio engine 790, the display device 106, and the one or more speakers 108 are integrated into the wearable electronic device 1002. In a specific example, the display device 106 is configured to display the video output 172, and the one or more speakers 108 are configured to play the audio output 174 generated by the slow motion video and audio engine 790.

[0115] Fig.11 1100 is a specific implementation in which the device 102 is a wireless speaker and voice activated device 1102. The wireless speaker and voice activated device 1102 may have wireless network connectivity and be configured to perform auxiliary operations. The slow motion video and audio engine 790, one or more speakers 108, one or more cameras 402, and one or more microphones 404 are included in the wireless speaker and voice activated device 1102. During operation, the wireless speaker and voice activated device 1102 may monitor the audio captured by the one or more microphones 404 and determine whether the user has spoken a keyword. In response to determining that a keyword has been spoken, the wireless speaker and voice activated device 1102 may perform auxiliary operations, such as via executing an integrated auxiliary application. These auxiliary operations may include initiating playback of a slow motion video, such as activating the slow motion video and audio engine 790 to generate an audio output 174 that can be broadcast at one or more speakers 108, and a video output 172 that the wireless speaker and voice activated device 1102 can send to a second device (not shown) for display. These auxiliary operations may also include adjusting the temperature, playing music, turning on lights, etc. For example, these auxiliary operations are performed in response to receiving a command after a key word or key phrase (eg, "Hello, assistant").

[0116] Fig.12 A specific implementation 1200 is depicted in which the device 102 is a portable electronic device corresponding to a camera device 1202. In addition to one or more image capture devices (e.g., a video capture unit), a slow motion video and audio engine 790 and one or more microphones 404 are included in the camera device 1202. During operation, the slow motion video and audio engine 790 can be used to process captured video data and audio data to generate data 132 that the camera device 1202 can send to a second device (not shown) for playback.

[0117] Fig.13An implementation 1300 is depicted in which the device 102 includes a portable electronic device corresponding to an extended reality ("XR") head-mounted device 1302, such as a virtual reality ("VR") head-mounted device, an augmented reality ("AR") head-mounted device, or a mixed reality ("MR") head-mounted device. A slow-motion video and audio engine 790, a display device 106 (e.g., a visual interface device positioned in front of the user's eyes), one or more speakers 108, or a combination thereof are integrated into the XR head-mounted device 1302. The slow-motion video and audio engine 790 may be used to generate a video output 172 and an audio output 174, the display device 106 being configured to display the video output 172, and the one or more speakers 108 being configured to play the audio output 174, so that a user of the XR head-mounted device 1302 can experience playback of slow-motion video and audio.

[0118] Fig.14 An implementation 1400 is depicted in which the device 102 includes a portable electronic device corresponding to augmented reality or mixed reality glasses 1402. The glasses 1402 include a holographic projection unit 1404 configured to project visual data onto a surface of a lens 1406 or reflect the visual data from the surface of the lens 1406 onto the retina of the wearer. A slow motion video and audio engine 790, one or more speakers 108, one or more cameras 402, one or more microphones 404, or a combination thereof are integrated into the glasses 1402. The slow motion video and audio engine 790 can be used to process audio data and video data received via one or more microphones 404 and one or more cameras 402, respectively, to generate data 132 (e.g., playback structure 432), processing the data 132 to generate coordinated slow motion video and audio playback via the holographic projection unit 1404 and the one or more microphones 404, or a combination thereof.

[0119] Fig.15 A specific implementation 1500 is depicted in which the disclosed technology is implemented in a vehicle 1502, which is illustrated as a manned or unmanned aerial device (e.g., a personal aircraft, a surveillance drone, etc.). Slow motion video and audio engine 790, one or more cameras 402, one or more microphones 404, or a combination thereof are integrated into vehicle 1502.

[0120] In some implementations where the vehicle 1502 is configured as a surveillance drone, one or more of the microphone 404 and the camera 402 may be pointed at a particular person or object being monitored to capture HFR video and accompanying audio. The slow motion video and audio engine 790 may process the captured HFR video and audio to generate data 132 (e.g., playback structure 432), which may be stored at the vehicle 1502 or sent to a second device for playback as coordinated slow motion video and audio.

[0121] Fig.16 Another specific implementation 1600 in which the disclosed technology is implemented in a vehicle 1602, illustrated as a car, is depicted. Slow motion video and audio engine 790, one or more microphones 404, one or more cameras 402, one or more speakers 108, display device 106, or a combination thereof are integrated into vehicle 1602.

[0122] One or more of the microphone 404 and the camera 402 may be pointed toward a user (e.g., an operator or passenger of the vehicle 1602) to capture audio and video of the user, pointed outward to capture audio and video of the surrounding environment, or a combination thereof. The captured audio and video (e.g., HFR video) may be processed by the slow motion video and audio engine 790 for playback as coordinated slow motion video and audio via the display device 106 and one or more speakers 108, sent to another device (e.g., a user's computing device, a remote server, etc.) for storage and later playback, or a combination thereof. In some implementations, HFR video recording with audio may be automatically initiated based on one or more motion sensors or other sensors of the vehicle 1602, such as when one or more motion sensors indicate that the vehicle 1602 is experiencing emergency braking, sudden acceleration that may indicate a collision, etc.

[0123] Fig.17 An example of a method 1700 of coordinating speed adjusted video and audio is depicted. The method 1700 may be performed by a device 102, such as a media stream playback engine 120, for example.

[0124] The method 1700 includes obtaining, at 1702, data corresponding to a media stream at one or more processors, the data including a video recorded at a first frame rate and a plurality of audio segments. The plurality of audio segments include one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. For example, the media stream playback engine 120 obtains the data 132 from the memory 112.

[0125] The method 1700 includes initiating playback of the media stream at the one or more processors at 1704. The playback of the media stream includes playing the video at a second frame rate different from the first frame rate, playing the one or more first audio segments at a first playback speed corresponding to the first frame rate, and playing the one or more second audio segments at a second playback speed corresponding to the second frame rate. For example, the audio playback unit 124 generates a video output 172 at the second frame rate 138 for display at the display device 106, and generates an audio output 174 based on the one or more first audio segments 140 being played at the first playback speed 144 and the one or more second audio segments 150 being played at the second playback speed 154.

[0126] In some implementations, each of the plurality of audio segments is associated with a corresponding specific frame of the video that represents a playback start point of the audio segment. For example, each of the one or more first audio segments 140 can be associated with a corresponding original frame in the original frames 260, and each of the one or more second audio segments 150 can be associated with a corresponding anchor frame in the anchor frames 270, such as Figure 2 According to some aspects, the one or more second audio segments have substantially equal lengths and are associated with regularly spaced anchor frames in the video, such as Figure 2 According to some aspects, each of the one or more first audio segments is associated with a corresponding original frame corresponding to an audio generation event in the video, such as a reference frame 270. Figure 2 Audio segments 241 and 242 and Figure 4 As described by the media stream data structure generator 170.

[0127] In some implementations, the second frame rate is slower than the first frame rate, and the second playback speed is slower than the first playback speed. For example, when the data 132 corresponds to HFR video and accompanying audio, the second frame rate 138 is slower than the first frame rate 136, and the second playback speed 154 for playback of audio indicated as having the second playback type 152 (e.g., perceptible audio) is slower than the first playback speed 144 for playback of audio indicated as having the first playback type 142 (e.g., non-perceptible audio).

[0128] By playing the audio of the one or more first audio segments having the first playback type using the first playback speed, sound perception issues caused by adjusting the playback speed of the audio can be avoided. By playing the one or more second audio segments as a segment sequence associated with an anchor frame sequence in the video, synchronization drift associated with playing out the audio using the second playback speed 154 (e.g., slower than the original speed) can be reduced or eliminated. Therefore, the method 1700 can be used to provide speed-adjusted video playback with enhanced sound quality.

[0129] Fig.18 An example of a method 1800 for coordinating speed-adjusted video and audio is depicted. For example, the method 1800 may be performed by a device 102 (such as a media stream data structure generator 170). In a specific implementation, the method 1800 is included in Fig.17 The following steps in method 1700 (e.g., performed as part thereof): obtain data.

[0130] Method 1800 includes, at 1802, recording a media stream to generate a recorded media stream, the recorded media stream including a recorded video stream and a recorded audio stream corresponding to the video. Figure 4 Recorder 410 records media stream 130 to generate media stream 412 including recorded video stream 414 and recorded audio stream 416 .

[0131] The method 1800 includes, at 1804, processing the recorded audio stream using a neural network to generate a plurality of audio streams, the plurality of audio streams including a first audio stream and a second audio stream, the first audio stream including audio classified as having a first playback type, the second audio stream including audio classified as having a second playback type. For example, the media stream data structure generator 170 processes the recorded audio stream 416 using the neural network 420 to generate the first audio stream 422 and the second audio stream 424.

[0132] Method 1800 includes generating a playback structure for the media stream to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed at 1806. For example, audio to video frame linker 430 generates playback structure 432 for media stream 130, such as Figure 4 Described in .

[0133] Fig.19 An example of a method 1900 for coordinating speed-adjusted video and audio is depicted. For example, the method 1900 may be performed by a device 102 (such as a media stream data structure generator 170). In a specific implementation, the method 1900 is included in Fig.18In the following steps of method 1800 (eg, performed as part thereof): generating a data structure.

[0134] The method 1900 includes, at 1902 , identifying, in a recorded video stream, original frames associated with a particular segment of a first audio stream, such as described for the audio to video frame linker 430 identifying original frames for sounds in the first audio stream 422 .

[0135] The method 1900 includes associating, at 1904, a specific segment of the first audio stream with an original frame in the playback structure such that during playback of the media stream, initiation of playback of the specific segment of the first audio stream and initiation of playback of the original frame are substantially simultaneous. For example, the audio to video frame linker 430 generates one or more links, tables, or other metadata to associate the segments of the one or more first audio segments 140 with the original frame, thereby achieving the following. Figure 2 Substantially simultaneous playback as described in.

[0136] The method 1900 also includes, at 1906 , assigning a first playback speed to a particular segment, such as described with reference to the audio to video frame linker 430 .

[0137] Fig. 20 An example of a method 2000 for coordinating speed-adjusted video and audio is depicted. For example, the method 2000 may be performed by a device 102 (such as a media stream data structure generator 170). In a specific implementation, the method 2000 is included in Fig.18 In the following steps of method 1800: using a neural network to process the recorded audio stream.

[0138] Method 2000 includes identifying a first sound that is imperceptible to a user at a reduced playback speed, at 2002, and generating a first audio stream based on the first sound, at 2004. For example, neural network 420 processes recorded audio stream 416 to identify imperceptible sounds (e.g., speech of person 162 and chirping of bird 166), and outputs first audio stream 422 based on (e.g., including) the imperceptible sounds.

[0139] Method 2000 includes identifying a second sound perceptible to a user at a reduced playback speed, at 2006, and generating a second audio stream based on the second sound, at 2008. For example, neural network 420 processes recorded audio stream 416 to identify perceptible sounds (e.g., the sound of ball 164 bouncing and the sound of stream 168), and outputs a second audio stream 424 based on (e.g., including) the perceptible sounds.

[0140] Method 2000 includes: at 2012, identifying a third sound that is not perceptible to a user at a reduced playback speed; and at 2014, determining that the third sound is to be omitted during playback based on an omission parameter, such as reference Figure 4 The ignored parameters are described in 426.

[0141] Method 2000 includes bypassing generating an audio stream based on the third sound in response to determining that the third sound is to be omitted, at 2016. For example, based on ignore parameter 426 indicating that a particular sound is to be ignored, neural network 420 does not output the sound to any of multiple audio streams 421, thereby preventing the sound from being included in playback structure 432 for later playback.

[0142] Fig.17 , Fig.18 , Fig.19 or Fig. 20 The methods of any or all of the figures in the drawings may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processing unit (DSP), a controller, another hardware device, a firmware device, or any combination thereof. For example, Figures 17 to 20 Any one or more of the methods may be performed by a processor executing instructions, such as reference Fig.21 as described.

[0143] refer to Fig.21 , depicts a block diagram of a particular exemplary implementation of a device and generally designated as 2100. In various implementations, the device 2100 may have more Fig.21 More or fewer components may be illustrated. In an exemplary implementation, device 2100 may correspond to device 102. In an exemplary implementation, device 2100 may execute the reference Figures 1 to 20 One or more operations described.

[0144] In a particular implementation, the device 2100 includes a processor 2106 (e.g., a CPU). The device 2100 may include one or more additional processors 2110 (e.g., one or more DSPs). In certain aspects, the processor 116 corresponds to the processor 2106, the processor 2110, or a combination thereof. The processor 2110 may include a speech and music coder-decoder (codec) 2108, which includes a speech decoder ("vocoder") encoder 2136, a vocoder decoder 2138, a slow motion video and audio engine 790, or a combination thereof.

[0145] The device 2100 may include a memory 2186 and a codec 2134. The memory 2186 may include instructions 2156 that are executable by one or more additional processors 2110 (or processor 2106) to implement the functionality described with reference to the slow motion video and audio engine 790. In certain aspects, the memory 2186 corresponds to the memory 112. The device 2100 may include a modem 2170 coupled to the antenna 2152 via the transceiver 2150. The modem 2170 may be configured to send signals to a second device (not shown), receive signals from the second device, or both. According to a specific implementation, the modem 2170 may correspond to Figure 1 Modem 190.

[0146] The device 2100 may include a display device 106 coupled to a display controller 2126. One or more speakers 108 and one or more microphones 404 may be coupled to a codec 2134. The codec 2134 may include a digital-to-analog converter (DAC) 2102, an analog-to-digital converter (ADC) 2104, or both. In a particular implementation, the codec 2134 may receive analog signals from the one or more microphones 404, convert the analog signals to digital signals using the analog-to-digital converter 2104, and provide the digital signals to the voice and music codec 2108. The voice and music codec 2108 may process the digital signals, and the digital signals may be further processed by the slow motion video and audio engine 790. In a particular implementation, the voice and music codec 2108 may provide the digital signals to the codec 2134. The codec 2134 may use the digital-to-analog converter 2102 to convert the digital signals to analog signals, and may provide the analog signals to the one or more speakers 108.

[0147] In a particular implementation, the device 2100 can be included in a system-in-package or system-on-chip device 2122. In a particular implementation, the memory 2186, the processor 2106, the processor 2110, the display controller 2126, the codec 2134, and the modem 2170 are included in the system-in-package or system-on-chip device 2122. In a particular implementation, the input device 2130, the one or more cameras 402, and the power supply 2144 are coupled to the system-in-package or system-on-chip device 2122. In addition, in a particular implementation, as shown in FIG. Fig.21, the display device 106, the input device 2130, the one or more speakers 108, the one or more microphones 404, the one or more cameras 402, the antenna 2152, and the power supply 2144 are located outside the system-on-chip device 2122. In a particular implementation, each of the display device 106, the input device 2130, the one or more speakers 108, the one or more microphones 404, the one or more cameras 402, the antenna 2152, and the power supply 2144 can be coupled to a component of the system-on-chip device 2122, such as an interface or a controller.

[0148] Device 2100 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet device, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, an extended reality (XR) device, a base station, a mobile device, or any combination thereof.

[0149] In conjunction with the described implementations, an apparatus includes means for storing data corresponding to a media stream, the data including video recorded at a first frame rate and a plurality of audio segments, the plurality of audio segments including one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type. For example, the means for storing data may correspond to memory 112, one or more processors 116 or components thereof (e.g., one or more caches or buffers coupled to or integrated in media stream playback engine 120, media stream data structure generator 170, or both), media stream playback engine 120, audio playback unit 124, memory 2186, processor 2106, processor 2110, one or more other circuits or components configured to store data corresponding to a media stream, or any combination thereof.

[0150] The apparatus also includes a component for initiating playback of the media stream, the playback of the media stream comprising: playing back the video at a second frame rate different from the first frame rate, playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate, and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate. For example, the component for initiating playback of the media stream may correspond to one or more processors 116, media stream playback engine 120, video playback unit 122, audio playback unit 124, display device 106, one or more speakers 108, processor 2106, processor 2110, one or more other circuits or components configured to initiate playback of the media stream, or any combination thereof.

[0151] In some embodiments, a non-transitory computer-readable medium (e.g., memory 2186) includes instructions (e.g., instructions 2156) that, when executed by one or more processors (e.g., one or more processors 116), cause the one or more processors to obtain data (e.g., data 132) corresponding to a media stream (e.g., media stream 130), the data including a video (e.g., video 134) recorded at a first frame rate (e.g., first frame rate 136) and a plurality of audio segments, the plurality of audio segments including one or more first audio segments (e.g., one or more first audio segments 140) indicated as having a first playback type (e.g., first playback type 142) and one or more second audio segments (e.g., one or more second audio segments 150) indicated as having a second playback type (e.g., second playback type 152). When executed by the one or more processors, these instructions also cause the one or more processors to initiate playback of a media stream, wherein the playback of the media stream includes: playing back the video at a second frame rate (e.g., second frame rate 138) that is different from the first frame rate, playing back the one or more first audio segments at a first playback speed (e.g., first playback speed 144) corresponding to the first frame rate, and playing back the one or more second audio segments at a second playback speed (e.g., second playback speed 154) corresponding to the second frame rate.

[0152] The present disclosure includes the following set of embodiments.

[0153] According to embodiment 1, a device includes: a memory, which is configured to store data corresponding to a media stream, the data including: a video recorded at a first frame rate; and a plurality of audio segments, the plurality of audio segments including one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; and one or more processors, the one or more processors being coupled to the memory and being operably configured to initiate playback of the media stream, the playback of the media stream including: playing back the video at a second frame rate different from the first frame rate; playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate; and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0154] Embodiment 2 includes the apparatus of embodiment 1, wherein the first playback type corresponds to a speed-maintained playback type, and wherein the second playback type corresponds to an adjustable-speed playback type.

[0155] Embodiment 3 includes the apparatus of embodiment 1 or embodiment 2, wherein each audio segment of the plurality of audio segments is associated with a corresponding specific frame of the video representing a playback start point of the audio segment.

[0156] Embodiment 4 includes the apparatus of embodiment 3, wherein the one or more second audio segments are of substantially equal length and are associated with regularly spaced anchor frames in the video.

[0157] Embodiment 5 includes the apparatus of embodiment 3 or embodiment 4, wherein each of the one or more first audio segments is associated with a respective original frame corresponding to an audio generating event in the video.

[0158] Embodiment 6 includes the apparatus of any one of embodiments 1 to 5, wherein the second frame rate is slower than the first frame rate, and the second playback speed is slower than the first playback speed.

[0159] Embodiment 7 includes an apparatus according to any one of Embodiments 1 to 6, wherein the playback of the media stream includes generating the following items: a first audio stream including the one or more first audio segments, a second audio stream including the one or more second audio segments, and an output audio stream corresponding to the superposition of the first audio stream and the second audio stream.

[0160] Embodiment 8 includes an apparatus according to any one of embodiments 1 to 7, wherein, in order to obtain the data, the one or more processors are configured to: record the media stream to generate a recorded media stream, the recorded media stream comprising a recorded video stream and a recorded audio stream corresponding to the video; use a neural network to process the recorded audio stream to generate multiple audio streams, the multiple audio streams comprising a first audio stream and a second audio stream, the first audio stream comprising audio classified as having the first playback type, and the second audio stream comprising audio classified as having the second playback type; and generate a playback structure for the media stream to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed.

[0161] Embodiment 9 includes the apparatus according to embodiment 8, wherein, in order to generate the playback structure, the one or more processors are configured to: identify the original frame associated with the specific segment of the first audio stream in the recorded video stream; associate the specific segment of the first audio stream with the original frame in the playback structure, so that during the playback of the media stream, the initiation of the playback of the specific segment of the first audio stream and the initiation of the playback of the original frame are substantially simultaneous; and assign the first playback speed to the specific segment.

[0162] Embodiment 10 includes an apparatus according to embodiment 9, wherein, in order to identify the original frame associated with the particular segment, the one or more processors are configured to: identify one or more objects in the recorded video stream from which the sound associated with the beginning of the particular segment originates.

[0163] Embodiment 11 includes a device according to any one of embodiments 8 to 10, wherein, in order to generate the playback structure, the one or more processors are configured to: identify anchor frames in the recorded video stream based on the length and anchor frequency of the recorded video stream; split the second audio stream into a specific number of sequential segments with matching durations, the specific number of sequential segments being based on the number of identified anchor frames; associate a first segment of the sequential segments with a first identified anchor frame in the playback structure, so that during playback of the media stream, initiation of playback of the first segment of the sequential segments and initiation of playback of the first identified anchor frame are substantially simultaneous; and associate a second segment of the sequential segments with a second identified anchor frame in the playback structure, so that during playback of the media stream, initiation of playback of the second segment and initiation of playback of the second identified anchor frame are substantially simultaneous.

[0164] Embodiment 12 includes the apparatus of embodiment 11, wherein the anchor frequency is dynamic.

[0165] Embodiment 13 includes the apparatus of any one of Embodiments 8 to 12, wherein the neural network includes a recurrent neural network.

[0166] Embodiment 14 includes the apparatus of any one of Embodiments 8 to 13, further comprising one or more cameras configured to generate video data corresponding to the recorded video stream.

[0167] Embodiment 15 includes the device of any one of Embodiments 8 to 14, further comprising one or more microphones configured to generate audio data corresponding to the recorded audio stream.

[0168] Embodiment 16 includes the device of any one of Embodiments 1-15, further comprising a modem coupled to the one or more processors, the modem configured to receive the media stream from a second device.

[0169] Embodiment 17 includes a device according to any one of Embodiments 1 to 16, wherein the one or more processors are integrated in at least one of the following: a mobile phone, a tablet computer device, a wearable electronic device, or a camera device.

[0170] Embodiment 18 includes a device according to any one of embodiments 1 to 16, wherein the one or more processors are integrated into a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device.

[0171] Embodiment 19 includes the apparatus of any one of Embodiments 1-16, wherein the one or more processors are integrated into a vehicle.

[0172] According to embodiment 20, a method includes: obtaining data corresponding to a media stream at one or more processors, the data including a video recorded at a first frame rate and a plurality of audio segments, the plurality of audio segments including one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; initiating playback of the media stream at the one or more processors, the playback of the media stream including: playing the video at a second frame rate different from the first frame rate; playing the one or more first audio segments at a first playback speed corresponding to the first frame rate; and playing the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0173] Embodiment 21 includes the method according to embodiment 20, wherein each audio segment of the plurality of audio segments is associated with a corresponding specific frame of the video representing a playback start point of the audio segment.

[0174] Embodiment 22 includes the method of embodiment 21, wherein the one or more second audio segments have substantially equal lengths and are associated with regularly spaced anchor frames in the video.

[0175] Embodiment 23 includes the method of Embodiment 21 or Embodiment 22, wherein each of the one or more first audio segments is associated with a respective original frame corresponding to an audio generating event in the video.

[0176] Embodiment 24 includes the method of any one of Embodiments 20 to 23, wherein the second frame rate is slower than the first frame rate, and the second playback speed is slower than the first playback speed.

[0177] Embodiment 25 includes a method according to any one of Embodiments 20 to 24, wherein obtaining the data includes: recording the media stream to generate a recorded media stream, the recorded media stream including a recorded video stream and a recorded audio stream corresponding to the video; using a neural network to process the recorded audio stream to generate multiple audio streams, the multiple audio streams including a first audio stream and a second audio stream, the first audio stream including audio classified as having the first playback type, and the second audio stream including audio classified as having the second playback type; and generating a playback structure for the media stream to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed.

[0178] Embodiment 26 includes a method according to embodiment 25, wherein generating the playback structure includes: identifying an original frame associated with a specific segment of the first audio stream in the recorded video stream; associating the specific segment of the first audio stream with the original frame in the playback structure, so that during playback of the media stream, initiation of playback of the specific segment of the first audio stream and initiation of playback of the original frame are substantially simultaneous; and assigning the first playback speed to the specific segment.

[0179] Embodiment 27 includes a method according to embodiment 25 or embodiment 26, wherein using the neural network to process the recorded audio stream includes: identifying a first sound that is not perceptible to a user at a reduced playback speed; generating the first audio stream based on the first sound; identifying a second sound that is perceptible to the user at the reduced playback speed; generating the second audio stream based on the second sound; identifying a third sound that is not perceptible to the user at the reduced playback speed; determining that the third sound is to be omitted during playback based on an ignore parameter; and bypassing generation of an audio stream based on the third sound in response to determining that the third sound is to be omitted.

[0180] According to embodiment 28, a non-transitory computer-readable medium includes instructions, which when executed by one or more processors cause the one or more processors to: obtain data corresponding to a media stream, the data including: a video recorded at a first frame rate; and a plurality of audio segments, the plurality of audio segments including one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; and initiate playback of the media stream, the playback of the media stream including: playing back the video at a second frame rate different from the first frame rate; playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate; and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0181] Embodiment 29 includes a non-transitory computer-readable medium according to embodiment 28, wherein, to obtain the data, the instructions cause the one or more processors to: record the media stream to generate a recorded media stream, the recorded media stream comprising a recorded video stream and a recorded audio stream corresponding to the video; use a neural network to process the recorded audio stream to generate multiple audio streams, the multiple audio streams comprising a first audio stream and a second audio stream, the first audio stream comprising audio classified as having the first playback type, and the second audio stream comprising audio classified as having the second playback type; and generate a playback structure for the media stream to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed.

[0182] According to embodiment 30, a device includes: a component for storing data corresponding to a media stream, the data including a video and multiple audio segments recorded at a first frame rate, the multiple audio segments including one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; and a component for initiating playback of the media stream, the playback of the media stream including: playing back the video at a second frame rate different from the first frame rate; playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate; and playing back the one or more second audio segments at a second playback speed corresponding to the second frame rate.

[0183] It will also be appreciated by the skilled person that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above in terms of their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. The skilled person can implement the described functions in different ways for each specific application, and such specific implementation decisions will not be interpreted as causing a departure from the scope of the present disclosure.

[0184] The steps of the method or algorithm described in conjunction with the specific implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor, including that the processor and the storage medium may reside in an application-specific integrated circuit (ASIC), including that the ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0185] The previous description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but should be granted the broadest scope that may be consistent with the principles and novel features as defined by the following embodiments.

Claims

1. A device, comprising: A memory, the memory being configured to store data corresponding to a media stream, the data comprising: video recorded at a first frame rate; and a plurality of audio segments, the plurality of audio segments comprising one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; and one or more processors coupled to the memory and operably configured to initiate playback of the media stream, the playback of the media stream comprising: Playing back the video at a second frame rate different from the first frame rate; playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate; and The one or more second audio segments are played back at a second playback speed corresponding to the second frame rate. 2 . The apparatus of claim 1 , wherein the first playback type corresponds to a speed-maintained playback type, and wherein the second playback type corresponds to a speed-adjustable playback type.

3. The apparatus of claim 1, wherein each of the plurality of audio segments is associated with a corresponding specific frame of the video that represents a playback start point for the audio segment.

4. The apparatus of claim 3, wherein the one or more second audio segments have substantially equal lengths and are associated with regularly spaced anchor frames in the video.

5. The apparatus of claim 3, wherein each of the one or more first audio segments is associated with a respective original frame corresponding to an audio generating event in the video.

6. The device of claim 1, wherein the second frame rate is slower than the first frame rate, and the second playback speed is slower than the first playback speed.

7. A device according to claim 1, wherein the playback of the media stream includes generating the following items: a first audio stream including the one or more first audio segments, a second audio stream including the one or more second audio segments, and an output audio stream corresponding to the superposition of the first audio stream and the second audio stream.

8. The apparatus of claim 1 , wherein to obtain the data, the one or more processors are configured to: recording the media streams to generate recorded media streams, the recorded media streams comprising a recorded video stream and a recorded audio stream corresponding to the video; processing the recorded audio stream using a neural network to generate a plurality of audio streams, the plurality of audio streams comprising a first audio stream and a second audio stream, the first audio stream comprising audio classified as having the first playback type, the second audio stream comprising audio classified as having the second playback type; as well as A playback structure for the media streams is generated to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed.

9. The apparatus of claim 8, wherein to generate the playback structure, the one or more processors are configured to: identifying, in the recorded video stream, an original frame associated with a particular segment of the first audio stream; associating the particular segment of the first audio stream with the original frame in the playback structure such that during playback of the media stream, initiation of playback of the particular segment of the first audio stream and initiation of playback of the original frame are substantially simultaneous; and The first playback speed is assigned to the particular segment.

10. The apparatus of claim 9, wherein, in order to identify the original frame associated with the particular segment, the one or more processors are configured to: identify one or more objects in the recorded video stream from which the sound associated with the start of the particular segment originates.

11. The apparatus of claim 8, wherein to generate the playback structure, the one or more processors are configured to: identifying an anchor frame in the recorded video stream based on a length of the recorded video stream and an anchor frequency; splitting the second audio stream into a specific number of sequential segments having matching durations, the specific number of sequential segments being based on the number of identified anchor frames; associating a first one of the sequential segments with a first identified anchor frame in the playback structure such that during playback of the media stream, initiation of playback of the first one of the sequential segments and initiation of playback of the first identified anchor frame are substantially simultaneous; and A second one of the sequential segments is associated with a second identified anchor frame in the playback structure such that during playback of the media stream, initiation of playback of the second segment and initiation of playback of the second identified anchor frame are substantially simultaneous.

12. The apparatus of claim 11, wherein the anchor frequency is dynamic.

13. The apparatus of claim 8, wherein the neural network comprises a recurrent neural network.

14. The device of claim 8, further comprising one or more cameras configured to generate video data corresponding to the recorded video stream.

15. The device of claim 8, further comprising one or more microphones configured to generate audio data corresponding to the recorded audio stream.

16. The device of claim 1, further comprising a modem coupled to the one or more processors, the modem configured to receive the media stream from a second device.

17. The device of claim 1, wherein the one or more processors are integrated in at least one of: a mobile phone, a tablet computer device, a wearable electronic device, or a camera device.

18. The device of claim 1, wherein the one or more processors are integrated into a virtual reality head-mounted device, a mixed reality head-mounted device, or an augmented reality head-mounted device.

19. The apparatus of claim 1, wherein the one or more processors are integrated into a vehicle.

20. A method comprising: obtaining, at one or more processors, data corresponding to a media stream, the data comprising video recorded at a first frame rate and a plurality of audio segments, the plurality of audio segments comprising one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; as well as Initiating playback of the media stream at the one or more processors, the playback of the media stream comprising: Playing the video at a second frame rate different from the first frame rate; playing the one or more first audio segments at a first playback speed corresponding to the first frame rate; and The one or more second audio segments are played at a second playback speed corresponding to the second frame rate.

21. The method of claim 20, wherein each of the plurality of audio segments is associated with a corresponding specific frame of the video representing a playback start point for the audio segment.

22. The method of claim 21, wherein the one or more second audio segments are of substantially equal length and are associated with regularly spaced anchor frames in the video.

23. The method of claim 21, wherein each of the one or more first audio segments is associated with a respective original frame corresponding to an audio generating event in the video.

24. The method of claim 20, wherein the second frame rate is slower than the first frame rate and the second playback speed is slower than the first playback speed.

25. The method of claim 20, wherein obtaining the data comprises: recording the media streams to generate recorded media streams, the recorded media streams comprising a recorded video stream and a recorded audio stream corresponding to the video; processing the recorded audio stream using a neural network to generate a plurality of audio streams, the plurality of audio streams comprising a first audio stream and a second audio stream, the first audio stream comprising audio classified as having the first playback type, the second audio stream comprising audio classified as having the second playback type; as well as A playback structure for the media streams is generated to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed.

26. The method of claim 25, wherein generating the playback structure comprises: identifying, in the recorded video stream, an original frame associated with a particular segment of the first audio stream; associating the particular segment of the first audio stream with the original frame in the playback structure such that during playback of the media stream, initiation of playback of the particular segment of the first audio stream and initiation of playback of the original frame are substantially simultaneous; as well as The first playback speed is assigned to the particular segment.

27. The method of claim 25, wherein processing the recorded audio stream using the neural network comprises: identifying a first sound that is not perceptible to a user at the reduced playback speed; generating the first audio stream based on the first sound; identifying a second sound perceptible to the user at the reduced playback speed; generating the second audio stream based on the second sound; identifying a third sound that is imperceptible to the user at the reduced playback speed; determining, based on an ignore parameter, that the third sound is to be omitted during playback; as well as Generating an audio stream based on the third sound is bypassed in response to determining that the third sound is to be omitted.

28. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: Obtain data corresponding to the media stream, the data comprising: Video recorded at a first frame rate; and a plurality of audio segments, the plurality of audio segments comprising one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; as well as Initiating playback of the media stream, the playback of the media stream comprising: Playing back the video at a second frame rate different from the first frame rate; playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate; and The one or more second audio segments are played back at a second playback speed corresponding to the second frame rate.

29. The non-transitory computer readable medium of claim 28, wherein to obtain the data, the instructions cause the one or more processors to: recording the media streams to generate recorded media streams, the recorded media streams comprising a recorded video stream and a recorded audio stream corresponding to the video; processing the recorded audio stream using a neural network to generate a plurality of audio streams, the plurality of audio streams comprising a first audio stream and a second audio stream, the first audio stream comprising audio classified as having the first playback type, the second audio stream comprising audio classified as having the second playback type; as well as A playback structure for the media streams is generated to facilitate playback of the recorded video stream at the second frame rate, playback of the first audio stream at the first playback speed, and playback of the second audio stream at the second playback speed.

30. An apparatus, comprising: means for storing data corresponding to a media stream, the data comprising video recorded at a first frame rate and a plurality of audio segments, the plurality of audio segments comprising one or more first audio segments indicated as having a first playback type and one or more second audio segments indicated as having a second playback type; and A component for initiating playback of the media stream, the playback of the media stream comprising: Playing back the video at a second frame rate different from the first frame rate; playing back the one or more first audio segments at a first playback speed corresponding to the first frame rate; and The one or more second audio segments are played back at a second playback speed corresponding to the second frame rate.

Images