Audio or visual input interacting with video creation

Abstract

During creation of a video item, a first type of input may be received via a first component of a computing device. The first type of input may correspond to a first type of element associated with the video item. A characteristic of the signal in the first type of input may be determined in real-time based on the first type of input. At least one modification may be made to a second type of element associated with the video item based at least in part on a characteristic of the signal in the first type of input. In some examples, the first type of input may be an audio input, and the second type of element may include a visual element. In other examples, the first type of input may be a video input, and the second type of element may include an audio element.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial No. 17 / 843,891, filed on June 17, 2022, and entitled “AUDIO OR VISUAL INPUT INTERACTING WITH VIDEO CREATION,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Communication is increasingly being conducted using internet-based tools. Internet-based tools can be any software or platform. Existing social media platforms enable users to communicate with each other by sharing images, videos and other information via static applications or web pages. As communication devices such as mobile phones become more sophisticated, people continue to expect new ways of entertainment, social networking and communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The following detailed description will be better understood when read in conjunction with the accompanying drawings.For illustrative purposes, example embodiments of various aspects of the disclosure are shown in the drawings; however, the invention is not limited to the specific methods and instrumentalities disclosed.

[0005] Figure 1 An example system for distributing content that can be in accordance with the present disclosure is shown.

[0006] Figure 2 Example diagrams depicting audio and visual element modifications that may be made in accordance with the present disclosure are shown.

[0007] Figure 3 An example method is shown that can be used in accordance with the present disclosure for element modification based on video creation input.

[0008] Figure 4 Example methods for visual element modification based on audio input that may be consistent with the present disclosure are shown.

[0009] Figure 5 Another example method for visual element modification based on audio input that may be consistent with the present disclosure is shown.

[0010] Figure 6 Another example method for visual element modification based on audio input that may be consistent with the present disclosure is shown.

[0011] Figure 7 Another example method for visual element modification based on audio input that may be consistent with the present disclosure is shown.

[0012] Figure 8 Another example method for visual element modification based on audio input that may be consistent with the present disclosure is shown.

[0013] Fig. 9 An example method for audio element modification based on video input that may be in accordance with the present disclosure is shown.

[0014] Fig.10 Another example method for audio element modification based on video input that may be in accordance with the present disclosure is shown.

[0015] Fig.11A An example user interface that can be consistent with the present disclosure is shown.

[0016] Fig. 11B Another example user interface that can be consistent with the present disclosure is shown.

[0017] Fig. 12A Another example user interface that can be consistent with the present disclosure is shown.

[0018] Fig. 12B Another example user interface that can be consistent with the present disclosure is shown.

[0019] Fig.13A Another example user interface that can be consistent with the present disclosure is shown.

[0020] Fig. 13B Another example user interface that can be consistent with the present disclosure is shown.

[0021] Fig.14 An example computing device is shown that can be used to perform any of the techniques disclosed herein. DETAILED DESCRIPTION

[0022] Users can use content creation platforms to generate content, such as video items. In some examples, video items can be created based on video input, such as can be captured via a camera. Video input can sometimes include a user's video, such as including the user's face. In addition, the video item can have a corresponding audio output. The audio output can be generated based on an audio input, such as can be captured via a microphone. The audio input can sometimes include the user's audio, such as including the user's voice. The user may wish to add effects to the video item, such as one or more visual effects. The user may also wish to add audio effects to the output audio accompanying the video item.

[0023] One disadvantage of existing video creation platforms is that the technology for controlling audio and visual effects can be limited. For example, a user may have limited input technology to indicate when to apply a visual or audio effect to a video item or output audio. The technology for selecting the type of visual and audio effects and for controlling the magnitude and duration of the visual and audio effects may also be limited. Some existing visual and audio effect input technologies may be inconvenient and may require the user to perform additional actions or steps that were not originally required to create the video. This can potentially lead to user confusion, delay the video creation process, prevent users from adopting desired effects, and otherwise degrade the user experience for content creators and content viewers.

[0024] This article describes the technology for creating audio and visual effects based on video input. In some examples, the described technology can allow users to create and control visual effects based on audio input, optionally in real time. In one example, changes in the audio input from a microphone (such as changes in the user's voice) can be used to control one or more visual effects in a video item. These changes can be included in the characteristics of sound features (such as pitch, tone, volume, energy or duration). Visual effects can include, for example, stretching or shaking visual elements, inserting animated graphic elements into video items, moving visual elements, etc. In addition, in some examples, the described technology can allow users to create and control audio effects based on video input, optionally in real time. In one example, changes in the video input from a camera (such as the movement of one or more body parts) can be used to control one or more audio effects in an audio output. Audio effects can include, for example, modifying sound features in an audio output, such as pitch, tone, volume, energy, echo, duration, etc.

[0025] The techniques described herein for creating audio and visual effects based on video input may be used by a system for distributing content. Figure 1 An example system 100 for distributing content is illustrated. System 100 may include a server 102 and a plurality of client devices 104. Server 102 and plurality of client devices 104a-n may communicate with each other via one or more networks 132.

[0026] The servers 102 may be located at a data center (such as a single location) or distributed across different geographic locations (e.g., at several locations). The servers 102 may provide services via one or more networks 132. The one or more networks 132 include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and the like. The one or more networks 132 may include physical links, such as coaxial cable links, twisted pair links, fiber optic links, combinations thereof, and the like. The one or more networks 132 may include wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0027] Server 102 may include multiple computing nodes hosting various services. In an embodiment, the node hosts video service 112. Video service 112 may include content streaming services, such as Internet protocol video streaming services. Video service 112 may be configured to distribute content 123 via various transmission technologies. Video service 112 is configured to provide content 123, such as video, audio, text data, combinations thereof, etc. Content 123 may include content streams (e.g., video streams, audio streams, information streams), content files (e.g., video files, audio files, text files) and / or other data. Content 123 may be stored in database 122. For example, video service 112 may include video sharing services, video hosting platforms, content distribution platforms, collaborative gaming platforms, etc.

[0028] In an embodiment, the content 123 distributed or provided by the video service 112 includes a video. The video may have a duration less than or equal to a predetermined time limit, such as one minute, five minutes, or other predetermined number of minutes. As an example and not limitation, the video may include at least one but not more than four fifteen-second segments strung together. The short duration of the video may provide quick entertainment for the viewer, which allows the user to watch a large amount of video in a short time frame. Such quick entertainment may be popular on social media platforms.

[0029] The video may include a pre-recorded audio overlay, such as a clip of a pre-recorded song or the audio of a television program or movie. If the short video includes a pre-recorded audio overlay, the short video may feature one or more individuals lip-syncing, dancing, or otherwise moving their bodies to the pre-recorded audio. For example, a short video may feature an individual completing a "dance challenge" to a popular song, or a short video may feature two individuals participating in a lip-sync or dance duet. As another example, a short video may feature an individual completing a challenge that requires them to move his or her body in a manner corresponding to the pre-recorded audio overlay, such as in a manner corresponding to the beat or rhythm of a pre-recorded song featuring the pre-recorded audio overlay. Other videos may not include a pre-recorded audio overlay. For example, these videos may feature individual sports, pranks, or giving advice (such as beauty and fashion advice, cooking tips, or home improvement tips).

[0030] In an embodiment, content 123 may be output to different client devices 104 via network 132. Content 123 may be streamed to client devices 104. The content stream may be a stream of video received from video service 112. Multiple client devices 104 may be configured to access content 123 from video service 112. In an embodiment, client device 104 may include content application 106. Content application 106 outputs (e.g., displays, renders, presents) content to a user associated with client device 104. Content 123 may include video, audio, comments, text data, etc.

[0031] The plurality of client devices 104 may include any type of computing device, such as a mobile device, a tablet device, a laptop computer, a desktop computer, a smart TV or other smart device (e.g., a smart watch, a smart speaker, smart glasses, a smart helmet), a gaming device, a set-top box, a digital streaming device, a robot, etc. The plurality of client devices 104 may be associated with one or more users. A single user may access the server 102 using one or more of the plurality of client devices 104. The plurality of client devices 104 may travel to various locations and use different networks to access the server 102.

[0032] The video service 112 may be configured to receive input from a user. A user may be registered as a user of the video service 112 and may be a user of the content application 106 operating on the client device 104. The user input may include a video created by the user, a user comment associated with the video, or a "like" associated with the video. The user input may include a connection request and user input data, such as text data, digital image data, or user content. The connection request may include a request from a client device 104a-d to connect to the video service 112. The user input data may include information that the user connected to the video service 112 wants to share with other connected users of the video service 112, such as videos and / or user comments.

[0033] The video service 112 may be able to receive different types of input from users using different types of client devices 104. For example, a user using a content application 106 on a first user device, such as a mobile phone or tablet, may be able to create and upload videos using the content application 106. Users using the content application 106 on different mobile phones or tablets may also be able to view, comment on, or "like" videos or comments written by other users. In another example, a user using the content application 106 on a smart TV, laptop, desktop, or gaming device may not be able to create and upload videos or comment on videos using the content application 106. Instead, a user using the content application 106 on a smart TV, laptop, desktop, or gaming device may only be able to use the content application 106 to view videos, view comments left by other users, and "like" videos.

[0034] In an embodiment, a user can create a video (such as a short video) using a content application 106 on a client device 104 and upload the video to a server 102. The client device 104 can access an interface 108 of the content application 106. The interface 108 may include an input element. For example, the input element may be configured to allow the user to create a video. In order to create a short video, the user may give permission for the content application 106 to access an image capture device (such as a camera or microphone) of the client device 104. Using the content application 106, the user may select the duration of the video or set the speed of the video, such as "slow motion" or "speed up".

[0035] The user may edit the video using the content application 106. After the user has created the video, the user may upload the video to the server 102 and / or save the video locally to the client device 104a-n using the content application 106. When the user uploads the video to the server 102, they may select whether they want the video to be viewable by all other users of the content application 106 or only by a subset of the users of the content application 106. The video service 112 may store the uploaded video and any metadata associated with the video in one or more databases 122.

[0036] In an embodiment, a user may provide input about a video using a content application 106 on a client device 104. The client device 104 may access an interface 108 of the content application 106 that allows the user to provide input associated with the video. The interface 108 may include an input element. For example, the input element may be configured to receive input from the user, such as a comment or "like" associated with a particular video. If the input is a comment, the content application 106 may allow the user to set an emoticon associated with his or her input. The content application 106 may determine timing information for the input, such as when the user posted a comment. The content application 106 may send the input and associated metadata to the server 102. For example, the content application 106 may send the comment, an identifier of the user who posted the comment, and timing information for the comment to the server 102. The video service 112 may store the input and associated metadata in a database 122.

[0037] The video service 112 may be configured to output uploaded videos and user inputs to other users. A user may register as a user of the video service 112 to view videos created by other users. The user may be a user of a content application 106 operating on a client device 104. The content application 106 may output (display, render, present) videos and user comments to a user associated with the client device 104. The client device 104 may access an interface 108 of the content application 106. The interface 108 may include output elements. The output elements may be configured to display information about different videos so that the user may select a video to view. For example, the output elements may be configured to display multiple cover images, titles, or topic tags associated with a video. The output elements may also be configured to arrange videos according to categories associated with each video.

[0038] In an embodiment, user comments associated with a video can be output to other users who watch the same video. For example, all users who access a video can view comments associated with the video. Video service 112 can output videos and associated comments simultaneously. Comments can be output by video service 112 in real time or near real time. Content application 106 can display videos and comments on client device 104 in various ways. For example, comments can be displayed in an overlay above the content or in an overlay next to the content. As another example, a user who wants to view comments of other users associated with a video may need to select a button to view comments. Comments can be animated when displayed. For example, comments can be displayed as scrolling across the video or across the overlay.

[0039] According to the techniques described herein, characteristics of input signals (such as signals in input video and input audio) may be used to control modification of elements (such as visual or audio elements) associated with a video item. Figure 2 An example diagram depicting audio and visual element modifications that may be made in accordance with the present disclosure is shown. Figure 2 As shown in FIG. 1 , system 200 generates video item 154 based on video input 114. Video input 114 may be captured by camera 107, such as may include Figure 1 In one or more client devices 104a-n. Figure 2 In the example of , video input 114 includes a video of user 101, such as a video including the user's face and / or other body parts of user 101. Video item 154 may include at least some of the content of video input 114, and in some cases all of the content, but the content of video input 114 may be modified in one or more ways when included in video item 154.

[0040] The video item 154 has a corresponding audio output 155. The audio output 155 may be generated based on the audio input 115. The audio input 115 may be captured by the microphone 105, such as may be included in Figure 1 In one or more client devices 104a-n. Figure 2 In the example of , audio input 115 includes audio of user 101, such as audio including the voice of user 101. Audio output 155 may include at least some of the content of audio input 115, and in some cases all of the content, but the content of audio input 115 may be modified in one or more ways when included in audio output 155. During playback, video item 154 and audio output 155 may be played together in synchronization with each other. For example, audio output 155 may include words spoken by user 101, and these words may be played in audio output 155 in synchronization with the display of corresponding mouth movements of user 101 in video item 154.

[0041] like Figure 2 As shown in , the audio analysis component 125 can perform audio analysis on the audio input 115 to determine the audio signal characteristics 135. The audio analysis can be performed during the creation of the video item 154. The audio signal characteristics 135 can change over time in the audio input 115. For example, the audio signal characteristics 135 can include sound characteristics, such as pitch, pitch, volume, energy and / or duration. In some examples, the audio signal characteristics 135 can include sound characteristics, such as the pitch, pitch, volume, energy and / or duration of the voice of the user 101, including the sound emitted by the user 101. In some cases, the audio analysis component 125 can use one or more machine learning models (such as one or more neural network models) to perform audio analysis on the audio input 115. In some examples, the audio analysis can include converting the audio input 115 to the frequency domain, such as by performing one or more frequency domain transforms (e.g., Fourier transforms or another similar transform) on the audio input 115. In some examples, the audio analysis can include determining the audio signal characteristics 135 at different times, such as pitch, pitch, volume and energy, and determining and tracking the changes of those characteristics over time.

[0042] After being determined by the audio analysis component 125, the audio signal characteristics 135 can be provided to the modification component 140. Then, the modification component 140 can evaluate the audio signal characteristics 135 in conjunction with the audio-based modification instructions 145 to determine one or more modifications made to the video item 154. For example, the audio-based modification instructions 145 can indicate various conditions for modifying the visual elements 164 within the video item 154. Specifically, the audio-based modification instructions 145 can indicate changes in the audio signal characteristics 135 that can trigger the modification of the visual elements 164. For example, these triggering conditions can include changes in the pitch, tone or volume of the user's voice or another sound (such as background sound). In some cases, the triggering condition can require that the pitch, tone or volume meet the selected criteria at least within the threshold duration. The selected criteria can include, for example, exceeding a minimum threshold value, being lower than a maximum threshold value, remaining within a given value range, matching or being associated with a given mode, etc.

[0043] In addition to specifying the trigger conditions, the audio-based modification instructions 145 can also specify the modifications caused by the trigger conditions. In some examples, the modification can include stretching, shaking, changing color and / or moving the visual element 164. Moreover, in some examples, the modification can include inserting an animated graphic element into the video item 154 to replace, cover or otherwise modify the visual element 164. In some examples, the visual element 164 can include the body parts of the user 101, such as the user's eyes, eyebrows, mouth, lips, nose, hands, arms, etc. In other examples, the visual element 164 can be different kinds of objects, such as controllable characters in a game-related video. Moreover, in some examples, the visual element 164 can include all or any part of the image frame in the video item.

[0044] In a specific example, visual element 164 can stretch and / or shake in one or more directions based on the pitch of the voice of user 101, such as when the pitch increases to above certain levels. In another specific example, increasing the intensity (e.g., volume, energy, etc.) of the voice of the user can cause an animated graphic element (e.g., a fireball) to be inserted into visual element 164 (e.g., the eyeball of the user), for example, to convey a feeling of anger or fury. In addition, as the intensity of the voice of the user increases, the voice of the user can gradually present a more metallic texture. In another specific example, the change of the pitch of the voice of the user can control the movement of visual element 164. For example, visual element 164 can be a character in a game. The character can move upwards as the pitch of the voice of the user increases, and the character can move downwards as the pitch of the voice of the user decreases. After detecting the audio signal characteristic 135 that triggers modification, any or all of these modifications to visual element 164 can be performed in real time.

[0045] like Figure 2As shown in , the video analysis component 124 can perform video analysis on the video input 114 to determine the video signal characteristics 134. The video analysis can be performed during the creation of the video item 154. The video signal characteristics 134 can change over time in the video input 114. For example, the video signal characteristics 134 can include the position and movement of different objects in the video input 114. In some examples, the video signal characteristics 134 can include the position and movement of at least one body part of the user 101 (such as eyes, eyebrows, mouth, head, hands, etc.). In some cases, the video analysis component 124 can use one or more machine learning models (such as one or more neural network models) to perform video analysis on the video input 114. In some examples, the video analysis can include performing object detection and / or object recognition analysis on different frames of the video input. The object detection and / or object recognition analysis can include facial detection and / or facial recognition analysis. In some examples, the video analysis can include determining the video signal characteristics 134 at different times, such as object positions and determining and tracking changes in those positions over time.

[0046] After being determined by the video analysis component 124, the video signal characteristics 134 may be provided to the modification component 140. The modification component 140 may then evaluate the video signal characteristics 134 in conjunction with the video-based modification instructions 144 to determine one or more modifications to be made to the audio output 155. For example, the video-based modification instructions 144 may indicate various conditions for modifying the audio elements 165 within the audio output 155. Specifically, the video-based modification instructions 144 may indicate changes in the video signal characteristics 134 that may trigger modifications to the audio elements 165. For example, these triggering conditions may include movement of an object (such as a body part of the user 101, e.g., the user's eyes, eyebrows, mouth, lips, nose, hands, arms, etc.). In some cases, the triggering conditions may require the object to move in one or more selected directions, exceed a given movement speed threshold and / or movement distance threshold, follow a selected movement pattern, etc.

[0047] In addition to specifying the trigger condition, the modification instruction 144 based on the video can also specify the modification caused by the trigger condition. In some examples, the modification can include changing the audio element 165. In some examples, the audio element 165 can include the characteristics of the voice of the user 101, such as pitch, tone, volume, energy, echo or duration. In a specific example, the user 101 can move his or her eyebrows to change the pitch of the user's voice. For example, the eyebrows can be moved upward to improve the pitch of the user's voice, and the eyebrows can be moved downward to reduce the pitch of the user's voice. In another specific example, the movement of body parts or other objects can cause an echo effect. For example, the body parts can be raised upward to increase the echo effect, such as real-time transformation of the user's voice to achieve an angelic echo. The body parts can also be lowered to reduce the echo effect. After detecting the video signal characteristics 134 that trigger the modification, any or all of these modifications to the audio element 165 can be performed in real time.

[0048] In some examples, any or all of the above-described video analysis techniques, audio analysis techniques, visual element modification techniques, and audio element modification techniques may be performed by Figure 1 One or more of the client devices 104a-n and / or the server 102 executes. Return to Figure 1 , shows an example in which the video analysis component 124, the audio analysis component 125, and the modification component 140 are each included in the server 102 and the client devices 104a-n. Figure 1 In some examples, each of the client devices 104a-n and the server 102 can perform any or all of the above-described video analysis techniques, audio analysis techniques, visual element modification techniques, and audio element modification techniques. In some examples, the above-described video analysis techniques, audio analysis techniques, visual element modification techniques, and / or audio element modification techniques can be performed entirely at the client device 104. In some other examples, the above-described video analysis techniques, audio analysis techniques, visual element modification techniques, and / or audio element modification techniques can be performed entirely at the server 102.

[0049] In other examples, the execution of the above-described video analysis techniques, audio analysis techniques, visual element modification techniques, and / or audio element modification techniques can be distributed between the client device 104 and the server 102. For scenarios in which the server 102 is employed to perform any or all of these techniques, the video input 114 and / or the audio input 115 can be captured by the client device 104a-n (e.g., via the camera 107 and / or the microphone 105) and then provided to the server 102 via one or more networks 132 for analysis, processing, and / or modification. In addition, in some examples, the results of the analysis, processing, and / or modification can be transmitted from the server 102 to the client device 104a-n via the one or more networks 132. Specifically, in some examples, the server 102 can perform modifications to the visual element 164 and / or the audio element 165. In other examples, the server 102 can only determine when the modification should be performed, and the server can transmit instructions to perform this modification back to the client device 104a-n to enable the modification to be performed on the client side.

[0050] Figure 3 An example process 300 for creating an element modification based on a video input according to the present disclosure is shown. At operation 310, a first type of input is received via a first component of a computing device during creation of a video item, wherein the first type of input corresponds to a first type of element associated with the video item. In some examples, receiving the first type of input via the first component of the computing device may include receiving audio input via a microphone. As described above with reference to Figure 2 As described, the first type of input may be audio input 115, which may be received via microphone 105 during creation of video item 154. Audio input 115 may include or may otherwise correspond to an element of the first type, such as audio element 165. Audio element 165 may include characteristics of the user's 101 voice, such as pitch, tone, volume, energy, echo, or duration.

[0051] In some other examples, receiving the first type of input via the first component of the computing device may include receiving video input via a camera. As described above, the first type of input may be video input 114, which may be received via camera 107 during creation of video item 154. Video input 114 may include or may otherwise correspond to an element of the first type, such as visual element 164. In some examples, visual element 164 may include a body part of user 101, such as the user's eyes, eyebrows, mouth, lips, nose, hands, arms, etc.

[0052] At operation 312, the characteristics of the signal in the input of the first type are determined in real time based on the input of the first type. In the input of the first type, the characteristics may change over time. In some examples, the characteristics may include sound features, and the sound features may include at least one of pitch, pitch, volume, energy, or duration. As described above, the audio analysis component 125 may perform audio analysis on the audio input 115 to determine the audio signal characteristics 135. The audio signal characteristics 135 may include a measurement of at least one of pitch, pitch, volume, energy, or duration. In some examples, the audio analysis component 125 may use one or more machine learning models (such as one or more neural network models) to perform audio analysis on the audio input 115. In some cases, the audio analysis may include converting the audio input 115 to the frequency domain, such as by performing one or more frequency domain transforms (e.g., Fourier transform or another similar transform) on the audio input 115. In some examples, the audio analysis may include determining the audio signal characteristics 135 at different times, such as pitch, pitch, volume, and energy, and determining and tracking the changes of those characteristics over time.

[0053] In some other examples, the characteristic may include movement of at least one body part within the video input. As described above, the video analysis component 124 may perform video analysis on the video input 114 to determine the video signal characteristic 134. The video signal characteristic 134 may include calculations of movement of at least one body part within the video input 114. In some examples, the video analysis component 124 may employ one or more machine learning models (such as one or more neural network models) to perform video analysis on the video input 114. In some cases, the video analysis may include performing object detection and / or object recognition analysis on different frames of the video input. The object detection and / or object recognition analysis may include facial detection and / or facial recognition analysis. In some examples, the video analysis may include determining audio signal characteristics 135 at different times, such as object locations and determining and tracking changes in those locations over time.

[0054] At operation 314, at least one modification is made to an element of a second type associated with the video item based at least in part on a characteristic of the signal in the input of the first type. Upon determining the characteristic of the signal in the input of the first type that triggers the at least one modification, the at least one modification may be caused and performed in real time. In some examples, the element of the second type may include a visual element in the video item. As described above with reference to Figure 2As described, modification component 140 may modify visual elements 164 in video item 154 based on audio signal characteristics 135 and audio-based modification instructions 145. For example, audio-based modification instructions 145 may indicate various conditions for modifying visual elements 164 within video item 154. Specifically, audio-based modification instructions 145 may indicate that audio signal characteristics 135 may trigger changes to the modification of visual elements 164. In addition to specifying triggering conditions, audio-based modification instructions 145 may also specify modifications caused by the triggering conditions. In one specific example, operation 314 may include stretching the visual element in at least one direction based on a change in at least one of the characteristics of the signal within the audio input. Refer to FIG. 314 below. Figure 5 This example is described in detail. In another specific example, operation 314 may include shaking the visual element based on a change in at least one of the characteristics of the signal within the audio input. Figure 6 This example is described in detail. In another specific example, operation 314 may include inserting at least one animated graphic element into the video item based on a change in at least one of the characteristics of the signal within the audio input. Figure 7 This example is described in detail. In another specific example, operation 314 may include causing movement of the visual element based on a change in at least one of the characteristics of the signal within the audio input. Figure 8 Describe this example in detail.

[0055] In some other examples, the second type of element may include an audio element in an audio output associated with the video item. Figure 2 As described, the modification component 140 can modify the audio element 165 in the audio output 155 based on the video signal characteristics 134 and the video-based modification instructions 144. For example, the video-based modification instructions 144 can indicate various conditions for modifying the audio element 165 within the audio output 155. Specifically, the video-based modification instructions 144 can indicate changes in the video signal characteristics 134 that can trigger the modification of the audio element 165. In addition to specifying the triggering condition, the video-based modification instructions 144 can also specify the modification caused by the triggering condition. In a specific example, operation 314 can include modifying a sound feature in the audio output based on the movement of at least one body part within the video input, wherein the sound feature includes at least one of pitch, intonation, volume, energy, echo, or duration. Refer to the following Fig.10 Describe this example in detail.

[0056] Figure 4An example process 400 for modifying a visual element based on audio input according to the present disclosure is shown. At operation 410, audio input is received via a microphone during creation of a video item, wherein the audio input corresponds to a speech element associated with the video item. Figure 2 As described, audio input 115 may be received via microphone 105 during creation of video item 154. Audio input 115 may include or may otherwise correspond to an element of a first type, such as audio element 165. Audio element 165 may include characteristics of the user's 101 voice, such as pitch, tone, volume, energy, echo, or duration.

[0057] At operation 412, characteristics of the signal in the input audio are determined in real time based on the audio input. These characteristics may change over time in the audio input. In some examples, these characteristics may include sound features, and the sound features may include at least one of pitch, pitch, volume, energy, or duration. As described above, the audio analysis component 125 may perform audio analysis on the audio input 115 to determine the audio signal characteristics 135. The audio signal characteristics 135 may include a measurement of at least one of pitch, pitch, volume, energy, or duration. In some examples, the audio analysis component 125 may use one or more machine learning models (such as one or more neural network models) to perform audio analysis on the audio input 115. In some cases, the audio analysis may include converting the audio input 115 to the frequency domain, such as by performing one or more frequency domain transforms (e.g., Fourier transform or another similar transform) on the audio input 115. In some examples, the audio analysis may include determining the audio signal characteristics 135 at different times, such as pitch, pitch, volume, and energy, and determining and tracking the changes of those characteristics over time.

[0058] At operation 414, at least one modification is made to a visual element in the video item based at least in part on the characteristic of the signal in the audio input. Upon determining the characteristic of the signal in the audio input that triggers the at least one modification, the at least one modification may be caused and performed in real time. Figure 2As described, the modification component 140 can modify the visual elements 164 in the video item 154 based on the audio signal characteristics 135 and the audio-based modification instructions 145. For example, the audio-based modification instructions 145 can indicate various conditions for modifying the visual elements 164 within the video item 154. Specifically, the audio-based modification instructions 145 can indicate changes in the audio signal characteristics 135 that can trigger the modification of the visual elements 164. In addition to specifying the triggering conditions, the audio-based modification instructions 145 can also specify the modifications caused by the triggering conditions. In some examples, sound characteristics such as pitch, tone, volume, energy, or duration can cause the modification of the visual elements. The sound characteristics can be characteristics of the user's voice and / or other sounds (such as background sounds). Refer to the following Figure 5-8 Some specific examples of modifications that may be performed on the visual element at operation 414 are described in detail. In some examples, for example, object detection (e.g., face detection) and / or object recognition (e.g., face recognition) analysis may be performed on frames of video input 114 and / or video item 154 by video analysis component 124 to determine the location of visual element 164 (such as a body part (e.g., facial feature) or other object) using the frame. After determining the location of the visual element within the frame, the visual element may be modified by modification component 140, such as by modifying pixel values ​​or otherwise modifying the frame at the determined location of visual element 164.

[0059] Reference now Figure 5 , describes an example process 500 in which a visual element is stretched in at least one direction based on a change in at least one of the characteristics of a signal within an audio input. Note that, Figure 5 Operations 510 and 512 are related to Figure 4 Operations 410 and 412 are identical. Therefore, the descriptions of operations 410 and 412 may be considered applicable to operations 510 and 512, respectively, and these descriptions are not repeated here. Figure 5 At operation 514, the visual element is stretched in at least one direction based on a change in at least one of the characteristics of the signal in the audio input. The visual element may be stretched in real time after at least one of the characteristics of the signal in the audio input changes. Figure 2As described, the visual element 164 can be a body part of the user 101, such as the user's face, eyes, eyebrows, mouth, lips, nose, hands, arms, etc. Therefore, the user's body parts can be stretched in one or more directions (e.g., horizontally, vertically, diagonally, etc.). For example, in some cases, the user's face, including facial features such as eyes, mouth, nose, etc., can be stretched in one or more directions based on changes in the user's voice (such as changes in pitch, tone, volume, energy, etc.). Other objects, such as objects worn by the user (e.g., glasses, hats, etc.) and other background or foreground objects, can also be stretched. In a specific example, the user's face can be stretched in one or more directions based on changes in the pitch of the user's 101 voice. For example, in some cases, the stretching of the user's face can increase as the pitch of the user's voice increases, and the amount of stretching of the user's face can decrease as the pitch of the user's voice decreases. In one example, once the pitch of the user's voice reaches the selected threshold pitch value, the user's face can begin to stretch, and the amount of stretching can increase as the pitch of the user's voice increases. Once the user stops speaking or the pitch of the user's voice falls below a threshold pitch value, the user's face stops stretching. In some examples, the entire center portion of the image frame (which often may include the user's face) may be stretched horizontally so that the left and right portions of the image frame are not displayed (e.g., cut off) and only the stretched center portion of the image frame is displayed. Fig.11A -B describes in detail some example user interfaces depicting example extrusions of visual elements.

[0060] Reference now Figure 6 , describes an example process 600 in which a visual element is caused to jitter based on a change in at least one of the characteristics of a signal within an audio input. Note that, Figure 6 Operations 610 and 612 are related to Figure 4 Operations 410 and 412 are identical. Therefore, the descriptions of operations 410 and 412 may be considered applicable to operations 610 and 612, respectively, and these descriptions are not repeated here. Figure 6 At operation 614, the visual element is jittered based on a change in at least one of the characteristics of the signal within the audio input. The visual element may be jittered in real time after at least one of the characteristics of the signal within the audio input changes. Figure 2As described, visual element 164 can be a body part of user 101, such as the user's face, eyes, eyebrows, mouth, lips, nose, hands, arms, etc. Therefore, the user's body parts can be made to shake. For example, in some cases, the user's face (including facial features such as eyes, mouth, nose, etc.) can shake based on changes in the user's voice (such as changes in pitch, tone, volume, energy, etc.). Other objects, such as objects worn by the user (e.g., glasses, hats, etc.) and other background or foreground objects, can also shake. In a specific example, the user's face may shake based on changes in the pitch of the user's 101 voice. For example, in some cases, the amount of shaking of the user's face can increase as the pitch of the user's voice increases, and the amount of shaking of the user's face can decrease as the pitch of the user's voice decreases. In one example, once the pitch of the user's voice reaches a selected threshold pitch value, the user's face begins to shake, and the amount of shaking can increase as the pitch of the user's voice increases. Once the user stops talking or the pitch of the user's voice is below the threshold pitch value, the user's face stops shaking. Refer to the following Fig.11A -B describes in detail some example user interfaces depicting example dithering of visual elements.

[0061] Reference now Figure 7 , describes an example process 700 in which at least one animated graphic element is inserted into a video item based on a change in at least one of the characteristics of a signal within an audio input. Note that, Figure 7 Operations 710 and 712 are related to Figure 4 Operations 410 and 412 are identical. Therefore, the descriptions of operations 410 and 412 may be considered applicable to operations 710 and 712, respectively, and these descriptions are not repeated here. Figure 7 At operation 714, at least one animated graphic element is inserted into the video item based on a change in at least one of the characteristics of the signal within the audio input. After at least one of the characteristics of the signal within the audio input changes, the at least one animated graphic element can be inserted into the video item in real time. For example, in some cases, increasing the intensity (e.g., volume, energy, etc.) of the user's voice can cause an animated graphic element (such as a fireball) to be inserted onto a visual element (such as the user's eyeball), for example to convey a feeling of anger or fury. In addition, in some examples, other animated graphic elements can also be displayed. For example, the background can be modified by making animated rays or spikes appear to be emitted from the user's face to convey a feeling of anger or fury. In some examples, the emission of the animated rays or spikes can be synchronized with the user's speech and can increase as the intensity of the user's voice increases. In addition, in some cases, as the intensity of the user's voice increases, the user's voice can gradually take on a more metallic texture. Refer to the following Fig. 12A-B describes in detail some example user interfaces depicting examples of inserting animated graphic elements into video items.

[0062] Reference now Figure 8 , describes an example process 800 in which movement of a visual element is caused based on a change in at least one of the characteristics of a signal within an audio input. Note that, Figure 8 Operations 810 and 812 are related to Figure 4 Operations 410 and 412 are identical. Therefore, the descriptions of operations 410 and 412 may be considered applicable to operations 810 and 812, respectively, and these descriptions are not repeated here. Figure 8 At operation 814, the visual element is moved based on a change in at least one of the characteristics of the signal within the audio input. The movement of the visual element can be caused in real time after at least one of the characteristics of the signal within the audio input changes. For example, in some cases, the visual element can be an object in a game, such as a character controlled by a user. In some cases, a change in the pitch or other voice characteristics of the user's voice can control the movement of the visual element 164. For example, the character can move up as the pitch of the user's voice increases, and the character can move down as the pitch of the user's voice decreases. As another example, the character can move up as the volume of the user's voice increases, and the character can move down as the volume of the user's voice decreases. Refer to the following Fig.13A -B describes in detail some example user interfaces depicting example movements of a visual element based on a change in at least one of the characteristics of a signal within an audio input.

[0063] Fig. 9 An example process 900 for modifying audio elements based on video input according to the present disclosure is shown. At operation 910, during creation of a video item, a video input is received via a camera, where the video input corresponds to a visual element associated with the video item. Figure 2 As described, during creation of video item 154, video input 114 may be received via camera 107. Video input 114 may include or may otherwise correspond to an element of a first type, such as visual element 164. In some examples, visual element 164 may include a body part of user 101, such as the user's eyes, eyebrows, mouth, lips, nose, hands, arms, etc.

[0064] At operation 912, characteristics of a signal in the video input are determined in real time based on the video input. The characteristics may change over time in the video input. In some examples, the characteristics may include movement of at least one body part within the video input. As described above, the video analysis component 124 may perform video analysis on the video input 114 to determine video signal characteristics 134. The video signal characteristics 134 may include calculations of movement of at least one body part within the video input 114. In some examples, the video analysis component 124 may employ one or more machine learning models (such as one or more neural network models) to perform video analysis on the video input 114. In some examples, the video analysis may include performing object detection and / or object recognition analysis on different frames of the video input. The object detection and / or object recognition analysis may include facial detection and / or facial recognition analysis. In some examples, the video analysis may include determining video signal characteristics 134 at different times, such as object locations and determining and tracking changes in those locations over time.

[0065] At operation 914, at least one modification is made to an audio element in an audio output associated with the video item based at least in part on a characteristic of the signal in the video input. Upon determining the characteristic of the signal in the video input that triggers the at least one modification, the at least one modification may be caused and performed in real time. Figure 2 As described, the modification component 140 can modify the audio element 165 in the audio output 155 based on the video signal characteristics 134 and the video-based modification instructions 144. For example, the video-based modification instructions 144 can indicate various conditions for modifying the audio element 165 within the audio output 155. Specifically, the video-based modification instructions 144 can indicate changes in the video signal characteristics 134 that can trigger the modification of the audio element 165. In addition to specifying the triggering condition, the video-based modification instructions 144 can also specify the modification caused by the triggering condition. In a specific example, operation 914 can include modifying a sound feature in the audio output based on the movement of at least one body part within the video input, wherein the sound feature includes at least one of pitch, tone, volume, energy, echo, or duration. Refer to the following Fig.10 This example is described in detail. In some examples, the sound characteristics of the user's voice, such as pitch, tone, volume, energy, echo, or duration, can be modified by modifying the audio input 115 to create a desired sound effect. For example, in some cases, the audio analysis component 125 can analyze the audio input 115 to detect data corresponding to the user's voice within the audio input, such as by filtering out background sounds, noise, and other non-speech audio data. The modification component 140 can then modify the remaining data corresponding to the user's voice in the audio input 115 to achieve the desired effect.

[0066] Reference now Fig.10 , describes an example process 1000 in which a visual element is stretched in at least one direction based on a change in at least one characteristic of a signal within an audio input. Note that, Fig.10 Operation 1010 and Fig. 9 Therefore, the description of operation 910 can be considered to be applicable to operation 1010, and this description is not repeated here. Fig.10 At operation 1012 of , characteristics of a signal in the video input are determined in real time based on the video input, wherein the characteristics include movement of at least one body part within the video input. As described above, the video analysis component 124 can perform video analysis on the video input 114 to determine the video signal characteristics 134. The video signal characteristics 134 may include calculations of movement of at least one body part within the video input 114. In some examples, the video analysis component 124 may perform video analysis on the video input 114 using one or more machine learning models, such as one or more neural network models. In some cases, the video analysis may include performing object detection and / or object recognition analysis on different frames of the video input. The object detection and / or object recognition analysis may include facial detection and / or facial recognition analysis. In some examples, the video analysis may include determining video signal characteristics 134 at different times, such as object locations and determining and tracking changes in those locations over time.

[0067] At operation 1014, the sound features in the audio output are modified based on the movement of at least one body part in the video input, wherein the sound features include at least one of pitch, tone, volume, energy, echo or duration. After the movement of at least one body part in the video input, the sound features can be modified in real time. In a specific example, the user 101 can move his or her eyebrows to change the pitch of the user's voice. For example, the eyebrows can be moved upward to improve the pitch of the user's voice, and the eyebrows can be moved downward to reduce the pitch of the user's voice. In another specific example, the movement of a body part or other object can cause an echo effect. For example, a body part can be raised upward to increase the echo effect, such as transforming the user's voice in real time to achieve an angelic echo. The body part can also be lowered to reduce the echo effect.

[0068] Some example user interfaces depicting examples of the above effects will now be described. In particular, Fig.11A -B depicts examples related to stretching and dithering visual elements based on a change in at least one of the characteristics of a signal within an audio input. Specifically, Fig.11AThe user interface 1100 is shown before the stretching and shaking effects are performed. As shown, the user interface 1100 shows a frame 1120 of a video item including a user's face 1103. The face 1103 includes a mouth 1102 and a chin 1104. The user is wearing glasses 1101. Now refer to Fig. 11B , shows a user interface 1110 including a frame 1121 of the same video item, in which a stretch effect and a dithering effect are applied. As shown in frame 1121, face 1103 has been stretched horizontally. For example, face 1103 is wider in frame 1121 than in frame 1120. In particular, glasses 1101, mouth 1102, and chin 1104 (as well as other facial features) are not as wide as in frame 1121. Fig. 11B 1121 are stretched more than they are in Fig.11A 1120. In addition, a shaking effect is applied in frame 1121. For example, in frame 1121, glasses 1101 and chin 1104 appear wavy, such as some edges have an up-and-down pattern instead of being straight or curved. When applied to multiple frames in a video, this wavy appearance can produce a shaking effect in which the object appears to be shaking.

[0069] The stretching and jittering effects in frame 1121 may be applied based on a change in at least one of the characteristics of the signal within the audio input. For example, in some cases, visual elements (such as face 1103, glasses 1101, mouth 1102, and / or chin 1104) may be stretched in one or more directions based on changes in the user's voice (such as changes in pitch, tone, volume, energy, etc.). In a specific example, the visual element may be stretched in one or more directions based on a change in the pitch of the user's voice. For example, in some cases, the amount of stretching of the user's face may increase as the pitch of the user's voice increases, and the amount of stretching of the user's face may decrease as the pitch of the user's voice decreases. Therefore, when generating frame 1121, the pitch of the user's voice may be higher than the pitch of the user's voice when generating frame 1120. The increase in pitch may cause the stretching and jittering effects to be applied to frame 1121.

[0070] Fig. 12A -B depicts an example of inserting at least one animated graphic element into a video item based on a change in at least one of the characteristics of a signal within an audio input. Specifically, Fig. 12A A user interface 1200 is shown showing a frame 1220 of a video item including a user's face 1203. The face 1203 includes eyes 1201A-B. The user is positioned in front of a background 1202. Referring now to Fig. 12B, a user interface 1210 including a frame 1221 of the same video item is shown, wherein an animated graphic object insertion is performed. As shown in frame 1221, fireballs 1211A-B are inserted into the positions of eyes 1201A-B in frame 1221, for example to convey a feeling of anger or fury. Therefore, in FIG. 12, the user's eyes 1201A-B are blocked by fireballs 1211A-B. In some examples, the user's eyes 1201A-B may only be partially blocked by fireballs 1211A-B, and therefore may be partially visible. Thus, fireballs 1211A-B modify visual elements (e.g., eyes 1201A-B) by blocking sight and changing appearance. In addition, background 1202 is blocked by background animation 1212. Background animation 1212 includes animated rays emitted from the user's face 1203 to convey a feeling of anger or fury. In some examples, the emission of animated rays or spikes may be synchronized with the user's voice and may increase as the intensity of the user's voice increases. After at least one of the characteristics of the signal within the audio input changes, both the fireballs 1211A-B and the background animation 1212 can be inserted into the video item in real time. In some examples, increasing the intensity (e.g., volume, energy, etc.) of the user's voice can cause animated graphic elements (such as the fireballs 1211A-B and the background animation 1212) to be inserted into the video item. In addition, decreasing the intensity (e.g., volume, energy, etc.) of the user's voice can cause animated graphic elements (such as the fireballs 1211A-B and the background animation 1212) to cease being inserted into the video item.

[0071] Fig.13A -B depicts an example of moving a visual element based on a change in at least one of the characteristics of a signal within an audio input. Specifically, Fig.13A A user interface 1300 is shown showing a frame 1320 of a video item. In this example, the video item is a video game in which a user controls the movement of a character 1301 in the video game. The character 1301 is a movable sea creature that includes an image of the user's face. The game includes sea corals 1302 that move on the screen from right to left, and the user can score points in the game by causing the character 1301 to interact with (e.g., touch) the sea corals 1302 that move on the screen. Referring now to FIG. Fig. 13B, user interface 1310 including frame 1321 of the same video item is shown. As shown in frame 1321, character 1301 has moved upward from its previous position in frame 1320. In this example, the movement of character 1301 is based on a change in at least one of the characteristics of the signal within the audio input. The movement of character 1301 can be made in real time after at least one of the characteristics of the signal within the audio input changes. In some cases, changes in the pitch or other voice characteristics of the user's voice can control the movement of character 1301. For example, the character can move upward as the pitch of the user's voice increases, and the character can move downward as the pitch of the user's voice decreases. As another example, the character can move upward as the volume of the user's voice increases, and the character can move downward as the volume of the user's voice decreases.

[0072] Fig.14 The diagram shows a method that can be used in various aspects (such as Figure 1 ) of computing devices that are not limited to the services, networks, modules, and / or devices depicted in the . Figure 1 The example architecture of the message service, interface service, processing service, content service, cloud network and client can be Fig.14 The invention may be implemented by one or more instances of computing device 1400. Fig.14 The computer architecture shown in the figure illustrates a conventional server computer, workstation, desktop computer, laptop computer, tablet computer, network appliance, PDA, e-reader, digital cellular telephone or other computing node, and can be used to perform any aspects of the computer described herein, such as for implementing the methods described herein.

[0073] The computing device 1400 may include a baseboard or "motherboard," which is a printed circuit board to which multiple components or devices may be connected via a system bus or other electrical communication path. One or more central processing units (CPUs) 1404 may operate in conjunction with a chipset 1406. The CPU(s) 1404 may be a standard programmable processor that performs the arithmetic and logic operations required for the operation of the computing device 1400.

[0074] The CPU 1404 can perform the necessary operations by manipulating switch elements that distinguish and change these states from one discrete physical state to the next. Switch elements can generally include electronic circuits that maintain one of two binary states, such as flip-flops, and electronic circuits that provide output states based on logical combinations of the states of one or more other switch elements, such as logic gates. These basic switch elements can be combined to create more complex logic circuits, including registers, adders-subtractors, arithmetic logic units, floating point units, etc.

[0075] CPU 1404 may be augmented or replaced with other processing units such as GPU(s) 1405. GPU(s) 1405 may include processing units specialized for, but not necessarily limited to, highly parallel computations such as graphics and other visualization-related processing.

[0076] Chipset 1406 may provide an interface between CPU(s) 1404 and the rest of the components and devices on the baseboard. Chipset 1406 may provide an interface to random access memory (RAM) 1408 used as main memory in computing device 1400. Chipset 1406 may also provide an interface to a computer-readable storage medium, such as read-only memory (ROM) 1420 or non-volatile RAM (NVRAM) (not shown), for storing basic routines that may help boot computing device 1400 and transfer information between various components and devices. ROM 1420 or NVRAM may also store other software components required for operation of computing device 1400 in accordance with various aspects described herein.

[0077] The computing device 1400 can operate in a networked environment using logical connections to remote computing nodes and computer systems through a local area network (LAN). Chipset 1406 may include functionality for providing network connectivity through a network interface controller (NIC) 1422, such as a Gigabit Ethernet adapter. NIC 1422 may be capable of connecting computing device 1400 to other computing nodes through network 1416. It should be appreciated that multiple NICs 1422 may be present in computing device 1400, connecting the computing device to other types of networks and remote computer systems.

[0078] The computing device 1400 can be connected to a mass storage device 1428 that provides non-volatile storage for the computer. The mass storage device 1428 can store system programs, application programs, other program modules, and data, which have been described in more detail herein. The mass storage device 1428 can be connected to the computing device 1400 via a storage controller 1424 connected to the chipset 1406. The mass storage device 1428 can be composed of one or more physical storage units. The mass storage device 1428 can include a management component 1410. The storage controller 1424 can interface with the physical storage unit via a serial attached SCSI (SAS) interface, a serial advanced technology connection (SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transmitting data between the computer and the physical storage unit.

[0079] The computing device 1400 can store data on the mass storage device 1428 by transforming the physical state of the physical storage unit to reflect the information being stored. The specific transformation of the physical state may depend on various factors and different implementations of the present description. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage unit and whether the mass storage device 1428 is characterized as a primary storage device or an auxiliary storage device, etc.

[0080] For example, the computing device 1400 can store information to the mass storage device 1428 by issuing instructions via the storage controller 1424 to change the magnetic properties of specific locations within the disk drive unit, the reflective or refractive properties of specific locations in the optical storage unit, or the electrical properties of specific capacitors, transistors, or other discrete components in the solid-state storage unit. Other transformations of the physical media are possible without departing from the scope and spirit of the present description, and the foregoing examples are provided only for the convenience of the present description. The computing device 1400 can also read information from the mass storage device 1428 by detecting the physical state or properties of one or more specific locations within the physical storage unit.

[0081] In addition to the mass storage device 1428 described above, the computing device 1400 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be appreciated by those skilled in the art that computer-readable storage media may be any available media that provides storage of non-transitory data and can be accessed by the computing device 1400.

[0082] As an example and not limitation, computer-readable storage media may include volatile and non-volatile, transient computer-readable storage media and non-transient computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid-state storage technology, compact disc ROM ("CD-ROM"), digital versatile disk ("DVD"), high-definition DVD ("HD-DVD"), BLU-RAY or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0083] Such as Fig.14A mass storage device such as the mass storage device 1428 depicted in the figure can store an operating system for controlling the operation of the computing device 1400. The operating system can include a version of the LINUX operating system. The operating system can include a version of the WINDOWS SERVER operating system from MICROSOFT. According to another aspect, the operating system can include a version of the UNIX operating system. Various mobile phone operating systems such as IOS and ANDROID can also be utilized. It should be appreciated that other operating systems can also be utilized. The mass storage device 1428 can store other systems or applications and data used by the computing device 1400.

[0084] The mass storage device 1428 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into the computing device 1400, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. These computer-executable instructions transform the computing device 1400 by specifying how the CPU(s) 1404 transition between states, as described above. The computing device 1400 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computing device 1400, may perform the methods described herein.

[0085] Such as Fig.14 A computing device such as the computing device 1400 depicted in FIG. 14 may also include an input / output controller 1432 for receiving and processing input from a number of input devices such as a keyboard, a mouse, a touch pad, a touch screen, an electronic stylus, or other types of input devices. Similarly, the input / output controller 1432 may provide output to a display such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other types of output devices. It will be appreciated that the computing device 1400 may not include Fig.14 All components shown in the Fig.14 Other components not explicitly shown in the figure or may be used in conjunction with Fig.14 The architecture shown in is completely different from the architecture shown in .

[0086] As described herein, a computing device may be a physical computing device such as Fig.14 The computing device 1400 of the present invention may also include a virtual machine host process and one or more virtual machine instances. The computer executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0087] It should be understood that the methods and systems are not limited to specific methods, specific components or specific implementations. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0088] As used in this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when a value is expressed as an approximation by using the antecedent "about", it will be understood that the particular value forms another embodiment. It should also be understood that the endpoints of each range are significant both relative to the other endpoint and independently of the other endpoint.

[0089] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not.

[0090] Throughout the description and claims of this specification, the word "comprise" and variations of the word, such as "comprising" and "comprises", means "including but not limited to", and is not intended to exclude, for example, other components, integers or steps. "Exemplary" means "an example of..." and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is not used in a limiting sense, but for explanation purposes.

[0091] Components that can be used to perform the described methods and systems are described. When combinations, subsets, interactions, groupings, etc. of these components are described, it should be understood that, while specific references to each of the various individual and collective combinations and arrangements of these may not be explicitly described, for all methods and systems, each is specifically contemplated and described herein. This applies to all aspects of the application, including, but not limited to, the operations in the described methods. Therefore, if there are multiple additional operations that can be performed, it should be understood that each of these additional operations can be performed using any specific embodiment or combination of embodiments of the described methods.

[0092] The present method and system may be understood more readily by reference to the following detailed description of the preferred embodiment and examples included therein and to the figures and descriptions thereof.

[0093] As will be appreciated by those skilled in the art, the method and system may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. In addition, the method and system may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) implemented in the storage medium. More particularly, the method and system may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be utilized, including a hard disk, a CD-ROM, an optical storage device, or a magnetic storage device.

[0094] Embodiments of the method and system are described below with reference to the block diagrams and flow charts of the method, system, device and computer program product. It should be understood that each box in the block diagram and flow chart and the combination of boxes in the block diagram and flow chart can be implemented by computer program instructions respectively. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create components for implementing the functions specified in one or more flow chart boxes.

[0095] These computer program instructions may also be stored in a computer-readable memory, which may direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device, thereby producing a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0096] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, certain methods or processing blocks can be omitted in some embodiments. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith can be performed in other appropriate orders. For example, the described blocks or states can be performed in an order different from the order specifically described, or multiple blocks or states can be combined in a single block or state. The example blocks or states can be performed serially, in parallel, or in some other way. Blocks or states can be added to or removed from the described example embodiments. The example systems and components described herein can be configured differently from those described. For example, compared to the described example embodiments, elements can be added, removed, or rearranged.

[0097] It should also be appreciated that various items are shown as being stored in memory or on storage devices when in use, and these items or portions thereof may be transferred between memory and other storage devices for the purpose of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially implemented or provided in firmware and / or hardware, including but not limited to one or more application specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium, such as a hard disk, memory, a network, or a portable media medium article read by an appropriate device or via an appropriate connection. Systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wire / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). In other embodiments, such computer program products may also take other forms. Thus, the present invention may be practiced with other computer system configurations.

[0098] Although the method and system have been described in conjunction with preferred embodiments and specific examples, it is not intended to limit the scope to the particular embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0099] Unless expressly stated otherwise, any method described herein is in no way intended to be interpreted as requiring that its operations be performed in a particular order. Thus, in the event that a method claim does not actually recite the order in which its operations are to be followed, or in the event that the operations are not otherwise specifically stated in the claim or description to be limited to a particular order, it is by no means intended that an order can be inferred from any aspect. This applies to any possible non-express basis for interpretation, including: matters of logic relative to the arrangement of steps or operational flows; plain meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0100] It will be apparent to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit of the present disclosure. Other embodiments will be apparent to those skilled in the art by considering the specification and practice described herein. The specification and example drawings are to be considered exemplary only, with the true scope and spirit being indicated by the appended claims.

Claims

1. A method comprising: receiving, via a first component of a computing device, a first type of input during creation of a video item, wherein the first type of input corresponds to a first type of element associated with the video item; determining, in real time, a characteristic of a signal in the first type of input based on the first type of input, wherein the characteristic changes over time in the first type of input; as well as At least one modification is made to an element of a second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type. 2 . The method of claim 1 , wherein the receiving, via the first component of the computing device, the first type of input comprises receiving audio input via a microphone, wherein the second type of element comprises a visual element in the video item.

3. The method of claim 2, wherein the characteristic comprises a sound feature, and wherein the sound feature comprises at least one of pitch, tone, volume, energy, or duration.

4. The method of claim 2, wherein said modifying the at least one element of the second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type comprises: The visual element is stretched in at least one direction based on a change in at least one of the characteristics of the signal within the audio input.

5. The method of claim 2, wherein said modifying the at least one element of the second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type comprises: The visual element is dithered based on a change in at least one of the characteristics of the signal within the audio input.

6. The method of claim 2, wherein said modifying said at least one element of said second type associated with said video item based at least in part on said characteristic of said signal in said first type of input comprises: At least one animated graphical element is inserted into the video item based on a change in at least one of the characteristics of the signal within the audio input.

7. The method of claim 2, wherein said modifying said at least one element of said second type associated with said video item based at least in part on said characteristic of said signal in said first type of input comprises: The movement of the visual element is caused based on a change in at least one of the characteristics of the signal within the audio input.

8. The method of claim 1, wherein the receiving, via the first component of the computing device, the first type of input comprises receiving a video input via a camera, and wherein the second type of element comprises an audio element in an audio output associated with the video item.

9. The method of claim 8, wherein the characteristic comprises movement of at least one body part within the video input.

10. The method of claim 9, wherein said modifying the at least one element of the second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type comprises: A sound characteristic in the audio output is modified based on the movement of the at least one body part within the video input, wherein the sound characteristic includes at least one of pitch, tone, volume, energy, echo, or duration.

11. A system comprising: one or more computer processors; as well as One or more computer memories including computer readable instructions that, when executed by one or more computer processors, configure the system to perform operations including: receiving, via a first component of a computing device, a first type of input during creation of a video item, wherein the first type of input corresponds to a first type of element associated with the video item; determining, in real time, a characteristic of a signal in the first type of input based on the first type of input, wherein the characteristic changes over time in the first type of input; as well as At least one modification is made to an element of a second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type.

12. The system of claim 11, wherein the receiving, via the first component of the computing device, the first type of input comprises receiving audio input via a microphone, wherein the second type of element comprises a visual element in the video item.

13. The system of claim 12, wherein the characteristic comprises a sound feature, and wherein the sound feature comprises at least one of pitch, tone, volume, energy, or duration.

14. The system of claim 12, wherein the modifying the at least one element of the second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type comprises at least one of: stretching the visual element in at least one direction based on a change in at least one of the characteristics of the signal within the audio input; inserting at least one animated graphical element into the video item based on a change in at least one of the characteristics of the signal within the audio input; or The visual element is dithered based on a change in at least one of the characteristics of the signal within the audio input.

15. The system of claim 11, wherein the receiving of the first type of input via the first component of the computing device comprises receiving video input via a camera, and wherein the second type of element comprises an audio element in an audio output associated with the video item.

16. The system of claim 15, wherein said modifying said at least one element of said second type associated with said video item based at least in part on said characteristic of said signal in said first type of input comprises: A sound characteristic in the audio output is modified based on movement of the at least one body part within the video input, wherein the sound characteristic comprises at least one of pitch, tone, volume, energy, echo, or duration.

17. A non-transitory computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, cause the processor to perform operations comprising: receiving, via a first component of a computing device, a first type of input during creation of a video item, wherein the first type of input corresponds to a first type of element associated with the video item; determining, in real time, a characteristic of a signal in the first type of input based on the first type of input, wherein the characteristic changes over time in the first type of input; as well as At least one modification is made to an element of a second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type.

18. The non-transitory computer-readable storage medium of claim 17, wherein the receiving, via the first component of the computing device, the first type of input comprises receiving audio input via a microphone, wherein the second type of element comprises a visual element in the video item.

19. The non-transitory computer-readable storage medium of claim 18, wherein the modifying the at least one element of the second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type comprises at least one of: stretching the visual element in at least one direction based on a change in at least one of the characteristics of the signal within the audio input; inserting at least one animated graphical element into the video item based on a change in at least one of the characteristics of the signal within the audio input; or The visual element is dithered based on a change in at least one of the characteristics of the signal within the audio input.

20. The non-transitory computer-readable storage medium of claim 17, wherein the receiving of the first type of input via the first component of the computing device comprises receiving video input via a camera, and wherein the second type of element comprises an audio element in an audio output associated with the video item.

21. The non-transitory computer-readable storage medium of claim 20, wherein modifying the at least one element of the second type associated with the video item based at least in part on the characteristic of the signal in the input of the first type comprises: A sound characteristic in the audio output is modified based on the movement of the at least one body part within the video input, wherein the sound characteristic includes at least one of pitch, tone, volume, energy, echo, or duration.

Images