Keyword-based object insertion in video stream

Abstract

An apparatus includes one or more processors configured to obtain an audio stream and detect one or more keywords in the audio stream. The one or more processors are also configured to adaptively classify one or more objects associated with the one or more keywords. The one or more processors are also configured to insert the one or more objects into the video stream.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of commonly owned priority to U.S. non-provisional patent application No. 17 / 933,425, filed on September 19, 2022, the contents of which are expressly incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to inserting one or more objects in a video stream based on one or more keywords. Background Art

[0004] Advances in technology have led to smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones, such as mobile and smart phones, tablets and laptop computers, which are small, lightweight and easy to carry by users. These devices can transmit voice and data packets over wireless networks. In addition, many such devices incorporate additional functionality, such as digital still cameras, digital video cameras, digital recorders and audio file players. In addition, such devices can process executable instructions, including software applications that can be used to access the Internet, such as web browser applications. Therefore, these devices can include significant computing capabilities.

[0005] Such computing devices typically include the functionality of receiving audio captured by a microphone and playing the audio via a speaker. The device typically also includes the functionality of displaying a video captured by a camera. In some examples, the device includes the functionality of receiving a media stream and playing the audio of the media stream via a speaker while displaying the video of the media stream. For live media streams displayed simultaneously with reception or capture, there is typically not enough time for a user to edit the video before display. Therefore, when presenting a live media stream, enhancements that might otherwise be performed to improve viewer retention, add relevant content, etc. are unavailable, which can result in a reduced viewer experience. Summary of the invention

[0006] According to one embodiment of the present disclosure, a device includes one or more processors configured to obtain an audio stream and detect one or more keywords in the audio stream. The one or more processors are also configured to adaptively classify one or more objects associated with the one or more keywords. The one or more processors are also configured to insert the one or more objects into a video stream.

[0007] According to another embodiment of the present disclosure, a method includes obtaining an audio stream at a device. The method also includes detecting one or more keywords in the audio stream at the device. The method also includes adaptively classifying one or more objects associated with the one or more keywords at the device. The method also includes inserting one or more objects into a video stream at the device.

[0008] According to another embodiment of the present disclosure, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to obtain an audio stream and detect one or more keywords in the audio stream. When executed by one or more processors, the instructions also cause the one or more processors to adaptively classify one or more objects associated with the one or more keywords. When executed by one or more processors, the instructions also cause the one or more processors to insert one or more objects into a video stream.

[0009] According to another embodiment of the present disclosure, a device includes a component for obtaining an audio stream. The device also includes a component for detecting one or more keywords in the audio stream. The device also includes a component for adaptively classifying one or more objects associated with the one or more keywords. The device also includes a component for inserting one or more objects into a video stream.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent after reviewing the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of certain illustrative aspects of a system operable to insert keyword-based objects into a video stream and illustrative examples of inserting keyword-based objects into a video stream in accordance with some examples of the present disclosure.

[0012] Figure 2 is a specific implementation of a method for inserting keyword-based objects into a video stream according to some examples of the present disclosure and can be performed by Figure 1 FIG. 1 is a diagram of an illustrative example of inserting keyword-based objects into a video stream performed by a device.

[0013] Figure 3 is a diagram of another specific implementation of a method for inserting keyword-based objects into a video stream according to some examples of the present disclosure and can be performed by Figure 1 FIG. 1 is a diagram of an illustrative example of inserting keyword-based objects into a video stream performed by a device.

[0014] Figure 4 According to some examples of the present disclosure Figure 1FIG. 1 is a diagram of illustrative aspects of an example of a keyword detection unit of a system.

[0015] Figure 5 is a diagram of illustrative aspects of operations associated with keyword detection according to some examples of the present disclosure.

[0016] Figure 6 is another specific implementation of the method for object generation according to some examples of the present disclosure and can be performed by Figure 1 FIG. 1 is a diagram of an illustrative example of object generation performed by a device.

[0017] Figure 7 According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of an example of one or more components of an object determination unit of a system.

[0018] Figure 8 is a diagram of illustrative aspects of operations associated with object classification according to some examples of the present disclosure.

[0019] Fig.9A According to some examples of the present disclosure Figure 1 A diagram of another illustrative aspect of the operation associated with an object classification neural network of a system.

[0020] Fig. 9B According to some examples of the present disclosure and by Figure 1 A diagram of illustrative aspects of operations associated with feature extraction performed by a neural network of a system for object classification.

[0021] Fig. 9C According to some examples of the present disclosure and by Figure 1 A diagram of illustrative aspects of the operations performed by a system's object classification neural network and associated classification and probability distribution.

[0022] Fig. 10A According to some examples of the present disclosure, Figure 1 A specific embodiment of a method for determining an insertion position performed by a device and an example diagram of determining an insertion position.

[0023] Fig. 10B According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operations performed by a system's positional neural network.

[0024] Fig.11 is a block diagram of illustrative aspects of a system operable to perform keyword-based object insertion into a video stream according to some examples of the present disclosure.

[0025] Fig.12is a block diagram of another illustrative aspect of a system operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure.

[0026] Fig.13 is a block diagram of another illustrative aspect of a system operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure.

[0027] Fig.14 is a block diagram of another illustrative aspect of a system operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure.

[0028] Fig.15 is a block diagram of another illustrative aspect of a system operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure.

[0029] Fig.16 According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operation of components of a system.

[0030] Fig.17 An example of an integrated circuit operable to insert keyword-based objects into a video stream in accordance with some examples of the present disclosure is shown.

[0031] Fig.18 is a diagram of a mobile device operable to perform keyword-based object insertion into a video stream according to some examples of the present disclosure.

[0032] Fig.19 is a diagram of a headset operable to perform keyword-based object insertion into a video stream according to some examples of the present disclosure.

[0033] Fig. 20 is a diagram of a wearable electronic device operable to perform keyword-based object insertion into a video stream according to some examples of the present disclosure.

[0034] Fig.21 is a diagram of a voice-controlled speaker system operable to insert keyword-based objects into a video stream according to some examples of the present disclosure.

[0035] Fig. 22 is a diagram of a camera operable to insert keyword-based objects into a video stream according to some examples of the present disclosure.

[0036] Fig.23 is a diagram of a headset, such as an extended reality headset, operable to perform keyword-based object insertion into a video stream, according to some examples of the present disclosure.

[0037] Fig.24 is a diagram of an extended reality eyewear device operable to insert keyword-based objects into a video stream according to some examples of the present disclosure.

[0038] Fig.25 is a diagram of a first example of a vehicle operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure.

[0039] Fig.26 is a diagram of a second example of a vehicle operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure.

[0040] Fig. 27 According to some examples of the present disclosure, Figure 1 FIG. 1 is a diagram of a specific implementation of a method for inserting keyword-based objects into a video stream performed by a device.

[0041] Fig.28 is a block diagram of a particular illustrative example of a device operable to perform keyword-based object insertion into a video stream in accordance with some examples of the present disclosure. DETAILED DESCRIPTION

[0042] Computing devices often include functionality to play back a media stream by providing an audio stream to speakers while displaying a video stream. For live media streams that are displayed simultaneously with reception or capture, there is often insufficient time for a user to perform enhancements prior to display to improve viewer retention, add relevant content to the video stream, etc.

[0043] Systems and methods for inserting keyword-based objects into a video stream are disclosed. For example, a video stream updater performs keyword detection in an audio stream to generate a keyword, and determines whether a database includes any objects associated with the keyword. In response to determining that the database includes an object associated with the keyword, the video stream updater inserts the object into the video stream. Alternatively, in response to determining that the database does not include any objects associated with the keyword, the video stream updater applies an object generation neural network to the keyword to generate an object associated with the keyword, and inserts the object into the video stream. Optionally, in some examples, the video stream updater designates the newly generated object as associated with the keyword and adds the object to the database. Thus, the video stream updater can enhance the video stream using pre-existing objects or newly generated objects associated with the keyword detected in the audio stream.

[0044] Enhancements can improve audience retention, add relevant content, and the like. For example, it can be a challenge to keep the audience interested during playback of a video stream of a person speaking at a podium. Adding objects to a video stream can make the video stream more interesting to the audience during playback. To illustrate, adding a background image showing the results of tree planting to a live media stream discussing climate change can increase audience retention for the live media stream. As another example, adding an image of a local restaurant to a video stream about traveling to an area with the same kinds of food served at the restaurant can entice viewers to visit the local restaurant or can lead to increased orders for the restaurant. In some examples, the video stream can be enhanced based on an audio stream obtained separately from the video stream. To illustrate, the video stream can be updated based on user voice included in an audio stream received from one or more microphones.

[0045] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In the specification, common features are represented by common reference numerals. As used herein, various terms are used only for the purpose of describing specific embodiments and are not intended to limit the embodiments. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some embodiments and plural in other embodiments. For illustration, Figure 1 Depicts a system including one or more processors ( Figure 1 102 ), indicating that in some embodiments, the device 130 includes a single processor 102 , and in other embodiments, the device 130 includes multiple processors 102 .

[0046] In some of the drawings, multiple instances of a particular type of feature are used. Although the features are physically and / or logically distinct, each feature uses the same reference numeral, and different instances are distinguished by adding a letter to the reference numeral. When referring to features as a group or type herein, for example, when no reference is made to a specific one of the features, the reference numeral is used without a distinguishing letter. However, when referring to a specific feature among multiple features of the same type, the reference numeral is used with a distinguishing letter. For example, referring to Figure 1 , multiple objects are shown and are associated with reference numerals 122A and 122B. When referring to a specific one of these objects, such as object 122A, a distinguishing letter "A" is used. However, when referring to any one of these objects or these objects as a group, reference numeral 122 is used without a distinguishing letter.

[0047] As used herein, the terms "include", "it includes", and "include..." may be used interchangeably with "contain", "it includes", or "include...". In addition, the term "wherein" may be used interchangeably with "wherein". As used herein, "exemplary" indicates an example, implementation, and / or aspect, and should not be interpreted as limiting or indicating a preference or preferred implementation. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) used to modify elements (such as structures, components, operations, etc.) do not themselves indicate any priority or order of the element relative to another element, but only distinguish the element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more of a particular element, and the term "plurality" refers to a plurality of (e.g., two or more) of a particular element.

[0048] As used herein, "coupling" may include "communicative coupling", "electrical coupling" or "physical coupling", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicative coupling, electrical coupling or physical coupling) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronic devices, one or more connectors or inductive coupling. In some embodiments, two devices (or components) that are communicatively coupled (such as electrically communicating) may directly or indirectly send and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled or physically coupled) without an intermediate component.

[0049] In the present disclosure, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that such terms should not be interpreted as limiting, and other technologies may be used to perform similar operations. In addition, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access", and "determine" may be used interchangeably. For example, "generate", "calculate", "estimate" or "determine" a parameter (or signal) may refer to actively generating, estimating, calculating or determining a parameter (or signal) or may refer to using, selecting or accessing a parameter (or signal) such as one that has been generated by another component or device.

[0050] refer to Figure 1, discloses certain illustrative aspects of a system 100. The system 100 is configured to insert keyword-based objects into a video stream. Figure 1 Also shown in FIG. 1 are examples 190 and 192 of inserting keyword-based objects into a video stream.

[0051] The system 100 includes a device 130 including one or more processors 102 coupled to a memory 132 and a database 150. The one or more processors 102 include a video stream updater 110 configured to perform keyword-based object insertion in a video stream 136, and the memory 132 is configured to store instructions 109 executable by the one or more processors 102 to implement the functionality described with reference to the video stream updater 110.

[0052] The video stream updater 110 includes a keyword detection unit 112 coupled to an object insertion unit 116 via an object determination unit 114. Optionally, in some embodiments, the video stream updater 110 further includes a position determination unit 170 coupled to the object insertion unit 116.

[0053] The device 130 also includes a database 150 accessible to one or more processors 102. However, in other aspects, the database 150 can be external to the device 130, such as stored in a storage device, a network device, a cloud-based storage, or a combination thereof. The database 150 is configured to store a collection of objects 122, such as object 122A, object 122B, one or more additional objects, or a combination thereof. As an illustrative, non-limiting example, an "object" as used herein refers to a visual digital element, such as an image, clip art, a photo, a drawing, a graphics interchange format (GIF) file, a portable network graphic (PNG) file, or a video clip. An "object" is primarily or entirely image-based and is therefore different from a text-based addition, such as a subtitle.

[0054] In some implementations, the database 150 is configured to store object keyword data 124 that indicates one or more keywords 120, if any, associated with one or more objects 122. In a particular example, the object keyword data 124 indicates that an object 122A (e.g., an image of the Statue of Liberty) is associated with one or more keywords 120A (e.g., "New York" and "Statue of Liberty"). In another example, the object keyword data 124 indicates that an object 122B (e.g., clip art representing a clock) is associated with one or more keywords 120B (e.g., "clock," "alarm," "time").

[0055] The video stream updater 110 is configured to process the audio stream 134 to detect one or more keywords 180 in the audio stream 134 and insert objects associated with the detected keywords 180 into the video stream 136. In some examples, a media stream (e.g., a live media stream) includes an audio stream 134 and a video stream 136, as shown in FIG. Fig.11 Optionally, in some examples, at least one of the audio stream 134 or the video stream 136 corresponds to decoded data generated by a decoder by decoding encoded data received from another device, as described in reference to Fig.12 Optionally, in some examples, the video stream updater 110 is configured to receive the audio stream 134 from one or more microphones coupled to the device 130, as described in reference Fig.13 Optionally, in some examples, the video stream updater 110 is configured to receive the video stream 136 from one or more cameras coupled to the device 130, as shown in FIG. Fig.14 Optionally, in some embodiments, audio stream 134 is obtained separately from video stream 136. For example, audio stream 134 is received from one or more microphones coupled to device 130, and video stream 136 is received from another device or generated at device 130, as described at least in reference to Fig.13 , Fig.23 and Fig.26 Further described.

[0056] For illustration, the keyword detection unit 112 is configured to determine one or more detected keywords 180 in at least a portion of the audio stream 134, as shown in FIG. Figure 5 As further described herein, a "keyword" as used herein may refer to a single word or a phrase including multiple words. In some embodiments, the keyword detection unit 112 is configured to apply a keyword detection neural network 160 to at least a portion of the audio stream 134 to generate one or more detected keywords 180, as shown in FIG. Figure 4 Further described.

[0057] The object determination unit 114 is configured to determine (e.g., select or generate) one or more objects 182 associated with the one or more detected keywords 180. The object determination unit 114 is configured to select one or more of the objects 122 stored in the database 150 that are indicated by the object keyword data 124 as being associated with the one or more detected keywords 180 to be included in the one or more objects 182. In particular aspects, the selected objects correspond to pre-existing and pre-classified objects associated with the one or more detected keywords 180.

[0058] The object determination unit 114 includes an adaptive classifier 144 configured to adaptively classify one or more objects 182 associated with the one or more detected keywords 180. Classifying the object 182 includes generating the object 182 based on the one or more detected keywords 180 (e.g., a newly generated object), performing classification of the object 182 to designate the object 182 as being associated with the one or more keywords 120 (e.g., the newly classified object), and determining whether any of the keyword(s) 120 matches any of the keyword(s) 180, or both. In some aspects, the adaptive classifier 144 is configured to refrain from classifying the object 182 in response to determining that a pre-existing and pre-classified object is associated with at least one of the one or more detected keywords 180. Alternatively, the adaptive classifier 144 is configured to classify (e.g., generate, perform classification, or both) the object 182 in response to determining that none of the pre-existing objects are indicated by the object keyword data 124 as being associated with any of the one or more detected keywords 180.

[0059] In some aspects, the adaptive classifier 144 includes an object generation neural network 140, an object classification neural network 142, or both. The object generation neural network 140 is configured to generate an object 122 (e.g., a newly generated object) associated with one or more objects 182. For example, the object generation neural network 140 is configured to process one or more detected keywords 180 (e.g., "alarm clock") to generate one or more objects 122 (e.g., clip art of a clock) associated with the one or more detected keywords 180, as shown in FIG. Figure 6 and Figure 7 As further described. The adaptive classifier 144 is configured to add the one or more objects 122 (e.g., newly generated objects) to the one or more objects 182 associated with the one or more detected keywords 180. In certain aspects, the adaptive classifier 144 is configured to update the object keyword data 124 to indicate that the one or more objects 122 (e.g., newly generated objects) are associated with the one or more keywords 120 (e.g., one or more detected keywords 180).

[0060] The object classification neural network 142 is configured to classify objects 122 (e.g., pre-existing objects) stored in the database 150. For example, the object classification neural network 142 is configured to process an object 122A (e.g., an image of the Statue of Liberty) to generate one or more keywords 120A (e.g., "New York" and "Statue of Liberty") associated with the object 122A, as shown in FIG. Figure 9A-9CAs further described. As another example, the object classification neural network 142 is configured to process the object 122B (e.g., a clip art of a clock) to generate one or more keywords 120B (e.g., "clock," "alarm," and "time"). The adaptive classifier 144 is configured to update the object keyword data 124 to indicate that the object 122A (e.g., an image of the Statue of Liberty) and the object 122B (e.g., the clip art of a clock) are associated with the one or more keywords 120A (e.g., "New York" and "Statue of Liberty") and the one or more keywords 120B (e.g., "clock," "alarm," and "time"), respectively.

[0061] The adaptive classifier 144 is configured to determine whether the set of objects 122 includes at least one object 122 associated with the one or more detected keywords 180 after generating (e.g., updating) the one or more keywords 120 associated with the set of objects 122. The adaptive classifier 144 is configured to add the object 122A (e.g., a newly classified object) to the one or more objects 182 associated with the one or more detected keywords 180 in response to determining that at least one of the one or more keywords 120A (e.g., "New York" and "Statue of Liberty") matches at least one of the one or more detected keywords 180 (e.g., "New York City").

[0062] In some aspects, in response to determining that object keyword data 124 indicates that object 122 is associated with at least one keyword 120 that matches at least one of the one or more detected keywords 180 , adaptive classifier 144 determines that object 122 is associated with one or more detected keywords 180 .

[0063] In some embodiments, the adaptive classifier 144 is configured to determine that the keyword 120 matches the detected keyword 180 in response to determining that the keyword 120 is the same as the detected keyword 180 or that the keyword 120 is a synonym of the detected keyword 180. Alternatively, in some embodiments, the adaptive classifier 144 is configured to generate a first vector representing the keyword 120 and generate a second vector representing the detected keyword 180. In these embodiments, the adaptive classifier 144 is configured to determine that the keyword 120 matches the detected keyword 180 in response to determining that the vector distance between the first vector and the second vector is less than a distance threshold.

[0064] The adaptive classifier 144 is configured to adaptively classify the one or more objects 182 associated with the one or more detected keywords 180. For example, in certain embodiments, the adaptive classifier 144 is configured to refrain from classifying the one or more objects 182 in response to selecting one or more of the objects 122 (e.g., pre-existing and pre-classified objects) stored in the database 150 for inclusion in the one or more objects 182. Alternatively, the adaptive classifier 144 is configured to classify the one or more objects 182 associated with the one or more detected keywords 180 in response to determining that no objects 122 (e.g., pre-existing and pre-classified objects) are associated with the one or more detected keywords 180.

[0065] In some examples, classifying one or more objects 182 includes generating, using object generation neural network 140, at least one of one or more objects 182 (e.g., a newly generated object) associated with at least one of the one or more detected keywords 180. In some examples, classifying one or more objects 182 includes designating, using object classification neural network 142, one or more of objects 122 (e.g., the newly classified object) as being associated with one or more keywords 120, and adding at least one of objects 122 having a keyword 120 that matches the at least one detected keyword 180 to one or more objects 182.

[0066] Optionally, in some examples, the adaptive classifier 144 uses the object generation neural network 140 and does not use the object classification neural network 142 to classify the one or more objects 182. For illustration, in these examples, the adaptive classifier 144 includes the object generation neural network 140, and the object classification neural network 142 may be disabled or optionally omitted from the adaptive classifier 144.

[0067] Optionally, in some examples, the adaptive classifier 144 uses the object classification neural network 142 and does not use the object generation neural network 140 to classify the one or more objects 182. For illustration, in these examples, the adaptive classifier 144 includes the object classification neural network 142, and the object generation neural network 140 may be disabled or optionally omitted from the adaptive classifier 144.

[0068] Optionally, in some examples, the adaptive classifier 144 uses the object generation neural network 140 and uses the object classification neural network 142 to classify the one or more objects 182. For illustration, in these examples, the adaptive classifier 144 includes the object generation neural network 140 and the object classification neural network 142.

[0069] Optionally, in some examples, adaptive classifier 144 uses object generation neural network 140 in response to determining that use of object classification neural network 142 has not resulted in any object 122 being classified as being associated with one or more detected keywords 180. To illustrate, in these examples, object generation neural network 140 is adaptively used based on the results of use of object classification neural network 142.

[0070] The adaptive classifier 144 is configured to provide one or more objects 182 associated with the one or more detected keywords 180 to the object insertion unit 116. The one or more objects 182 include one or more pre-existing and pre-classified objects selected by the adaptive classifier 144, one or more objects newly generated by the object generation neural network 140, one or more objects newly classified by the object classification neural network 142, or a combination thereof. Optionally, in some embodiments, the adaptive classifier 144 is further configured to provide the one or more objects 182 (or at least type information of the one or more objects 182) to the position determination unit 170.

[0071] Optionally, in some embodiments, the position determination unit 170 is configured to determine one or more insertion positions 164 and provide the one or more insertion positions 164 to the object insertion unit 116. In some embodiments, the position determination unit 170 is configured to determine the one or more insertion positions 164 based at least in part on the object type of the one or more objects 182, as described in reference to Figure 2 to Figure 3 In some embodiments, the position determination unit 170 is configured to apply the position neural network 162 to at least a portion of the video stream 136 to determine one or more insertion locations 164, as further described with reference to FIG.

[0072] In certain aspects, the insertion locations 164 correspond to specific locations in an image frame of the video stream 136 (e.g., background, foreground, top, bottom, specific coordinates, etc.) or specific content in an image frame of the video stream 136 (e.g., a shirt, a picture frame, etc.). For example, during live media processing, the one or more insertion locations 164 may indicate a location (e.g., foreground), content (e.g., a shirt), or both (e.g., a shirt in the foreground) within each of one or more specific frames of the video stream 136 that are presented substantially simultaneously with the corresponding detected keyword 180 being played out. In some aspects, the one or more specific image frames are time-aligned with one or more audio frames of the audio stream 134 that are processed to determine the one or more detected keywords 180, as referenced in the example of FIG. Fig.16 Further described.

[0073] In some embodiments where there is no position determination unit 170 to determine the one or more insertion positions 164, the one or more insertion positions 164 correspond to one or more predetermined insertion positions that can be used by the object insertion unit 116. Non-limiting illustrative examples of predetermined insertion positions include background, bottom right, bottom scroll, or a combination thereof. In certain aspects, the one or more predetermined positions are based on default data, configuration settings, user input, or a combination thereof.

[0074] The object insertion unit 116 is configured to insert the one or more objects 182 at the one or more insertion locations 164 in the video stream 136. In some examples, the object insertion unit 116 is configured to perform round-robin insertion of the one or more objects 182 if the one or more objects 182 include multiple objects to be inserted at the same insertion location 164. For example, the object insertion unit 116 performs round-robin insertion of a first subset (e.g., multiple images) of the one or more objects 182 at a first insertion location 164 (e.g., background), performs round-robin insertion of a second subset (e.g., multiple clip art, GIF files, etc.) of the one or more objects 182 at a second insertion location 164 (e.g., shirt), and so on. In other examples, the object insertion unit 116 is configured to insert the object 122A of the one or more objects 182 at a first insertion location (e.g., background) of the one or more insertion locations 164, insert the object 122B of the one or more objects 182 at a second insertion location (e.g., bottom right), and so on in response to determining that the one or more objects 182 include multiple objects and the one or more insertion locations 164 include multiple locations. The object insertion unit 116 is configured to output the video stream 136 (with the one or more objects 182 inserted).

[0075] In some implementations, device 130 corresponds to or is included in one of various types of devices. In the illustrative example, one or more processors 102 are integrated into a headphone device, such as a headset. Fig.19 In other examples, one or more processors 102 are integrated in a Fig.18 The mobile phone or tablet computer device, as referenced Fig. 20 The wearable electronic device, as shown in reference Fig.21 The voice controlled speaker system, as shown in reference Fig. 22 The camera device, as referenced Fig.23 The extended reality (XR) headset or as referenced Fig.24 In at least one of the XR glasses devices described above. In another illustrative example, one or more processors 102 are integrated into a vehicle, such as reference Fig.25 and Fig.26Further described.

[0076] During operation, the video stream updater 110 obtains an audio stream 134 and a video stream 136. In certain aspects, the audio stream 134 is a live stream that the video stream updater 110 receives in real time from a microphone, a network device, another device, or a combination thereof. In certain aspects, the video stream 136 is a live stream that the video stream updater 110 receives in real time from a camera, a network device, another device, or a combination thereof.

[0077] Optionally, in some embodiments, the media stream (eg, live media stream) includes an audio stream 134 and a video stream 136, as shown in FIG. Fig.11 Optionally, in some embodiments, at least one of the audio stream 134 or the video stream 136 corresponds to decoded data generated by a decoder by decoding encoded data received from another device, as described in reference to Fig.12 Optionally, in some embodiments, the video stream updater 110 receives the audio stream 134 from one or more microphones coupled to the device 130, as described in reference Fig.13 Optionally, in some embodiments, the video stream updater 110 receives the video stream 136 from one or more cameras coupled to the device 130, as shown in FIG. Fig.14 Further described.

[0078] The keyword detection unit 112 processes the audio stream 134 to determine one or more detected keywords 180 in the audio stream 134. In some examples, the keyword detection unit 112 processes a predetermined count of audio frames of the audio stream 134, audio frames of the audio stream 134 corresponding to a predetermined playback time, or both. In particular aspects, the predetermined count of audio frames, the predetermined playback time, or both are based on default data, configuration settings, user input, or a combination thereof.

[0079] In some embodiments, the keyword detection unit 112 is omitted (or not used), and instead uses speech recognition technology to determine one or more words represented in the audio stream 134, and uses semantic analysis technology to process the one or more words to determine one or more detected keywords 180. Optionally, in some embodiments, the keyword detection unit 112 applies the keyword detection neural network 160 to process one or more audio frames of the audio stream 134 to determine (e.g., detect) one or more detected keywords 180 in the audio stream 134, as described in reference to FIG. Figure 4Further described. In some aspects, applying the keyword detection neural network 160 includes extracting acoustic features of one or more audio frames to generate input values, and processing the input values ​​using the keyword detection neural network 160 to determine one or more detected keywords 180 corresponding to the acoustic features. Technical effects of applying the keyword detection neural network 160 can include using fewer resources (e.g., time, computing cycles, memory, or a combination thereof) and improving the accuracy of determining one or more detected keywords 180 compared to using speech recognition and semantic analysis.

[0080] In the example, the adaptive classifier 144 first performs a database search or lookup operation based on a comparison of the one or more database keywords 120 to the one or more detected keywords 180 to determine whether the set of objects 122 includes any objects associated with the one or more detected keywords 180. In response to determining that the set of objects 122 includes at least one object 122 associated with the one or more detected keywords 180, the adaptive classifier 144 refrains from classifying the one or more objects 182 associated with the one or more detected keywords 180.

[0081] In example 190, the keyword detection unit 112 determines one or more detected keywords 180 (e.g., “New York City”) in the audio stream 134 associated with the video stream 136A. In response to determining that the set of objects 122 includes an object 122A (e.g., an image of the Statue of Liberty) associated with the one or more keywords 120A (e.g., “New York” and “Statue of Liberty”) and determining that at least one of the one or more keywords 120A matches at least one of the one or more detected keywords 180 (e.g., “New York City”), the adaptive classifier 144 determines that the object 122A is associated with the one or more detected keywords 180. In response to determining that the object 122A is associated with the one or more detected keywords 180, the adaptive classifier 144 includes the object 122A in the one or more objects 182 and refrains from classifying the one or more objects 182 associated with the one or more detected keywords 180.

[0082] In example 192, the keyword detection unit 112 determines one or more detected keywords 180 (e.g., “alarm”) in the audio stream 134 associated with the video stream 136A. The keyword detection unit 112 provides the one or more detected keywords 180 to the adaptive classifier 144. In response to determining that the set of objects 122 includes an object 122B (e.g., a clip art of a clock) associated with one or more keywords 120B (e.g., “clock,” “alarm,” and “time”) and determining that at least one of the one or more keywords 120B matches at least one of the one or more detected keywords 180 (e.g., “alarm”), the adaptive classifier 144 determines that the object 122B is associated with the one or more detected keywords 180. In response to determining that the object 122B is associated with the one or more detected keywords 180, the adaptive classifier 144 includes the object 122B in the one or more objects 182 and refrains from classifying the one or more objects 182 associated with the one or more detected keywords 180.

[0083] In an alternative example where a database search or lookup operation does not detect any objects associated with one or more detected keywords 180 (e.g., “New York City” in Example 190 or “alarm clock” in Example 192), the adaptive classifier 144 classifies one or more objects 182 associated with the one or more detected keywords 180.

[0084] Optionally, in some aspects, classifying the one or more objects 182 includes using the object classification neural network 142 to determine whether any object in the set of objects 122 can be classified as being associated with the one or more detected keywords 180, as described in reference to Figure 9A-9C For example, using the object classification neural network 142 may include performing feature extraction of an object 122 in the set of objects 122 to determine an input value representing the object 122, performing classification based on the input value to determine one or more potential keywords that may be associated with the object 122, and generating a probability distribution indicating a likelihood that each of the one or more potential keywords is associated with the object 122. The adaptive classifier 144 designates the one or more potential keywords as the one or more keywords 120 associated with the object 122 based on the probability distribution. The adaptive classifier 144 updates the object keyword data 124 to indicate that the object 122 is associated with the one or more keywords 120 generated by the object classification neural network 142.

[0085] As an example, the adaptive classifier 144 processes the object 122A (e.g., an image of the Statue of Liberty) using the object classification neural network 142 to generate one or more keywords 120A (e.g., "New York" and "Statue of Liberty") associated with the object 122A. The adaptive classifier 144 updates the object keyword data 124 to indicate that the object 122A (e.g., the image of the Statue of Liberty) is associated with the one or more keywords 120A (e.g., "New York" and "Statue of Liberty"). As another example, the adaptive classifier 144 processes the object 122B (e.g., a clip art of a clock) using the object classification neural network 142 to generate one or more keywords 120B (e.g., "clock," "alarm," and "time") associated with the object 122B. The adaptive classifier 144 updates the object keyword data 124 to indicate that the object 122B (e.g., the clip art of a clock) is associated with the one or more keywords 120B (e.g., "clock," "alarm," and "time").

[0086] After updating the object keyword data 124 (e.g., after applying the object classification neural network 142 to each object 122), the adaptive classifier 144 determines whether any of the objects in the set of objects 122 are associated with one or more detected keywords 180. In response to determining that the object 122 is associated with the one or more detected keywords 180, the adaptive classifier 144 adds the object 122 to the one or more objects 182. In example 190, the adaptive classifier 144 adds the object 122A to the one or more objects 182 in response to determining that the object 122A (e.g., an image of the Statue of Liberty) is associated with one or more detected keywords 180 (e.g., "New York City"). In example 192, the adaptive classifier 144 adds the object 122B to the one or more objects 182 in response to determining that the object 122B (e.g., a clip art of a clock) is associated with one or more detected keywords 180 (e.g., "alarm clock"). In some implementations, in response to determining that at least one object is already included in one or more objects 182 , adaptive classifier 144 refrains from applying object generation neural network 140 to determine one or more objects 182 associated with one or more detected keywords 180 .

[0087] Optionally, in some embodiments, classifying the one or more objects 182 includes applying the object generation neural network 140 to the one or more detected keywords 180 to generate the one or more objects 182. In some aspects, the adaptive classifier 144 applies the object generation neural network 140 in response to determining that the one or more objects 182 do not include an object. For example, the adaptive classifier 144 applies the object generation neural network 140 in embodiments that do not include applying the object classification neural network 142, or after applying the object classification neural network 142 but no matching objects are detected for the one or more detected keywords 180.

[0088] In some aspects, the object determination unit 114 applies the object classification neural network 142 independently of whether any pre-existing objects are already included in the one or more objects 182 to update the classification of the object 122. For example, in these aspects, the adaptive classifier 144 includes the object generation neural network 140, and the object classification neural network 142 is external to the adaptive classifier 144. To illustrate, in these aspects, classifying the one or more objects 182 includes selectively applying the object generation neural network 140 in response to determining that no objects (e.g., no pre-existing objects) are included in the one or more objects 182, and applying the object classification neural network 142 independently of whether any pre-existing objects are already included in the one or more objects 182. In these aspects, resources are used to classify the objects 122 of the database 150, and resources are selectively used to generate new objects.

[0089] In some aspects, the object determination unit 114 applies the object generation neural network 140 independently of whether any pre-existing objects are already included in the one or more objects 182 to generate one or more additional objects to be added to the one or more objects 182. For example, in these aspects, the adaptive classifier 144 includes the object classification neural network 142, and the object generation neural network 140 is external to the adaptive classifier 144. To illustrate, in these aspects, classifying the one or more objects 182 includes selectively applying the object classification neural network 142 in response to determining that no objects (e.g., no pre-existing and pre-classified objects) are already included in the one or more objects 182, and applying the object generation neural network 140 independently of whether any pre-existing objects are already included in the one or more objects 182. In these aspects, resources are used to add the newly generated objects to the database 150, and resources are selectively used to classify objects 122 of the database 150 that may have already been classified.

[0090] In some embodiments, the object generation neural network 140 includes a stacked generative adversarial network (GAN). For example, applying the object generation neural network 140 to the detected keyword 180 includes generating an embedding representing the detected keyword 180, using a stage 1 GAN to generate a lower resolution object based at least in part on the embedding, and using a stage 2 GAN to refine the lower resolution object to generate a higher resolution object, as described in reference Figure 7 As further described, adaptive classifier 144 adds the newly generated higher resolution object to the set of objects 122 , updates object keyword data 124 indicating that the high resolution object is associated with detected keyword 180 , and adds the newly generated object to one or more objects 182 .

[0091] In example 190, if no object 122 is associated with one or more detected keywords 180 (e.g., "New York City"), the adaptive classifier 144 applies the object generation neural network 140 to the one or more detected keywords 180 (e.g., "New York City") to generate an object 122A (e.g., an image of the Statue of Liberty). The adaptive classifier 144 adds the object 122A (e.g., the image of the Statue of Liberty) to the set of objects 122 in the database 150, updates the object keyword data 124 to indicate that the object 122A is associated with the one or more detected keywords 180 (e.g., "New York City"), and adds the object 122A to the one or more objects 182.

[0092] In example 192, if no object 122 is associated with the one or more detected keywords 180 (e.g., "alarm clock"), the adaptive classifier 144 applies the object generation neural network 140 to the one or more detected keywords 180 (e.g., "alarm clock") to generate an object 122B (e.g., a clip art of a clock). The adaptive classifier 144 adds the object 122B (e.g., a clip art of a clock) to the set of objects 122 in the database 150, updates the object keyword data 124 to indicate that the object 122B is associated with the one or more detected keywords 180 (e.g., "alarm clock"), and adds the object 122B to the one or more objects 182.

[0093] The adaptive classifier 144 provides the one or more objects 182 to the object insertion unit 116 to insert the one or more objects 182 at one or more insertion locations 164 in the video stream 136. In some embodiments, the one or more insertion locations 164 are predetermined. For example, the one or more insertion locations 164 are based on default data, configuration settings, user input, or a combination thereof. In some aspects, the predetermined insertion locations 164 can include position-specific locations of the video frame, such as background, foreground, bottom, corner, center, etc.

[0094] Optionally, in some embodiments where the video stream updater 110 includes a position determination unit 170, the adaptive classifier 144 also provides the one or more objects 182 (or at least type information of the one or more objects 182) to the position determination unit 170 to dynamically determine the one or more insertion locations 164. In some examples, the one or more insertion locations 164 may include location-specific locations, such as background, foreground, top, middle, bottom, corner, diagonal, or a combination thereof. In some examples, the one or more insertion locations 164 may include content-specific locations, such as the front of a shirt, a sports field, a television, a whiteboard, a wall, a picture frame, another element depicted in a video frame, or a combination thereof. Using the position determination unit 170 enables dynamic selection of elements in the content of the video stream 136 as the one or more insertion locations 164.

[0095] In some implementations, the position determination unit 170 performs an image comparison of portions of a video frame of the video stream 136 with stored images of potential locations to identify the one or more insertion locations 164. Optionally, in some implementations where the position determination unit 170 includes a position neural network 162, the position determination unit 170 applies the position neural network 162 to the video stream 136 to determine the one or more insertion locations 164 in the video stream 136. For example, the position determination unit 170 applies the position neural network 162 to the video frames of the video stream 136 to determine the one or more insertion locations 164, as further described with reference to FIG. 10. Technical effects of using the position neural network 162 to identify the insertion locations may include using fewer resources (e.g., time, computing cycles, memory, or a combination thereof), having a higher degree of accuracy, or both when determining the one or more insertion locations 164, as compared to performing an image comparison to identify the insertion locations.

[0096] The object insertion unit 116 receives one or more objects 182 from the adaptive classifier 144. In some implementations, the object insertion unit 116 uses one or more predetermined locations as the one or more insertion locations 164. In other implementations, the object insertion unit 116 receives one or more insertion locations 164 from the location determination unit 170.

[0097] The object insertion unit 116 inserts one or more objects 182 at one or more insertion locations 164 in the video stream 136. In example 190, the object insertion unit 116 inserts the object 122A as the background into one or more video frames of the video stream 136A to generate the video stream 136B in response to determining that the insertion location 164 (e.g., background) is associated with an object 122A (e.g., an image of the Statue of Liberty) included in the one or more objects 182. In example 192, in response to determining that the insertion location 164 (e.g., foreground) is associated with an object 122B (e.g., a clip art of a clock) included in the one or more objects 182, the object insertion unit 116 inserts the object 122B as a foreground object into one or more video frames of the video stream 136A to generate the video stream 136B.

[0098] In some embodiments, the insertion position 164 corresponds to an element depicted in a video frame (e.g., the front of a shirt). The object insertion unit 116 inserts the object 122 (e.g., a shirt) at the insertion position 164, and the insertion position 164 may change position in one or more video frames of the video stream 136A to follow the movement of the element. For example, the object insertion unit 116 determines a first position of an element (e.g., a shirt) in a first video frame and inserts the object 122 at the first position in the first video frame. As another example, the object insertion unit 116 determines a second position of an element (e.g., a shirt) in a second video frame and inserts the object 122 at the second position in the second video frame. If the element has changed position between the first video frame and the second video frame, the first position may be different from the second position.

[0099] In a particular example, the one or more objects 182 include a single object 122, and the one or more insertion locations 164 include a plurality of insertion locations 164. In some implementations, the object insertion unit 116 selects one of the insertion locations 164 for inserting the object 122, while in other implementations, the object insertion unit 116 inserts copies of the object 122 at two or more of the plurality of insertion locations 164 in the video stream 136. In some implementations, the object insertion unit 116 performs a looped insertion of the object 122 at the plurality of insertion locations 164. For example, the object insertion unit 116 inserts the object 122 at a first position of the plurality of insertion locations 164 in a first set of video frames of the video stream 136, inserts the object 122 at a second position (not the first position) of the one or more insertion locations 164 in a second set of video frames of the video stream 136 that is different from the first set of video frames, and so on.

[0100] In a specific example, the one or more objects 182 include a plurality of objects 122, and the one or more insertion positions 164 include a plurality of insertion positions 164. In some implementations, the object insertion unit 116 performs a looped insertion of the plurality of objects 122 at the plurality of insertion positions 164. For example, the object insertion unit 116 inserts a first object 122 at a first insertion position 164 in a first set of video frames of the video stream 136, inserts a second object 122 at a second insertion position 164 in a second set of video frames of the video stream 136 that is different from the first set of video frames (without the first object 122 in the first insertion position 164), and so on.

[0101] In a specific example, the one or more objects 182 include a plurality of objects 122, and the one or more insertion locations 164 include a single insertion location 164. In some implementations, the object insertion unit 116 performs a looped insertion of the plurality of objects 122 at the single insertion location 164. For example, the object insertion unit 116 inserts the first object 122 at the insertion location 164 in a first set of video frames of the video stream 136, inserts the second object 122 (instead of the first object 122) at the insertion location 164 in a second set of video frames of the video stream 136 that is different from the first set of video frames, and so on.

[0102] The object insertion unit 116 outputs the video stream 136 after inserting the one or more objects 182 into the video stream 136. In some implementations, the object insertion unit 116 provides the video stream 136 to a display device, a network device, a storage device, a cloud-based resource, or a combination thereof.

[0103] Thus, the system 100 enables the video stream 136 to be enhanced with one or more objects 182 associated with one or more detected keywords 180. The enhancement of the video stream 136 can improve viewer retention, create advertising opportunities, etc. For example, adding objects to the video stream 136 can make the video stream 136 more interesting to the viewer. To illustrate, when the audio stream 134 includes one or more detected keywords 180 associated with the object 122A (e.g., "New York City"), adding the object 122A (e.g., an image of the Statue of Liberty) can increase viewer retention of the video stream 136. In another example, the object 122A can be associated with a related entity associated with the one or more detected keywords 180 (e.g., a restaurant in New York, a restaurant that serves food associated with New York, another business that sells New York-related food or services, a travel website, or a combination thereof).

[0104] Although the video stream updater 110 is shown as including the position determination unit 170, in some other implementations, the position determination unit 170 is excluded from the video stream updater 110. For example, in an embodiment where the position determination unit 170 is disabled or omitted from the position determination unit 170, the object insertion unit 116 uses one or more predetermined positions as the one or more insertion positions 164. Using the position determination unit 170 enables dynamic determination of the one or more insertion positions 164, including content-specific insertion positions.

[0105] Although the adaptive classifier 144 is shown as including the object generation neural network 140 and the object classification neural network 142, in some other embodiments, the object generation neural network 140 or the object classification neural network 142 is excluded from the video stream updater 110. For example, adaptively classifying one or more objects 182 may include selectively applying the object generation neural network 140. In some embodiments, the object determination unit 114 does not include the object classification neural network 142, and thus does not use resources to reclassify objects that may have already been classified. In other embodiments, the object determination unit 114 includes the object classification neural network 142 external to the adaptive classifier 144, and thus classifies objects independently of the adaptive classifier 144. In an example, adaptively classifying one or more objects 182 may include selectively applying the object classification neural network 142. In some embodiments, the object determination unit 114 does not include the object generation neural network 140, and thus resources are not used to generate new objects. In other embodiments, the object determination unit 114 includes the object generation neural network 140 external to the adaptive classifier 144, and thus generates new objects independently of the adaptive classifier 144.

[0106] The use of object generation neural network 140 to generate new objects is provided as an illustrative example. In other examples, another type of object generator that does not include a neural network can be used as an alternative or in addition to object generation neural network 140 to generate new objects. The use of object classification neural network 142 to perform classification of objects is provided as an illustrative example. In other examples, another type of object classifier that does not include a neural network can be used as an alternative or in addition to object classification neural network 142 to perform classification of objects.

[0107] Although the keyword detection unit 112 is illustrated as including a keyword detection neural network 160, in some other implementations, the keyword detection unit 112 may process the audio stream 134 to determine the one or more detected keywords 180 independently of any neural network. For example, the keyword detection unit 112 may use speech analysis and semantic analysis to determine the one or more detected keywords 180. Using the keyword detection neural network 160 (e.g., as compared to speech recognition and semantic analysis) may include using fewer resources (e.g., time, computing cycles, memory, or a combination thereof), having a higher degree of accuracy, or both when determining the one or more detected keywords 180.

[0108] Although the position determination unit 170 is shown as including the position neural network 162, in some other embodiments, the position determination unit 170 can determine the one or more insertion locations 164 independently of any neural network. For example, the position determination unit 170 can use image comparison to determine the one or more insertion locations 164. Using the position neural network 162 (e.g., as compared to image comparison) can include using fewer resources (e.g., time, computing cycles, memory, or a combination thereof), having a higher accuracy, or both when determining the one or more insertion locations 164.

[0109] refer to Figure 2 , showing a specific implementation of a method 200 for inserting a keyword-based object into a video stream, and an example 250 of inserting a keyword-based object into a video stream. In certain aspects, one or more operations of the method 200 are performed by Figure 1 The method may be performed by one or more of the keyword detection unit 112, the adaptive classifier 144, the position determination unit 170, the object insertion unit 116, the video stream updater 110, one or more processors 102, the device 130, the system 100, or a combination thereof.

[0110] Method 200 includes obtaining at least a portion of an audio stream at 202. For example, Figure 1 The keyword detection unit 112 obtains one or more audio frames of the audio stream 134, as shown in FIG. Figure 1 described.

[0111] Method 200 also includes detecting a keyword at 204. For example, Figure 1 The keyword detection unit 112 processes one or more audio frames of the audio stream 134 to determine one or more detected keywords 180, as shown in FIG. Figure 1 In example 250, the keyword detection unit 112 processes the audio stream 134 to determine one or more detected keywords 180 (eg, "New York City").

[0112] Method 200 further includes determining whether any background object corresponds to a keyword at 206. In an example, Figure 1 The set of objects 122 corresponds to background objects. For illustration, each of the set of objects 122 can be inserted into the background of a video frame. The adaptive classifier 144 determines whether any object in the set of objects 122 corresponds to (e.g., is associated with) one or more detected keywords 180, as described in reference to FIG. Figure 1 described.

[0113] The method 200 also includes, in response to determining that the background object corresponds to a keyword at 206, inserting the background object at 208. For example, in response to determining that the object 122A corresponds to the one or more detected keywords 180, the adaptive classifier 144 adds the object 122A to the one or more objects 182 associated with the one or more detected keywords 180. In response to determining that the object 122A is included in the one or more objects 182 corresponding to the one or more detected keywords 180, the object insertion unit 116 inserts the object 122A into the video stream 136. In example 250, the object insertion unit 116 inserts the object 122A (e.g., an image of the Statue of Liberty) into the video stream 136A to generate the video stream 136B.

[0114] Otherwise, in response to determining at 206 that no background objects correspond to the keyword, method 200 includes maintaining the original background at 210. For example, in response to adaptive classifier 144 determining that the set of objects 122 does not include any background objects associated with one or more detected keywords 180, video stream updater 110 bypasses object insertion unit 116 and outputs one or more video frames of video stream 136 unchanged (e.g., without inserting any background objects into one or more video frames of video stream 136A).

[0115] Thus, the method 200 enables the video stream 136 to be enhanced with background objects associated with one or more detected keywords 180. When no background objects are associated with one or more detected keywords 180, the background of the video stream 136 remains unchanged.

[0116] refer to Figure 3 , showing a specific implementation of a method 300 for inserting a keyword-based object into a video stream, and a diagram 350 of an example of inserting a keyword-based object into a video stream. In certain aspects, one or more operations of the method 300 are performed by Figure 1The method may be performed by one or more of the keyword detection unit 112, the adaptive classifier 144, the position determination unit 170, the object insertion unit 116, the video stream updater 110, one or more processors 102, the device 130, the system 100, or a combination thereof.

[0117] Method 300 includes obtaining at least a portion of an audio stream at 302. For example, Figure 1 The keyword detection unit 112 obtains one or more audio frames of the audio stream 134, as shown in FIG. Figure 1 described.

[0118] The method 300 also includes detecting keywords using a keyword detection neural network at 304. For example, Figure 1 The keyword detection unit 112 uses the keyword detection neural network 160 to process one or more audio frames of the audio stream 134 to determine one or more detected keywords 180, as shown in FIG. Figure 1 described.

[0119] The method 300 further includes determining whether the keyword is mapped to any object in the database at 306. For example, Figure 1 The adaptive classifier 144 determines whether any object in the set of objects 122 stored in the database 150 corresponds to (eg, is associated with) one or more detected keywords 180, as described in reference to Figure 1 described.

[0120] Method 300 includes, in response to determining at 306 that the keyword maps to an object in the database, selecting the object at 308. For example, Figure 1 In response to determining that one or more detected keywords 180 (e.g., "New York City") are associated with object 122A (e.g., an image of the Statue of Liberty), adaptive classifier 144 selects object 122A to be added to one or more objects 182 associated with the one or more detected keywords 180, as shown in FIG. Figure 1 As another example, Figure 1 In response to determining that one or more detected keywords 180 (e.g., "New York City") are associated with object 122B (e.g., a clip art of an apple with the letters "NY"), the adaptive classifier 144 selects object 122B to be added to one or more objects 182 associated with the one or more detected keywords 180, as shown in FIG. Figure 1 described.

[0121] Otherwise, in response to determining at 306 that the keyword does not map to any object in the database, method 300 includes generating an object using an object generation neural network at 310. For example, Figure 1In response to determining that none of the set of objects 122 is associated with one or more detected keywords 180, adaptive classifier 144 uses object generation neural network 140 to generate object 122A (e.g., an image of the Statue of Liberty), object 122B (e.g., a clip art of an apple with the letters “NY”), one or more additional objects, or a combination thereof, as described in reference to FIG. Figure 1 After generating the object at 310, method 300 includes adding the generated object to a database at 312 and selecting the object at 308. For example, Figure 1 The adaptive classifier 144 adds the object 122A, the object 122B, or both to the database 150, and selects the object 122A, the object 122B, or both to be added to the one or more objects 182 associated with the one or more detected keywords 180, as shown in FIG. Figure 1 described.

[0122] The method 300 also includes determining whether the object is of a background type at 314. For example, Figure 1 The position determination unit 170 may determine whether the object 122 included in the one or more objects 182 is of the background type. The position determination unit 170 specifies the insertion position 164 for the object 122 based on the determination of whether the object 122 is of the background type, as shown in FIG. Figure 1 In a specific example, Figure 1 In response to determining that object 122A among the one or more objects 182 is of the background type, the position determination unit 170 specifies a first insertion position 164 (e.g., background) for object 122A. As another example, in response to determining that object 122B among the one or more objects 182 is not of the background type, the position determination unit 170 specifies a second insertion position 164 (e.g., foreground) for object 122B. In some embodiments, the position determination unit 170 selects another position (e.g., foreground) of the video frame as the insertion position 164 in response to determining that a position (e.g., background) of a video frame of the video stream 136 includes at least one object associated with the one or more detected keywords 180.

[0123] In a particular embodiment, a first subset of the set 122 of objects is stored in a background database, and a second subset of the set 122 of objects is stored in a foreground database, both of which may be included in the database 150. In this embodiment, in response to determining that the object 122A is included in the background database, the position determination unit 170 determines that the object 122A is of the background type. In the example, in response to determining that the object 122B is included in the foreground database, the position determination unit 170 determines that the object 122B is of the foreground type instead of the background type.

[0124] In some embodiments, the first subset and the second subset are non-overlapping. For example, object 122 is included in the background database or the foreground database, but not in both. However, in other embodiments, the first subset at least partially overlaps with the second subset. For example, a copy of object 122 can be included in each of the background database and the foreground database.

[0125] In certain embodiments, the object type of object 122 is based on the file type (e.g., image file, GIF file, PNG file, etc.) of object 122. For example, in response to determining that object 122A is an image file, position determination unit 170 determines that object 122A is of background type. In another example, in response to determining that object 122B is not an image file (e.g., object 122B is a GIF file or a PNG file), position determination unit 170 determines that object 122B is of foreground type instead of background type.

[0126] In certain embodiments, the metadata of object 122 indicates whether object 122 is of the background type or of the foreground type. For example, in response to determining that the metadata of object 122A indicates that object 122A is of the background type, position determination unit 170 determines that object 122A is of the background type. As another example, in response to determining that the metadata of object 122B indicates that object 122B is of the foreground type, position determination unit 170 determines that object 122B is of the foreground type instead of the background type.

[0127] Method 300 includes, in response to determining at 314 that the object is of a background type, inserting the object into the background at 316. For example, Figure 1 The object insertion unit 116 inserts the object 122A at the first insertion position (e.g., background) in one or more video frames of the video stream 136 in response to determining that the first insertion position 164 (e.g., background) is designated for the object 122A in the one or more objects 182, as shown in FIG. Figure 1 described.

[0128] Otherwise, in response to determining at 314 that the object is not of the background type, method 300 includes inserting the object into the foreground at 318. For example, Figure 1 The object insertion unit 116 inserts the object 122B at the second insertion position (e.g., foreground) in one or more video frames of the video stream 136 in response to determining that the second insertion position 164 (e.g., foreground) is designated for the object 122B in the one or more objects 182, as shown in FIG. Figure 1 described.

[0129] Thus, when there is no pre-existing object 122 associated with the one or more detected keywords 180, the method 300 can generate a new object 122 associated with the one or more detected keywords 180. The object 122 can be added to the background or foreground of the video stream 136 based on the object type of the object 122. The object type of the object 122 can be based on the file type of the object 122, the storage location, the metadata, or a combination thereof.

[0130] In diagram 350, keyword detection unit 112 processes audio stream 134 using keyword detection neural network 160 to determine one or more detected keywords 180 (e.g., "New York City"). In particular aspects, adaptive classifier 144 determines that object 122A (e.g., an image of the Statue of Liberty) is associated with one or more detected keywords 180 (e.g., "New York City") and adds object 122A to one or more objects 182. In response to determining that object 122A is of background type, position determination unit 170 designates object 122A as being associated with first insertion position 164 (e.g., background). In response to determining that object 122A is associated with first insertion position 164 (e.g., background), object insertion unit 116 inserts object 122A into one or more video frames of video stream 136A to generate video stream 136B.

[0131] According to an alternative aspect, adaptive classifier 144 may alternatively determine that object 122B (e.g., clip art of an apple with the letters "NY") is associated with one or more detected keywords 180 (e.g., "New York City") and add object 122B to one or more objects 182. In response to determining that object 122B is not of the background type, position determination unit 170 designates object 122B as being associated with second insertion position 164 (e.g., foreground). In response to determining that object 122B is associated with second insertion position 164 (e.g., foreground), object insertion unit 116 inserts object 122B into one or more video frames of video stream 136A to generate video stream 136C.

[0132] refer to Figure 4 , a diagram 400 of an illustrative implementation of the keyword detection unit 112 is shown. The keyword detection neural network 160 includes a speech recognition neural network 460 coupled to a keyword selector 464 via a potential keyword detector 462.

[0133] The speech recognition neural network 460 is configured to process at least a portion of the audio stream 134 to generate one or more words 461 detected in the portion of the audio stream 134. In certain aspects, the speech recognition neural network 460 comprises a recurrent neural network (RNN). In other aspects, the speech recognition neural network 460 can comprise another type of neural network.

[0134] In an illustrative embodiment, the speech recognition neural network 460 includes an encoder 402, a RNN transducer (RNN-T) 404, and a decoder 406. In a particular aspect, the encoder 402 is trained as a connection time classification (CTC) network. During training, the encoder 402 is configured to process one or more acoustic features 412 to predict phonemes 414, graphemes 416, and wordpieces 418 from a long short-term memory (LSTM) layer 420, an LSTM layer 422, and an LSTM layer 426, respectively. The encoder 402 includes a temporal convolutional layer 424 that reduces the encoder time series length (e.g., by a factor of three). The decoder 406 is trained to predict one or more fields 458 by processing an input embedding 454 of one or more input fields 452 using an LSTM layer 456. According to some aspects, the decoder 406 is trained to reduce cross entropy loss.

[0135] The RNN-T 404 is configured to process one or more acoustic features 432 of at least a portion of the audio stream 134 using LSTM layers 434, LSTM layers 436, and LSTM layers 440 to provide a first input (e.g., a first field) to a feedforward 448 (e.g., a feedforward layer). The RNN-T 404 also includes a temporal convolutional layer 438. The RNN-T 404 is configured to process an input embedding 444 of one or more input fields 442 using an LSTM layer 446 to provide a second input (e.g., a second field) to the feedforward 448. In a particular aspect, the one or more acoustic features 432 correspond to real-time test data, and the one or more input fields 442 correspond to existing training data on which a speech recognition neural network 460 is trained. The feedforward 448 is configured to process the first input and the second input to generate a field 450. The speech recognition neural network 460 is configured to output one or more words 461 corresponding to the one or more fields 450.

[0136] The RNN-T 404 (eg, the weights of the RNN-T 404) is initialized based on the encoder 402 (eg, the trained encoder 402) and the decoder 406 (eg, the trained decoder 406). In the example (by Figure 4), initialize the weights of LSTM layer 434 based on the weights of LSTM layer 420, initialize the weights of LSTM layer 436 based on the weights of LSTM layer 422, initialize the weights of LSTM layer 440 based on the weights of LSTM layer 426, initialize the weights of temporal convolution layer 438 based on the weights of temporal convolution layer 424, initialize the weights of LSTM layer 446 based on the weights of LSTM layer 456, initialize the weights of generated input embedding 444 based on the weights of generated input embedding 454, or a combination thereof.

[0137] LSTM layer 420 including 5 LSTM layers, LSTM layer 422 including 5 LSTM layers, LSTM layer 426 including 2 LSTM layers, and LSTM layer 456 including 2 LSTM layers are provided as illustrative examples. In other examples, LSTM layer 420, LSTM layer 422, LSTM layer 426, and LSTM layer 456 may include any number of LSTM layers. In a particular aspect, LSTM layer 434, LSTM layer 436, LSTM layer 440, and LSTM layer 446 include the same number of LSTM layers as LSTM layer 420, LSTM layer 422, LSTM layer 426, and LSTM layer 456, respectively.

[0138] Potential keyword detector 462 is configured to process one or more words 461 to determine one or more potential keywords 463, as shown in FIG. Figure 5 The keyword selector 464 is configured to select one or more detected keywords 180 from one or more potential keywords 463, as described in reference Figure 5 Further described.

[0139] refer to Figure 5 , a diagram 500 showing illustrative aspects of operations associated with keyword detection. In certain aspects, keyword detection is performed by Figure 1 The keyword detection neural network 160, the keyword detection unit 112, the video stream updater 110, the one or more processors 102, the device 130, the system 100, Figure 4 The speech recognition neural network 460, the potential keyword detector 462, the keyword selector 464 or a combination thereof is performed.

[0140] Keyword detection neural network 160 obtains at least a portion of audio stream 134 representing speech. Keyword detection neural network 160 uses speech recognition neural network 460 on the portion of audio stream 134 to detect one or more words 461 of speech (e.g., "Wishing you on your birthday, anything you ask for may be received, anything you wish may be fulfilled on your birthday and always happy birthday"), as referenced in the example above. Figure 4as described.

[0141] Potential keyword detector 462 performs semantic analysis on one or more words 461 to identify one or more potential keywords 463 (e.g., "wish", "request", "birthday"). For example, potential keyword detector 462 ignores conjunctions, articles, prepositions, etc. in one or more words 461. One or more potential keywords 463 are indicated by underlining in one or more words 461 in diagram 500. In some embodiments, one or more potential keywords 463 may include one or more words (e.g., "wish", "request", "birthday"), one or more phrases (e.g., "New York City", "alarm clock"), or a combination thereof.

[0142] The keyword selector 464 selects at least one of the one or more potential keywords 463 (e.g., "wish", "request", "birthday") as one or more detected keywords 180 (e.g., "birthday"). In some implementations, the keyword selector 464 performs semantic analysis on the one or more words 461 to determine which of the one or more potential keywords 463 corresponds to a subject in the one or more words 461, and selects at least one of the one or more potential keywords 463 corresponding to the subject as one or more detected keywords 180. In a specific example, the keyword selector 464 selects the potential keyword 463 (e.g., "birthday") as one or more detected keywords 180 based at least in part on determining that the potential keyword 463 (e.g., "birthday") appears more frequently (e.g., three times) in the one or more words 461 than other keywords in the one or more potential keywords 463. The keyword selector 464 selects at least one of the one or more potential keywords 463 (e.g., "wish", "request", "birthday") corresponding to the subject of the one or more words 461 (e.g., "birthday") as one or more detected keywords 180.

[0143] In certain aspects, object 122A (e.g., clip art of an elf) is associated with one or more keywords 120A (e.g., "wishes" and "elves"), and object 122B (e.g., an image with balloons and a birthday banner) is associated with one or more keywords 120B (e.g., "balloons," "birthday," "birthday banner"). In certain aspects, in response to determining that one or more keywords 120B (e.g., "balloons," "birthday," "birthday banner") match one or more detected keywords 180 (e.g., "birthday"), adaptive classifier 144 selects object 122B to include in one or more objects 182 associated with one or more detected keywords 180, as described in reference to FIG. Figure 1 described.

[0144] refer to Figure 6 , showing method 600, example 650, example 652, and example 654 of object generation. In certain aspects, one or more operations of method 600 are performed by Figure 1 The object generation neural network 140, the adaptive classifier 144, the video stream updater 110, one or more processors 102, the device 130, the system 100, or a combination thereof is executed.

[0145] Method 600 includes preprocessing at 602. For example, Figure 1 The object generation neural network 140 pre-processes at least a portion of the audio stream 134. To illustrate, the pre-processing may include reducing noise in at least a portion of the audio stream 134 to increase the signal-to-noise ratio.

[0146] Method 600 also includes feature extraction at 604. For example, Figure 1 The object generation neural network 140 extracts features 605 (eg, acoustic features) from the pre-processed portion of the audio stream 134 .

[0147] Method 600 also includes performing semantic analysis using a language model at 606. For example, Figure 1 The object generation neural network 140 may obtain one or more words 461 and one or more detected keywords 180 corresponding to the pre-processed portion of the audio stream 134. For illustration, the object generation neural network 140 obtains the one or more words 461 based on the operation of the keyword detection unit 112. For example, Figure 1 The keyword detection unit 112 performs pre-processing (eg, denoising, one or more additional enhancements, or a combination thereof) of at least a portion of the audio stream 134 to generate a pre-processed portion of the audio stream 134 . Figure 4 The speech recognition neural network 460 performs speech recognition on the preprocessing part to generate one or more words 461, and the one or more words 461 can be provided to Figure 4 The potential keyword detector 462 is also provided to the object generation neural network 140.

[0148] The object generation neural network 140 can perform semantic analysis on the features 605, one or more words 461 (e.g., "flower with long pink petals and raised orange stamens"), one or more detected keywords 180 (e.g., "flower"), or a combination thereof to generate one or more descriptors 607 (e.g., "long pink petals; raised orange stamens"). In certain aspects, the object generation neural network 140 performs semantic analysis using a language model. In some examples, the object generation neural network 140 performs semantic analysis on one or more detected keywords 180 (e.g., "New York") to determine one or more related words (e.g., "Statue of Liberty", "port", etc.).

[0149] Method 600 also includes generating an object using an object generation network at 608. For example, Figure 1 The adaptive classifier 144 uses the object generation neural network 140 to process one or more detected keywords 180 (e.g., "flower"), one or more descriptors 607 (e.g., "long pink petals" and "raised orange stamens"), related words, or a combination thereof to generate one or more objects 182, as shown in FIG. Figure 7 As further described. Thus, the adaptive classifier 144 enables a plurality of words corresponding to one or more detected keywords 180 to be used as input to an object generation neural network 140 (e.g., a GAN) to generate an object 182 (e.g., an image) associated with the plurality of words. In some aspects, the object generation neural network 140 generates the object 182 (e.g., an image) in real time while processing the audio stream 134 of the live media stream, so that the object 182 can be inserted into the video stream 136 substantially simultaneously with the determination of the one or more detected keywords 180 (e.g., with an imperceptible or nearly imperceptible delay). Optionally, in a particular embodiment, the object generation neural network 140 selects an existing object (e.g., an image of a flower) that matches the one or more detected keywords 180 (e.g., "flower") and modifies the existing object to generate the object 182. For example, the object generation neural network 140 modifies the existing object based on the one or more detected keywords 180 (e.g., "flower"), one or more descriptors 607 (e.g., "long pink petals" and "raised orange stamens"), related words, or a combination thereof to generate the object 182.

[0150] In example 650, the adaptive classifier 144 uses the object generation neural network 140 to process one or more words 461 (e.g., "flower with long pink petals and raised orange stamens") to generate an object 122 (e.g., a generated image of a flower with various pink petals, orange stamens, or a combination thereof). In example 652, the adaptive classifier 144 uses the object generation neural network 140 to process one or more words 461 ("blue bird") to generate an object 122 (e.g., a generated photo-realistic image of a bird). In example 654, the adaptive classifier 144 uses the object generation neural network 140 to process one or more words 461 ("blue bird") to generate an object 122 (e.g., a generated clip art of a bird).

[0151] refer to Figure 7 , a diagram 700 of an example of one or more components of the object determination unit 114 is shown and includes the object generation neural network 140. In certain aspects, the object determination unit 114 may include one or more additional components that are not shown for ease of illustration.

[0152] In particular embodiments, object generation neural network 140 includes a stacked GAN. To illustrate, object generation neural network 140 includes a stage 1 GAN coupled to a stage 2 GAN. Stage 1 GAN includes conditioning enhancer 704 coupled to stage 1 discriminator 708 via stage 1 generator 706. Stage 2 GAN includes conditioning enhancer 710 coupled to stage 2 discriminator 714 via stage 2 generator 712. Stage 1 GAN generates a lower resolution object based on embedding 702. Stage 2 GAN generates a higher resolution object (e.g., a photo-realistic image) based on embedding 702 and also based on the lower resolution object from stage 1 GAN.

[0153] The object generation neural network 140 is configured to generate an embedding ( ) 702. In some aspects, the text description 701 corresponds to Figure 4 One or more words 461, Figure 1 One or more detected keywords 180, Figure 6 In certain embodiments, some details of the text description 701 that were ignored by the stage 2 GAN when generating the lower resolution object are considered by the stage 1 GAN when generating the higher resolution object.

[0154] The object generation neural network 140 provides the embedding 702 to each of the conditioning enhancer 704, the stage 1 discriminator 708, the conditioning enhancer 710, and the stage 2 discriminator 714. The conditioning enhancer 704 processes the embedding using a fully connected layer ( ) 702 to generate a Gaussian distribution The mean (μ0) 703 and variance (σ0) 705 of corresponds to being embedded as ( ) 702. The variance (σ0) 705 corresponds to The value on the diagonal of ). Adjust the enhancer 704 to be from the Gaussian distribution The sampled embedding 702 generates a Gaussian conditioning variable ( ) 709 to capture the meaning of the embedding 702 with changes. For example, the moderating variable ( ) 709 is based on the following equation:

[0155] , Equation 1

[0156] in, corresponds to the adjustment variable ( ) 709 (eg, adjustment vector), corresponds to a mean value (μ0) of 703, corresponds to a variance (σ0) of 705, corresponds to element-wise multiplication, Corresponding to Gaussian distribution N(0,1). Regulation enhancer 704 adjusts the variable ( ) 709 (e.g., adjustment vector) is provided to the stage 1 generator 706.

[0157] The stage-1 generator 706 generates a lower resolution object 717 conditioned on the text description 701. For example, the stage-1 generator 706 modulates the variable ( ) 709 and a random variable (z) are conditioned to generate a lower resolution object 717. In an example, the lower resolution object 717 (e.g., an image, clip art, GIF file, etc.) represents a primitive shape and basic color. In a particular aspect, the random variable (z) corresponds to random noise (e.g., a dimensional noise vector). In a particular example, the stage 1 generator 706 cascades the adjustment variable ( ) 709 and a random variable (z), and the cascade is processed by a series of upsampling blocks 715 to generate a lower resolution object 717.

[0158] The stage 1 discriminator 708 spatially replicates the embedding ( ) 702 to generate a text tensor. The stage 1 discriminator 708 processes the lower resolution object 717 using a downsampling block 719 to generate an object filter map. The object filter map is concatenated with the text tensor to generate an object text tensor that is fed into the convolutional layer. A fully connected layer 721 with one node is used to generate a decision score.

[0159] In some aspects, the stage 2 generator 712 is designed as an encoder-decoder with a residual block 729. Similar to the conditioning enhancer 704, the conditioning enhancer 710 processes the embedding ( )702 to generate the adjustment variable ( ) 723, which is spatially replicated at the stage 2 generator 712 to form a text tensor. The lower resolution object 717 is processed by a series of downsampling blocks (e.g., encoders) to generate an object filter map. The object filter map is concatenated with the text tensor to generate an object text tensor, which is processed by a residual block 729. In a particular aspect, the residual block 729 is designed to learn a multi-model representation across features of the lower resolution object 717 and features of the text description 701. A series of upsampling blocks 731 (e.g., decoders) are used to generate a higher resolution object 733. In a particular example, the higher resolution object 733 corresponds to a photo-realistic image.

[0160] The stage 2 discriminator 714 spatially replicates the embedding ( ) 702 to generate a text tensor. The stage 2 discriminator 714 processes the higher resolution object 733 using a downsampling block 735 to generate an object filter map. In a particular aspect, the count of the downsampling block 735 is greater than the count of the downsampling block 719 due to the larger size of the high resolution object 733 compared to the lower resolution object 717. The object filter map is concatenated with the text tensor to generate an object text tensor, which is fed to the convolutional layer. A fully connected layer 737 with one node is used to generate a decision score.

[0161] During the training phase, the stage 1 generator 706 and the stage 1 discriminator 708 may be jointly trained. During training, the stage 1 discriminator 708 is trained (e.g., modified based on feedback) to improve its ability to distinguish images generated by the stage 1 generator 706 from real images of similar resolution, while the stage 1 generator 706 is trained to improve its ability to generate images classified as real images by the stage 1 discriminator 708. Similarly, the stage 2 generator 712 and the stage 2 discriminator 714 may be jointly trained. During training, the stage 2 discriminator 714 is trained (e.g., modified based on feedback) to improve its ability to distinguish images generated by the stage 2 generator 712 from real images of similar resolution, while the stage 2 generator 712 is trained to improve its ability to generate images classified as real images by the stage 2 discriminator 714. In some implementations, after the training phase is complete, the stage 1 generator 706 and the stage 2 generator 712 may be used in the object generation neural network 140, while the stage 1 discriminator 708 and the stage 2 discriminator 714 may be omitted (or deactivated).

[0162] In certain aspects, lower resolution object 717 corresponds to an image having basic colors and primitive shapes, and higher resolution object 733 corresponds to a photo-realistic image. In certain aspects, lower resolution object 717 corresponds to a baseline drawing (e.g., no shading, a single color, or both), and higher resolution object 733 corresponds to a detailed drawing (e.g., having shading, multiple colors, or both).

[0163] In certain aspects, the object determination unit 114 adds the higher resolution object 733 as the object 122A to the database 150 and updates the object keyword data 124 to indicate that the object 122A is associated with one or more keywords 120A (e.g., the text description 701). In certain aspects, the object determination unit 114 adds the lower resolution object 717 as the object 122B to the database 150 and updates the object keyword data 124 to indicate that the object 122B is associated with one or more keywords 120B (e.g., the text description 701). In certain aspects, the object determination unit 114 adds the lower resolution object 717, the higher resolution object 733, or both to the one or more objects 182.

[0164] refer to Figure 8 , a method 800 of object classification is shown. In certain aspects, one or more operations of the method 800 are performed by Figure 1 The object classification neural network 142, the object determination unit 114, the adaptive classifier 144, the video stream updater 110, one or more processors 102, the device 130, the system 100, or a combination thereof is executed.

[0165] Method 800 includes selecting a next object from a database at 802. For example, Figure 1 Adaptive classifier 144 may select an initial object (eg, object 122A) from database 150 during an initial iteration of a processing loop over all objects 122 in database 150 , as described further below.

[0166] Method 800 also includes determining whether the object is associated with any keywords at 804. For example, Figure 1 The adaptive classifier 144 determines whether the object keyword data 124 indicates any keywords 120 associated with the object 122A.

[0167] Method 800 includes, in response to determining at 804 that the object is associated with at least one keyword, determining whether there are more objects in the database at 806. For example, in response to determining that the object keyword data 124 indicates that the object 122A is associated with one or more keywords 120A, Figure 1 The adaptive classifier 144 determines whether there are any additional objects 122 in the database 150. For illustration, the adaptive classifier 144 analyzes the objects 122 in order based on the object identifiers and determines whether there are additional objects corresponding to the next identifier after the identifier of the object 122A in the database 150. If there are no more unprocessed objects in the database, the method 800 ends at 808. Otherwise, the method 800 includes selecting a next object from the database for the next iteration of the processing loop at 802.

[0168] Method 800 includes, in response to determining at 804 that the object is not associated with any keyword, applying an object classification neural network to the object at 810. For example, Figure 1 In response to determining that the object keyword data 124 indicates that the object 122A is not associated with any keyword 120, the adaptive classifier 144 applies the object classification neural network 142 to the object 122A to generate one or more potential keywords, such as reference Figure 9A-9C Further described.

[0169] The method 800 also includes associating the object with the generated potential keyword having the highest possible score at 812. For example, each potential keyword generated for the object by the object classification neural network 142 may be associated with a score indicating a probability that the potential keyword matches the object. The adaptive classifier 144 may designate the keyword with the highest score among the potential keywords as the keyword 120A and update the object keyword data 124 to indicate that the object 122A is associated with the keyword 120A, as shown in FIG. Fig. 9C Further described.

[0170] refer to Fig.9A, showing the Figure 1 900 is a diagram of illustrative aspects of the operations associated with the object classification neural network 142. The object classification neural network 142 is configured to perform feature extraction 902 on the object 122A to generate features 926, as shown in FIG. Fig. 9B Further described.

[0171] The object classification neural network 142 is configured to perform classification 904 of the features 926 to generate a classification layer output 932, as shown in FIG. Fig. 9C As further described, the object classification neural network 142 is configured to process the classification layer output 932 to determine a probability distribution 906 associated with the one or more potential keywords and select at least one of the one or more potential keywords as the one or more keywords 120A based on the probability distribution 906 .

[0172] refer to Fig. 9B , a diagram showing illustrative aspects of feature extraction 902. In particular embodiments, the object classification neural network 142 includes a convolutional neural network (CNN) that includes a plurality of convolution stages 922 configured to generate an output feature map 924. The convolution stages 922 include a first set of convolution, ReLU, and pooling layers of a first stage 922A, a second set of convolution, ReLU, and pooling layers of a second stage 922B, and a third set of convolution, ReLU, and pooling layers of a third stage 922C. The output feature map 924 output from the third stage 922C is converted to a vector corresponding to a feature 926 (e.g., a flattened layer). Although three convolution stages 922 are shown, in other embodiments, any other number of convolution stages 922 may be used for feature extraction.

[0173] refer to Fig. 9C , a diagram showing illustrative aspects of classification 904 and determining probability distribution 906. In certain aspects, object classification neural network 142 includes fully connected layers 928, such as layer 928A, layer 928B, layer 928C, one or more additional layers, or a combination thereof. Object classification neural network 142 performs classification 904 by processing features 926 using fully connected layers 928 to generate classification layer outputs 932. For example, the output of the last layer 928D corresponds to the classification layer output 932.

[0174] The object classification neural network 142 applies a softmax activation function 930 to the classification layer output 932 to generate a probability distribution 906. For example, the probability distribution 906 indicates the probability of one or more potential keywords 934 being associated with the object 122A. For illustration, the probability distribution 906 indicates a first probability (e.g., 0.5), a second probability (e.g., 0.7), and a third probability (e.g., 0.1) that a first potential keyword 934 (e.g., "bird"), a second potential keyword 934 (e.g., "bluebird"), and a third potential keyword 934 (e.g., "white bird") are associated with the object 122A (e.g., an image of a bluebird), respectively.

[0175] Object classification neural network 142 selects at least one of one or more potential keywords 934 to include in one or more keywords 120A associated with object 122A (e.g., image of a bluebird) based on probability distribution 906. In the illustrated example, object classification neural network 142 selects second potential keyword 934 (e.g., "bluebird") in response to determining that second potential keyword 934 (e.g., "bluebird") is associated with a highest probability (e.g., 0.7) in probability distribution 906. In another embodiment, object classification neural network 142 selects at least one of potential keywords 934 based on the selected one or more potential keywords having at least a threshold probability (e.g., 0.5) as indicated by probability distribution 906. For example, in response to determining that each of the first potential keyword 934 (e.g., “bird”) and the second potential keyword 934 (e.g., “bluebird”) is associated with a first probability (e.g., 0.5) and a second probability (e.g., 0.7), respectively, that are greater than or equal to a threshold probability (e.g., 0.5), the object classification neural network 142 selects the first potential keyword 934 (e.g., “bird”) and the second potential keyword 934 (e.g., “bluebird”) to include in the one or more keywords 120A.

[0176] refer to Fig. 10A , showing a method 1000 and an example 1050 of inserting a position determination. In certain aspects, one or more operations of the method 1000 are performed by Figure 1 The position neural network 162, the position determination unit 170, the object insertion unit 116, the video stream updater 110, one or more processors 102, the device 130, the system 100, or a combination thereof is executed.

[0177] Method 1000 includes applying a position neural network to a video frame at 1002. In example 1050, position determination unit 170 applies position neural network 162 to video frame 1036 of video stream 136 to generate features 1046, as shown in FIG. Fig. 10B Further described.

[0178] The method 1000 also includes performing segmentation at 1022. For example, the location determination unit 170 performs segmentation based on the features 1046 to generate one or more segmentation masks 1048. In some aspects, performing segmentation includes applying a neural network to the features 1046 according to various techniques to generate segmentation masks. Each segmentation mask 1048 corresponds to an outline of a segmentation of the video frame 1036, the segmentation corresponding to a region of interest, such as a person, a shirt, a pair of pants, a hat, a picture frame, a television, a sports field, one or more other types of regions of interest, or a combination thereof.

[0179] The method 1000 further includes applying a mask at 1024. For example, the position determination unit 170 applies one or more segmentation masks 1048 to the video frame 1036 to generate one or more segmentations 1050. To illustrate, the position determination unit 170 applies a first segmentation mask 1048 to the video frame 1036 to generate a first segmentation corresponding to a shirt, applies a second segmentation mask 1048 to the video frame 1036 to generate a second segmentation corresponding to pants, and so on.

[0180] The method 1000 also includes applying detection at 1026. For example, the location determination unit 170 performs detection to determine whether any of the one or more segmentations 1050 matches a location criterion. To illustrate, the location criterion may indicate a valid insertion location of the video stream 136, such as a person, a shirt, a sports field, etc. In some examples, the location criterion is based on default data, a configuration setting, a user input, or a combination thereof. The location determination unit 170 generates detection data 1052 indicating whether any of the one or more segmentations 1050 matches the location criterion. In a particular aspect, the location determination unit 170 generates the detection data 1052 indicating the at least one segmentation in response to determining that at least one segmentation of the one or more segmentations 1050 matches the location criterion.

[0181] Optionally, in some embodiments, method 1000 includes applying detection to each of one or more objects 182 based on an object type of the one or more objects 182. For example, one or more objects 182 include object 122A having a particular object type. In some embodiments, the location criteria indicate valid locations associated with the object types. For example, the location criteria indicate a first valid location (e.g., shirt, hat, etc.) associated with a first object type (e.g., GIF, clip art, etc.), a second valid location (e.g., wall, playground, etc.) associated with a second object type (e.g., image), etc. In response to determining that object 122A is the first object type, location determination unit 170 generates detection data 1052 indicating at least one segmentation of one or more segmentations 1050 that matches the first valid location. Alternatively, location determination unit 170 generates detection data 1052 indicating at least one segmentation of one or more segmentations 1050 that matches the second valid location in response to determining that object 122A is the second object type.

[0182] In some embodiments, the position criteria indicates that if the one or more objects 182 include the object 122 associated with the keyword 120 and another object associated with the keyword 120 is included in the background of the video frame, then the object 122 is to be included in the foreground of the video frame. For example, in response to determining that the one or more objects 182 include the object 122A associated with the one or more keywords 120A, the video frame 1036 includes the object 122B associated with the one or more keywords 120B in a first position (e.g., background), and at least one of the one or more keywords 120A matches at least one of the one or more keywords 120B, the position determination unit 170 generates detection data 1052 indicating at least one of the one or more segmentations 1050 matching a second position (e.g., foreground) of the video frame 1036.

[0183] The method 1000 also includes determining whether a location is identified at 1008. For example, the location determination unit 170 determines whether the detection data 1052 indicates that any of the one or more segmentations 1050 matches the location criteria.

[0184] The method 1000 includes, in response to determining at 1008 that a location is identified, designating an insertion location at 1010. In example 1050, in response to determining that the detection data 1052 indicates that the segmentation 1050 (e.g., a shirt) satisfies the location criteria, the location determination unit 170 designates the segmentation 1050 as the insertion location 164. In a particular example, the detection data 1052 indicates that a plurality of segmentations 1050 satisfy the location criteria. In some aspects, the location determination unit 170 selects one of the plurality of segmentations 1050 to designate as the insertion location 164. In other examples, the location determination unit 170 selects two or more (e.g., all) of the plurality of segmentations 1050 to add to the one or more insertion locations 164.

[0185] The method 1000 includes skipping insertion at 1012 in response to determining that no location was identified at 1008. For example, in response to determining that the detection data 1052 indicates that no segmentation 1050 matches the location criteria, the location determination unit 170 generates a "no location" output indicating that no insertion location was selected. In this example, in response to receiving the no location output, the object insertion unit 116 outputs the video frame 1036 without inserting any object in the video frame 1036.

[0186] refer to Fig. 10B , diagram 1070 showing illustrative aspects of operations performed by position neural network 162 of position determination unit 170. In particular aspects, position neural network 162 comprises a residual neural network (resnet), such as resnet 152. For example, position neural network 162 comprises a plurality of convolutional layers (e.g., CONV1, CONV2, etc.) and pooling layers (“POOL”) for processing video frames 1036 to generate features 1046.

[0187] refer to Fig.11 , a diagram of a system 1100 including certain devices of embodiment 130 is shown. System 1100 is operable to insert keyword-based objects into a video stream. In certain aspects, Figure 1 The system 100 includes one or more components of the system 1100. For ease of description, Figure 1 Some components of the device 130 are not Fig.11 In some aspects, Figure 1 The device 130 may include Fig.11 One or more of the components of device 130 shown in , one or more additional components, one or more fewer components, one or more different components, or a combination thereof.

[0188] System 1100 includes device 130 coupled to device 1130 and coupled to one or more display devices 1114. In certain aspects, device 1130 includes a computing device, a server, a network device, a storage device, a cloud storage device, a camera, a communication device, a broadcast device, or a combination thereof. In certain aspects, one or more display devices 1114 include a touch screen, a monitor, a television, a communication device, a playback device, a display screen, a vehicle, an XR device, or a combination thereof. In certain aspects, the XR device may include an augmented reality device, a mixed reality device, or a virtual reality device. As an illustrative example, one or more display devices 1114 are described as being external to device 130. In other examples, one or more display devices 1114 may be integrated into device 130.

[0189] Device 130 includes a demultiplexer (demux) 1172 coupled to video stream updater 110. Device 130 is configured to receive media stream 1164 from device 1130. In an example, device 130 receives media stream 1164 from device 1130 via a network. The network may include a wired network, a wireless network, or both.

[0190] The demultiplexer 1172 demultiplexes the media stream 1164 to generate the audio stream 134 and the video stream 136. The demux 1172 provides the audio stream 134 to the keyword detection unit 112 and provides the video stream 136 to the position determination unit 170, the object insertion unit 116, or both. The video stream updater 110 updates the video stream 136 by inserting one or more objects 182 into one or more portions of the video stream 136, as shown in FIG. Figure 1 described.

[0191] In certain aspects, media stream 1164 corresponds to a live media stream. Video stream updater 110 updates video stream 136 of the live media stream and provides video stream 136 (eg, an updated version of video stream 136) to one or more display devices 1114, one or more storage devices, or a combination thereof.

[0192] In some examples, video stream updater 110 selectively updates a first portion of video stream 136, as shown in FIG. Figure 1As described. Video stream updater 110 provides the first portion (e.g., after the selective update) to one or more display devices 1114, one or more storage devices, or a combination thereof. Optionally, in some aspects, device 130 outputs the updated portion of video stream 136 to one or more display devices 1114 while receiving a subsequent portion of video stream 136 included in media stream 1164 from device 1130. Optionally, in some aspects, video stream updater 110 provides audio stream 134 to one or more speakers while providing video stream 136 to one or more display devices 1114.

[0193] refer to Fig.12 , a diagram of a system 1200 is shown. The system 1200 is operable to insert keyword-based objects into a video stream. The system 1200 includes a device 130 coupled to a device 1206 and coupled to one or more display devices 1114.

[0194] In certain aspects, device 1206 includes a computing device, a server, a network device, a storage device, a cloud storage device, a camera, a communication device, a broadcast device, or a combination thereof. Device 130 includes a decoder 1270 coupled to video stream updater 110 and configured to receive encoded data 1262 from device 1206. In an example, device 130 receives encoded data 1262 from device 1206 via a network. The network may include a wired network, a wireless network, or both.

[0195] The decoder 1270 decodes the encoded data 1262 to generate decoded data 1272. In a particular aspect, the decoded data 1272 includes the audio stream 134 and the video stream 136. In a particular aspect, the decoded data 1272 includes one of the audio stream 134 or the video stream 136. In this aspect, the video stream updater 110 obtains the decoded data 1272 (e.g., one of the audio stream 134 or the video stream 136) from the decoder 1270 and obtains the other of the audio stream 134 or the video stream 136 separately from the decoded data 1272 (e.g., from another component or device). The video stream updater 110 selectively updates the video stream 136, as described in reference to FIG. Figure 1 As described, the video stream 136 is provided (eg, after selective updating) to one or more display devices 1114, one or more storage devices, or a combination thereof.

[0196] refer to Fig.13 , a diagram of a system 1300 is shown. The system 1300 is operable to insert keyword-based objects into a video stream. The system 1300 includes a device 130 coupled to one or more microphones 1302 and to one or more display devices 1114.

[0197] As an illustrative example, one or more microphones 1302 are shown as being external to device 130. In other examples, one or more microphones 1302 may be integrated into device 130. Video stream updater 110 receives audio stream 134 from one or more microphones 1302 and obtains video stream 136 separately from audio stream 134. In certain aspects, audio stream 134 includes the user's voice. Video stream updater 110 selectively updates video stream 136, as described with reference to FIG. Figure 1 As described above, the video stream 136 is provided to one or more display devices 1114. In certain aspects, the video stream updater 110 provides the video stream 136 to the display screen of one or more authorized devices (e.g., one or more display devices 1114). For example, the device 130 captures the voice of a performer while the performer is backstage at a concert and sends the enhanced video content (e.g., video stream 136) to the device of the premium ticket holder.

[0198] refer to Fig.14 , a diagram of a system 1400 is shown. The system 1400 is operable to insert keyword-based objects into a video stream. The system 1400 includes a device 130 coupled to one or more cameras 1402 and to one or more display devices 1114.

[0199] As an illustrative example, one or more cameras 1402 are shown as being external to device 130. In other examples, one or more cameras 1402 may be integrated into device 130. Video stream updater 110 receives video stream 136 from one or more cameras 1402 and obtains audio stream 134 separately from video stream 136. Video stream updater 110 selectively updates video stream 136, as described with reference to FIG. Figure 1 As described, the video stream 136 is provided to one or more display devices 1114.

[0200] Fig.15 is a block diagram of illustrative aspects of a system 1500 operable to insert keyword-based objects into a video stream according to some examples of the present disclosure, wherein the one or more processors 102 include an always-on power domain 1503 and a second power domain 1505, such as an on-demand power domain. In some implementations, the first stage 1540 and buffer 1560 of the multi-stage system 1520 are configured to operate in an always-on mode, and the second stage 1550 of the multi-stage system 1520 is configured to operate in an on-demand mode.

[0201] The normally-on power domain 1503 includes a buffer 1560 and a first stage 1540. Optionally, in some implementations, the first stage 1540 includes a position determination unit 170. The buffer 1560 is configured to store at least a portion of the audio stream 134 and at least a portion of the video stream 136 to be accessible for processing by components of the multi-stage system 1520. For example, the buffer 1560 stores one or more portions of the audio stream 134 accessible for processing by components of the second stage 1550, and stores one or more portions of the video stream 136 accessible for processing by components of the first stage 1540, the second stage 1550, or both.

[0202] The second power domain 1505 includes the second stage 1550 of the multi-stage system 1520 and further includes the activation circuit 1530. Optionally, in some embodiments, the second stage 1550 includes the keyword detection unit 112, the object determination unit 114, the object insertion unit 116, or a combination thereof.

[0203] The first stage 1540 of the multi-stage system 1520 is configured to generate at least one of a wake-up signal 1522 or an interrupt 1524 to initiate one or more operations at the second stage 1550. In one example, the wake-up signal 1522 is configured to transition the second power domain 1505 from the low power mode 1532 to the active mode 1534 to activate one or more components of the second stage 1550.

[0204] For example, the activation circuit 1530 may include or be coupled to a power management circuit, a clock circuit, a head switch or foot switch circuit, a buffer control circuit, or any combination thereof. The activation circuit 1530 may be configured to initiate power-on of the second stage 1550, such as by selectively applying or raising the voltage of the power supply of the second stage 1550 of the second power domain 1505, or both. As another example, the activation circuit 1530 may be configured to selectively gate or ungate a clock signal to the second stage 1550, such as to prevent or enable circuit operation without removing power.

[0205] In some embodiments, the first stage 1540 includes the position determination unit 170, and the second stage 1550 includes the keyword detection unit 112, the object determination unit 114, the object insertion unit 116, or a combination thereof. In these embodiments, the first stage 1540 is configured to generate at least one of the wake-up signal 1522 or the interrupt 1524 to initiate operation of the keyword detection unit 112 of the second stage 1550 in response to the position determination unit 170 detecting at least one insertion position 164.

[0206] In some embodiments, the first stage 1540 includes the keyword detection unit 112, and the second stage 1550 includes the position determination unit 170, the object determination unit 114, the object insertion unit 116, or a combination thereof. In these embodiments, the first stage 1540 is configured to generate at least one of a wake-up signal 1522 or an interrupt 1524 to initiate operation of the position determination unit 170, the object determination unit 114, or both of the second stage 1550 in response to the keyword detection unit 112 determining one or more detected keywords 180.

[0207] The output 1552 generated by the second stage 1550 of the multi-stage system 1520 is provided to an application 1554. The application 1554 may be configured to output the video stream 136 to one or more display devices, the audio stream 134 to one or more speakers, or both. For illustration, as illustrative non-limiting examples, the application 1554 may correspond to a voice interface application, an integrated assistance application, a vehicle navigation and entertainment application, a gaming application, a social networking application, or a home automation system.

[0208] By selectively activating the second stage 1550 based on the results of processing data at the first stage 1540 of the multi-stage system 1520, the overall power consumption associated with keyword-based object insertion into a video stream may be reduced.

[0209] refer to Fig.16 , according to some examples of the present disclosure Figure 1 1600 of illustrative aspects of the operation of components of the system 100. The keyword detection unit 112 is configured to receive a sequence 1610 of audio data samples, such as a sequence of continuously captured frames of the audio stream 134, which are illustrated as a first frame (A1) 1612, a second frame (A2) 1614, and one or more additional frames including an Nth frame (AN) 1616 (where N is an integer greater than 2). The keyword detection unit 112 is configured to output a sequence 1620 of sets of detected keywords 180, the sets including a first set (K1) 1622, a second set (K2) 1624, and one or more additional sets including an Nth set (KN) 1626.

[0210] The object determination unit 114 is configured to receive a sequence 1620 of sets of detected keywords 180. The object determination unit 114 is configured to output a sequence 1630 of sets of one or more objects 182, including a first set (O1) 1632, a second set (O2) 1634, and one or more additional sets including an Nth set (ON) 1636.

[0211] The position determination unit 170 is configured to receive a sequence 1640 of video data samples, such as a sequence of consecutively captured frames of the video stream 136, illustrated as a first frame (V1) 1642, a second frame (V2) 1644, and one or more additional frames including an Nth frame (VN) 1646. The position determination unit 170 is configured to output a sequence 1650 of one or more sets of insertion positions 164, including a first set (L1) 1652, a second set (L2) 1654, and one or more additional sets including an Nth set (LN) 1656.

[0212] Object insertion unit 116 is configured to receive sequence 1630, sequence 1640, and sequence 1650. Object insertion unit 116 is configured to output sequence 1660 of video data samples, such as frames of video stream 136, e.g., first frame (V1) 1642, second frame (V2) 1644, and one or more additional frames including frame N (VN) 1646.

[0213] During operation, the keyword detection unit 112 processes the first frame 1612 to generate a first set 1622 of detected keywords 180. In some examples, the keyword detection unit 112 generates the first set 1622 indicating that no keywords were detected (e.g., an empty set) in response to determining that no keywords were detected in the first frame 1612. The position determination unit 170 processes the first frame 1642 to generate a first set 1652 of insertion positions 164. In some examples, the position determination unit 170 generates the first set 1652 indicating detected insertion positions (e.g., an empty set) in response to determining that no insertion positions were detected in the first frame 1642.

[0214] Optionally, in some aspects, the first frame 1612 is time-aligned with the first frame 1642. For example, a particular time (e.g., capture time, playback time, reception time, creation time, etc.) indicated by a first timestamp associated with the first frame 1612 is within a threshold duration of a corresponding time of the first frame 1642.

[0215] The object determination unit 114 processes the first set 1622 of detected keywords 180 to generate a first set 1632 of one or more objects 182. In some examples, in response to determining that the first set 1622 (e.g., an empty set) indicates that no keywords were detected, there are no objects associated with the first set 1622 (e.g., no pre-existing objects and no generated objects), or both, the object determination unit 114 generates the first set 1632 (e.g., an empty set) indicating that there are no objects associated with the first set 1622 of detected keywords 180.

[0216] The object insertion unit 116 processes the first frame 1642 of the video stream 136, the first set 1652 of insertion positions 164, and the first set 1632 of one or more objects 182 to selectively update the first frame 1642. The sequence 1660 includes the selectively updated version of the first frame 1642. As an example, in response to determining that the first set 1652 (e.g., an empty set) indicates that no insertion positions are detected, the first set 1632 (e.g., an empty set) indicates that there are no objects (e.g., no pre-existing objects and no generated objects), or both, the object insertion unit 116 adds the first frame 1642 (without inserting any objects) to the sequence 1660. Alternatively, when the first set 1632 includes one or more objects and the first set 1652 indicates one or more insertion positions 164, the object insertion unit 116 inserts the one or more objects in the first set 1632 at the one or more insertion positions 164 indicated by the first set 1652 to update the first frame 1642 and add the updated version of the first frame 1642 in the sequence 1660.

[0217] Optionally, in some examples, in response to updating the first frame 1642, the object insertion unit 116 updates one or more additional frames of the sequence 1640. For example, the first set 1632 of objects 182 may be inserted into multiple frames of the sequence 1640 such that the objects persist for more than a single video frame during playback. Optionally, in some aspects, in response to updating the first frame 1642, the object insertion unit 116 instructs the keyword detection unit 112 to skip processing of one or more frames of the sequence 1610. For example, the one or more detected keywords 180 may remain the same for at least a threshold count of frames of the sequence 1610 such that an update to a frame of the sequence 1660 corresponds to the same keywords 180 for at least the threshold count of frames.

[0218] In an example, the insertion position 164 indicates a specific position in the first frame 1642, and generating an updated version of the first frame 1642 includes inserting at least one object in the first set 1632 at the specific position in the first frame 1642. In another example, the insertion position 164 indicates specific content (e.g., a shirt) represented in the first frame 1642. In this example, generating the updated version of the first frame 1642 includes performing image recognition to detect a position of the content (e.g., the shirt) in the first frame 1642, and inserting at least one object in the first set 1632 at the detected position in the first frame 1642. In some examples, the insertion position 164 indicates one or more specific image frames (e.g., a threshold count of image frames). To illustrate, in response to updating the first frame 1642, the object insertion unit 116 selects up to the threshold count of image frames after the first frame 1642 in the sequence 1640 as one or more additional frames for insertion. Updating the one or more additional frames includes performing image recognition to detect a position of the content (e.g., the shirt) in each of the one or more additional frames. In response to determining that content is detected in the additional frame, the object insertion unit 116 inserts at least one object at the detected position of the content in the additional frame. Alternatively, in response to determining that content is not detected in the additional frame, the object insertion unit 116 skips insertion in the additional frame and processes the next additional frame for insertion. For illustration, when the content (e.g., shirt) changes position in the additional frame, the inserted object changes position, and the object is not inserted in any additional frame where the content is not detected.

[0219] Such processing continues, including the keyword detection unit 112 processing the Nth frame 1616 of the audio stream 134 to generate the Nth set 1626 of detected keywords 180, the object determination unit 114 processing the Nth set 1626 of detected keywords 180 to generate the Nth set 1636 of objects 182, the position determination unit 170 processing the Nth frame 1646 of the video stream 136 to generate the Nth set 1656 of insertion positions 164, and the object insertion unit 116 selectively updating the Nth frame 1646 of the video stream 136 based on the Nth set 1636 of objects 182 and the Nth set 1656 of insertion positions 164 to generate the Nth frame 1646 of the sequence 1660.

[0220] Fig.17An embodiment 1700 of the device 130 is depicted as an integrated circuit 1702 that includes one or more processors 102. The one or more processors 102 include a video stream updater 110. The integrated circuit 1702 also includes an audio input 1704, such as one or more bus interfaces, to enable the audio stream 134 to be received for processing. The integrated circuit 1702 includes a video input 1706, such as one or more bus interfaces, to enable the video stream 136 to be received for processing. The integrated circuit 1702 includes a video output 1708, such as a bus interface, to enable the transmission of output signals, such as the video stream 136 (e.g., when inserted into a video device). Figure 1 Integrated circuit 1702 implements an implementation of inserting keyword-based objects into a video stream as a component in a system, such as Fig.18 A mobile phone or tablet computer, such as Fig.19 The headphones depicted in Fig. 20 Wearable electronic devices such as Fig.21 The voice controlled speaker system described in Fig. 22 The camera depicted in Fig.23 The XR headsets depicted in Fig.24 The XR glasses depicted in Fig.25 or Fig.26 The vehicle depicted in .

[0221] As an illustrative, non-limiting example, Fig.18 An implementation 1800 is depicted in which device 130 includes a mobile device 1802, such as a phone or tablet. Mobile device 1802 includes one or more microphones 1302, one or more cameras 1402, and a display screen 1804. Components of one or more processors 102, including video stream updater 110, are integrated into mobile device 1802 and are shown using dashed lines to indicate internal components that are not generally visible to a user of mobile device 1802. In a specific example, video stream updater 110 operates to detect user voice activity in audio stream 134 and then processes audio stream 134 to perform one or more operations at mobile device 1802, such as inserting a video stream 136 into video stream 136. Figure 1 one or more objects 182 and launch a graphical user interface or otherwise display the video stream 136 (e.g., with inserted objects 182) at display screen 1804 (e.g., via an integrated “smart assistant” application).

[0222] Fig.19An implementation 1900 is depicted in which the device 130 includes a headphone device 1902. The headphone device 1902 includes one or more microphones 1302, one or more cameras 1402, or a combination thereof. Components of the one or more processors 102, including the video stream updater 110, are integrated into the headphone device 1902. In a specific example, the video stream updater 110 operates to detect user voice activity in the audio stream 134 and then processes the audio stream 134 to perform one or more operations at the headphone device 1902, such as converting the audio stream 134 into a user's voice activity. Figure 1 The one or more objects 182 are inserted into the video stream 136 and the video data corresponding to the video stream 136 (eg, with the inserted objects 182) is sent to a second device (not shown) (such as Fig.11 One or more display devices 1114) for display.

[0223] Fig. 20 An embodiment 2000 is depicted in which the device 130 includes a wearable electronic device 2002, illustrated as a “smart watch”, a video stream updater 110, one or more microphones 1302, one or more cameras 1402, or a combination thereof, integrated into the wearable electronic device 2002. In a specific example, the video stream updater 110 operates to detect user voice activity in the audio stream 134, which is then processed to perform one or more operations at the wearable electronic device 2002, such as converting the user voice activity to the audio stream 134. Figure 1 The wearable electronic device 2002 inserts one or more objects 182 into the video stream 136 and launches a graphical user interface or otherwise displays the video stream 136 (e.g., with the inserted object 182) at the display screen 2004 of the wearable electronic device 2002. For illustration, the wearable electronic device 2002 may include a display screen that is configured to display a notification based on a user's voice detected by the wearable electronic device 2002. In a specific example, the wearable electronic device 2002 includes a tactile device that provides a tactile notification (e.g., a vibration) in response to detecting user voice activity. For example, the tactile notification may enable a user to view the wearable electronic device 2002 to see a displayed notification corresponding to the keyword 180 detected to be spoken by the user (e.g., an object 182 inserted into the video stream 136). Thus, the wearable electronic device 2002 may alert a hearing-impaired user or a user wearing a headset that the user's voice activity is detected.

[0224] Fig.2121 is an embodiment 2100 in which the device 130 includes a wireless speaker and voice activated device 2102. The wireless speaker and voice activated device 2102 may have wireless network connectivity and is configured to perform auxiliary operations. One or more processors 102 including a video stream updater 110, one or more microphones 1302, one or more cameras 1402, or a combination thereof are included in the wireless speaker and voice activated device 2102. The wireless speaker and voice activated device 2102 also includes a speaker 2104. During operation, in response to receiving a verbal command (e.g., one or more detected keywords 180) recognized in the audio stream 134 via operation of the video stream updater 110, the wireless speaker and voice activated device 2102 may perform auxiliary operations, such as inserting one or more objects 182 into the video stream 136 and providing the video stream 136 (e.g., with the inserted objects 182) to another device, such as Fig.11 One or more display devices 1114. For example, in response to receiving one or more detected keywords 180 (e.g., "I'm hungry") after a key phrase (e.g., "Hi Assistant"), the wireless speaker and voice activation device 2102 performs an auxiliary operation, such as displaying an image associated with a restaurant.

[0225] Fig. 22 An embodiment 2200 is depicted in which the device 130 includes a portable electronic device corresponding to a camera device 2202. The video stream updater 110, one or more microphones 1302, or a combination thereof are included in the camera device 2202. In certain aspects, Fig.14 The one or more cameras 1402 include a camera device 2202. During operation, in response to receiving a verbal command (e.g., one or more detected keywords 180) recognized in the audio stream 134 via operation of the video stream updater 110, the camera device 2202 can perform operations in response to the verbal user command, such as inserting one or more objects 182 in the video stream 136 captured by the camera device 2202, and Fig.11 The video stream 136 (eg, with inserted object 182) is displayed at one or more display devices 1114 of the camera device 2202. In some aspects, the one or more display devices 1114 may include a display screen of the camera device 2202, another device, or both.

[0226] Fig.23An embodiment 2300 is depicted in which the device 130 includes a portable electronic device corresponding to an XR headset 2302. The XR headset 2302 may include a virtual reality, mixed reality, or augmented reality headset. The video stream updater 110, one or more microphones 1302, one or more cameras 1402, or a combination thereof are integrated into the XR headset 2302. In certain aspects, user voice activity detection may be performed on an audio stream 134 received from one or more microphones 1302 of the XR headset 2302. The visual interface of the device is positioned in front of the user's eyes to enable an augmented reality, mixed reality, or virtual reality image or scene to be displayed to the user when the XR headset 2302 is worn. In a specific example, the video stream updater 110 inserts one or more objects 182 in the video stream 136, and the visual interface device is configured to display the video stream 136 (e.g., with the inserted objects 182). In particular aspects, the video stream updater 110 provides the video stream 136 (eg, with the inserted object 182 ) to a shared environment displayed by the XR headset 2302 , one or more additional XR devices, or a combination thereof.

[0227] Fig.24 An embodiment 2400 is depicted in which the device 130 includes a portable electronic device corresponding to XR glasses 2402. The XR glasses 2402 may include virtual reality, augmented reality, or mixed reality glasses. The XR glasses 2402 include a holographic projection unit 2404 configured to project visual data onto a surface of a lens 2406 or to reflect visual data off a surface of the lens 2406 and onto a retina of the wearer. The video stream updater 110, one or more microphones 1302, one or more cameras 1402, or a combination thereof are integrated into the XR glasses 2402. The video stream updater 110 may be used to insert one or more objects 182 into the video stream 136 based on one or more detected keywords 180 detected in an audio stream 134 received from the one or more microphones 1302. In a specific example, the holographic projection unit 2404 is configured to display the video stream 136 (e.g., with the inserted object 182). In particular aspects, the video stream updater 110 provides the video stream 136 (eg, with the inserted object 182 ) to a shared environment displayed by the holographic projection unit 2404 , one or more additional XR devices, or a combination thereof.

[0228] In a particular example, the holographic projection unit 2404 is configured to display one or more of the inserted objects 182 indicating the detected audio event. For example, the one or more objects 182 may be superimposed on the user's field of view at a particular location that coincides with the location of the sound source associated with the audio event detected in the audio stream 134. For illustration, the sound may be perceived by the user as emanating from the direction of the one or more objects 182. In an illustrative embodiment, the holographic projection unit 2404 is configured to display the one or more objects 182 (e.g., one or more detected keywords 180) associated with the detected audio event.

[0229] Fig.25 An embodiment 2500 is depicted in which the device 130 corresponds to or is integrated within a vehicle 2502, which is illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). The video stream updater 110, one or more microphones 1302, one or more cameras 1402, or a combination thereof are integrated into the vehicle 2502. User voice activity detection may be performed based on an audio stream 134 received from the one or more microphones 1302 of the vehicle 2502, such as for delivery instructions from an authorized user of the vehicle 2502. In certain aspects, the video stream updater 110 updates the video stream 136 (e.g., assembling instructions) with one or more objects 182 based on one or more detected keywords 180 detected in the audio stream 134, and provides the video stream 136 (e.g., with the inserted objects 182) to the user. Fig.11 One or more display devices 1114. The one or more display devices 1114 may include a display screen of the vehicle 2502, a user device, or both.

[0230] Fig.26 Another embodiment 2600 is depicted in which the device 130 corresponds to or is integrated within a vehicle 2602, which is shown as a car. The vehicle 2602 includes one or more processors 102, which include a video stream updater 110. The vehicle 2602 also includes one or more microphones 1302, one or more cameras 1402, or a combination thereof.

[0231] In some examples, one or more microphones 1302 are positioned to capture speech of an operator of the vehicle 2602. User voice activity detection may be performed based on an audio stream 134 received from one or more microphones 1302 of the vehicle 2602. In some implementations, user voice activity detection may be performed based on an audio stream 134 received from an internal microphone (e.g., one or more microphones 1302), such as for voice commands from authorized passengers. For example, user voice activity detection may be used to detect a voice command from an operator of the vehicle 2602 (e.g., from a parent requesting the location of a sushi restaurant) and ignore the voice of another passenger (e.g., a child requesting the location of an ice cream shop).

[0232] In certain embodiments, the video stream updater 110 inserts one or more objects 182 into the video stream 136 and provides the video stream 136 (e.g., with the inserted objects 182) to the display 2620 in response to determining one or more detected keywords 180 in the audio stream 134. In certain aspects, the audio stream 134 includes speech (e.g., "Sushi is my favorite") of a passenger of the vehicle 2602. The video stream updater 110 determines the one or more detected keywords 180 (e.g., "Sushi") based on the audio stream 134 and determines a first location of the vehicle 2602 based on global positioning system (GPS) data at a first time.

[0233] The video stream updater 110 determines one or more objects 182 corresponding to one or more detected keywords 180, as shown in FIG. Figure 1 Optionally, in some aspects, the video stream updater 110 adaptively classifies the one or more objects 182 associated with the one or more detected keywords 180 and the first location using the adaptive classifier 144. For example, the video stream updater 110 adds the object 122A to the one or more objects 182 (e.g., without classifying the one or more objects 182) in response to determining that the set of objects 122 includes an object 122A (e.g., a sushi restaurant image) associated with one or more keywords 120A (e.g., "sushi", "restaurant") that matches the one or more detected keywords 180 (e.g., "sushi") and is associated with a particular location within a threshold distance of the first location.

[0234] In certain aspects, in response to determining that the set of objects 122 does not include any objects associated with the one or more detected keywords 180 and associated with a location within a threshold distance of the first location, the video stream updater 110 uses the adaptive classifier 144 to classify the one or more objects 182. In certain aspects, classifying the one or more objects 182 includes using the object generation neural network 140 to determine the one or more objects 182 associated with the one or more detected keywords 180 and the first location. For example, the video stream updater 110 retrieves an address of a restaurant within the threshold distance of the first location from the navigation database and applies the object generation neural network 140 to the address and the one or more detected keywords 180 (e.g., "sushi") to generate an object 122A (e.g., clip art indicating a sushi roll and the address) and adds the object 122A to the one or more objects 182.

[0235] In certain aspects, classifying the one or more objects 182 includes determining, using the object classification neural network 142, that the one or more objects 182 are associated with the one or more detected keywords 180 and the first location. For example, the video stream updater 110 processes the object 122A (e.g., an image indicating a sushi roll and an address) using the object classification neural network 142 to determine that the object 122A is associated with the keyword 120A (e.g., “sushi”) and the address. In response to determining that the keyword 120A (e.g., “sushi”) matches the one or more detected keywords 180 and the address is within a threshold distance of the first location, the video stream updater 110 adds the object 122A to the one or more objects 182.

[0236] The video stream updater 110 inserts the one or more objects 182 into the video stream 136 and provides the video stream 136 (e.g., with the inserted objects 182) to the display 2620. For example, the inserted objects 182 are overlaid on the navigation information shown in the display 2620. In certain aspects, the video stream updater 110 determines a second location of the vehicle 2602 at a second time based on the GPS data. In certain embodiments, the video stream updater 110 dynamically updates the video stream 136 based on the change in location of the vehicle 2602. The video stream updater 110 uses the adaptive classifier 144 to classify the one or more second objects associated with the one or more detected keywords 180 and the second location and insert the one or more second objects into the video stream 136.

[0237] In certain aspects, the fleet includes the vehicle 2602 and one or more additional vehicles, and the video stream updater 110 provides the video stream 136 (eg, with the inserted object 182 ) to a display device of one or more vehicles in the fleet.

[0238] refer to Fig. 27 , showing a specific implementation of a method 2700 for inserting a keyword-based object into a video stream. In certain aspects, one or more operations of the method 2700 are performed by Figure 1 The method is performed by at least one of the keyword detection unit 112, the object determination unit 114, the adaptive classifier 144, the object insertion unit 116, the video stream updater 110, one or more processors 102, the device 130, the system 100, or a combination thereof.

[0239] Method 2700 includes obtaining an audio stream at 2702. For example, Figure 1 The keyword detection unit 112 obtains the audio stream 134, as shown in FIG. Figure 1 described.

[0240] Method 2700 also includes detecting one or more keywords in the audio stream at 2704. For example, Figure 1 The keyword detection unit 112 detects one or more detected keywords 180 in the audio stream 134, as shown in FIG. Figure 1 described.

[0241] The method 2700 further includes adaptively classifying one or more objects associated with the one or more keywords at 2706. For example, Figure 1 In response to determining that none of the set of objects 122 stored in the database 150 is associated with the one or more detected keywords 180, the adaptive classifier 144 may classify (e.g., identify, generate, or both via a neural network-based classification) the one or more objects 182 associated with the one or more detected keywords 180. Alternatively, in response to determining that at least one of the set of objects 122 is associated with at least one of the one or more detected keywords 180, Figure 1 The adaptive classifier 144 may designate at least one of the set of objects 122 (eg, not classify the one or more objects 182 ) as the one or more objects 182 associated with the one or more detected keywords 180 .

[0242] Optionally, in some embodiments, adaptively classifying at 2706 includes generating one or more objects based on one or more keywords using an object generation neural network at 2708. For example, Figure 1 The adaptive classifier 144 uses the object generation neural network 140 to generate at least one of the one or more objects 182 based on the one or more detected keywords 180, as shown in FIG. Figure 1 described.

[0243] Optionally, in some embodiments, adaptively classifying at 2706 includes using an object classification neural network at 2710 to determine that one or more objects are associated with one or more detected keywords 180. For example, Figure 1 The adaptive classifier 144 uses the object classification neural network 142 to determine that at least one of the objects 122 is associated with one or more detected keywords 180, and adds at least one of the objects 122 to the one or more objects 182, as shown in FIG. Figure 1 described.

[0244] Method 2700 includes inserting one or more objects into the video stream at 2712. For example, Figure 1 The object insertion unit 116 inserts one or more objects 182 into the video stream 136, as shown in FIG. Figure 1 described.

[0245] Thus, the method 2700 enables the video stream 136 to be enhanced with one or more objects 182 associated with one or more detected keywords 180. The enhancement of the video stream 136 may improve viewer retention, create advertising opportunities, etc. For example, adding objects to the video stream 136 may make the video stream 136 more interesting to the viewer. To illustrate, when the audio stream 134 includes one or more detected keywords 180 associated with the object 122A (e.g., "New York City"), adding the object 122A (e.g., an image of the Statue of Liberty) may increase viewer retention of the video stream 136. In another example, the object 122A may correspond to a visual element representing a related entity associated with the one or more detected keywords 180 (e.g., an image associated with a restaurant in New York, a restaurant that serves food associated with New York, another business that sells New York-related goods or services, a travel website, or a combination thereof).

[0246] Fig. 27 The method 2700 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a neural processing unit (NPU), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig. 27 The method 2700 may be performed by a processor executing instructions, such as reference Fig.28 as described.

[0247] refer to Fig.28 , depicts a block diagram of a particular illustrative device of an embodiment and generally designated 2800. In various embodiments, device 2800 may have more than Fig.28In an illustrative embodiment, device 2800 may correspond to device 130. In an illustrative embodiment, device 2800 may perform the operations described in reference to Figure 1-Figure 27 Describes one or more operations.

[0248] In certain embodiments, device 2800 includes a processor 2806 (eg, a CPU). Device 2800 may include one or more additional processors 2810 (eg, one or more DSPs). In certain aspects, Figure 1 The one or more processors 102 may correspond to processor 2806, processor 2810, or a combination thereof. Processor 2810 may include a speech and music coder-decoder (CODEC) 2808, which includes a speech coder ("vocoder") encoder 2836, a vocoder decoder 2838, a video stream updater 110, or a combination thereof.

[0249] The device 2800 may include a memory 2886 and a CODEC 2834. The memory 2886 may include instructions 109 executable by one or more additional processors 2810 (or processor 2806) to implement the functionality described with reference to the video stream updater 110. The device 2800 may include a modem 2870 coupled to an antenna 2852 via a transceiver 2850.

[0250] In certain aspects, the modem 2870 is configured to receive data from one or more devices and to send data. For example, the modem 2870 is configured to receive data from the device 1130. Fig.11 The modem 2870 is configured to receive the video stream 136 from the video stream updater 110 and provide the video stream 136 to the demultiplexer 1172. Fig.11 In another example, the modem 2870 is configured to receive data from the device 1206. Fig.12 The modem 2870 is configured to receive the encoded data 1262 and provide the encoded data 1262 to the decoder 1270. Fig.13 The one or more microphones 1302 receive the audio stream 134, the one or more cameras 1402 receive the video stream 136, or a combination thereof.

[0251] Device 2800 may include a display 2828 coupled to a display controller 2826. In certain aspects, Figure 1The one or more display devices 1114 of the video stream updater 110 include a display 2828. One or more speakers 2892, one or more microphones 1302, or a combination thereof may be coupled to the CODEC 2834. The CODEC 2834 may include a digital / analog converter (DAC) 2802, an analog / digital converter (ADC) 2804, or both. In a particular embodiment, the CODEC 2834 may receive an analog signal from the one or more microphones 1302, convert the analog signal into a digital signal using the analog / digital converter 2804, and provide the digital signal (e.g., as an audio stream 134) to the voice and music codec 2808. The voice and music codec 2808 may process the digital signal, and the digital signal may be further processed by the video stream updater 110. In a particular embodiment, the voice and music codec 2808 may provide the digital signal to the CODEC 2834. The CODEC 2834 may convert the digital signal into an analog signal using the digital-to-analog converter 2802, and may provide the analog signal to the one or more speakers 2892.

[0252] In particular embodiments, device 2800 may be included in a system-in-package or system-on-chip device 2822. In particular embodiments, memory 2886, processor 2806, processor 2810, display controller 2826, CODEC 2834, and modem 2870 are included in the system-in-package or system-on-chip device 2822. In particular embodiments, input device 2830, one or more cameras 1402, and power supply 2844 are coupled to the system-in-package or system-on-chip device 2822. Additionally, in particular embodiments, as shown in FIG. Fig.28 As shown, the display 2828, input device 2830, one or more cameras 1402, one or more speakers 2892, one or more microphones 1302, antenna 2852, and power supply 2844 are external to the system-in-package or system-on-chip device 2822. In particular embodiments, each of the display 2828, input device 2830, one or more cameras 1402, one or more speakers 2892, one or more microphones 1302, antenna 2852, and power supply 2844 can be coupled to a component of the system-in-package or system-on-chip device 2822, such as an interface or a controller.

[0253] Device 2800 may include a smart speaker, a speaker pole, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disk (DVD) player, a playback device, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aircraft, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a computing device, a communication device, an Internet of Things (IoT) device, an extended reality (XR) device, a base station, a mobile device, or any combination thereof.

[0254] In conjunction with the described embodiments, the apparatus includes a component for obtaining an audio stream. For example, the component for obtaining may correspond to Figure 1 The keyword detection unit 112, the video stream updater 110, the one or more processors 102, the device 130, the system 100, Figure 4 Speech recognition neural network 460, Fig.11 Demultiplexer 1172, Fig.12 Decoder 1270, Fig.15 The buffer 1560, the first stage 1540, the normally open power domain 1503, the second stage 1550, the second power domain 1505, Fig.17 Integrated circuit 1702, audio input 1704, Fig.18 Mobile devices 1802, Fig.19 Headphones device 1902, Fig. 20 Wearable electronic devices 2002, Fig.21 Voice activated device 2102, Fig. 22 Camera device 2202, Fig.23 XR headset 2302, Fig.24 XR glasses 2402, Fig.25 Vehicles 2502, Fig.26 Vehicles 2602, Fig.28 The codec 2834, ADC 2804, speech and music codec 2808, vocoder decoder 2838, processor 2810, processor 2806, device 2800, one or more other circuits 2800 or components configured to obtain an audio stream, or any combination thereof.

[0255] The apparatus also includes a component for detecting one or more keywords in the audio stream. For example, the component for detecting may correspond to Figure 1 The keyword detection unit 112, the video stream updater 110, the one or more processors 102, the device 130, the system 100, Figure 4 A speech recognition neural network 460, a potential keyword detector 462, a keyword selector 464, Fig.15 The first stage 1540, the normally open power domain 1503, the second stage 1550, the second power domain 1505, Fig.17 Integrated circuit 1702, Fig.18 Mobile devices 1802, Fig.19 Headphones device 1902, Fig. 20 Wearable electronic devices 2002, Fig.21 Voice activated device 2102, Fig. 22 Camera device 2202, Fig.23 XR headset 2302, Fig.24 XR glasses 2402, Fig.25 Vehicles 2502, Fig.26 Vehicles 2602, Fig.28 The processor 2810, the processor 2806, the device 2800, one or more other circuits or components configured to detect one or more keywords, or any combination thereof.

[0256] The apparatus further includes a component for adaptively classifying one or more objects associated with one or more keywords. For example, the component for adaptively classifying may correspond to Figure 1 object determination unit 114, adaptive classifier 144, object generation neural network 140, object classification neural network 142, video stream updater 110, one or more processors 102, device 130, system 100, Fig.15 The second stage 1550, the second power domain 1505, Fig.17 Integrated circuit 1702, Fig.18 Mobile devices 1802, Fig.19 Headphones device 1902, Fig. 20 Wearable electronic devices 2002, Fig.21 Voice activated device 2102, Fig. 22 Camera device 2202, Fig.23 XR headset 2302, Fig.24 XR glasses 2402, Fig.25 Vehicles 2502, Fig.26 Vehicles 2602, Fig.28 The processor 2810, the processor 2806, the device 2800, one or more other circuits or components configured to adaptively classify, or any combination thereof.

[0257] The apparatus also includes a component for inserting one or more objects into the video stream. For example, the component for inserting may correspond to Figure 1object insertion unit 116, video stream updater 110, one or more processors 102, device 130, system 100, Fig.15 The second stage 1550, the second power domain 1505, Fig.17 Integrated circuit 1702, Fig.18 Mobile devices 1802, Fig.19 Headphones device 1902, Fig. 20 Wearable electronic devices 2002, Fig.21 Voice activated device 2102, Fig. 22 Camera device 2202, Fig.23 XR headset 2302, Fig.24 XR glasses 2402, Fig.25 Vehicles 2502, Fig.26 Vehicles 2602, Fig.28 The processor 2810, the processor 2806, the device 2800, one or more other circuits or components configured to selectively insert one or more objects into a video stream, or any combination thereof.

[0258] In some embodiments, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 2886) includes instructions (e.g., instructions 109) that, when executed by one or more processors (e.g., one or more processors 2810 or processor 2806), cause the one or more processors to obtain an audio stream (e.g., audio stream 134) and detect one or more keywords (e.g., one or more detected keywords 180) in the audio stream. The instructions, when executed by the one or more processors, also cause the one or more processors to adaptively classify one or more objects (e.g., one or more objects 182) associated with the one or more keywords. When executed by the one or more processors, the instructions also cause the one or more processors to insert the one or more objects into a video stream (e.g., video stream 136).

[0259] Certain aspects of the present disclosure are described below in sets of interrelated examples:

[0260] According to Example 1, a device includes: one or more processors configured to: obtain an audio stream; detect one or more keywords in the audio stream; adaptively classify one or more objects associated with the one or more keywords; and insert the one or more objects into a video stream.

[0261] Example 2 includes the device of example 1, wherein the one or more processors are configured to classify the one or more objects associated with the one or more keywords based on determining that no objects in the set of objects are indicated as being associated with the one or more keywords.

[0262] Example 3 includes the apparatus of Example 1 or Example 2, wherein classifying the one or more objects comprises generating the one or more objects based on the one or more keywords using an object generation neural network.

[0263] Example 4 includes the apparatus of Example 3, wherein the object generation neural network comprises a stacked generative adversarial network (GAN).

[0264] Example 5 includes the apparatus of any of Examples 1 to 4, wherein classifying the one or more objects comprises determining, using an object classification neural network, that the one or more objects are associated with the one or more keywords.

[0265] Example 6 includes the apparatus of Example 5, wherein the object classification neural network comprises a convolutional neural network (CNN).

[0266] Example 7 includes the device of any of Examples 1 to 6, wherein the one or more processors are configured to apply a keyword detection neural network to the audio stream to detect one or more keywords.

[0267] Example 8 includes the apparatus of Example 7, wherein the keyword detection neural network comprises a recurrent neural network (RNN).

[0268] Example 9 includes the apparatus of any of Examples 1 to 8, wherein the one or more processors are configured to: apply a position neural network to a video stream to determine one or more insertion locations in one or more video frames of the video stream; and insert one or more objects at the one or more insertion locations in the one or more video frames.

[0269] Example 10 includes the apparatus of Example 9, wherein the position neural network comprises a residual neural network (resnet).

[0270] Example 11 includes the device of any one of Examples 1 to 10, wherein the one or more processors are configured to: insert a specific object into the foreground or background of a video stream based at least on a file type of the specific object among the one or more objects.

[0271] Example 12 includes the device of any of Examples 1 to 11, wherein the one or more processors are configured to: insert one or more objects into the foreground of the video stream in response to determining that the background of the video stream includes at least one object associated with the one or more keywords.

[0272] Example 13 includes the apparatus of any one of Examples 1 to 12, wherein the one or more processors are configured to perform loop insertion of one or more objects in the video stream.

[0273] Example 14 includes the device of any of Examples 1 to 13, wherein the one or more processors are integrated into at least one of a mobile device, a vehicle, an augmented reality device, a communication device, a playback device, a television, or a computer.

[0274] Example 15 includes the apparatus of any of Examples 1 to 14, wherein the audio stream and the video stream are included in a live media stream received at the one or more processors.

[0275] Example 16 includes the device of Example 15, wherein the one or more processors are configured to receive the live media stream from the network device.

[0276] Example 17 includes the device of Example 16, further comprising a modem, wherein the one or more processors are configured to receive the live media stream via the modem.

[0277] Example 18 includes the device of any of Examples 1 to 17, further comprising one or more microphones, wherein the one or more processors are configured to receive the audio stream from the one or more microphones.

[0278] Example 19 includes the device of any one of Examples 1 to 18, further comprising a display device, wherein the one or more processors are configured to provide the video stream to the display device.

[0279] Example 20 includes the device of any of Examples 1 to 19, further comprising one or more speakers, wherein the one or more processors are configured to output the audio stream via the one or more speakers.

[0280] Example 21 includes the device of any of Examples 1 to 20, wherein the one or more processors are integrated into a vehicle, wherein the audio stream includes speech of a passenger of the vehicle, and wherein the one or more processors are configured to provide the video stream to a display device of the vehicle.

[0281] Example 22 includes the apparatus of Example 21, wherein the one or more processors are configured to: determine a first location of the vehicle at a first time; and adaptively classify one or more objects associated with the one or more keywords and the first location.

[0282] Example 23 includes the device described in Example 22, wherein the one or more processors are configured to: determine a second position of the vehicle at a second time; adaptively classify one or more second objects associated with the one or more keywords and the second position; and insert the one or more second objects into the video stream.

[0283] Example 24 includes the apparatus of any of Examples 21 to 23, wherein the one or more processors are configured to send the video stream to a display device of one or more second vehicles.

[0284] Example 25 includes the device of any of Examples 1 to 24, wherein the one or more processors are integrated into an extended reality (XR) device, wherein the audio stream includes the voice of a user of the XR device, and wherein the one or more processors are configured to provide the video stream to a shared environment displayed by at least the XR device.

[0285] Example 26 includes the device of any of Examples 1 to 25, wherein the audio stream includes the user's voice, and wherein the one or more processors are configured to send the video stream to a display of one or more authorized devices.

[0286] According to Example 27, a method includes: obtaining an audio stream at a device; detecting one or more keywords in the audio stream at the device; selectively applying a neural network at the device to determine one or more objects associated with the one or more keywords; and inserting the one or more objects into a video stream at the device.

[0287] Example 28 includes the method of Example 27, further comprising classifying the one or more objects associated with the one or more keywords based on determining that no objects in the set of objects include any objects indicated as being associated with the one or more keywords.

[0288] Example 29 includes the method of Example 27 or Example 28, wherein classifying the one or more objects comprises generating the one or more objects based on the one or more keywords using an object generation neural network.

[0289] Example 30 includes the method of Example 29, wherein the object generation neural network comprises a stacked generative adversarial network (GAN).

[0290] Example 31 includes the method of any of Examples 27 to 30, wherein classifying the one or more objects comprises determining, using an object classification neural network, that the one or more objects are associated with one or more keywords.

[0291] Example 32 includes the method of Example 31, wherein the object classification neural network comprises a convolutional neural network (CNN).

[0292] Example 33 includes the method of any of Examples 27 to 32, further comprising applying a keyword detection neural network to the audio stream to detect one or more keywords.

[0293] Example 34 includes the method of example 33, wherein the keyword detection neural network comprises a recurrent neural network (RNN).

[0294] Example 35 includes the method of any one of Examples 27 to Example 34, further comprising: applying a position neural network to a video stream to determine one or more insertion locations in one or more video frames of the video stream; and inserting one or more objects at the one or more insertion locations in the one or more video frames.

[0295] Example 36 includes the method of Example 35, wherein the position neural network comprises a residual neural network (resnet).

[0296] Example 37 includes the method of any one of Examples 27 to 36, further comprising: inserting a specific object into the foreground or background of the video stream based at least on a file type of the specific object among the one or more objects.

[0297] Example 38 includes the method of any one of Examples 27 to 37, further comprising: in response to determining that the background of the video stream includes at least one object associated with the one or more keywords, inserting one or more objects into the foreground of the video stream.

[0298] Example 39 includes the method of any one of Examples 27 to 38, further comprising performing loop insertion of one or more objects in the video stream.

[0299] Example 40 includes the method of any one of Examples 27 to 39, wherein the device is integrated into at least one of a mobile device, a vehicle, an augmented reality device, a communication device, a playback device, a television, or a computer.

[0300] Example 41 includes the method of any one of Examples 27 to 40, wherein the audio stream and the video stream are included in a live media stream received at the device.

[0301] Example 42 includes the method of Example 41, further comprising receiving the live media stream from the network device.

[0302] Example 43 includes the method of Example 42, further comprising receiving the live media stream via a modem.

[0303] Example 44 includes the method of any one of Examples 27 to 43, further comprising receiving an audio stream from one or more microphones.

[0304] Example 45 includes the method of any one of Examples 27 to 44, further comprising: providing the video stream to a display device.

[0305] Example 46 includes the method of any one of Examples 27 to 45, further comprising providing the audio stream to one or more speakers.

[0306] Example 47 includes the method of any one of Examples 27 to Example 46, further comprising: providing the video stream to a display device of the vehicle, wherein the audio stream includes the voice of a passenger of the vehicle.

[0307] Example 48 includes the method of Example 47, further comprising: determining a first position of the vehicle at a first time; and adaptively classifying one or more objects associated with the one or more keywords and the first position.

[0308] Example 49 includes the method described in Example 48, further comprising: determining a second position of the vehicle at a second time; adaptively classifying one or more second objects associated with the one or more keywords and the second position; and inserting the one or more second objects into the video stream.

[0309] Example 50 includes the method of any one of Examples 47 to 49, further comprising: sending the video stream to a display device of one or more second vehicles.

[0310] Example 51 includes the method of any one of Examples 27 to 50, further comprising providing the video stream to a shared environment displayed by at least one extended reality (XR) device, wherein the audio stream includes a voice of a user of the XR device.

[0311] Example 52 includes the method of any one of Examples 27 to 51, further comprising sending the video stream to a display of one or more authorized devices, wherein the audio stream includes the user's voice.

[0312] According to Example 53, a device includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform any one of the methods of Examples 27 to 52.

[0313] According to Example 54, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the method described in any one of Examples 27 to 52.

[0314] According to Example 55, an apparatus includes components for performing a method as described in any one of Examples 27 to 52.

[0315] According to Example 56, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to: obtain an audio stream; detect one or more keywords in the audio stream; adaptively classify one or more objects associated with the one or more keywords; and insert the one or more objects into a video stream.

[0316] According to Example 57, an apparatus includes: a component for obtaining an audio stream; a component for detecting one or more keywords in the audio stream; a component for adaptively classifying one or more objects associated with the one or more keywords; and a component for inserting the one or more objects into a video stream.

[0317] Those skilled in the art will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, and such implementation decisions should not be interpreted as resulting in a departure from the scope of the present disclosure.

[0318] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, compact disk read-only memory (CD-ROM), or any other form of non-temporary storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0319] The previous description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but should conform to the broadest possible scope consistent with the principles and novel features as defined by the appended claims.

Claims

1. A device comprising: One or more processors configured to: Get the audio stream; detecting one or more keywords in the audio stream; adaptively classifying one or more objects associated with the one or more keywords; and The one or more objects are inserted into the video stream.

2. The device according to claim 1, wherein: The one or more processors are configured to classify the one or more objects associated with the one or more keywords based on determining that no objects in the set of objects are indicated as being associated with the one or more keywords.

3. The device according to claim 1, wherein: Classifying the one or more objects includes generating the one or more objects based on the one or more keywords using an object generation neural network.

4. The device according to claim 3, wherein: The object generation neural network includes a stacked generative adversarial network (GAN).

5. The device according to claim 1, wherein: Classifying the one or more objects includes determining that the one or more objects are associated with the one or more keywords using an object classification neural network.

6. The device according to claim 5, wherein: The object classification neural network includes a convolutional neural network (CNN).

7. The device according to claim 1, wherein: The one or more processors are configured to apply a keyword detection neural network to the audio stream to detect the one or more keywords.

8. The device according to claim 7, wherein: The keyword detection neural network includes a recurrent neural network (RNN).

9. The device according to claim 1, wherein: The one or more processors are configured to: applying a position neural network to the video stream to determine one or more insertion locations in one or more video frames of the video stream; and The one or more objects are inserted at the one or more insertion locations in the one or more video frames.

10. The device according to claim 9, wherein: The position neural network includes a residual neural network (resnet).

11. The device according to claim 1, wherein: The one or more processors are configured to insert a particular object among the one or more objects into a foreground or background of the video stream based at least on a file type of the particular object.

12. The device according to claim 1, wherein: The one or more processors are configured to, in response to determining that a background of the video stream includes at least one object associated with the one or more keywords, insert the one or more objects into a foreground of the video stream.

13. The apparatus according to claim 1, wherein: The one or more processors are configured to perform loop insertion of the one or more objects in the video stream.

14. The apparatus according to claim 1, wherein: The one or more processors are integrated into at least one of a mobile device, a vehicle, an augmented reality device, a communication device, a playback device, a television, or a computer.

15. The apparatus according to claim 1, wherein: The audio stream and the video stream are included in a live media stream received at the one or more processors.

16. The device according to claim 15, wherein: The one or more processors are configured to receive the real-time media stream from a network device.

17. The device of claim 16, further comprising a modem, wherein: The one or more processors are configured to receive the real-time media stream via the modem.

18. The device of claim 1, further comprising one or more microphones, wherein: The one or more processors are configured to receive the audio stream from the one or more microphones.

19. The device according to claim 1, further comprising a display device, wherein: The one or more processors are configured to provide the video stream to the display device.

20. The device of claim 1, further comprising one or more speakers, wherein The one or more processors are configured to output the audio stream via the one or more speakers.

21. The apparatus of claim 1, wherein: The one or more processors are integrated in a vehicle, wherein the audio stream includes speech of an occupant of the vehicle, and wherein the one or more processors are configured to provide the video stream to a display device of the vehicle.

22. The apparatus according to claim 21, wherein The one or more processors are configured to: determining a first position of the vehicle at a first time; and The one or more objects associated with the one or more keywords and the first location are adaptively classified.

23. The apparatus of claim 22, wherein: The one or more processors are configured to: determining a second position of the vehicle at a second time; adaptively classifying one or more second objects associated with the one or more keywords and the second location; as well as The one or more second objects are inserted into the video stream.

24. The apparatus of claim 21, wherein: The one or more processors are configured to transmit the video stream to a display device of one or more second vehicles.

25. The apparatus of claim 1, wherein: The one or more processors are integrated in an extended reality (XR) device, wherein the audio stream includes a voice of a user of the XR device, and wherein the one or more processors are configured to provide the video stream to a shared environment displayed by at least the XR device.

26. The apparatus of claim 1, wherein: The audio stream includes a user's voice, and wherein the one or more processors are configured to send the video stream to a display of one or more authorized devices.

27. A method comprising: Get the audio stream at the device; detecting, at the device, one or more keywords in the audio stream; adaptively classifying, at the device, one or more objects associated with the one or more keywords; as well as The one or more objects are inserted into a video stream at the device.

28. The method according to claim 27, wherein: Classifying the one or more objects includes generating the one or more objects based on the one or more keywords using an object generation neural network.

29. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Get the audio stream; detecting one or more keywords in the audio stream; adaptively classifying one or more objects associated with the one or more keywords; and The one or more objects are inserted into the video stream.

30. An apparatus comprising: A component for obtaining an audio stream; means for detecting one or more keywords in the audio stream; means for adaptively classifying one or more objects associated with the one or more keywords; as well as Means for inserting the one or more objects into a video stream.

Images