Audio-visual analysis for object rendering at capture

Abstract

A system and method for generating an automatic audiovisual analysis for object rendering at the time of capture is disclosed. One example provides a method of processing audiovisual content. The method includes: receiving content including a plurality of audio frames and a plurality of video frames; classifying each of the plurality of audio frames into a plurality of audio classifications; and classifying each of the plurality of video frames into a plurality of video classifications. The method includes: processing a plurality of audio frames based on respective audio classifications; and processing the plurality of video frames based on the respective video classifications. Each audio classification is processed with a different audio processing operation, and each video classification is processed with a different video processing operation. The method includes generating an audio / video representation of the content by merging the processed plurality of audio frames and the processed plurality of video frames.

Description

[0001] 1. Cross-reference to related applications

[0002] This application claims the benefit of priority to PCT patent application publication No. PCT / CN2022 / 118437, filed on September 13, 2022, and U.S. Provisional Patent Application No. 63 / 449,726, filed on March 3, 2023, both of which are incorporated herein by reference in their entirety. 2. Technical Field

[0003] Various example embodiments generally relate to media processing of multimedia content. In particular, example embodiments relate to a system, method, or computer program product configured for generating an automated audiovisual analysis for processing and rendering. Summary of the Invention

[0004] Various embodiments are disclosed herein for rendering objects captured within audiovisual data. The audiovisual portion of the content is divided into visual frames (e.g., image frames, video frames) and audio frames. The audio frames are analyzed to identify audio objects within the audio frames. The visual frames are analyzed to identify visual objects within the visual frames. The audio objects and visual objects are classified based on the detected objects and scenes. The classification can indicate, for example, whether the audiovisual content is indoors or outdoors, whether the captured content includes sports, people, landscapes, and can also indicate specific object types, etc. The audio frames and visual frames are processed separately using the detected objects and classifications and recombined to create the final audiovisual output.

[0005] Various examples, instances, and aspects of the present disclosure provide a method for classifying and categorizing objects based on their contribution to the clarity, immersion, and spatiality of the overall audio. By classifying and categorizing both audio and visual frames, the audio and visual aspects of the audiovisual content can be processed separately, thereby improving the quality of the final audiovisual output.

[0006] According to an example embodiment, a method for processing audiovisual content is provided. The method includes receiving content including a plurality of audio frames and a plurality of video frames; classifying each of the plurality of audio frames into a plurality of audio classifications; and classifying each of the plurality of video frames into a plurality of video classifications. The method includes processing the plurality of audio frames based on the corresponding audio classifications; and processing the plurality of video frames based on the corresponding video classifications. Each audio classification is processed using a different audio processing operation, and each video classification is processed using a different video processing operation. The method includes generating an audio / video representation of the content by merging the processed plurality of audio frames and the processed plurality of video frames.

[0007] According to another exemplary embodiment, a non-transitory computer-readable medium storing instructions is provided that, when executed by an electronic processor, causes the electronic processor to perform operations comprising: receiving content comprising a plurality of audio frames and a plurality of video frames; classifying each of the plurality of audio frames into a plurality of audio classifications; and classifying each of the plurality of video frames into a plurality of video classifications. The instructions include processing the plurality of audio frames based on the respective audio classifications; and processing the plurality of video frames based on the respective video classifications. Each audio classification is processed using a different audio processing operation, and each video classification is processed using a different video processing operation. The instructions include generating an audio / video representation of the content by merging the processed plurality of audio frames and the processed plurality of video frames.

[0008] According to yet another example embodiment, a video system for processing audiovisual content is provided. The system includes a processor for performing processing of the audiovisual content. The processor is configured to receive content including a plurality of audio frames and a plurality of video frames; classify each of the plurality of audio frames into a plurality of audio classifications; and classify each of the plurality of video frames into a plurality of video classifications. The processor is configured to process the plurality of audio frames based on the corresponding audio classifications; and to process the plurality of video frames based on the corresponding video classifications. Each audio classification is processed using a different audio processing operation, and each video classification is processed using a different video processing operation. The processor is configured to generate an audio / video representation of the content by merging the processed plurality of audio frames and the processed plurality of video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent from the following detailed description and accompanying drawings, by way of example, in which:

[0010] Figure 1 Depicts an example process of a video / image transmission pipeline.

[0011] Figure 2 An example audio-visual analysis-based rendering system is depicted.

[0012] Figure 3 An example visual analytics system is depicted.

[0013] Figure 4 Depicted are example video frames including noise.

[0014] Figure 5 Depicts the after resizing Figure 4 Example video frames of .

[0015] Figure 6 A block diagram depicts an example method for detecting scene changes.

[0016] 7A to 7C An example 3×3 neighborhood of pixels is depicted.

[0017] Figure 8 Another block diagram of an example method for detecting scene changes is depicted.

[0018] Figure 9 A block diagram depicts an example method for processing audiovisual content. DETAILED DESCRIPTION

[0019] The present disclosure and its aspects may be embodied in various forms, including: hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​the various aspects of the present disclosure and is not intended to limit the scope of the disclosure in any way.

[0020] Example video / image transfer pipeline

[0021] Figure 1 An example process of a video transmission pipeline 100 according to an embodiment is depicted, which shows the various stages from video / image capture to video / image content display. An image generation block 105 can be used to capture or generate a sequence of video / image frames 102. Frames 102 can be digitally captured (e.g., by a digital camera, by a cell phone camera, etc.) or generated by a computer (e.g., using computer animation) to provide video and / or image data 107. Audio data can also be provided as part of data 107 and associated with the video / image data. Alternatively, frames 102 can be captured on film by a film camera. The film can then be converted into a digital format to provide video / image data 107.

[0022] Audio data can include channel-based audio that assigns sound sources to specific channels (e.g., stereo, 5.1, 7.1, etc.), object-based audio that assigns sound objects to specific channels, and any associated metadata. For example, an audio object can include one or more audio signals (e.g., an audio data stream or a data stream encoding the audio essence, also referred to herein as an "audio object signal") and associated metadata. The metadata describes one or more characteristics of the audio signal (e.g., information metadata) or indicates how the audio signal should be processed by downstream processes (such as rendering) (e.g., control metadata).

[0023] In some embodiments, the metadata includes audio object position data, audio object size data, audio object gain data, audio object trajectory data, content type data (e.g., dialogue, effects, etc.), rendering constraint data, etc. Metadata corresponding to discrete audio sources in the audio signal (also referred to as parametric source descriptions) includes a spatial audio description for the corresponding source (e.g., source position / 3D coordinates and source size).

[0024] Some audio objects may be static, while other audio objects may have time-varying metadata. Time-varying audio objects may move, change size, and / or have other characteristics that change over time.

[0025] When an audio object is monitored or played back in a reproduction environment, the audio object can be rendered based on positional metadata using the reproduction speakers present in the reproduction environment, rather than outputting the audio object to predetermined physical channels as in traditional channel-based systems (such as Dolby 5.1 and Dolby 7.1).

[0026] While some metadata associated with audio objects may be received concurrently with the audio data, additional metadata may be generated during the audiovisual analysis process described herein.

[0027] During the production phase 110, a processor may process the data 107 to provide a visual video / image production stream 112. The data from the video / image production stream 112 may be provided to a processor (or one or more processors, such as a central processing unit (CPU)) at a post-production block 115 for post-production editing. Post-production editing may be performed by a user of the video transmission pipeline 100 (e.g., the creator of the captured frame 102). Post-production editing at block 115 may include, for example, adjusting or modifying the color or brightness in specific areas of the image to enhance the image quality or achieve a specific look based on the video creator's creative intent. This portion of post-production editing is sometimes referred to as "color timing" or "color grading." Other edits may be performed at block 115 (e.g., scene selection and sequencing, image cropping, the addition of computer-generated visual effects, artifact removal, etc.) to produce a "final" version 117 of the work for distribution. In some examples, the operations performed at block 115 include detecting and classifying objects within the data 107. During post-production editing 115 , the video and / or images may be viewed on a reference display 125 .

[0028] After post-production 115, the data of the final version 117 can be transmitted to an encoding block 120 for further downstream transmission to decoding and playback devices such as televisions, set-top boxes, movie theaters, etc. In some embodiments, the encoding block 120 may include audio and video encoders (such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats) to generate an encoded bitstream 122. In the receiver, the encoded bitstream 122 is decoded by a decoding unit 130 to generate a corresponding decoded signal 132 representing a replica or a close approximation of the signal 117. The receiver can be attached to a target display 140, which can have slightly different or completely different characteristics than the reference display 125. In this case, a display management (DM) block 135 can be used to map the decoded signal 132 to the characteristics of the target display 140 by generating a display mapping signal 137. Depending on the embodiment, the decoding unit 130 and the display management block 135 can include separate processors or can be based on a single integrated processing unit.

[0029] The codecs used in the encoding block 120 and / or decoding block 130 are capable of video / image data processing and compression / decompression. Compression is used in the encoding block 120 to reduce the size of the corresponding file(s) or stream(s). The decoding process performed by the decoding block 130 typically involves decompressing the received video / image data file(s) or stream(s) into a form that can be used for playback and / or further editing. Example encoding / decoding operations that can be used in the encoding block 120 and decoding block 130 according to various embodiments are described in more detail below.

[0030] Audiovisual rendering system

[0031] Figure 2 A block diagram of an audiovisual analysis-based object rendering system 200 is illustrated. The operations of the described audiovisual analysis-based object rendering system 200 can be performed by an electronic processor of the post-production block 115. The audiovisual video input is separated into visual frames and audio frames by a visual frame extraction block 202 and an audio frame extraction block 204, respectively. As referred to herein, visual frames include image frames or video frames captured by a camera. As referred to herein, audio frames include audio data captured by a microphone, the audio data being associated with the visual frames (e.g., captured simultaneously with the visual frames).

[0032] The visual frames are provided to a visual scene / object classifier block 206 for visual scene and / or object classification. The audio frames are provided to an audio scene / object classifier block 208 for audio scene and / or object classification.

[0033] The scene class can contain several different classifiers, each with a different purpose. For example, one scene classifier can be used to analyze the captured location, such as outdoor location, indoor location, or traffic type. Another scene classifier can be used to distinguish the captured content type, such as sports, food, landscape, people, etc. Classifiers can also perform object localization and segmentation. The output classes generated by the classification of the audio classifier and the visual classifier can be the same classifier, different classifiers, or related classifiers. For example, for a bird song object, the audio class can be "bird song", but the visual class can be "tree", "bird", or "bird cage", etc.

[0034] In some instances, in addition to classification, audio objects are assigned to four main categories based on the perceived importance of the audio. These categories can include: "very important objects" (e.g., the first category), "high importance objects" (e.g., the second category), "important objects" (e.g., the third category), and "low importance objects" (e.g., the fourth category). Objects assigned as "very important objects" are objects that help improve the clarity and spatiality of the audio data. For example, detected speech can be classified as a "very important object," as well as any object that provides high-level information that can give the audio an increased sense of spatiality. Objects assigned as "high-level importance objects" include non-speech sounds that contribute to audio immersion, such as animal sounds and directional objects with clear time-frequency patterns. Objects assigned as "important objects" include background sounds that contain environmental information about the scene. These objects can be attenuated to improve speech clarity with only a small impact on the sound scene immersion. Objects assigned as "low-level importance objects" include environmental sounds that can be highly attenuated to improve the clarity of the audio. Thus, audio objects in one category (e.g., "important objects") may be attenuated at a first level, while audio objects in a second category (e.g., "low-importance objects") may be attenuated at a second level that is higher than the first level, and vice versa. While specific categories are provided, these categories are merely examples. Fewer or more categories may be provided to provide an order of importance for categorizing audio objects.

[0035] In some embodiments, object classification is performed automatically by an electronic processor. For example, the electronic processor may implement a machine learning model configured to receive and classify audio data. Each classification may be assigned a specific category level that the electronic processor automatically associates with the audio object. For example, when the electronic processor detects that an audio component of the audio data is "speech," the electronic processor classifies the audio object as speech and automatically categorizes the audio object as a "very important object." In another embodiment, the electronic processor receives input from an editor of the audio frame indicating the classification and categorization of the audio frame.

[0036] The visual output class and the audio output class are combined in the audio-visual combination block 210. The combination of the visual output class and the audio output class can be based on deep learning, rules, etc. The combined audio-visual information is used as context information for further audio and visual processing, which will be described in more detail below. The combined audio-visual information and the original visual frame are provided to the visual processing block 212 for visual processing.

[0037] The combined audio-visual information and the original audio frame are provided to the audio object separation block 214. The audio object separation block 214 separates the objects classified by the audio scene / object classifier block 208 from the original audio frame. The separated audio objects and the combined audio-visual information are provided to the object selection / metadata generation block 216. The object selection / metadata generation block 216 generates metadata describing the different audio objects. The audio rendering block 218 renders the audio objects.

[0038] The classification of audio objects may affect the processing of audio frames at the audio object separation block 214, the object selection / metadata generation block 216, and / or the audio rendering block 218. For example, audio data may be assigned to specific speakers based on their classification. Audio data associated with "high-importance objects" may be expanded or contracted to improve spatiality. Audio data associated with "low-importance objects" may be suppressed to improve overall audio clarity while preserving more important audio objects.

[0039] The processed video from visual processing block 212 and the rendered audio from audio rendering block 218 are provided to audio-visual composition block 220. Audio-visual composition block 220 combines the rendered audio and processed video to generate a video output.

[0040] Figure 3 Illustrated is a block diagram of an example visual analysis system 300. The operations described with respect to the visual analysis system 300 may be performed by the visual scene / object classifier block 206, the audio-visual combination block 210, the visual processing block 212, or a combination thereof.

[0041] The visual frames from the visual frame extraction block 202 are received by the color richness detection block 302. The color richness detection block 302 is configured to filter the received video data and remove frames containing only monochrome or frames whose color richness is below a set threshold. In this way, the color richness detection block 302 removes frames that are captured when, for example, the camera lens is obscured or when the camera is too close to an object, resulting in a loss of environmental or object outline information.

[0042] In one example of detecting color distance within a visual frame, the color distance between two RGB pixels is calculated according to Equation 1:

[0043]

[0044] Where R, G, and B are values ​​between 0 and 255 for the red, green, and blue channels, respectively. To simplify Equation 1, these terms can be redefined as:

[0045] ΔR=(R2-R1) 2

[0046] ΔG=(G2-G1) 2

[0047] ΔB=((B2-B1) 2

[0048] Since humans have different sensitivities to different RGB color values, different color components can be weighted differently. Therefore, Equation 1 can be rewritten as Equation 2:

[0049]

[0050] In one embodiment, the weight value is w R =2,w G =4, and w B =3, because people are more sensitive to changes in the green channel and less sensitive to changes in the red channel. In another embodiment, w is modified based on different color types. R 、w G , and w B For example, consider that the weighted value is defined as:

[0051]

[0052] Then, Equation 2 is written as Equation 3:

[0053]

[0054] In Equation 4, the green channel has a fixed weight, while the red and blue channels have dynamic weights based on the red channel value. For example, when the red channel value is close to 255, the red channel gets the maximum weight (w R =~3), and when the red channel value is close to 0, the blue channel gets the maximum weight (w B =~3).

[0055] In some instances, calculating the weighting function for all pixels in a visual frame is computationally demanding. Therefore, to reduce computational requirements, color distances can be calculated for a limited set of dominant colors. Statistical analysis can be performed on the image to calculate the most representative colors in the image. In this example, the color richness of the visual frame is obtained by Equation 4:

[0056]

[0057] where N is the number of most representative colors and τ is the RGB vector of these colors, given by Equation 5:

[0058] τ = [RGB1, RGB2, ..., RGB N ] [Equation 5]

[0059] When cr < 100, the visual frame contains little color information.

[0060] In the case where the visual frame is characterized by noise, the most representative color may be mistakenly identified. For example, Figure 4 An example of a video frame 400 with scattered colors (indicated by a white dot) is provided. The video frame has a black background that is selected as the most representative color. Even though the video frame 400 has no color richness, the video frame 400 has a cr value of approximately 150 due to the white dot. To overcome this error, the video frame 400 can be resized to a smaller resolution, such as Figure 5 . The smaller resolution reduces the effect of noise. Video frame 500 has a cr value of approximately 43, and the noisy frame is effectively discarded. The output of the color richness detection block 302 can be an indication of whether the color richness of the visual frame cr is greater than or equal to a color richness threshold.

[0061] return Figure 3 , the output of the color richness detection block 302 is provided to a primary object detection block 304 and a scene cut detection block 306. The primary object detection block 304 is configured to identify the location of the primary object of interest within the visual frame. For example, face and / or body detection methods can be performed to segment each person within the visual frame, estimate the location of each person within the image coordinate system of the visual frame, the orientation of the face relative to the camera reference system, and the distance of the face from the camera. Primary objects are not limited to faces and bodies, and can also include objects such as animals, plants, buildings, or other objects in the video frame. The primary object detection block 304 outputs an indication of the primary object and data associated with the primary object.

[0062] In an example where the primary object is a person's head, the detected head position and head pose can be used as input for audio processing. For example, the size of a face in a visual frame can be used to control the volume of an audio object associated with the face in an audio mix (e.g., audio associated with a person's voice). If the face is closer to the camera in a second visual frame, the volume of the associated audio object is increased for the second visual frame. If the face is farther from the camera in the second visual frame, the volume of the associated audio object is reduced for the second visual frame. In some instances, the position of the face within the visual frame is mentioned when spatializing the associated audio object. Facial orientation can also be referred to as voice spatialization and reverberation. This concept can be applied to other video objects. As the video object moves frame by frame within the video, the image coordinates of the video object are used to spatialize the corresponding audio object.

[0063] The scene cut detection block 306 determines a frame-by-frame change in an object of interest (e.g., a scene change). For example, a first visual frame may have a first object of interest (e.g., a person), and a second visual frame may have a second object of interest (e.g., an animal). The scene cut detection block 306 outputs an indication of the change in the object of interest.

[0064] Three example scene change detection algorithms include grayscale-based, edge-based, and motion-based algorithms. Grayscale-based scene detection algorithms use the difference between the grayscale values ​​of a reference frame image and the current frame to determine whether a scene change has occurred. For example, the difference between the grayscale histograms of the two frames is compared. Grayscale-based scene detection algorithms have relatively low computational complexity but may encounter errors when video objects move between frames. Edge-based scene detection methods compare the edge contours of corresponding objects between the current frame and the reference frame. This algorithm effectively detects soft cuts such as dissolves, fade-ins, and fade-outs. However, for detecting abrupt scene changes, they have no significant advantage over grayscale-based detection methods and have increased complexity compared to grayscale-based detection methods. Motion-based scene change detection algorithms detect discontinuities in the motion of video objects before and after a scene change. The average of the residual error of the motion estimation is used as the decision criterion for scene changes. However, reliable motion-based scene change algorithms typically have high computational complexity.

[0065] To alleviate these problems, embodiments described herein provide a hybrid scene change detection algorithm that uses luminance (Y), chrominance (U), and chroma (V) of a visual frame. Figure 6 An example method 600 is provided for detecting scene changes, which is performed at the scene cut detection block 306. At step 602, the method 600 includes converting an RGB color frame to a YUV frame.

[0066] At step 604, method 600 includes converting the Y component into a feature frame using binary weighting. For example, Figure 7A An example 3*3 neighborhood of pixels at c(x,y) is shown. The position of each neighboring pixel is "p" and its corresponding value is denoted as g(p). Figure 7A In , the position p of each pixel is labeled clockwise from 0 to 7. That is, g(0)=7, g(1)=3, g(2)=1, g(3)=2, g(4)=4, g(5)=7, g(6)=9, and g(7)=8. Figure 7B The binary map obtained by comparing g(p) and g(c) is shown. Figure 7C The decimal value of each neighborhood pixel is shown. Figure 7B The binary mapping of is obtained according to Equation 6:

[0067]

[0068] in,

[0069] At step 606, method 600 includes generating a histogram of the feature frames. For example, Equation 7 provides a method for calculating the absolute sum difference between the current frame t and the previous frame t-1.

[0070]

[0071] Among them, H t are the histogram values.

[0072] At step 608, method 600 includes determining whether a scene change occurs based on the histogram. For example, a threshold is then set for detecting a scene change, as provided by Equation 8:

[0073]

[0074] For the first frame, C1(t) may be set to 1. The value C1(t) may be the output of the scene cut detection block 306. Although the method 600 is described as calculating the difference between the current frame and the previous frame, the method 600 may instead calculate the difference between the current frame and a future frame.

[0075] Figure 8 Another example method 800 for detecting scene changes is provided. At step 802, the method 800 includes converting an RGB color frame into a YUV frame. At step 804, the method 800 includes calculating an average YUV value for the current frame. For example, the average YUV value for the current frame is determined according to equations 9-11:

[0076]

[0077] At step 806, the method 800 includes calculating the difference in average YUV values ​​between the current frame and the subsequent frame. For example, the difference between frame t and frame t-1 can be calculated using Equation 12:

[0078]

[0079] in:

[0080] D Y =|mean t (Y)-mean t-1 (Y)|

[0081] D U =|mean t (U)-mean t-1 (U)|

[0082] D V =|mean t (V)-mean t-1 (V)|

[0083] At step 808, method 800 includes determining whether a scene change has occurred based on the difference. For example, a threshold is then set for detecting a scene change, as provided by Equation 13:

[0084]

[0085] The value C2(t) may be the output of the scene cut detection block 306. Although the method 800 is described as calculating the difference between the current frame and the previous frame, the method 800 may instead calculate the difference between the current frame and a future frame.

[0086] In another example, multiple historical frames or future frames can be analyzed to make the detection of scene changes more robust. In this example, the average YUV value of k historical frames is provided as:

[0087]

[0088] And the relative difference is provided as:

[0089]

[0090] In this example, the threshold can be set as provided in Equation 18:

[0091]

[0092] In some embodiments, a combination of parameters C1(t), C2(t), and C3(t) and their corresponding thresholds are used to detect scene changes. For example, a scene change can be detected when C1(t) = 1, C2(t) = 1, and C3(t) = 1. In other instances, a scene change is detected when at least two of C1(t), C2(t), and C3(t) = 1. In such an embodiment, the scene change detection block 306 can effectively detect soft cuts such as dissolves, fade-ins, and fade-outs, as well as abrupt scene cut detection with minimal additional computational complexity. As an example of potential thresholds for scene detection, th1 = 2500, th2 = 80, and th3 = 0.3.

[0093] The outputs of the primary object detection block 304 and the scene cut detection block 306 (e.g., whether a scene change is occurring) are provided to a scene / object classifier block 308. In some instances, the output of the color richness detection block 302 is also provided to the scene / object classifier block 308. The scene / object classifier block 308 identifies the scene content, content type, and / or primary video objects in the scene captured within the visual frame and classifies them. In some instances, the frequency with which the scene / object classifier block 308 operates depends on the frequency of scene changes. For example, the scene / object classifier block 308 can classify the content of the scene in response to a scene change occurring. In another instance, when no scene change is detected, the scene / object classifier block 308 operates at set time intervals.

[0094] In some examples, the classification results for each visual frame or scene are weighted based on an analysis associated with the scene. For example, the color richness value determined by the color richness detection block 302 can be provided as a weighting value to the scene / object classifier block 308. Scenes with high color richness can be given a higher weight than scenes with relatively low color richness.

[0095] Thus, the audiovisual analysis-based object rendering system 200 provides a system and process for identifying and classifying objects detected in both visual and audio frames. In some embodiments, visual classes are defined to match audio scenes or objects. Visual classes are mapped to audio classes to create linked classes that define both visual and audio frames. For example, voice audio can be linked to a detected face.

[0096] The visual processing block 310 receives the classified visual frames and renders the video based on the object classification. Different classification results may lead to different processing strategies. For example, the video processing block 212 and the audio rendering block 218 independently process the video content and audio content for different content types, different scenes and different objects. For example, automatic scaling can be performed on a specific visual object based on its classification. Different audio processing, such as level adjustment, equalization and spatialization operations, can be performed on different audio objects based on the classification or classification of different audio objects. Audio objects classified as "very important objects" can have their absolute or relative levels raised (or increased), while audio objects classified as "low importance objects" can have their absolute or relative levels weakened (or reduced). Raising and weakening audio objects includes adjusting the gain associated with the audio objects. In some embodiments, raising and weakening audio objects includes using other forms of audio processing (e.g., dialogue enhancement, applying filters, etc.) to increase or decrease clarity. In addition, audio and visual objects can be processed based on the scene / object analysis performed in the visual scene / object classifier block 206 and the audio scene / object classifier block 208. For example, when a visual frame is classified as a forest scene, audio objects associated with insect and bird sounds may be added or augmented to improve immersion.

[0097] Figure 9 An example method 900 for processing audiovisual content is provided. The various steps described herein with respect to method 900 can be performed simultaneously, in parallel, or in a sequence different from the serial and iterative execution illustrated. Method 900 can be performed by post-production block 115. At block 902, method 900 includes receiving content including a plurality of audio frames and a plurality of video frames. For example, audiovisual analysis-based object rendering system 200 receives a video input. A plurality of audio frames are extracted from the video input by audio frame extraction block 204. A plurality of video frames are extracted by visual frame extraction block 202.

[0098] At block 904, method 900 includes classifying each of a plurality of audio frames into a plurality of audio classifications. For example, audio scene / object classifier block 208 receives audio frames from audio frame extraction block 204 and classifies the audio frames into audio objects. Classification of the audio frames can be performed by audio scene / object classifier block 208 in conjunction with audio object separation block 214 and / or object selection / metadata generation block 216. Each audio frame includes metadata indicating the classification of the detected audio object. At block 906, method 900 includes classifying each of a plurality of video frames into a plurality of video classifications. For example, visual scene / object classifier block 206 receives video frames from visual frame extraction block 202 and classifies objects within the video frames (e.g., the detected primary object). Each video frame includes metadata indicating the classification of the detected visual object.

[0099] At block 908, method 900 includes processing a plurality of audio frames based on corresponding audio classifications. For example, the audio rendering block 218 processes the audio frames based on the classification and categorization of the audio frames. In some embodiments, each audio frame is processed using a different audio processing operation. In other embodiments, some audio frames may be processed using the same audio processing operation, while other audio frames may be processed using different audio processing operations. For example, a first group of audio frames may be processed using a first audio processing operation, a second group of audio frames may be processed using a second audio processing operation, and a third group of audio frames may be processed using a third audio processing operation.

[0100] At block 910, method 900 includes processing a plurality of video frames based on corresponding video classifications. For example, visual processing block 212 processes the video frames based on the classification of visual objects within the video frames. In some embodiments, each video frame is processed using a different video processing operation. In other embodiments, some video frames may be processed using the same video processing operation, while other video frames may be processed using different video processing operations. For example, a first group of video frames may be processed using a first video processing operation, a second group of video frames may be processed using a second video processing operation, and a third group of video frames may be processed using a third video processing operation.

[0101] At block 912 , method 900 includes generating an audio / video representation of the content. For example, audiovisual synthesis block 220 generates a video output by merging the processed plurality of audio frames from audio rendering block 218 and the processed plurality of video frames from visual processing block 212 .

[0102] Systems, methods, and devices according to the present disclosure may employ any one or more of the following configurations.

[0103] (1) A method for processing audiovisual content, the method comprising: receiving content including a plurality of audio frames and a plurality of video frames; classifying each of the plurality of audio frames into a plurality of audio classifications; classifying each of the plurality of video frames into a plurality of video classifications; processing the plurality of audio frames based on the corresponding audio classifications, wherein each audio classification is processed using a different audio processing operation; processing the plurality of video frames based on the corresponding video classifications, wherein each video classification is processed using a different video processing operation; and generating an audio / video representation of the content by merging the processed plurality of audio frames and the processed plurality of video frames.

[0104] (2) The method according to (1), further comprising: classifying each of the plurality of audio classifications into one of a plurality of priority categories, wherein processing the plurality of audio frames comprises processing the plurality of audio frames based on the corresponding priority categories.

[0105] (3) A method according to (2), wherein the multiple priority categories include a first category and a second category, the first category indicates a higher priority than the second category, and wherein processing the multiple audio frames includes performing at least one item selected from the group consisting of: boosting audio frames classified as the first category, and attenuating audio frames classified as the second category.

[0106] (4) The method according to (3), wherein the first category includes speech objects.

[0107] (5) The method according to (4), wherein the first category further includes objects having height information.

[0108] (6) The method according to any one of (4) to (5), wherein the second category does not include speech objects.

[0109] (7) A method according to any one of (3) to (6), wherein the multiple priority categories include a third category indicating a lower priority than the first category, and wherein processing the multiple audio frames includes attenuating audio frames classified into the third category at a different attenuation level than audio frames classified into the second category.

[0110] (8) The method according to any one of (1) to (7), further comprising: extracting the plurality of audio frames from the content to separate the plurality of audio frames from the plurality of video frames.

[0111] (9) The method according to any one of (1) to (8), further comprising: determining the color richness of the color frame for each video frame; comparing the color richness with a color richness threshold for each video frame; and discarding the video frame for each video frame in response to the color richness being less than the color richness threshold.

[0112] (10) The method according to (9) further includes: for each video frame, determining a weight value of the video frame based on the color richness of the video frame, wherein each of the multiple video frames is classified based on the weight value.

[0113] (11) The method according to any one of (1) to (10), further comprising: determining whether a scene change occurs between a current video frame and a subsequent video frame, wherein classifying each of the multiple video frames into multiple video classifications comprises classifying the video frame for each video frame in response to determining that a scene change has occurred.

[0114] (12) The method according to (11), wherein determining whether a scene change occurs includes: converting the current frame into a first luma-chroma-chrome (YUV) frame; converting the subsequent frame into a second YUV frame; generating a first histogram based on the first YUV frame; generating a second histogram based on the second YUV frame; and determining whether the scene change occurs based on the first histogram and the second histogram.

[0115] (13) The method according to (12), wherein determining whether the scene change occurs based on the first histogram and the second histogram includes: calculating the absolute sum difference between the first histogram and the second histogram; and comparing the absolute sum difference with a scene change threshold.

[0116] (14) The method according to (11), wherein determining whether a scene change occurs includes: converting the current frame into a first luma-chroma-chrome (YUV) frame; converting the subsequent frame into a second YUV frame; calculating a difference between a first average YUV value of the first YUV frame and a second average YUV value of the second YUV frame; and determining whether the scene change occurs based on the difference between the first average YUV value and the second average YUV value.

[0117] (15) A method according to any one of (1) to (14), wherein classifying each of the multiple video frames into multiple video classifications includes: performing at least one of main object detection and scene detection to generate an intermediate result; and classifying the multiple video frames based on the intermediate result.

[0118] (16) A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method according to any one of (1) to (15).

[0119] (17) A video system for processing audiovisual content, the system comprising: a processor for performing processing of the audiovisual content, the processor being configured to: receive content comprising a plurality of audio frames and a plurality of video frames; classify each of the plurality of audio frames into a plurality of audio classifications; classify each of the plurality of video frames into a plurality of video classifications; process the plurality of audio frames based on the corresponding audio classifications, wherein each audio classification is processed using a different audio processing operation; process the plurality of video frames based on the corresponding video classifications, wherein each video classification is processed using a different video processing operation; and generate an audio / video representation of the content by merging the processed plurality of audio frames and the processed plurality of video frames.

[0120] (18) A system according to (17), wherein the processor is further configured to: classify each of the multiple audio classifications into one of a plurality of priority categories, wherein, in order to process the multiple audio frames, the processor is configured to process the multiple audio frames based on the corresponding priority category.

[0121] (19) A system according to (18), wherein the multiple priority categories include a first category and a second category, the first category indicates a higher priority than the second category, and wherein, in order to process the multiple audio frames, the processor is configured to: boost audio frames classified as the first category; and weaken audio frames classified as the second category.

[0122] (20) The system of any one of (17) to (19), wherein the processor is further configured to extract the plurality of audio frames from the content to separate the plurality of audio frames from the plurality of video frames.

[0123] (21) A system according to any one of (17) to (20), wherein the processor is further configured to: determine the color richness of the color frame for each video frame; compare the color richness with a color richness threshold for each video frame; and discard the video frame for each video frame and in response to the color richness being less than the color richness threshold.

[0124] (22) A system according to any one of (17) to (21), wherein the processor is further configured to: determine whether a scene change occurs between a current video frame and a subsequent video frame, wherein, in order to classify each of the multiple video frames into multiple video classifications, the processor is configured to classify the video frame for each video frame in response to determining that a scene change has occurred.

[0125] (23) A system according to (22), wherein, in order to determine whether a scene change occurs, the processor is configured to: convert the current frame into a first luma-chroma-chrome (YUV) frame; convert the subsequent frame into a second YUV frame; generate a first histogram based on the first YUV frame; generate a second histogram based on the second YUV frame; and determine whether the scene change occurs based on the first histogram and the second histogram.

[0126] (24) A system according to (23), wherein, in order to determine whether the scene change occurs based on the first histogram and the second histogram, the processor is configured to: calculate the absolute sum difference between the first histogram and the second histogram; and compare the absolute sum difference with a scene change threshold.

[0127] (25) A system according to (22), wherein, in order to determine whether a scene change occurs, the processor is configured to: convert the current frame into a first luma-chroma-chrome (YUV) frame; convert the subsequent frame into a second YUV frame; calculate the difference between a first average YUV value of the first YUV frame and a second average YUV value of the second YUV frame; and determine whether the scene change occurs based on the difference between the first average YUV value and the second average YUV value.

[0128] With respect to the processes, systems, methods, heuristics, and the like described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a particular ordered order, these processes can be practiced with the described steps performed in an order different from that described herein. It should also be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0129] Accordingly, it should be understood that the above description is intended to be illustrative and not restrictive. Upon reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but rather with reference to the appended claims and the full scope of equivalents to which such claims are entitled. It is anticipated and hoped that the technology discussed herein will develop in the future, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present application is capable of modification and change.

[0130] All terms used in the claims are intended to be given their broadest reasonable constructions and ordinary meanings as understood by those skilled in the art upon review of the description herein, unless an explicit indication to the contrary appears herein. In particular, use of singular articles such as "a," "an," "the," and "said" should be understood to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0131] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. This Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of linking the disclosure together as a whole. The disclosed approach should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Accordingly, the appended claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0132] Some embodiments may be implemented as a circuit-based process, including possible implementation on a single integrated circuit.

[0133] Aspects of the system described herein can be implemented in a suitable computer-based sound processing network environment to process digital or digitized audio files. Portions of the adaptive audio system can include one or more networks comprising any desired number of independent machines including one or more routers (not shown) for buffering and routing data transmitted between the computers. Such networks can be constructed on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0134] Some embodiments may be embodied in the form of methods and devices for practicing these methods. Some embodiments may also be embodied in the form of program code recorded in a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard drive, or any other non-transitory machine-readable storage medium, wherein when the program code is loaded into a machine (such as a computer, etc.) and executed by it, the machine becomes a device for practicing (multiple) patented inventions. Some embodiments may also be embodied in the form of program code, such as stored in a non-transitory machine-readable storage medium (including being loaded into a machine and / or executed by a machine), wherein when the program code is loaded into a machine (such as a computer or processor, etc.) and executed by it, the machine becomes a device for practicing (multiple) patented inventions. When implemented on a general-purpose processor, the program code segments are combined with the processor to provide a unique device that operates similarly to a specific logic circuit.

[0135] Unless expressly stated otherwise, each numerical value and range should be interpreted as being approximate as if the value or range were preceded by the word "about" or "approximately."

[0136] The use of figure numbers and / or reference numerals in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate interpretation of the claims. Such use should not be construed as necessarily limiting the scope of the claims to the embodiments shown in the corresponding figures.

[0137] Although elements in the method claims (if any) are recited in a specific order with corresponding labels, these elements are not necessarily intended to be implemented in that specific order unless the claim recitation otherwise implies a specific order for implementing some or all of these elements.

[0138] Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present disclosure. The appearances of the phrase "in one embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. The same applies to the term "implementation."

[0139] Unless otherwise specified herein, the use of ordinal adjectives "first," "second," "third," etc. to refer to an object among a plurality of similar objects indicates merely that different instances of such similar objects are being referred to and is not intended to imply that the similar objects so referred to are necessarily in a corresponding order or sequence, whether temporally, spatially, in ranking, or in any other manner.

[0140] Unless otherwise specified herein, in addition to its plain meaning, the conjunction "if" may also or alternatively be interpreted to mean "when" or "at the time" or "in response to determining" or "in response to detecting," which interpretation may depend on the specific context. For example, the phrase "if it is determined that" or "if [stated condition] is detected" may be interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]."

[0141] Also for the purposes of this description, the terms "couple," "coupling," "coupled," "connect," "connecting," or "connected" refer to any means known in the art or later developed that allow energy to be transferred between two or more elements, and the insertion of one or more additional elements is contemplated, although not required. In contrast, the terms "directly coupled," "directly connected," etc., imply the absence of such additional elements.

[0142] As used herein with respect to components and standards, the term "compliant" means that the component communicates with other components in a manner specified in whole or in part by the standard and will be recognized by other components as sufficiently capable of communicating with other components in the manner specified by the standard. A compliant component is not required to operate internally in the manner specified by the standard.

[0143] The functions of the various elements shown in the figure, including any functional blocks marked as "processors" and / or "controllers", can be provided by using dedicated hardware and hardware capable of executing software associated with appropriate software. When provided by a processor, the function can be provided by a single dedicated processor, by a single shared processor, or by multiple separate processors (some of which can be shared). In addition, the explicit use of the term "processor" or "controller" should not be interpreted as referring exclusively to hardware capable of executing software, and can implicitly include but are not limited to digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read-only memories (ROMs), random access memories (RAMs), and non-volatile memories for storing software. Other traditional and / or customized hardware may also be included. Similarly, any switches shown in the figure are only conceptual. Their functions can be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, and specific techniques can be selected by the implementer, as more specifically understood from the context.

[0144] As used in this application, the terms "circuit" and "circuitry" may refer to one or more or all of the following: (a) a pure hardware circuit implementation (such as an implementation in a pure analog circuit and / or digital circuit); (b) a combination of hardware circuitry and software, such as (if applicable): (i) a combination of analog hardware circuitry and / or digital hardware circuitry with software / firmware, and (ii) any portion of a hardware processor with software (including a digital signal processor), software, and memory that work together to enable a device (such as a mobile phone or server, etc.) to perform various functions; and (c) a hardware circuitry and / or processor, such as a microprocessor or a portion of a microprocessor, that requires software (e.g., firmware) to operate, but may not be present when the software is not required for operation. This definition of circuitry applies to all uses of the term in this application (including all claims). As a further example, as used in this application, the term "circuitry" also covers implementations of only a hardware circuitry or a processor (or multiple processors) or a portion of a hardware circuitry or processor and its accompanying software and / or firmware. The term "circuitry" also covers, for example, a baseband integrated circuit or a processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or networking device, if applicable to a particular claim element.

[0145] Those skilled in the art will recognize that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be recognized that any flow charts, flow diagrams, state transition diagrams, pseudocode, etc. represent various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not the computer or processor is explicitly shown.

[0146] The Summary of the Invention in this specification is intended to introduce some example embodiments, and additional embodiments are described in the Detailed Description and / or with reference to one or more of the accompanying drawings. The Summary is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0147] Although the present disclosure includes references to illustrative embodiments, this description is not intended to be construed as limiting. Various modifications of the described embodiments, as well as other embodiments within the scope of the present disclosure, that are apparent to those skilled in the art to which the present disclosure relates are deemed to be within the principles and scope of the present disclosure, for example, as expressed in the claims.

Claims

1. A method for processing audiovisual content, the method comprising: receiving content including a plurality of audio frames and a plurality of video frames; classifying each of the plurality of audio frames into a plurality of audio classes; classifying each of the plurality of video frames into a plurality of video classifications; processing the plurality of audio frames based on respective audio classifications, wherein each audio classification is processed using a different audio processing operation; processing the plurality of video frames based on corresponding video classifications, wherein each video classification is processed with a different video processing operation; and An audio / video representation of the content is generated by merging the processed plurality of audio frames and the processed plurality of video frames.

2. The method of claim 1, further comprising: Each of the plurality of audio classifications is categorized into one of a plurality of priority categories, wherein processing the plurality of audio frames includes processing the plurality of audio frames based on the corresponding priority category.

3. The method of claim 2, wherein: The multiple priority categories include a first category and a second category, the first category indicating a higher priority than the second category, and wherein processing the multiple audio frames includes performing at least one item selected from the group consisting of: boosting audio frames classified as the first category, and attenuating audio frames classified as the second category.

4. The method of claim 3, wherein: The first category includes speech objects.

5. The method of claim 4, wherein: The first category further includes objects having height information.

6. The method according to any one of claims 4 to 5, wherein: The second category does not include speech objects.

7. The method according to any one of claims 3 to 6, wherein: The plurality of priority categories include a third category indicating a lower priority than the first category, and wherein processing the plurality of audio frames includes attenuating audio frames classified in the third category at a different attenuation level than audio frames classified in the second category.

8. The method according to any one of claims 1 to 7, further comprising: The plurality of audio frames are extracted from the content to separate the plurality of audio frames from the plurality of video frames.

9. The method according to any one of claims 1 to 8, further comprising: determining a color richness of a color frame for each video frame; For each video frame, comparing the color richness with a color richness threshold; as well as For each video frame and in response to the color richness being less than the color richness threshold, the video frame is discarded.

10. The method of claim 9, further comprising: For each video frame, determining a weight value of the video frame based on the color richness of the video frame, Wherein, each of the multiple video frames is classified based on the weight value.

11. The method of any one of claims 1 to 10, further comprising: Determine whether a scene change occurs between the current video frame and the subsequent video frame, Wherein, classifying each of the plurality of video frames into a plurality of video classifications includes, for each video frame, classifying the video frame in response to determining that a scene change has occurred.

12. The method of claim 11, wherein: Determining whether a scene change has occurred includes: converting the current frame into a first luma-chroma-chrome (YUV) frame; Converting the subsequent frame into a second YUV frame; generating a first histogram based on the first YUV frame; generating a second histogram based on the second YUV frame; and It is determined whether the scene change occurs based on the first histogram and the second histogram.

13. The method of claim 12, wherein: Determining whether the scene change occurs based on the first histogram and the second histogram includes: calculating an absolute sum difference between the first histogram and the second histogram; and The absolute sum difference is compared to a scene change threshold.

14. The method of claim 11, wherein: Determining whether a scene change has occurred includes: converting the current frame into a first luma-chroma-chrome (YUV) frame; Converting the subsequent frame into a second YUV frame; calculating a difference between a first average YUV value of the first YUV frame and a second average YUV value of the second YUV frame; and Whether the scene change occurs is determined based on the difference between the first average YUV value and the second average YUV value.

15. The method according to any one of claims 1 to 14, wherein: Classifying each of the plurality of video frames into a plurality of video classifications comprises: performing at least one of main object detection and scene detection to generate an intermediate result; and The plurality of video frames are classified based on the intermediate results.

16. A non-transitory computer readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including the method of any one of claims 1 to 15.

17. A video system for processing audio-visual content, the system comprising: A processor, the processor is used to perform processing of audio-visual content, the processor is configured to: receiving content including a plurality of audio frames and a plurality of video frames; classifying each of the plurality of audio frames into a plurality of audio classes; classifying each of the plurality of video frames into a plurality of video classifications; processing the plurality of audio frames based on respective audio classifications, wherein each audio classification is processed using a different audio processing operation; processing the plurality of video frames based on corresponding video classifications, wherein each video classification is processed with a different video processing operation; and An audio / video representation of the content is generated by merging the processed plurality of audio frames and the processed plurality of video frames.

18. The system of claim 17, wherein: The processor is further configured to: Each of the plurality of audio classifications is classified into one of a plurality of priority categories, wherein, to process the plurality of audio frames, the processor is configured to process the plurality of audio frames based on the corresponding priority category.

19. The system of claim 18, wherein: The plurality of priority categories include a first category and a second category, the first category indicating a higher priority than the second category, and wherein, to process the plurality of audio frames, the processor is configured to: boosting audio frames classified as the first category; and Audio frames classified as the second category are attenuated.

20. The system of any one of claims 17 to 19, wherein: The processor is further configured to: The plurality of audio frames are extracted from the content to separate the plurality of audio frames from the plurality of video frames.

21. The system of any one of claims 17 to 20, wherein: The processor is further configured to: determining a color richness of a color frame for each video frame; For each video frame, comparing the color richness to a color richness threshold; and For each video frame and in response to the color richness being less than the color richness threshold, the video frame is discarded.

22. The system of any one of claims 17 to 21, wherein: The processor is further configured to: Determine whether a scene change occurs between the current video frame and the subsequent video frame, In order to classify each of the plurality of video frames into a plurality of video classifications, the processor is configured to classify the video frame in response to determining that a scene change has occurred for each video frame.

23. The system of claim 22, wherein: In order to determine whether a scene change occurs, the processor is configured to: converting the current frame into a first luma-chroma-chrome (YUV) frame; Converting the subsequent frame into a second YUV frame; generating a first histogram based on the first YUV frame; generating a second histogram based on the second YUV frame; as well as It is determined whether the scene change occurs based on the first histogram and the second histogram.

24. The system of claim 23, wherein: In order to determine whether the scene change occurs based on the first histogram and the second histogram, the processor is configured to: calculating an absolute sum difference between the first histogram and the second histogram; and The absolute sum difference is compared to a scene change threshold.

25. The system of claim 22, wherein: In order to determine whether a scene change occurs, the processor is configured to: converting the current frame into a first luma-chroma-chrome (YUV) frame; Converting the subsequent frame into a second YUV frame; Calculating a difference between a first average YUV value of the first YUV frame and a second average YUV value of the second YUV frame; as well as Whether the scene change occurs is determined based on the difference between the first average YUV value and the second average YUV value.

Images