Method, system and medium for converting image frames of a video between aspect ratios
Through machine learning, the video lens features are extracted and panning and scanning framing decisions are predicted, and the conversion of wide-screen aspect ratio image frames to home aspect ratio is automated, solving the laborious problem of manual decision-making in the prior art and improving conversion efficiency and consistency.
Patent Information
- Application Number
- CN202010199104.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-20
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-11-29
AI Technical Summary
The prior art requires manual decision-making when converting movie image frames with wide-screen aspect ratios to aspect ratios supported by home display devices, panning and scanning framing is laborious and time-consuming and difficult to automatically process.
Using machine learning methods, the aspect ratio conversion function is determined by extracting visual and audio features in the video lens, the panning and scanning framing decisions that human operators will make, and the image frames are converted between aspect ratios.
Automated aspect ratio conversion is achieved, reducing manual intervention, improving conversion efficiency and consistency, and enabling the creation of relatively efficient and creative decisions between different aspect ratios and in line with film producers.
Smart Images

Figure CN111724302B_ABST
Abstract
Description
Technical Field
[0001] Aspects presented in this disclosure generally relate to converting image frames of a video between aspect ratios. Background Art
[0002] The aspect ratio of an image is the proportional relationship between the width of the image and the height of the image. Motion picture content is often created for theatrical release in a widescreen aspect ratio (e.g., a 2.39:1 aspect ratio), which is wider than the aspect ratio supported by home display devices (e.g., a 1.78:1 aspect ratio for high-definition television). As a result, image frames of widescreen aspect ratio motion pictures need to be converted to a smaller aspect ratio for home video release. One method of aspect ratio conversion is pan and scan, which requires reframing the image frame to retain the primary focus of the scene depicted therein while cropping out the remainder of the image frame (e.g., the sides of the widescreen aspect ratio image frame). Framing decisions can include moving the retained window left or right to follow the action in the scene, creating the effect of a "panning" shot, and ensuring that the focus (i.e., "scan") of each shot is preserved. Traditional pan and scan requires manual decisions on how to represent the on-screen action throughout the movie in the new aspect ratio, which is laborious and time-consuming. Summary of the invention
[0003] One aspect of the present disclosure provides a computer-implemented method for converting image frames between aspect ratios. The method generally includes using machine learning to determine at least one aspect ratio conversion function based at least in part on: (i) extracted features that indicate at least one of visual elements or audio elements associated with a first image frame of a first aspect ratio in a shot from a video, and (ii) manual framing decisions in pan and scan conversion to convert the first image frame to a corresponding second image frame of a second aspect ratio. The method also includes identifying shots in the received video, and for each image frame in each of the identified shots, converting the image frame from the first aspect ratio to the second aspect ratio using one of the at least one aspect ratio conversion functions.
[0004] Other aspects include, but are not limited to, computer-readable media including instructions that enable a processing unit to implement one or more aspects of the disclosed methods and systems configured to implement one or more aspects of the disclosed methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] In order that the aforementioned aspects may be obtained and understood in detail, a more particular description of the aspects of the present disclosure briefly summarized above may be obtained by reference to the appended drawings.
[0006] It is to be noted, however, that the appended drawings illustrate only typical aspects of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective aspects.
[0007] Figure 1 A method for converting image frames of a video between aspect ratios according to aspects of the present disclosure is shown.
[0008] Figure 2 An aspect ratio conversion module according to aspects of the present disclosure is shown in more detail.
[0009] Figure 3 Training of a model and inference using the trained model according to aspects of the present disclosure are shown.
[0010] Figure 4 Example pan and scan conversion according to aspects of the present disclosure is shown.
[0011] Figure 5 A method for converting image frames between aspect ratios according to aspects of the present disclosure is shown.
[0012] Figure 6 A system is shown in which aspects of the present disclosure may be implemented. DETAILED DESCRIPTION
[0013] Aspects presented herein provide a method for converting image frames (such as those in movies) from one aspect ratio to another while predicting pan and scan framing decisions that a human operator would make. In one configuration, one or more functions for predicting pan and scan framing decisions are determined using machine learning and training data, the training data including historical pan and scan conversions performed manually. Each of the (one or more) functions may include one or more rules and / or models for mapping inputs (such as features extracted from shots in a video) to outputs (such as pan and scan framing predictions for image frames in the shot). As used herein, a "shot" refers to a series of interrelated, continuous image frames in a video that is continuously captured by one or more cameras and represents continuous action in time and space. The training data (and subsequent inference data) can be prepared by extracting features that indicate visual and / or audio elements associated with a particular shot in the training video. The (one or more) functions can be determined using machine learning, which takes such extracted features as input and outputs predicted pan and scan framing decisions. After obtaining such function(s), image frames of a received video may be converted between aspect ratios on a shot-by-shot basis by extracting the same features and using the function(s) to perform pan and scan framing predictions on image frames in each shot in the video.
[0014] In the following, reference is made to aspects of the present disclosure. However, it should be understood that the present disclosure is not limited to the specifically described aspects. On the contrary, any combination of the following features and elements, whether or not related to different aspects, is contemplated as implementing and practicing the aspects disclosed herein. In addition, although various aspects may achieve advantages relative to other possible solutions and / or relative to the prior art, whether a given aspect achieves a particular advantage does not limit the present disclosure. Therefore, the following aspects, features, configurations, and advantages are merely illustrative and are not considered to be elements or limitations of the appended claims unless expressly stated in (one or more) claims. Similarly, references to "aspects" should not be interpreted as a generalization of any inventive subject matter disclosed herein, and should not be considered to be elements or limitations of the appended claims unless expressly stated in (one or more) claims.
[0015] As will be appreciated by those skilled in the art, aspects of the present disclosure may be embodied as systems, methods, or computer program products. Thus, aspects of the present disclosure may take the form of a complete hardware configuration, a complete software configuration (including firmware, resident software, microcode, etc.), or a configuration combining software and hardware aspects, which configurations may generally be referred to herein as "circuits," "modules," or "systems." Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.
[0016] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0017] A computer readable signal medium may include a propagated data signal containing computer readable program code, such as in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium, and may communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0018] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0019] Computer program code for performing operations of various aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection may be established with an external computer (e.g., through the Internet using an Internet service provider).
[0020] Aspects of the present disclosure are described below with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to aspects of the present disclosure. It will be understood that each block of the flowchart and / or block diagram and the combination of blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device create a unit for implementing the function / action specified in the flowchart and / or block diagram block or multiple blocks.
[0021] These computer program instructions may also be stored in a computer-readable medium, which may instruct a computer, other programmable data processing apparatus, or other device to operate in a particular manner so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in the flowchart and / or block diagram block or multiple blocks.
[0022] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.
[0023] Aspects of the present disclosure may be provided to end users via a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as computing capabilities that provide an abstraction between computing resources and their underlying technology architecture (e.g., servers, storage devices, networks), thereby enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be quickly configured and released with minimal management effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage devices, data, applications, and even complete virtualized computing systems) in a "cloud" without having to consider the underlying physical systems (or the location of those systems) used to provide the computing resources.
[0024] Typically, cloud computing resources are provided to users on a pay-per-use basis, and users are charged only for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualized systems instantiated by the user). Users can access any resource residing in the cloud at any time from anywhere on the Internet. In the context of the present disclosure, users can access applications (e.g., aspect ratio conversion applications) or related data available in the cloud. For example, an aspect ratio conversion application can be executed on a computing system in the cloud, and image frames in a received video are converted from one aspect ratio to another aspect ratio according to the technology disclosed herein, and the converted image frames are stored in a storage location in the cloud. Doing so allows users to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).
[0025] Reference now Figure 1, a method for converting image frames of a video between aspect ratios according to aspects of the present disclosure is shown. As shown, the aspect ratio conversion application 100 includes a shot identification module 110, a feature extraction module 120, and an aspect ratio conversion module 130. The aspect ratio conversion application 100 receives as input a video 102 (e.g., an episode of a movie or a series) and associated metadata 104 and an indication of aspect ratios to convert between. The metadata 104 may include information associated with the entire input video 102 (e.g., the filmmaker who created the video, the film studio, etc.) and / or information associated with a particular shot or image frame in the video 102 (e.g., the time of day or the objects appearing in the particular shot or image frame). The aspect ratio conversion application 100 is configured to determine pan and scan framing to predict what a user, such as a filmmaker, will choose when converting image frames of the video 102 between different image frame aspect ratios. The aspect ratio conversion application 100 then crops the image frames of the video 102 based on the determined pan and scan framing and outputs a video 140, the image frames of the video 140 being represented in the aspect ratio. For example, the image frames of the input video 102 may be original versions of the image frames created for theatrical release in a widescreen aspect ratio (e.g., a 2.39:1 aspect ratio), and the aspect ratio conversion application 100 may convert such image frames to image frames of the output video 140 in an aspect ratio supported by a home display device (e.g., a 1.78:1 aspect ratio), while anticipating the creative choices that a filmmaker will make.
[0026] The shot identification module 110 is configured to identify different shots 112 within the input video 102. i As mentioned above, each lens 112 i A sequence of interrelated, continuous image frames captured continuously by a camera (or multiple cameras) and representing continuous action in time and space. Automated techniques for detecting shot boundaries in a video are known, and the aspect ratio conversion application 100 may use such shot boundary detection techniques in certain configurations. In another configuration, metadata 104 associated with the video 102 may indicate shot boundaries, in which case the shot identification module 110 may identify shots 112 in the input video 102 from such metadata 104. i . Each identified shot 112 i are processed (either sequentially or in parallel with other identified shots) to determine pan and scan conversion of image frames therein.
[0027] The feature extraction module 120 is configured to determine the feature of each shot 112 identified from the input video 102. iA set of features 126 associated with an image frame in the image. Features are properties and / or characteristics of the data that can be used during: (1) during the training phase with examples of desired outputs (sometimes also referred to as labeled data) to derive rules and / or models via machine learning, and (2) during the inference phase to use the rules and / or models given new data and make predictions. Each desired feature can be obtained from the identified shot 112 i The feature extraction module 120 may be used to extract the image frame or from associated metadata, which may provide certain features and / or information from which features may be derived. Although shown as being performed by the feature extraction module 120 for illustrative purposes, in some configurations, feature extraction may also be performed elsewhere. For example, the aspect ratio conversion module 130 may employ a model such as a convolutional neural network, which itself includes a feature extraction layer. As another example, in the case of a scene with lens 112 i Some features may be automatically determined or manually labeled and indicated elsewhere in the associated metadata or in a database, in which case the feature extraction module 120 may obtain such features from the metadata or the database.
[0028] As shown, the feature extraction module 120 includes a visual feature extraction module 122 and an audio feature extraction module 124. The visual feature extraction module 122 and the audio feature extraction module 124 are responsible for extracting features indicating visual and audio elements associated with the shot in the input video 102, respectively. In one configuration, the visual elements include objects, such as characters appearing in the image frames of the shot, the direction in which the characters face, the tracking movement of the objects in 2D space (i.e., the xy position of the objects) during the filming process, etc., as discussed in more detail below. In addition, the audio elements may include characters speaking in the shot, the subject of the dialogue in the shot, and the music playing in the shot if there is no dialogue, as discussed in more detail below. In this case, the feature extraction module 120 may determine the characters, the direction in which the characters face, the movement of the characters, the characters speaking, the subject of the dialogue, and / or the music as features 126 to input to the prediction technology, which outputs panning and scanning framing predictions based on these features 126. The prediction technology may include (one or more) functions determined at least in part via machine learning to identify and protect important areas of each image frame in the shot. For example, areas depicting speaking characters, areas depicting characters when only music is playing, and areas depicting the subject of dialogue may be protected when predicting pan and scan framing decisions that a user will make, as discussed in more detail below.
[0029] The feature extraction module 120 may employ any suitable technique to identify and locate objects such as people in image frames of a shot, track the movement of such objects, determine the direction in which a person is facing, identify a person who is speaking or music that is playing, and determine the subject of a conversation. For example, one or more machine learning models (e.g., convolutional neural networks (CNNs), support vector machines (SVMs), or variants thereof) may be trained based on labeled training data to classify and locate objects such as people, creatures, props, screen text and / or environments, and the direction in which a person is facing, and the visual feature extraction module 122 in the feature extraction module 120 may input each image frame from the shot into such trained (one or more) machine learning models to detect the objects therein and the direction in which the person is facing. As another example, the visual feature extraction module 122 may identify a person who is speaking based on mouth movement, in which case the trained (one or more) machine learning models may be used again to detect the mouth of the person in the image frame, and the visual feature extraction module 122 may also determine whether the detected mouth changes from frame to frame to indicate that the person is speaking. Alternatively, the speaking person may be identified via speech recognition and matching of the recognized speech to visually recognized persons (via, for example, facial recognition) or based on an associated script such as a combined continuity script (CCSL) indicating which persons are speaking in the shot. As yet another example, the audio feature extraction module 124 may compare the audio fingerprint in the shot with fingerprints of songs in a database to determine whether music is playing and / or identify a particular song that may indicate the mood of the shot. As an additional example, the topic of dialogue in the shot may be determined by applying natural language processing techniques (e.g., topic extraction) to a portion of the CCSL associated with the shot or to text generated via speech-to-text techniques after identifying the person's voice. As described, one or more features may also be indicated in the received metadata (or elsewhere such as in a database), in which case the feature extraction module 120 may obtain such features directly from the metadata (or elsewhere).
[0030] In some configurations, in addition to identifying objects, their movement, the direction a person is facing, the person or music that is speaking, and the subject of the dialogue, the feature extraction module 120 may also determine other visual and / or audio elements. For example, visual elements other than objects, their movement, and the direction a person is facing may be identified, including image characteristics of image frames in a shot, such as whether the image frame is bright or dark; whether the image frame depicts a daytime scene or a nighttime scene; the contrast, saturation, graininess, and / or sharpness of the image frame; animation depicted in the image frame, etc.; and technical information, such as the camera lens used to capture the image frame of the shot, whether the image frame depicts a digital image or a film image, whether the image frame depicts live action or animated content, the year, etc. Whether the image frame is bright or dark, depicts daytime or nighttime, etc. are also referred to herein as the nature of the scene depicted. Additional features that the feature extraction module 120 may determine include features that indicate creative information, such as a filmmaker, brand, genre, film company, TV series, franchise, etc. associated with the input video 102. Likewise, image features, technical information, and creative information features may be determined by the feature extraction module 120 itself and / or from manually tagged information in the metadata using any suitable technique. For example, creative information such as filmmaker, brand, genre, film company, series, franchise, etc. may be indicated in the metadata 104 received with the video 102. As another example, the feature extraction module 120 may query a database to obtain such information. Similarly, technical information such as the camera lens used to capture the image frame of the shot, whether the image frame depicts a digital image or a film image, whether the image frame depicts a live action shot or animated content, the year, etc. may be indicated in the received metadata 104 or determined by the feature extraction module 120 itself. As another example, automated techniques are known for predicting the time of day of a scene, and the visual feature extraction module 122 may use such techniques to determine whether a daytime scene or a nighttime scene is depicted in the image frame of the shot.
[0031] The aspect ratio conversion module 130 is configured to employ one or more prediction techniques to determine, for each identified shot in the received video 102 , the pan and scan framing of each image frame in the shot based on extracted features, including features extracted by the feature extraction module 120 . Figure 2 The aspect ratio conversion module 130 according to aspects of the present disclosure is shown in more detail. As shown, given an identified shot 112 i and the extracted features 126, the aspect ratio conversion module 130 uses one or more functions 210 i To determine the conversion between aspect ratios, the one or more functions 210 iThe aspect ratio conversion module 130 may be determined using machine learning, but may also include user-defined rules. As described, each function may include one or more rules and / or models for mapping inputs to outputs. Although rules and models are sometimes discussed as being different here, in some configurations, the model itself may include rules. In other configurations, only rules or only models may be used. That is, the aspect ratio conversion module 130 may generally utilize any appropriate rules and / or models.
[0032] The function may include one or more rules defined by automatic reasoning and / or a user. In particular, the rule may include an if-then rule. In one configuration, the user defines an if-then rule to constrain the output of the aspect ratio conversion, thereby providing a frame-by-frame or scene-by-scene restriction on the panning and scanning framing predictions made by the aspect ratio conversion module 130. For example, a user-defined rule may specify that if a character in a corner speaks in a shot, the character who speaks must appear in the re-framed image frame of the shot, i.e., the automatically predicted panning and scanning framing must retain the character who speaks. As another example, a user-defined rule may specify that when a character appears in the image frame of a shot, if only music is playing, the character must also appear in the re-framed image frame of the shot. Another user-defined rule may specify that if the image frame of a shot depicts a scene with the sun during the day, the re-framed image frame of the shot should not pan toward the sun. Although described herein as a user-defined rule, it should be understood that such a rule may also be learned or adjusted via machine learning, as discussed in more detail below.
[0033] In addition to or in lieu of user-defined rules, the rules in the function may also include if-then rules automatically determined by machine learning techniques. In particular, the automatically determined rules can solve the regression problem of predicting pan and scan framing for converting image frames of a video from one aspect ratio to another. In one configuration, the rules include rules formed by a path from a root node to a leaf node of a decision tree or a variant thereof, which may also be viewed as a model. A decision tree is a tree-like model whose internal nodes represent tests of feature values, leaves specify classification or regression outputs, and branches to different feature values at each node. For example, one rule in a decision tree may specify that if the filmmaker is A, the brightness in the image frames of a shot is above a threshold (indicating, for example, a daytime scene), a single speaking person appears in the image frames of the shot, the size of the person is within a certain range, the direction in which the person's face is pointing is within another range, and the person moves from one xy position range to another xy position range from the first image frame to the last image frame of the shot, then the pan and scan conversion should crop the image frames of the shot by cutting certain percentages from the left and right sides of the first image frame, while moving the cutting to other percentages from the left and right sides of the last image frame, where the cutting performed on the intermediate image frames is between the cutting performed on the first image frame and the last image frame, thereby creating a "panning" shot effect. Another rule in such a decision tree may specify that for the same filmmaker, brightness level, and speaking person, but the speaking person is located at different positions in the first and last image frames of the shot, the pan and scan conversion should crop different percentages from the left and right sides of the image frames of the shot and / or create different panning effects, etc. A decision tree (sometimes called a regression tree) that produces this regression output (an image frame that indicates how to crop a shot based on input features (filmmaker, brightness, person speaking, facing direction, person size, and person movement in the above example) can be learned via any suitable decision tree algorithm based on historical pan and scan conversion examples (from which features can be extracted) and manual framing decisions used as examples of desired output in training. Examples of algorithms that can be used to learn decision trees include ID3 (Iterative Dichotomizer 3), C4.5, CART (Classification and Regression Trees), CHAID (Chi-Square Automatic Interaction Detector), and MARS (Multiple Adaptive Regression Splines). See below for more information. Figure 3 The process of training models such as decision trees is discussed in more detail. In some configurations, multiple decision trees or variants thereof, such as decision trees of a random forest, may be learned. For example, multiple decision trees may be trained using appropriate training data, such as different decision trees for predicting pan and scan conversions between different input and output aspect ratios, or different decision trees for different classifications (e.g., live-action, animation, action, comedy, etc.), movie franchises, or overall genres.
[0034] In another configuration, the rules in the function may include rules in a knowledge-based system. In this case, the rules may be stored in a knowledge base along with facts about historical pan and scan conversions, and the inference engine may apply the stored rules to perform pan and scan framing predictions on image frames of an input video. Furthermore, the stored rules may include rules learned in any suitable manner (e.g., using rule-based machine learning techniques), and / or the stored rules may include user-defined rules such as those described above. For example, rules that fail to predict historical manual aspect ratio conversions may be discarded, or their parameter values may be changed, through machine learning processing.
[0035] In some configurations, the rules in the function may also be relearned as more examples of manual aspect ratio conversions become available. Additionally, the rules may be adjusted by, for example, enhancing rules that make correct predictions while adjusting parameters of rules that make incorrect predictions.
[0036] In addition to or in lieu of the rules, the function may include one or more models trained via machine learning techniques to solve the regression problem of predicting panning and scanning framing for converting image frames of a video from one aspect ratio to another. As described, an example of such a model is a decision tree or a variant thereof, which can be used in some configurations of the aspect ratio conversion application 100. In an optional configuration, the model in the function may include one or more deep learning models. The deep learning model includes an artificial neural network with multiple layers between the input layer and the output layer. Such a deep learning model can be trained using, for example, back propagation with gradient descent or a variant thereof, and any appropriate loss function, such as a manually specified loss function or an automatically determined loss function, can be minimized during training under a loss minimization framework. Like the decision tree described above, in some configurations, multiple deep learning models can be trained using appropriate training data, such as different deep learning models for predicting panning and scanning conversions between different input and output aspect ratios, or for different classifications (e.g., live action, animation, action, comedy, etc.), movie franchises, or overall styles. As another example, different deep learning models can be trained to predict the creative decisions that different filmmakers or studios will make, so that if, for example, a particular filmmaker's or studio's films tend to have a unique feel after being manually panned and scanned due to specific creative choices, the model can make predictions that mimic the same feel. After the models are trained, the aspect ratio conversion module 130 can select one of the trained models (e.g., for the appropriate input and output aspect ratios, filmmaker, etc.) and then input features extracted from the identified shots in the input video 102 and / or the image frames from the identified shots themselves to the selected deep learning model, which in turn outputs a predicted pan and scan framing for each image frame.
[0037] Convolutional neural networks and their variants are examples of deep learning architectures that can be trained to predict the pan and scan decisions that a filmmaker would make when converting image frames of a shot in a video from one aspect ratio to another. A CNN typically includes multiple feature extraction layers, including convolutional and pooling layers, and fully connected layers that make predictions based on the features extracted by the feature extraction layers. In one configuration, feature extraction layers previously trained on a large dataset of images (e.g., the ImageNet image set) are combined with fully connected layers trained to make pan and scan framing predictions for aspect ratio conversion purposes (with or without being retrained), and the output layer of the CNN can include nodes (e.g., four nodes) that output pan and scan framing predictions as coordinates where an input image frame of one aspect ratio should be cropped to represent an image frame of another aspect ratio. Such a CNN can be trained using manually made pan and scan framing decisions as examples of desired outputs, essentially learning to mimic manually made framing decisions. In addition, the "panning" effect can be achieved by, for example, using a CNN to predict how to reframe the first and last image frames of a shot (or key frames in a shot) and determining the reframe between the first and last image frames (key frames) for the intermediate image frames in the shot, or in any other suitable manner. Although CNN is described herein as a reference example, other suitable deep learning models (e.g., a generative adversarial network (GAN), an autoencoder, or a variant thereof) can be used in combination with or in place of CNN.
[0038] In some configurations, function 210 iOne or more of may include rules for constraining output content before and / or after employing the model. Continuing with the example above, where the user-defined rule specifies that if a person is speaking in a shot, then the speaking person must appear in a reframed image frame of the shot, the aspect ratio conversion application 100 may extract a portion of the image frame that can be cropped to the desired aspect ratio size while retaining the speaking person in the output image frame before inputting each image frame in the shot to the deep learning model, and then input only the extracted portion of the image frame (resized to the size that the deep learning model takes as input, if appropriate) to the deep learning model. Doing so ensures that the output of the deep learning model is a prediction of the framing that, when applied to the cropped original image frame, does not remove the speaking person. As another example, after the aspect ratio conversion application 100 checks the fact that the pan and scan framing predictions made by deep learning produce output image frames depicting a speaking person, the same rule that the speaking person must appear in the reframed image frame of the shot can be applied, and if the output image frame does not depict the speaking person, the aspect ratio conversion application 100 can adjust the pan and scan framing output by the deep learning model to satisfy the rule and / or issue an alarm so that the user can manually adjust the pan and scan framing output through the deep learning model. In yet another configuration, the deep learning model itself can be trained (e.g., by training a fully connected layer of a CNN or training a different type of model such as a conditional GAN or a variant thereof) to account for speaking people and other features, such as the features described above that indicate visual (e.g., image features and technical information) and audio (e.g., speaking person, music, subject matter of dialogue) elements of the shot and / or creative information.
[0039] Although decision trees, knowledge-based systems, and deep learning models are described above as reference examples of rules and models that functions may include, optional configurations may utilize other types of functions or combinations thereof. For example, in one configuration, the aspect ratio conversion application 100 may use a nearest neighbor technique (e.g., a k-nearest neighbor algorithm). The k-nearest neighbor algorithm determines a classification or regression output based on a feature similarity measure (e.g., a distance function) with a training example. In one configuration, the aspect ratio conversion application 100 may use a k-nearest neighbor algorithm or a variant thereof to determine, based on features determined for the lens, a number of most similar lens examples that have been manually converted between desired aspect ratios via panning and scanning in a database of historical panning and scanning conversions, and then the aspect ratio conversion application 100 may use the average of the panning and scanning framing in the most similar examples as the framing of the image frame in the lens. More generally, any function suitable for solving the regression problem of predicting panning and scanning conversion of image frames between aspect ratios on a shot-by-shot basis may be used. Although discussed herein primarily with respect to regression outputs, in some embodiments, the function may also be trained to output a classification, i.e., an identified category, such as one of a plurality of predefined pan and scan views for each image frame and / or shot.
[0040] return Figure 1 , after predicting the pan and scan framing of image frames of input video 102 on a shot-by-shot basis, aspect ratio conversion module 130 crops the image frames based on the predicted framing and then outputs the cropped image frames as output video 140. In some configurations, aspect ratio conversion application 100 (or other application) may also receive and apply user modifications to the predicted pan and scan framing. For example, a movie studio may have a contractual obligation to obtain a filmmaker's approval for a pan and scan conversion of the filmmaker's film. Aspect ratio conversion application 100 (or other application) may allow a user (e.g., the filmmaker in the above example) to manually modify the automatically predicted pan and scan framing by, for example, providing a preview of the predicted framing and tools to change those framings, as described below with respect to Figure 4 Described in detail.
[0041] Figure 3 The training of the model 325 and the use of the trained model 325 for reasoning according to aspects of the present disclosure are shown. For example, the trained model 325 can be included in the above function 210 iAlthough training of the model 325 is shown for illustrative purposes, the model training application (which may be the aspect ratio conversion application 100 itself or another application) may generally use machine learning training techniques to obtain any number of functions including models and / or rules. Different functions may be obtained using appropriate training data, for example, to predict conversions between different input and output aspect ratios; to predict conversion decisions for different filmmakers or film companies; to predict conversion decisions for different genres (e.g., live-action, animation, action, comedy, etc.), film franchises, or overall styles; or for any other purpose.
[0042] As shown, during the training phase 320, training data 310 is first used (e.g., an offline model training application) to train a model 325. In one configuration, the training data 310 includes historical videos of different aspect ratios, some of which are manually panned and scanned conversions of other videos. Such training data can be obtained from, for example, a database or library containing versions of past movies and / or series episodes represented in different aspect ratios, including manually created panning and scanning versions (e.g., by a filmmaker or artist approved by the filmmaker). In addition, the training data used to train different models can include different historical panning and scanning conversions performed manually. For example, training data specific to a filmmaker, film company, genre, etc. can be used to train a model trained to predict the panning and scanning framing of a specific filmmaker, film company, genre, etc. After training, additional reasoning data 330 is input to the trained model 325 during the reasoning phase to generate an output 340, which includes the panning and scanning framing predictions made by the training model 325. Although not shown, the training data 310 may also be prepared prior to use for training, such as by extracting features from the training data 310 and determining historical pan and scan framing decisions by comparing versions of the same training data video at different aspect ratios for use as output examples (if such output examples are not otherwise provided, such as by a user). As described, in some configurations, the extracted features may include features indicative of visual and / or audio elements associated with a shot, and these features may be determined from the video itself, associated metadata, and / or elsewhere.
[0043] The training process at 320 can use any appropriate training technique, and the specific technique selected will generally depend on the type of model (or rule) being trained. For example, the above-mentioned back propagation with gradient descent can be used to train a deep learning model. As another example, the above-mentioned ID3, C4.5, CART, CHAID or MARS techniques can be used to train a decision tree. In some configurations, the training process at 320 uses a subset of the training data 310 to train the model 325, and then uses another subset of the training data 310 to evaluate the model 325. For example, most of the training data 310 can be used to train the model 325, and the rest of the training data 310 is used to evaluate the trained model 325. The evaluation of the trained model 325 can include verifying that the model 325 performs well enough when making panning and scanning framing predictions, such as producing panning and scanning frame predictions that are less than the expected error from the manual panning and scanning framing in the evaluation data set. The goal of training is to obtain a model (or rules) that is able to predict what a user, such as a filmmaker, chooses as the pan and scan framing of an image frame of a video being converted from one aspect ratio to another. Although one set of training data 310 is shown for illustrative purposes, additional training data (e.g., automatically predicted pan and scan framing that is subsequently modified by a filmmaker) may be used to retrain and improve a previously trained model.
[0044] As shown, the inference phase includes receiving inference data 330, which may include a video (e.g., a movie or TV series episode or portion thereof) and associated metadata, which has not been seen before by the model 325, assuming that the model 325 is selected for use by the aspect ratio conversion module 130, and after preparing such data in the same manner as the training data 310 (e.g., by extracting features), the inference data 330 is input into the model 325. In turn, the model 325 generates as output 340 a pan and scan framing for image frames of a shot in the video, which can predict what a user (e.g., a filmmaker) would choose.
[0045] Figure 4 An example pan and scan conversion according to aspects of the present disclosure is shown. Although for illustrative purposes, a conversion of a single image frame 400 from a widescreen aspect ratio to a smaller aspect ratio is shown, the aspect ratio conversion application 100 may generally convert multiple frames of a video between aspect ratios on a shot-by-shot basis, as described above. Furthermore, for simplicity, the actions performed by the aspect ratio conversion application 100 are described as follows: Figure 4 However, it should be understood that this action can actually be performed by components of the aspect ratio conversion application 100 (e.g., the above-mentioned lens recognition module 110, feature extraction module 120 and aspect ratio conversion module 130), or any other system configured to perform the above-mentioned operations.
[0046] Figure 4 The illustrated example assumes that the aspect ratio conversion application 100 uses a function including a trained model (e.g., a decision tree or deep learning model) that predicts where to crop the widescreen image frame 400, and the user-defined rule described above, where if a speaking person is identified in the shot, the speaking person must appear in the reframed image frame of the shot. As shown in panel A, the aspect ratio conversion application 100 extracts features including visual and audio elements (shown as bounding boxes 420 and 422, 424 and 426, and 428 and 430 around the persons 412, 414, and 416 and their faces, respectively), as well as features that determine that the person 412 is speaking in the shot. As described above, other features that indicate visual (e.g., image characteristics and technical information) and audio (e.g., the subject matter of music or dialogue) elements of the shot and / or creative information may also be extracted.
[0047] Based on the rule that the speaking person must appear in the output image frame of the shot, the aspect ratio conversion application 100 determines an area 410 of the widescreen image frame 400 that can be cropped to a smaller aspect ratio while retaining the speaking person 412 in the reframed image frame. For example, if the rightmost side of the widescreen image frame 400 is to be selected as a framing with a smaller aspect ratio, the speaking person 412 may be (at least partially) cropped out when the speaking person 412 appears on the right side of the widescreen image frame. The aspect ratio conversion application 100 may prevent such removal of the speaking person 412 by determining an area 410 where no pan and scan framing is predicted to result in the speaking person 412 being cropped out. For example, the aspect ratio conversion application 100 may determine an area of the widescreen image frame 400 where the rightmost and leftmost framing will retain the speaking person 412 in the output image frame at a smaller aspect ratio.
[0048] As shown in Panel B, the aspect ratio conversion application 100 uses a function, which may include trained models and / or rules, to determine pan and scan framing predictions 420 for an image frame 400. In one configuration, a determined region 410 of a widescreen image frame 400 is (appropriately sized and) input to the function, which in turn outputs a pan and scan framing prediction 420.
[0049] As shown in panel C, a user modification is made to the predicted pan and scan framing 420, resulting in a different framing 430. Panel D shows an output image frame 440 after the widescreen image frame 400 is cropped according to the framing 430. In one configuration, the aspect ratio conversion application 100 (or another application) provides a user interface that is displayed via a display device and allows a user to preview the automatically predicted pan and scan framing (e.g., framing prediction 420), as well as to modify such framing predictions. As described, a film studio may be contractually obligated to obtain a filmmaker's approval of a pan and scan conversion for the filmmaker's film, in which case the filmmaker may be allowed to modify the automatically predicted pan and scan framing.
[0050] Figure 5 A method 500 for converting image frames between aspect ratios according to aspects of the present disclosure is shown. Although the method 500 is described with respect to actions performed by the aspect ratio conversion application 100, it should be understood that the steps of the method 500 may actually be performed by components of the aspect ratio conversion application 100 (e.g., the shot identification module 110, the feature extraction module 120, and the aspect ratio conversion module 130), or more generally, by any system configured to perform the method steps in any order in some configurations.
[0051] As shown, the method 500 begins at step 510, where the aspect ratio conversion application 100 (or another application) uses machine learning techniques and training data (including manual panning and scan conversion) to determine one or more aspect ratio conversion functions. As described above, each aspect ratio conversion function may include one or more rules and / or models (e.g., decision tree rules, rules in knowledge-based systems, and the above-mentioned deep learning models), which may be obtained by appropriate machine learning training techniques (e.g., decision tree learning algorithms for decision trees, back propagation with gradient descent for deep learning models, etc.). Features such as features indicating visual and audio elements associated with shots from a video may be extracted for training (and subsequent reasoning) purposes, and output examples of manual panning and scanning framing decisions may also be determined by, for example, comparing versions of the same training data video of different aspect ratios (if such output examples are not, for example, provided by a user). The aspect ratio conversion function may (or may not) include user-defined rules.
[0052] At step 520 , the aspect ratio conversion application 100 receives a video, associated metadata, and an indication of an aspect ratio to convert between. The received video is not part of the training data set, but is input to the one or more aspect ratio conversion functions determined at step 510 .
[0053] At step 530, the aspect ratio conversion application 100 identifies shots in the received video. The aspect ratio conversion application 100 may use any suitable technique to identify shots, such as an automatic technique that detects shot boundaries, or by determining shots from received metadata if such metadata indicates shot boundaries.
[0054] The aspect ratio conversion application 100 loops and processes each identified shot at step 540. Although described as a loop for simplicity, the processing of some (or all) identified shots (as well as other processing) is optionally performed in parallel.
[0055] For one of the identified shots, the aspect ratio conversion application 100 determines features associated with the shot and / or a particular image frame therein at step 550. As described, these features may indicate visual elements (e.g., objects that appear in the image frames of the shot); tracking motion of these objects; the direction a person is facing; image features, such as whether the image frame depicts a bright scene or a dark scene, whether the image frame depicts a day scene or a night scene, contrast, saturation, graininess, sharpness, and / or animation effects; as well as technical information (e.g., the camera lens used to capture the image frames in the shot, whether the image frames are digital or film images, whether the image frames are live action or animated content, year, etc.), audio elements (e.g., a person speaking, music being played, the subject of the dialogue, etc.), and / or creative information (e.g., a filmmaker, brand, genre, film company, series, etc. associated with the received video). Each feature can be determined from metadata associated with the received video if such metadata includes information related to the feature, or the aspect ratio conversion application 100 can independently determine the feature in any suitable manner, such as using a CNN to classify and locate objects, speech recognition, and matching with visually identified people or mouth movements to determine the person who is speaking, etc., as described above.
[0056] At step 560, the aspect ratio conversion application 100 determines the pan and scan framing for each image frame in the shot based at least on the features determined at step 530 and using one or more functions determined at step 510. As described, the aspect ratio conversion application 100 can select and use appropriate functions (e.g., functions that are specific to input and output aspect ratios or for a particular filmmaker or film studio), and each selected function includes appropriate automatically learned or user defined rules and / or models as described above. In addition, the aspect ratio conversion application 100 (or another application) can also allow a user to modify the automatically determined pan and scan framing predictions.
[0057] At step 570, the aspect ratio conversion application 100 crops each image frame in the shot based on the pan and scan framing determined for the image frame (or the user's modification of the automatically determined pan and scan framing). Then, at step 580, if there are additional shots to be processed, the method 500 returns to step 540, where the aspect ratio conversion application 100 continues to loop and process the identified shots in the video.
[0058] Figure 6 A system 600 in which aspects of the present disclosure may be implemented is shown. As shown, the system 600 includes, but is not limited to, a central processing unit (CPU) 605, a network interface 615 that connects the system 600 to a network, an interconnect 617, a memory 620, and a storage device 630. The system 600 also includes an I / O device interface 610 that connects an I / O device 612 (e.g., a keyboard, a display, and a mouse device) to the system 600.
[0059] CPU 605 retrieves and executes programming instructions stored in memory 620. Similarly, CPU 605 stores and retrieves application data residing in memory 620. Interconnect 617 facilitates the transmission of such as programming instructions and application data between CPU 605, I / O device interface 610, storage device 630, network interface 615 and memory 620. CPU 605 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, one or more graphics processing units (GPUs), a combination of such processors, and other types of processors. In addition, memory 620 is generally included to represent random access memory. Storage device 630 can be a disk drive storage device. Although shown as a single unit, storage device 630 can be a combination of fixed and / or removable storage devices, such as a disk drive, a flash drive, a removable memory card or an optical storage device, a network attached storage device (NAS) or a storage area network (SAN). In addition, system 600 is included to represent a physical computing system and a virtual machine instance hosted on a set of underlying physical computing systems. Furthermore, although shown as a single computing system, one of ordinary skill in the art will recognize that Figure 6 The components of the illustrated system 600 may be distributed across multiple computing systems connected by a data communications network.
[0060] As shown, the memory 620 includes an operating system 621 and an aspect ratio conversion application 100. The operating system 621 can be, for example, Linux® or Windows®. The aspect ratio conversion application 100 is configured to convert image frames between aspect ratios, such as image frames of a movie or a TV series episode. In one configuration, the aspect ratio conversion application 100 is configured to determine one or more aspect ratio conversion functions using machine learning techniques (multiple machine learning techniques) and training data including manual pan and scan conversion; receive a video, associated metadata, and an indication of an aspect ratio between which to convert; identify shots in the received video; and process each identified shot by: determining features associated with the shot and / or a particular image frame therein, at least based on the determined features and using at least one or more determined functions (multiple functions), determining pan and scan framing for each image frame in the shot, and determining the pan and scan framing for each image frame in the shot according to the above for the aspect ratio conversion function. Figure 5 The described method 500 crops each image frame in a shot based on a pan and scan framing determined for the image frame (or a user's modification of an automatically determined pan and scan framing).
[0061] Although discussed above primarily with respect to converting a widescreen aspect ratio image frame to a smaller aspect ratio for home distribution, which requires cropping the sides of the widescreen aspect ratio image frame, the techniques disclosed herein may also be applied to cropping the top and bottom of an image frame if the original aspect ratio is higher than the final aspect ratio (also known as "tilt and scan" or "reverse pan and scan").
[0062] Although the above discussion is mainly focused on specific examples of features, in optional configurations, other features may be used. For example, such other features may indicate whether the scene depicted in the image frame of the lens includes direct sunlight or ambient light, whether the scene is indoors or outdoors, the atmosphere of the scene, etc.
[0063] Advantageously, the techniques disclosed herein employ specific features, including features indicative of visual and audio elements of image frames in a shot, and creative information, to use machine learning to derive functions for human prediction of the creative decisions that an operator would make in panning and scanning conversions between different aspect ratios. As a result, the techniques disclosed herein can be used to replace or assist panning and scanning conversions that are traditionally performed manually. The techniques disclosed herein can also allow for relatively efficient and consistent creation of content represented in different aspect ratios while satisfying the expectations of filmmakers or other individuals whose historical panning and scanning decisions are used as examples in the machine learning process. Those filmmakers or other individuals can then be free to, for example, spend more time editing the original version of a film or series episode without having to do as much work on the derived panning and scanning versions.
[0064] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various aspects of the present disclosure. Therefore, each block in the flow chart or block diagram can represent a module, segment or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some optional embodiments, the functions shown in the blocks may not occur in the order shown in the figure. For example, depending on the functions involved, in fact, two blocks shown in succession can be executed substantially simultaneously, or sometimes these blocks can be executed in reverse order or out of order. It should also be noted that each block of the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs a specified function or action, or by a combination of dedicated hardware and computer instructions.
[0065] While the foregoing is directed to aspects of the present disclosure, other and further aspects of the disclosure may be devised without departing from the basic scope thereof and the scope of the disclosure is determined by the claims that follow.
Claims
1. A method for converting image frames of a video between aspect ratios, the method comprising: determining an aspect ratio conversion function for converting image frames from a first aspect ratio to a second aspect ratio using a machine learning (ML) model, wherein the ML model is trained using: (i) a first extracted feature indicative of at least one of a visual element or an audio element associated with a first plurality of image frames having the first aspect ratio in footage from a video, and (ii) manual framing decisions in pan and scan conversion of the first plurality of image frames having the first aspect ratio to a corresponding second plurality of image frames having the second aspect ratio; identifying shots in the received video, wherein each of the identified shots includes one or more image frames having the first aspect ratio; as well as For each of the one or more image frames for each identified shot, the corresponding image frame is converted from the first aspect ratio to the second aspect ratio using the determined aspect ratio conversion function.
2. The method according to claim 1, wherein: Converting the one or more image frames from the first aspect ratio to the second aspect ratio includes inputting a second extracted feature indicative of at least one of a visual element or an audio element associated with the one or more image frames to the determined aspect ratio conversion function.
3. The method according to claim 2, further comprising: identifying that one or more characters in the received video are speaking based at least in part on at least one of: (i) mouth movement of the one or more characters in the one or more image frames; (ii) recognition of one or more speech sounds and matching of the recognized one or more speech sounds with a visual description of the one or more persons in the one or more image frames, The second extracted features indicate audio elements related to speech of one or more characters identified as speaking.
4. The method according to claim 2, wherein: The second extracted feature is indicative of an audio element associated with the one or more image frames, the audio element comprising music being played.
5. The method according to claim 2, wherein: The second extracted features indicate visual elements associated with the one or more image frames, the visual elements comprising at least one of: an object appearing in the one or more image frames, a camera lens used to capture the one or more image frames, whether the one or more image frames are digital image frames or film image frames, whether the one or more image frames include live-action or animated content, or features of the one or more image frames.
6. The method according to claim 5, wherein: The visual elements associated with the one or more image frames include objects appearing in the one or more image frames, and wherein the objects include at least one of characters, creatures, props, screen text, or environmental objects.
7. The method according to claim 5, wherein: The visual elements associated with the one or more image frames include features of the one or more image frames, and wherein the features of the one or more image frames include at least one of the following: whether the one or more image frames depict a daytime scene or a nighttime scene, whether the one or more image frames are bright or dark, whether the one or more image frames are captured under direct sunlight or ambient light, whether the one or more image frames depict an indoor scene or an outdoor scene, or the atmosphere of the scene depicted in the one or more image frames.
8. The method according to claim 2, wherein: Converting the one or more image frames from the first aspect ratio to the second aspect ratio is further based on at least one of a film producer, brand, genre, film company, series, or film franchise associated with the received video.
9. The method according to claim 1, wherein: The one of the determined aspect ratio conversion functions comprises at least one of a rule in a knowledge-based system, a decision tree, or a deep learning model.
10. The method according to claim 1, further comprising: A user modification is received and applied to the framing of respective image frames determined in converting the one or more image frames from the first aspect ratio to the second aspect ratio.
11. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing unit, cause the processing unit to convert an image frame between aspect ratios by: determining an aspect ratio conversion function for converting image frames from a first aspect ratio to a second aspect ratio using a machine learning (ML) model, wherein the ML model is trained using: (i) first extracted features indicative of at least one of visual elements or audio elements associated with a first plurality of image frames of the first aspect ratio in footage from a video, and (ii) manual framing decisions in pan and scan conversion to convert the first plurality of image frames to a corresponding second plurality of image frames having the second aspect ratio; Identify shots in the received video, where Each of the identified shots includes one or more image frames having the first aspect ratio; as well as For each of the one or more image frames for each identified shot, the corresponding image frame is converted from the first aspect ratio to the second aspect ratio using the determined aspect ratio conversion function.
12. The computer-readable storage medium of claim 11, wherein: Converting the one or more image frames from the first aspect ratio to the second aspect ratio includes inputting a second extracted feature indicative of at least one of a visual element or an audio element associated with the one or more image frames to the determined aspect ratio conversion function.
13. The computer-readable storage medium of claim 12, wherein: The operations also include: identifying that one or more people in the received video are speaking based at least in part on at least one of: (i) mouth movement of the one or more people in the one or more image frames; or (ii) recognition of one or more speech sounds and matching the recognized one or more speech sounds with a visual description of the one or more people in the one or more image frames, The second extracted features indicate audio elements related to speech of one or more characters identified as speaking.
14. The computer-readable storage medium of claim 12, the second extracted feature is indicative of an audio element associated with the one or more image frames, the audio element comprising music being played.
15. The computer-readable storage medium of claim 12, wherein: The second extracted features indicate visual elements associated with the one or more image frames, the visual elements comprising at least one of: an object appearing in the one or more image frames, a camera lens used to capture the one or more image frames, whether the one or more image frames are digital image frames or film image frames, whether the one or more image frames include live-action or animated content, or features of the one or more image frames.
16. The computer-readable storage medium of claim 15, wherein: The visual elements associated with the one or more image frames include objects appearing in the one or more image frames, and wherein the objects include at least one of characters, creatures, props, screen text, or environmental objects.
17. The computer-readable storage medium of claim 15, wherein: The visual elements associated with the one or more image frames include features of the one or more image frames, and wherein the features of the one or more image frames include at least one of the following: whether the one or more image frames depict a daytime scene or a nighttime scene, whether the one or more image frames are bright or dark, whether the one or more image frames are captured under direct sunlight or ambient light, whether the one or more image frames depict an indoor scene or an outdoor scene, or the atmosphere of the scene depicted in the one or more image frames.
18. The computer-readable storage medium of claim 11, wherein: Converting the one or more image frames from the first aspect ratio to the second aspect ratio is further based on at least one of a film producer, brand, genre, film company, series, or film franchise associated with the received video.
19. The computer-readable storage medium of claim 11, the operations further comprising: A user modification is received and applied to the framing of respective image frames determined when converting the one or more image frames from the first aspect ratio to the second aspect ratio.
20. A system for converting image frames of a video between aspect ratios, comprising: one or more processors; as well as A memory including a program, wherein when the program is executed on the one or more processors, the program performs operations, the operations comprising: determining an aspect ratio conversion function for converting image frames from a first aspect ratio to a second aspect ratio using a machine learning (ML) model, wherein the ML model is trained using: (i) extracted features indicative of at least one of visual elements or audio elements associated with a first plurality of image frames having the first aspect ratio in footage from a video, and (ii) manual framing decisions in pan and scan conversion of the first plurality of image frames having the first aspect ratio to a corresponding second plurality of image frames having the second aspect ratio; identifying shots in the received video, wherein each of the identified shots includes one or more image frames having the first aspect ratio; and For each of the one or more image frames for each identified shot, the corresponding image frame is converted from the first aspect ratio to the second aspect ratio using the determined aspect ratio conversion function.
Citation Information
Patent Citations
Face detection device
US20060177110A1
