System and method for camera processing in live environment

Abstract

At least one system and method herein for camera processing in a live environment. This includes determining a music score to be performed in a live environment; a live version of the music score and a recorded version of the music score are analyzed in the live environment, where the analysis is to determine a time-based difference of at least one frame in the live version from at least one frame in the recorded version, and where the time-based difference is greater than the time-based difference of at least one frame in the recorded version. A camera device selection or camera device adjustment of one or more camera devices in a live environment will be made using a time-based difference and at least one primary effect of a recorded version or a live version.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a PCT application of U.S. non-provisional application No. 18 / 241,708, filed on September 1, 2023 and entitled “SYSTEM AND METHOD FOR CAMERAHANDLING IN LIVE ENVIRONMENTS”, which in turn is related to and claims the benefit of priority of U.S. provisional application No. 63 / 403,540, filed on September 2, 2022 and entitled “SYSTEM AND METHOD FOR CAMERAHANDLING IN LIVE ENVIRONMENTS”, and the entire disclosures of both U.S. provisional application No. 63 / 403,540 and U.S. non-provisional application No. 18 / 241,708 are incorporated herein by reference for all intents and purposes. Background Art 1. Field of the Invention

[0003] The disclosure herein relates generally to camera processing in a live environment, and in particular to addressing delays in tracking a live version of audio in a live environment and to the benefits of camera direction based on a live version of audio in a live environment.

[0004] 2. Description of the Prior Art

[0005] Audio recognition allows computer-based applications to determine songs by capturing a live version of the song. For known music scores, computer-based applications allow the detection of artists or scores using a short recording of a live version of the music score or a recorded version of the music score from an output device such as a radio, television, or even a person. Some computer-based applications allow the use of humming or other utterances to attempt to detect the music score. Summary of the invention

[0006] A processor-implemented method includes determining, by at least one processor, that at least one frame of a live version of a musical score performed in a live environment corresponds to a recorded frame of a recorded version of the musical score. The method also includes performing, by at least one camera, camera movement based on a time-based reference between the at least one frame and the recorded frame and based on at least one primary effect of the recorded version or the live version.

[0007] The system herein is used to track a live version of a musical score to enable camera processing in a live environment. The system may be associated with at least one camera, a memory, and at least one processor to execute instructions stored in the memory to perform the following steps or functions, the steps or functions including determining that at least one frame of the live version of the musical score performed in the live environment corresponds to a recorded frame of a recorded version of the musical score. The steps or functions also include camera movement by the at least one camera, the camera movement being performed based on a time-based reference between the at least one frame and the recorded frame and based on at least one primary effect of the recorded version or the live version. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0009] Figure 1 An example of a live environment for performing camera processing is shown;

[0010] Figure 2 An example system for camera processing in a live environment is shown;

[0011] Figure 3A and Figure 3B shows example line diagrams associated with methods and systems for camera processing in a live environment;

[0012] Figure 4A A diagram illustrating camera processing in a live environment wherein one or more cameras are selected or adjusted in the live environment to point in a direction associated with at least one primary effect;

[0013] Figure 4B Aspects associated with an immersive experience in a live environment based in part on tracking a live version of audio in the live environment are shown;

[0014] Figure 5A An example method for camera processing in a live environment is shown;

[0015] Figure 5B Another example method for camera processing in a live environment is shown; and

[0016] Figure 6 Another example system for camera processing in a live environment is shown. DETAILED DESCRIPTION

[0017] In the following description, various embodiments will be described. For the purpose of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will also be clear to those skilled in the art that the embodiments may be practiced without the specific details. In addition, known features may be omitted or simplified to avoid obscuring the described embodiments. Various other functions may be implemented within the various embodiments, and may be discussed and suggested elsewhere herein.

[0018] In addition, the disclosure herein generally relates to cameras and devices having cameras, including smartphones, stand-alone cameras, laptops, notebooks, etc. Therefore, references to one or more cameras cover such other devices. The disclosure herein addresses the processing of such cameras in a live environment, such as during a live classical music performance of a piece or work. However, the disclosure herein may also be applied to concerts of other musical genres having multiple performances and cameras to capture different views of the concert. In an example, the camera processing herein is to resolve time-based differences between a live version of the audio (e.g., corresponding to a particular live classical music piece or performance) and a recorded version of the audio (also referred to herein as a reference audio recording) in real time.

[0019] Recorded versions can have many differences, such as different interpretations of different conductors, which may come from studio recordings or from different performances of the same specific classical music piece. The differences from different interpretations can also include different lengths of the same specific classical music piece (such as different total audio lengths). Although some methods can allow the use of computer-based applications for audio recognition to determine the song by capturing the live version of the song, these methods do not involve determining the difference based on time for camera processing. For example, computer-based applications can allow the use of part of the recorded version or even the live version in the live environment (such as from a live performance or output device, including radio, television, or from human speech) to detect artists, music scores, and other related information. For example, some computer-based applications can allow recognition of humming or other human speech for detecting music scores. In contrast, the method herein determines the time-based difference (including the use of AI / ML) between the music score of the recorded version (such as a specific classical music piece) and the same piece of the live version, but these versions may be performed in different ways by conductors, soloists, or orchestras, and, for example, may result in different performance lengths.

[0020] The systems and methods herein may address such deficiencies presented and pointed out throughout this document by systems and methods on systems for camera processing in a live environment. Video production of a live environment (such as a classical music performance or concert) may require a team of trained people, and is therefore a costly undertaking with an element of uncertainty in the results. The methods or systems herein may enable a user to conduct a concert in advance: such as by using a recorded version of a musical score with associated tags, where the musical score will be performed in a live environment; by tracking time-based differences; and by using at least one primary effect of the time-based differences and tags from the live version to address camera selection (selection from a plurality of different cameras) or camera adjustment (changing pitch, tilt, or zoom) of one or more cameras in a live environment.

[0021] Therefore, the method or system herein is an automatic method to perform camera selection or camera adjustment of one or more cameras during a live performance (such as during a concert) representing a live environment. The method allows the preset direction in the tag associated with the recorded version to be modified to be used with one or more cameras. For example, the preset direction can be associated with the music score of the recorded version. In addition, the preset direction can be associated with at least one main effect in the music score, such as which section of the orchestra (wind instrument section, string instrument section, or percussion instrument section) or which musical feature (rhythm, tone, pitch, timbre, intensity, etc.) is dominant in the recorded version, and at which reference moment in the music score. Then, based in part on the input of the position of the live version in the music score relative to the recorded version, the preset direction can be used to guide one or more cameras during the live performance.

[0022] The methods or systems herein also allow for initial camera selection and camera adjustment using, for example, at least one primary effect from a recorded version of a live performance. However, in this approach, the recorded version of the music score is used to ensure that any time-based differences relative to the live version can be resolved by overriding the initial camera selection and camera adjustment with further camera selection and camera adjustment from the live version of the live performance or by overriding the initial camera selection and camera adjustment with a default orientation setting for the camera selection or camera adjustment. For example, a default orientation setting for camera selection or camera adjustment can ensure that if there are unexplained time differences, such as a delayed or extended tune in the live version relative to the recorded version of the music score, a conductor's view is provided.

[0023] Method or system herein can reduce cost by eliminating the intervention of each camera in one or more cameras by the live camera operator, the music reader and the artistic director, and no matter what kind of change may occur in the live version, the reproducible result of the live performance different from the music score can be realized in the video displayed. In addition, although the musician locates himself according to the stage planning on the stage, the musician or even the part of the musician can be located using multiple cameras, so that the display of the music performance of the live version is matched with the stage planning. This can at least identify the musician individually and in parts in an effective way. The camera direction herein allows following the artist or part, such as when the performer moves on the stage.

[0024] The methods or systems herein include one or more cameras that can be pointed in a direction associated with at least one primary effect, such as a flute solo in a wind section. This in turn highlights the flute performer and other such individual musicians or sections when desired. In addition, the methods herein not only consider frequency or rhythm to determine at least one primary effect, but can also use timbre to make such a determination. For example, the methods or systems herein also incorporate speed improvements for video tracking of a live performance for display by selecting the smallest time-based difference from multiple recorded versions of the same musical score to be performed in a live environment.

[0025] The method or system herein includes an artificial intelligence / machine learning engine (AI / ML engine) that is trained to use the time difference between the determined live version of the music score and the recorded version of the music score to infer the tags to be sent to the automatic director module. The AI / ML engine herein can convert the data associated with the live version of the music score and the data associated with the recorded version of the music score into a numerical vector. For example, tokenization can be performed to divide the data into multiple parts, such as one or more frames of a music score. These parts can be further subdivided. The method or system herein enables tokenization to generate tags that summarize the relationship between the recorded version of the data and the tags to be associated with the data, such as by transmitting it to the automatic director module for camera selection and camera adjustment. As a result of the tokenization, a set of unique tags can be generated for the data, and these tags can be associated with the tags.

[0026] In addition, vectorization can be performed as part of an AI / ML engine. Vectorization of tags associated with time-based differences between a live version and a recorded version provides a numeric vector. The numeric vector can be used with feature selection to determine to retain certain features that will be used to automatically assign tags to the time-based differences between the live version of the music score and the recorded version of the music score. The AI / ML engine herein includes storing, at least temporarily, distances between tags. Such distances can be used to reduce the number of vectors required and can reduce the need to compare vectors as part of the AI / ML engine to add tags to time-based differences.

[0027] In at least one embodiment, a separate AI / ML engine may be used in a similar manner to extract features for use with a frame referencing module to provide time-based differences between versions of a musical score. Also in this case, the AI / ML engine used for frame referencing herein includes at least temporarily storing distances between tokens from the tokenization process. Such distances may be used to reduce the number of vectors required to extract features and may reduce the need to compare vectors as part of an AI / ML engine to determine features to use with a frame referencing module.

[0028] In addition, the live version of the music score can be analyzed with respect to multiple recorded versions using this aforementioned method at the same time. This can be performed using parallel processing to improve the robustness and tolerance of various interpretations of the music score in a live environment. The method or system herein may only require that a single vector of a main effect (such as pitch) be compared with a matrix of the same main effect from multiple recorded versions, wherein each matrix is ​​referenced by a reference number in the matrix. In addition, pooling can be used to reduce the matrix size.

[0029] The system associated with the method herein includes a memory and at least one processor to execute instructions stored in the memory to perform the steps in the method. The steps in the method include analyzing audio from a live version (such as live audio from a live performance of a classical music piece) and audio from a recorded version (such as pre-recorded audio of different performances of the same piece of music). The analysis is to determine a time-based difference, which can represent a difference over a period of time. In at least one embodiment, the time-based difference is based on at least one frame of the live version (live performance) relative to the recorded version (pre-recorded performance). In addition, the time-based difference can be used with at least one main effect (such as a flute performer soloing in a classical music piece) to enable camera selection or camera adjustment of one or more cameras during a live classical music performance. In this method, one or more cameras can be directed to at least one main effect of the live environment based on at least one main effect associated with the recorded version, even if there is a time-based difference between the position where the at least one main effect appears in the recorded version and the position where the at least one main effect appears in the live version of the same classical music piece.

[0030] Figure 1 An example live environment 100 for performing camera processing as described herein is shown. The live environment 100 is shown as a classical arrangement for an orchestra, but live concerts of any musical genre can benefit from the disclosure herein. The live environment 100 shown may include the following: a string section 102, 104, 106 having at least a violin and a cello; a woodwind (wind) section 114 having, for example, a clarinet, a bass clarinet, and a bassoon; a brass section 118 having a tuba and a trumpet, and in some cases, the brass section may also be combined with wind instruments; a percussion (or double bass) section 108; a vocal section 110; an auxiliary instrument section 116; and a conductor's podium 112. The string section 102, 104, 16 may be the largest section in a classical arrangement for an orchestra. In addition, the auxiliary instrument section may include a piano, a harp, and other large specialized musical instruments. The orchestra is on stage 130 and the entire stage could be a live environment.

[0031] In the method or system herein, the conductor can use each part in different ways in different parts of the music score. Some music scores may require multiple members of the string section in one part, including a specific string section, such as violin or cello or combination. In addition, some music scores may include a combination of multiple parts, and some music scores may include a separate performance in each part. In the method or system herein, multiple cameras 122 to 130 can be provided, which can be pointed to a single part, a single person, or these cameras can be used for a specific part of a specific person in the live environment. Multiple cameras capture the live environment in real time for display or record the live environment for display later. In addition, one or more cameras 122 to 130 can also be pointed to the audience or bystander 120.

[0032] Figure 2 An example system 200 for performing camera processing in a live environment is shown. The system 200 includes at least one processor and a memory having instructions to be executed on the at least one processor to perform the functions described herein. Further description of this example system may be found herein according to Figure 6 The system includes a live audio / video feedback line 214 for feeding a live version of the music score from the live environment 212 to the sound tracking module 208 through the live version module 202. The live version module 202 includes at least a memory to provide the live version to the sound tracking module 208 in real time.

[0033] The system includes a recorded version module 204 to provide the recorded version of the music score to the sound tracking module 208. Therefore, the recorded version module 204 includes the recorded version of the music score and a library of predetermined information 222 for each of the recorded versions. The predetermined information 222 may include tags (such as meta tags) for the main effects at various time points or reference moments in the recorded version of the music score. In addition, the recorded version module 204 can also retain the music score of the live versions of different performances, which will become different recorded versions of the same music score, thereby providing further robustness to the system 200 herein.

[0034] Thus, a predetermined information 222 may be created and stored with different recorded versions. In an application, different recorded versions of the same music score may be used to select a recorded version with the minimum time-based difference from the live version using the methods herein. For example, parallel processing may be used to process at least one frame of each different recorded version in parallel with the frames of the live version in real time. A reliable recorded version (reference audio) may then be selected from the different recorded versions based at least in part on a reliability coefficient. In an example, a time-based difference may be generated and minimized to a minimum value, wherein the minimum value represents a reliability coefficient. Thus, a recorded version with the minimum time-based difference may be selected so that the associated recorded version tag is used with the live version. The systems and methods herein may then react more quickly to resolve camera selection and camera adjustment based on the minimum time-based difference. In the methods or systems herein, the live version and the recorded version may be provided to the vectorization module 206A in a combined manner as digital data.

[0035] The sound tracking module 208 can perform vectorization of the digital data via a vectorization module 206A implemented by at least one processor. The vectorization module 206A can also be used to generate vectors based on tags for use with the frame reference AI / ML engine 206B, but can also be used with the tag-associated AI / ML engine 210. The tag-associated AI / ML engine 210 can primarily use the time-based differences 216 determined by the sound tracking module 208 to perform tokenization and vectorization of the data, respectively. The frame reference AI / ML engine 206B generates features from one or more of the recorded version of the music score or the live version of the music score. These features are provided to the frame reference module 206C.

[0036] The frame referencing module 206C of the sound tracking module 208 performs frame referencing using an appropriate algorithm, such as using online dynamic time warping (ODTW). The frame referencing module 206C provides the time-based differences 216 from the output of the sound tracking module 208 to the tag-related AI / ML engine 210, which then infers or predicts in real time the tags 218 to be provided to the live version. The tag-related AI / ML engine 210 can learn how to tag the live version based in part on the time-based differences 216 between the music score of the recorded version and the music score of the live version and in part on the tags from the predetermined information 222. The tag-related AI / ML engine 210 can then associate those tags from the predetermined information 222 of the recorded version with the frames of the live version based on the time-based differences 216.

[0037] The tags 218 may be provided to the automatic directing module 220 to identify at least one major effect (such as a flute solo) in the recorded version of the music score that also appears in the live version of the music score. The tag-associated AI / ML engine 210 may notify the automatic directing module 220 of these tags 218, taking into account time-based differences between the live version of the music score and the recorded version of the music score. For example, an initial time-based difference (such as a 2 second delay) may be learned after starting the live version, and the tag-associated AI / ML engine 210 may project the tags 218 from the predetermined information 222 to the rest of the live version when it occurs, so that the cameras 122 to 130 may be correctly directed at a future point in time and in real time.

[0038] In addition, the tag-related AI / ML engine 210 can beneficially account for the ongoing time-based differences as the live version continues. Therefore, the tag-related AI / ML engine 210 can be trained using time-based differences to infer or predict additional points in the live version that correspond to the recorded version, so that when these time points in the live version are approaching, the tag information is notified to the automatic broadcast module 220. Since the live version may have additional time-based differences, the tag-related AI / ML engine 210 is able to update the tags 218 for these points as the time-based differences change. The tag-related AI / ML engine 210 can also be used to create tags based on the provided predetermined information 222. For example, these tags 218 can be a combination of one or more tags from the predetermined information 222. The tags 218 can be converted into a specific camera preset in the automatic broadcast module 220, which can send a signal to one or more cameras to make the selections or adjustments described throughout this document. In one example, the NN of the tag-related AI / ML engine 210 receives the predetermined information 222, which may include a stage plan and metadata (with tags) predetermined and associated with the recorded version.

[0039] Thus, once trained, the tag-associated AI / ML engine 210 can infer tags 218 associated with future points in one or more frames of the live version of the music score. The tag-associated AI / ML engine 210 transmits the tags 218 to the auto-directing module 220. The tags allow the auto-directing module 220 to send a signal to one or more cameras 122 to 130. The signal is used to perform camera selection or camera adjustment of one or more cameras 122 to 130 in the live environment. The sound tracking module 208 and the tag-associated AI / ML engine 210 can analyze the live version of the music score frame by frame. The time-based differences 216 are transmitted to the tag-associated AI / ML engine 210 for each received frame of the live version analyzed in the live environment relative to one or more frames of the one or more recorded versions of the music score.

[0040] The recorded version module 204 includes predetermined information 222 indicating at least one main effect in the musical score. The predetermined information 222 can then be passed to the AI / ML engine 210 associated with the tag when a time-based difference 216 (such as a delay, a slower tempo, etc.) is determined. The AI / ML engine 210 associated with the tag uses the predetermined information 222 and the time-based difference 216 provided from the frame reference module 206C to determine a tag 218 to be notified to the automatic guide module 220. The tag 218 transmitted to the automatic guide module 220 allows the module 220 to select or adjust one or more cameras 122 to 130 in the live environment to point in a direction associated with the at least one main effect using the time-based difference from the frame-by-frame analysis. Therefore, since the tag 218 can include a future tag for a predetermined time before the current point in the live version, the direction occurs immediately or in an expected manner. Thus, system 200 is capable of determining delays or tempo changes in a live version of a musical score, is capable of determining at least one major effect that is delayed, faster, or otherwise different than a recorded version, and is capable of selecting or adjusting at least one camera to account for the at least one major effect.

[0041] The method or system herein includes feature extraction of a live version or a recorded version, which can be performed in a frame reference AI / ML engine 206B, which is different from the tag-related AI / ML engine 210 associated with the tagging method described herein. The sound tracking module 208 may include a frame reference AI / ML engine 206B for feature extraction. For example, the frame reference AI / ML engine 206B may include an autoencoder to perform such feature extraction. For example, the autoencoder may be a vector quantized variational autoencoder (VQ-VAE). VQ-VAE attempts to find a potential space (such as a variable) with an underlying representation in the provided data (such as data associated with a live version or a recorded version). VQ-VAE maps the live version and the recorded version from the received digital data in a frame-by-frame manner, VQ-VAE performs the tokenization described elsewhere herein, and then VQ-VAE provides a vector associated with the tag. The tokenization can be represented by an embedded index. The method herein can use VQ-VAE with ODTW to select features and use these features to determine time-based differences.

[0042] The frame reference AI / ML engine 206B can be provided with a stored embedding vector so that there is no need to perform a comparison of the vectors to learn the features to be used with the ODTW algorithm. In addition, a similar method can be performed on the tag-related AI / ML engine 210, which can learn how to add tags to the live version based on tags from the recorded version and time-based differences, and can provide the tags to be notified to the automatic director module 222. The frame reference AI / ML engine 206B and the tag-related AI / ML engine 210 can be regularly updated to the average of the most recently labeled data. The method or system herein can use a moving average in a method. In addition, the frame reference AI / ML engine 206B can be trained to infer the differences between the live version of the music score and the recorded version of the music score. The tag-related AI / ML engine 210 is used to infer the tags to be notified to the automatic director module 220.

[0043] The neural network (NN) of the AI / ML engine 210 associated with the tag can learn from the time-based differences of the sound tracking module 208 to dynamically align the input layer with the weights within the NN for the described tags available to it. In addition, the frame reference AI / ML engine 206B can be dynamically updated in a similar manner to dynamically align the input layer with the weights within its NN for the described features of the audio version. The tags 218 can be defined for use with different main effects, or can be generated by an adversarial neural network. For example, a discriminator of such an adversarial neural network can be trained to classify forced changes to a recorded version, where a generator is used to generate forced changes. The forced changes can have specific main effects with known tags from the predetermined information 222, which are notified to the discriminator after training. Therefore, in this method, automatic tags can be added to specific main effects, or tags can be generated for use with the NN of the AI / ML engine 210 associated with the tag.

[0044] The weights of the NN for the tag-related AI / ML engine 210 can be updated based on time-based differences and can be used with neurons in subsequent layers. For example, when the ODTW analysis indicates that there are no differences between the versions, the weights of the NN remain unchanged, and when frames of the live version of the musical score are processed in the AI / ML engine 210, the NN may not assign or notify any labels. This may be a feed-forward approach for the NN for the tag-related AI / ML engine 210. For differences pointed out by the ODTW analysis, the NN may be offset, resulting in the output identifying at least one label associated with the difference, and the NN may be associated with a major effect in the live version or the recorded version. Therefore, time-based differences in the versions can be automatically resolved and time-based differences in the versions can be automatically labeled without resolving specific major effects of the live version relative to the recorded version.

[0045] Therefore, in Figure 2 In the system 200, the system 200 may include at least one processor as part of the automatic navigation module 220, the tag-related AI / ML engine 210, and / or the immersive experience module 224. The at least one processor may perform a determination that at least one frame of the live version 202 of the music score performed in the live environment 212 corresponds to a recorded frame of the recorded version 204 of the music score. Then, at least one camera 122 to 130 may be caused to perform a camera movement that is performed based on a time-based reference, including a time-based difference 216 between the at least one frame and the recorded frame, and the time-based difference 216 is based on at least one primary effect of the recorded version 204 or the live version 202.

[0046] In addition, Figure 2 In the embodiment of the present invention, the primary effect is in the form of instructions stored in a memory associated with at least one processor or in a different memory so that the music score of the recorded version 204 can be accessed from such a memory. The instructions can be executed by at least one processor. In addition, these instructions for the primary effect can be different from the instructions for the functions executed at the at least one processor to determine at least one frame and perform camera movement. For at least one frame, the instructions associated with the primary effect can include at least a first instruction for capturing at least one musical performer using camera movement through at least one camera 122 to 130, or the instructions associated with the primary effect can include at least a second instruction for generating and capturing a type of immersive experience by at least one camera 122 to 130. In addition, additional steps or functions to be performed by at least one processor (such as at least one processor of the automatic broadcast module 220) include: performing camera movement to capture musical performers or immersive experiences; and combining or including camera movement with the live version 202 of the live environment 212.

[0047] In another example, using Figure 2 The system 200 in the embodiment of the present invention can be implemented to provide instructions for the main effect based on computer input. For example, a character or graphical user interface can be used to receive instructions for the main effect. Figure 6 The primary effect provided by the computer-based input may be based on a musical score of the recorded version 204. In one example, the immersive experience is generated for at least one frame of the live version 202. In addition, the immersive experience may be at least one of the following types: Figure 4B 452 in the live environment 212 / 402, for example, the graphical visualization 452 is displayed during the live performance; or text 454 is displayed to the audience of the live performance. For example, the automatic broadcast module 220 provides input to the immersive experience module 224 to cause the projection 226 and other forms of immersive experience features to provide the graphical visualization 452 or text 454. Although described with respect to a specific portion of the view 410 (percussion section 410), the immersive experience can be provided in any part of the live environment 212 / 402 itself.

[0048] The system 200 also supports adjusting the camera movement or immersive experience based on features derived from the audio and video of the audio / video feedback line 214 of the live performance. For example, such features may include at least one of the volume or emotional expression level of the music performer. The adjustment may include at least one of the camera zoom associated with at least one camera 122 to 130 or the color of the graphic visualization 452 or the text 454. In one example, the emotional expression level of the music performer can be determined based at least in part on the object tracking AI / ML algorithm of the object tracking module 228, which tracks 230 the facial matrix of the music performer in the live version of the music score. The AI / ML algorithm can be different from the frame reference AI / ML engine 206B or the AI / ML engine 210 associated with the tag. The object tracking AI / ML algorithm can be trained using facial matrices for different emotional expressions. For example, an object tracking AI / ML algorithm may be able to determine different emotional expressions of a musical performer by receiving a facial matrix filtered (or processed in any manner to extract facial details) from a capture of a musical performer and, for example, by classifying the facial matrix into at least one of different emotional expressions.

[0049] In one aspect, adjustments to camera movement or immersive experience are also based in part on object tracking 230 performed for the music score of the live version 202. For example, the live version 202 can accept input from the object tracking module 228, and the live version 202 can allow faster processing at near real-time speeds compared to real-time processing. This may be due to one or more of the additional video processing steps in the audio / video feedback line 214 having been performed for the live version compared to the live version captured in real time. For example, the object tracking module 228 may need to perform additional processing when directly capturing the live environment 212. For example, this is shown by the dotted line pointing to the object tracking module 228.

[0050] In addition, the system 200 includes using multiple cameras to perform camera movement of at least one camera 126, 128. For example, the camera movement includes a specific action performed on a specific camera of the at least one camera 122, 124, 126, 128, 130 to provide views 406 to 414 of a specific portion, the views 406 to 414 including views of a string instrument section 408, a wind instrument section 414, and a percussion instrument section 410. The camera movement can be modified in real time based at least in part on a determination of a problem with the at least one camera or another camera associated with the at least one camera. For example, if a camera is not working, the automatic broadcast module 220 can use the remaining cameras to compensate for the loss of views 406 to 414 of the specific portion.

[0051] The system 200 also supports determining, for at least one frame of the live version of the music score, a recorded frame of the recorded version of the music score based on a frame referencing AI / ML engine 206B for extracting features describing at least one frame of the live version 202 and a recorded frame of the recorded version 204. The frame referencing AI / ML engine 206B can use online dynamic time warping (ODTW) to determine associations between audio features.

[0052] One or more AI / ML engines 210 herein may be based at least in part on feedback provided by a user. For example, the feedback may be collected via a computer-based input, and wherein the feedback includes errors in frame matching. For example, a user may use Figure 6 The associated described handheld device provides such computer-based input remotely. Feedback can be added to the training data set for the AI / ML engine 210. The system herein enables any AI / ML algorithm herein (such as Figure 2 Modules 206B, 210, 228 shown in can use the results of a previous live performance of a music score. For example, using at least one determined recorded frame can achieve continuous frame matching between the live version and the recorded version.

[0053] Any AI / ML algorithm herein may use a confidence level, wherein a first predetermined value of the confidence level allows the AI / ML algorithm to affect camera movement by one or more of the following: switching to another type or setting of primary effects, or maintaining at least one camera 122 to 130 or multiple cameras in a switching mode until the machine learning algorithm reacquires a second predetermined value of the confidence level for the next frame of the live version of the musical score or the recorded version of the musical score. The switching herein may include at least one of the following: redirecting at least one camera 122, 124, 126, 128, 130 or two or more cameras 122 to 130 to capture a conductor of the musical score, or reducing an effect associated with at least one primary effect based on an immersive experience associated with the live version of the musical performance.

[0054] Any AI / ML algorithm herein may take the form of a collection of models. This allows each model in the collection to take as input the same or different set of features extracted from the audio signal of the audio / video feedback line 214. Each of these model collections may have ODTW applied separately. The separate outputs of the ODTW program for each model collection in the model collection may be synthesized together, and a confidence may be calculated for the synthesized output. The synthesized output of two or more model collections may be an average of the outputs. In one example, the confidence may be based in part on a comparison of whether the outputs of two or more model collections in the model collection are different from each other. In addition, if all or nearly all of the outputs are the same (or similar, for example, according to a predetermined variation), the confidence for two or more model collections in the model collection is equal to or nearly equal to a maximum value, such as a value of "1".

[0055] In another example, the more different the outputs of two or more ensemble models are from one another, the more likely it is that a lower confidence determination will be made. Still further, when the confidence is too low relative to a predetermined value (also referred to herein as a confidence acceptance threshold) (which is subject to manual or automatic adjustment), the model ensemble may report a problem with the certainty of its output. This may cause the system 200 to operate in an erroneous scenario, where it performs a safety operation, such as to provide a default orientation setting that captures the perspective of a conductor, an audience, or a general orchestra, rather than an individual musician. The safety operation may exist until the model ensemble begins to deliver outputs with a confidence that exceeds the predetermined value.

[0056] Recording version 204 (one of which is in Figure 2 ) may be different versions of a musical score that, along with associated primary effects, may be stored in the system 200 (and / or Figure 6 In a system 600 of the present invention, a recorded version to be used with the live version at the beginning of the music score can be determined from the plurality of recorded versions. The determination can be based in part on a difference, such as the time-based difference 216, that is part of a time-based reference between at least one frame and a separate one of the recorded frames of the recorded version.

[0057] In an example, a recorded version from a plurality of recorded versions may be determined as a recorded version to be used with a live version of a musical score based in part on a confidence level, and an AI / ML algorithm (e.g., one or more of the frame-reference AI / ML engine 206B or the tag-associated AI / ML engine 210) provides an output using at least one frame and a plurality of recorded frames with the confidence level. For example, one or more of the frame-reference AI / ML engine 206B or the tag-associated AI / ML engine 210 may be responsible for determining a recorded frame associated with at least one frame and may be responsible for making such a determination by providing a confidence level. The confidence level may be considered together with the determination of the recorded frame to determine whether to retain the recorded frame or to change the recorded frame. Although the determination of the recorded frame is made, the confidence level provided may be lower than a predetermined value. In this way, the determination of the recorded frame to be used may be rejected, and another recorded frame of a different recorded version having a higher confidence level relative to the predetermined value and to be used with the live version may be used.

[0058] The system 200 includes using an automatic broadcast module 220 to use preliminary instructions to perform camera movement for at least one camera or more cameras 122 to 130 until a recording version with a predetermined confidence value is determined. The determination of the recording version can use a scalable cloud environment (such as Figure 6 The system 200 may be configured to execute in real time or near real time using the plurality of recorded versions available in the system 600. Information associated with the recorded versions may then be sent to at least one camera or to the plurality of cameras 122 to 130. The system 200 may also support the use of edge environments (e.g., Figure 6 In the system 600 , additional recorded frames to be used with the live version are additionally determined in real time by one or more of the frame reference AI / ML engines 206B or the tag-related AI / ML engines 210 .

[0059] In addition, the system 200 supports capturing a live version in real-time synchronization with the live audio using at least one camera or multiple cameras 122 to 130, such as via the provided feedback feature 214. The synchronized capture can be used for at least one of online broadcast, television-based live entertainment production, or the production of a musical score recording. Thus, the camera movement can obtain different live versions of the musical score for different reproductions, such as for online broadcast, television-based live entertainment production, or the production of a musical score recording.

[0060] Figure 3A and Figure 3BExample line graphs 300, 350 associated with camera processing in a live environment of at least one embodiment herein are shown. The method or system herein includes dynamic time warping (ODTW) to perform non-linear normalization of the time between at least two frames for each version of a musical score. In addition, for example, ODTW works best to provide at least a sum of distance values ​​for pairs of indexes that are aligned or ordered in sequence. ODTW provides a static metric that can benefit from optimizing the alignment or order therein. This may be part of dynamic programming performed in ODTW. For example, ODTW compares a numerical representation of a live version to a stored recorded version.

[0061] In the examples herein, a recorded version of a music score and a live version of a music score are received as numerical representations of the music score. The numerical representation of the versions can rely on specific coefficients, such as chromatic features, cepstral coefficients, mel-frequency cepstral coefficients, or perceptual linear prediction (PLP) coefficients. In addition, feature extraction via a frame-referenced AI / ML engine uses pattern comparison by searching for representations between versions of the music score. Figure 3A A line graph 300 is shown of frame coefficients over time 306 for a recorded version 308 relative to a live version 310, with time 306 representing a reference moment of the music score. In addition, the line graph 300 shows y-axes 302, 304 having normalized values ​​for the music score of the recorded version 308 and the music score of the live version 310, for example, including at least one effect that can be used to determine a dominant effect in the recorded version. In addition, a mapping 312 is shown between the versions 308, 310, which represents the time-based differences in the versions.

[0062] The method or system herein utilizes ODTW to minimize the distance between different coefficients at one or more time points 306. For example, the method herein minimizes the distance between coefficients (points in each line 308, 310) for one or more time points 306 by duplicating and matching each such point in the one or more time points 306. In addition, a sliding window may be defined, and each point in the live version 308 is used with all points in the sliding window of the recorded version 310. The cost function of ODTW is used to determine the cost between such points.

[0063] In addition, the distance (and minimum distance) between the initial point and the current point in the sliding window can be determined by minimizing the cost function. For the matching point from the initial point to the recorded version 308, it is also performed in a similar manner. A cost function with a cost value for the final point on the sliding window can be obtained, and then by matching the final points between the versions in the sliding window until the end of the frame in the frame-by-frame analysis, the cost value can be used to define the minimum distance between the versions. When the recorded version is known relative to the live version, the estimated start time can be used and the distance between the points in the lines 308, 310 of each version can be determined. The distance can be minimized in each frame to determine the position of the points of the live version relative to the points of the recorded version, wherein the time information associated with the x-axis is used to define the time-based differences. In addition, a frame can be used in subsequent frame analysis because the time-based differences may run through the entire music score.

[0064] In an example line graph 350 associated with camera processing in a live environment, minimized differences and time-based differences between versions of a musical score are provided for illustrative purposes. The methods and systems herein process numerical representations without generating such line graphs. For example, the x-axis 352, y-axis 354 in the line graph 350 are associated with ranges of a sequence of coefficients used in the ODTW analysis for the versions of the musical score. The ranges may be roughly the same. In the line graph 352, the proportional linear portion 356 implies a match between the versions, while any tilt 358 represents a contraction or expansion in terms of time-based differences between the versions for at least one frame.

[0065] Figure 4A An exemplary camera process 400 is shown in accordance with at least one embodiment herein in a live environment, wherein one or more cameras are selected or adjusted in the live environment to point in a direction associated with at least one primary effect. For the live environment 402, at least one camera provides a default view 404 and a plurality of views 406 to 414 of specific parts. The views 406 to 414 of specific parts include views of a string instrument section 408, a wind instrument section 414, and a percussion instrument section 410. In addition, if one camera is used for two views in the views 404, 406 to 414, one camera can be adjusted using time-based differences and at least one primary effect. However, multiple cameras can be selected and adjusted using time-based differences and at least one primary effect. In addition, a camera is selected at least according to a requirement for making adjustments (such as panning, tilting, or zooming) to focus on one or more subjects of the live environment 402.

[0066] The methods or systems herein allow for initial camera selections and camera adjustments to be made using, for example, at least one primary effect from a recorded version of a live performance. These initial camera selections and camera adjustments may be for providing a view to a viewer on a display screen (as shown) or may be for recording with a live version of the music score. The recorded version of the music score is used to ensure that any time-based differences relative to the live version can be resolved by overwriting the initial camera selections and camera adjustments with further camera selections and camera adjustments from the live version of the live performance or by overwriting the initial camera selections and camera adjustments with a default orientation setting for camera selections or camera adjustments, such as the default view 404. For example, the default orientation setting for camera selections or camera adjustments ensures that if there are unexplained time differences, such as a delayed or extended tune in the live version of the music score relative to the recorded version of the music score, a view of the conductor is provided.

[0067] Figure 4B Aspects 450 associated with immersive experiences 452, 454 in a live environment are shown, based in part on tracking the audio of a live version in the live environment. For example, a camera can automatically point to an object 456 (such as, to a specific musical performer), and the immersive experience 452, 454 can be provided in real time using the provided immersive experience module 224. The musical performer 456 may move in time relative to the performance of the live version. In an example herein that is beneficial to the immersive experience 452, 454, the system 200, using its object tracking module 228 and the camera 120 to 130, can track the musical performer 456 to provide the immersive experience 452, 454. However, when the object tracking module 228 loses the musical performer 456, the system 200 can switch to another operating scenario associated with performing camera processing as a backup, but the immersive experience module 224 can be used to keep the immersive experience 452, 454 ongoing.

[0068] In another example, the object tracking module 228 may include emotion detection of an emotional expression of the music performer 456. The output associated with the emotional expression may be used to provide one of the additional primary effects that affect the camera processing being performed. For example, zooming in or out of at least one of the provided cameras 122 to 130 may be based on the detected emotional expression of the music performer 456. Additionally, when the emotional expression of the music performer 456 is negative, the camera processing may be switched to a default orientation setting.

[0069] System 200 can also be integrated with different types of presentation media software or display media software.For example, immersive experience module 224 can be able to provide graphic visualization 452 and text 454 (such as, "band (THE BAND)" reflects words, lyrics, names, etc.), but can include any other software for generating immersive experience except graphic visualization 452 and text 454.In addition, graphic visualization 452 can move with the audio of music performer 456 and / or live version, and graphic visualization 452 can include three-dimensional (3D) visual effects, face, image, cartoon and other virtualization.Therefore, the graphic visualization, text or any other form of display media of immersive experience herein can be synchronized with one or more of the live audio and live video from the live environment.Displayed media can change according to the characteristics obtained from the live audio.For example, during live performance, when a specific main effect occurs, the graphic visualization software of immersive experience module 224 connected to the display screen of live environment 212 can project corresponding graphic visualization onto the display screen or by the augmentation or virtual reality device projection used with system 200.

[0070] Figure 5A An example method 500 for performing camera processing in a live environment is shown. The method 500 includes determining a musical score to be performed (502). Once the performance begins, the method 500 includes analyzing a live version of the musical score and a recorded version of the musical score in the live environment (504). The analysis (504) may include the ODTW method described throughout this document. For example, the analysis (504) may include determining, by at least one processor, that at least one frame of the live version of the musical score performed in the live environment corresponds to a recorded frame of the recorded version of the musical score. Determining a time-based difference based on the analysis in step 504 (506). In addition, a verification may be performed as to whether at least one main effect is in the live version or the recorded version (508).

[0071] The primary effect may be predetermined and may be associated with the recorded version. For example, a solo from an orchestra member, a performance from a particular section, and other such aspects may be a primary effect that requires the camera to focus on the member or particular section. However, when the primary effect occurs for a live version of the musical score, a time-based difference between the live version of the musical score and the recorded version of the musical score may change. The time-based difference forms the basis for the AI / ML engine to provide a tag to perform camera selection or camera adjustment (510) of one or more cameras in a live environment. For example, at least one processor uses the tag to determine one or more cameras to be selected or adjustments to be made to one or more cameras. For example, the step of performing (510) may be performed using at least one processor and / or by at least one camera. The step of performing (510) may be a camera movement performed based on a time-based reference between at least one frame and the recorded frame and based on at least one primary effect of the recorded version or the live version.

[0072] Figure 5A Also shown is an example method 500 for performing camera processing in a live environment. The method 500 includes determining (502) a musical score (such as a musical composition) to be performed. Once the performance begins, the method 500 includes analyzing (504) the live performance of the musical composition with a recorded version (such as a reference musical composition) in the live environment. The analysis (504) may include the entire ODTW method described herein. Based on the analysis in step 504, time-based differences (506) are determined. In addition, major effects in the live version may appear in the live version in a delayed or otherwise varied form relative to the recorded version. The method herein will generate a default view, such as to select a camera pointing at the conductor, or to zoom, pan, or tilt the camera in the direction of the conductor to obtain such major effects in the live version of the musical score.

[0073] Figure 5B Another example method 550 for performing camera processing in a live environment is shown. The method 550 may replace Figure 5A Method 500 in, at least because Figure 5B The method 550 in will result in a change in the camera device. For example, Figure 5BThe method 550 in the embodiment includes determining a musical score to be performed (552). Once the performance begins, the method 550 includes determining at least one primary effect in a live version of the musical score or a recorded version of the musical score (554). The method 550 also includes analyzing the live version of the musical score and the recorded version of the musical score frame by frame in a live environment (556). The analysis (556) may include the entire ODTW method described herein. A validation may be performed for a time-based difference determined according to the analysis in step 556 (558). In addition, the method 550 includes selecting or adjusting one or more cameras in the live environment to point in a direction associated with at least one primary effect (560) using the time-based difference according to the frame-by-frame analysis of step 556. The system 200, 600 herein supports turning on or off at least one function associated with the processor-implemented method 500, 550 using a computer-based input that reports at least one aspect of the musical score. For example, at least one function that can be turned on or off is controlling the level of automation of camera movement.

[0074] Figure 6 Another example system 600 for performing camera processing in a live environment is shown. The system 600 may include computer and network aspects to perform at least Figure 2 Figure 3 Figure 5A ,and Figure 5B All or part of the features described in . These computer and network aspects 600 may include a distributed system. The distributed system 600 may include one or more computing devices 612, 614. The one or more computing devices 612, 614 may be suitable for executing and running with a client application (such as a browser or a stand-alone application) and for executing and running on one or more networks 606.

[0075] In at least one embodiment, the computing device herein may be a unit of a system for performing certain portions of the features described herein, such as vectorization. However, such a computing device may include a smartphone to perform at least portions of such features described herein. For example, a smartphone with a camera may be used instead of a camera to receive information about directions, such as adjustments or directions pointing to the smartphone. In one example, different types of devices may be selected or adjusted for use throughout the camera processing described herein, including devices for recording after registration for use with the system herein. In addition, since some live environments may involve the movement of one or more artists, it is beneficial to bring other audiences / onlookers into the system to incorporate their cameras into the system through instructions sent to their smartphones. These directors at least indicate some adjustments to enable their smartphones to be used for directing in a live environment.

[0076] In addition, the server 604 having components 604A to 604N can be communicatively coupled to computing devices 612, 614 via a network 606 and via a receiver device 608 (if provided). The components 612, 614 include processors, memory, and random access memory (RAM). The server 604 can be adapted to operate a service or application to manage functions and sessions associated with database 602 access and computing devices 612, 614. The server 604 can be associated with one or more cameras 608 of the system 620 for camera processing in a live environment 622.

[0077] The encoder / decoder 618 and the transmitter / receiver 616 may be associated with the system 620 to allow remote monitoring and control of various aspects of camera processing in a live environment. The encoder / decoder 618 and the transmitter / receiver 616 may include a processor and a memory having instructions that, when executed by the processor, may cause the encoder / decoder 618 and the transmitter / receiver 616 to collectively perform the functions described throughout this document and at least with reference to Figure 2 3, and 5. For example, camera control may include signals for camera selection and adjustment sent directly from server 604 to camera 608. However, the automatic guide module may be brought closer to the camera via encoder / decoder 618 and transmitter / receiver 616 to perform camera control on camera 608.

[0078] Server 604 can be located in the on-site environment, but can also be located in a location different from the on-site environment to perform the entire present invention and at least reference to Figure 23, and 5. Such a server 604 may support a system 620 for camera processing in a live environment 622. Such a system 620 may operate partially within the live environment 622. Such a tool 620 may include subsystems for performing the functions described throughout this document.

[0079] The subsystem may be a module capable of testing or training portions of the system 620, as described herein at least with respect to ODTW and tags. The subsystem may be packaged in one or more computing devices having at least one processor and memory, such that the at least one processor may perform functions based in part on instructions from the memory executed in the at least one processor. Although shown together, the system boundary 620 may be part of one or more cameras 608 and encoders 618 and transmitters / receivers 616. The server 604 and computing devices 610 to 614 may be located in a different geographic location than the live environment.

[0080] One or more cameras 608 of the system 620 for camera processing in a live environment are provided so as to be able to capture views associated with a live environment 622. The system for camera processing in a live environment can be adapted to transmit the received information and the additional information processed therein by wire or wirelessly. The encoder and transmitter 616 can communicate with one or more components within the system 620 and external to the system 620.

[0081] One or more components 604A to 604N may be adapted to function as a provisioning device within the server 604. In addition, one or more components 604A to 604N may include one or more processors and one or more memory devices, thereby being adapted to perform camera processing in a field environment, while other processors and memory devices in the server 604 may perform other functions.

[0082] The server 604 may also provide software-based services or applications in a virtual or physical environment (such as to support the simulations cited herein). When the server 604 is a virtual environment, components 604A to 604N are software components that can be implemented on the cloud. This feature allows remote access to information received and communicated between any of the aforementioned devices. One or more components 604A to 604N of the server 604 may be implemented in the form of hardware or firmware, rather than the software implementation described herein as a whole. The methods or systems herein also allow the use of their combinations.

[0083] One computing device 610 to 614 may be a smart monitor or display having at least a microcontroller and memory with instructions for being able to display information from camera 608. One computing device 610 may be a transmitter device to transmit directly to a receiver device or to a receiver device via network 606 and to server 604, and the other computing devices 612, 614, which may be part of an encoder and transmitter 616.

[0084] Other computing devices 612, 614 may include portable handheld devices, including but not limited to smartphones, cellular phones, tablet computers, personal digital assistants (PDAs), and wearable devices (head-mounted displays, watches, etc.). In addition, other computing devices 612, 614 may operate one or more operating systems, including Microsoft Windows (Any generation) and / or various mobile operating systems such as Windows Palm and / or variations thereof.

[0085] Other computing devices 612, 614 may support applications designed as Internet-related applications, electronic mail (email), text messaging or multimedia messaging service (SMS or MMS) applications, and may use other communication protocols. Other computing devices 612, 614 may also include applications running Microsoft Apple and / or Other computing devices 612, 614 may be general-purpose personal computers and / or laptops running such operating systems. or UNIX-like operating systems or other GNU / Linux operating systems (such as Google Chrome ) workstations. Including gaming systems (Microsoft )'s thin client device can be used as other computing devices 612, 614.

[0086] The methods or systems herein use a network 606, which can be any type of network that can support data communications using various protocols, including TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), One or more networks 606 can be based on Ethernet, token ring, wide area network, Internet, virtual network, virtual private network (VPN), local area network (LAN), intranet, extranet, public switched telephone network (PSTN), infrared network, wireless network (such as in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, and / or any other wireless protocol, etc.), and / or any combination of these and / or other networks.

[0087] The server 604 runs a suitable operating system, including any operating system described throughout this document. In addition, the server 604 can also run some server applications, including HTTP (Hypertext Transfer Protocol) server, FTP (File Transfer Protocol) server, CGI (Common Gateway Interface) server, Server, database server, and / or variants thereof. Database 602 is supported by a database server feature of server 604 equipped with front-end capabilities. Such database server functionality includes commercially available (International Business Machines), and / or variations thereof.

[0088] Server 604 can provide feeds and / or real-time updates for media sources. Server 604 is part of a plurality of server boxes distributed in the region but functioning for the presently described process. Server 604 includes applications for measuring network performance through network monitoring and traffic management. The provided database 602 can store information from the field environment including user interactions, usage pattern information, adaptation rule information, and other information.

[0089] The method or system herein is a SaaS (Software as a Service) method that uses instructions on a non-transitory medium that, when installed on and executed using at least one processor, can cause a processor of a system for camera processing in a live environment to perform certain functions. For example, the production of a video from a concert may be a function that ties together one or more of the aspects described herein. The method herein can replace an entire video production team. At the same time, the method herein ensures a higher standard for quality and artistic integrity within the industry.

[0090] Organizers of classical concerts and other types of concerts and live performances may be able to use this method and system via a web-based or standalone platform. This gives at least full control and flexibility to the event organizer to record and / or stream each live concert. In addition, using the AI / ML method in this article, a combination of a live version and multiple video streams can be automatically generated according to predefined and changing scenarios (such as fixed direction or changing direction or adjustment). This video is then the guide video of the concert. The system in this article also ensures that every performance of classical music will be recorded, and each recording will reflect the guide carefully planned for the live environment.

[0091] The method and system herein enable concert organizers to use tools that will allow them to carry out their activities in the digital and Internet fields. For example, the software method herein automates the process of recording a live environment (including the live version of the music score in the performance) by tracking the music score relative to the recorded version. Predefined and dynamic concert guides that are consistent with the music score and consistent with the idea of ​​the guide can be implemented, such as by moving the camera device to point to the performer or part. In addition, the method and system herein ensure that the solution can be generally applied to conventional concerts, and can also be applied to progressive avant-garde (modern) philharmonic orchestras, music universities, music academies, opera houses, independent artists, wherein reducing the high recording costs is achievable.

[0092] The methods and systems herein can match a live version of a music score with a recorded version of the music score using ODTW. In a live environment, ODTW can determine in real time the points of the live version of the music score relative to the recorded version of the music score. Then, using a pre-prepared script, control of the direction of one or more cameras can be performed to record or display a video associated with a main effect in the live version. For this purpose, a camera that allows remote control of the lens and saves the frames used is used. The camera is then broadcast or transmitted only based on the main effects that may be marked as instructions in the recorded version. In addition, images from the camera can be recorded in a multi-track format and in a mixed form, and images from one camera are displayed based on the tags.

[0093] The methods and systems herein incorporate software with a set of libraries that enable, among other things, the multiple features described herein. For example, one feature is tracking music scores for tonal music references, as well as tracking music scores for references in contemporary and atonal music. Additionally, live environments (including concerts) are recorded according to directions set by concert organizers, eliminating the need to manually calibrate cameras and convert high-level labels to specific camera settings. Still further, multi-track camera recordings and mixes (such as video synchronized with a live version) can be used with images from one camera selected by an administrator. Thus, the live environment can be at least a replay of a previously recorded live version, and the video is altered in part based on one or more frames of additional camera footage that are available and selected or adjusted to fit the replay of the live version.

[0094] The methods or systems herein may enable fully automated directing by storing or implementing tags in the recorded version without the need for additional information. For example, such information related to the main effects may be converted into high-level tags and provided for a particular camera setup. Then, using the knowledge of the tags that now represent the established scene layout and techniques for automatic scene calibration (using tools such as Deep Camera), a directing of a multi-camera recording may be recorded in the form of a mixed synchronized video and sound, showing images that are always from one selected camera.

[0095] The method or system herein can be based on an audio recognition problem that can solve the problem of the performer's current position in the music score. In addition, the sound in the current live version can be adjusted to the recorded version. An administrator (such as a director) can use a streaming media floor plan to assign or determine the recorded version. This determination must be annotated in the system for performing camera processing in a live environment to implement the method herein, and therefore the determination of the music score to be performed is an action within the system, such as a computer-based input, to enable the features described herein to begin to be used for camera processing in a live environment.

[0096] In the method or system herein, a live version can be matched to a recorded version using ODTW, which can operate on appropriately processed sound (such as using chromatic features, variational autoencoders, vector quantization, and variational autoencoders). In addition, in a live environment, since the ODTW module can determine in real time the portion of the music score being played, this determination can be used to control camera movement. The camera is capable of remote lens movement, as well as saving frames unless in a current recording mode. In addition, images from the cameras can be recorded in the form of multi-track and synchronized video and sound based on prepared scenes, thereby displaying at least one image from one camera at all times.

[0097] The systems and methods herein include software running on a server at a remote location rather than in a live environment. For example, portions of the software may be located in the cloud, which allows the client panel to make different adjustments. In addition, audio / video signal retransmission may be performed after the audio / video signal is received from the live server in the live environment to the remote environment. The systems and methods herein are scalable to different numbers of cameras, which can adapt the system for camera processing in a live environment to different customer requirements.

[0098] The method or system herein is capable of tracking classical music tracks and recording and / or broadcasting a live environment, such as a live concert. There may be necessary programming layers and data streaming between the live version of the music score and the recorded version of the music score. There may be key layers supported by innovative programming libraries that are responsible for preprocessing of the live version, tonal music tracking, or novel preprocessing methods for contemporary and atonal music tracking using variational autoencoders or VQ-VAE neural networks.

[0099] The method or system herein includes that online dynamic time warping (ODTW) can be used during a live performance to determine the position of the live version relative to the recorded version. The method and system herein enable a complete transformation of the tracking system based on multiple recorded versions of the same music score. This element stabilizes the operation of the algorithm described herein with respect to ODTW and AI / ML and ensures the correct process of tracking tracks that may have different renditions and different tempos.

[0100] The functionality of automatic scene calibration can be to perform fully automatic broadcasting without the need for additional information that converts high-level tags into the form of specific camera settings. In addition, there may be other components in the methods and systems herein to correspond to common programming requirements in organizational infrastructure. Such requirements can schedule recorded versions to track live versions according to a schedule.

[0101] The systems and methods herein rely on a programming library that implements methods for processing audio signals; supports determining the location of a live version relative to the location of a recorded version and implements the ODTW method; supports the process of updating the systems herein for camera processing in a live environment; supports the implementation of tools for director preparation, with particular emphasis on determining camera movements relative to the recorded version; and supports the process for mapping tags to camera movements, with particular emphasis on remote movement of camera lenses. In summary, frames and their development can be saved according to the output order of the library that implements the tools for camera processing in a live environment.

[0102] In addition, the systems and methods herein include: a programming library for implementing methods for recording audio with multiple tracks and mixed tracks; a development library for implementing tools for synchronizing a live version with a mixed recorded version; and a programming library for implementing methods for automatic scene calibration.

[0103] In addition, the programming libraries herein can be adapted to determine the type of musician (such as an instrument) to implement methods for determining the positioning (coordinates) of people on the stage (detected using a neural network) based in part on a calibration space specified by the library that implements tools for automatic scene calibration, and then comparing those coordinates to the coordinates of the musicians' parts on an inaccurate stage plan.

[0104] Additional programming libraries herein support directing of the preparation process. The software architecture used in the method and system may have dedicated common patterns or standards. This approach may consider the necessary programming layers and the data / information flowing between them. In addition, there may be key layers corresponding to libraries / modules.

[0105] The method or system herein includes instructions (including documentation and example implementations) for integrating a programming layer into the features mentioned above, which may illustrate adaptations to existing and missing components. The system and method herein is capable of recording and / or transmitting a live environment (including live performances) based on the use of multiple cameras mixed in real time to address the complexity and cost of arranging live concert recordings and / or broadcasts, which is handled by professional live production and production companies.

[0106] Additionally, there may be many people involved in a live production, including camera operators, artistic directors, music readers, etc. There may be associated costs for streaming management and setup for technical staff. There may be logistical issues in such live productions, such as associated with recording audio from multiple cameras.

[0107] The systems and methods herein also enable concert organizers and individual artists to participate in and share concerts online in digital form, making it easier to reach a wider audience. In the case of the developed solution, achieving such a goal would require hiring a specialized company engaged in live broadcast production and realization, which would lead to complexity and cost overruns, which would be beneficial to the community seeking such content. Using the developed system to automate the entire process of live broadcasting of classical concerts enables concert organizers to save costs and complexity while increasing potential audiences.

[0108] The systems and methods herein can act as a camera operator, sheet music reader, and artistic director, all rolled into one. The systems and methods use a neural network model supported by VQ-VAE aspects for preprocessing to solve problems associated with atonal classical music, as described with respect to one or more of the figures in the figures herein. The systems and methods herein enable everyone to watch high-quality multi-camera concert broadcasts and make classical music more accessible than ever before. In addition, this approach can replace the per-concert model that requires advance planning and high costs. The unlimited model of the present systems and methods can enable customers to quickly and easily record each performance.

[0109] The method herein is for tracking a live version of a musical score to enable camera processing in a live environment. A system associated with the method includes a memory and at least one processor for executing instructions stored in the memory to perform the steps of the method. The steps of the method include determining a musical score to be performed in a live environment. The method also includes analyzing the live version of the musical score and a recorded version of the musical score in the live environment. The analysis is used to determine a time-based difference of at least one frame in the live version with respect to the recorded version, wherein the time-based difference is used with at least one primary effect of the recorded version or the live version to perform camera selection or camera adjustment of one or more cameras in the live environment.

[0110] Another method herein is for performing camera processing in a live environment. A system associated with the method includes a memory and at least one processor to execute instructions stored in the memory to perform the steps in the method. The steps in the method include determining a musical score to be performed in the live environment. The method also includes determining at least one primary effect in a live version of the musical score or a recorded version of the musical score. The live version of the musical score and the recorded version of the musical score are analyzed frame by frame in the live environment. The method includes selecting or adjusting one or more cameras in the live environment to point in a direction associated with the at least one primary effect using time-based differences from the frame-by-frame analysis.

[0111] Although the technology herein may be modified and alternatively constructed, these changes are within the spirit and scope of the present disclosure. Therefore, certain illustrated embodiments are shown in the drawings and have been described in detail above, but these embodiments do not limit the present disclosure to the specific one or more forms disclosed; on the contrary, all modifications, alternative constructions, and equivalent contents are included, which fall within the spirit and scope of the present disclosure, as defined in the appended claims.

[0112] In the context of describing the disclosed embodiments (especially in the context of the following claims), terms such as "a," "an," "the," and similar references are understood to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context, and are not intended to serve as definitions of the terms. Include, have, include, and contain are understood to be open-ended terms (meaning phrases such as including, but not limited to), unless otherwise noted. Connected (when unmodified and referring to a physical connection) may be understood as partially or fully contained within, attached to, or joined together, even if there are things in between.

[0113] Recitation of numerical ranges herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. Use of terms such as "set" (for a group of items) or "subset" are to be understood to include a non-empty set of one or more members, unless otherwise noted or contradicted by context. Further, the term "subset of a corresponding set" does not necessarily mean a true subset of the corresponding set, but a subset and a corresponding set may be equal, unless otherwise noted or contradicted by context.

[0114] Connective language of phrases of the form "at least one of A, B, and C," or "at least one of A, B, and C," are generally understood from the context to be generally used to indicate that an item, term, etc. may be A or B or C, or any non-empty subset of the set of A, B, and C, unless expressly stated otherwise or clearly contradicted by the context. In addition, a set having three members, such as "at least one of A, B, and C," and "at least one of A, B, and C," refers to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective language is generally not intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. In addition, unless expressly stated otherwise or contradicted by the context, terms such as "plurality" denote plural status (e.g., a plurality of terms refers to a plurality of terms). The number of terms in a plural is at least two, but may be more when expressly or contextually indicated. Further, phrases such as "based on" and the like mean based at least in part on rather than solely on, unless stated otherwise or clearly indicated by context.

[0115] Unless otherwise indicated herein or clearly contradicted by context, Figure 5A and Figure 5BThe operations of the methods and sub-steps described herein may be performed in any suitable order. The methods or systems herein include processes, such as those described herein (or variations and / or combinations thereof), which may be performed under the control of one or more computer systems configured with executable instructions, and which may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed collectively or exclusively by hardware or a combination thereof on one or more processors.

[0116] Such code may be stored on a computer-readable storage medium. Such code may be a computer program having instructions executable by one or more processors. A computer-readable storage medium is a non-transient computer-readable storage medium that does not include transient signals (such as propagating transient electrical or electromagnetic transmissions), but includes non-transient data storage circuits (such as buffers, caches, and queues) within a transceiver of transient signals. In addition, code (such as executable code or source code) is stored on a set of one or more non-transient computer-readable storage media, on which executable instructions (or other memories for storing executable instructions) are stored, and when the executable instructions are executed by one or more processors of a computer system (such as the result of being executed), the computer system is caused to perform the operations described herein.

[0117] A set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more individual non-transitory storage media in the plurality of non-transitory computer-readable storage media lack all code, while the plurality of non-transitory computer-readable storage media collectively store all code. The executable instructions are executed such that different instructions are executed by different processors - the non-transitory computer-readable storage medium stores the instructions and the main central processing unit (CPU) executes some instructions while other processing units execute other instructions. In addition, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0118] The computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such computer system is configured with applicable hardware and / or software that can implement the performance of the operations. The computer system that implements at least one embodiment of the present disclosure is a single device or a distributed computer system having multiple devices that operate in different ways so that the distributed computer system performs the operations described herein and so that a single device does not perform all operations.

[0119] Although the above discussion provides at least one embodiment of an implementation of the described technology, other architectures may be used to implement the described functionality and are intended to be within the scope of the present disclosure. In addition, although specific responsibilities may be assigned to components and processes, they are defined above only for ease of discussion, and various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0120] Although the subject matter has been described in language specific to structures and / or methods or processes, it should be understood that the subject matter claimed in the appended claims is not limited to the specific structures or methods described. Rather, the specific structures or methods are disclosed as example forms of how the claims may be implemented.

[0121] Based on the above, it will be readily understood by those skilled in the art that the tools of the present disclosure provide many technical and commercial advantages and can be used in a variety of applications. Various embodiments can be combined or modified based in part on the present disclosure, which is readily understood to support such combinations and modifications to achieve the benefits described above.

Claims

1. A processor-implemented method, the processor-implemented method comprising: determining, by at least one processor, that at least one frame of a live version of a musical score performed in a live environment corresponds to a recorded frame of a recorded version of the musical score; as well as Camera movement is performed by at least one camera, the camera movement being performed according to a time-based reference between the at least one frame and the recorded frame and according to at least one main effect of the recorded version or the live version.

2. The processor-implemented method of claim 1 , wherein: The primary effect is in the form of instructions that are stored in a memory with the recorded version of the music score and are to be executed by the at least one processor, and wherein, for the at least one frame, the instructions include: capturing, by the at least one camera, at least one musical performer using the camera movement; or generating and capturing, by the at least one camera, a type of immersive experience, and wherein the processor-implemented method further includes: performing movement of the camera device to capture the musical performer or the immersive experience; and The camera movement is combined with the live version of the live environment.

3. The processor-implemented method of claim 2 , further comprising: The instructions are enabled to be provided as computer based input and the instructions are based on a recorded version of the musical score.

4. The processor-implemented method of claim 2, wherein: The immersive experience is generated for at least one frame of the live version, and wherein a type of the immersive experience is at least one of: a graphical visualization displayed during the live performance; or text displayed to an audience of the live environment.

5. The processor-implemented method of claim 4, further comprising: The camera movement or the immersive experience is adjusted based on characteristics obtained from the audio and video of the live environment, wherein the characteristics include at least one of the volume or emotional expression level of the musical performer, and wherein the adjustment includes at least one of camera zoom or adjusting the color of the graphic visualization or the text.

6. The processor-implemented method of claim 5, further comprising: The level of emotional expression of the music performer is determined based at least in part on a machine learning algorithm tracking a facial matrix of the music performer in a live version of the music score.

7. The processor-implemented method of claim 5, wherein: Adjustments to the camera movement or the immersive experience are also based in part on object tracking performed for the live version of the music score.

8. The processor-implemented method of claim 1 , further comprising: The camera movement of the at least one camera is performed using a plurality of cameras, wherein the camera movement comprises a specific action performed on a specific camera of the plurality of cameras.

9. The processor-implemented method of claim 8, further comprising: The camera movement is modified in real time based at least in part on a determination of a problem with the at least one camera or another camera associated with the at least one camera.

10. The processor-implemented method of claim 1 , further comprising: For the at least one frame of the live version of the music score, a recorded frame of the recorded version of the music score is determined based on a machine learning algorithm, the machine learning algorithm extracts features used to describe the at least one frame and the recorded frame, and the machine learning algorithm uses online dynamic time warping (ODTW) to determine the association between audio features.

11. The processor-implemented method of claim 10, further comprising: continuously improving the machine learning algorithm based at least in part on feedback provided by a user, wherein the feedback is collected via a computer-based input, and wherein the feedback includes errors in frame matching; and The feedback is added to a training data set for the machine learning algorithm.

12. The processor-implemented method of claim 10 , the computer-implemented method further comprising: The machine learning algorithm is enabled to use results from a music score from a previous live environment, wherein the recorded frames determined using the at least one frame enable continuous frame matching between the live version and the recorded version.

13. The processor-implemented method of claim 10 , further comprising: Enabling the machine learning algorithm to use a confidence score, wherein a first predetermined value of the confidence score allows the machine learning algorithm to affect camera movement by one or more of: switching to another type or setting of primary effects, or maintaining the at least one camera or multiple cameras in a switching mode until the machine learning algorithm regains a second predetermined value of confidence for a next frame of a live version of the music score or a recorded version of the music score.

14. The processor-implemented method of claim 13, wherein: The switching includes at least one of redirecting the at least one camera or cameras to capture a conductor of the musical score, or reducing an effect associated with the at least one primary effect based on an immersive experience associated with a live version of the musical performance.

15. The processor-implemented method of claim 1 , further comprising: storing a plurality of recordings and associated primary effects, the plurality of recordings being different versions of a musical score; as well as A recorded version is determined from the plurality of recorded versions to be used with the live version at the beginning of the musical score based in part on differences between the at least one frame and respective ones of a plurality of recorded frames of the plurality of recorded versions.

16. The processor-implemented method of claim 1 , further comprising: A recorded version of a plurality of recorded versions is determined to be used with the live version of the musical score based in part on a confidence level, and a machine learning algorithm responsible for determining a recorded frame associated with at least one frame uses the at least one frame and a plurality of the recorded frames to provide an output using the confidence level.

17. The processor-implemented method of claim 1 , further comprising: Using preliminary instructions, camera movement for the at least one camera or the plurality of cameras is performed until a recording version having a predetermined confidence value is determined, wherein the determination of the recording version is performed in real time or near real time using a scalable cloud environment and using a plurality of recording versions; and sending information associated with the recorded version to the at least one camera or cameras; and Additional recorded frames to be used with the live version are additionally determined in real time using an edge environment associated with the scalable cloud environment.

18. The processor-implemented method of claim 1 , further comprising: The live version is captured using the at least one camera or multiple cameras, wherein the capture is synchronized with the live audio in real time and is used for at least one of online broadcast, television-based live entertainment production, or production of a musical score recording.

19. The processor-implemented method of claim 18, further comprising: The camera movement is performed to obtain multiple live versions of the music score for use in different ones of the online broadcast, the television-based live entertainment production, or the production of the music score recording.

20. The processor-implemented method of claim 1 , further comprising: At least one function associated with the processor-implemented method can be turned on or off using a computer-based input that reports at least one aspect of the musical score, the at least one function being for controlling a level of automation of camera movement.

Images