Network-based processing and distribution of multimedia content for live music performances

Abstract

Methods, systems, and computer program products for network-based processing and distribution of multimedia content for live music performances are disclosed. In some implementations, a recording device can be configured to record a multimedia event (e.g., a music performance). The recording device can provide the recording to a server while the event is in progress. The server automatically synchronizes, mixes, and masters the recording. The server performs the automatic mixing and mastering using reference audio data previously captured during a rehearsal. The server streams the mastered recording over the Internet or other public or private network to a plurality of end users. The streaming can be live.

Description

[0001] Information related to divisional application

[0002] This case is a divisional application. The parent application of this divisional application is the invention patent application filed on July 20, 2017, with application number 201780045266.X, entitled "Network-based processing and distribution of multimedia content for live music performances". Technical Field

[0003] This invention generally relates to the capture, processing, and distribution of multimedia content from live music performances. Background Technology

[0004] Distributing high-quality audio and video of live performances via the Internet can be challenging. Many amateur video recordings uploaded to the Internet have poor video and sound quality. When a band plays in an acoustically untreated venue, the sound quality can be poor if the recording is uploaded directly without further processing. For example, if a drum kit is used, it may be played so loudly that the other instruments in the band cannot be heard clearly. Additionally, if the band does not properly set up their recording equipment (e.g., including multiple microphones, preamps, and a mixing console), the recording of the performance may have low sound quality. Even with proper equipment setup, the band may lack the technical expertise to use it effectively. Similarly, professional-quality video recording and editing of a performance may require technical expertise beyond the performer's skills. Summary of the Invention

[0005] This invention discloses systems, program products, and methods for video editing based on rehearsal and live data. The system receives rehearsal data about a performance from one or more microphones and one or more video cameras. The system matches sound with performers based on the rehearsal data. During a live performance, the system receives live audio and video data of the performance. Based on analysis of the rehearsal data, the system derives the performer's grade relative to the rehearsal and the performer's representative position in the one or more video cameras during the rehearsal. The system then edits the video data based on rules for highlighting the performer using the derived grade and position; the system optionally uses analysis of its performance to improve the editing, the analysis generating, for example, rhythm or beat data and performer movement tracking data. The system then associates the audio data with the edited video data for storage and streaming to one or more user devices.

[0006] This invention discloses systems, program products, and methods for video processing under limited network bandwidth. A video camera can capture high-definition video (e.g., 4K video) of a performance. The video may be difficult to stream live (or even upload offline) via a communication network. The video camera can submit one or more frames of the video (optionally at a lower resolution and optionally compressed using a lossy video codec) to a server system. Based on the one or more frames and audio data, as described in previous paragraphs, the server system can generate editing decisions for the video data. The server system can instruct the video camera to trim a portion of the high-definition video corresponding to a performer or group of performers, and submit that portion of the video as medium or low-definition video (e.g., 720p video) (optionally compressed using a lossy video codec) to the server system. The video camera device can continuously store a long buffer (e.g., tens of seconds) of high-definition video (e.g., 4K) corresponding to the last captured frame, allowing instructions received from the server system to be applied to frames captured seconds earlier. The server system can then store the medium or low resolution video or stream the medium or low resolution video to a user device.

[0007] This invention discloses a network-based processing and distribution scheme for multimedia content used in live performances. In some schemes, a recording device may be configured to record an event (e.g., a live musical performance). The recording device provides the recording to a server during the performance. The server automatically synchronizes, mixes, and masters the recording. In one scheme, the server performs the automated mixing and mastering using reference audio data captured during rehearsals, in which the recording device and sound source are placed in the same acoustic (and, in the case of video recording devices, visual) arrangement as in the event. The server provides the mastered recording to multiple end-user devices, for example, via live streaming.

[0008] In some implementations, the server streams video signals of a live event to multiple users. Using reference audio data (also referred to as rehearsal data) recorded during rehearsals, the server determines the locations of various instruments and singers (hereinafter also referred to as "sound sources") and the locations of performers at the recording locations. During a live performance, the server determines one or more dominant sound sources based on one or more parameters (e.g., volume). An image capture device (e.g., a video camera) can capture live video of the performance and send it to the server. Using the locations of the dominant sound sources, the server determines a portion of the video to apply video editing operations (e.g., scaling, transformation, visual effects). The video editing operations can occur in real time on the live video or on previously recorded video data. The server streams a portion of the video corresponding to the dominant sound source (e.g., a close-up of the lead singer or lead guitarist) to the end-user device. In some implementations, the audio and video streaming can be interactive. For example, the server may provide a video overlay or graphical user interface on the end-user device that allows the end-user to control audio mixing (e.g., increasing the volume of a singer or solo instrument) and video editing (e.g., zooming in on a specific performer). In some embodiments, the server may issue commands to the audio or video recording device to adjust one or more recording parameters (e.g., adjusting the recording level of a microphone preamplifier, the zoom level of a video recorder, turning a specific microphone or video recorder on or off, or any combination thereof).

[0009] The features described in this specification offer one or more advantages over conventional audio and video technologies. These features improve upon conventional manual audio and video processing techniques by automating track mixing and mastering based at least in part on reference audio data obtained from reference audio data. Therefore, bands do not need audio recording or mixing expertise to produce well-sounding recordings of their live performances. Using the automated mixing and mastering processes disclosed herein, bands can produce a harmonious sound without resorting to professional recording, mixing, and mastering engineers. If a band desires a mixing style from a specific expert, they can use the web-based platform disclosed herein to hire an expert for remote mixing and mastering of their recordings.

[0010] Similarly, the disclosed implementation improves upon conventional video processing techniques by replacing manual camera operations (e.g., panning and zooming) with automated camera operations based at least in part on audio and video rehearsal data. The video can be automatically zoomed in or focused on individual performers or instruments. Bands can produce and edit professional-quality videos of their live performances without employing professional cameramen.

[0011] Bands can use various technologies (e.g., live streaming) to deliver high-quality audio and video to multiple end-user devices. To enhance the end-user experience, streaming can be made interactive, allowing end-users to control various aspects of audio mixing and video editing. In this specification, for convenience, the term "band" may refer to an ensemble of one or more performers and instruments. The term may also refer to a group of one or more participants in a non-musical environment (e.g., performers in a play, speakers in a conference, or speakers in a broadcasting system).

[0012] The features and processes disclosed herein improve upon conventional server computers by configuring a server computer to perform automated synchronization, mixing, and mastering of audio tracks for live performances, as well as editing of video data. The server computer can stream processed audio and video to end-user devices and provide control that allows end-users to further mix or edit the audio and video. In various embodiments, the server computer can store raw data from the live performance for offline use, mixing, mastering, repurposed, segmented, and curated. The server computer can store processed data for later distribution. The server computer can store data that has undergone various processing stages (anywhere between raw data and fully processed data, inclusive). The server can store data on storage devices (e.g., hard drives, optical discs (CDs), remote storage websites (e.g., cloud-based audio and video services), or memory modules).

[0013] The features and processes disclosed herein improve upon conventional server computers by allowing server computers to automatically edit video data based on various rules. A server computer implementing the disclosed technology can guide a recording device (e.g., a video camera) to automatically focus on a performer (e.g., a soloist) when the performer sings differently from other performers (e.g., louder), when the performer moves, or when the performer sings unaccompanied (e.g., a cappella). The server computer can cut and change scenes according to the rhythm and beat of the music. The server computer guides the recording device to track the movement of a sound source, including (for example) switching from a first performer to a second performer; therefore, the switching can be a hard cut or a slow pan from the first performer to the second performer. Tracking can be performed on the recorded data without physically moving the recording device. Therefore, based on audio data analysis, the server computer can mimic what a human cinematographer can do. The disclosed technology therefore has the technical advantage of moving the view of an event without physically moving the recording device.

[0014] The features and processes disclosed herein improve upon conventional server computers by reducing the bandwidth requirements for transmitting high-definition video data. High-definition video data (e.g., 4K video) can require high bandwidth for transmission. The disclosed features selectively transmit highlighted areas of the video (e.g., the portion corresponding to the soloist's position) and focus on said position. The system can transmit portions of video data at lower resolutions, such as 720p video. Therefore, when viewers are only watching the soloist, the system does not need to transmit the entire stage in 4K video. The system still maintains the perceived clarity and transparency of the soloist. Thus, the system achieves the technical advantage of transmitting high-quality video with reduced bandwidth.

[0015] In another embodiment, one or more camera devices capture high-resolution (e.g., 4K) video, and the method described herein is used to stream intermediate edited lower-resolution video (e.g., 1080), allowing the server system to further make decisions on editing within 1080 frames and supplying 720p to the audience.

[0016] Details of one or more embodiments of the disclosed subject matter are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the disclosed subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0017] Figure 1 It is a diagram illustrating the first instance of the arrangement of the recording device at the event location.

[0018] Figure 2 This is a diagram illustrating the second instance of the arrangement of the recording device at the event location.

[0019] Figure 3 It is a block diagram illustrating an example architecture of a recording device.

[0020] Figure 4 It is a diagram illustrating the architecture of an example audio and video system used for network-based audio processing.

[0021] Figure 5 It is a block diagram illustrating an example signal path used in audio and video processing.

[0022] Figure 6 It is a flowchart of an instance of audio processing.

[0023] Figure 7 This is a block diagram illustrating an example of an automated mixing and mastering unit.

[0024] Figure 8 It is a flowchart illustrating an example of automated leveling.

[0025] Figure 9 It is a flowchart illustrating an example of automated translation.

[0026] Figure 10 A diagram illustrating an instance of angular transformation in maximum warping.

[0027] Figure 11 It is a flowchart illustrating an example of how to infer energy levels based on microphone signals.

[0028] Figure 12 It is a flowchart illustrating an instance of the process of inferring energy levels in a frequency band.

[0029] Figure 13 It is a flowchart illustrating an example of the automatic equalization process for individual sound sources.

[0030] Figure 14A It is a diagram representing an instance of a three-instrument mix after equalization.

[0031] Figure 14B This is a diagram illustrating the instance gain in automatic equalization.

[0032] Figure 15 It is a flowchart illustrating an instance of video segmentation based on novel accumulations in audio data.

[0033] Figure 16 It is a diagram illustrating the process of accumulating examples of novelty.

[0034] Figure 17 This is a flowchart of an example process for synchronizing signals from multiple microphones.

[0035] Figure 18 The diagram illustrates an example sequence for synchronizing five microphones.

[0036] Figure 19A and 19B The diagram illustrates an instance of the user interface showing the results of automated video editing.

[0037] Figure 20 This is a flowchart illustrating an example of automated video editing.

[0038] Figure 24 This is a flowchart illustrating an example of noise reduction process 2400.

[0039] Figure 25 It is a flowchart illustrating an example technique for video editing based on rehearsal data.

[0040] Figure 26 It is a flowchart illustrating an example of video editing based on rehearsal data.

[0041] Figure 27 This is a block diagram illustrating an exemplary technique for selecting subframe regions from full-frame video data.

[0042] Figure 28 It is a flowchart of an instance of selecting a subframe region from full-frame video data, executed by the server system.

[0043] Figure 29 This is a flowchart of an instance of a process performed by a video capture device to select a subframe region from full-frame video data.

[0044] Figure 21 This is a diagrammatic explanation of the implementation reference. Figures 1 to 20 And a block diagram of an example device architecture of a mobile device with the features and operations described in sections 24 to 29.

[0045] Figure 22 yes Figures 1 to 20 A block diagram of an instance network operating environment for mobile devices numbered 24 to 29.

[0046] Figure 23 Implementation reference Figures 1 to 20 And a block diagram of an instance system architecture of the server system with the features and operations described in sections 24 to 29.

[0047] Similar reference symbols indicate similar elements in various diagrams. Detailed Implementation

[0048] Instantiated upstream architecture

[0049] Figure 1 This is a diagram illustrating a first exemplary arrangement of the recording apparatus at a live performance event 100. Event 100 can be any event in which audio content (e.g., speech, vocal, or instrumental) and optionally video content are produced. Specifically, event 100 can be a concert in which one or more instruments and / or one or more singers perform. One or more sound sources can be present at event 100. Each sound source can be an instrument, a singer, a loudspeaker, or any item that produces sound. For simplicity, sound sources that include non-instrumental sound sources are collectively referred to as instruments in various parts of this specification.

[0050] In some embodiments, devices 102, 104, and 106 may be configured to record audio and video of event 100. Devices 102 and 104 may be mobile devices (e.g., smartphones, wearable devices, or portable audio and video recorders). Devices 102 and 104 may include an internal microphone, be coupled to an external microphone, or both. If an external microphone is used, it may be coupled to one or more microphone preamplifiers. The external microphone may be coupled to devices 102 and 104 using wired or wireless connections. In some embodiments, each of devices 102 and 104 may be coupled to one or more external sound generating devices, which generate audio signals directly based on analog electrical signals (e.g., keyboard output) or digital signals (e.g., digital sound generated by a laptop computer). These signals may be fed directly to devices 102 and 104 via corresponding adapters.

[0051] Each of devices 102 and 104 can execute an application for recording audio content for event 100. The application can transmit the recorded audio track to a remote server computer via communication network 110. Communication network 110 can be a personal area network (PAN, e.g., Bluetooth). TM The network can be a local area network (LAN), a cellular network (e.g., a 4G or 5G data network), a wide area network (WAN, e.g., the Internet), or an ad hoc network. Communication can be made via a gateway (e.g., wireless device 108) or individually. In some embodiments, the server computer may be local to event 100. For example, the server computer may be either device 102 or 104.

[0052] In some implementations, each of devices 102 and 104 may include a client application linked to an online user account for audio processing. The client application may perform user authentication and authorization before sending audio tracks to the online user account. The client application may include audio processing capabilities that respond to commands from a remote server, such as commands to adjust filters (e.g., low-pass filters, high-pass filters, layered filters), the gain of microphone preamplifiers incorporated into or coupled to devices 102 or 104, or to devices 102 or 104. Alternatively, the commands may control the bit depth and sampling rate of the recording (e.g., 16 bits @ 44.1 Hz).

[0053] Each of devices 102 and 104 can submit the recorded audio track to the server via a wired device (e.g., a wired or wireless router) or other wired or wireless device 108. Wireless device 108 can be a wireless access point (AP) for a wireless local area network (WLAN) or a cell tower. Wireless device 108 can be connected to a communication network 110. Devices 102 and 104 can send the recorded live audio track to the server via the communication network 110. Data submission can occur in real-time (e.g., while a performance is in progress) or offline (e.g., simultaneously or sequentially by devices 102 and 104 after a performance has partially or completely concluded). Devices 102 and 104 can store the recorded audio track for offline submission.

[0054] In some embodiments, device 106 is an image capture device configured to capture images and audio of event 100. For example, device 106 may be configured to capture high-definition video (e.g., 4K resolution video). Device 106 may capture still images and video of event 100. Device 106 may transmit the captured still images or video to a server via wireless device 108 and communication network 110. Device 106 may transmit still images or video in real time or offline. In some embodiments, device 106 may perform the operations of a server computer.

[0055] In some implementations, one or two microphones of devices 102, 104, and 106 are designated as one or more primary microphones (e.g., “room” microphones) for capturing audio from all instruments and vocals. The signal output from one or two primary microphones may be designated as a primary signal (e.g., a primary mono or primary stereo signal) or a primary channel signal. Sometimes, other microphones placed at corresponding individual sound sources (e.g., vocal microphones) or groups of sound sources (e.g., drum microphones) are designated as point microphones, also known as satellite microphones. Point microphones can enhance primary microphones (e.g., bass drum microphones, snare drum microphones, hi-hat microphones, top microphones for capturing cymbals, guitar and bass amplifier microphones, etc.) by providing more localized capture of the sound sources.

[0056] In some embodiments, each of devices 102, 104, and 106 may be configured to perform server operations by executing one or more computer programs. In these embodiments, audio signal processing may be performed in situ at event 100. The device performing the operation (e.g., device 106) can then upload the processed signal to a storage device or to an end-user device via communication network 110.

[0057] Figure 2This is a diagram illustrating a second exemplary arrangement of the recording apparatus at event 100. The integrated recorder 200 can be configured to record audio and video signals of event 100. The integrated recorder 200 may include microphones 202 and 204. Each of microphones 202 and 204 may be an omnidirectional, directional, or bidirectional microphone, or a microphone with any directional mode. Each of microphones 202 and 204 may be arranged to point in a given direction. Microphones 202 and 204 may be designated as primary microphones. In various embodiments, the integrated recorder 200 may be coupled to one or more point microphones for additional audio input.

[0058] The integrated recorder 200 may include an image capture device 206 for capturing still images or video of event 100. The integrated recorder 200 may include or be coupled to a user interface for specifying one or more attributes (e.g., target loudness level) of sound sources in event 100. For example, the integrated recorder 200 may be associated with a mobile application via a device identifier. The mobile application may have a graphical user interface (GUI) for display on a touch-sensitive surface of the mobile device 207. The GUI may include one or more user interface items configured to accept user input for specifying attributes of sound sources (e.g., target volume or gain level for a guitar, lead singer, bass, or drums). For example, the attributes may include how many decibels (dB) should be separated between two sound sources (e.g., between a lead singer and other sound sources), which sound source should be dominant (e.g., by playing at a higher volume level than other sound sources), and how much separation is required (e.g., up to X dB). One or more user interface projects can accept user input for specifying rehearsal phases, where reference audio data from sound sources is collected for automated mixing.

[0059] The integrated recorder 200 may optionally perform one or more operations, such as synchronizing signals from a main and spot microphone, separating sound sources from the recorded signal, mixing signals from different sound sources based on reference audio data, and mastering the mixed signal. The integrated recorder 200 may submit the mastered signal as a stereo or multi-channel signal to a server via a wireless device 208 connected to the communication network 210. Similarly, the integrated recorder 200 may provide video signals to the server. The server may then broadcast the stereo or multi-channel signal and video signal substantially in real time to end-user devices during event 100. The communication network 210 may be a PAN, LAN, cellular data network (e.g., a 4G or 5G network), WAN, or ad hoc network.

[0060] Figure 3This is a block diagram illustrating an example architecture of the recording device 302. The recording device 302 can be... Figure 1 Device 102 or 104 or Figure 2 The integrated recorder 200.

[0061] Recording device 302 may include or be coupled to a main microphone 304 and a video camera 306. The main microphone 304 may be a built-in microphone or a dedicated microphone coupled to recording device 302. The main microphone 304 may provide a baseline (also referred to as a foundation) for audio signal processing, as described in further detail below. The video camera 306 may be a built-in camera or a dedicated camera coupled to recording device 302. The video camera 306 may be a Digital Cinema Advocacy Consortium (DCI) 4K, DCI 2K, or Full HD video camera, configured to capture video at a sufficiently high resolution such that a portion of the captured video, if enlarged, can still utilize the full capacity of a conventional monitor with a medium resolution (e.g., 1080p, 1080i, 720p, or 720i).

[0062] Recording apparatus 302 may include an external microphone interface 308 for connection to one or more spot microphones 310. The external microphone interface 308 is configured to receive signals from the one or more spot microphones 310. In some embodiments, the external microphone interface 308 is configured to provide control signals to the one or more spot microphones 310. Recording apparatus 302 may include an external camera interface 312 for connection to one or more external cameras 314. The external camera interface 312 is configured to receive signals from the one or more external cameras 314 and to provide control signals to the one or more external cameras 314.

[0063] The recording apparatus 302 may include one or more processors 320. The one or more processors 320 may be configured to perform analog-to-digital conversion of audio signals from a microphone and to perform digital compression of digital audio signals and video signals from a camera. In some embodiments, the one or more processors 320 are further configured to synchronize audio signals from various channels, separate sound sources from the audio signals, automatically mix individual sound sources, and perform mastering processing on the mixed signals.

[0064] Recording apparatus 302 may include a network interface 322 for submitting digital audio and video signals to a server via a network device. In some embodiments, network interface 322 may submit mastered digital audio and video signals to the server. Network interface 322 may be configured to receive commands from the server for adjusting one or more parameters of the audio or video recording. For example, network interface 322 may receive commands for panning a video camera to a specified direction and zooming in (or out) or adjusting the recording level of a specific microphone.

[0065] Recording device 302 may include a user interface 324 for receiving various user inputs for controlling recording attributes. User interface 324 may be included in a GUI displayed on a touch-sensitive surface of recording device 302. User interface 324 may be displayed on a device separate from recording device 302, such as a smartphone or tablet computer executing a client application.

[0066] Figure 4 This is a diagram illustrating an example architecture of an audio and video system 400 for network-based audio and video processing. In network-based audio and video processing, a communication network 402 links events to end-user devices, allowing the end-user of the device to access events at event 100 (…). Figure 1 The communication network 402 can be a PAN, LAN, cellular network, WAN (e.g., the Internet), or ad hoc network. The audio and video system 400 may include one or more subsystems, each of which is described below.

[0067] Studio-side system 404 is a subsystem of audio system 400, comprising equipment located and arranged at the location where events occur (e.g., in a studio, concert hall, theater, stadium, living room, or other venue). Studio-side system 404 may include references Figure 1 The described architecture includes multiple general-purpose devices (e.g., smartphones, tablets, laptops), each executing an audio or video processing application, recording and transmitting the recorded signals to server 408. Alternatively, studio-side system 404 may include reference... Figure 2 The described instance architecture involves a dedicated integrated recorder that records and sends the recorded signal to server 408.

[0068] Server 408 is a subsystem of audio system 400 and includes one or more computers or one or more discrete or integrated electronic circuits (e.g., one or more processors). Server 408 is configured to receive, process, and provide the live audio and video content of event 100 via communication network 402 to an end-user device via communication network 402. Server 408 may include one or more processors programmed to perform audio processing. In some embodiments, server 408 may control various aspects of studio-side system 404. For example, server 408 may increase or decrease the microphone volume level when clipping is detected, increase or decrease the sample bit rate or bit depth, or select the compression type based on detected bandwidth limitations.

[0069] In some implementations, server 408 performs automatic mixing and mastering of the audio signal. Server 408 can also automatically select specific portions from the video stream corresponding to the playing instrument. See below for reference. Figure 5 Describe additional details about the components and operation of server computer 408.

[0070] In some implementations, server 408 allows devices in editor-side system 420 to perform mixing, mastering, and scene selection. Editor-side system 420 is a subsystem of audio system 400, configured to allow third-party editors to edit audio or video content during live content streaming. Editor-side system 420 may include one or more mixer devices 422. Mixer devices 422 can be operated by end users, musicians in a band or orchestra performing a live event, or professional mixing engineers. Editor-side system 420 may include one or more video editing devices 424. Video editing devices 424 can be operated by end users, performers, or professional cameramen.

[0071] End users can listen to or watch the live content of event 100 on various end-user systems 410. The live or stored content can be played on user audio devices 412 (e.g., a stereo or multi-channel audio system with multiple speakers), user video devices 414 (e.g., one or more computer monitors), or a combination of both (e.g., a television, smartphone, desktop computer, laptop or tablet computer, or wearable device).

[0072] In some implementations, the audio system 400 allows end users to provide feedback on and control various aspects of the live content using their end-user devices. For example, the audio system 400 may allow real-time rating of live content based on voting, or audio or video panning performed by a specific authorized end user.

[0073] Figure 5 This is a block diagram illustrating an example signal path in audio and video processing. It can be viewed on server 408 (…). Figure 4 Components that implement the signal path on the server computer 408. These components may include a synchronizer 502, a source splitter 504, a mixing and mastering unit 506, a distribution front end 508, and an estimator 522. In some embodiments, some or all of these components may be implemented in software on the server computer 408. In other embodiments, some or all of these components may include one or more electronic circuits configured to perform various operations. Each electronic circuit may include one or more discrete components (e.g., resistors, transistors, or vacuum tubes) or integrated components (e.g., integrated circuits, microprocessors, or computers).

[0074] Synchronizer 502 can receive digital audio data for event 100 from one or more recording devices. The digital audio data may be (for example) sampled audio data. Each recording device, or each microphone coupled to a recording device, may correspond to an audio channel or track of a musical performance. The signal from the recording device is referred to as a channel signal. Therefore, synchronizer 502 can receive Nm channel signals, where Nm is the total number of microphones recording event 100, or more generally, the set of all sound signals captured in event 100. For example, the Nm channel signals may include one or more channels from direct output from a keyboard or from line audio output from a computing device or portable music player. The Nm channel signals may include a main channel signal from an ambient microphone and spot channel signals (also referred to as beams) from spot microphones. The Nm channel signals may be recorded by microphones on the recording device and sampled locally by an analog-to-digital converter at the recording device. The recording device may transmit the sampled audio data to synchronizer 502 via a network in an audio format in data packets. Therefore, the Nm channel signals may refer to digitized audio signals rather than analog signals directly from the microphone.

[0075] The Nm channel signals may not be time-synchronized. For example, data packets of digital signals may arrive at the server without regard to the physical time order in which the corresponding captured audio signals are generated. Synchronizer 502 may, for example, generate an output containing Nm synchronized channel signals based on the timestamps associated with the data packets. Synchronizer 502 may feed the Nm synchronized channel signals to source splitter 504. See below for reference. Figure 17 and Figure 18 Additional details describing the operation of synchronizer 502.

[0076] Source splitter 504 is a component of server 408 configured to separate each sound source from Nm synchronized signals. Each sound source may correspond to (for example) an instrument, singer, instrument group, or singer group. Source splitter 504 outputs Ns signals, each signal corresponding to a sound source. The number of sound sources (Ns) may be the same as or different from the number of synchronized signals Nm. In some implementations, source splitter 504 may be bypassed.

[0077] The outputs of Ns signals from source splitter 504 or the outputs of Nm synchronized signals from synchronizer 502 (assuming they bypass source splitter 504) can be fed into one or more mixing and mastering processing units 506. The mixing and mastering processing unit 506 can be a software and / or hardware component of server 408, configured to perform mixing operations on individual sound source channels based at least in part on reference audio data, and to perform mastering operations on the mixed audio signals to produce a final N-channel audio signal (e.g., stereo audio, surround sound). The mixing and mastering processing unit 506 can output the N-channel audio signal to distribution front end 508. In various embodiments, the mixing and mastering processing unit 506 can perform operations such as applying mixing gain, equalizing each signal, performing dynamic range correction (DRC) on each signal, and performing noise reduction on each signal. The mixing and mastering processing unit 506 can perform these operations individually for each signal or simultaneously for multiple signals in various combinations.

[0078] The reference audio data may include audio content recorded by a microphone during rehearsals and processed by estimator 522. During rehearsals, the microphone and sound sources are placed in the same acoustic arrangement as in the live event 100. The microphone then records audio signals as each sound source is played individually. Additionally, the microphone may record noise samples where the sound source is not being played.

[0079] Estimator 522 is a component configured to collect and process audio data from the rehearsal stage. Estimator 522 can instruct each performer at a sound source in their performance position to play their instrument or sing individually. For example, estimator 522 can instruct (e.g., through prompts via a device user interface) each performer to play their instrument at a low volume for X seconds and at a high volume for Y seconds. Nm signals from the microphone can be recorded during rehearsal. Estimator 522 can process the Nm signals, determine the loudness matrix, derive the sound source characteristics and location, and provide the instrument characteristics and location to the mixing and mastering unit 506 for mixing operations. Estimator 522 can receive additional input for configuring parameters to determine instrument characteristics and location. See below for further details. Figure 8 , Figure 9 , Figure 10 and Figure 13 Additional details describe the components and operation of estimator 522.

[0080] The distribution front end 508 may include an interface (e.g., a streaming or web server) for providing N-channel audio to a storage device or an end-user device for downloading, including live streaming (e.g., Hypertext Transfer Protocol (HTTP) live streaming, Real-Time Streaming Protocol (RTSP), Real-Time Transport Protocol (RTP), RTP Control Protocol (RTCP)). Live streaming can occur substantially in real time during event 100.

[0081] Server 408 may include video editor 530. Video editor 530 is a component of a server configured to receive the video signal from event 100 and to automatically edit the video signal, at least in part, based on the audio content. Automatic video editing may include, for example, amplifying a particular instrument or performer (e.g., a close-up) when video editor 530 determines that a particular instrument is the dominant sound source. See below for reference. Figure 19A and 19B and Figure 20 Additional details describing the operation of video editor 530.

[0082] Figure 6 This is a flowchart of an example audio processing procedure 600. (For example...) Figure 4 Server 408 executes process 600. Process 600 improves conventional audio processing techniques by automating various mixing and mastering operations, at least in part, based on reference audio data recorded during rehearsals. In this specification, the term "rehearsal" refers to a phase within that process.

[0083] Server 408 may receive (602) reference audio data from one or more channel signal sources. The reference audio data may contain acoustic information of one or more sound sources individually played during rehearsal. The reference audio data may contain, for example, acoustic information of background noise during rehearsal when the sound sources are not played. Each channel signal source may include a microphone or line output. Each sound source may be, for example, an instrument, a singer, or a synthesizer. Server 408 may be connected via a communication network (e.g., Figure 4 The communication network 402) receives reference audio data. The first channel signal may be captured by a first channel signal source (e.g., device 102) recording rehearsals at a first position (e.g., in front of the left side of the stage or at a specific instrument). The second channel signal may be captured by a second channel signal source (e.g., device 104) recording rehearsals at a second position (e.g., in front of the right side of the stage or at a specific instrument).

[0084] Server 408 may receive (604) one or more channel signals of a performance event (e.g., event 100) from one or more channel signal sources. Each channel signal may be a digital or analog signal from a corresponding channel signal source. Each channel signal may contain audio signals from one or more sound sources played at the performance event. During the performance event, the sound sources and channel signal sources are located in the same acoustic arrangement (e.g., at the same location). In some embodiments, server 408 may automatically synchronize the first channel signal with the second channel signal in the time domain. After synchronization, server 408 may determine the first sound source and the second sound source based on the first channel signal and the second channel signal.

[0085] Server 408 may automatically mix one or more channel signals during event 100 or after event 100 has ended (606). The automated mixing operation may include adjusting one or more properties of the acoustic effects of one or more sound sources of event 100 based on reference audio data. For example, the automated mixing operation may include performing noise reduction individually on each sound source, thereby balancing or leveling each sound source and translating each sound source.

[0086] The mixing operation may also include automatically adjusting the attributes of signals from one or more sound sources from event 100, at least in part, based on reference audio data. Automatically adjusting the attributes of one or more sound sources may include increasing or decreasing the gain of one or more sound sources according to the corresponding volume level of each sound source. Automatically adjusting the attributes of one or more sound sources may include increasing or decreasing the gain of each channel signal (each signal from the corresponding sound source) or both, causing each of the one or more sound sources to reach or approximately reach a target volume level. Server computer 408 may use estimator 522 to determine each corresponding volume level, at least in part, based on reference audio data. Other mixing operations may include, but are not limited to: applying compression, equalization, saturation or distortion, delay, reverb, modulation, stereo, filtering, and riding vocal or instrument volume.

[0087] The reference audio data may include audio signals recorded by the first and second recording devices during the rehearsal phase prior to event 100. Reference audio data may be recorded individually for each sound source or group of sound sources during the rehearsal phase. The reference audio data may include a first sound level signal (e.g., a first sound level signal designated as soft or low volume) and a second sound level signal (e.g., a second sound level signal designated as loud or high volume) for each sound source. Reference audio data may be recorded against background noise while the sound sources are playing. In some embodiments, the reference audio data may include a single sound level signal (e.g., when each sound source is playing at a medium volume).

[0088] Server 408 may determine the corresponding gain for each sound source in event 100, at least in part, based on reference audio data. Determining the corresponding gain may involve receiving input at a specified target volume level for each sound source or group of sound sources (e.g., a guitar group, a drum group, background vocals). Server computer 408 may use estimator 522 to determine the corresponding volume level of the signal in the reference audio data. Server 408 may determine each corresponding gain based on the difference between the volume level of the signal in the reference audio data and the target volume level.

[0089] In some implementations, the automated mixing (606) operation may include adjusting the gain of signals from one or more sound sources based on input from a remote human mixing or mastering engineer (who is logged into the server system via a communication network). Thus, a remote mixing or mastering engineer not present at event 100 can mix or master the sound sources of event 100 during live streaming.

[0090] Server 408 may, for example, provide downmixed audio from the server system (608) to a storage device via live streaming, or provide it as live content of event 100 to an end-user device. The end-user device may play the content on one or more speakers integrated into or coupled to the end-user device. In some embodiments, server 408 may automate video editing for event 100. Video editing may be live editing during event 100 or offline editing of previously recorded video of event 100. Figure 19A , 19B Additional details regarding automated video editing operations are described in section 20. In some implementations, a remote human video editor may use the platform to provide video editing during event 100.

[0091] In some implementations, server 408 may provide commands to the first and second recording devices based on a first channel signal or a second channel signal. The commands may adjust recording parameters of the recording devices. For example, the commands may instruct the recording devices to adjust gain, compression type, compression or sampling rate (e.g., 44.1 Hz) or bit depth (e.g., 16 or 24 bits).

[0092] Figure 7 This is a block diagram illustrating the components of an exemplary mixing and mastering unit 506. The mixing and mastering unit 506 may include various electronic circuits configured to perform mixing and mastering operations. The mixing and mastering unit 506 improves upon conventional mixing and mastering techniques by automating signal leveling and shifting during the mixing stage and by automating novel signal segmentation when a continuous and long volume increase is present.

[0093] The mixing and mastering unit 506 may include a mixing unit 702 and a mastering unit 704. The mixing unit 702 is a component of the mixing and mastering unit 704 configured to automatically perform mixing operations using reference audio data from one or more remote or local mixing consoles and inputs to Ns signals from the source splitter 504 or Nm synchronized signals from the synchronizer 502.

[0094] Among other components, the mixing unit 702 may also include a leveling unit 706, a shifter 708, a source equalizer 710, and a noise reduction unit 711. The leveling unit 706 is a component of the mixing unit 702 configured to adjust the gain for each sound source or each microphone. This adjustment may be based at least in part on reference audio data, input from a mixing console, or a combination of both. See below for reference. Figure 8 Additional details describing the operation of leveling unit 706.

[0095] The panner 708 is a component of the mixing unit 702 configured to spatially place each sound source at a location (e.g., left, right, center) on a virtual music stage. See below for reference. Figure 9 and Figure 10 Additional details describing the operation of the translation device 708.

[0096] The sound source equalizer 710 is a component of the mixing unit 702 configured to perform equalization (EQ) operations on individual sound sources rather than on the mixed audio signal as a whole. See below for reference. Figure 13 , Figure 14A and Figure 14B Additional details describing the operation of the sound source equalizer 710.

[0097] Noise reduction unit 711 is a component of mixing unit 702 configured to perform noise reduction (NR) operation on individual signals rather than across the spectrum of all signals. See below for reference. Figure 24 Additional details describing the operation of the noise reduction unit 711.

[0098] Among other components, the mastering unit 704 may also include an equalizer 712 and a splitting unit 714. The equalizer 712 is a module of the mastering unit 704 configured to smooth the sound levels across different frequencies of the mixed audio signal as a whole. The splitting unit 714 is a module of the mastering unit 704 configured to divide the video signal into multiple segments based on the inherent characteristics of the audio signal. In some embodiments, the splitting unit 714 is... Figure 5 The components of video editor 530, or those coupled to video editor 530. See below for reference. Figure 15 and 16 Additional details describing the operation of segmentation unit 714.

[0099] Figure 8 This is a flowchart illustrating an example process 800 of automatically leveling a sound source. This can be achieved by the leveling unit 706 ( Figure 7 Process 800 is executed. In automatic leveling, leveling unit 706 automatically adjusts the corresponding volume level of the sound source to the target level. Process 800 improves conventional mixing techniques by automatically performing gain adjustments based at least in part on reference audio data rather than on manual human adjustments. This allows for rapid, real-time processing of large amounts of music content.

[0100] The leveling unit 706 may receive (802) reference audio data (also referred to as rehearsal data). The reference audio data may include representations of channel signals from multiple sound sources (e.g., a main microphone and spot microphones). The representations may be channel signals directly from the channel signal sources, or partially processed signals (e.g., equalized or dynamically calibrated).

[0101] The leveling unit 706 can determine (804) the corresponding correlation between each pair of channel signal sources (e.g., microphones). The details of determining said correlation are described below with reference to equation (3).

[0102] The leveling unit 706 can assign (806) the corresponding energy level of each primary microphone to a baseline associated with unity gain or some other reference level (e.g., -18dB).

[0103] In some implementations, the leveling unit 706 can determine (808) the corresponding contribution of each microphone point to the baseline.

[0104] The leveling unit 706 can receive (810) target level data, which specifies the target level for each sound source. The target level data can be received from a user interface.

[0105] The leveling unit 706 can determine (812) a cost function for rescaling the audio signal to the target level according to the corresponding gain, based on the corresponding contribution. The cost function can be a function of variables (in this case, the gain) that are solved to minimize the function. Solving for the variables of the cost function is called minimizing the cost function. Details and examples of solving for the variables of the cost function are described below in a section titled “Minimizing the Cost Function by Best Guessing”.

[0106] The leveling unit 706 calculates the corresponding gain for each of the (814) channel signals by minimizing the cost function. The leveling unit 706 can apply the corresponding gain to the channel signals in the live audio data to achieve the target level for each sound source. The leveling unit 706 can provide the resulting signal to other components for further processing and playback on speakers or headphones in the end-user device. Additional details and examples of process 800 are described below.

[0107] Index set i = 1, ..., N i This can represent the number of sound sources, where N i For event 100 ( Figure 1 The total number of sound sources at position (). Index set b = 1,...,N b This can represent the number of beams, where each beam, as described earlier, is a channel signal from the corresponding point microphone. N b This represents the total number of point microphones. The index set M = L, R, 1, ..., N b This can represent the combination of the main left microphone (L) and the main right microphone (R) with a beam index. In the case where the main microphone is a mono microphone, the index set can be M = Mono,1,...,N. b The term Mono represents a single-channel microphone. Subsequent processing is similar. Multiple sound sources can be assigned to the same beam. Therefore, in some scenarios, N b <N i This is (for example) a case where a spot microphone is placed close to a guitarist who is also singing. In this example, both vocals and guitar are assigned to the same spot microphone. Therefore, the leveling unit 706 can specify the total number of signals that will exist in the final mix as N. M .

[0108] One of the inputs to the algorithm executed by the leveling unit 706 is a loudness matrix L that quantizes the loudness level (e.g., in dB) of each instrument i in each beam M. iM The estimator 522 can calculate the loudness matrix L. iM The leveling unit 706 can be used for loudness matrix L iM Using a linear scaling, the leveling unit 706 can achieve the following energy matrix E iM This represents the energy of each instrument in each microphone:

[0109]

[0110] If the leveling unit 706 will increase the gain g M When applied to beam b, the energy of the beam can be changed to In addition, the energy matrix can specify how much energy from each sound source exists in the two main stereo channels (by E). iL E iR specified).

[0111] The leveling unit 706 is configured to determine the gain g. M This is a vector representing the corresponding gain of each channel (including the main channel and the point channels), where the gain of the two main channels is represented first. The leveling unit 706 can fix the absolute scale so that all energy is referred to as energy in the main stereo channels. The leveling unit 706 can determine the energy to which no gain is applied, and thus no gain is applied. In this method, the main stereo channels can be designated as a baseline with unity gain. The leveling unit 706 can calculate the contribution of each point microphone on this baseline. Therefore, the leveling unit 706 can adjust the gain of g... M The first two entries are set as units:

[0112]

[0113] To estimate the energy after mixing different signals, the leveling unit 706 can first obtain the normalized correlation matrix (Ci) between beam M and beam M' based on the reference audio data of each sound source i. i ) M,M′ Each C can be obtained solely from the rehearsal of sound source i. i The leveling unit 706 can use s iM Let M represent the signal (mainly stereo plus beam) captured by M microphones during instrument i rehearsal. The leveling unit 706 can calculate the normalized covariance matrix as follows:

[0114]

[0115] Where "<>" represents the time average over a time period. The time period can be the entire rehearsal time. Alternatively, the leveling unit 706 can obtain the average within the gated portion of the rehearsal, allowing it to remove nearly silent portions (if present). Therefore, the leveling unit 706 can maintain portions with a good signal-to-noise ratio (SNR). The normalized covariance satisfies the following:

[0116] ·(C i ) M,M =1, which means that the microphone is always completely related to itself.

[0117] ·(C i ) M,M′ =0, if the signals from the two microphones are completely uncorrelated.

[0118] ·(C i ) M,M′=1, if the signals from the two microphones are perfectly correlated in positive phase: s iM =αs iM′ , where α is a value and α>0.

[0119] ·(C i ) M,M′ =-1, if the signals from the two microphones are perfectly correlated in out-of-phase mode: s iM =αs iM′ , where α is a value and α < 0.

[0120] Using this covariance matrix, the leveling unit 706 can express the total energy E of sound source i as follows: i :

[0121]

[0122] The equivalent, shorter form (however, summing the repeated terms) is:

[0123]

[0124] Another input to the leveling unit 706 can be the target loudness level (or target energy T) of each sound source i in the final mix. i (In a linear scale). In principle, it is only related to the target level. If the leveling unit 706 has fixed the global volume by fixing the gain of the main stereo channels to unit, then this gives physical meaning to the absolute target volume. The leveling unit 706 can determine one or more criteria to set it to the appropriate level.

[0125] To achieve this, the leveling unit 706 can acquire specific data items to reach the desired relative target loudness level T. i The method for defining the absolute scale by the leveling unit 706 among all possible approaches is determined such that the leveling unit 706 can control the fraction of the total energy ultimately derived from the main stereo microphone to the point microphone. In some implementations, the leveling unit 706 may set this fraction as a user input parameter.

[0126] In some implementations, leveling unit 706 can infer this ratio by targeting a given ratio between direct energy and reverberant energy (specified as the direct-to-reverberant ratio). For example, in an audio environment with strong reverberation (e.g., a church), leveling unit 706 can apply a high level of relative point microphone energy. By contrast, in an audio environment with low reverberation where the main stereo microphone is in its optimal position (e.g., an acoustically treated room), leveling unit 706 can allow most of the energy to come from the main stereo microphone. Therefore, leveling unit 706 can obtain the specified point-to-main energy ratio R from the user or through automatic calculation. spotsThe input. The leveling unit 706 can then use equation (6) below to determine the terms in the cost function:

[0127]

[0128]

[0129]

[0130]

[0131] Here, E main For the energy from the main microphone, E spot The energy comes from the point microphone, and E total This represents the total energy.

[0132] The leveling unit 706 can reduce this equation to simplify the process. Assuming the final sound source energy is correctly reached, such that E in the following reduction... i ≈T i :

[0133]

[0134]

[0135] In this approximation, these energies do not depend on g. M Therefore, the leveling unit 706 can impose almost no point and all constraints before minimization. The leveling unit 706 can rescale the target energy. And solve for the scaling factor r:

[0136]

[0137] in T is scaled by the scaling factor r. i .

[0138] The leveling unit 706 can then be appropriately rescaled. To determine the cost function. Even if the leveling unit 706 is set to R spots =0, in the case described, E total ≈∑ i T i =E main It may also require some point microphones to achieve the target level T of a certain sound source i. i This violates the setting R. spots The intuition is that =0, and it is attributed to the fact that after minimization, the leveling unit 706 can become close to E. i ≈T iThe approximation.

[0139] The leveling unit 706 can determine the cost function as (using shorthand: dB) p [·] = 20log 10 [·] and dB I [·] = 10log 10 [·]):

[0140]

[0141] Where F is N b One unknown number g M The function, all dependencies are via E i This arises from implicit dependency. In equation (9) and other equations below, terms in dB are expressed as squares, for example, (dB). 2 In various implementations, these terms can be replaced with absolute values ​​(e.g., |dB|).

[0142] The normalization factor ensures that the absolute value of the first term can be compared across cases with varying numbers of sound sources. The leveling unit 706 can use a cost function F to represent the average squared error (e.g., in dB) of each sound source failing to reach the target. In some implementations, the leveling unit 706 can obtain the cost function and avoid the approximations described above. The leveling unit 706 can facilitate additional cost items:

[0143] F[r,g M ]=F0[r,g M ]+F1[g M ],

[0144]

[0145] (10)

[0147] In equation (6), based on g b Define E spots E main This allows the leveling unit 706 to control the amount of R given by increasing α. spots The importance of constraints. Note that in this implementation, the leveling unit 706 also needs to find the minimum value of r, which can be the global scale of the target.

[0148] In various implementations, the leveling unit 706 may use an algorithm that provides a better result for achieving a specified loudness target, given information about which sound sources are more important. For example, the input information might specify that the lead vocals should be 3 dB higher than other instruments. This information is crucial in determining the quality of the mix. Other instruments might be a few dB off from their correct target, but the mix will not be judged as bad as the lead vocals being below target. The leveling unit 706 can determine an importance weight for each sound source. This aspect is captured. The leveling unit 706 can be defined as follows, incorporating a cost function with importance weights:

[0149]

[0150] The leveling unit 706 can minimize the cost function F as described above for g M Solve this problem. In some implementations, the leveling unit 706 can assign importance weights based on whether the instrument is the primary instrument. For example, the leveling unit 706 can assign importance weights to non-lead instruments. Set the importance weight to 1, and assign importance weights to the main instrument. Set to a value between 2 and 5.

[0151] Add a dedicated dot microphone

[0152] In some scenarios, the algorithm tends to use almost no energy from a specific channel signal source (e.g., a dedicated spot microphone) because other microphones can be used to correctly achieve the level of the corresponding sound source. This can occur in situations with leakage (such as when the spot microphone is omnidirectional (e.g., the internal microphone of a smartphone)). Generally, when using a dedicated spot microphone, the leveling unit 706 can be configured to obtain most of the energy from the corresponding instrument through these microphones.

[0153] During the rehearsal phase, the leveling unit 706 can define the level of specialization of a given point microphone for a given sound source. If the point microphone has almost no leakage from other sound sources, then the leveling unit 706 can set the specialization to 1. If the leakage from other sound sources is significant (e.g., exceeding a threshold), then the leveling unit 706 can set the specialization to 0. Therefore, for sound source i with beam b(i), this specialization D(i) should be:

[0154]

[0155]

[0156] Where SNR(i) is the signal-to-noise ratio of sound source i, dBMaxRatio is a first threshold. If the value is higher than the first threshold, the leveling unit 706 sets the level to 1, and dBMinRatio is a second threshold. If the value is lower than the second threshold, the leveling unit 706 sets the level to 0. The thresholds can be predefined or can be derived from user input.

[0157] The leveling unit 706 can clamp to D(i)∈[0,1]. In some embodiments, the leveling unit 706 can set the values ​​of these parameters as follows: dBMaxRatio = 3dB, dBMinRatio = -6dB. These settings imply that if the relevant instrument is at least 3dB higher than the sum of all other instruments in the microphone, then the level is 1, and if it is -6dB or less, then the level is 0.

[0158] The leveling unit 706 can use D(i) to adjust the new term N in the cost function. ded Weighted:

[0159]

[0160] Where N ded Let E be a measure (real number) of how many dedicated microphones exist, and E i,spots Total energy in the mix originating from the point microphone and the instrument i:

[0161]

[0162]

[0163] The leveling unit 706 can calculate g by minimizing the cost function that includes this new term. M .

[0164] Minimize the cost function through the best guess.

[0165] In some implementations, these cost functions may be nonlinear. To minimize the nonlinear cost function, the leveling unit 706 may employ a guessing method. The leveling unit 706 may discretize all g in 1dB steps over a range of, for example, ±5dB. M And find the combination that minimizes F. The leveling unit 706 can start from the best guess and find the combination by starting at the best guess and moving several steps away from the best guess across the range until the leveling unit 706 finds the minimum value of the cost function.

[0166] To achieve this, leveling unit 706 can perform a first guess. This can be (for example) obtained by ignoring leakage, such that E is diagonal (assuming estimator 522 or leveling unit 706 categorizes rows and columns such that the corresponding beams of the sound sources are diagonal). In this case, only one beam contributes to each i. This beam is labeled b(i). Therefore:

[0167]

[0168] Where E i,main The energy of instrument i in the main microphone. Assuming the leveling unit 706 achieves its target, The leveling unit 706 can solve for g:

[0169]

[0170] In the case where the leveling unit 706 repeats the same beam for more than one instrument, the leveling unit 706 can solve for g as follows:

[0171]

[0172] Where <> represents the average value, which is the average value for various instruments assigned to the same beam b. The leveling unit 706 can specify the solution as the best guess and increase or decrease the solution by X dB steps to find the best solution.

[0173] Notes on the signs of the molecules. The only fact that is well-assured is that the total target energy from all beams is greater than or equal to the energy from the main microphone.

[0174]

[0175]

[0176] However, some individual terms in the equation can be negative for a given i. This means that for a given sound source i, there is already sufficient loudness in the main stereo channels to achieve the target. In these cases, the leveling unit 706 can set the gain of the corresponding beam to zero. In some implementations, the leveling unit 706 can search for a range of possibilities, for example, -15 dB.

[0177] The leveling unit 706 can target T i Using the same loudness model. Instead of expressing loudness in dB, the leveling unit 706 can express loudness in dB and uses the loudness model that the leveling unit 706 has been using to convert back to dB.

[0178] Automatic translation device

[0179] Figure 9This is a flowchart illustrating an example of automatic translation process 900. It can be derived from... Figure 7 The translation device 708 executes process 900. By executing process 900, the translation device 708 improves upon conventional translation techniques by automatically placing the instrument in its correct position on the music stage.

[0180] The shifter 708 may receive (902) channel signals of event 100. The channel signals may be the output of the leveling unit 706. Each channel signal may correspond to a microphone. The shifter 708 may receive (904) reference audio data of the sound sources in event 100. The reference audio data may be generated based on signals recorded during rehearsals. The shifter 708 may calculate (906) based on the reference audio data, such as the total energy in the left channel and the total energy in the right channel contributed by each sound source. The shifter 708 may calculate (908) left-right imbalance based on the total energy. The shifter 708 may determine a cost function to minimize the imbalance. The shifter 708 may calculate (910) the natural translation of sound sources captured by the main microphone. The shifter 708 may determine a cost function to maximize the natural translation. The shifter 708 may, for example, determine (912) non-translatable sound sources based on an input that designates the sound source as non-translatable. The 708 translation unit can determine the cost function that takes into account non-translatable sound sources.

[0181] The translater 708 can determine (914) a cost function having a translation angle as a variable for each channel signal. The cost function may have a first component corresponding to the imbalance, a second component corresponding to the translationable sound source, and a third component corresponding to the non-translatable sound source.

[0182] The shifter 708 determines (916) the shift position of each channel signal by minimizing the cost function. The shift position can be parameterized as a shift angle, the ratio between the left and right output channels, or the percentage of the left output signal to the right output channel. The shifter 708 can apply the shift position to the channel signals to position the sound source between the left and right sides of the stereo stage for audio effects output to the speakers.

[0183] In some implementations, the panner 708 may perform audio panning based on video data. The panner 708 may use facial tracking or instrument tracking of the video data to determine the position of a specific sound source (e.g., a singer or instrument). The panner 708 may then determine the panning position of the sound source based on the position.

[0184] Once the leveling unit 706 has calculated the required gain (g) for each beam... bThe translater 708 can then determine how to split each beam into left and right L / R segments in an energy-preserving manner. The translater 708 can calculate the translation angle θ for each beam b. b :

[0185] l b =cosθ b g b ,r b =sinθ b g b ,θ b =0,...,π / 2 (19)

[0187] Among them l b Let r be the left channel component of beam b, and r b Let θ be the right channel component of beam b. The extreme left is θ = 0, the extreme right is θ = π / 2, and the center is θ = π / 4.

[0188] Translation 708 extends the index to M, assuming that translation 708 keeps the primary stereo channel unchanged, where l L =r L =1. The resulting mix, varying with angle, is:

[0189]

[0190] Where s M Let L be the signal picked up by microphone M, L be the left component of the mix, and R be the right component of the mix.

[0191] Based on reference audio data, the shifter 708 can calculate the total energy in the L / R channels produced by each instrument:

[0192]

[0193]

[0194] in For translation angle θ b The total energy of sound source i in the left channel, and For translation angle θ b The total energy of sound source i in the right channel. These expressions now depend only on θ. b Because of the gain g b It has been tuned by leveling unit 706.

[0195] One thing that the shifter 708 can impose is to balance the overall mix between L and R. Therefore, the shifter 708 can cause the L / R imbalance cost function H. LR-balance Minimize:

[0196]

[0197] On the other hand, the translation unit 708 can be configured to take into account the natural translation of sound sources arranged in event 100 from the perspective of event 100. Natural translation is fully captured by the primary stereo energy: E iL E iR Therefore, the following can also be imposed on the translation unit 708:

[0198]

[0199] In some implementations, the panner 708 may receive the desired position as external input, rather than determining the position after analyzing natural translations obtained by the left and right channels. For example, the panner 708 may determine the natural position based on an image or video. Alternatively, the panner 708 may determine the natural position based on user input.

[0200] Furthermore, certain sound sources (e.g., lead vocals, bass, etc.) should never be translated. The translator 708 can be configured to account for this as much as possible. These sound sources can be designated as non-translatable sound sources. The translator 708 can be configured separately by I... P / I U This represents the set of sound sources that can be translated / cannot be translated. The translater 708 can then generalize the above to...

[0201]

[0202]

[0203] The translater 708 can then determine the cost function H of the non-translatable source. unpanneble and the cost function H of the sound source that can be translated panneble .

[0204]

[0205]

[0206] Translation 708 controls the amount of translation that indicates a tendency to translate the instrument wider (opposite to placing it in the center of the stage). In some embodiments, translation 708 may introduce another. In some embodiments, translation 708 may amplify estimates from the main microphone. Translation 708 may receive parameters d∈[0,1] (which indicate divergence) as presets or user input. Translation 708 may perform the following transformations on the perceived main channel energy, inducing changes in the instrument's angle:

[0207] E iL E iR →(E iL EiR ) 1+4d ,

[0208] θ0→θ final =tan -1 (tanθ0) 1+4d , (26)

[0210] Where θ0 is the original translation angle, θ final This is the final translation angle. For d=0, there is no change, θ final =θ0. For the extreme case d=1, the following text... Figure 10 The warping is shown in the middle.

[0211] For d, the translater 708 can use the following translateable cost function:

[0212]

[0213] The 708 translation module can use the final cost function:

[0214] H[θ b ] = H LR-balance +α u H unpannable +α p H pannable . (28)

[0216] Where the weight α u and α p The importance of controlling each aspect of the cost function to be applied to both non-translatable and translatable sound sources. The translater 708 can be discretized via θ. i The possible values ​​(for example, every 10 degrees from -50 to 50) are discretized and iterated until the minimum value of the cost function is found, such that the cost function H[θ] is minimized. b ] minimize.

[0217] Figure 10 This diagram illustrates an example of an angle transformation for maximum warp. The angle transformation for maximum warp can be performed by the translater 708 (d = 1). The horizontal axis represents the original angle θ0 of one or more sound sources. The vertical axis represents the final angle θ of one or more sound sources. final The center is translated at an angle of 45 degrees.

[0218] Joint Minimization

[0219] The leveling unit 706 can use fewer spot microphones than it can handle. For example, various configurations of the input gain can compete, where all input gains produce the same loudness. This can have a negative impact on the panner 708, as it significantly reduces the range of possibilities for the panner 708 when only one or two spot microphones are used.

[0220] In some implementations, to reduce this uncertainty in the automatic leveling phase and to support configurations using more point microphones, the automatic leveling phase operation of the leveling unit 706 may be linked to the translation phase operation of the panner 708.

[0221] In these embodiments, the leveling and translation unit can combine the circuitry and functionality of leveling unit 706 and translater 708. The leveling and translation unit can receive reference audio data. The reference audio data may include representations of channel signals from multiple channel signal sources recorded during rehearsals of one or more sound sources. The leveling and translation unit can receive target level data, which specifies a target level for each sound source. The leveling and translation unit can receive live audio data. The live audio data may include recorded or real-time signals from one or more sound sources playing at live event 100. The leveling and translation unit can determine a joint cost function for leveling and translating the live audio data based on the reference audio data. The joint cost function may have a first component for leveling the live audio data and a second component for translating the live audio data. The first component may be based on the target level data. The second component may be based on a first representation of imbalance between the left and right channels, a second representation of translateable sources among the sound sources, and a third representation of non-translatable sources among the sound sources. The leveling and shifting unit calculates the corresponding gain to be applied to each channel signal and the corresponding shift position of each channel signal by minimizing the joint cost function. The leveling and shifting unit can apply the gain and shift position to the signal of the live audio data of the event to level the sound sources in the live audio data and place the sound sources in the live audio data between the left and right sides of the stereo stage for audio effects output to a storage device or to a stereo reproduction system.

[0222] The joint cost function is shown below in equation (29), in which some of the terms mentioned above are renamed:

[0223] H automixer [g b ,θ b ] = H level [g b ]+α panner H panner [g b ,θ b],

[0224] H level =H targets +α d H dedicated ,

[0225] H panner =H LR-balance +α u H unpannable +α p H pannable , (29)

[0227] Where H automixer For the cost function of the combined leveling unit 706 and the translater 708, H level For the cost function of leveling unit 706, H panner For the cost function of the translater 708, H targets Let H be the cost function when considering the objective of important sound sources, and H dedicated This is the cost function considering dedicated beams. Additionally, α... d For the weights of the dedicated microphone, α u Let be the weights of the non-translatable sound source, and α p These are the weights for the translateable sound source. The translater 708 can receive these weights as preset parameters or user input.

[0228] The cost function in the joint cost function is defined below.

[0229]

[0230]

[0231]

[0232]

[0233]

[0234] in

[0235]

[0236]

[0237]

[0238]

[0239] l b =cosθ b g b,r b =sinθ b g b ,θ b =0,...,π / 2. (31)

[0241] Here, the automatic leveling processing of the leveling unit 706 is independent of the translation angle. It measures the overall loudness of the mono downmix. Besides the translation angle, the automatic translation processing of the shifter 708 also depends on the beam gain g. b Inferring the RMS of the instrument based on microphone signals.

[0242] Figure 11 This is a flowchart illustrating an example of inferring energy levels based on microphone signals 1100. Figure 5 and 7 The estimator 522 can execute a process 1100 to measure the RMS of an instrument. The RMS of an instrument can be represented as the root mean square of the energy levels of various sound sources.

[0243] Estimator 522 can receive (1102) reference audio data. The reference audio data may include audio recorded during rehearsals with respect to i = 1,...,N. i The sound source contains channel signals from m = 1, ..., M microphones.

[0244] The estimator 522 can calculate (1104) the corresponding level (e.g., loudness level, energy level, or both) of each instrument in each microphone based on reference audio data.

[0245] Estimator 522 can determine (1108) a cost function for the corresponding gain of each sound source. In the cost function, estimator 522 can assign less weight to the signal from the main microphone than to the point microphone. In the cost function, estimator 522 can penalize the loudness of instruments in the estimated live data that are significantly louder than those represented in the reference audio data. In the cost function, estimator 522 can scale the cost function to the average (across microphones) difference between the levels measured between the performance and rehearsal.

[0246] Estimator 522 determines (1110) the appropriate gain for each sound source by minimizing the cost function. Estimator 522 can provide the corresponding gain in the energy matrix or loudness matrix to a processor (e.g., video editor 530) for processing video signals, for example, for identifying which instrument is being played at a higher level than the thresholds for other instruments, and for focusing on said instrument or the player of said instrument. Additional details and examples of process 1100 are described below.

[0247] The audio scene of event 100 may contain m = 1, ..., M microphones and i = 1, ..., N microphones. i Each sound source. During rehearsals, each instrument is played individually. The estimator 522 is configured to calculate the E signal for each microphone. i,m Server 408 (which calculates the loudness of each instrument and converts those values ​​into energy) Figure 4 The components of ). In some implementations, loudness measurement may be based on (for example) the European Broadcasting Union (EBU) R128 standard, and via 10 L / 10 This is converted into energy. Therefore, during rehearsals, the estimator 522 can calculate matrix e based on the following relationship: i,m :

[0248]

[0249] in For the loudness measured in each microphone during instrument i rehearsal, and This refers to the loudness of each instrument during the rehearsal phase (unknown and immeasurable).

[0250] When all bands perform together, the estimator 522 can only measure with weight. The total loudness in each microphone. If the transfer functions from the instruments and microphones remain constant and equal to those during rehearsal, and the signals from all instruments are statistically independent, then the following relationship holds:

[0251]

[0252] The estimator 522 can use the gain g i To compare the grade of each instrument with its grade during rehearsals:

[0253]

[0254] In some implementations, the estimator 522 may use a cost function C (which may be a gain g). i The function is used to ensure that the estimator 522 estimates the performance level such that the model is optimally satisfied in the least squares sense:

[0255]

[0256] Where C1(g) i ) is the first component of the cost function C.

[0257] In some implementations, the estimator 522 can improve the results by giving less importance to the primary stereo microphone, since the primary stereo microphone may be less discriminative than the point microphone. The estimator 522 can apply a set of weights w separately for each microphone. m , making

[0258] For the main signal, w m =w main <1

[0259] Otherwise, w m =1 (36)

[0261] And therefore:

[0262]

[0263] The problem of estimating energy levels can be uncertain, where there are fewer microphones than instruments. This uncertainty can correspond to achieving the same overall loudness in each microphone by simultaneously boosting the estimates of some instruments while dimming the estimates of others. To reduce this uncertainty, in some implementations, the estimator 522 may introduce a term that penalizes instruments whose estimated loudness is significantly higher than the loudness measured during rehearsal. One possible term is defined below.

[0264]

[0265] Where C2(g) i ) is the second component of the cost function C, and α2 and n are the parameters of the penalty.

[0266] If (for example) α² = 0.1 and n = 6, then there is essentially no penalty when the gain is lower than that of rehearsal. However, the penalty becomes 6.4 when the inferred level is higher than the rehearsal level by 6 dB. When this C2 is added, its scale can differ from C1. The estimator 522 can introduce the scale △ as follows:

[0267]

[0268]

[0269] Where △ represents the average (across microphones) squared difference between the grades measured between the performance and rehearsal.

[0270] Therefore, the estimator 522 can apply the following cost function:

[0271]

[0272]

[0273] In some implementations, the estimator 522 can be measured in dB. Measuring everything in dB provides better performance when estimating low levels.

[0274]

[0275]

[0276] Where dB[·] = 10log 10 [·]. The instance value of the parameter is: α² = 0.001, n = 4, w main =0.2. In some implementations, the estimator 522 may ignore all of the above 1 / M factors when minimizing the cost function.

[0277] In some implementations, estimator 522 may apply an initial filtering stage before minimizing the cost function. In this initial stage, estimator 522 may use the specialization D(i) of sound source i to determine the specificity of a given channel signal relative to sound sources that have leakage from other instruments. For example, estimator 522 may determine whether the specialization D(i) of sound source i is above a given threshold. For each such sound source, estimator 522 may obtain a corresponding gain by limiting the cost function to include only the corresponding specialized channel signal. For example, if estimator 522 determines that an instrument... and dedicated microphone A pair If the threshold is met, then the estimator 522 can determine the gain by minimizing the reduced cost function. In the case where the estimator 522 selects the cost function in equation (40), the reduced cost function will be equation (40.1) below.

[0278]

[0279] Equation (40.1) allows estimator 522 to perform minimization using equation (40.2) below.

[0280]

[0281] The estimator 522 can use other cost functions described above by applying a simplification similar to that described in reference equations (40.1) and (40.2) to include only a pair of signal sources (sometimes dedicated microphones) and determine the gain of each of the pair.

[0282] After determining these gains in the initial filtering stage, estimator 522 immediately reduces the problem of minimizing the cost function to determine only the gains of instruments that do not have dedicated channel signals. Estimator 522 can fix these gains to the gains present in the initial filtering stage. Estimator 522 can then minimize the cost function with respect to the remaining gains.

[0283] Inferring the RMS of musical instruments in the frequency band

[0284] Using the estimator 522 described above, the RMS of the inferred signal can be extended in a frequency-dependent manner to improve the estimation in cases where different instruments contribute to different parts of the overall spectrum. Figure 12 This is a flowchart illustrating an example of the process 1200 for inferring energy levels in a frequency band. The estimator 522 can execute the operation of process 1200.

[0285] The estimator 522 can receive (1202) reference audio data. The reference audio data may include audio signals recorded from rehearsals, in which the sound source and microphone are placed in the same arrangement as during the live performance.

[0286] In the first stage, estimator 522 can calculate the corresponding rehearsal loudness of each sound source in the (1204) frequency band. The frequency band mentioned therein can be an octave band centered at the frequency f = {32, 65, 125, 250, 500, 1000, 2000, 4000, 8000}, obtained by using a standard filter conforming to ANSI (American National Standards Institute) specifications.

[0287] In the next stage, estimator 522 can use the following cost function to calculate (1206) the total cost as the sum of the costs of each sound source:

[0288]

[0289] Where C1(g) i ) is the first component of the cost function across the microphone and frequency band.

[0290] Estimator 522 can calculate the quality term in the (1208) frequency band:

[0291]

[0292]

[0293] Where C2(g) i ) is the second component of the cost function across the microphone and frequency band.

[0294] Estimator 522 determines the corresponding gain in the (1210) frequency band by minimizing cost. Estimator 522 may provide the gain to the processor for processing the live data of event 100. For example, estimator 522 may provide the gain to video editor 530 to identify sound sources playing at a higher level than other sound sources so that video editor 530 can focus on or amplify said sound source.

[0295] In some implementations, estimator 522 may be biased towards estimating whether the instrument is on or off. This can be done by modifying the second term in equation (43) to have a minimum value at g = 0, 1. For example, estimator 522 may apply a function f to g. i :

[0296]

[0297] Here, p is the control value, and a is the desired point where the item has the minimum value. Here, a = 1.

[0298] exist Below, this function is symmetric because it has a minimum value only at x = 0 and x = ±a, and it only has a minimum value at x = 0 and x = ±a. There is a minimum value at a point where f = p. Therefore, the estimator 522 can use the value of p to control the value at the maximum value (and thus, the size of the wall between the minimum and the maximum values ​​at x = 0, a). More generally,

[0299]

[0300] It has the same properties, except that the maximum value is now at x=3. -2n a. The second term in the cost function becomes:

[0301]

[0302] The parameters in equation (45) are instantiated as a = 1, n = 1, p = 5.

[0303] In some implementations, the estimator 522 may implement the following functions.

[0304]

[0305] Where m and n are control values. The parameters in equation (45) are set to n = 1.2, m = 1e-5, a = 1.

[0306] In some implementations, the estimator 522 may implement a sixth-order polynomial symmetric under x-parity, with coincidence points (0,0), (x... p ,y p ),(x a ,0),(xl ,y l ):

[0307]

[0308] Automated EQ in the loudness domain

[0309] Figure 13 This is a flowchart illustrating an example of an automatic equalization process for individual sound sources 1300. (For example...) Figure 7 The sound source equalizer 710 performs process 1300. The sound source equalizer 710 is configured to apply equalization (EQ) at the level of individual sound sources (as opposed to the overall stereo mix) for the specific purpose of automatically removing, emphasizing, or attenuating instruments.

[0310] The sound source equalizer 710 can receive (1302) audio data. The audio data may be live audio data of event 100 or rehearsal data. The audio data may include channel signals from the sound source.

[0311] The sound source equalizer 710 can map (1304) the corresponding signal of each sound source to the excitation in each frequency band. The sound source equalizer 710 can sum the sounds from different sources in the excitation space and sum the effects from different frequency bands in the loudness space.

[0312] The source equalizer 710 then automatically equalizes one or more sound sources. The source equalizer 710 can generate a list of source-band pairs that map each sound source to each frequency band. The source equalizer 710 can determine a corresponding demand value for each source-band pair in the list (1306). The demand value can indicate the relative importance of the sound source represented in the pair relative to other sound sources and other frequency bands in the frequency band of the pair after equalization. The demand value can be the product of the relative importance value and the masking level of other sound source pairs, or it can be any mathematically representable relationship where the demand increases or decreases when the relative importance or masking level is increased or decreased, respectively.

[0313] The sound source equalizer 710 compares the demand values ​​with a threshold. If all demand values ​​are below the threshold, the sound source equalizer 710 can terminate process 1300.

[0314] Once the demand value for a source-band pair exceeds a threshold, the source equalizer 710 can immediately perform equalization on the sound sources represented in the pair to highlight the source-band pair (1308). The equalization may include (for example) reducing other sound sources in the band.

[0315] The sound source equalizer 710 can remove (1310) prominent source-frequency bands from the possible list and return to stage 1306.

[0316] The sound source equalizer 710 can apply process 1300 only to the reference audio data. The sound source equalizer 710 can then use the fixed settings of event 100. Alternatively or additionally, the sound source equalizer 710 can optionally adaptively run these operations and features during live event 100 after using the reference audio data as a seed.

[0317] The sound source equalizer 710 can use indices i∈{1,...,N} i The sound source is represented by a} (e.g., i=1 for bass, etc.). In some scenarios, this allows for good separation of all instruments in the mix. The sound source equalizer 710 can separate the sound from each instrument signal s. i Excitation mapped to each frequency band b:

[0318] s i →E(i,b), (49)

[0320] Where E(i,b) is the excitation of sound source i in frequency band b. Frequency band b can be an ERB (Equivalent Rectangular Bandwidth) band.

[0321] This mapping can be represented in the Glasberg-Moore loudness model. By performing this mapping, the sound source equalizer 710 considers the effects on a head incident from the front (or, alternatively, in the diffuse field), and the inverse filter of the silence threshold. Similarly, the sound source equalizer 710 can be mapped from the excitation space to a specific loudness L[E(i,b)]:

[0322] s i →E(i,b)→L[E(i,b)], (50)

[0324] It will be modeled by the compression applied by the substrate. Instances of this function above 1 kHz are L = α(E+1). 0.2 The term "specific loudness" can refer to the loudness per frequency band, and "loudness" can refer to the sum across all frequency bands. The dependence on b, as shown in the equation, indicates the specific loudness. The independence from b indicates the sum.

[0325] The sound source equalizer 710 can sum sounds from different sources in the excitation space and sum effects from different frequency bands in the loudness space:

[0326]

[0327] Where E sources (b) is the excitation for all sound sources in frequency band b, and L all-bands (i) represents the loudness of sound source i across all frequency bands.

[0328] The significance quantity is the partial loudness of the signal in the presence of noise (or in the presence of several other signals). The sound source equalizer 710 can split all sound sources into one sound source designated as signal pL with index i and all other sound sources with index i′:

[0329]

[0330] Its maximum value is precisely the loudness of the sound source i, pL(i,b) = L(i,b), where L(i,b) is the loudness of sound source i in frequency band b. This value occurs when there is no masking between sound sources, and therefore, the compressor works alone: ​​L(∑E) = ∑L(E). Masking can reduce this value.

[0331] The source equalizer 710 then automatically equalizes some of the sound sources. In some implementations, equalization prevents some sound sources from being masked by others. The source equalizer 710 therefore assumes that all sound sources have been tuned during the initial premixing stage so that they emit sound at a given target loudness. For example, all sound sources may have equal loudness (except for the lead vocals), and the source equalizer 710 may make the lead vocals 3dB louder than all other sound sources.

[0332] The premixing stage can focus on individual sound sources. However, when all sound sources are played at the same time, masking can occur. The source equalizer 710 can perform equalization operations to emphasize some sources, for example, to help some sound sources stand out. A typical example is the bass. When the bass is played with an organ or other wideband instruments, the source equalizer 710 can apply a high-pass operation to these wideband instruments, making the bass more prominent at the low end. Conversely, in an organ solo audio event, the source equalizer 710 does not apply equalization for this reason. Therefore, this problem is a cross-instrument problem.

[0333] One way to keep the source equalizer 710 running is to detect which sound source or which frequency band of which sound source has a greater need for equalization. The source equalizer 710 can specify this quantity as Need(i,b) for instrument i and frequency band b. This need can depend on the following factors: i) how important the frequency band is to the sound source; and ii) how the sound source is masked by all other sound sources. The source equalizer 710 can quantify the importance and the degree of masking using I(i,b) and M(i,b) respectively.

[0334] The importance of a musical instrument's frequency band depends solely on the instrument itself (unlike masking). For example, the low-end frequency band of a bass may be important for a bass. By comparison, the low-end frequency band of an organ may be less important because the organ extends further across the frequency range. The sound source equalizer 710 can measure and define the importance of frequencies bounded to [0,1] as follows:

[0335]

[0336] Where I(i,b) represents the importance level of instrument i in frequency band b after equalization.

[0337] To measure the masking effect of all other instruments, the sound source equalizer 710 can designate all other sound sources as noise. The sound source equalizer 710 can utilize a portion of the loudness of the other sound sources. To obtain an indicator bounded to [0,1]:

[0338]

[0339] Where M(i,b) represents the masking level of other instruments in frequency band b for instrument i.

[0340] Therefore, the sound source equalizer 710 can implement the demand function as follows:

[0341]

[0342] Need(i,b) is the required value indicating the relative importance of sound source i in frequency band b relative to other sound sources and other frequency bands after equalization.

[0343] The sound source equalizer 710 simplifies the notation for the loudness of all sound sources except i: L({i′},b)=L[∑ i′ The final implementation scheme can be expressed as follows: E(i′,b)].

[0344]

[0345] The sound source equalizer 710 can implement the following algorithm to achieve automatic equalization. Improvements are described in the next section. For convenience, the quantity Need(i,b) is simplified to N(i,b).

[0346] Step 1: The sound source equalizer 710 can find the sound source and frequency band with the highest N(i,b). The sound source equalizer 710 can designate this as a source-band pair: For example: (bass, third band). The sound source equalizer 710 can compare this highest N(i,b) with the threshold t∈[0,1]. If N(i,b)>t, then proceed to step 2; otherwise, stop (nothing else needs to be equalized).

[0347] Step 2: The sound source equalizer 710 equalizes the remaining instruments to highlight the selected pair. The sound source equalizer 710 can use i′ to represent the difference between the selected pair and the others. All other sound sources. The Sound Source Equalizer 710 can be paired with it. The resulting masking is distributed proportionally among each sound source i′. This is done by reducing the gain to the frequency band of each instrument:

[0348]

[0349] Where g is the only unknown that will be fixed by the requirement that it be sufficiently prominent. Figure 14A and 14B The text demonstrates how the gain of each instrument changes with g.

[0350] If g = 1, then all gains are 1, and each source-band pair is masked by its own gain. The degree of reduction in its excitation is proportional to the extent of the effect. Based on the same equation (54), i′ is obtained for... The resulting masking, but treating i′ as noise and... Consider it a signal:

[0351]

[0352] in This indicates that sound source i′ corresponds to sound sources in frequency band b. The degree of concealment achieved.

[0353] At the same time, the sound source equalizer 710 can also boost the selected instrument-frequency band.

[0354]

[0355] Where α represents the amount of relative boost that the control provides. (Regarding the sound source equalizer 710, it enables all other...) The parameter (how much attenuation) is used. For example, if α = 0, then the source equalizer 710 will never boost. If α = 1, then the source equalizer 710 will boost by up to the amount that the source equalizer 710 attenuates the rest.

[0356] Finally, the sound source equalizer 710 solves for g by defining the target masking level of the source-band pair such that it is not sufficiently masked. Where M thresholdThe masking threshold is given by an equation representing an unknown (g). Equation (54) shows that the equation is nonlinear. The source equalizer 710 can solve for g by reducing the discrete step order of g in dB fractions (for example) until a limit is met. The source equalizer 710 can reverse the gain-loudness mapping and thus return from the loudness domain to the linear domain.

[0357] In this context, the sound source equalizer 710 can be configured by setting a minimum allowed value of g or (for better control) by directly limiting it. and The maximum level of tolerable equilibrium is imposed based on the allowed values.

[0358] Step 3: The sound source equalizer 710 can remove pairs from the possible pair list. To make it stand out. Return to step 1.

[0359] The algorithm described above selects candidate pairs only by examining the pairs with the greatest demand. The algorithm does not guarantee that the benefit is global across all instruments. The global approach mimics spatial encoding: find the pair that minimizes the global requirement immediately after the enhancement; then iterate.

[0360] For example, the Sound Source Equalizer 710 can define the global requirements of the mix to be equalized as:

[0361]

[0362] Here, Need(global) refers to the global requirement.

[0363] The sound source equalizer 710 then performs the following operations. First, the sound source equalizer 710 calculates the initial global demand Need(global). Second, the sound source equalizer 710 then obtains all possible pairs (i,b), or selects several pairs (e.g., 10 pairs) with higher N(i,b). For each pair, the sound source equalizer 710 designates it as a candidate to be boosted and runs the previous algorithm to find a gain g(i,b) to be applied to boost the candidate and attenuate the others. For each pair thus considered, the sound source equalizer 710 recalculates the new global demand. Third, the sound source equalizer 710 selects the pair that minimizes the global demand and applies its boost gain. The sound source equalizer 710 then replaces Need(global) with its new value and returns to the first step.

[0364] The sound source equalizer 710 may terminate the above iteration if any of the following occurs: 1. Need(global) is already below a given threshold; 2. The selection of (i,b) does not result in a decrease in Need(global); or 3. The sound source equalizer 710 has iterated more than the given maximum number of times.

[0365] Figure 14A This is a diagram representing the three-instrument mix to be equalized. The horizontal axis represents the frequency f. The three-instrument mix includes the bass, organ, and other instruments. The vertical axis represents energy. As shown, the bass's energy is concentrated in the lower ERB band. Therefore, the sound source equalizer 710 can determine that the bass has a higher demand for equalization in the lower ERB band compared to the organ and other instruments.

[0366] Figure 14B This is a diagram illustrating the gain in automatic equalization. Starting at g=1, the gain is decreased, increasing the bass gain in the lower ERB band while attenuating all other instruments in the lower ERB band. The organ masks the bass more than the "other" instrument, and therefore the organ attenuates more.

[0367] Based on novel segmentation

[0368] Figure 15 This is a flowchart illustrating an example of video segmentation based on novel accumulations in audio data.1500 Figure 7 The segmentation unit 714 executes process 1500. In some implementations, it can be... Figure 5 The video editor 530 implements segmentation unit 714.

[0369] Segmentation unit 714 can receive (1502) an audio signal. Segmentation unit 714 can construct (1504) a novelty index of the audio signal over time. Segmentation unit 714 can determine the peak value of the audio signal above a threshold. Segmentation unit 714 can determine (1506) a segment length based on an average cut length. The cut length can be an input, a preset value, or derived from past cuts (e.g., by averaging over X past cuts). Segmentation unit 714 can determine (1508) a sum of novelty indices since the last cut. The sum can be an integral of the novelty indices over time.

[0370] After determining a novelty threshold, segmentation unit 714 immediately determines (1510) the random time of the next cut, wherein the randomness of the next cut time is averaged to the average segment length. Segmentation unit 714 can cut (1512) the audio signal or the corresponding video signal synchronized with the audio signal into new segments at the random time and begin summing the novelty index of the next cut. Segmentation unit 714 can provide the new segments to a user device for streaming or downloading, and for playback on a speaker. Additional details and examples of process 1500 are described below.

[0371] Video editing can be based on novelty, which indicates points of significant change in audio or video. Segmentation unit 714 can construct an index to measure novelty, called the novelty index. Segmentation unit 714 constructs the novelty index by comparing the extracted feature set across different segments of the audio recording. Segmentation unit 714 can construct a similarity index and convolve it with a checkers board kernel to extract novelty.

[0372] Segmentation unit 714 can calculate the novelty index across time. Segmentation unit 714 can first select peaks above a certain threshold. The size of the segments used to extract features determines the scale of the novelty operation. Short segments allow for the differentiation of individual notes. Long segments allow for the differentiation of coarser concepts, such as distinguishing between an introduction and a chorus. The threshold for segments considered novel can affect the frequency of cuts, and therefore can be set as a function of the desired average cut length, which in turn can be set as a function of rhythm. Therefore, segmentation unit 714 can perform the following operations:

[0373] • Search rhythm → Set average shear length → Set threshold.

[0374] Segmentation unit 714 can appropriately handle sections of a song with crescendo. These are characterized by a prolonged, smooth increase in a novel index that does not lead to a prominent peak, and thus results in no cuts over a very long period. Segmentation unit 714 may include an accumulation processing module that quantifies the need for a cut over the duration, independent of the fact that a peak may occur. Segmentation unit 714 can characterize the cut from the last cut t last This requirement for integrals based on novel indexes:

[0375]

[0376] Where N(t) represents the demand and novelty(t) represents the novelty index at time t.

[0377] When N(t) is determined to be higher than the threshold N thr Subsequently, segmentation unit 714 is tuned to ensure that there is an average probability of shearing over the next T seconds and begins to randomly sample. Segmentation unit 714 can then process N... thrValues ​​are assigned to those considered to be in high demand, for example, a novelty duration of at least 3 seconds = 0.6. Similarly, the value of T can be related to the required average shear length, as described above.

[0378] Figure 16 It is a diagram illustrating the process of accumulating examples of novelty. In Figure 16 The diagram illustrates the effect of novel accumulation processing on a song with a gradual increase in volume. The X-axis represents time in seconds. The Y-axis represents the value of the integral. Long volume increases exist between 150 and 180 seconds, as shown by curve 1602. Curve 1602 shows the integral without the novelty of the accumulation post-processing. If only the peak value of the index is used, the event will be undetectable in this segment. Curve 1604 shows the integral after the accumulation post-processing, revealing both the presence of the new cut and the controlled randomness of its appearance. This appearance may be based on a hard threshold or preferably on probability.

[0379] synchronous

[0380] Figure 17 This diagram illustrates an exemplary process for synchronizing audio signals from multiple microphones (1700). Figure 5 Synchronizer 502 executes process 1700. Synchronizer 502 in process 1700 improves upon conventional synchronization techniques because it can synchronize audio signals based solely on the audio signal.

[0381] Synchronizer 502 can synchronize microphones used in an audio scene simply by analyzing the audio. In some implementations, synchronizer 502 can use various correlation determination techniques (e.g., cross-correlation algorithms) to synchronize all microphones to the primary stereo channel.

[0382] Synchronizer 502 can receive (1702) an audio signal. The audio signal can be a channel signal from a microphone. Synchronizer 502 can calculate (1704) the corresponding quality value of the correlation between each pair of audio signals. Synchronizer 502 can assign (1706) the quality value to a mapping vector.

[0383] Synchronizer 502 iteratively determines a series of delays and inserts these delays into the mapping vector (1708). Synchronizer 502 identifies a pair of signals in the mapping vector with the highest quality value. Synchronizer 502 aligns the audio signals in the pair and downmixes the pair to a mono signal, appending the aligned delay to the mapping vector. Synchronizer 502 replaces the first audio signal in the pair with the downmixed mono signal and removes the second audio signal from the list of maximum values. Synchronizer 502 keeps the downmixed mono signal fixed and recalculates the quality value. Synchronizer 502 iterates from the identification stage until only one signal remains.

[0384] After completing the iteration, synchronizer 502 can immediately synchronize the audio signal using the corresponding delays in the mapping vector according to the order in which the delays were inserted into the mapping vector (1710). Synchronizer 502 can then submit the synchronized signal to other components (e.g., Figure 5 The source splitter 504 is used for additional processing and streaming. Further details and examples of process 1700 are described below.

[0385] In some implementations, synchronizer 502 performs a globally synchronized algorithm, thus placing greater emphasis on delays calculated based on cross-correlation with strong peaks. Synchronizer 502 can label microphones in the scene as m = 1, ..., M. If one of the microphones is stereo, then it has previously undergone polarity checking and downmixing to mono. Synchronizer 502 can determine its correlation C over time as follows: m,m′ (t):

[0386]

[0387] Where s m This indicates the signal from microphone m. For t=0, this conforms to the normalized correlation used in equation (3) above.

[0388] For each pair, synchronizer 502 can obtain results that lead to higher and lower correlation, respectively. t max t min The value of . The quality factor (or the quality Q of the correlation) describing how good the correlation is would be:

[0389]

[0390] If the minimum value is stronger than the maximum value, then Q can be negative. This will not affect the result, because the synchronizer 502 can seek the maximum value of Q.

[0391] Synchronizer 502 can execute a recursive algorithm as follows. First, synchronizer 502 can initialize an empty mapping vector Map with (M-1) entries. Synchronizer 502 will only consider the upper diagonal of Q (due to its symmetry), and therefore... Where m1 <m2。

[0392] 1. Find the one with the largest The pair of m1 and m2.

[0393] 2. Align with And downmix to mono. Attach to the Map.

[0394] 3. Replace with this downmixing. Remove m2 from the index list to scan for the maximum value of Q.

[0395] 4. Recalculate all values ​​for all m. And m1 is fixed.

[0396] 5. Repeat step one until only one microphone remains.

[0397] The synchronizer 502 has a set of M-1 delays t m,m′ The second index appears only once for each microphone except the first microphone (which is typically the downmixer for the main stereo). To reconstruct the delay required by microphone m to be synchronized with (for example) the first microphone, synchronizer 502 may follow a chain leading to the first microphone:

[0398] t m =t m′,m +t m″,m′ +...+t 1,m″′ . (64)

[0400] In some implementations, synchronizer 502 can improve computation speed by avoiding recalculating all correlations after each downmix to mono. It is assumed that synchronizer 502 has already calculated all C in the first stage. m,m′ This means that synchronizer 502 has calculated all c m,m′ (T)= m (t)s m′ (t+T)>. After aligning m′ to m and downmixing, synchronizer 502 can obtain a new signal (check t). m,m′ (positive and negative signs):

[0401]

[0402] To calculate the new information about s m～ Correlation:

[0403]

[0404] The molecule has two terms. Synchronizer 502 can switch from C. n,m Export the first item. Synchronizer 502 can export from C. n,m′ The known term t m,m′ A cyclic shift of one unit derives the second term. The denominator also has two terms. Synchronizer 502 can obtain from |s m |and|s m′ The second term is derived from its correlation:

[0405]

[0406] ​Therefore, synchronizer 502 only needs to calculate the initial correlation matrix.

[0407] Noise reduction

[0408] Figure 24 This is a flowchart illustrating an example of noise reduction process 2400. (It can be derived from...) Figure 7 The noise reduction unit 711 performs process 2400. The noise reduction unit 711 can apply noise reduction to each channel signal. For example, the advantages of the disclosed method include: by individually applying various gains to each channel, the noise reduction unit 711 can reduce noise when a particular channel has a sufficiently high audio level to mask channel signals from other channels. Furthermore, the channel signals can originate from different channel signal sources (e.g., microphones of different models and modes) that may be located at individual points in the venue (e.g., more than two to three meters away).

[0409] The noise reduction unit 711 can receive (2402) reference audio data. The reference audio data contains channel signals recorded during a silence period in the rehearsal phase. The silence period can be a period during which no instrument is played (e.g., X seconds).

[0410] The noise reduction unit 711 may include a noise estimator component. The noise estimator can estimate (2404) the corresponding noise level in each channel signal of the reference audio data. The noise estimator can designate the estimated noise level as the noise floor. The estimation of the corresponding noise level in each channel signal of the reference audio data can be performed within multiple frequency bands, referred to as frequency bands.

[0411] The noise reduction unit 711 can receive (2406) live performance data. The live performance data includes channel signals recorded during event 100, in which one or more instruments that were silent during the rehearsal phase are played during event 100.

[0412] Noise reduction unit 711 may include a noise reduction component. The noise reduction component can individually reduce (2408) the corresponding noise level in each channel signal of the live performance data. After determining that the difference between the noise level in the channel signal of the live performance data and the estimated noise level meets a threshold, the noise reduction component can immediately apply a corresponding suppression gain to each channel signal of the live performance data. The reduction of the corresponding noise level in each channel signal of the live performance data can be performed in each frequency band.

[0413] After reducing the noise level, the noise reduction unit 711 can provide the channel signal (2410) to downstream devices for further processing, storage, or distribution to one or more end-user devices. The downstream devices may be (for example) Figure 5 The distribution front end 508 or Figure 7The master tape processing unit 704.

[0414] The estimation (2404) and reduction (2408) phases can be performed based on noise reduction parameters including threshold, slope, attack time, decay time, and octave band size. Instance values ​​for the parameters are: threshold of 10 dB; slope of 20 dB per dB; and attack time and decay time of 50 ms. Additional details and examples of the noise reduction operation are described below.

[0415] During the estimation (2404) phase, the noise estimator may perform the following operations individually on each channel signal in the reference audio data. The noise estimator may segment the channel signals in a buffer of X samples (e.g., 2049 samples). The buffers may have a half-length overlap. The noise estimator may apply the square root of a discrete window function (e.g., a Hanning window) to each buffer. The noise estimator may apply a discrete Fourier transform. The noise estimator may calculate the noise level using equation (68) below.

[0416] n(f) = 10 * log10(|·| 2 ), (68)

[0418] Where n(f) is the noise level of a specific frequency band f.

[0419] The noise estimator can determine the noise floor by averaging the noise level within the buffer using the following equation (69).

[0420] n estimate (f)=<n(f)> b uffers , (69)

[0422] Where n estimate (f) represents the noise level of the noise floor specified as frequency band f, and <> represents the average value. Therefore, the noise estimator can determine the number n of each frequency band for each channel signal. estimate (f).

[0423] During the noise reduction (2408) phase, the noise reducer can suppress the noise level of each channel signal in the live performance data of event 100 by performing the following operations individually for each channel signal. The noise reducer can divide the channel signals in the live performance data into buffers of X samples (e.g., 2049 samples). The buffers may have a half-length overlap. The noise reducer can apply the square root of a discrete window function (e.g., a Hanning window) to each buffer. The noise reducer can apply a discrete Fourier transform. The noise reducer can calculate the noise level n(f) using equation (68) above.

[0424] The noise reducer can use equation (70) below to calculate the difference between the noise level n(f) and the background noise in the live performance data.

[0425] d(f) = n(f) - n estimate (f), (70)

[0427] Where d(f) is the difference.

[0428] The noise reducer can then apply a suppression gain to the channel signal in the live performance data in expander mode. Applying the suppression gain in expander mode may involve determining whether the difference d(f) is less than a threshold. After determining that d(f) is less than the threshold, the noise reducer can immediately apply a gain of dB based on the number of dB of the suppression difference, according to the slope parameter.

[0429] Noise reducers can smooth all suppression gains across frequency bands or within a given bandwidth specified in parameters such as octave band size. Noise reducers can use attack time and decay time parameters to smooth all suppression gains over time. Noise reducers can apply an inverse discrete Fourier transform, followed by the square root of a discrete window function. Noise reducers can then be superimposed and their results added.

[0430] Figure 18 The diagram illustrates an exemplary sequence for synchronizing five microphones. First, synchronizer 502 aligns the signal from microphone 3 with the signal from microphone 2. Synchronizer 502 determines the delay t. 23 And will delay t 23 Add to list. Synchronizer 502 downmixes the aligned signal to a mono signal. Synchronizer 502 then replaces the signal from microphone 2 with the mono signal. Synchronizer 502 can continue this process by aligning the mono signal with the signal from microphone 4, and then aligning the signals from microphone 1 to microphone 5. Finally, synchronizer 502 aligns the signal from microphone 1 with the signal from microphone 2. Synchronizer 502 can finally obtain list {t 23 ,t 24 ,t 15 ,t 12 In this case, t2 = t 12 ,t3=t 23 +t 12 ,t4=t 24 +t 12 ,t5=t 15 .

[0431] Video editing

[0432] Figure 19A and 19BThe illustration shows an example user interface displaying the results of automatic video editing. This user interface can be presented on the display surface of a user device (e.g., video device 414). It can be provided by a video editor 530 ( Figure 5 ) Implemented in Figure 19A and 19B The features described in [the document].

[0433] Figure 19A The diagram illustrates event 100 ( Figure 1 The user interface for the first video scene is shown. In the example presented, a band is playing during event 100. A video camera captures live video of the band playing. Each sound source in the band (e.g., singer 192, guitar 194, and other sound sources) plays at a similar level. Video editor 530 can receive the live video and audio data of the band playing. The live video may contain live video of event 100 or pre-stored video. Video editor 530 can determine, based on the audio data, that the difference between the energy (or loudness) levels of each sound source is less than a threshold. In response, video editor 530 can determine that the entire video scene 196 of event 100 can be presented. Video editor 530 can then provide the entire video scene 196 in the live video for streaming. Video device 414 can receive the entire video scene 196 and present the scene for display.

[0434] Figure 19B The diagram illustrates event 100 ( Figure 1 The user interface for the second video scene. (See reference...) Figure 19A During the described live performance period, video editor 530 can determine, based on audio data, that one or more sound sources are playing at a significantly higher level than other instruments. For example, video editor 530 can determine that the loudness level or energy level of singer 192 and guitar 194 is above a threshold level higher than the loudness level or energy level of other instruments. In response, video editor 530 can determine the translation angle of one or more sound sources and focus on and magnify a portion of the video data to obtain a partial video scene 198. In the example shown, video editor 520 focuses on and magnifies the position of singer 192 and guitar 194. Video editor 530 can then provide the partial video scene 198 containing singer 192 and guitar 194 in the live video for streaming. Video device 414 can receive the partial video scene 198 containing singer 192 and guitar 194. Video device 414 can present the partial video scene 198 for display.

[0435] Figure 20 This is a flowchart illustrating an example of an automated video editing process 2000. It can be used by video editor 530 (…). Figure 5) Execution process 2000. Video editor 530 is a component of a server configured to receive live video recordings of event 100.

[0436] Video Editor 530 can receive (2002) Event 100 ( Figure 1 The video data and audio data of event 100. The video data and audio data may be on-site data. On-site data may be real-time data or pre-stored data. The video data may contain images of sound sources located at different locations in event 100. The audio data may contain the energy level or loudness level of the sound sources and the translation angle of the sound sources.

[0437] The video editor 530 can determine (2004) a specific sound source as the dominant sound source based on the audio data. For example, the video editor 530 can determine that the signal of the sound source represented in the audio data indicates that the sound source is playing at a volume level that is higher than the volume level of other sound sources represented in the audio data by a threshold amount.

[0438] Video editor 530 can determine the location of a sound source in video data (2006). In some embodiments, video editor 530 can determine the location based on the translation angle of the sound source in the audio data. For example, video editor 530 can determine the angular width of the entire scene in the video data and determine the location corresponding to the angle in the entire space (corresponding to the translation angle of the sound source). Video editor 530 can determine the translation position of the sound source based on the audio data. Video editor 530 can specify the translation position of the sound source as the location of the sound source in the video data. In some embodiments, video editor 530 can determine the location based on the video data, for example, by using face tracking or instrument tracking.

[0439] The video editor 530 can determine (2008) a portion of the live video data corresponding to the location of the sound source. For example, the video editor 530 can magnify a portion of the live video data based on the translation angle of the sound source.

[0440] The video editor 530 can simultaneously provide (2010) portions of audio data and live video data for streaming to a storage device or end-user device. Thus, for example, if a singer or guitarist is performing a solo, the live video played on the end-user device can automatically zoom in on the singer or guitarist without interference or control from a camera operator.

[0441] Additionally, in some implementations, the video editor 530 may receive input identifying the location of various sound sources in event 100. For example, the video editor 530 may be included in or coupled to a client-side application with a user interface. The user interface may receive one or more touch inputs on a still image or video input to event 100, each touch input associating a location with a sound source. For example, a user may specify "guitar" by touching a guitarist in a still image or video. During rehearsal, the user may specify "guitar" and then record a guitar performance. Thus, the sound labeled "guitar" can be associated with a location in the still image or video.

[0442] Although event 100 is in progress, video editor 530 can receive live video recordings and receive Ns signals from Ns sound sources from source separator 504. Video editor 530 can identify one or more dominant signals from the multiple signals. For example, video editor 530 can determine that the signal from a particular sound source (e.g., a singer) is louder than every other signal by X dB, where X is the threshold number. In response, video editor 530 can identify a tag (e.g., "singer") and identify the location of the live video recording corresponding to the tag. Video editor 530 can focus on the location, for example, by clipping a portion of the original video recording corresponding to the location or by zooming in on a portion of the original video recording. For example, if the original video recording is in 4K resolution, then video editor 530 can clip a 720p resolution video corresponding to the location. Video editor 530 can provide the clipped video to distribution front end 508 for streaming to end-user devices.

[0443] Rehearsal-based video processing

[0444] Figure 25 This is a block diagram illustrating an example technique for video editing based on rehearsal data. The example server system 2502 is configured to provide editing decisions for live video data based on rehearsal video data. The server system 2502 may contain one or more processors.

[0445] Server system 2502 is configured to automatically edit live data 2504 based on rehearsal video data and rehearsal audio data, such as live audio and video of a musical performance or live audio and video of any event. The live data includes M video signals 2506 of the performance captured by M video capture devices (e.g., one or more video cameras). Audio data 2508 includes N audio signals from N audio capture devices (e.g., one or more microphones). The number and location of the audio capture devices can be arbitrary. Therefore, the corresponding input gain of the audio capture devices can be unknown. Due to the arrangement of the audio capture devices, the level of the audio signals may not be directly related to the natural or perceived level of the performer's playing.

[0446] Server system 2502 can determine, based on live data 2504 and rehearsal data 2510, which performer is playing and their approximate level. Each performer can be an instrument, a person playing an instrument, a person performing as a singer, or a person operating a device that otherwise generates electronic or physical sound signals. As previously indicated, instruments, singers, and devices are referred to as sound sources. For example, in live data 2504, the feed corresponding to the first sound source (e.g., bass) of the first performer may be lower than that of the second sound source (e.g., guitar) of the second performer, or even louder than the second instrument played during the actual performance. This difference can result from recording configurations where the various input level stages involved in the chain of each sound source and the physical distance between the audio capture device and the sound source can vary.

[0447] Typically, human operators (e.g., sound engineers, cinematographers, or video directors) use knowledge of who is playing and at what level to determine how to edit the video. Server system 2502 can derive knowledge from rehearsal data 2510 and apply one or more editing rules that specify user preferences (e.g., art direction) to perform editing that simulates human operator editing.

[0448] Server system 2502 can determine the location of each performer in the camera feed during the rehearsal phase using rehearsal data 2510. Server system 2502 then forms a mapping between the sound source and the performer's voice without requiring the performer or operator to manually enter the mapping.

[0449] During rehearsals, the band positions its sound sources at various locations on stage in the same layout as in a live performance. One or more audio capture devices and one or more video capture devices are also positioned at the rehearsal location in the same layout as in a live performance. Each audio capture device may be a barometric (barometric) microphone, a direct input feed (e.g., from an electronic keyboard), or a device that captures signals in the digital domain generated by a digital sound source (e.g., a laptop computer running music production software). At least one video capture device is a video camera positioned to capture all sound sources and performers in the band in a single video frame. Server system 2502 can use the audio and video recordings of the rehearsal as rehearsal data 2510 to configure parameters for editing live data 2504.

[0450] Rehearsal data 2510 includes rehearsal audio data 2512 and rehearsal video data 2514. The analysis module 2516 of the server system 2502 correlates the loudness range of the sound sources with the digital loudness range present in the final digital stream. The analysis module 2516 thus calibrates the multiple level stages involved between signal capture and final digital representation. In some embodiments, the analysis module 2516 determines a corresponding average digital range for each sound source, as captured by each of the audio capture devices. This average may be a weighted average of EBU loudness levels between a low-level soft play and a high-level loud play.

[0451] Analysis module 2516 analyzes rehearsal video data 2514 to determine the location of each performer within a video frame. Analysis module 2516 can use human detection, face detection algorithms, torso detection algorithms, pre-filtering via background subtraction, and any combination of these with other object recognition algorithms to make this determination. Some example algorithms include Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Local Binary Pattern Recognition (LBP), Facial Feature Code (FTC), Evolutionary Voting Algorithm (EVA), Deep Learning Networks (DLN), and others.

[0452] In some implementations, the analysis module 2516 includes a sound source detector. The sound source detector is configured to analyze rehearsal audio data 2512 to identify each individual sound source, applying media intelligence to determine what type of sound source it is at a low level (e.g., bass, piano, or vocals), a high level (e.g., harmonics, percussion), or both. In some implementations, one or more instrument identification (MIR) processes performed in the analysis module 2516 can retrieve a global descriptor for the event, which (for example) indicates whether the genre of the played music is rock, classical, jazz, etc. The analysis module 2516 can provide the sound source type and global descriptor to an automated video editing engine (AVEE) 2518 for editing live data 2504.

[0453] Analysis module 2516 associates each performer detected by analysis module 2516 with a corresponding sound source. For example, analysis module 2516 may map faces identified by analysis module 2516 with specific sounds (e.g., the sound of a guitar). In some embodiments, analysis module 2516 determines the mapping by sorting the sounds and faces. For example, rehearsal audio data 2512 may contain sound sources played in, for example, from left to right as seen in a video. Analysis module 2516 then associates the leftmost detected face with the first sound source played during rehearsal. In another instance, data is acquired via direct human input through a customized graphical interface (e.g., by displaying still frames of the band captured by one of the video devices and prompting the user to tap each performer to then select which sound sources the performer is playing from a pre-filled menu).

[0454] After the band completes rehearsal, the band can begin its live performance. Server system 2502 captures live data 2504 in the same manner as during rehearsal. Sound sources (e.g., performers) can be located in substantially the same locations during the live performance and during rehearsal. Audio and video capture devices are placed in the same locations as during rehearsal. Server system 2502 feeds live audio data 2508 to inference module 2520 and feature extraction module 2522. Inference module 2520 is configured to determine the loudness of each sound source or group of sound sources at a given time. The output of inference module 2520 may include the sound level of each sound source or group of sound sources, for example, referring to the loudness played during rehearsal, in dB, for example, the difference from a low level, the difference from a high level, or the difference from the average by X dB. Referring to the loudness during rehearsal can eliminate ambiguity related to the possibly different level stages used during analog-to-digital conversion of each sound source.

[0455] The feature extraction module 2522 is configured to (for example) retrieve temporal variation features of the live audio data 2508 using an MIR algorithm. The feature extraction module 2522 can perform operations including (for example) beat detection (including downbeat detection), calculation of novelty index, rhythm, tonality, and others.

[0456] Server system 2502 can feed live video data 2506 to adaptive tracking module 2524. Adaptive tracking module 2524 is configured to perform adaptive face tracking, performer tracking, or other object tracking. Adaptive tracking module 2524 therefore considers performers who may leave the stage and therefore should not be in focus. Adaptive tracking module 2524 is also configured to track performers who have moved significantly from their original position (e.g., when a singer is walking and dancing on stage).

[0457] Analysis module 2516, inference module 2520, and feature extraction module 2522 provide their output to AVEE 2518. AVEE 2518 is a component of system 2502 configured to perform operations including framing performers. Conventional face detection algorithms accurately describe where each person's face is located, but not accurately describe how to frame them for scaling and cropping. AVEE 2518 uses the corresponding size and position of each face to derive the corresponding size and position of the corresponding subframe of the original high-resolution video frame providing a focused view of the corresponding performer. The subframe can be a lower resolution (e.g., 720p) frame cropped from a higher resolution (e.g., 4K) frame by AVEE 2518 or a video capture device. AVEE 2518 can render the subframe as an image of the event. AVEE 2518 can determine the size and position of frames in making cropping decisions based on size, position, and salience.

[0458] In making size-based cropping decisions, AVEE 2518 uses a facial proportioning framing algorithm to determine the size of subframes, where AVEE 2518 determines that the size of the subframe is proportional to the performer's identified face. For example, AVEE 2518 may determine that the height of the performer's subframe is X (e.g., five) times the face diameter. AVEE 2518 may determine that the width of the subframe is a multiple of the height to achieve a predetermined aspect ratio. Similarly, AVEE 2518 may determine that the width of the performer's subframe is Y (e.g., eight) times the face diameter. AVEE 2518 may determine that the height of the subframe is a multiple of the weights used to achieve the aspect ratio. AVEE 2518 may determine that the facial horizontality is concentrated in the subframe and in one-third of the path from the top of the subframe downwards.

[0459] Alternatively or additionally, in some embodiments, AVEE 2518 uses a hand-proportioning algorithm to determine the size of the subframe, wherein AVEE 2518 determines the size of the subframe to be proportional to one or more identified hands of the performer. Alternatively or additionally, in some embodiments, AVEE 2518 uses a sound source-proportioning algorithm to determine the size of the subframe, wherein AVEE 2518 determines the size of the subframe to be proportional to an identified musical sound source or an identified sound source or one or more other regions of interest.

[0460] In making position-based cut decisions, AVEE 2518 can use motion tracking to determine the position of subframes within a high-resolution frame. For example, when the adaptive tracking module 2524 notifies a performer that they are moving across the stage and provides a movement path, AVEE 2518 can follow the performer, identified by their face, and move the subframe of the focus view along the path.

[0461] In making salience-based cut decisions, AVEE 2518 places subframes on prominent performers or groups of prominent performers. AVEE 2518 can determine the salience of a performer based on various states from the live audio data 2508. For example, AVEE 2518 can determine the probability that a performer is prominent in a performance at a given moment based on the outputs of the inference module 2520 and the feature extraction module 2522. AVEE 2518 can select a performer for the next video cut based on this probability; the higher the probability, the higher the probability that the performer will be selected for the next cut. The probability that AVEE 2518 will select a subframe covering the performer is positively correlated with the probability that the performer is prominent; for example, the higher the probability that the performer is prominent, the higher the probability that AVEE 2518 will select a subframe covering the performer. AVEE 2518 can determine the likelihood of a performer being salience based on audio characteristics, including, for example, the energy (e.g., RMS energy) of the corresponding audio signal for the performer, the RMS energy Δ (increase or decrease) compared to the last N seconds of the performance, note start frequencies, rhythm changes, and other characteristics. Alternatively, AVEE 2518 can use various video characteristics to determine salience. These video characteristics may include, for example, motion within subframe boundaries.

[0462] AVEE 2518 can create video edits that match the tempo and flow of music. For example, AVEE 2518 can determine cuts in a way that the average frequency of the cut is related to the rhythm of the music (as estimated by feature extraction module 2522). AVEE 2518 can determine the precise timing of each specific cut by aligning the cut with novel changes in live audio data 2508 above a given threshold, which is optionally related to the rhythm of the music. Higher rhythms correspond to lower thresholds, and therefore higher frequencies of the cut. These changes may include, for example, a change in overall loudness or timbre, or the start or stop of one or more performers. AVEE 2518 can time cuts based on an assessment of the musical structure of the performance. For example, AVEE 2518 can align cuts temporally with musical measurements or phrases.

[0463] AVEE 2518 can determine the selection of the subframe to be cut based on performance metrics including outputs from analysis module 2516, inference module 2520, feature extraction module 2522, and optionally adaptive tracking module 2524. The performance metrics may include a corresponding salience metric for each performer as described above, the corresponding specifications of each performer's subframe, and cut decisions based on size, position, and salience. AVEE 2518 can use the following exemplary procedure to determine which subframe to cut.

[0464] AVEE 2518 can detect the next peak of a novel index. AVEE can define a peak using the maximum loudness followed by a decay exceeding a predefined and / or configurable threshold time, threshold level, or both.

[0465] AVEE 2518 determines the elapsed time since the last cut and the elapsed time since the full-frame shot showing all performers. After determining that the elapsed time since the last cut is less than a predefined and / or configurable minimum cut length, AVEE 2518 immediately returns to the first stage to detect the next peak of the novelty index. After determining that the elapsed time since the full-frame shot exceeds a threshold, AVEE 2518 immediately cuts to a full frame. AVEE 2518 can define the threshold based on the number of cuts or duration derived from the rhythm.

[0466] AVEE 2518 can select to eliminate one or more performers (if their subframes have been displayed for more than a threshold time). AVEE 2518 can determine this threshold time based on tempo. For example, a faster tempo may correspond to a shorter threshold time.

[0467] AVEE 2518 can increase the salience of performers designated as having a lead role, matching the highest salience among all performers. AVEE 2518 can designate one or more performers as having a lead role based on input received from the user interface.

[0468] AVEE 2518 can construct a list of performers whose salience value is within X (e.g., three) dB of the maximum salience among all performers. AVEE 2518 can add additional entries to this list of lead performers. AVEE 2518 can determine the number of additional entries to be added to the list. For example, the number of additional entries can be related to the total number of performers. AVEE 2518 can randomly select performers according to the method for selecting performers described above.

[0469] Based on video editing decisions, the AVEE 2518 can, for example, edit live video data 2506 in real time during a performance. The AVEE 2518 can provide edited video data for storage or streaming to one or more user devices. In streaming scenarios, the AVEE 2518 can use a pre-set time within the AVEE 2518 for buffering to perform the processing described above. The pre-set time can be preconfigured to X seconds, for example, greater than 1 second, 5 to 10 seconds, etc. The AVEE 2518 can determine the pre-set time based on the buffering requirements of the cloud service application receiving the streaming. In offline scenarios where content is stored but not streaming, the AVEE 2518 can set the pre-set time to infinity or any sufficiently large time period that would cover the entire performance or song.

[0470] For convenience, the term "tracking" is used to describe the performer. In various implementations, tracking is not limited to the performer. For example, tracking might involve an instrument (e.g., a guitar), a part of the instrument (e.g., the guitar neck), or a part of the performer (a pianist's hand). AVEE 2518 can designate these areas as potential candidates to be focused and framed.

[0471] exist Figure 25 For convenience, the analysis module 2516, AVEE 2518, inference module 2520, feature extraction module 2522, and adaptive tracking module 2524 are presented as separate modules. In various embodiments, these modules may be combined or subdivided. For example, in some embodiments, the functions of the analysis module 2516, inference module 2520, feature extraction module 2522, and adaptive tracking module 2524 may be implemented by AVEE 2518. In some embodiments, AVEE makes video editing decisions and provides these decisions as instructions to one or more video capture devices. The one or more video capture devices then implement the decisions.

[0472] Figure 26 This is a flowchart illustrating an example of video editing based on rehearsal data 2600. It can be generated by a server system (e.g., Figure 25 The server system 2502) executes process 2600.

[0473] The server system receives (2602) rehearsal data, comprising rehearsal video data and rehearsal audio data, from one or more recording devices. The rehearsal data represents a rehearsal of the event performed by one or more performers of the event. The one or more recording devices include one or more microphones and one or more video cameras. The one or more video cameras may include at least one video camera designated as a high-resolution video camera (e.g., a 4K-capable video camera).

[0474] The server system identifies (2604) a corresponding image of each of one or more performers based on rehearsal video data. The identification of the corresponding image may be based on video-based tracking of at least one of the performers or the instruments played by the performers. For example, the identification may be based on facial recognition, instrument recognition, or other object recognition.

[0475] The server system determines (2606) the corresponding sound attributes associated with each identified image based on the rehearsal audio data. Sound attributes may include sound type, sound level, or both. Sound type may indicate the type of instrument used by the performer, such as guitar, drums, or vocals.

[0476] The server system receives (2608) on-site data containing on-site video and audio data of the event from one or more recording devices. In some implementations, the on-site data may be buffered on one or more recording devices, on the server system, or both, corresponding to a time period corresponding to the data processing time and whether the result will be stored or streamed to a user device.

[0477] The server system determines (2610) the prominence of each performer based on the identified image and associated sound attributes. The system can derive the corresponding level of each performer's performance relative to the rehearsal and the corresponding position of each performer during the rehearsal, as captured by one or more video cameras. As described above, the server system can use techniques for determining the dominant sound source to determine the prominent performer. In some implementations, determining the first performer as the prominent performer may include the following operations: The server system normalizes each loudness level among the performers based on live rehearsal audio data. The service system determines that, in the live audio data, at least one performer performs at a level that, after normalization, is at least a threshold amount higher than the normalized loudness levels of other performers. The service system can then determine the first performer as the prominent performer.

[0478] In some implementations, normalizing each loudness level may involve the following steps: The server system determines a first-level sound and a second-level sound for each performer based on rehearsal audio data. The first level is lower than the second level. The server system then normalizes each loudness level by scaling and aligning it to both the first and second levels.

[0479] In some implementations, determining the first performer as the prominent performer may include the following steps: The server system determines, based on live video data, that the first performer's movement exceeds the movement of other performers by at least a threshold. The server system then determines the first performer as the prominent performer based on the movement.

[0480] The server system edits (2612) the live video and audio data according to one or more editing rules. In editing the live data, the server system emphasizes at least one performer based on a corresponding salience. For example, editing may emphasize the singers, instruments, or ensembles of a band or orchestra (e.g., brass or woodwind sections) that includes multiple performers. Editing of the live video data can be performed by the server system. In some implementations, editing can be performed by the recording device. For example, the server system may provide editing instructions to the recording device, causing the recording device to perform the editing operation.

[0481] Editing live video and audio data may include determining the speed and tempo of an event based on the live audio data. The server system can then cut the live video data according to the speed and tempo. Editing live video and audio data may also include determining when a performer (e.g., the first performer) has started or stopped playing. The server system can then, in response, cut the live video data at the point where playback has started or stopped (e.g., during execution).

[0482] In some implementations, editing live video and audio data involves the following operations: The server system determines that the elapsed time since full-frame footage of all performers has exceeded a threshold time. The server system may cut the live video data in response. The server system may determine the threshold time based on the number of cuts or the duration derived from the rhythm of the live audio data.

[0483] The server system then provides (2614) edited data for playback. The server system may store the association between the edited live video data and the edited live audio data on a storage device, or stream the association between the edited live video data and the edited live audio data to the user device.

[0484] Frame region selection

[0485] Figure 27 This is a block diagram illustrating an exemplary technique for selecting subframe regions from full-frame video data. At a live event (e.g., a concert), at least one video capture device 2702 captures video of the event. At least one audio capture device 2704 captures audio of the event. Devices 2702 and 2704 submit the live video and audio data to a server system 2706 via a communication network 2708 (e.g., the Internet). Video capture device 2702 can record video at high resolution (e.g., 4K). High-resolution video may consume too much bandwidth on the communication network 2708 for uploading to or downloading from the server system 2706 to a user device.

[0486] Server system 2706 may be the same as or different from the reference. Figure 25 The described server system 2502. Server system 2706 can store or stream medium-resolution (e.g., 720p) video to user devices. Video capture device 2702 can be configured as a slave device of server system 2706. As a slave device of server system 2706, video capture device 2702 follows commands from server system 2706 to zoom, crop, and select focus on video data. Server system 2706, which receives live audio data from audio capture device 2704, identifies performers and corresponding instruments, makes decisions about selecting subframes, and guides video capture device 2702 to submit the selected subframes to server system 2706.

[0487] Server system 2706 receives at least one video frame across the entire frequency band at the event location. The video frame contains all performers and does not need to be in full resolution, and may be optionally compressed using a lossy codec. Server system 2706 then makes a decision about where to focus and which subframe to select based on the live audio data. Server system 2706 instructs video capture device 2702 to submit medium-resolution live video of only the selected subframe to server system 2706.

[0488] Video capture device 2702 may include a video buffer 2710. The video buffer 2710 is a data storage device configured to store X seconds (e.g., ten seconds) of video data at full resolution. The video data may include a series of full-band frames 2712 and associated timing information. Video capture device 2702 includes a video converter 2714. The video converter 2714 converts the full-band frames 2712 from full resolution to a series of lower resolution (e.g., 720p or 640×480) images. The video converter 2714 submits the lower resolution images to server system 2706 at a reduced frame rate (e.g., 1fps). Over an average time, video capture device 2702 converts the video stream in video buffer 2710 into medium-resolution video and submits the medium-resolution video to server system 2706 at a standard frame rate (e.g., 24fps).

[0489] At the initial time t0, the submitted video can be a video covering the entire frequency band with frames consistent with the lower resolution image. The video capture device 2702 then waits for instructions on editing decisions from the server system 2706 while continuing to submit medium-resolution video data and images to the server system 2706.

[0490] Server system 2706 includes AVEE 2718. AVEE 2718 may be the same as or different from AVEE 2718. Figure 25 The AVEE 2518. The AVEE 2718 receives full-frame image and live audio data. The AVEE 2718 is configured to determine which performer or instrument to focus on based on the full-frame image and live audio data received from the audio capture device 2704. For example, the AVEE 2718 can determine that the singer is the prominent performer at time t1. The AVEE 2718 can then issue instructions to zoom in on the prominent performer (in this example, the singer) in the video starting from time t1. The instructions can be associated with sensor pixel coordinates (e.g., X pixels from the left, Y pixels from the bottom), size, and time t1.

[0491] In response to an instruction, video capture device 2702 performs editing according to the instruction. Video capture device 2702 retrieves video data corresponding to time t1 from video buffer 2710. Video capture device 2702 crops the video data to the desired position based on coordinates. Video capture device 2702 resizes the cropped video data to a specified size and submits the resized video data to server system 2706, while simultaneously submitting converted full-frame images to server system 2706. Video capture device 2702 can submit the resized video data at a standard frame rate (e.g., 24fps). Therefore, server system 2706 will receive video focused on highlighting the performer (e.g., singer) from time t1.

[0492] Based on the live audio data and images submitted to server system 2706, server system 2706 can determine that the second performer (e.g., the violinist) becomes the prominent performer at a second time t2. Server system 2706 then provides instructions to video capture device 2702 to focus on a portion of the live video containing the violinist. Upon receiving instructions specifying the position and size of the subframe and time t2, video capture device 2702 immediately changes the cropping coordinates and submits the cropped and optionally resized video containing the violinist to server system 2706. Therefore, server system 2706 receives the medium-resolution video of the violinist from time t2.

[0493] Server system 2706 includes an assembly unit 2720. Assembly unit 2720 is configured to assemble medium-resolution video data from video capture device 2702 and live audio data from audio capture device 2704 for storage or streaming to a user device. For live streaming, assembly unit 2720 can add a delay to the beginning of the assembled video stream. Both video buffer 2710 and the delay can compensate for delays in decision-making and data transmission. For example, server system 2706 can decide to zoom in on a drummer when he kicks the drum, instructing video capture device 2702 to focus on the drummer at a position in a buffer held in memory corresponding to the time the drummer kicks. This time can be X (e.g., 0.2) seconds before video capture device 2702 receives the command. Server system 2706 then receives the newly edited video and supplies it to the audience, thus using the delay to mask the time for decision-making and command transmission.

[0494] Figure 28 This is a flowchart of an instanced process 2800 performed by a server system to select a subframe region from full-frame video data. The server system may be... Figure 27 Server system 2706.

[0495] The server system receives (2802) at least one frame of audio data of an event and video data of the event from one or more audio capture devices. The video data is captured by a video capture device configured to record video at a first resolution. The first resolution may be 4K or higher. The frame may have a resolution equal to or lower than the first resolution. The video data frame of the event may be a frame from a series of frames of video data. The series of frames may be received at the server system at a frame rate lower than the frame capture rate of the video capture device (e.g., 24fps or higher) (e.g., 1fps or lower). In some embodiments, capturing a single frame of all performers of the event is sufficient. In some embodiments, the video capture device submits multiple frames to the server system to cover performers who may have moved during the event.

[0496] The server system determines the corresponding position of each of the individual performers in the (2804) event based on audio data and images of individual performers identified in frames of video data. The server system may make this determination based on rehearsal data.

[0497] After the server system determines, based on the audio data, that the first performer among the individual performers is the prominent performer at the first moment, the server system immediately instructs (2806) the video capture device to submit a first portion of the video data to the server system at a second resolution. The first portion of the video data is spatially directed to the position of the first performer captured at the first moment. The second resolution may be 1080p or lower.

[0498] After the server system determines, based on the audio data, that the second performer among the individual performers is the prominent performer at the second time point, the server system immediately instructs (2808) the video recorder to submit a second portion of the video data to the server system at a second resolution. This second portion of the video data is spatially directed to the position of the second performer captured at the second time point.

[0499] The server system designates (2810) the first and second portions of the video data as video of an event at a second resolution. The server system then provides (2812) the association between the audio data and the video of the event at the second resolution as audio and video recordings of the event to a storage device or user device. For example, the server system may add a delay to the video of the event at the second resolution. The server system then streams the delayed video and associated audio data to one or more user devices.

[0500] In some implementations, the video capture device buffers a period of video data at a first resolution. In response to a command from the server system, the video capture device selects the position of a frame of buffered video data corresponding to the first and second performers and submits it to the server system.

[0501] Figure 29 This is a flowchart of an exemplary process 2900 performed by a video capture device to select a subframe region from full-frame video data. The video capture device may be... Figure 27 The video capture device 2702.

[0502] The video capture device records (2902) video data at a first resolution. The first resolution may be 4K or higher. The video capture device stores (2904) video data in a local buffer of the video capture device. The video capture device determines (2906) a series of one or more images based on the recorded video data. The video capture device submits (2908) the series of one or more images to a server system at a first frame rate. The first frame rate may be one frame per second or less. The video capture device receives (2910) an instruction from the server system to focus on a portion of the video data, the instruction indicating the temporal and spatial location of said portion of the recorded video data.

[0503] In response to the instruction, the video capture device converts (2912) a portion of the video data stored in its local buffer into video data with a second resolution and a second frame rate higher than the first frame rate, based on the indicated time and spatial location. The second resolution may be 1080p or lower. The second frame rate may be 24 frames per second or higher. The video capture device then submits the converted video data at the second resolution as live video data of the event to the server (2914).

[0504] Exemplary recording device architecture

[0505] Figure 21 This is a diagrammatic explanation of the implementation reference. Figures 1 to 20 A block diagram of an exemplary device architecture 2100 for the apparatus described in sections 24 to 29, illustrating the features and operation of the device. The apparatus may be (for example) Figure 1 Recording device 102 or 104 or Figure 3The recording device 302 may include a memory interface 2102, one or more data processors, an image processor and / or a processor 2104, and a peripheral device interface 2106. The memory interface 2102, the one or more processors 2104, and / or the peripheral device interface 2106 may be separate components or may be integrated into one or more integrated circuits. The processor 2104 may include an application processor, a baseband processor, and a wireless processor. For example, various components in a mobile device may be coupled by one or more communication buses or signal lines.

[0506] Sensors, devices, and subsystems may be coupled to peripheral interface 2106 to facilitate multiple functionalities. For example, motion sensor 2110, light sensor 2112, and proximity sensor 2114 may be coupled to peripheral interface 2106 to facilitate orientation, illumination, and proximity functions of the mobile device. Location processor 2115 may be connected to peripheral interface 2106 to provide geolocation. In some embodiments, location processor 2115 may be programmed to perform the operation of a GNSS receiver. Electronic magnetometer 2116 (e.g., an integrated circuit chip) may also be connected to peripheral interface 2106 to provide direction data that can be used to determine magnetic north. Thus, electronic magnetometer 2116 can be used as an electronic compass. Motion sensor 2110 may include one or more accelerometers configured to determine changes in the speed and direction of movement of the mobile device. Barometer 2117 may include one or more devices connected to peripheral interface 2106 and configured to measure atmospheric pressure around the mobile device.

[0507] The camera subsystem 2120 and optical sensor 2122 (e.g., charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) optical sensor) can be used to enhance camera functions, such as recording photos and video editing.

[0508] Communication functionality can be facilitated by one or more wireless communication subsystems 2124 (which may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters). The specific design and implementation of the communication subsystem 2124 may depend on the communication network through which the mobile device intends to operate. For example, the mobile device may include subsystems designed to operate via GSM networks, GPRS networks, EDGE networks, Wi-Fi... TM or WiMax TM Network and Bluetooth TM A network-operated communication subsystem 2124. Specifically, the wireless communication subsystem 2124 may include a managed protocol that enables a mobile device to be configured as a base station for other wireless devices.

[0509] The audio subsystem 2126 can be coupled to the speaker 2128 and microphone 2130 to facilitate voice-enabled functions such as speech recognition, speech copying, digital recording, and telephone voice functions. The audio subsystem 2126 can be configured to receive voice commands from the user.

[0510] I / O subsystem 2140 may include touch surface controller 2142 and / or other input controller 2144. Touch surface controller 2142 may be coupled to touch surface 2146 or touchpad. Touch surface 2146 and touch surface controller 2142 may (for example) use any of a variety of touch sensitivity technologies (including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies) and other proximity sensor arrays or other elements to detect their contact and movement or interruption for determining one or more contact points with touch surface 2146. Touch surface 2146 may include (for example) a touch screen.

[0511] Other input controllers 2144 may be coupled to other input / control devices 2148, such as one or more buttons, rocker switches, thumbwheels, infrared ports, USB ports, and / or pointer devices (e.g., styluses). One or more buttons (not shown) may include up / down buttons for volume control of the speaker 2128 and / or microphone 2130.

[0512] In one implementation, pressing the button for a first duration unlocks the touch surface 2146; and pressing the button for a second duration longer than the first duration switches power to or off the mobile device. Users may be able to customize the functionality of one or more of the buttons. The touch surface 2146 may also (for example) be used to implement virtual or soft buttons and / or a keyboard.

[0513] In some implementations, the mobile device may display recorded audio and / or video files, such as MP3, AAC, and MPEG files. In some implementations, the mobile device may include MP3 player functionality. Other input / output and control devices may also be used.

[0514] Memory interface 2102 may be coupled to memory 2150. Memory 2150 may include high-speed random access memory and / or non-volatile memory, such as one or more disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Memory 2150 may store operating system 2152, such as iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system (e.g., VxWorks). Operating system 2152 may contain instructions for handling basic system services and performing hardware-dependent tasks. In some embodiments, operating system 2152 may include a kernel (e.g., a UNIX kernel).

[0515] The memory 2150 may also store communication instructions 2154 to facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. The memory 2150 may include: graphical user interface instructions 2156 for facilitating graphical user interface processing; sensor processing instructions 2158 for facilitating sensor-related processing and functions; telephone instructions 2160 for facilitating telephone-related processes and functions; electronic message sending and receiving instructions 2162 for facilitating electronic message sending and receiving processes and functions; web browsing instructions 2164 for facilitating web browsing-related processes and functions; media processing instructions 2166 for facilitating media processing-related processes and functions; GNSS / location instructions 2168 for facilitating general GNSS and location-related processes and instructions; camera instructions 2170 for facilitating camera-related processes and functions; and magnetometer data 2172 and calibration instructions 2174 for facilitating magnetometer calibration. Memory 2150 may also store other software instructions (not shown), such as security instructions, network video instructions to facilitate network video-related processes and functions, and / or network shopping instructions to facilitate network shopping-related processes and functions. In some embodiments, media processing instructions 2166 are divided into audio processing instructions and video processing instructions to facilitate audio processing-related processes and functions and video processing-related processes and functions, respectively. Activation records and International Mobile Equipment Identity (IMEI) or similar hardware identifiers may also be stored in memory 2150. Memory 2150 may store audio processing instructions 2176 that, when executed by processor 2104, cause processor 2104 to perform various operations (e.g., including joining a recording service group by logging into a user account, designating one or more microphones of the device as spot microphones or a primary microphone, recording the audio signal of the group using one or more microphones, and submitting the recorded signal to a server). In some embodiments, audio processing instructions 2176 may cause processor 2104 to perform reference... Figure 4And the operation of server 408 as described in the other figures. Memory 2150 may store references that, when executed by processor 2104, cause processor 2104 to execute. Figures 25 to 29 The video processing instructions described for various operations.

[0516] Each of the instructions and applications identified above may correspond to an instruction set for performing one or more functions described above. These instructions do not need to be implemented as a separate software program, program step, or module. Memory 2150 may contain additional or fewer instructions. Furthermore, various functions of the mobile device may be implemented in hardware and / or software, including implementation in one or more signal processing and / or application-specific integrated circuits.

[0517] Figure 22 yes Figures 1 to 20 Block diagram of an exemplary network operating environment 2200 for mobile devices 24 to 29. Devices 2202a and 2202b may (for example) communicate via one or more wired and / or wireless networks 2210 in data communication. For example, wireless network 2212 (e.g., cellular network) may communicate with wide area network (WAN) 2214 (e.g., the Internet) via gateway 2216. Similarly, access device 2218 (e.g., 802.11g wireless access point) may provide communication access to WAN 2214. Each of devices 2202a and 2202b may be Figure 1 Device 102 or device 104 or Figure 3 The recording device 302.

[0518] In some implementations, both voice and data communications can be established via wireless network 2212 and access device 2218. For example, device 2202a can make and receive telephone calls (e.g., using voice via Internet Protocol (VoIP)) via wireless network 2212, gateway 2216, and wide area network 2214 (e.g., using Transmission Control Protocol / Internet Protocol (TCP / IP) or User Datagram Protocol (UDP)), send and receive email messages (e.g., using Post Office Protocol 3 (POP3)), and retrieve electronic documents and / or streams (e.g., web pages, photos, and videos). Similarly, in some implementations, device 2202b can make and receive telephone calls, send and receive email messages, and retrieve electronic documents via access device 2218 and wide area network 2214. In some implementations, device 2202a or 2202b can be physically connected to access device 2218 using one or more cables, and access device 2218 can be a personal computer. In this configuration, device 2202a or 2202b may be referred to as a “connected” device.

[0519] Devices 2202a and 2202b can also establish communication through other means. For example, wireless device 2202a can communicate with other wireless devices (e.g., other mobile devices, mobile phones, etc.) via wireless network 2212. Similarly, devices 2202a and 2202b can communicate using one or more communication subsystems (e.g., Bluetooth). TM Communication devices can establish peer-to-peer communication, such as personal area networks (LANs). Other communication protocols and topologies can also be implemented.

[0520] Devices 2202a or 2202b may (for example) communicate with one or more services 2230, 2240, and 2250 via one or more wired and / or wireless networks. For example, one or more audio and video processing services 2230 may provide audio processing services including automatic synchronization, automatic leveling, automatic panning, automatic source equalization, automatic splitting, and streaming (as described above). Mixing service 2240 may provide a user interface that allows mixing professionals to log in via a remote console to perform mixing operations on live audio data. Visual effects service 2250 may provide a user interface that allows visual effects professionals to log in via a remote console to edit video data.

[0521] Devices 2202a or 2202b may also access other data and content via one or more wired and / or wireless networks. For example, content publishers (such as news websites, Real Simple Aggregator (RSS) feeds, websites, blogs, social networking sites, developer networks, etc.) may be accessed by devices 2202a or 2202b. This access may be provided by invoking web browsing functionality or an application (e.g., a browser) in response to a user touch (for example) a web object.

[0522] Instantiated system architecture

[0523] Figure 23 Implementation reference Figures 1 to 20 A block diagram of the system architecture (e.g., a server system) featuring and operating as described in sections 24 to 29. Other architectures are also possible, including those with more or fewer components. In some implementations, architecture 2300 includes one or more processors 2302 (e.g., dual-core). The components include a processor, one or more output devices 2304 (e.g., LCD), one or more network interfaces 2306, one or more input devices 2308 (e.g., mouse, keyboard, touchscreen display), and one or more computer-readable media 2312 (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components can exchange communication and data via one or more communication channels 2310 (e.g., bus), which can utilize various hardware and software to facilitate the transmission of data and control signals between the components.

[0524] The term "computer-readable media" refers to media that participates in providing instructions to processor 2302 for execution, and includes, without limitation, non-volatile media (e.g., optical discs or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media includes, without limitation, coaxial cables, copper wires, and optical fibers.

[0525] Computer-readable media 2312 may further include an operating system 2314 (e.g., The system includes an operating system, a network communication module 2316, an audio processing manager 2320, a video processing manager 2330, and a field content distributor 2340. The operating system 2314 can be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, etc. The operating system 2314 performs basic tasks including, but not limited to: identifying input from network interface 2306 and / or device 2308 and providing output to network interface 2306 and / or device 2308; tracking and managing files and directories on computer-readable media 2312 (e.g., memory or storage device); controlling peripheral devices; and managing services on one or more communication channels 2310. The network communication module 2316 includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP / IP and HTTP).

[0526] The audio processing manager 2320 may include computer instructions that, upon execution, cause the processor 2302 to perform various audio estimation and manipulation operations as described above (e.g., with reference to server 408). The video processing manager 2330 may include computer instructions that, upon execution, cause the processor 2302 to perform video editing and manipulation operations as described above (e.g., with reference to video editor 530, AVEE 2518, or AVEE 2718). The field content distributor 2340 may include computer instructions that, upon execution, cause the processor 2302 to perform operations such as receiving reference audio data and field data of audio events and, after processing the audio and visual data, streaming the processed field data to one or more user devices.

[0527] Architecture 2300 can be implemented in parallel processing or peer-to-peer infrastructure or on a single device with one or more processors. The software may comprise multiple software components or may be a single code body.

[0528] The described features can be implemented in one or more computer programs executable on a programmable system, the programmable system comprising at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the data storage system. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform a specific activity or produce a specific result. Computer programs can be written in any form of programming language (e.g., Objective-C, Java) that includes compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0529] A suitable processor for executing instructions includes, by way of example, both general-purpose microprocessors and special-purpose microprocessors of any type of computer, as well as a single processor or one or more processors or cores. Generally, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data files or operatively coupled to and communicating with one or more mass storage devices; these devices include disks (e.g., internal hard disks and removable disks), magneto-optical disks, and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, by way of example including: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by or incorporated into an ASIC (Application-Specific Integrated Circuit).

[0530] To provide interaction with the user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a retinal display device for displaying information to the user. The computer may have a touch surface input device (e.g., a touchscreen) or a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. The computer may have a voice input device for receiving voice commands from the user.

[0531] Features may be implemented in a computer system that includes back-end components (e.g., a data server), middleware components (e.g., an application server or an internet server), or front-end components (e.g., a client computer with a graphical user interface or an internet browser), or any combination thereof. The components of the system may be connected via any form of digital data communication or media (e.g., a communication network). Examples of communication networks include, for example, LANs, WANs, and computers and networks that form the Internet.

[0532] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers that have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of user interaction) may be received at the server from the client device.

[0533] A system of one or more computers may be configured to perform the action by means of software, firmware, hardware, or a combination thereof installed on the system (which, in operation, cause the system to perform a specific action). A system of one or more computer programs may be configured to perform the action by means of instructions contained in which, when executed by a data processing device, the device performs a specific action.

[0534] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Specific features described herein in the context of a single embodiment may also be implemented in combination within that single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in some combination and even initially claimed, in some cases, one or more features from said combination may be removed from the claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0535] Similarly, although operations are depicted in a specific order in the diagrams, this should not be construed as requiring the execution of such operations in the shown specific order or in sequential order, or performing all illustrated operations to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring this separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0536] Therefore, specific embodiments of the subject matter have been described. Other embodiments exist within the scope of the appended claims. In some cases, the actions stated in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0537] Several embodiments of the present invention have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the invention.

[0538] Various aspects of the invention can be understood from the following exemplary embodiments (EEE):

[0539] EEE 1. A method for leveling audio, comprising:

[0540] A leveling unit comprising one or more electronic circuits receives reference audio data, the reference audio data comprising representations of channel signals from multiple channel signal sources;

[0541] The leveling unit receives target level data, which specifies the target level for each sound source;

[0542] The leveling unit determines, based on the reference audio data, a cost function for rescaling the audio signal to the target level according to the corresponding gain; and

[0543] The corresponding gain to be applied to each of the channel signals in the live audio data is calculated by minimizing the cost function.

[0544] EEE 2. The method according to EEE 1, wherein the representation of the channel signal comprises an original channel signal or a processed channel signal, the processed channel signal comprising a channel signal that has been processed by a noise reduction unit, an equalizer, a dynamic range correction unit, or a sound source separator.

[0545] EEE 3. The method according to EEE 1 or EEE 2, which includes determining the corresponding correlation between each pair of the channel signal sources by the leveling unit.

[0546] EEE 4. A method for translating audio, comprising:

[0547] Reference audio data of a sound source is received by a translater comprising one or more electronic circuits, the sound source comprising one or more sources designated as one or more translateable sources and one or more sources designated as one or more non-translatable sources.

[0548] Receive the audio channel signal of the event in which the sound source is playing;

[0549] A cost function is determined based on the reference audio data. The cost function has a translation position for each channel signal as a variable. The cost function includes a first component to represent the imbalance between the left and right channels, a second component to represent the one or more translationable sources, and a third component to represent the one or more non-translatable sources among the sound sources.

[0550] The corresponding translation position of each channel signal is determined by minimizing the cost function; and

[0551] The translation position is applied to the channel signal to place the sound source of the event between the left and right sides of the stereo stage for audio effects output to the stereo reproduction system.

[0552] EEE 5. The method according to EEE 4, wherein the translation position comprises at least one of the translation angles or the ratio between the left and right channels, and wherein the stereo reproduction system comprises headphones or speakers.

[0553] EEE 6. A method for leveling and translating audio, comprising:

[0554] The reference audio data is received by a leveling and translation unit comprising one or more electronic circuits, the reference audio data comprising a representation of channel signals from multiple channel signal sources recorded during rehearsals of one or more sound sources.

[0555] The leveling and translation unit receives target level data, which specifies the target level for each sound source.

[0556] The leveling and translation unit receives live audio data, which includes recorded or real-time signals from one or more sound sources playing at the location of the live event.

[0557] The leveling and translation unit determines a joint cost function for leveling the live audio data and translating the live audio data based on the reference audio data. The joint cost function has a first component for leveling the live audio data and a second component for translating the live audio data. The first component is based on the target level data, and the second component is based on a first representation of the imbalance between the left and right channels, a second representation of the translationable sources among the sound sources, and a third representation of the non-translatable sources among the sound sources.

[0558] The corresponding gain to be applied to each of the channel signals and the corresponding translation position of each channel signal are calculated by minimizing the joint cost function; and

[0559] The gain and shift position are applied to the signal of the live audio data of the event to level the sound source in the live audio data and place the sound source in the live audio data between the left and right sides of the stereo stage for audio effects output to a storage device or stereo reproduction system.

[0560] EEE 7. The method described in EEE 6, wherein each level is an energy level or a loudness level.

[0561] EEE 8. A method for determining audio levels, comprising:

[0562] Reference audio data is received by an estimator comprising one or more electronic circuits, the reference audio data comprising channel signals representing one or more sound sources played during rehearsal;

[0563] The estimator calculates the corresponding level for each sound source in each microphone based on the reference audio data;

[0564] Determining the grade difference between the live audio data and the reference audio data includes comparing each sound source as represented in the live audio data with the corresponding grade of the sound source as represented in the reference audio data.

[0565] A cost function for the corresponding level of each sound source is determined based on the difference;

[0566] The corresponding level is determined by minimizing the cost function; and

[0567] The level is provided as input to the audio or video processor.

[0568] EEE 9. The method according to EEE 8, comprising:

[0569] The estimator calculates the corresponding level for each sound source in each of the multiple frequency bands, wherein

[0570] The cost function includes the sum of the costs for each sound source across all frequency bands;

[0571] The corresponding level is determined in each frequency band.

[0572] EEE 10. A method for audio equalization, comprising:

[0573] An equalizer, comprising one or more electronic circuits, receives audio data containing signals from multiple sound sources.

[0574] The equalizer maps the corresponding signal for each sound source to the excitation in each frequency band.

[0575] Determine the demand value for each source-band pair in the source-band pair list, where each source-band pair represents a sound source and a frequency band. The demand value indicates the relative importance of the sound source represented in the pair relative to other sound sources and other frequency bands in the frequency band of the pair after equalization, and the masking level of one or more other sound source pairs represented in the pair.

[0576] Iteratively equalize the signals of the sound sources represented by the source-band pairs with the highest demand values ​​in the list and remove the equalized source-band pairs from the list until the highest demand value of the remaining source-band pairs is below a threshold; and

[0577] The equalized signal is provided for playback on one or more speakers.

[0578] EEE 11. The method according to EEE 10, wherein the demand value is the product of one or more values ​​representing the relative importance and one or more values ​​representing the masking level of the sound source.

[0579] EEE 12. A method for segmenting video content, comprising:

[0580] Audio signals are received by a segmentation unit comprising one or more electronic circuits;

[0581] The segmentation units construct a novel index about the audio signal across time;

[0582] The cutting time for the next cut is determined based on the peak value in the novelty index;

[0583] Cut the video content at the cut time; and

[0584] The cut video content is provided as a new video clip to a storage device or to one or more end-user devices.

[0585] EEE 13. The method according to EEE 12, wherein determining the shearing time includes:

[0586] The segment length is determined based on the average cut length, and the segment length corresponds to the length of the audio segment; and

[0587] The cutting time is determined based on the fragment length.

[0588] EEE 14. The method according to EEE 13, wherein determining the shearing time based on the segment length includes:

[0589] Determine the sum of the novelty indices over the time since the last cut; and

[0590] After determining that the sum is above the novelty threshold, the cutting time is immediately determined according to the time from the time when the sum of the novelty index satisfies the novelty threshold to the time of the next cutting, and the randomness of the cutting time is averaged over the segment length.

[0591] EEE 15. A method for synchronizing audio, comprising:

[0592] Receive audio signals from multiple microphones;

[0593] Determine the corresponding quality value of the correlation between each pair of audio signals and assign the quality value to a mapping vector;

[0594] Iteratively determining a series of delays and inserting the delays into the mapping vector, wherein iteratively determining the series of delays includes iteratively aligning and downmixing a pair of audio signals having the highest quality value; and

[0595] After the iteration is completed, the audio signal is immediately synchronized using the corresponding delay in the mapping vector according to the order in which the delay is inserted into the mapping vector.

[0596] EEE 16. A method for noise reduction, comprising:

[0597] Reference audio data, which includes vocal tract signals recorded during the silent rehearsal phase, is received by a noise reduction unit comprising one or more electronic circuits.

[0598] The noise level in each channel signal of the reference audio data is estimated by the noise estimator of the noise reduction unit;

[0599] Receive live performance data, which includes acoustic signals recorded during the event, in which one or more instruments that were silent during the rehearsal phase are played during the event;

[0600] The noise reduction unit individually reduces the corresponding noise level in each channel signal of the live performance data, including applying a corresponding suppression gain to the channel signal in the live performance data immediately after determining that the difference between the noise level in each channel signal of the live performance data and the estimated noise level meets a threshold; and

[0601] After reducing the noise level, the audio signal is provided to downstream devices for further processing, storage, or distribution to one or more end-user devices.

[0602] EEE 17. The method according to EEE 16, wherein:

[0603] Estimating the corresponding noise level in each channel signal of the reference audio data across multiple frequency bands.

[0604] In the frequency band, the corresponding noise level in each channel signal of the live performance data is reduced, and

[0605] The estimation and reduction are performed based on noise reduction parameters including the threshold, slope, attack time, decay time, and octave size.

[0606] EEE 18. A method comprising:

[0607] The system receives reference audio data from one or more channel signal sources, the reference audio data including acoustic information of one or more sound sources that are individually played during rehearsals;

[0608] The server system receives one or more channel signals of a performance event from one or more channel signal sources, each channel signal coming from a corresponding channel signal source and including audio signals from the one or more sound sources played at the performance event.

[0609] The server system mixes the one or more channel signals, the mixing including automatically adjusting one or more audio attributes of one or more sound sources of the performance event based on the reference audio data;

[0610] The mixed recording of the performance event is provided from the server system to a storage device or to multiple end-user devices; and

[0611] The server system provides the one or more channel signals of the performance event and a separate file describing at least one or more audio attributes from the server system to the storage device.

[0612] EEE 19. A method comprising:

[0613] The system receives reference audio data from one or more channel signal sources, the reference audio data including acoustic information of one or more sound sources played individually.

[0614] The server system receives one or more channel signals of a performance event from one or more channel signal sources, each channel signal coming from a corresponding channel signal source and including audio signals from the one or more sound sources played at the performance event.

[0615] The server system mixes the one or more channel signals, the mixing including automatically adjusting one or more audio attributes of one or more sound sources of the performance event based on the reference audio data; and

[0616] The mixed recording of the performance event is provided from the server system to a storage device or to multiple end-user devices.

[0617] EEE 20. The method according to any of the aforementioned EEEs, wherein:

[0618] Each audio signal source contains a microphone or sound signal generator with signal output.

[0619] Each sound source is a singer, an instrument, or a synthesizer.

[0620] The server system includes one or more computers connected to the one or more channel signal sources via a communication network, and

[0621] The one or more channel signal sources and the one or more sound sources have the same acoustic arrangement in the rehearsal and in the performance event.

[0622] EEE 21. The method according to any of the aforementioned EEEs, wherein:

[0623] The one or more channel signals include a first channel signal from a first channel signal source among the one or more channel signal sources and a second channel signal from a second channel signal source among the one or more channel signal sources, and

[0624] The method includes synchronizing the first channel signal and the second channel signal in the time domain by the server system.

[0625] EEE 22. The method according to any of the preceding EEEs, comprising: separating a first sound source and a second sound source from the one or more channel signals, including separating the first sound source and the second sound source from a plurality of sound sources represented in the one or more channel signals, wherein the one or more channel signals include a first signal representing the first sound source and a second signal representing the second sound source.

[0626] EEE 23. The method according to any of the preceding EEEs, wherein the mixing includes leveling the first sound source and the second sound source by the server system and translating the first sound source and the second sound source.

[0627] EEE 24. The method according to EEE 23, wherein leveling the first sound source and the second sound source includes increasing or decreasing the gain of the one or more sound sources according to a corresponding energy level of each sound source, each corresponding energy level being determined by the server system based on the reference audio data.

[0628] EEE 25. The method according to any of the preceding EEEs, wherein the reference audio data includes at least one of the following:

[0629] The signal of each sound source played in accordance with the first level designated as the lower level and the second level designated as the higher level; or

[0630] The signal of each sound source is played at a single level.

[0631] EEE 26. The method according to any of the preceding EEEs, comprising determining a corresponding gain for each sound source in the event based on the reference audio data, wherein determining the corresponding gain includes, for each sound source:

[0632] Receive input at the specified target level;

[0633] Determine the corresponding level of the signal in the reference audio data; and

[0634] Each corresponding gain is determined based on the difference between the level of the signal in the reference audio data and the target level.

[0635] EEE 27. The method according to any of the preceding EEEs, wherein mixing the one or more channel signals includes adjusting the gain of the one or more channel signals, the signals from the one or more sound sources, or both, based on input from a mixer device logged into the server system.

[0636] EEE 28. The method according to any of the preceding EEEs, comprising performing video editing in response to the event, wherein performing the video editing includes:

[0637] The video editor of the server system receives video and audio data of the event. The video data includes video of sound sources visibly located at different locations in the event, and the audio data includes the energy levels of the sound sources.

[0638] The signal of the first sound source represented in the audio data, determined based on the audio data, indicates that the first sound source is playing at a level that is higher than the level of other sound sources represented in the audio data by a threshold amount.

[0639] Determine the location of the first sound source in the video data;

[0640] Determine a portion of the video data corresponding to the location of the first sound source; and

[0641] The audio data and a portion thereof are synchronously provided to the storage device or to the end-user device.

[0642] EEE 29. The method according to EEE 28, wherein determining the location of the sound source in the video data comprises:

[0643] The translation position of the first sound source is determined based on the audio data; and

[0644] The translation position of the first sound source is designated as the position of the sound source in the video data.

[0645] EEE 30. The method according to EEE 28, wherein determining the location of the sound source in the video data includes using facial tracking or instrument tracking to determine the location of the sound source.

[0646] EEE 31. The method according to any of the aforementioned EEEs, comprising:

[0647] A command is provided from the server system to the one or more channel signal sources based on the one or more channel signals, the command being configured to adjust the recording parameters of the one or more channel signal sources, the recording parameters including at least one of gain, compression type, bit depth, or data transfer rate.

Claims

1. A method for video editing, comprising: The server system receives rehearsal data from one or more recording devices. The rehearsal data includes rehearsal video data and rehearsal audio data, and the rehearsal data represents the rehearsal of the event performed by one or more performers of the event. The server system identifies the corresponding image of each of the one or more performers based on the rehearsal video data; Based on the rehearsal audio data, determine the corresponding sound attributes associated with each identified image; The server system receives on-site data, including on-site video data and on-site audio data of the event, from one or more recording devices; The server system determines the prominence of each performer based on the identified image and associated audio attributes in the following manner: The loudness level of the performers is determined based on the rehearsal audio data; Based on the live audio data, determine the loudness level of each performer over time relative to the corresponding loudness level determined based on the rehearsal audio data; The relative prominence of each performer over time is determined by comparing their corresponding determined relative loudness levels. The on-site video data and the on-site audio data are edited according to one or more editing rules, the editing rules being based on corresponding salience and the identified image; as well as Providing edited on-site video data and edited on-site audio data for playback includes storing the edited on-site video data and edited on-site audio data in association on a storage device, or streaming the edited on-site video data and edited on-site audio data. The server system described therein includes one or more computer processors.

2. The method according to claim 1, wherein the corresponding sound attribute includes sound type and loudness level.

3. The method according to claim 1, wherein determining the loudness level based on the rehearsal audio data comprises: Based on the rehearsal audio data, a first-level voice and a second-level voice for each performer are determined, wherein the first-level voice is lower than the second-level voice. as well as Calculate the weighted average between the first and second levels.

4. The method according to claim 1, further comprising: Based on the live video data, it is determined that the movement of the first performer exceeds the movement of the other performers by at least a threshold. Based on the amount of movement, the first performer is determined to be the prominent performer; as well as The one or more prominent performers are framed by enlarging or cropping one or more images of the prominent performer.

5. The method according to claim 1, wherein editing the on-site video data and the on-site audio data comprises: The speed and rhythm of the event are determined based on the on-site audio data; as well as The live video data is cut according to the speed and rhythm.

6. The method according to claim 1, wherein editing the on-site video data and the on-site audio data comprises: Determine whether at least one performer has started or stopped playing; as well as In response, the live video data is cut.

7. The method according to claim 1, wherein editing the on-site video data and the on-site audio data comprises: In response to determining, based on the live video data, that the elapsed time since the display of full-frame footage of all performers exceeds a threshold time, the live video data is cut, wherein the threshold time is determined based on the number of cuts or the duration.

8. The method according to claim 1, wherein editing the on-site video data and the on-site audio data comprises: Determine that the elapsed time since at least one performer appeared in the edited live video data is less than a threshold time, wherein the threshold time is determined based on the number of cuts or the duration; as well as It is stipulated that during the period when the elapsed time is less than the threshold time, the at least one performer will not be shown in the next edited cut.

Images