Media synchronous playback and reverse control system of mobile terminal

Abstract

The invention discloses a media synchronous playback and reverse control system and method of a mobile terminal, and belongs to the technical field of multimedia interaction. The system comprises a media server, a main player and a mobile terminal which are interconnected through a synchronization and control signaling management engine. The media server is responsible for media analysis and content distribution; the main player renders the main media stream and provides a parameterized effect interface; and the mobile terminal synchronously presents the enhanced information flow, and converts the multi-mode sensor data into a standardized reverse control instruction through an interactive logic mapper. According to the invention, bidirectional synchronous and differentiated content presentation between the mobile terminal and the main player is realized, and a user is innovatively allowed to parametrically control the visual or audio effect of the main playing content in real time through sensors of the mobile terminal, such as a gyroscope, a microphone and a camera, so that brand new immersive interactive experience is created; and meanwhile, the method has the advantages of low cost and high universality.

Description

Technical Field

[0001] This invention relates to the field of multimedia playback and interactive control technology, specifically to a media synchronous playback and reverse control system for a mobile terminal. Background Technology

[0002] With the widespread adoption of smart mobile devices (such as smartphones and tablets) and the rapid development of streaming media technology, users have placed higher demands on how they consume multimedia content. Traditional screen mirroring technologies (such as DLNA, AirPlay, and Miracast) enable the one-way transmission of media content from mobile devices to large-screen displays (such as smart TVs and projectors) for playback. However, existing technologies have the following main drawbacks: 1. Limited Functionality: Primarily enables "one-way mirroring" or "pushing" of media content from a mobile device to a large-screen display for playback. After screen mirroring, the mobile device typically functions only as a simple remote control (play / pause, volume adjustment) or enters a black / locked screen state, thus wasting its screen resources.

[0003] 2. Disjointed Interaction: The content between the mobile device and the display is a simple copy, lacking two-way, in-depth interaction. The mobile device cannot acquire and display enhanced information that complements but differs from the main screen.

[0004] 3. No reverse control: It cannot convert the rich sensor data (such as gyroscope, touch screen, microphone, camera) and computing power on the mobile terminal into real-time, creative control signals for the content played on the main display.

[0005] Meanwhile, existing second-screen applications (such as the "TV assistant" in some video apps) can display progress bars, program introductions, or act as keyboards on mobile devices. However, their functions are limited, serving only as auxiliary controls and unable to achieve real-time parametric reverse control of the main screen's visual and audio content (for example, using the phone's tilt angle to control the 3D rotation of the video screen, or using the phone's microphone to collect ambient sounds to affect the background music).

[0006] Existing visual media players typically limit interaction to a single device (such as a computer keyboard and mouse) or use dedicated hardware controllers. There is a lack of solutions that leverage the widespread availability of mobile devices as low-cost, multifunctional, and personalized "rich interactive controllers."

[0007] In view of this, in-depth research was conducted on the above issues, which led to the creation of this case. Summary of the Invention

[0008] The purpose of this invention is to provide a media synchronization playback and reverse control system for mobile terminals, so as to solve the problems mentioned in the background art of the prior art where mobile terminals have limited functions, fragmented interaction, and are unable to achieve creative reverse control in screen projection scenarios.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A media synchronization playback and reverse control system for a mobile terminal, comprising: The system includes a media server module, a main player module, a mobile terminal module, and a synchronization and control signaling management engine. The synchronization and control signaling management engine is used to maintain network connections and time synchronization between modules; The media server module includes an intelligent media analyzer and a content and instruction distributor; The main player module includes a main rendering engine and a parametric effects interface; The mobile terminal module includes an enhanced information renderer, a multimodal sensor collector, and an interactive logic mapper.

[0010] Preferably, the intelligent media analyzer is used to preprocess and analyze the original media file to extract at least one metadata from timecode, chapter markers, keyframes, audio spectrum, and sentiment markers; The content and instruction distributor is used to distribute different media data streams to the main player and the mobile terminal based on the metadata and control instructions from the mobile terminal.

[0011] Preferably, the parameterized effects interface exposes a set of audiovisual parameters that can be adjusted in real time, including at least one of color filters, particle motion speed, and audio reverberation intensity; The interface is configured to receive reverse control instructions from the content and instruction dispatcher and apply the instructions to the media content currently being rendered.

[0012] Preferably, the interactive logic mapper provides a user-configurable mapping scheme to convert raw sensor data from the multimodal sensor collector into standardized control commands; The sensor data includes at least one of the following: touch gestures, gyroscope attitude, accelerometer data, microphone input, and camera footage.

[0013] Preferably, the mapping scheme supports a graphical configuration interface, allowing users to associate sensor input sources with controlled effect parameters and set the mapping curve type.

[0014] A method for synchronized media playback and reverse control includes the following steps: Connection and synchronization establishment phase: The mobile terminal discovers and connects to the main player and media server through local area network scanning or QR code recognition, and completes time synchronization based on the media server's clock. Two-way synchronous playback phase: The main player plays the main media stream, and the mobile terminal simultaneously receives and renders the enhanced data stream; Reverse control phase: The mobile terminal collects sensor data, generates control commands through the interactive logic mapper, and forwards them to the main player through the content and command distributor to adjust its playback effect in real time.

[0015] Preferably, the reverse control phase further includes: Control commands are encapsulated in JSON or Protocol Buffers format and include at least the command type, target parameters, parameter values, and high-precision media timestamps; The instructions are transmitted with low latency via WebSocket or RTC channels to ensure that the control effect takes effect at the corresponding media moment.

[0016] Preferably, the method further includes a dynamic role switching step: The mobile terminal can dynamically switch between enhanced information display mode, reverse controller mode and traditional remote control mode or run multiple modes simultaneously.

[0017] Preferably, the synchronization establishment phase adopts a time synchronization mechanism based on NTP-like algorithm and real-time communication technology, with a target audio-visual synchronization error of less than 40 milliseconds.

[0018] A mobile terminal, comprising: An enhanced information renderer is used to receive and display enhanced data streams from a media server; Multimodal sensor data acquisition unit, used to acquire sensor data from equipment in real time; An interactive logic mapper is used to map the sensor data into control commands for the content played by the main player; A communication module is used to establish a connection with the media server and the main player, and to transmit the control commands. The mobile terminal is configured to execute the media synchronous playback and reverse control method.

[0019] Compared with the prior art, the present invention aims to provide a system and method to achieve: 1. Two-way synchronization and enhanced playback: Media content is played simultaneously on mobile terminals and the main display, but with differentiated and complementary information (such as playing the full movie on TV and simultaneously displaying director's comments, character backgrounds, or visual sound spectrum on the mobile phone).

[0020] 2. Mobile-to-Mobile Reverse Deep Control: Transform the mobile terminal into a powerful physical interaction controller, utilizing its multimodal input capabilities to control the visual presentation, audio effects, or playback logic of content played on the main display in real time, creating a completely new interactive experience.

[0021] 3. Low latency and high synchronization: Ensures extremely low latency and high-precision synchronization of media playback and control command transmission between master and slave devices, guaranteeing a smooth experience.

[0022] The beneficial effects of the media synchronization playback and reverse control system of this mobile terminal include: 1. Efficient resource utilization and enhanced experience: Transforming mobile terminals from "idle remote controls" into "enhanced interactive screens and intelligent controllers" greatly improves the value of the second screen and the immersive experience of the user.

[0023] 2. Creating a brand-new interactive paradigm: For the first time, it systematically proposed and implemented real-time parameterized reverse control of the main playback content using multimodal input from mobile terminals, opening up new interactive possibilities for games, education, art exhibitions, and family entertainment.

[0024] 3. Low cost and high accessibility: Utilizing existing and widely available smartphones and tablets as controllers eliminates the need for users to purchase dedicated hardware, lowering the barrier to entry for users.

[0025] 4. Flexible architecture: The modular design allows the system to adapt to devices with different performance levels, and the media server can be cloud-based to support more complex media analysis and content distribution. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the data flow and core interaction logic between the modules of the system of this invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figure 1 The present invention provides the following technical solutions: The core architecture of this invention's system includes: a media server module, a main player module, a mobile terminal App module, and a synchronization and control signaling management engine connecting each module. For example... Figure 1 The flowchart shown clearly illustrates the data flow and core interaction logic between the various modules of the system.

[0029] I. Detailed Explanation of System Components 1. Media Server Module: It can exist independently (such as a NAS or cloud server) or be integrated into the main playback device (such as a smart TV or computer).

[0030] Intelligent Media Analyzer: Preprocesses and analyzes raw media files (video / audio) to extract metadata such as timecode, chapter markers, subtitles, keyframes, audio spectrum, and sentiment markers.

[0031] Content and instruction distributor: As a data hub, it receives control instructions from the mobile device and distributes different data streams to the main player and mobile terminal based on the instructions and media metadata.

[0032] 2. Main player module (running on the display device): Main rendering engine: Receives the main media stream (usually high bitrate raw video / audio) from the server, decodes and renders it, and outputs it to the main display.

[0033] Parametric effects interface: Exposes a set of audiovisual parameters that can be adjusted in real time (such as color_filter_hue, particle_speed, audio_reverb_mix), receives reverse control instructions from the server, and applies them immediately to the current rendering.

[0034] 3. Mobile Terminal App Module: Enhanced information renderer: Receives enhanced data streams (such as visualized graphics, text comments, and interactive game interfaces) from the server and renders them on the mobile terminal screen, forming a "second perspective" that is synchronized with but different from the main screen content.

[0035] Multimodal sensor acquisition unit: continuously collects data from terminal sensors, including touch gestures, screen pressure, gyroscope / accelerometer (attitude), microphone audio, camera footage, geolocation, etc.

[0036] Interactive Logic Mapper: Provides user-defined or preset mapping schemes to convert raw sensor data into semantic control commands. For example, "tilt the phone up and down" is mapped to "control the vertical perspective distortion of the video image"; "blow into the phone microphone" is mapped to "increase the intensity of snow particles in the image".

[0037] 4. Synchronization and Control Signaling Management Engine: Responsible for maintaining network connections between all devices (Wi-Fi P2P, Bluetooth, LAN).

[0038] An adaptive heartbeat mechanism and timestamp synchronization algorithm are used to ensure that all devices operate on the same media timeline, thus offsetting network jitter.

[0039] When transmitting reverse control commands, the commands are accompanied by a high-precision timestamp to ensure that the control effect takes effect at the precise media moment.

[0040] II. Main Work Processes and Methods 1. Connection and synchronization establishment phase: The mobile app discovers and connects to the main player and media server by scanning a local area network or identifying a QR code.

[0041] The three parties negotiated through the signaling engine, using the media server's clock as a reference, to complete the initial time synchronization.

[0042] 2. Two-way synchronous playback stage: The user initiates playback on the main media player or mobile device. The media server begins pushing content.

[0043] The main player receives and plays the main media stream (video).

[0044] The mobile terminal simultaneously receives and plays the enhanced data stream. For example: While playing music, the phone displays real-time generated, interactive music visualizations.

[0045] While watching a movie, your phone will display an interactive map of the current scene, a chart of character abilities, or behind-the-scenes interview clips.

[0046] 3. Reverse Control Stage (Core Innovation): The sensor data from the mobile terminal is converted into standardized control commands by the interactive logic mapper (such as {"cmd":"set_visual_param","name":"wind_strength","value":0.8","timestamp":123.456}).

[0047] The instruction is sent to the content and instruction distributor of the media server with low latency via the signaling engine.

[0048] The distributor forwards instructions to the parameterized effects interface of the main player.

[0049] The main player adjusts the visual effects or audio processing parameters of the current screen in real time according to instructions. For example, if a user shakes their phone, the visual particles of music on the TV screen will scatter; if a user uses their phone's camera to capture colors in reality, the overall tone of the TV screen will change accordingly.

[0050] 4. Dynamic role switching: The system supports mobile terminals to dynamically switch between three modes: "enhanced information display", "reverse controller" and "traditional remote control", or to perform multiple functions simultaneously.

[0051] In summary, the key points of this invention include: 1. Synchronization Protocol: A custom signaling protocol based on WebSocket can be used. Media synchronization uses streaming media protocols such as RTSP / RTMP, or adaptive bitrate distribution based on HTTP-FLV, DASH / HLS. Synchronization accuracy is guaranteed by combining an NTP-like algorithm with RTC (Real-Time Communication) technology, with a target audio-visual synchronization error of less than 40ms.

[0052] 2. Reverse control instruction encoding: The instructions are in lightweight JSON or Protocol Buffers format and include instruction type, target parameters, parameter values, high-precision media timestamps and sequence numbers.

[0053] 3. Mapping Configuration: The mobile app provides a graphical interface that allows users to drag "sensor input sources" (such as the Y-axis of the accelerometer) onto "controlled effect parameters" (such as particle gravity coefficients) and set the mapping curve (linear, exponential, logarithmic).

[0054] The following is an example of an application of a home interactive music playback system based on a local media server: 1. System Hardware Environment Main player: A smart TV that supports internet access (running the main player software).

[0055] Media server: A home NAS (Network Attached Storage) with media server software deployed.

[0056] Mobile terminal: A smartphone (with the dedicated app for this invention installed).

[0057] Network environment: Home Wi-Fi LAN (802.11ac standard).

[0058] 2. System Module Configuration and Initialization 2.1 Media Server Module Configuration The media server software running on the NAS device includes the following components: Intelligent Media Analyzer: Developed using FFmpeg and librosa libraries, this preprocesses music files (such as MP3 and FLAC formats) stored on the NAS. The analysis includes: Audio spectrum data (extracted via short-time Fourier transform).

[0059] Beat detection and beat timestamp.

[0060] Pitch, loudness, and timbre characteristics.

[0061] Song structure markers (intro, verse, chorus, interlude).

[0062] Content and instruction dispatcher: Developed using Node.js, it implements the following functions: Control the connection by listening through a WebSocket Server.

[0063] The main audio stream is pushed to the main player via the HTTP-FLV protocol.

[0064] Enhanced data streams (including spectrum arrays, beat markers, song lyrics, and background information) are pushed to mobile devices via WebSocket Data Channel.

[0065] Receive and forward JSON-formatted control commands from mobile terminals.

[0066] 2.2 Main Player Module Configuration The main media player software (developed based on Android TV) running on the smart TV includes: Main rendering engine: Based on ExoPlayer, it receives HTTP-FLV audio streams from the server, decodes them, and outputs them through the TV speakers.

[0067] Parameterized effects interface: Provided as a dynamic link library, exposing the following parameters that can be adjusted in real time: Visual parameters: Particle velocity (0.0~5.0) Angle of particle motion direction (0~360 degrees) Hue shift (0.0~1.0) Waveform amplitude coefficient (0.5~3.0) Audio effect parameters Mixing ratio (0.0~1.0) Low-pass filter cutoff frequency (20~20000Hz) This interface interacts with the main rendering engine via JNI, ensuring that parameter changes take effect in the next frame rendering cycle.

[0068] 2.3 Mobile Terminal App Module Configuration The smartphone app (developed based on Android) includes: Enhanced Information Renderer: Developed based on OpenGL ES 3.0, it receives enhanced data streams pushed by the server and renders the following content: Real-time audio spectrum waterfall plot.

[0069] A visual particle system that pulsates with the beat.

[0070] The lyrics and song background story cards are displayed in sync.

[0071] Multimodal sensor data acquisition unit: Acquires the following sensor data at a frequency of 100Hz: Gyroscope (three-axis angular velocity).

[0072] Accelerometer (triaxial accelerometer).

[0073] Touchscreen (multi-touch position and pressure).

[0074] Microphone (ambient sound amplitude).

[0075] Interactive logic mapper: Provides both preset and custom mapping schemes. Preset Plan A (Tilt Control): javascript / / Map device pitch angle to particle direction function mapTiltToParticleDirection(pitch) { / / pitch: -90 degrees (phone vertically downwards) to +90 degrees (phone vertically upwards) return (pitch + 90) * 2; / / Map to 0~360 degrees } Preset Plan B (Blowing Control): javascript / / Map microphone peak volume to particle velocity function mapBlowToParticleSpeed(micPeak) { / / micPeak: 0.0~1.0 normalized value return Math.min(micPeak * 10, 5.0); } Customizable configuration interface: Users can establish the association between sensor sources and effect parameters by dragging and dropping.

[0076] 2.4 Synchronization and Control Signaling Management Engine A hybrid synchronization strategy is adopted: Initial synchronization: Based on the improved NTP algorithm, with the media server as the time source, the client and server exchange timestamps three times, calculate network latency and clock deviation, and synchronize the client clock to the server clock reference, with the error controlled within ±10ms.

[0077] Continuous synchronization: An adaptive heartbeat mechanism is used to exchange synchronization packets every 500ms, dynamically adjusting the playback buffer to offset network jitter.

[0078] Command transmission channel: Control commands are transmitted through a separate WebSocket channel, separate from the data stream, to ensure priority transmission of control commands and low latency (target latency <50ms).

[0079] 3. Detailed Workflow 3.1 Connection and Synchronization Establishment Phase The user opens the main media player software on the smart TV. After the software starts, it enters the "waiting for connection" state and displays a QR code on the TV screen.

[0080] Users open the mobile app, click the "Scan to Connect" button, and scan the QR code on the TV.

[0081] The QR code encodes the TV's IP address and service port.

[0082] The app connects to the TV via WebSocket, and the TV returns the address of the media server to the app.

[0083] The app then connects to the media server's WebSocket service port.

[0084] After the three-way connection is established, initiate time synchronization negotiation: The phone calculates round-trip delay and clock skew, and adjusts the local media clock accordingly.

[0085] 3.2 Two-way synchronous playback stage Users select a song from the NAS on the mobile app and click play.

[0086] The media server received a playback command: The intelligent media analyzer performs real-time analysis, extracting spectrum data (50 frames per second, 1024 frequency points per frame) and beat information.

[0087] Content and instruction dispatcher: The original audio is converted into an HTTP-FLV stream and then streamed to the TV.

[0088] The augmented data (spectrum + beat markers) is pushed to the mobile phone via WebSocket.

[0089] The TV receives and plays the audio stream, while simultaneously rendering a basic visual background (such as gradient colors) according to the song's progress.

[0090] Mobile phone synchronously receives enhanced data: Displays a real-time spectrum waterfall plot.

[0091] During the chorus, a background card for the song's creation pops up at the bottom of the screen (such as "This song was created in 1971 and expresses a longing for a peaceful world...").

[0092] 3.3 Reverse Control Phase (Core Process) Users select a preset "tilt control" mapping scheme.

[0093] The user tilted the phone 30 degrees to the left.

[0094] The multimodal sensor data acquisition unit reads gyroscope data: text Gyroscope: { x: 0.0, y: -0.5, z: 0.0} / / The y-axis represents the rate of change of pitch angle. The interactive logic mapper converts raw data into standardized control commands: json { "cmd": "set_visual_param", "target": "particle_direction", "value": 120.5, / / Calculated using mapTiltToParticleDirection "timestamp": 1625097605.456, / / High-precision media timestamp "seq": 12345 } Instructions are sent via WebSocket to the media server's content and instruction dispatcher.

[0095] After verifying the validity of the timestamp, the distributor immediately forwards the instruction to the main TV player.

[0096] The TV's main player's parametric effects interface receives the instruction and adjusts the particle system's motion direction to 120.5 degrees in the next rendering frame (approximately 16.7ms later).

[0097] The visualized particle stream on the television screen immediately drifted to the upper left (corresponding to a 120.5-degree direction).

[0098] 3.4 Dynamic Role Switching Users can switch control modes at the bottom of the App interface: Enhanced Information Mode: Displays only spectrum and song information; sensor data is not transmitted.

[0099] Reverse control mode: Sensor data is mapped to control commands, and the enhanced information display is reduced to a floating window.

[0100] Remote control mode: Displays traditional playback control buttons (play / pause, previous / next track, volume), with sensor-controlled pause.

[0101] Contents not described in detail in this specification are prior art known to those skilled in the art. Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A media synchronization playback and reverse control system for a mobile terminal, characterized in that, include: The system includes a media server module, a main player module, a mobile terminal module, and a synchronization and control signaling management engine. The synchronization and control signaling management engine is used to maintain network connections and time synchronization between modules; The media server module includes an intelligent media analyzer and a content and instruction distributor; The main player module includes a main rendering engine and a parametric effects interface; The mobile terminal module includes an enhanced information renderer, a multimodal sensor collector, and an interactive logic mapper.

2. The media synchronization playback and reverse control system for a mobile terminal according to claim 1, characterized in that: The intelligent media analyzer is used to preprocess and analyze the original media files, extracting at least one metadata from timecode, chapter markers, keyframes, audio spectrum, and sentiment markers; The content and instruction distributor is used to distribute different media data streams to the main player and the mobile terminal based on the metadata and control instructions from the mobile terminal.

3. The media synchronization playback and reverse control system for a mobile terminal according to claim 1, characterized in that: The parametric effects interface exposes a set of audiovisual parameters that can be adjusted in real time, including at least one of color filters, particle motion speed, and audio reverberation intensity. The interface is configured to receive reverse control instructions from the content and instruction dispatcher and apply the instructions to the media content currently being rendered.

4. The media synchronization playback and reverse control system for a mobile terminal according to claim 1, characterized in that: The interactive logic mapper provides a user-configurable mapping scheme to convert raw sensor data from the multimodal sensor acquisition unit into standardized control commands. The sensor data includes at least one of the following: touch gestures, gyroscope attitude, accelerometer data, microphone input, and camera footage.

5. The media synchronization playback and reverse control system for a mobile terminal according to claim 1, characterized in that: The mapping scheme supports a graphical configuration interface, allowing users to associate sensor input sources with controlled effect parameters and set the mapping curve type.

6. A method for media synchronization playback and reverse control using the system described in any one of claims 1 to 5, characterized in that, Includes the following steps: Connection and synchronization establishment phase: The mobile terminal discovers and connects to the main player and media server through local area network scanning or QR code recognition, and completes time synchronization based on the media server's clock. Two-way synchronous playback phase: The main player plays the main media stream, and the mobile terminal simultaneously receives and renders the enhanced data stream; Reverse control phase: The mobile terminal collects sensor data, generates control commands through the interactive logic mapper, and forwards them to the main player through the content and command distributor to adjust its playback effect in real time.

7. The media synchronization playback and reverse control method of the system according to claim 6, characterized in that: The reverse control phase further includes: Control commands are encapsulated in JSON or Protocol Buffers format and include at least the command type, target parameters, parameter values, and high-precision media timestamps; The instructions are transmitted with low latency via WebSocket or RTC channels to ensure that the control effect takes effect at the corresponding media moment.

8. The media synchronization playback and reverse control method of the system according to claim 6, characterized in that: The method also includes a dynamic role switching step: The mobile terminal can dynamically switch between enhanced information display mode, reverse controller mode and traditional remote control mode or run multiple modes simultaneously.

9. The media synchronization playback and reverse control method of the system according to claim 6, characterized in that: The synchronization establishment phase adopts a time synchronization mechanism based on NTP-like algorithm and real-time communication technology, with a target audio-visual synchronization error of less than 40 milliseconds.

10. A mobile terminal, characterized in that, include: An enhanced information renderer is used to receive and display enhanced data streams from a media server; Multimodal sensor data acquisition unit, used to acquire sensor data from equipment in real time; An interactive logic mapper is used to map the sensor data into control commands for the content played by the main player; A communication module is used to establish a connection with the media server and the main player, and to transmit the control commands. The mobile terminal is configured to perform the method as described in any one of claims 6 to 9.

Images