Method, device and system for acousto-optic synchronization control based on Android device

Abstract

This application provides a method, apparatus, and system for audio-visual synchronization control based on an Android device. The method includes: capturing video frames and audio streams from the screen using the Android system's native screen recording API; dividing the screen into regions, sampling the color of each region in the video frames, and calculating color data for each region based on the sampled data; performing a Fourier transform on the audio stream, calculating the audio beat, identifying the rhythmic characteristics of the audio, and generating beat data; establishing a communication connection with a USB signal generator, and sending the color data and beat data to a smart light fixture via the USB signal generator, so that the smart light fixture can adjust the corresponding color and flashing frequency according to the received color data and beat data, achieving synchronization with the image and sound. This application achieves adaptive and intelligent control of the lighting atmosphere, effectively enhancing the immersiveness and convenience of the smart lighting experience.

Description

Technical Field

[0001] This application belongs to the field of intelligent lighting control technology, and in particular relates to the field of sound and light synchronization control technology based on Android devices. Background Technology

[0002] With the widespread adoption of smart TVs and streaming media players, users' demand for immersive audiovisual experiences for watching movies and listening to music continues to rise. Ambient lighting, which can expand the visual experience, is gradually becoming a core component of home entertainment systems. Currently, the ambient lighting industry's technological iteration has mainly gone through three stages: the traditional ambient lighting stage, the initial smart device linkage stage, and the initial multimodal perception attempt stage. Early traditional ambient lights relied solely on physical buttons and remote controls for manual adjustment, only allowing switching of fixed colors, brightness, and simple dynamic lighting effects. Their simple structure and limited functionality lacked environmental perception and intelligent control capabilities. With the popularization of smart audio-visual devices, some ambient lights on the market achieved basic linkage with these devices, allowing control of the lights via remote control and mobile apps, triggering fixed lighting effects during video playback. However, the linkage logic was rigid and unable to adapt to different scenarios and autonomously optimize lighting effects. Later, high-end ambient lighting products began to incorporate external sensors such as cameras and microphones, attempting to achieve multi-dimensional environmental perception. However, the degree of multimodal data fusion was low, and the reliance on external dedicated hardware to build the perception system significantly increased the cost of device setup and the complexity of system operation.

[0003] Current ambient lighting technologies generally suffer from several technical shortcomings. First, hardware utilization is low; most systems require dedicated sensors such as cameras and microphones, making it impossible to reuse the built-in sensors on Android multimedia terminals. This results in high user procurement and maintenance costs and cumbersome usage procedures. Second, usage pattern recognition accuracy is insufficient; it cannot autonomously identify user scenarios such as watching movies or listening to music, relying on manual switching of lighting modes and relying on single-dimensional information to determine the scene, making it difficult to match the user's true usage intentions. Third, personalized adaptation capabilities are weak; lighting output relies on fixed effect templates and cannot learn to adapt to user preferences, making it difficult to meet diverse usage needs. Fourth, system deployment has high barriers to entry and significant privacy risks; high-end products rely on distributed multi-sensor architecture networking, requiring numerous indoor devices, resulting in high installation and maintenance costs. Furthermore, external acquisition devices can easily lead to privacy leaks in the user's audio-visual environment. Fifth, scene content analysis capabilities are shallow; it cannot deeply extract core content features such as color, audio beat, and musical rhythm, resulting in low compatibility between lighting effects and audio-visual content, making it difficult to create high-quality immersive audio-visual interactive effects.

[0004] Existing ambient lighting technology has evolved from simple manual control to intelligent linkage, but it still has significant shortcomings in areas such as hardware resource utilization, pattern recognition accuracy, sound and light synchronization effects, personalized adaptation, and system complexity. These issues limit the user experience and widespread adoption of ambient lighting systems. Summary of the Invention

[0005] This application provides a sound and light synchronization control method, device and system based on Android devices, which is used to realize adaptive and intelligent control of lighting atmosphere and improve the user experience of smart lighting.

[0006] In a first aspect, embodiments of this application provide a sound and light synchronization control method based on an Android device, comprising: capturing video frames and audio streams of the screen using the native screen recording API of the Android system; dividing the screen into regions, sampling the color of each region in the video frame, and calculating the color data of each region based on the sampling data; performing Fourier transform on the audio stream, calculating the audio beat, identifying the rhythmic features of the audio, and generating beat data; establishing a communication connection with a USB signal generator, and sending the color data and the beat data to a smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data, thereby achieving synchronization with the picture and sound.

[0007] In one implementation of the first aspect, capturing video frames and audio streams of the screen based on the Android system's native screen recording API includes: obtaining real-time capture permissions for screen image data and system audio through the Android system's native screen recording API; creating a virtual display module to mirror the current display content of the Android device's screen, setting the display resolution and pixel density, capturing video frames of the screen through an image data reading interface; and continuously reading and buffering the audio stream currently output by the Android device through a system audio data recording interface.

[0008] In one implementation of the first aspect, the method further includes: recording parameter optimization; the recording parameter optimization includes: real-time monitoring of the hardware performance status of the current Android device, dynamically adjusting the resolution and frame rate of video recording according to the hardware performance status to balance recording quality and system resource usage; and encoding the video frames using a hardware encoding acceleration method.

[0009] In one implementation of the first aspect, the step of dividing the screen into regions, sampling the colors of each region in the video frame, and calculating the color data of each region based on the sampled data includes: dividing the screen into N×M grid regions, determining the pixel range of each region based on the screen resolution and the number of grids, uniformly sampling the pixels in each region; calculating the RGB average value of all sampled pixels in each region, converting the RGB values ​​to the HSV color space, using the K-means clustering algorithm to identify the dominant color tone of each region, and calculating the standard deviation and saturation distribution statistical characteristics of the colors.

[0010] In one implementation of the first aspect, the step of performing a Fourier transform on the audio stream, calculating the audio beat intensity and audio features, and generating beat data includes: performing a fast Fourier transform on each frame of data to convert the time-domain signal into a frequency-domain signal; calculating the spectral amplitude and phase information of each frame based on the frequency-domain signal, and analyzing the energy distribution of different frequency components; calculating the energy value of each frame according to the energy distribution, generating an energy envelope curve, identifying the peak points in the energy envelope curve, calculating the number of beats per minute, and calculating the audio beat intensity based on the peak points and the number of beats per minute; extracting the average energy, energy change rate, spectral centroid, spectral bandwidth, zero-crossing rate, and autocorrelation function of the audio as audio features; wherein, the beat data includes the audio beat intensity and the audio features.

[0011] In one implementation of the first aspect, before performing the Fast Fourier Transform, the method further includes: after the audio stream undergoes normalization and denoising preprocessing, dividing the audio stream into frames of fixed length and applying a Hanning window to each frame, so as to perform a Fast Fourier Transform on the windowed frame data.

[0012] In one implementation of the first aspect, before sending the color data and the beat data to the smart lamp via the USB signal generator, the method further includes: packaging the color data and the beat data according to a custom frame format, wherein the frame format includes a synchronization word, frame type, length, data segment, and checksum.

[0013] Secondly, this application provides an audio-visual synchronization control device based on an Android device, comprising: a capture module for capturing video frames and audio streams of the screen using the native screen recording API of the Android system; a color data acquisition module for dividing the screen into regions, sampling the color of each region in the video frame, and calculating the color data of each region based on the sampled data; a beat data acquisition module for performing Fourier transform on the audio stream, calculating the audio beat, identifying the rhythmic characteristics of the audio, and generating beat data; and a communication connection module for establishing a communication connection with a USB signal generator and sending the color data and the beat data to a smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

[0014] Thirdly, this application provides an audio-visual synchronization control system based on an Android device, comprising: an Android device for capturing video frames and audio streams of the screen using the Android system's native screen recording API, dividing the screen into regions, sampling the color of each region in the video frame, and calculating the color data of each region based on the sampled data; performing a Fourier transform on the audio stream, calculating the audio beat, identifying the rhythmic features of the audio, and generating beat data; and a USB signal generator for establishing communication connections with the Android device and a smart lamp, respectively, for sending the color data and the beat data to the smart lamp, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

[0015] In one implementation of the third aspect, it further includes: a smart lamp, used to adjust the corresponding color and flashing frequency according to the received color data and beat data, so as to achieve synchronization with the picture and sound.

[0016] The audio-visual synchronization control method, apparatus, and system based on Android devices provided in this application have the following beneficial effects:

[0017] This application achieves adaptive and intelligent control of lighting atmosphere by real-time acquisition of screen and audio data from the Android device, analyzing the color and tempo characteristics of the media content. The entire control process requires no manual user intervention; the lights proactively respond to and adapt to changes in the media content, effectively enhancing the user experience of smart lighting. Attached Figure Description

[0018] Figure 1 The diagram shown is an overall flowchart of an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0019] Figure 2 The flowchart shown is a process for capturing video frames and audio streams from the screen in an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0020] Figure 3 The diagram shown illustrates the principle and flow of recording parameter optimization in an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0021] Figure 4 The diagram shown is a schematic representation of the principle flow of obtaining color data in an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0022] Figure 5 The flowchart shown is a process for generating beat data in an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0023] Figure 6 The diagram shown is a schematic representation of the principle of generating beat data in an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0024] Figure 7 The diagram shown is a data communication schematic of an audio-visual synchronization control method based on an Android device according to an embodiment of this application.

[0025] Figure 8 The diagram shown illustrates the overall implementation principle of an audio-visual synchronization control method based on an Android device, according to an embodiment of this application.

[0026] Figure 9 The diagram shown is a schematic representation of an audio-visual synchronization control device based on an Android device in one embodiment of this application.

[0027] Figure 10 The diagram shown is a schematic representation of an audio-visual synchronization control system based on an Android device in one embodiment of this application.

[0028] Component designation

[0029] 10- Sound and light synchronization control device based on Android device; 11- Capture module; 12- Color data acquisition module; 13- Beat data acquisition module; Communication connection module-14; Sound and light synchronization control system based on Android device-100; Android device-110; USB signal generator-120; Smart lamp-130; Steps-S100~S400; Steps-S110~S130; Steps-S310~S340. Detailed Implementation

[0030] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0031] The following will refer to the appendices in the embodiments of this application. Figure 1 To be continued Figure 10 This application provides a detailed description of the technical solutions for the audio-visual synchronization control method, apparatus, and system based on Android devices in the embodiments of this application. This allows those skilled in the art to understand and implement the audio-visual synchronization control method, apparatus, and system based on Android devices in this embodiment without inventive effort.

[0032] First, this embodiment provides a sound and light synchronization control method based on Android devices, which can be used in Android network TVs, smart set-top boxes, multimedia players and various audio-visual entertainment terminals based on the Android system, and is especially suitable for ambient lighting control in home theaters, e-sports games, music appreciation and smart home scenarios. Figure 1 The flowchart shown is a presentation of an audio-visual synchronization control method based on an Android device, as described in an embodiment of this application. Figure 1 As shown, the audio-visual synchronization control method based on an Android device provided in this application includes the following steps S100 to S400.

[0033] Step S100: Capture video frames and audio streams from the screen using the Android system's native screen recording API;

[0034] Step S200: Divide the screen into regions, sample the colors of each region in the video frame, and calculate the color data of each region based on the sampled data;

[0035] Step S300: Perform Fourier transform on the audio stream, calculate the audio beat, identify the rhythmic features of the audio, and generate beat data;

[0036] Step S400: Establish a communication connection with the USB signal generator, and send the color data and the beat data to the smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

[0037] The audio-visual synchronization control method based on Android devices provided in this embodiment captures screen images and audio streams in real time through the Android native screen recording API. On the one hand, the screen is divided into regions for color analysis to extract color data of each region. On the other hand, the audio stream is subjected to Fourier transform to detect beat information. The two parts of data are sent to the smart lamps via a USB signal transmitter, driving the lamps to change color with the screen content and move dynamically with the music beat, thus achieving precise synchronization of screen, sound and light.

[0038] The following combination Figures 2 to 8 The above steps S100 to S400 of the sound and light synchronization control method based on Android device in this embodiment will be described in detail.

[0039] Step S100: Capture video frames and audio streams from the screen using the Android system's native screen recording API.

[0040] In this embodiment, the native screen recording API (MediaProjection API) provided by the Android system is used to achieve real-time capture of screen images and audio streams.

[0041] Figure 2 The flowchart shown is a process for capturing video frames and audio streams from the screen in an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 2 As shown, in one implementation of this embodiment, capturing video frames and audio streams of the screen based on the native screen recording API of the Android system includes the following steps S110 to S130.

[0042] Step S110: Obtain real-time capture permissions for screen data and system audio through the Android system's native screen recording API.

[0043] Step S120: Create a virtual display module to mirror the current screen display content of the Android device, set the display resolution and pixel density, and capture video frames of the screen through the image data reading interface.

[0044] Step S130: Continuously read and cache the audio stream currently output by the Android device through the system audio data recording interface.

[0045] In this embodiment, when capturing video frames, a channel for obtaining screen pixel data is established through permission requests and virtual display. When capturing audio streams, digital signals are read directly from the system audio output. This acquisition method does not require additional hardware such as external cameras or microphones, making full use of the Android device's own hardware and software resources to provide a real-time, clean audio and video data source for subsequent color analysis and beat extraction.

[0046] The process involves creating a screen recording permission request via MediaProjectionManager, obtaining a MediaProjection instance after user authorization, and then creating a VirtualDisplay to mirror the screen content. Each frame of the image is then read in RGBA format using ImageReader. For audio capture, the currently playing digital audio stream is directly read from the system audio output mixer via the AudioRecord API. The stream has a sampling rate of 44.1kHz, is stereo, and is 16-bit deep. These two data sets together form the raw input for subsequent color analysis and beat detection, enabling real-time acquisition of media content.

[0047] In detail, this embodiment uses the Android system's standard MediaProjection mechanism to obtain screen recording permission. First, an Intent object for the screen capture permission request is created using the `createScreenCaptureIntent` method of the `MediaProjectionManager` class. This Intent triggers a system-level permission confirmation pop-up, explicitly informing the user that the application will obtain permission to capture screen content and system audio. Then, the `startActivityForResult` method is called in the current Activity to send this Intent to the system, which then displays a permission confirmation interface. After the user makes a "allow" or "deny" selection in the permission confirmation interface, the system returns the selection result to the current Activity via the `onActivityResult` callback. The application receives and judges the return result in this callback method: if the user confirms the authorization, a `MediaProjection` instance is obtained using the `getMediaProjection` method of `MediaProjectionManager`. This instance serves as the core handle for subsequent screen capture operations, used to create a virtual display device; if the user denies the authorization, the application prompts the user that screen recording permission is required and guides the user to re-initiate the authorization request. Once a valid MediaProjection instance is obtained, the application acquires the legal permission to record the current screen and system audio output, and can proceed with the subsequent video frame capture and audio stream capture process.

[0048] Based on a MediaProjection instance, this embodiment captures video frames by creating a virtual display device. Specifically, the createVirtualDisplay method of the MediaProjection instance is called, passing in parameters such as display name, screen width, screen height, screen pixel density (DPI), and flags, to create a virtual display mirror that is synchronized in real time with the content of the current physical screen. The display parameters of this virtual display device are matched with the actual resolution of the current screen to ensure that the captured image is completely consistent with the image viewed by the user.

[0049] Meanwhile, this embodiment creates an ImageReader instance as a receiver of image data. During creation, its image format is set to ImageFormat.RGBA to directly acquire RGBA four-channel pixel data containing complete color information. This format eliminates the need for subsequent format conversion steps before color analysis, thus improving processing efficiency. The width and height of the ImageReader are also set to match the screen resolution, and its maximum number of images is set to 2 to 3 frames to balance memory usage and real-time requirements. Then, the ImageReader's Surface object is used as the target output for virtual display, so that each frame of the screen is automatically rendered into this Surface, which is then received and cached by the ImageReader.

[0050] Regarding frame data acquisition, this embodiment employs a callback-driven approach for real-time capture: a listener is registered using the `setOnImageAvailableListener` method of `ImageReader`. When a new video frame arrives, the system automatically triggers the `onImageAvailable` callback. Within this callback method, the application calls the `acquireLatestImage` method of `ImageReader` to obtain the latest `Image` object. This method automatically discards cached old frames, ensuring that the latest frame is always processed, thereby reducing latency. After obtaining the `Image` object, its `getPlanes` method retrieves an `ImagePlane` array containing pixel data. A `ByteBuffer` object is then extracted from the `ImagePlane`, storing complete RGBA pixel data, which can be directly passed to the color analysis module for subsequent regional color calculations. After each frame is processed, the `close` method of the `Image` object is called to release its occupied memory resources, and the `setOnImageAvailableListener` method of `ImageReader` is called with null to either unlisten or ensure the next frame callback is triggered correctly, thus forming a complete closed loop of frame acquisition and resource release, preventing memory leaks.

[0051] This embodiment achieves internal loopback capture of system audio through the AudioRecord API, rather than relying on external microphones. First, an AudioRecord instance is created, and the audio source is set to the system's internal audio output mixer to directly capture the currently playing media audio data from the system's underlying audio layer, thus avoiding environmental noise interference and sound quality loss caused by secondary analog-to-digital conversion. The audio parameters are configured as follows: the sampling rate is set to 44.1kHz, which covers the audible frequency range and meets the accuracy requirements for beat detection, and is well supported on most Android devices; the number of channels is set to stereo (CHANNEL_IN_STEREO) to preserve the differences between the left and right channels in subsequent analysis, providing richer spatial features for spectral analysis; the audio encoding format and bit depth are set to AudioFormat.ENCODING_PCM_16BIT, meaning each sample point is represented by a 16-bit signed integer. This format is standard PCM encoding, highly compatible, and facilitates subsequent Fast Fourier Transform calculations; the audio recording mode is set to MODE_STREAM to support continuous streaming.

[0052] Regarding buffer configuration, the application obtains the system-recommended minimum buffer size through the `getMinBufferSize` method of `AudioRecord`, and sets the actual buffer size used to an integer multiple of this value (usually 2 or 4 times) to strike a balance between read latency and system load. Then, the `startRecording` method of `AudioRecord` is called to start audio acquisition.

[0053] In terms of audio data reading, this embodiment performs the reading operation in a separate background thread in a loop. Each time the `read` method of `AudioRecord` is called, the acquired audio data is filled from the hardware buffer into a byte array buffer pre-allocated by the application. The amount of data read is the size of one buffer. The return value of the `read` method is the number of bytes actually read. The application compares this with the expected number of bytes to be read. If the return value is greater than 0, it indicates that valid data has been successfully read. The byte array is then sent to the audio processing module for subsequent preprocessing and Fourier transform. If the return value is negative, it indicates a reading error. The application executes the corresponding exception handling strategy according to the error code, such as reinitializing `AudioRecord` or prompting the user to check the audio channel status. The entire audio capture process continues until the application actively calls the `stop` method of `AudioRecord` to stop the acquisition, thereby achieving continuous, low-latency, and uninterrupted real-time capture of the audio stream.

[0054] In one implementation of this embodiment, the method further includes: optimizing recording parameters. In this embodiment, the optimization of recording parameters aims to dynamically adjust video capture parameters according to different performance levels of the Android device and the current system load state, so as to achieve an adaptive balance between recording quality and system performance.

[0055] Figure 3 This diagram illustrates the principle and flow of recording parameter optimization in an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 3 As shown, the recording parameter optimization includes: real-time monitoring of the current Android device's hardware performance status, dynamically adjusting the video recording resolution and frame rate based on the hardware performance status to balance recording quality and system resource usage; and using hardware encoding acceleration to encode the video frames.

[0056] Specifically, the system obtains the real-time CPU utilization through the Android system's CPU load monitoring interface, acquires the available heap memory size of the application through the memory management interface, and combines this with static hardware information such as device model and chip platform to comprehensively assess the device's performance level and available computing resources. Based on the assessment results, the device performance status is divided into three levels: high load, normal, and low load, serving as the basis for subsequent dynamic parameter adjustments.

[0057] Based on the performance evaluation results, the video recording resolution and frame rate are adaptively adjusted. When the device is under low load and computing resources are sufficient, a higher recording resolution and frame rate are used to obtain finer color details and improve the subtlety of lighting effects. When the device is under high load or system resources are scarce, the recording resolution and frame rate are appropriately reduced to reduce CPU and memory usage, prioritizing smooth media playback and system responsiveness, and avoiding stuttering or audio interruptions due to resource contention. In specific implementation, the resolution is gradually adjusted downwards from full-screen resolution, and the frame rate is gradually reduced from 30fps to 15fps or lower. The adjustment range is dynamically calculated based on real-time performance data to ensure a smooth transition and avoid frequent fluctuations.

[0058] To reduce the computational burden on the central processing unit (CPU) in video frame encoding and image processing, this embodiment employs hardware encoding acceleration technology. Specifically, utilizing the hardware codec interface provided by the Android system, the video frame encoding task is offloaded to a dedicated hardware codec module built into the device. This module, as an independent hardware unit, can efficiently complete video frame compression and format conversion operations under low power consumption conditions. By offloading the encoding and decoding tasks from the CPU to the dedicated hardware module, the CPU's computational load is significantly reduced, reserving sufficient computing resources for color analysis algorithms and audio beat detection algorithms, thereby achieving more efficient concurrent processing under limited power consumption and heat dissipation conditions.

[0059] In this embodiment, performance evaluation and parameter adjustment operations are performed periodically at certain time intervals. When the performance status changes, the corresponding resolution and frame rate adjustments take effect immediately, ensuring that the recording quality and system performance are always kept in a dynamic balance, so that a smooth recording experience and stable light synchronization effect can be obtained on various Android devices.

[0060] Step S200: Divide the screen into regions, sample the colors of each region in the video frame, and calculate the color data of each region based on the sampled data.

[0061] In this embodiment, the captured video frames are subjected to regional color analysis to extract color features of different areas of the image, providing a data basis for subsequent color mapping of lamps. This mainly includes two parts: regional color analysis and color feature extraction.

[0062] Figure 4 This diagram illustrates the principle flow of acquiring color data in an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 4 As shown, in one implementation of this embodiment, dividing the screen into regions, sampling the color of each region in the video frame, and calculating the color data of each region based on the sampled data includes:

[0063] The screen is divided into N×M grid regions. The pixel range of each region is determined according to the screen resolution and the number of grids. Pixels in each region are uniformly sampled. The RGB average value of all sampled pixels in each region is calculated. The RGB values ​​are converted to HSV color space. The K-means clustering algorithm is used to identify the dominant color of each region. The standard deviation and saturation distribution statistical characteristics of the colors are calculated.

[0064] Region-based color analysis involves dividing the screen into several independent grid regions, sampling colors and calculating features for each region, thereby obtaining color distribution information at different locations on the screen. In this embodiment, the process of region-based color analysis includes:

[0065] 1) Grid Division: The screen is divided into N rows and M columns horizontally and vertically according to a preset division strategy, forming N×M rectangular grid areas. The grid density can be flexibly configured according to the actual application scenario. For example, in a movie-watching scenario, a 3×3 grid can be used to obtain the macroscopic color distribution of the image with lower computational overhead; in scenarios requiring more detailed color analysis of specific areas, a 3×5 or 5×5 grid can be used to obtain more differentiated color information for different areas. The number of rows and columns of the grid can be customized by the user through the configuration interface, or it can be automatically selected based on the type of content being played (movie, TV series, variety show, game, etc.).

[0066] 2) Region Size Calculation: After determining the grid division method, calculate the pixel range covered by each grid region based on the actual resolution of the current screen. Let the screen width be W pixels and the height be H pixels, divided into N rows and M columns. Then, the width of each grid region is W / M pixels and the height is H / N pixels. Taking a 1920×1080 resolution with a 3×5 grid division as an example, the width of each grid region is 384 pixels (1920 / 5), and the height is 360 pixels (1080 / 3). Based on this, calculate the starting and ending pixel positions of each grid region in the screen coordinate system, which will serve as the spatial range for subsequent color sampling.

[0067] 3) Color Sampling: To avoid excessive computation caused by traversing all pixels within each grid region, a uniform sampling strategy is adopted to subsample pixels within each region. Specifically, within each grid region, pixels are selected at equal intervals in both the horizontal and vertical directions according to a fixed sampling step size. The sampling step size is dynamically determined based on the grid size and the preset sampling density. For example, a smaller sampling step size can be used for larger grid regions to obtain more color samples, while a larger sampling step size is used for smaller grid regions to reduce redundant computation. Through uniform sampling, the number of pixels processed per frame can be reduced by more than an order of magnitude while maintaining the representativeness of color statistical features, significantly improving real-time processing efficiency.

[0068] After pixel sampling of each grid region is completed, further feature extraction is performed on the color data of each region, transforming it into color control parameters that can be used to drive the lighting fixtures. Specifically, this includes:

[0069] 1) Average Color Calculation: For all pixel samples collected within each grid area, the pixel values ​​of the red (R), green (G), and blue (B) channels are summed and divided by the total number of samples to calculate the average RGB value of that area. This average color reflects the overall color tendency of the image in that grid area and serves as the basis for color tuning of the corresponding lighting fixtures in that area.

[0070] 2) Color Space Conversion: Since the RGB color space is not intuitive enough in describing the perceptual attributes of color such as hue, saturation, and value, it is converted to the HSV (Hue, Saturation, Value) color space after calculating the average RGB value. During the conversion, the RGB values ​​are normalized to the 0-1 range, and the corresponding hue angle (0° to 360°), saturation value (0 to 1), and value (0 to 1) are calculated using the standard HSV conversion formula. The introduction of the HSV color space makes subsequent color feature analysis and light mapping rule design more convenient. For example, colors can be classified according to their emotional value, their vibrancy can be judged based on their saturation value, and the brightness output of lights can be adjusted based on their value.

[0071] 3) Dominant Color Extraction: For each grid region, the K-means clustering algorithm is used to identify the dominant color of that region. Specifically, the RGB values ​​of all sampled pixels within the region are used as clustering samples, and the number of clusters K is set to 3 to 5. Iterative clustering is performed in the color space using Euclidean distance as the similarity metric. After several iterations, each sample is assigned to the color cluster represented by the nearest cluster center. The number of samples contained in each color cluster is counted, and the central color of the color cluster with the largest number of samples is taken as the dominant color of that region. The extraction of the dominant color helps to filter out noise interference in the image, making the colors presented by the lights purer and more stable, and avoiding flickering of the lights due to sudden changes in local colors in the image.

[0072] 4) Color Distribution Statistics: To further characterize the color features of each grid area, this embodiment also calculates a series of color statistical indicators. Among them, the color standard deviation is calculated by statistically analyzing the deviation between the RGB values ​​of each pixel within the area and the average RGB value. A larger standard deviation indicates a more drastic color change in that area, which can be used to control the transition speed and smoothness of the lamp's color changes. The saturation distribution is calculated by statistically analyzing the HSV saturation values ​​of each pixel within the area, calculating their mean and variance, to determine whether the color vibrancy of the area is consistent, thereby deciding whether saturation compensation is needed for the lamp output. In addition, this embodiment also calculates the histogram distribution of each color channel to assist in identifying color gradients and transition areas in the image. These statistical features, combined with the average color and dominant hue, constitute a complete color feature vector for each grid area, providing multi-dimensional control basis for subsequent lamp color adjustment.

[0073] As shown above, the color data for each region is calculated based on the sampled data. Specifically, it includes the RGB average value obtained by averaging the red, green, and blue channels of all sampled pixels in that region, as well as the central color (primary hue) of the color cluster with the largest number of samples in that region, identified by the K-means clustering algorithm. It also includes the HSV values ​​of the hue, saturation, and lightness components converted from the RGB average value. Furthermore, the color data also includes the color standard deviation obtained by statistically analyzing the deviation between the RGB values ​​of each pixel in that region and the average RGB value, the saturation distribution characteristics formed by the mean and variance of the HSV saturation values ​​of each pixel in that region, and the histogram distribution data of the pixel values ​​of each color channel. The above-mentioned RGB average value, HSV value, primary hue, color standard deviation, saturation distribution, and histogram distribution together constitute the complete color feature vector of that region, serving as multi-dimensional control parameters for subsequently driving the color changes of the corresponding lighting fixtures in that region.

[0074] Step S300: Perform Fourier transform on the audio stream, calculate the audio beat, identify the rhythmic features of the audio, and generate beat data.

[0075] In this embodiment, signal analysis and feature extraction are performed on the captured audio stream to identify the beat position, rhythm intensity and spectral characteristics of the audio, providing data support for the rhythm control of the lighting fixtures.

[0076] Figure 5 The flowchart shown is a process for generating beat data in an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 5 As shown, in one implementation of this embodiment, performing Fourier transform on the audio stream, calculating the audio beat intensity and audio features, and generating beat data includes the following steps S310 to S340.

[0077] Step S310: After the audio stream is preprocessed by normalization and denoising, the audio stream is divided into frames of fixed length and a Hanning window is applied to each frame to perform a fast Fourier transform on the windowed frame data.

[0078] Step S320: Perform a fast Fourier transform on each frame of data to convert the time-domain signal into a frequency-domain signal. Calculate the spectral amplitude and phase information of each frame based on the frequency-domain signal and analyze the energy distribution of different frequency components.

[0079] Step S330: Calculate the energy value of each frame based on the energy distribution, generate an energy envelope curve, identify the peak points in the energy envelope curve, calculate the number of beats per minute, and calculate the audio beat intensity based on the peak points and the number of beats per minute.

[0080] Step S340: Extract the average energy, energy change rate, spectral centroid, spectral bandwidth, zero-crossing rate, and autocorrelation function of the audio as audio features.

[0081] The beat data includes the audio beat intensity and the audio characteristics.

[0082] In this embodiment, the captured audio stream is converted from a time-domain signal to a frequency-domain signal for analysis. First, the audio data is preprocessed by framing and windowing. Then, a Fast Fourier Transform is used to calculate the spectral amplitude and phase information of each frame to obtain the energy distribution of different frequency components. Based on this, the total energy value of each frame is calculated to form an energy envelope curve. A peak detection algorithm is used to identify peak points in the energy envelope as candidate beat positions, and the time interval between adjacent peaks is counted to calculate the beats per minute (BPM). Simultaneously, the intensity level of each beat is evaluated based on the peak height and frequency-domain energy distribution. Furthermore, time-frequency domain features such as spectral centroid, spectral bandwidth, and zero-crossing rate are extracted to assist in identifying advanced rhythmic features such as the rhythmic pattern, beat regularity, and emotional tone of the audio. The final generated beat information includes the beat timing position, beat intensity value, and BPM value, which are then used to drive the lights to move synchronously with the music rhythm.

[0083] Figure 6 This diagram illustrates the principle of generating beat data in an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 6 As shown, the principle process for generating beat data is as follows:

[0084] 1) Audio signal processing flow

[0085] The audio signal processing workflow includes three stages: preprocessing, framing, and windowing.

[0086] In the preprocessing stage, the raw Pulse Code Modulation (PCM) data in the audio stream is converted from integer format to floating-point format, so that the amplitude range of the signal is uniformly mapped to the interval of -1 to 1, in order to eliminate the impact of the difference in dimensions during audio acquisition by different devices on the accuracy of subsequent analysis; at the same time, moving average filtering or median filtering methods are used to smooth the audio signal, suppress burst noise and random interference introduced during the acquisition process, and retain the main characteristics of the audio signal.

[0087] In the framing process, the continuous audio stream is divided into several fixed-length frame segments. Since audio signals have short-term stationary characteristics—meaning their statistical properties remain approximately constant over a very short time range—analyzing them frame-by-frame effectively captures the temporal variation characteristics of the audio. In this embodiment, the frame length is set to 20 to 40 milliseconds. Within this time window, the spectral characteristics of the audio signal are relatively stable, balancing the temporal and frequency resolution of beat detection. An overlapping region is set between adjacent frames, typically with an overlap rate of 50%, to eliminate the boundary truncation effect caused by framing and ensure a smooth transition of feature parameters between frames.

[0088] In the windowing process, a Hanning window function is applied to each frame of audio data. The Hanning window is a cosine window function with both ends smoothly transitioning to zero. Each frame of data is multiplied point-by-point by the Hanning window value, causing the amplitude at the beginning and end of the frame to gradually become zero. The windowing operation effectively suppresses spectral leakage in the frequency domain caused by abrupt frame truncation, improving the accuracy of subsequent Fourier transform spectral analysis.

[0089] 2) Implementation of Fourier Transform

[0090] Each frame of windowed audio data is transformed from a time-domain signal to a frequency-domain signal for processing. The Fast Fourier Transform (FFT) algorithm is used to perform this transformation. The FFT algorithm reduces the computational complexity from O(N²) to O(N·log₂N) by performing a butterfly decomposition of the multiplication and addition operations in the Discrete Fourier Transform (DFT), significantly improving the computational efficiency of spectral analysis. After the transformation, for each frequency component, the square root of the sum of the squares of its real and imaginary parts is calculated to obtain the spectral amplitude value of that frequency component. The arctangent function of the ratio of the imaginary to the real part is calculated to obtain the phase information. The spectral amplitude reflects the energy strength of the audio signal at different frequencies, while the phase information carries the temporal relative position of each frequency component. Based on this, the power spectral density is obtained by squaring the spectral amplitude, and the power values ​​of each frequency point are accumulated from low to high frequency to obtain the energy distribution data of the audio signal in different frequency bands (such as low-frequency, mid-frequency, and high-frequency bands), providing a frequency domain analysis basis for subsequent beat detection and audio feature extraction.

[0091] 3) Beat detection

[0092] In this embodiment, beat detection identifies the beat position and intensity in the audio by analyzing the energy changes of the audio frame sequence.

[0093] 3-1) Energy Envelope Extraction: For each frame of audio data, the amplitude values ​​of each sampling point within the frame are squared and summed, then divided by the frame length to obtain the short-time energy value of that frame. This value reflects the loudness of the audio signal at that moment. The energy values ​​of all frames are arranged in chronological order to form an energy envelope curve that fluctuates with the audio playback time.

[0094] 3-2) Peak Detection: Peak detection is performed on the energy envelope curve to identify the locations of local maxima. Specifically, the energy value of each frame is compared with the energy values ​​of its preceding and following frames. If the energy value of the current frame is greater than the energy values ​​of the preceding and following frames and exceeds a predetermined threshold, then that location is determined to be a peak point. To avoid false detection of locations with weak volume but drastic amplitude fluctuations, a dynamic threshold is set, which is a multiple of the average energy value within the current time period (usually 1.2 to 1.5 times). The time positions of each peak point are the candidate beat time points.

[0095] 3-3) Beat interval calculation: Calculate the time interval between adjacent candidate beat points, and calculate the mode or median of all intervals as the baseline beat period. Based on this, calculate the number of beats per minute (BPM). The calculation formula is BPM = 60 / baseline beat period (seconds).

[0096] 3-4) Beat Intensity Assessment: For each detected beat point, its intensity level is comprehensively assessed by combining the peak height and frequency domain energy distribution of that point. The peak height reflects the energy impact intensity of the beat, while the frequency distribution characteristics reflect the timbre information of the beat. Combining both, the beat intensity is divided into multiple levels, including strong, medium, and weak, for use in dynamically adjusting the flicker brightness and dynamic amplitude when driving lighting fixtures.

[0097] 4) Audio Feature Extraction

[0098] In addition to beat information, this embodiment also extracts various audio features to provide auxiliary decision-making basis for lighting effect color adjustment and dynamic mode selection.

[0099] 4-1) Energy characteristics: Calculate the average energy of the audio signal and the rate of energy change between adjacent frames. The average energy reflects the overall loudness level of the audio and is used to adjust the overall brightness of the lighting fixtures; the rate of energy change reflects the dynamic fluctuation of the audio. A larger rate of change indicates a wide dynamic range and severe fluctuations in the audio, allowing for a larger adjustment range in the lighting response.

[0100] 4-2) Spectral Characteristics: Calculate the spectral centroid and spectral bandwidth. The spectral centroid is the weighted average frequency of the power spectrum, and its value reflects the brightness of the audio. A higher centroid indicates that the audio is generally bright or sharp, corresponding to cool-toned lighting. A lower centroid indicates that the audio is generally deep or warm, corresponding to warm-toned lighting. The spectral bandwidth reflects the degree of spectral diffusion. A larger bandwidth indicates a richer audio frequency composition, and lighting changes can be set to a higher dynamic range.

[0101] 4-3) Time-domain characteristics: Calculate the zero-crossing rate and autocorrelation function. The zero-crossing rate is the number of times the audio signal crosses the zero-level line per unit time. It has a high response to percussion and strong rhythm music and is used to help identify the rhythm density of audio. The autocorrelation function is used to detect the periodicity of the audio signal and can help distinguish between strong beat music and ambient music, so as to drive the lamps to use rhythmic mode or gradient mode respectively.

[0102] The aforementioned beat information and audio features together constitute an audio feature vector, including beat timing position, beat intensity level, BPM value, average energy, energy change rate, spectral centroid, spectral bandwidth, zero-crossing rate, and autocorrelation function, which serve as multi-dimensional parameter inputs for lamp rhythm control.

[0103] Step S400: Establish a communication connection with the USB signal generator, and send the color data and the beat data to the smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

[0104] In this embodiment, the color data of each region generated by the color analysis module and the beat information generated by the audio processing module are sent to the smart lamp via a USB signal transmitter to realize the data transmission from audio and video analysis results to light execution instructions.

[0105] In one specific implementation of this embodiment, the generated regions package the color data and the beat data according to a custom frame format. The frame structure includes a synchronization word, frame type, data length, valid data segment, and CRC check field to ensure the integrity and accuracy of data transmission. Subsequently, the packaged data is sent to an external USB signal transmitter (USB dongle) via a USB interface. After receiving the data, the signal transmitter converts it into a wireless signal (such as 2.4G radio frequency, Bluetooth, or Zigbee) and forwards it to the smart lamp. Through the above-mentioned wired and wireless transmission method, the high-speed and stable characteristics of the USB interface are utilized to ensure low-latency data transmission from the device to the transmitter, while wireless communication enables the flexibility of lamp deployment and the scalability of multi-lamp networking. This achieves efficient, real-time, and reliable transmission of color data and beat information from the Android device to the lamp.

[0106] A USB dongle serves as a communication bridge between Android devices and smart lighting fixtures. Its communication architecture includes hardware design, communication protocols, and connection management.

[0107] Specifically, the USB signal transmitter uses a USB-C interface to physically connect to the Android device. This interface supports reversible insertion and has high-speed data transmission capabilities to meet the bandwidth requirements for real-time transmission of color data and beat information. The transmitter integrates a microcontroller (MCU) and a wireless radio frequency chip. The microcontroller is responsible for receiving data from the Android device via the USB protocol and parsing it, while the wireless radio frequency chip is responsible for converting the parsed data into wireless signals for transmission.

[0108] Android devices and USB transmitters communicate using a custom frame-based protocol, which defines the frame structure and transmission rules for data exchange. The custom frame structure includes fields such as synchronization word, frame type, data length, valid data segment, and checksum, ensuring the integrity and parsability of each frame. The purpose of using a custom protocol instead of a generic one is to simplify protocol overhead, reduce unnecessary handshakes and control fields, thereby lowering the transmission latency of each frame and meeting the stringent real-time requirements of audio-visual synchronization.

[0109] When the USB transmitter is plugged into the USB port of an Android device, the device automatically detects the peripheral connection and completes the enumeration and initialization process to establish a data communication channel. During data transmission, the device sends heartbeat data packets to the USB transmitter at a fixed frequency to confirm the normal operation of the communication link. When the device detects that it has not received a response confirmation from the transmitter or that data transmission has failed, it triggers an automatic reconnection mechanism and re-executes the device enumeration and channel establishment process. Simultaneously, the transmitter has an error recovery function; when it detects a data packet verification error or data loss, it sends a retransmission request to the device. Upon receiving the request, the device retransmits the corresponding data frame. This mechanism ensures the continuity and reliability of data transmission.

[0110] In this embodiment, data transmission adopts a custom frame format. Each frame consists of five fields arranged sequentially: synchronization word, frame type, length, data segment, and checksum. The synchronization word, located at the beginning of the frame, is a fixed sequence of flag bits used by the receiver to identify and align the frame's start position, ensuring accurate positioning of the start boundary of each frame within the data stream. The frame type, following the synchronization word, identifies the data category carried by the current frame, including color data frames, beat information frames, heartbeat packets, and acknowledgment frames. The receiver uses the corresponding parsing logic to process the data segment content based on the frame type field. The length field records the effective number of bytes in the data segment, enabling the receiver to correctly extract complete data segments from the data stream. The data segment, immediately following the length field, carries payloads such as color data or beat information. The checksum, located at the end of the frame, uses a CRC (Cyclic Redundancy Check) algorithm to verify the synchronization word, frame type, length, and data segment, and the result is entered into this field. After receiving a complete frame, the receiver recalculates the checksum using the same algorithm and compares it with the checksum at the end of the frame. If they match, the data transmission is correct; otherwise, the frame data is considered erroneous, triggering a retransmission mechanism.

[0111] In this embodiment, the data transmission process includes data packaging, verification generation, and data transmission.

[0112] First, the color data of each region, including the RGB average value, dominant hue and HSV value of each region, is filled into the data segment of the frame structure according to the preset data format. Then, the beat information, including the beat timing position, beat intensity level and BPM value, is filled into the data segment of another frame. Finally, the corresponding frame type field is set according to the data type to complete the encapsulation from the raw analysis data to the standardized frame format.

[0113] After each data frame is packaged, a CRC check is performed on the synchronization word, frame type, length, and data segment of that frame. The calculated checksum value is then filled into the checksum field at the end of the frame. After the checksum generation is complete, the complete frame data is sent to an external USB signal transmitter via the USB interface. An asynchronous non-blocking write method is used during transmission to avoid data transmission blocking the main processing flow. After receiving the data frame, the USB signal transmitter first performs frame start boundary identification and checksum verification. After confirming that the data is complete and error-free, it extracts the payload from the data segment and converts it into a wireless signal through its internal wireless radio frequency chip for broadcasting or directional transmission to the smart lighting fixture, ultimately driving the lighting fixture to perform color changes and rhythmic movements.

[0114] Figure 7 This diagram illustrates the data communication process of an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 7 As shown, the data communication originates from an Android smart TV or media player. The device runs screen recording, color analysis, and audio processing software algorithms to package the captured screen color data and audio beat information into a custom frame format, generate a CRC checksum, and then transmit it via a wired connection through the USB interface.

[0115] The USB dongle acts as a data communication relay bridge, with its USB-C interface physically connected to the USB port of an Android device to receive packetized data from the device. The microcontroller inside the dongle performs verification and protocol parsing on the received data frames. Once the data is confirmed to be complete, the payload is passed to the internal wireless RF chip, which converts the data into a wireless signal according to a preset communication protocol and transmits it externally. The wireless communication method uses one or more combinations of proprietary 2.4G RF protocols, Bluetooth, Zigbee, or Thread protocols to adapt to the communication standards and networking requirements of different smart lighting fixtures.

[0116] In applications with a small number of lights or short communication distances, the USB transmitter directly sends wireless signals to each smart light. Each light receives the signal, extracts its corresponding color data and timing information, and then executes its lighting output. In applications with a large number of lights or requiring deployment across rooms, an optional gateway or network center acts as a data relay node. It receives wireless data from the USB transmitter, parses and repackages the data, and then distributes control commands to each light node in the network according to the networking protocol, achieving multi-light networking and centralized control.

[0117] Smart lighting fixtures, serving as the final execution end of data communication, include TV backlight strips, left-side ambient lights, right-side ambient lights, ambient auxiliary lights, and table lamps or light strip extensions. Each fixture receives wireless control signals directly from a USB signal transmitter or forwarded via a gateway, extracts RGB color data and beat intensity information for its respective area, and drives the internal LED light source to change color and brightness, achieving precise synchronization of the color and audio beat of the content played on the Android device.

[0118] In the aforementioned communication process, data is transmitted unidirectionally from the Android device to the transmitter via USB wired transmission, and then wirelessly to each lamp, forming a complete communication link. Among them, USB wired transmission ensures high-speed and stable data output from the device to the transmitter, while wireless transmission ensures the flexibility of lamp deployment and the convenience of multi-lamp expansion.

[0119] Figure 8 This diagram illustrates the overall implementation principle of an audio-visual synchronization control method based on an Android device, according to an embodiment of this application. Figure 8 As shown, the implementation principle of the audio-visual synchronization control method based on an Android device according to an embodiment of this application is as follows:

[0120] After the user enables the synchronization function, they first select the current media mode, either video or music, to determine the weighting and blending strategy between subsequent audio beats and screen colors. Then, real-time capture of the screen and audio stream is initiated. On one hand, the captured video frames are divided into a grid, and pixels within each grid area are uniformly sampled and their RGB averages calculated. The RGB values ​​are then converted to the HSV color space to extract the dominant hue and color statistical features of each area. On the other hand, the captured audio stream is framed, windowed, and subjected to Fast Fourier Transform (FFT) to calculate the spectral amplitude and energy distribution of each frame, using peak detection and... Energy envelope analysis extracts beat timing position, beat intensity level, and BPM value. After completing the above calculations, the color data and beat information of each region are packaged according to a custom frame format and a CRC check value is generated. This data is then sent to an external USB signal transmitter via a USB interface. The transmitter converts the data into a wireless signal and sends it outward. After receiving the wireless signal, the smart lights parse out their corresponding color control parameters and rhythm control parameters, driving the LED light source to change color with the screen content and move rhythmically with the music beat. No manual intervention is required throughout the process until the user actively stops the synchronization function.

[0121] The scope of protection of the audio-visual synchronization control method based on Android devices described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0122] Secondly, this application also provides an audio-visual synchronization control device based on an Android device. Figure 9 The diagram shown is a schematic representation of an audio-visual synchronization control device based on an Android device according to an embodiment of this application. Figure 9 As shown, the sound and light synchronization control device 10 based on an Android device in this embodiment includes: a capture module 11, a color data acquisition module 12, a beat data acquisition module 13, and a communication connection module 14.

[0123] In this embodiment, the capture module 11 captures video frames and audio streams of the screen based on the native screen recording API of the Android system; the color data acquisition module 12 is used to divide the screen into regions, sample the color of each region in the video frame, and calculate the color data of each region based on the sampled data; the beat data acquisition module 13 is used to perform Fourier transform on the audio stream, calculate the audio beat, identify the rhythm characteristics of the audio, and generate beat data; the communication connection module 14 is used to establish a communication connection with the USB signal generator, and send the color data and the beat data to the smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

[0124] In this embodiment, the capture module 11 captures video frames and audio streams from the screen based on the Android system's native screen recording API. Specifically, it obtains screen pixel data through screen recording permission requests and virtual display mirroring, and directly reads the currently playing digital audio signal from the device's internal audio output mixer, rather than relying on an external microphone. The color data acquisition module 12 divides the screen into several grid regions, uniformly samples the pixels in each region, calculates the RGB average value of each region's pixels in the red, green, and blue channels based on the sampled data, and identifies the dominant color tone of each region using the K-means clustering algorithm. Simultaneously, it converts the RGB values ​​to the HSV color space to obtain hue, saturation, and brightness information. These data together constitute the complete color features of each region. The beat data acquisition module 13 performs frame segmentation and windowing preprocessing on the captured audio stream, and... The Fast Fourier Transform (FFT) converts each frame of audio data from the time domain to the frequency domain, calculates the spectral amplitude and energy distribution of each frame, extracts the energy envelope curve, identifies the beat position through peak detection, calculates the BPM value by counting the beat interval, and evaluates the intensity level of each beat based on the peak height and frequency domain energy distribution. Finally, beat data containing beat timing position, beat intensity value, and BPM value is generated. The communication connection module 14 is used to establish a communication connection with the USB signal generator through the USB interface, package the color data and beat data according to a custom frame format and generate a CRC check value, and send it to the USB signal generator through the USB interface. The USB signal generator converts the data into a wireless signal and sends it to the smart lamp, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve real-time synchronization with the picture color and audio rhythm.

[0125] Each functional module provided in this embodiment corresponds one-to-one with each step in the aforementioned method embodiment, and is used to execute the corresponding method steps. For the specific implementation principle and data processing flow, please refer to the relevant description in the aforementioned method embodiment, which will not be repeated here.

[0126] Finally, this application also provides an audio-visual synchronization control system based on an Android device. Figure 10 The diagram shown is a schematic representation of an audio-visual synchronization control system based on an Android device, according to one embodiment of this application. Figure 10 As shown, the sound and light synchronization control system 100 based on an Android device in this embodiment includes at least: an Android device 110 and a USB signal generator 120; it may also include a smart lamp 130.

[0127] In this embodiment, the Android device 110 is used to capture video frames and audio streams from the screen based on the Android system's native screen recording API. The screen is divided into regions, and color samples are taken from each region of the video frame. Color data for each region is calculated based on the sampled data. The audio stream is subjected to Fourier transform to calculate the audio beat, identify the rhythmic features of the audio, and generate beat data. The USB signal generator 120 establishes communication connections with both the Android device 110 and the smart lamp 130, and sends the color data and beat data to the smart lamp 130. The smart lamp 130 then adjusts the corresponding color and flashing frequency based on the received color data and beat data to achieve synchronization with the image and sound. The smart lamp 130 adjusts the corresponding color and flashing frequency based on the received color data and beat data to achieve synchronization with the image and sound.

[0128] The Android device 110 is an Android network TV or player, running screen recording and data processing software to perform calculations such as real-time capture of screen images and audio streams, color analysis of screen regions, and audio beat detection. The USB signal generator 120 (USB dongle) serves as a signal transmitter, connected to the Android device 110 via a USB interface, receiving color and beat data sent by the Android device 110 and converting them into wireless signals for transmission. The smart lamp 130 serves as a lighting execution terminal, receiving wireless control signals sent by the USB signal generator 120 and adjusting the color and flashing frequency of the light according to the control signals to achieve synchronized changes in screen color and audio rhythm with the content played on the Android device 110. The Android device 110, USB signal generator 120, and smart lamp 130 together constitute a complete Android-based audio-visual synchronization control system 100. The data interaction and signal flow between the components are described in the corresponding descriptions in the aforementioned method embodiments and will not be repeated here.

[0129] For example, the audio-visual synchronization control method, apparatus, and system based on Android devices in this embodiment can be applied to the following scenarios:

[0130] Scene 1: Synchronized sound and light in a movie-watching setting

[0131] When a user watches a movie on an Android smart TV, the screen and audio are captured in real time via the Android native screen recording API. For the video, the screen is divided into several grid areas, and pixel sampling and color analysis are performed on each area to calculate the average RGB value and dominant hue of each area, obtaining color distribution data for different areas of the screen. For the audio stream, the audio signal is converted from the time domain to the frequency domain using a Fast Fourier Transform, and the spectral energy distribution of each frame is calculated. Peak detection and energy envelope analysis are used to extract the beat timing position and beat intensity level. The aforementioned color and beat data are sent to the smart lamp 130 via a USB signal generator 120, driving the smart lamp 130 to change color with the dominant hue of the movie screen and to move dynamically with the beat of the movie's background music. This achieves precise synchronization between the movie screen color, background music rhythm, and lighting effects, creating an immersive home theater viewing experience for the user.

[0132] Scene 2: Sound and light synchronization in a music playback scenario

[0133] When a user plays music on an Android player, the audio stream is captured via a screen recording API. After frame segmentation and windowing preprocessing, the spectral amplitude and energy distribution are extracted using a Fast Fourier Transform. Furthermore, the beat position is identified through energy envelope peak detection, the beat interval is calculated to determine the BPM value, and the intensity level of each beat is evaluated based on the peak height and frequency domain energy distribution. Simultaneously, the emotional color of the music is identified based on the spectral centroid and spectral bandwidth. Music segments dominated by high-frequency energy are mapped to cool-toned lighting, while those dominated by low-frequency energy are mapped to warm-toned lighting. The beat and color information are sent to the smart light fixture 130 via a USB signal generator 120, controlling the smart light fixture 130 to flash synchronously with the music rhythm and dynamically adjust the light color to match the music atmosphere, creating an immersive lighting effect that blends with the played music content. The above two scenarios are merely exemplary application scenarios in this implementation. This embodiment is also applicable to various media playback scenarios based on Android devices, such as synchronized response of game visuals and sound effects, and online video playback, which will not be listed here.

[0134] In summary, the audio-visual synchronization control method, apparatus, and system based on Android devices in this embodiment have the following beneficial effects:

[0135] First, this application utilizes the existing screen recording API of Android devices to complete audio and video data acquisition without the need for external cameras or microphones or other additional sensing devices. This reduces hardware procurement and deployment costs, avoids privacy risks associated with environmental sound pickup, and allows the lighting to respond instantly to the media content, achieving a low-threshold and lightweight system deployment.

[0136] Second, this application performs grid division, color sampling, and dominant color clustering on the video image to capture color features, while performing Fourier transform, energy envelope extraction, and peak detection on the audio stream to identify the beat. The dual-dimensional data fusion drives the lamp to change color with the image and move dynamically with the rhythm, achieving precise coordination of image, sound, and light.

[0137] Third, this application automatically analyzes the color and rhythm characteristics of media content through algorithms, eliminating the need for users to manually select scenes or adjust lighting parameters throughout the process. The lighting always actively adapts to the changes in the currently playing content, completely liberating users from tedious manual control and significantly reducing operating costs and the barrier to entry.

[0138] The fourth application achieves communication bridging with the lamps through a USB signal transmitter. Users only need to insert the transmitter into the USB-C port of the Android device and turn on the lamp power, and the system will automatically establish a communication link. No network configuration or gateway settings are required. It is plug-and-play and supports flexible expansion with multiple lamps, which is convenient for market promotion.

[0139] Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0140] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for controlling the synchronization of sound and light based on an Android device, characterized in that, include: Capture video frames and audio streams from the screen using the Android system's native screen recording API; The screen is divided into regions, color samples are taken from each region in the video frame, and color data for each region is calculated based on the sampled data. Perform a Fourier transform on the audio stream, calculate the audio beat, identify the rhythmic features of the audio, and generate beat data; A communication connection is established with a USB signal generator, and the color data and the beat data are sent to the smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

2. The sound and light synchronization control method based on an Android device according to claim 1, characterized in that, The method of capturing screen video frames and audio streams based on the Android system's native screen recording API includes: Obtain real-time capture permissions for screen data and system audio through the Android system's native screen recording API; Create a virtual display module to mirror the current screen display content of the Android device, set the display resolution and pixel density, and capture video frames from the screen through the image data reading interface; The system continuously reads and caches the audio stream currently output by the Android device through the system audio data recording interface.

3. The sound and light synchronization control method based on an Android device according to claim 1 or 2, characterized in that, Also includes: This also includes recording parameter optimization; The optimization of the recording parameters includes: The system monitors the hardware performance status of the current Android device in real time and dynamically adjusts the resolution and frame rate of video recording based on the hardware performance status to balance recording quality and system resource usage. The video frames are encoded using a hardware encoding acceleration method.

4. The sound and light synchronization control method based on an Android device according to claim 1, characterized in that, The step of dividing the screen into regions, sampling the color of each region in the video frame, and calculating the color data of each region based on the sampled data includes: The screen is divided into N×M grid regions. The pixel range of each region is determined according to the screen resolution and the number of grids. Pixels in each region are sampled uniformly. For each region, calculate the RGB average value of all sampled pixels, convert the RGB values ​​to the HSV color space, use the K-means clustering algorithm to identify the dominant color tone of each region, and calculate the standard deviation and saturation distribution statistical characteristics of the colors.

5. The sound and light synchronization control method based on an Android device according to claim 1, characterized in that, The step of performing a Fourier transform on the audio stream to calculate the audio beat intensity and audio features, and generating beat data, includes: Perform a Fast Fourier Transform on each frame of data to convert the time-domain signal into a frequency-domain signal. Calculate the spectral amplitude and phase information of each frame based on the frequency-domain signal and analyze the energy distribution of different frequency components. The energy value of each frame is calculated based on the energy distribution, an energy envelope curve is generated, the peak point in the energy envelope curve is identified, the number of beats per minute is calculated, and the audio beat intensity is calculated based on the peak point and the number of beats per minute. The average energy, rate of change of energy, spectral centroid, spectral bandwidth, zero-crossing rate, and autocorrelation function of the audio are extracted as audio features. The beat data includes the audio beat intensity and the audio characteristics.

6. The sound and light synchronization control method based on an Android device according to claim 5, characterized in that, Before performing the Fast Fourier Transform, the process includes: after the audio stream is preprocessed by normalization and denoising, the audio stream is divided into frames of fixed length and a Hanning window is applied to each frame to perform a Fast Fourier Transform on the windowed frame data.

7. The sound and light synchronization control method based on an Android device according to claim 1, characterized in that, Before sending the color data and the beat data to the smart lamp via the USB signal generator, the method further includes: The color data and the beat data are packaged according to a custom frame format, which includes a synchronization word, frame type, length, data segment, and checksum.

8. A sound and light synchronization control device based on an Android device, characterized in that, include: The capture module captures video frames and audio streams from the screen based on the Android system's native screen recording API. The color data acquisition module is used to divide the screen into regions, sample the colors of each region in the video frame, and calculate the color data of each region based on the sampled data. The beat data acquisition module is used to perform Fourier transform on the audio stream, calculate the audio beat, identify the rhythmic features of the audio, and generate beat data. The communication connection module is used to establish a communication connection with the USB signal generator and send the color data and the beat data to the smart lamp through the USB signal generator, so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

9. A sound and light synchronization control system based on an Android device, characterized in that, include: Android devices are used to capture video frames and audio streams from the screen based on the native screen recording API of the Android system, divide the screen into regions, sample the color of each region in the video frame, and calculate the color data of each region based on the sampled data. Perform a Fourier transform on the audio stream, calculate the audio beat, identify the rhythmic features of the audio, and generate beat data; The USB signal generator establishes communication connections with both the Android device and the smart lamp, and sends the color data and the beat data to the smart lamp so that the smart lamp can adjust the corresponding color and flashing frequency according to the received color data and beat data to achieve synchronization with the picture and sound.

10. The acoustic-optical synchronization control system based on an Android device according to claim 9, characterized in that, Also includes: The intelligent lighting fixture is used to adjust the corresponding color and flashing frequency according to the received color data and beat data, so as to achieve synchronization with the picture and sound.

Images