Low-latency dance-assist practice device and control method thereof

Abstract

The application discloses a low-delay dance auxiliary practice device and a control method thereof. The device comprises a processor integrating a hardware video acceleration unit and a display output controller, an image acquisition module connected through a video input interface, and a wireless receiving module supporting a wireless screen projection protocol. The core idea is to use the built-in hardware acceleration unit of the processor to combine the screen projection decoding frame and the local acquisition frame into a single combined picture in a zero-copy manner in the hardware layer in real time, and to build a full hardware straight-through pipeline from the data source to the display output interface. The end-to-end delay is stabilized at 50-100 ms, which is lower than the human visual perception threshold, thereby naturally ensuring that the screen projection teaching picture and the user real-time image are highly synchronized in time, and realizing a'mirror' follow-up practice experience. The scheme is originally compatible with standard screen projection protocols such as AirPlay, Miracast and DLNA, does not require an additional APP, can be realized by using a general system-level chip, is low in cost, and can be widely applied to home dance teaching, fitness follow-up practice and the like.

Description

Technical Field

[0001] This invention relates to the field of smart home and motion-assisted training equipment technology, and in particular to a dance-assisted practice device and its control method that utilizes hardware video acceleration to achieve low-latency image synthesis. Background Technology

[0002] With the increasing popularity of home fitness and online dance instruction, users need practice tools that allow them to compare and correct their movements in real time while following video lessons. The ideal user experience is for users to see instructional videos on their TVs while simultaneously viewing their own live movements on the same screen, displayed in split-screen or picture-in-picture format, with perfect synchronization and no perceptible delay. Current common solutions suffer from the following main drawbacks.

[0003] The latency of a purely software solution is uncontrollable. Using a mobile phone to play instructional videos and wirelessly project them to a TV while simultaneously taking a selfie with the phone's front-facing camera is currently the most common practice. However, this approach has multiple sources of latency in its data link: video encoding at the projection end introduces a 50-150ms latency; wireless transmission buffering introduces a 20-80ms latency; software decoding at the receiving end introduces a 30-100ms latency; camera preview and screen overlay are synthesized by GPU software, introducing an additional 30-60ms latency; and display output refresh introduces a 16-33ms latency. The cumulative latency across these stages typically exceeds 300ms, and in severe cases, can reach over 500ms. For dance practice, a rhythmic error exceeding 100ms can cause the learner to misalign their movements, rendering the practice ineffective.

[0004] The software has low synthesis efficiency and high system resource consumption. When performing multi-channel video compositing, existing devices typically rely on the CPU or general-purpose GPU to perform image scaling, color space conversion, and pixel blending. While functionally feasible, this approach suffers from the following problems: (a) each compositing operation requires copying multiple video frames from their respective buffers to the compositing buffer, resulting in significant memory and bandwidth consumption; (b) continuous high CPU / GPU load leads to severe overheating; and (c) the scheduling of compositing tasks is affected by operating system task switching, resulting in unstable output frame intervals and periodic stuttering.

[0005] Poor compatibility of screen mirroring protocols Most dedicated auxiliary devices rely on self-developed apps or proprietary protocols, requiring users to install additional software on their phones and keep the app version synchronized with the device. Standard screen mirroring protocols native to mobile operating systems (such as AirPlay and Miracast) are usually excluded from device capabilities, resulting in high barriers to entry and low user acceptance.

[0006] Dedicated hardware solutions are too expensive Professional equipment on the market with low-latency multi-view compositing capabilities (such as broadcast consoles and video splitters) mostly uses FPGAs or dedicated video processing chips to achieve hardware compositing. Although such solutions have low latency, the chip cost is usually tens to hundreds of yuan, and the development cycle is long and the flexibility is poor, making it difficult to promote in consumer-grade dance aids.

[0007] Limitations of camera field of view and installation method Standard USB cameras have a narrow field of view (typically 60°~80°), meaning even slight user movements will cause the camera to fall out of frame, making it impossible to capture full-body movements. Furthermore, USB cameras are usually standalone peripherals, requiring additional cables and stands, and users need to manually adjust the angle, resulting in a poor user experience.

[0008] The common thread among these shortcomings is that existing solutions fail to achieve a fully streamlined hardware pipeline within a single device, encompassing "wireless screen projection reception → hardware decoding → local real-time acquisition → hardware zero-copy synthesis → display output." Any software intervention in any intermediate step introduces uncontrollable latency and jitter. Summary of the Invention

[0009] Technical problems to be solved This invention aims to solve the following technical problem: how to construct a multi-channel real-time video synthesis output link with a stable end-to-end latency of 40-100ms without using FPGA or dedicated video processing chips. The low latency inherently ensures the synchronization of the projected video stream and the local camera video stream, making the time deviation between the teaching actions seen on the screen and the user's actual actions lower than the human visual perception threshold, achieving a "mirror-like" real-time follow-up experience. Simultaneously, it is natively compatible with mainstream standard wireless projection protocols to solve the problems of high latency, low software synthesis efficiency, and poor protocol compatibility in existing solutions, and makes the device feasible for consumer-grade mass production.

[0010] Technical solution The core concept of this invention lies in utilizing the hardware video acceleration unit integrated within the processor to scale, overlay, and synthesize two independent video data streams—the "wireless projection decoding frame" and the "local camera capture frame"—at the hardware level, constructing a zero-copy direct pipeline from the video source to the display. Specifically, this invention achieves the above objective through the following three technical means.

[0011] Method 1: Zero-copy connection between hardware decoding and hardware synthesis. The encoded video stream received by the wireless receiving module is directly decoded into raw frames by the processor's built-in hardware video decoder. These frames are stored in a shared memory buffer driven by the decoder. The handle to this buffer is directly passed to the hardware video acceleration unit, which accesses the buffer via its physical address without CPU copying.

[0012] Method 2: Hardware pass-through between image acquisition and display output. The image acquisition module sends digital image data to the processor via the video input interface. After preprocessing by the image signal processor (ISP), the data is also written to a shared memory buffer. The handle of this buffer is directly passed to the hardware video acceleration unit, which shares the same physical memory space with the projection decoding frame. The hardware acceleration unit directly reads both and performs the compositing operation.

[0013] Method 3: The composited output frame is directly scanned by the display output controller. The hardware video acceleration unit writes the composited image into a frame buffer that can be directly addressed by the display output controller. The display output controller (such as an HDMI TX controller) outputs the contents of the frame buffer to an external display using hardware scanning. There is no software interruption or copying between the composite and the output.

[0014] The above three components together form a zero-copy hardware pipeline from the data source to the display. The CPU is only responsible for task scheduling and parameter configuration and does not participate in any pixel data processing. This fundamentally eliminates the uncontrollable latency problem caused by multiple memory copies, task switching, and scheduling jitter in traditional software solutions.

[0015] The synchronization guarantee of the two video streams in this invention does not rely on any explicit timestamp pairing, frame alignment algorithm, or synchronization signal injection. The principle is as follows: when the input delays of both camera videos are compressed to a sufficiently small value (each ≤ 50ms), the time deviation between them is naturally limited to a very small range (typically less than a frame interval of 33ms), far below the time alignment threshold perceptible to human vision (approximately 120~150ms). Hardware decoding avoids the CPU scheduling overhead of software decoding, direct pass-through of the video input interface avoids queuing, the RGA / layer mixer avoids bandwidth contention during memory copying, and direct scanning of the display controller avoids secondary frame buffer relocation. These measures work together to compress the delays of both the projection link and the camera link to below 50ms, thus naturally reducing the deviation between the two streams to less than a frame interval.

[0016] Equipment Solution This invention also provides a control method for the aforementioned equipment. From the perspective of equipment control flow, this method specifies the necessary steps for achieving low-latency synthesis.

[0017] Step S1 (Initialization Phase): After the device powers on, the processor loads the operating system and initializes the video input interface controller, hardware video decoder, hardware video acceleration unit, and display output controller. Simultaneously, the wireless communication module's protocol stack service is started, and the device enters a projection waiting state.

[0018] Step S2 (Connection Establishment Phase): The external smart device discovers this device via the wireless LAN and initiates a screen mirroring connection request. After the wireless communication module completes the protocol handshake, it begins receiving the encoded video stream. The processor sends the received encoded data packets to the input queue of the hardware video decoder.

[0019] Step S3 (Parallel Data Stream Establishment Phase): While the projected video stream is being decoded, the image acquisition module continuously sends local image data to the processor through the video input interface. The acquisition / reception and decoding / preprocessing of the two data streams are executed asynchronously and in parallel, without blocking each other.

[0020] Step S4 (Hardware Compositing Trigger Stage): After the hardware video decoder outputs the first decoded frame and the image acquisition module outputs the first local frame, the processor sends a compositing configuration command to the hardware video acceleration unit. This configuration command specifies the display area, scaling ratio, and layer overlay order of each video stream in the combined image.

[0021] Step S5 (Continuous Composite Output Stage): Following configuration commands, the hardware video acceleration unit automatically performs scaling and overlay operations on each subsequent frame, writing the composite frame to a frame buffer directly addressable by the display output controller. The display output controller scans this buffer at a fixed refresh rate and outputs the composite image to an external display. The compositing and output rhythm is driven by a hardware clock and is not affected by operating system task scheduling.

[0022] Step S6 (End Phase): When the wireless communication module detects that the screen projection connection is disconnected or the user actively stops it, the processor sends a stop command to the hardware video acceleration unit, releases the relevant buffer resources, and the device returns to the waiting state of step S1.

[0023] The core features of the above control method are: the asynchronous parallel establishment of the two data streams in step S3, and the pure hardware-driven mechanism for synthesis and output in step S5. These two points together ensure the stability and predictability of the end-to-end latency.

[0024] Control methods and schemes The present invention also provides a control method for the above-mentioned dance-assisted practice device, including:

[0025] It receives encoded video streams sent by external mobile devices via wireless projection protocols and decodes them using the processor's built-in hardware decoder.

[0026] Simultaneously, user action videos are captured in real time through the video input interface to generate local image sequences.

[0027] The processor's hardware video acceleration unit is used to synthesize the decoded external video sequence and the local image sequence into a split-screen composite image in real time.

[0028] The combined image is output to an external display via a display output controller.

[0029] Furthermore, prior to the compositing step, the method includes: determining the split-screen layout based on a preset template or user selection, and allocating a display area and scaling ratio for each video stream within the combined screen. The hardware decoder output and the output of the local image sequence share memory via dmabuf or zero-copy mode, and are directly fed into the hardware video acceleration unit. This control method focuses on device hardware-level initialization, data stream scheduling, and compositing triggering, and does not involve specific video compositing algorithms.

[0030] Beneficial effects Compared with the prior art, the present invention has the following beneficial effects.

[0031] The low-latency architecture naturally enables time synchronization between the two video streams, giving users a real-time practice experience similar to "looking in a mirror." The projected video stream and the camera video stream share the same synthesis and output path after entering the hardware video acceleration unit, and their latency from synthesis to screen display is exactly the same. Therefore, the time deviation between the two streams depends only on the difference in their input latency before reaching the synthesis unit. This invention compresses the latency of each input stream to less than 50ms through a zero-copy hardware pipeline, and the difference is naturally less than a frame interval (about 33ms), far below the human visual perception threshold for time alignment (about 120~150ms). Users can see a frame-by-frame comparison of the teaching actions and their own actions on the same screen without any explicit synchronization mechanism, as if facing a real-time reflecting mirror, achieving a natural closed loop of "action imitation → visual feedback → action adjustment".

[0032] The end-to-end latency is stable at 40~100ms, which meets the timing requirements for dance practice. Pixel data is not copied by the CPU; synthesis and output are driven by a hardware clock, and latency does not fluctuate with system load or scheduling jitter. For dance movements with beat intervals of 300-500ms, the corresponding time deviation is less than 1 / 3 of a beat, and the follower will not experience a perceptible lag.

[0033] Natively compatible with mainstream screen mirroring protocols, no installation required: The wireless communication module supports standard protocols such as AirPlay, Miracast, and DLNA, as well as proprietary protocols such as LePlay. Users can connect through the native screen mirroring function of their mobile phone system without the need for an additional app.

[0034] Implemented based on a general-purpose SoC, it has low BOM cost and is ready for mass production. All video processing is completed by the built-in hardware unit of the mid-to-low-end system-on-a-chip, without the need for FPGA or dedicated video processing chips, and the cost of the whole machine can be controlled within 100 yuan.

[0035] The solution is not dependent on any specific hardware platform and has strong versatility. The "hardware decoding - zero-copy synthesis - hardware output" pipeline architecture is applicable to any processing unit with hardware video acceleration synthesis capabilities, including single SoC solutions, multi-chip discrete solutions, and combinations of different operating systems and projection protocol stacks. Attached Figure Description

[0036] Figure 1 This is a hardware structure block diagram of an embodiment of the dance-assisted practice device of the present invention; Figure 2 This is a schematic diagram of the video data processing and compositing pipeline in this invention; Figure 3 This is a simplified flowchart of an embodiment of the control method of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the present invention clearer, a detailed description is provided below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0038] Example 1: Implementation based on Rockchip RK3566 and OV5647.

[0039] Hardware structure (refer to) Figure 1 ) The dance-assisted practice device in this embodiment includes...

[0040] Processor 1: Employs the Rockchip RK3566 system-on-a-chip. This chip integrates a quad-core Cortex-A55 CPU (as the control core), a Mali-G52 GPU, a hardware video decoder (supporting H.264 / H.265 4K@60fps decoding), an RGA hardware 2D acceleration engine, a MIPI-CSI host controller, and an HDMI 2.0 display output controller. The RGA hardware 2D acceleration engine is the "hardware video acceleration unit" of this invention, and the HDMI 2.0 display output controller is the "display output controller" of this invention. The RK3566, as a single processing unit, simultaneously implements all functions of image acquisition interface control, hardware video decoding, hardware video acceleration synthesis, and display output control.

[0041] Image acquisition module 2: Employs an OmniVision OV5647 CMOS image sensor paired with a 150° field-of-view wide-angle lens. The sensor connects to the RK3566's MIPI CSI host via a 15-pin FPC cable through a MIPI-CSI interface (4-lane). The OV5647 outputs image data, which is processed by the RK3566's internal ISP into an NV12 format YUV image sequence with a resolution of 1920×1080@30fps.

[0042] Wireless communication module 3: It adopts the Realtek RTL8822CE Wi-Fi / Bluetooth dual-band module, which connects to the RK3566 through the SDIO interface and operates in the 2.4GHz / 5GHz dual-band.

[0043] The protocol stack processes of AirPlay, Miracast, DLNA, and LeBao screen mirroring SDK run on the processor's CPU as user-mode services. The wireless communication module 3 submits the received encoded data packets to the processor via the SDIO interface. The protocol stack running on the processor performs parsing and session management, and sends the encoded video data to the input queue of the hardware video decoder.

[0044] In use, the device is connected to an external display device (such as a TV or monitor) via an HDMI cable, and the external display device displays the composite image. The external display device is not part of this device.

[0045] Data flow and synthesis pipeline (see reference) Figure 2 ) Figure 2 This embodiment demonstrates in detail the complete data flow path of the two video data streams from the source to the display. The following sections describe each data flow direction in turn.

[0046] First data stream – screen-cast video stream: Phase A (Wireless Reception): The external mobile phone sends an encoded video stream via the AirPlay protocol. Wireless receiving module 3 receives the encoded data packets and submits the data to the processor via the SDIO interface. After the screen mirroring protocol stack process running on the processor completes protocol parsing, it writes the encoded video data into memory buffer A; Stage B (Hardware Decoding): The RK3566's hardware video decoder directly reads encoded data from buffer A via DMA, decodes it into raw frames, and writes them to the dmabuf buffer B driven by the decoder. This process is entirely completed by hardware; the CPU is only responsible for submitting the decoding task to the decoder and does not participate in pixel data handling. The format of the decoded output frame depends on the hardware configuration and can be a YUV series format (NV12, YUV420, etc.) or an RGB series format (RGB888, ARGB, etc.), as long as it meets the input requirements of the subsequent hardware synthesis unit. Phase C (Handle Transfer): After decoding is complete, the application layer obtains the dmabuf file descriptor of buffer B and directly passes the handle to the RGA hardware 2D acceleration engine driver. The RGA driver uses this handle to resolve the physical address of buffer B, achieving zero-copy connection from decoder output to synthesizer input.

[0047] Second data stream – Local camera video stream: Phase D (Image Acquisition): The OV5647 image sensor continuously outputs digital image data through its video input interface. The RK3566's video input interface controller receives this data stream in hardware and writes it directly into memory buffer C. Stage E (ISP Preprocessing): The RK3566's internal ISP reads image data from buffer C, performs preprocessing operations such as demosaicing, white balance, color correction, and format conversion, and outputs frame data that meets RGA input requirements, writing it to the dmabuf buffer D. ISP processing is also completed by hardware; preprocessing parameters are configured once by the CPU, and the processing process requires no CPU intervention. Stage F (Handle Transfer): Similar to Stage C, the dmabuf handle of buffer D is directly passed to the RGA driver.

[0048] Synthesis and Output Stages: Stage G (Hardware Compositing): The RGA hardware 2D acceleration engine reads data from buffer B (projection decoding frames) and buffer D (local acquisition frames) according to the compositing configuration commands pre-issued by the CPU, performs scaling operations (scaling the two 1920×1080 screens to 960×1080 respectively), and then overlays the scaled images into buffer E according to the left-right split-screen layout. Buffer E is a frame buffer that can be directly addressed by the display output controller. The entire scaling and overlay process is completed entirely by RGA hardware, without going through the CPU; Stage H (Display Output): The HDMI 2.0 display output controller scans buffer E at a fixed refresh rate (e.g., 60Hz), encodes the composite image into a display signal, and outputs it through the HDMI interface. The external display device receives this signal and displays the image. The operation of the display controller is driven by a hardware pixel clock and is independent of operating system scheduling.

[0049] Key Delay Analysis: In the above data stream, pixel data undergoes one hardware synthesis operation (RGA) from the screen projection decoding output (buffer B) to the final display (buffer E), taking approximately 1-3 ms. The process from camera capture to display output involves two hardware processing steps: ISP processing and RGA synthesis, taking approximately 5-10 ms. Adding the inherent latency of wireless transmission and decoding, the end-to-end latency of the entire link remains stable at around 65 ms and does not fluctuate significantly with changes in CPU load.

[0050] Control process (refer to) Figure 3 ) Figure 3 This demonstrates the complete control process of the device in this embodiment from startup to outputting the composite image.

[0051] Step S1 (Initialization): After the device powers on, the bootloader loads the Linux kernel. The kernel initializes the video input interface controller, RGA driver, HDMI display controller driver, and SDIO wireless interface. The user space starts the systemd service, loads the AirPlay / Miracast / LeBo protocol stack process on the CPU (running as a user-space service), and starts the camera acquisition process. The device enters the screen projection waiting state.

[0052] Step S2 (Connection Establishment): The user's mobile phone enables screen mirroring, discovers the device via mDNS / Bonjour service, and initiates a connection. After the wireless receiving module completes the protocol handshake, the mobile phone begins pushing the encoded video stream. The protocol stack process running on the processor parses the encoded data packets and sends the encoded video data sequentially into the input queue of the hardware decoder.

[0053] Step S3 (Parallel Data Stream Establishment): While the projected video stream begins decoding, the image acquisition module continuously sends digital image data through the video input interface. The acquisition framework submits the acquisition task to the ISP, which outputs frame data in units of frames. The two data streams run completely in parallel, without waiting for each other.

[0054] Step S4 (Synthesis Trigger): When the application layer detects that both the screen-casting decoding frame (buffer B) and the local acquisition frame (buffer D) are ready, it sends a synthesis configuration command to the RGA driver. This command includes: the dmabuf handles of input buffers B and D, their respective scaling ratios, their respective output positions, and the handle of output buffer E.

[0055] Step S5 (Continuous Composite Output): After receiving the configuration command, the RGA automatically performs scaling and overlay operations upon the arrival of each subsequent frame, without requiring the CPU to repeatedly issue commands. After completing the composite of each frame, the RGA sends a buffer update signal to the display controller. The display controller scans for new frames in the next vertical synchronization cycle and sends them to the external display device through the display output interface. The rhythm of composite and output is driven by both the RGA hardware clock and the display pixel clock.

[0056] Step S6 (End): When a screen mirroring connection is detected to be disconnected, the application layer sends a stop command to the RGA, releases buffers B, D, and E, and the device returns to state S1. The camera acquisition process can choose to continue running or stop altogether.

[0057] Those skilled in the art should understand that the specific chip and module models described above are merely examples and are not intended to limit the scope of protection of this invention. For example, the processor can also be other processors that integrate a hardware layer synthesizer, MIPI-CSI interface, and native HDMI interface, such as the Allwinner H618 or Amlogic S905 series. The camera interface is not limited to MIPI-CSI; it can also be a USB interface, DVP parallel interface, or LVDS interface. Correspondingly, the camera module can be replaced with a UVC camera module or other image sensors with equivalent performance. The wireless projection receiving module can also be expanded to a module that supports proprietary protocols. All of these fall within the scope of equivalent substitutions of this invention and can achieve the same technical effect.

[0058] Implementation based on Allwinner H618 and IMX219 This embodiment is an equivalent alternative to Embodiment 1, the difference being the selection of the processor and image sensor, as well as the different implementation of the hardware synthesis mechanism.

[0059] Processor 1: Employs the Allwinner H618 system-on-a-chip. Internally, it integrates a quad-core Cortex-A53 CPU, a Mali-G31 GPU, a hardware video decoder, and the DE2.0 display subsystem. Unlike the RK3566's independent RGA 2D accelerator, the H618's DE2.0 display subsystem uses a hardware composer mechanism for hardware compositing. This mechanism allows for the allocation of an independent display layer (Layer / Plane) to each video stream, with each layer having its own frame buffer and independent scaling control. During scan output, the display controller automatically reads data from all visible layers and performs pixel blending. This "layer blending" is essentially equivalent to the RGA's "2D compositing" function, both belonging to the "hardware video acceleration unit" of this invention.

[0060] Image acquisition module 2: It adopts a Sony IMX219 sensor with a field of view of 120°. It is connected through a video input interface and outputs 1080p@30fps data. The data is pre-processed by the H618's internal ISP to meet the frame format requirements of the synthesis unit input.

[0061] Data flow path and Figure 2 The pipeline shown is completely identical in principle, with the only difference being the implementation mechanism of "Stage G (hardware compositing)": In H618, the projection decoding frame and the camera frame are assigned as Layer 0 and Layer 1 respectively. The display subsystem DE2.0 automatically completes the overlay of the two layers by hardware during scanning output, without the need for an independent 2D acceleration engine to perform explicit compositing operations.

[0062] Control process and Figure 3 The process is the same, except that the content of the "composition configuration command" in step S4 is the layer parameter configuration (including the frame buffer address, display area, scaling factor and stacking order of each layer), rather than the operation command of RGA.

[0063] The measured end-to-end latency is approximately 80ms. This embodiment demonstrates that regardless of whether hardware compositing is implemented through a separate 2D acceleration engine or through the layer mixer of the display controller, as long as the compositing operation is completed in hardware and does not involve CPU copying, it falls under the equivalent implementation method claimed in this invention.

[0064] Example 3: Chipset-based discrete solution This embodiment illustrates that the "processing unit" in this invention is not limited to a single system-on-a-chip. The following is a functionally equivalent chipset solution.

[0065] Processor: NXP i.MX 8M Mini (containing a quad-core Cortex-A53) is used for system scheduling and parameter configuration, and runs the screen projection protocol stack.

[0066] Hardware video decoder: It adopts an independent Rockchip RK628D video decoding chip, which is connected to the processor through the PCIe interface and is responsible for decoding the encoded video stream received by the wireless receiving module into raw frames.

[0067] Hardware synthesizer: It uses an independent FPGA (such as Lattice ECP5), which is connected to the decoder and camera interface through a parallel bus. It receives two frame data streams, performs scaling and overlay, and outputs a synthesized frame.

[0068] Display output controller: It adopts an independent HDMI transmitter (such as ITE IT66121), receives the composite frame output by the FPGA through the RGB parallel interface, encodes it into a TMDS signal and outputs it through the display output interface.

[0069] Image acquisition module: Outputs digital video data through the video input interface and sends it directly to the FPGA.

[0070] Wireless receiver module: It connects to the processor via SDIO, receives the screen projection stream, and then forwards it to the independent decoding chip.

[0071] Although this chipset solution physically consists of four independent chips, it functionally fully implements the processing unit functions of "image acquisition interface control, hardware video decoding, hardware video acceleration synthesis, and display output control." The data flow also satisfies the "zero-copy" characteristic: the FPGA directly reads data from the decoder and camera interface, and the synthesized frame is directly output to the display output controller; the processor does not participate in any pixel data transfer. Therefore, this solution is an equivalent implementation of the present invention.

[0072] Other variations It should be understood that the specific chip models, sensor models, operating systems, communication modules, and synthesis mechanisms in the above embodiments are merely illustrative. Any device that meets the following combination of functional features falls within the protection scope of the claims of this invention: The processing unit has hardware video decoding capabilities and hardware video accelerated synthesis capabilities; The image acquisition module transmits image frames in real time via a wired digital video interface; The wireless receiving module supports real-time screen mirroring protocols or media streaming protocols based on wireless LAN. The compositing operation is performed by hardware, and pixel data is not copied by the CPU. The composite image is output to an external display device via the display output interface through the display output controller.

[0073] For example, the processor can also be Amlogic A311D, Qualcomm QCS610, HiSilicon Hi3519, etc.; the image acquisition module can also be OV8865, AR0234, GC4653, etc.; the video input interface can also be MIPI-CSI, DVP parallel interface, LVDS, USB video interface, etc.; the wireless projection protocol can also be Google Cast, Huawei Cast+, Samsung Smart View; the composite layout can also be split-screen or picture-in-picture; the display output interface can also be DisplayPort, V-by-One, or embedded eDP interface. The operating system can be Linux, Android, RTOS, or bare-metal program. All these variations are equivalent embodiments of the present invention.

Claims

1. A low-latency dance practice aid device, characterized in that, include: The processor integrates a hardware video acceleration unit and a display output controller; The image acquisition module is connected to the processor via a video input interface and is used to acquire the user's dynamic images in real time; The wireless receiving module is used to receive video streams sent by external smart devices via wireless screen mirroring protocols; The processor uses the hardware video acceleration unit to combine the real-time image captured by the image acquisition module with the video stream received by the wireless receiving module into a single combined image in real time, and outputs it to an external display through the display output controller.

2. The device according to claim 1, characterized in that, The processor is a processing unit with functions including image acquisition interface control, hardware video decoding, hardware video acceleration synthesis, and display output control.

3. The device according to claim 2, characterized in that, The processing unit is one of Rockchip RK3566, Allwinner H618, Amlogic A311D or Qualcomm QCS610.

4. The device according to claim 2, characterized in that, The processing unit is a chipset consisting of at least two of a central processing unit, a hardware video decoder, a hardware synthesizer, and a display controller.

5. The device according to claim 1, characterized in that, The hardware video acceleration unit includes at least one of the following: an RGA hardware 2D accelerator, a hardware layer mixer of the display subsystem, or a Plane compositing unit in DRM KMS.

6. The device according to claim 1, characterized in that, The video input interface is a wired interface capable of transmitting image frames in real time in digital form.

7. The device according to claim 6, characterized in that, The video input interface is one of the following: MIPI-CSI interface, DVP parallel interface, LVDS interface, or USB video interface.

8. The device according to claim 1, characterized in that, The image acquisition module includes an image sensor and a wide-angle lens with a field of view ranging from 120° to 150°.

9. The device according to claim 8, characterized in that, The image sensor is a CMOS image sensor.

10. The device according to claim 1, characterized in that, The processor is also used to preprocess image data from the image acquisition module to generate frame data that meets the input requirements of the hardware video acceleration unit.

11. The device according to claim 10, characterized in that, The preprocessing includes at least one of format conversion, image scaling, and color space transformation.

12. The device according to claim 1, characterized in that, The wireless projection protocol is a real-time screen mirroring protocol or a media streaming protocol based on a wireless local area network.

13. The device according to claim 12, characterized in that, The wireless screen casting protocol includes at least one of AirPlay, Miracast, DLNA, Google Cast, and LeBao's proprietary protocol.

14. The device according to claim 1, characterized in that, The display output controller outputs the combined image via an HDMI interface, a DisplayPort interface, a V-by-One interface, or an embedded display interface.

15. The device according to claim 1, characterized in that, The combined screen can be a left-right split screen, a top-bottom split screen, or a picture-in-picture screen.

16. The device according to claim 1, characterized in that, The image acquisition module's image acquisition and the wireless receiving module's video stream reception are performed asynchronously and in parallel.

17. The device according to claim 1, characterized in that, The processor uses the hardware video acceleration unit to combine the real-time image and the video stream into the combined image using a zero-copy method.

18. A control method for a low-latency dance-assisted practice device, applied to the device as described in any one of claims 1 to 17, characterized in that, include: It receives encoded video streams sent by external mobile devices via wireless screen mirroring protocols and decodes them using the processor's built-in hardware decoder. Simultaneously, user action videos are acquired in real time via a high-speed serial interface to generate local image sequences; The processor's hardware video acceleration unit is used to combine the decoded external video sequence with the local image sequence in real time to create a split-screen composite image. The combined image is output to an external display via a display output controller.

19. The control method according to claim 18, characterized in that, The process before compositing also includes: determining the split-screen layout based on a preset template or user selection, and assigning a display area and scaling ratio to each video stream in the combined screen.

20. The control method according to claim 18, characterized in that, The output of the hardware decoder and the output of the local image sequence are both sent directly to the hardware video acceleration unit without being copied by the CPU.

Images