Display device and human posture estimation method

Abstract

The application discloses a display device and a human body posture estimation method. The display device can access a collection device comprising a first camera and a second camera, and respectively collect RGB image data and event stream data of a target human body as bimodal data. After the display device acquires the bimodal data collected synchronously, a pre-constructed target posture estimation model is called. The target posture estimation model takes an original posture estimation model pre-trained by RGB as a main network basis, retains all pre-training parameters except a first convolution layer, and completes weight adaptation of the first convolution layer through cross-modal weight migration guided by RGB. The bimodal data is input into model fusion reasoning, and a human body posture estimation result containing multiple spine segment key points is output and displayed on a display. The application combines RGB texture information and event stream high-speed response characteristics, can improve spine posture estimation accuracy in a high-speed motion scene, and simultaneously reduces model training cost and data threshold.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for estimating human posture. Background Technology

[0002] Display devices are intelligent devices capable of presenting a user interface and supporting user interaction. Taking smart TVs as an example, smart TVs are television products based on Internet application technology, equipped with open operating systems and chips, possessing open application platforms, enabling two-way human-computer interaction, and integrating multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] In visual interaction scenarios such as motion-sensing interaction of display devices, home fitness guidance, and sports posture correction, multi-vertebral point human pose estimation is a core technology for achieving accurate motion analysis and human-computer interaction. Related technologies employ RGB frame-based cameras as the acquisition front end, acquiring continuous human images frame by frame. Based on a deep learning model, human appearance features are extracted from the images, and the coordinate information of each segment of the spine and joints throughout the body is regressed to complete human pose estimation. To improve the imaging effect in high-speed motion scenes, some solutions also introduce event cameras as supplementary acquisition devices. Features from RGB images and event streams are extracted separately and then fused. This, combined with a manually labeled dataset, trains the pose estimation model to achieve dual-modal fusion pose estimation output.

[0004] However, this fusion pose estimation method, which is based on RGB frame imaging and supplemented by event cameras, has the problem of insufficient accuracy in estimating human pose at multiple vertebral points in high-speed motion scenarios. It is difficult to meet the high-precision requirements for spinal motion capture in scenarios such as home fitness guidance and motion posture correction for display devices. Summary of the Invention

[0005] This application provides a display device and a human pose estimation method to solve the problem of low accuracy in multi-vertebral point human pose estimation.

[0006] In a first aspect, this application provides a display device, comprising: The monitor is configured to display the user interface; The device interface is configured to connect to a data acquisition device, which includes a first camera and a second camera. The first camera is used to acquire RGB image data of the target human body, and the second camera is used to acquire event stream data of the target human body. The controller is configured as follows: The RGB image data and the event stream data are obtained through the device interface to obtain bimodal data; The pre-built target pose estimation model is invoked; the target pose estimation model is based on the original pose estimation model pre-trained by RGB as the main network foundation, retains all pre-trained parameters of the original pose estimation model except for the first convolutional layer, and adapts the weights of the first convolutional layer through RGB-guided cross-modal weight transfer. The dual-modal data is input into the target pose estimation model, and after fusion inference by the target pose estimation model, the human pose estimation result is output; the human pose estimation result includes multiple key points of the spinal segment. The human pose estimation result is sent to the display, and the display is controlled to show the human pose estimation result.

[0007] The above technical solution has the following beneficial effects or advantages: By combining the texture detail advantages of RGB images with the high-speed response and motion blur-free characteristics of event cameras, high-precision spine pose estimation can be achieved in high-speed motion scenes, improving the shortcomings of pure RGB pose estimation, such as high latency and easy motion blur. Based on mature RGB pre-trained models, cross-modal weight transfer can be carried out without building a dual-modal network architecture from scratch. While retaining the original model's feature extraction capabilities, it can reduce model training costs and data requirements, thereby improving the accuracy of multi-vertebral point human pose estimation.

[0008] In some embodiments of this application, before the dual-modal data is input into the target attitude estimation model, the controller is further configured to: The RGB image data is normalized and distortion-reduced to obtain RGB input data; The asynchronous event stream data is aggregated according to a preset time window to count the number of brightness increase events and the number of brightness decrease events corresponding to each pixel position within the time window. A dual-channel polarity aggregation map is generated based on the number of brightness increase events and the number of brightness decrease events, serving as the event input data; the dual-channel polarity aggregation map includes a positive polarity channel and a negative polarity channel; The RGB input data and the event input data are spatially aligned so that they are in the same pixel coordinate system. The controller is configured to input the dual-modal data into the target attitude estimation model, specifically as follows: The RGB input data and the event input data in the same pixel coordinate system are input into the target pose estimation model.

[0009] The above technical solution has the following beneficial effects or advantages: The asynchronous, non-fixed-frame raw event stream is converted into a dual-channel structured feature map that matches the size of the RGB image, solving the format adaptation problem that event data cannot be directly input into convolutional neural networks. The dual channels are generated by polarity separation, which fully preserves the dynamic change information of pixel brightness rise and fall, and can give full play to the high temporal resolution advantage of the event camera. The pixel-level one-to-one correspondence of dual-modal data is achieved through spatial coordinate alignment, avoiding fusion errors caused by spatial misalignment, thereby ensuring the accuracy of dual-modal fusion inference.

[0010] In some embodiments of this application, the controller is further configured to: Samples are extracted from the target event dataset; Count the average number of pixel events in the positive and negative channels of each sample; The global information content ratio is calculated based on the average number of pixel events; the global information content ratio is the ratio of the average number of events in a single channel to the total average number of events in both channels, and the global information content ratio is used to guide the cross-modal weight fusion of the first convolutional layer.

[0011] The above technical solution has the following beneficial effects or advantages: Based on the proportion of channel information in real event data of the target scenario, the inherent distribution characteristics of the target data can be quantified, providing a reliable data foundation for subsequent cross-modal weight fusion and avoiding subjectivity and blindness in weight adaptation. By adopting a sampling statistical method, the amount of computation is reduced while ensuring data representativeness, and the channel ratio parameters that are adapted to the target dataset can be obtained quickly, making the execution efficiency of weight adaptation higher.

[0012] In some embodiments of this application, the controller adapts the weights of the first convolutional layer via RGB-guided cross-modal weight migration, specifically configured as follows: Based on the global information ratio, the original weights of the RGB three channels in the first convolutional layer are weighted and fused separately to obtain dual-channel event convolution weights; wherein, the RGB three channels include a first channel, a second channel, and a third channel, and the dual-channel event convolution weights include positive polarity event channel weights and negative polarity event channel weights; the positive polarity event channel weights are generated by weighted fusion of the weights of the first channel and the second channel, and the negative polarity event channel weights are generated by weighted fusion of the weights of the second channel and the third channel; The standard deviation of the dual-channel event convolution weights is normalized to make the output activation distribution of the dual-channel event convolution weights consistent with the pre-training output distribution of the original pose estimation model. The original weights and the dual-channel event convolution weights are concatenated along the channel dimension to form multi-channel convolution weights, thus completing the adaptation of the weights of the first convolutional layer.

[0013] The above technical solution has the following beneficial effects or advantages: Event channel weights are generated by weighted fusion of RGB weights, reusing the feature knowledge of the RGB pre-trained model. This eliminates the need to train the event branch network from scratch, thus narrowing the feature gap between modalities. Global standard deviation normalization ensures that the output activation distribution of the event channel is aligned with the original RGB pre-trained distribution, maximizing the preservation of the pre-trained feature extraction capability of the backbone network and avoiding significant degradation in model performance. Dual-modal adaptation is achieved by concatenating only the input layer channels, without modifying the original backbone network structure, ensuring compatibility.

[0014] In some embodiments of this application, before the target pose estimation model is trained, the controller is further configured as follows: Construct a three-dimensional human body model, wherein the three-dimensional human body model contains a preset number of spinal joints; Motion capture data carrying spinal segment motion information is imported into the three-dimensional human body model to drive the three-dimensional human body model to generate motion animation sequences; A first virtual camera and a second virtual camera are deployed in a virtual scene, wherein the hardware parameters of the first virtual camera and the first camera are matched, and the hardware parameters of the second virtual camera and the second camera are matched. A virtual image sequence is generated by rendering using the first virtual camera, and a virtual event stream data that is time-synchronized with the virtual image sequence is generated using the second virtual camera. Extract the corresponding multi-vertebra point coordinate labels from the motion capture data; The coordinates of the multi-vertebra points are matched with the virtual image sequence and the virtual event stream data to generate a bimodal virtual dataset carrying the multi-vertebra point annotations. The target pose estimation model is pre-trained based on the bimodal virtual dataset.

[0015] The above technical solution has the following beneficial effects or advantages: By automatically generating bimodal datasets with multi-vertebrae annotations through virtual simulation, training data can be acquired in large quantities at low cost. The virtual camera parameters match the real hardware, and the generated data distribution closely resembles the real scene, ensuring that the model pre-trained on the virtual data can be directly transferred to real devices. The annotations are automatically extracted from the motion capture data, which avoids the limitation of the upper limit of model training accuracy caused by manual annotation errors.

[0016] In some embodiments of this application, the controller performs pre-training of the target pose estimation model based on the bimodal virtual dataset, specifically configured as follows: A hierarchical learning rate strategy is adopted, with gradient-decreasing learning rates set for the input layer, intermediate feature extraction layer, and output head of the target pose estimation model, respectively. Using mean squared error as the loss function, the target pose estimation model is pre-trained in multiple rounds based on the bimodal virtual dataset and the learning rate.

[0017] The above technical solution has the following beneficial effects or advantages: The hierarchical learning rate strategy sets different training amplitudes for different network layers. The input layer adopts a higher learning rate to adapt to new modalities and quickly adapt to dual-modal inputs, while the backbone feature layer adopts a lower learning rate to retain pre-trained knowledge. This can balance the needs of modality adaptation and feature preservation, and can improve pre-training efficiency and model accuracy at the same time. Using mean squared error as the loss function optimizes the prediction error of key point coordinates, which can fit the core optimization goal of pose estimation task and ensure the accuracy of pre-training convergence direction.

[0018] In some embodiments of this application, after the controller performs multiple rounds of iterative pre-training on the target pose estimation model based on the bimodal virtual dataset, it is further configured to: Obtain a real, synchronous dataset labeled with key points of the spinal segments; The real synchronous dataset is divided into a training set and a validation set according to a preset ratio; A dynamic learning rate decay strategy is adopted, with mean squared error as the loss function, and the target pose estimation model is adaptively fine-tuned based on the training set and the validation set; the dynamic learning rate decay strategy includes decaying the learning rate to the original ratio after each preset training round.

[0019] The above technical solution has the following beneficial effects or advantages: By using domain-adaptive fine-tuning with real-world data, the model pre-trained on virtual data is transferred from the virtual domain to the real domain, eliminating the distribution differences between virtual and real data and improving the model's generalization ability and practical accuracy in real-world scenarios. The dynamic learning rate decay strategy can gradually reduce the magnitude of weight adjustments in the later stages of training, avoiding model parameter oscillations, ensuring stable convergence of the fine-tuning process, and further improving model accuracy.

[0020] In some embodiments of this application, when the controller performs domain adaptive fine-tuning of the target pose estimation model, it is further configured to: Based on the target pose estimation model, inference is performed on the unlabeled real event data in the real synchronization dataset to output the predicted coordinates of the spine points in consecutive frames; The predicted coordinates of the vertebral points are subjected to temporal smoothing to obtain the temporal smoothing result. Calculate the deviation between the predicted coordinates of the vertebral points and the temporal smoothing result; The prediction confidence level is calculated based on the aforementioned deviation; The real event data with a prediction confidence level lower than a preset threshold are labeled and supplemented.

[0021] The above technical solution has the following beneficial effects or advantages: Based on the temporal continuity of human motion, samples are automatically selected by the deviation between the prediction results and the temporal smoothing results, without the need to label the entire dataset. This allows for targeted supplementation and optimization of weak scenarios in the model. Furthermore, it can identify error-prone samples such as those with occlusion or complex motions. After supplementing the labels, the model can be iteratively optimized, continuously improving the robustness and accuracy of pose estimation in complex real-world scenarios.

[0022] In some embodiments of this application, the controller is specifically configured to output human pose estimation results as follows: The target posture estimation model outputs the key points of the spinal segment and the corresponding two-dimensional coordinates, three-dimensional coordinates and confidence scores, as the human posture estimation result. The controller is configured to send the human pose estimation result to the display, specifically as follows: Based on the key points of the spinal segments and the corresponding two-dimensional coordinates, three-dimensional coordinates and confidence values ​​of the key points of the spinal segments, a corresponding skeleton overlay image is generated, and the skeleton overlay image is displayed in real time on the display.

[0023] The above technical solution has the following beneficial effects or advantages: The output includes structured posture data with coordinates and confidence levels, which can be directly connected to upper-level applications such as posture assessment, health monitoring, and motion correction. The data has strong availability and scalability. The visualization method of skeleton overlay can intuitively present the real-time posture status of the human spine, making it easy for users to view and interact with, thereby improving the user experience of display devices.

[0024] Secondly, this application also provides a human pose estimation method, including: Obtain RGB image data and event stream data to obtain bimodal data; The pre-built target pose estimation model is invoked; the target pose estimation model is based on the original pose estimation model pre-trained by RGB as the main network foundation, retains all pre-trained parameters of the original pose estimation model except for the first convolutional layer, and adapts the weights of the first convolutional layer through RGB-guided cross-modal weight transfer. The dual-modal data is input into the target pose estimation model, and after fusion inference by the target pose estimation model, the human pose estimation result is output; the human pose estimation result includes multiple key points of the spinal segment. The human pose estimation result is sent to the display, and the display is controlled to show the human pose estimation result.

[0025] The above technical solution has the following beneficial effects or advantages: By fusing texture detail information from RGB images with the high temporal resolution and motion-blur-free characteristics of event stream data, it can stably output high-precision pose estimation results for multi-segment spinal key points in complex scenarios such as high-speed motion and low light. At the same time, it adopts a cross-modal weight transfer strategy based on RGB pre-trained models, eliminating the need to build a bimodal network architecture from scratch. While fully preserving the mature feature extraction capabilities of the original model, it can quickly adapt to bimodal inputs. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the operation scenarios of a display device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4 A flowchart illustrating a human pose estimation method provided in some embodiments of this application; Figure 5 A schematic flowchart of the preprocessing provided for some embodiments of this application; Figure 6 A schematic diagram illustrating the RGB-guided cross-modal weight transfer process provided in some embodiments of this application; Figure 7 A schematic diagram illustrating the process of generating a bimodal virtual dataset provided in some embodiments of this application; Figure 8 A schematic diagram of the pre-training process provided for some embodiments of this application; Figure 9 A flowchart illustrating the adaptive fine-tuning process provided for some embodiments of this application; Figure 10 A flowchart illustrating the process of hard case mining provided for some embodiments of this application; Figure 11 Interactive diagrams illustrating human pose estimation methods provided in some embodiments of this application; Figure 12aThis is a partial schematic diagram illustrating the generation of a bimodal virtual dataset provided in some embodiments of this application; Figure 12b This is another schematic diagram illustrating the generation of a bimodal virtual dataset provided in some embodiments of this application; Figure 12c This is another schematic diagram illustrating the generation of a bimodal virtual dataset provided in some embodiments of this application; Figure 12d This is another schematic diagram of the generation of a bimodal virtual dataset provided in some embodiments of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] In this embodiment, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0030] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0031] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0032] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0033] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0034] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0035] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0036] In some embodiments, detector 230 is used to collect signals from the external environment or signals interacting with the outside world. For example, detector 230 may include millimeter-wave radar, which can be used to detect whether a user is present within a preset range. Detector 230 may also include a voice acquisition unit to collect voice commands input by the user.

[0037] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0038] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0039] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0040] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0041] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0042] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0043] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0044] In some embodiments, the user input interface 280 can be used to receive instructions input by a user. The user input interface 280 may include at least one of a microphone, touchpad, sensor, remote control, etc. The display device 200 can then receive user-input instructions based on the user input interface 280 to perform interactive functions with the user.

[0045] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0046] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0047] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0048] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0049] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0050] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0051] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0052] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0053] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0054] In some embodiments, the display device 200 can realize human posture estimation, especially for applications requiring spinal motion capture, such as home fitness guidance and exercise posture correction. In related technologies, the display device 200 can connect to an RGB frame-type camera via a device interface 240 as a data acquisition front-end. The RGB camera acquires continuous RGB images of the target human body at a fixed frame rate (e.g., 30fps). After receiving the RGB images, the controller 250 extracts human appearance features from the images based on a deep learning model, and regresses to obtain two-dimensional or three-dimensional coordinate information of each segment of the spine and joints throughout the body, thereby completing human posture estimation.

[0055] However, in high-speed motion scenarios (such as rapid turns, jumps, and swings), the aforementioned RGB frame-based pose estimation method suffers from severe motion blur due to the exposure time limitations of the RGB camera. This results in blurred human body contours and loss of detailed spinal textures, making it difficult for the controller 250 to accurately extract the precise positional information of each spinal segment from the blurred image. Furthermore, limited by the frame rate (e.g., 30fps), the RGB camera has a time blind zone of approximately 33 milliseconds between adjacent frames, failing to capture rapid micro-movements of the spine between frames. This leads to insufficient temporal continuity in the pose estimation results, making it difficult to meet the high-precision requirements for spinal motion capture in scenarios such as home fitness guidance and posture correction (e.g., joint point positioning errors need to be controlled within the centimeter level).

[0056] Furthermore, to mitigate motion blur issues in high-speed motion scenes using RGB cameras, some solutions introduce event cameras as supplementary acquisition devices. Event cameras detect brightness changes pixel-by-pixel asynchronously, achieving microsecond-level temporal resolution and exhibiting characteristics of no motion blur and high dynamic range. However, the raw event stream output by the event camera is an asynchronous, sparse sequence of quadruples, a data modality entirely different from RGB images, making it unsuitable for direct input into convolutional neural network models pre-trained based on RGB images. Simultaneously, the multi-vertebrae pose datasets used in related technologies (such as SpineTrack) are all pure RGB image data, lacking corresponding event camera synchronization data and multi-vertebrae annotations. Constructing multi-vertebrae annotation data from scratch using event camera data is extremely costly and inconsistent annotation is difficult to guarantee. Moreover, multi-vertebrae pose estimation deep learning models are pre-trained on large-scale RGB image datasets, with the first convolutional layer having 3 input channels (corresponding to RGB three channels), creating a cross-modal difference from the 2-channel input of the event stream data. Directly training a bimodal model from scratch would not only consume significant computational resources and annotation data but also lose the mature feature extraction capabilities of the original RGB pre-trained model.

[0057] To address the aforementioned issues, this application provides a display device and a human pose estimation method. The display device 200 serves as the central hub for home visual interaction. It connects to a data acquisition device containing a first camera (e.g., an RGB camera) and a second camera (e.g., an event camera) via a device interface 240 to acquire synchronously acquired RGB image data and event stream data. The controller 250 invokes a pre-constructed target pose estimation model. This model uses a pre-trained RGB pose estimation model as its main network foundation, retaining all pre-trained parameters except for the first convolutional layer. The weights of the first convolutional layer are adapted using RGB-guided cross-modal weight transfer, enabling the adapted first convolutional layer to simultaneously receive fused input from five channels: three RGB channels and two event channels. After inputting the synchronously acquired dual-modal data into the model for fusion inference, the output is a human pose estimation result containing key points of multiple spinal segments.

[0058] In this way, by combining the advantages of texture details in RGB images with the high-speed response and motion blur-free characteristics of event cameras, high-precision multi-vertebrae pose estimation can be achieved in high-speed motion scenes. At the same time, cross-modal weight transfer can be carried out based on mature RGB pre-trained models, eliminating the need to build a bimodal network architecture from scratch. While fully retaining the mature feature extraction capabilities of the original model, it can quickly adapt to bimodal inputs, thereby reducing model training costs and data thresholds.

[0059] In the embodiments of this application, "key points of spinal segments" and "key points of the spine" have the same meaning and are used to refer to multiple key points distributed along the human spine. Each key point corresponds to a specific spinal segment and is used to describe the shape, curvature and posture of the spine.

[0060] In this embodiment, the first camera can be understood as an RGB camera or a color camera, used to acquire RGB color image data of the target human body at a fixed frame rate. The second camera can be understood as an event camera (such as DAVIS, Prophesee, etc.), used to detect brightness changes pixel by pixel in an asynchronous manner and output event stream data, each event stream data containing pixel coordinates, timestamps, and polarity information.

[0061] In some embodiments, the acquisition device may include an integrated event camera that combines an RGB sensor and an event sensor, or it may include a separate RGB camera and a separate event camera. The acquisition device is connected to the display device 200 via a device interface 240. The device interface 240 can be a wired communication interface such as a USB interface or an Ethernet interface, and can be an external interface or an internal interface of the display device 200, to provide a stable high-bandwidth data transmission channel to meet the requirements for synchronous transmission of RGB image data and event stream data.

[0062] Figure 4 This is a flowchart illustrating a human pose estimation method provided in some embodiments of this application. Figure 4 As shown, in some embodiments, controller 250 is configured to perform the following steps: S401. Obtain RGB image data and event stream data through the device interface to obtain bimodal data.

[0063] The controller 250 establishes a data connection with the acquisition device through the device interface 240 to acquire RGB image data and event stream data synchronously acquired by the first and second cameras. The first camera is an RGB camera that outputs continuous RGB color image frames at a preset frame rate (e.g., 30fps). Each RGB image frame is a three-dimensional tensor of H×W×3, with pixel values ​​ranging from [0,255]. The second camera is an event camera that outputs event stream data asynchronously. Each event contains four elements: pixel coordinates (x,y), timestamp t, and polarity p (+1 indicates a brightness increase event, -1 indicates a brightness decrease event). The two data streams are synchronously acquired through hardware triggering or software timestamp alignment to ensure that RGB image frames at the same time correspond to the event stream time window.

[0064] In some embodiments, the first camera and the second camera support hardware-triggered synchronization, which controls the two cameras to start exposure and acquisition simultaneously through the synchronization trigger signal output by the controller 250, ensuring that the exposure start time of the RGB image frame is precisely aligned with the start time of the event stream time window.

[0065] In other embodiments, a software timestamp alignment method may also be used, where the controller 250 records the timestamps of the data output by the two cameras, and frame-level synchronization is achieved through interpolation or nearest neighbor matching.

[0066] S402, Invoke the pre-built target pose estimation model.

[0067] The controller 250 invokes a pre-built target pose estimation model. This model uses the original RGB pre-trained pose estimation model as its main network foundation. While retaining all pre-trained parameters of the original pose estimation model except for the first convolutional layer, the weights of the first convolutional layer are adapted via RGB-guided cross-modal weight transfer. The adapted first convolutional layer has five input channels, with three channels receiving RGB input data and two channels receiving event input data.

[0068] S403. Input the dual-modal data into the target pose estimation model, and output the human pose estimation result after the target pose estimation model performs fusion inference.

[0069] The controller 250 first preprocesses the RGB image data and event stream data, and then inputs the preprocessed bimodal data (preprocessed RGB image data and event stream data) into a pre-constructed target pose estimation model. The bimodal data undergoes feature extraction layer by layer through the backbone network of the target pose estimation model, achieving deep fusion of RGB texture features and event stream temporal features at higher levels. Finally, the output head regresses to obtain a human pose estimation result containing key points of multiple spinal segments. These spinal segment key points can include representative joints of each segment of the spine, such as C1-C7 cervical vertebrae, T1-T12 thoracic vertebrae, and L1-L5 lumbar vertebrae, totaling nine types of spinal segment key points, as well as other human joints.

[0070] S404. Send the human pose estimation results to the display and control the display to show the human pose estimation results.

[0071] The controller 250 sends the human posture estimation results output by the target posture estimation model to the display 260 for visualization. The human posture estimation results include the two-dimensional coordinates, three-dimensional coordinates, and confidence scores of key points of each spinal segment and other human joints. Based on the coordinate information of each joint, the controller 250 generates a corresponding skeletal overlay image, which is displayed in real time on the display 260. This visually presents the motion state of the human spine to the user, making it convenient for the user to view and adjust their posture in scenarios such as fitness guidance and posture correction.

[0072] In this embodiment, the display device 200 serves as the central hub for home visual interaction. By connecting an RGB camera and an event camera, dual-modal synchronous acquisition is achieved. RGB-guided cross-modal weight transfer is used to quickly adapt the dual-modal input model. The texture detail advantage of RGB images is deeply integrated with the high-speed response and motion blur-free characteristics of the event camera, enabling the output of high-precision multi-vertebrae human pose estimation results in high-speed motion scenes. Simultaneously, cross-modal adaptation is carried out based on a mature RGB pre-trained model, which reduces model training costs and data thresholds while retaining the mature feature extraction capabilities of the original model.

[0073] Since the original event stream data is an asynchronous sparse sequence of quadruples without a fixed frame structure, it cannot be directly input into a target pose estimation model based on a convolutional neural network architecture. Furthermore, the RGB images and event stream data may have spatial coordinate deviations due to differences in camera mounting position or lens parameters. Therefore, before step S403, the controller 250 needs to preprocess the dual-modal data.

[0074] In some embodiments, the controller 250 is further configured to preprocess the bimodal data before it is input into the target attitude estimation model. For example... Figure 5 As shown, the preprocessing performed by the controller 250 includes the following steps: S501. Normalize and remove distortion from the RGB image data to obtain the RGB input data.

[0075] The controller 250 normalizes the channel values ​​of each pixel in the RGB image from the [0,255] interval to the [0,1] or [-1,1] interval to match the input value range of the target pose estimation model. At the same time, it uses the intrinsic parameter calibration data of the first camera (including radial distortion coefficient and tangential distortion coefficient) to perform distortion correction on the RGB image, eliminate the geometric deformation introduced by lens distortion, and restore the linear geometric relationship of the real scene.

[0076] S502. Perform polarity aggregation on asynchronous event stream data according to a preset time window to count the number of brightness increase events and brightness decrease events corresponding to each pixel position within the time window.

[0077] The controller 250 divides the continuous event stream data into multiple time window segments with a preset time window length ΔT (e.g., 10 to 30 milliseconds). For each time window, the controller counts the number of brightness increase events (polarity p=+1) and the number of brightness decrease events (polarity p=-1) corresponding to each pixel position (x,y) within the window, forming two independent counting matrices.

[0078] S503. Generate a dual-channel polarity aggregation map based on the number of brightness increase events and the number of brightness decrease events, and use it as event input data.

[0079] The dual-channel polarity aggregation map includes a positive polarity channel and a negative polarity channel: each pixel value in the positive polarity channel represents the cumulative number of brightness increase events at that pixel location within the time window, and each pixel value in the negative polarity channel represents the cumulative number of brightness decrease events at that pixel location within the time window. The dual-channel polarity aggregation map has a size of H×W×2, consistent with the spatial resolution (H×W) of the RGB input data.

[0080] In some implementations, the formula for calculating the dual-channel polarity aggregation map is as follows: ; in, It is a channel index. ,in Indicates a positive polarity channel. Indicates a negative polarity channel; It is the start time of the time window; This refers to the length of the time window, which can be 10-30 milliseconds. It is the Kronecker delta, when The value is 1 if the condition is met, and 0 otherwise. It is a location passage In the time window The cumulative event count (dimensionless integer); the final input to the model is 5-channel fused data: [H,W,3(RGB)+2(events)].

[0081] S504. Align the RGB input data and event input data in spatial coordinates so that the RGB input data and event input data are in the same pixel coordinate system.

[0082] The controller 250 uses the intrinsic and extrinsic parameter calibration data of the first and second cameras to calculate the spatial transformation matrix between the two cameras, and maps the pixel coordinates of the event input data to the pixel coordinate system of the RGB input data, or maps both to a common reference coordinate system, thereby achieving pixel-level spatial alignment of the dual-modal data.

[0083] After completing the above preprocessing, the controller 250 inputs the RGB input data and event input data in the same pixel coordinate system into the target pose estimation model for subsequent fusion inference.

[0084] This embodiment converts the asynchronous, non-fixed-frame structured raw event stream into a dual-channel polarity aggregation map that matches the size of the RGB image. This solves the format adaptation problem that event stream data cannot be directly input into a convolutional neural network. By separating the positive and negative polarities to generate dual channels, the dynamic change information of pixel brightness increase and decrease is fully preserved, which can give full play to the high temporal resolution advantage of the event camera. By aligning spatial coordinates, pixel-level one-to-one correspondence of dual-modal data is achieved, avoiding fusion errors caused by spatial misalignment, thereby ensuring the accuracy of dual-modal fusion inference.

[0085] In the aforementioned target pose estimation model, adapting the first-layer 3-channel convolution of the RGB pre-trained model to a 5-channel convolution that can simultaneously receive RGB three channels and event two channels is a key technical step in achieving dual-modal fusion inference. Since RGB images and event stream data belong to different data modalities, they differ significantly in pixel value distribution, information density, and statistical characteristics. Directly using randomly initialized weights to process the event input channel would destroy the feature extraction capability of the pre-trained model, leading to a significant degradation in model performance.

[0086] To solve the aforementioned cross-modal adaptation problem, such as Figure 6 As shown, Figure 6 This is a schematic flowchart illustrating RGB-guided cross-modal weight transfer provided for some embodiments of this application. In some embodiments, the controller 250 is further configured to perform the following steps: S601. Extract samples from the target event dataset.

[0087] The controller 250 randomly selects N independent event time window samples from the event dataset corresponding to the target application scenario. The value of N samples can be between 1000 and 5000. Each sample contains event stream data within a complete time window and the corresponding dual-channel polarity aggregation graph.

[0088] S602. Count the average number of pixel events in the positive and negative channels of each sample.

[0089] For each sample, calculate the average of the event counts at all pixel positions in the positive polarity channel (c=0) and the negative polarity channel (c=1) respectively, to obtain the average number of pixel events in the positive polarity channel and the average number of pixel events in the negative polarity channel for that sample. The calculation formula is as follows: ; in, It is a passage The average number of events, and These are the height and width of the image, respectively.

[0090] S603. Calculate the global information content ratio based on the average number of pixel events.

[0091] The global information content ratios are the ratio of the average number of events in the positive polarity channel to the total average number of events in both channels and the ratio of the average number of events in the negative polarity channel to the total average number of events in both channels. They can quantitatively reflect the information distribution characteristics of the positive and negative polarity channels in the target event dataset. The global information content ratios are used to guide the cross-modal weight fusion of the first convolutional layer.

[0092] In some implementations, the formula for calculating the global information content ratio is as follows: ; in, It is a positive polarity channel ( The average number of events, It is a negative polarity channel ( The average number of events; This indicates the proportion of positive polarity channels in the total information content of an event. This represents the proportion of negative polarity channels in the total information content of an event, and the two satisfy the constraint. It is used to reflect the true statistical characteristics of the target dataset.

[0093] In this embodiment of the disclosure, the proportion of channel information based on real event data of the target scene can quantify the inherent distribution characteristics of the target data, providing a reliable data foundation for subsequent cross-modal weight fusion and avoiding subjectivity and blindness in weight adaptation. By adopting a sampling statistical method, the amount of computation is reduced while ensuring data representativeness, and the channel ratio parameters that adapt to the target dataset can be obtained quickly, making the execution efficiency of weight adaptation higher.

[0094] After calculating the global information content ratio, the controller 250 further applies the global information content ratio to the cross-modal adaptation of the weights of the first convolutional layer. For example... Figure 6 As shown, in some embodiments, the controller 250 adapts the weights of the first convolutional layer via RGB-guided cross-modal weight transfer, specifically configured to perform the following steps: S604. Based on the proportion of global information, perform channel-weighted fusion of the original weights of the RGB three channels in the first convolutional layer to obtain the dual-channel event convolution weights.

[0095] The RGB three-channel convolution consists of a first channel (R channel), a second channel (G channel), and a third channel (B channel). The dual-channel event convolution weights include weights for the positive and negative event channels. For the positive event channel, the weights of the R and G channels are fused; for the negative event channel, the weights of the G and B channels are fused. The calculation formula is as follows: ; ; in, It is the first 3-channel convolutional layer of the original RGB image. The output channel, the first The weights of each input channel; It is the 2-channel convolutional layer for new event stream data. The output channel, the first The weights of each input channel; It is the output channel index. It refers to the height and width of the convolution kernel.

[0096] In the aforementioned weighted fusion method, the positive polarity channel corresponds to brightness increase events, which are related to moving edges and texture changes in the scene. The R and G channels are highly sensitive to brightness information in human vision, and fusing them can more comprehensively capture moving edge features. The negative polarity channel corresponds to brightness decrease events, and the G and B channels are more sensitive to shadows and fading areas of moving objects. Fusing them is beneficial for extracting spatial structural information of fading areas. By controlling the fusion ratio of the RGB channel weights through the global information content ratio, the generated event channel weights can adaptively match the data distribution characteristics of the target scene.

[0097] S605. Normalize the standard deviation of the dual-channel event convolution weights to make the output activation distribution of the dual-channel event convolution weights consistent with the pre-trained output distribution of the original pose estimation model.

[0098] The controller 250 performs global standard deviation normalization on the weights of the fused 2-channel convolutional layer to ensure that its output activation distribution is completely consistent with the original 3-channel model. The calculation formula is as follows: ; in, These are the final 2-channel convolutional layer weights after normalization, used to replace the first convolutional layer of the original model; This operator calculates the standard deviation of all elements within a tensor. Standard deviation normalization aims to ensure that the output activation values ​​after the event channel convolution maintain a consistent statistical distribution with the original RGB pre-trained model. Without normalization, the numerical scale differences between RGB weights and event channel weights can shift the input distribution of subsequent layers, disrupting the feature representations learned by the pre-trained model and causing a significant performance degradation. Normalization only adjusts the numerical scale of the weights, without altering their spatial structure or directional characteristics, thus preserving the feature extraction capabilities of the backbone network to the greatest extent possible.

[0099] S606. The original RGB three-channel weights and the normalized two-channel event convolution weights are concatenated along the channel dimension to form five-channel convolution weights, thus completing the adaptation of the weights of the first convolutional layer.

[0100] The original RGB weights and event convolution weights are concatenated along the input channel dimension to obtain new first-layer convolution weights, which replace the first convolutional layer of the original pose estimation model. After concatenation, the number of input channels of the target pose estimation model is expanded from 3 to 5, with the first 3 channels receiving RGB input data and the last 2 channels receiving event input data, thus achieving unified input for both modalities.

[0101] In this embodiment, event channel weights are generated by weighted fusion of RGB weights, which can reuse the feature knowledge of the RGB pre-trained model and narrow the feature gap between modalities. Global standard deviation normalization aligns the output activation distribution of the event channel with the original RGB pre-trained distribution, maximizing the preservation of the pre-trained feature extraction capability of the backbone network and avoiding significant degradation in model performance. Dual-modal adaptation is completed by concatenating only the input layer channels without modifying the original backbone network structure, ensuring compatibility.

[0102] Since the target pose estimation model needs to learn the bimodal fusion features of RGB images and event stream data, as well as the basic morphology and spatial distribution relationship of multiple vertebrae, and there is currently a lack of publicly available synchronous RGB-event bimodal training datasets with multiple vertebrae annotations, it is necessary to construct large-scale, high-quality training data before training.

[0103] Therefore, such as Figure 7 As shown, Figure 7 This is a schematic diagram illustrating the process of generating a bimodal virtual dataset provided in some embodiments of this application. In some embodiments, before training the target pose estimation model, the controller 250 is further configured to automatically generate a bimodal training dataset through virtual simulation, specifically by performing the following steps: S701. Construct a three-dimensional human body model.

[0104] The 3D human body model includes a preset number (e.g., 9) of spinal joints. The controller 250 creates a parametric human skeleton model conforming to human anatomy standards in a 3D modeling engine (e.g., Blender), defining representative spinal joints along the spine, such as cervical C1-C7, thoracic T1-T12, and lumbar L1-L5, totaling 9 independent joints distributed along the spine. Each joint possesses complete six-degree-of-freedom pose information (3D position coordinates and 3D rotation angles). The model supports free adjustment of bone length parameters, covering different heights (e.g., 150cm to 200cm), body types (e.g., thin, standard, and overweight), and clothing characteristics. The range of motion of all joints is limited according to ergonomic and kinesiological standards to ensure that the generated movements conform to the physiological structure of the real human body.

[0105] S702. Import motion capture data carrying spinal segment motion information into the 3D human body model to drive the 3D human body model to generate motion animation sequences.

[0106] The controller 250 supports importing professional motion capture data in various formats such as BVH and FBX. This type of data contains complete motion information for each segment of the spine, such as rotation angles and displacements of each spinal joint. After importing the motion data, it undergoes preprocessing: high-frequency noise introduced by the motion capture equipment is removed, missing frames are interpolated linearly or by spline interpolation, and data jitter is eliminated by mean filtering or Gaussian filtering. To enhance data diversity, the controller 250 also supports time scaling (e.g., from 0.5x slow motion to 2x fast motion) and spatial transformation (e.g., left and right flips, angle rotation) of motion sequences. A single original motion can generate dozens of motion variations with different speeds, amplitudes, and directions, covering typical motion patterns such as walking, running, jumping, turning, bending, and arm swinging.

[0107] S703. Deploy the first virtual camera and the second virtual camera in the virtual scene.

[0108] The controller 250 deploys multiple virtual cameras in the virtual scene of the 3D rendering engine and sets the shooting angle, distance, and height parameters of the virtual cameras to simulate the acquisition environment from different perspectives in a real scene. Among them, the hardware parameters of the first virtual camera are matched with those of the first camera (real RGB camera), accurately reproducing the key parameters of the real RGB camera such as resolution (e.g., 1920×1080), focal length, distortion coefficient, white balance, and exposure time; the hardware parameters of the second virtual camera are matched with those of the second camera (real event camera), accurately reproducing the key physical characteristics of the real event camera such as resolution, contrast threshold (e.g., 0.15 to 0.35), and basic noise level.

[0109] S704. A virtual image sequence is generated by rendering with the first virtual camera, and a virtual event stream data that is time-synchronized with the virtual image sequence is generated by the second virtual camera.

[0110] The controller 250 drives the 3D human model to execute a motion animation sequence. The first virtual camera renders and outputs RGB color images frame by frame at a preset frame rate (e.g., 30fps). Simultaneously, the second virtual camera uses an event simulator (e.g., ESIM) to generate continuous event stream data based on brightness changes between adjacent rendered frames. The temporal resolution of the event stream is controlled by simulator parameters (e.g., 1 microsecond level), and the time window length is configurable (preferably 20 milliseconds), ensuring that the output event stream is highly consistent with the physical characteristics of the real event camera. Finally, the event stream is segmented according to the time window and converted into a dual-channel polarity aggregation map.

[0111] S705. Extract the corresponding multi-vertebra point coordinate annotations from the motion capture data.

[0112] The controller 250 directly extracts the 3D coordinates of each spinal joint and other human joints in the virtual scene coordinate system from motion capture data or skeletal animation of a 3D human model, and projects them into 2D pixel coordinates using the projection matrix of the virtual camera. Since the annotation data comes directly from the virtual engine, no manual annotation is required, resulting in high annotation accuracy, and all 2D and 3D coordinates of the joints can be obtained simultaneously in a single frame.

[0113] S706. Match the multi-vertebral point coordinate annotations with the virtual image sequence and virtual event stream data to generate a bimodal virtual dataset carrying multi-vertebral point annotations.

[0114] The controller 250 uses time windows as units to associate and bind the RGB image frames rendered within the same time window, the corresponding event dual-channel polarity aggregation map, and the 2D / 3D coordinate annotations of each spinal joint point to form a complete training sample.

[0115] S707. Pre-train the target pose estimation model based on a dual-modal virtual dataset.

[0116] The controller 250 pre-trains the model based on the dual-modal virtual dataset, enabling the model to initially grasp the dual-modal fusion characteristics of RGB and event stream data, as well as the basic spatial morphology and distribution patterns of multiple vertebrae, laying a data foundation for subsequent fine-tuning with real data.

[0117] In this embodiment, a bimodal dataset with multi-vertebrae annotations is automatically generated through virtual simulation, enabling low-cost, large-scale acquisition of training data without relying on expensive manual annotation processes. The virtual camera parameters are matched one-to-one with the real hardware, and the generated data distribution closely resembles the real scene, ensuring that the model pre-trained on the virtual data can be directly transferred to real devices. Annotations are automatically extracted from motion capture data, with high annotation accuracy and consistency, avoiding errors and inconsistencies in manual annotation.

[0118] After constructing the bimodal virtual dataset and initializing the target pose estimation model, model pre-training is required to enable the model to learn the fused feature representation of the bimodal data and the basic pose morphology of multiple vertebrae. Since different network layers of the target pose estimation model play different functional roles—the input layer is responsible for adapting to the bimodal data, the intermediate feature extraction layer is responsible for extracting general visual features, and the output head is responsible for regressing joint coordinates—the training requirements of different layers differ. If a uniform learning rate is used for training, it is difficult to simultaneously ensure modality adaptation efficiency and pre-training knowledge retention.

[0119] Therefore, such as Figure 8 As shown, in some embodiments, the controller 250 performs pre-training of the target pose estimation model based on a bimodal virtual dataset, specifically configured to perform the following steps: S801. A hierarchical learning rate strategy is adopted, with gradient-decreasing learning rates set for the input layer, intermediate feature extraction layer, and output head of the target pose estimation model.

[0120] The input layer (i.e., the adapted first five-channel convolutional layer) needs to quickly learn the input representation of the bimodal data, so a high learning rate is set, such as 1×10. -3 The intermediate feature extraction layers (i.e., the convolutional, pooling, and normalization layers of the backbone network excluding the first layer) have acquired mature and general feature extraction capabilities from RGB pre-training. A low learning rate, such as 5×10, is set. -4 To preserve pre-trained knowledge, the output head (i.e., the final fully connected layer or convolutional output layer) needs to relearn the regression mapping of multi-vertebral joint coordinates, setting a suitable learning rate, such as 2×10. -3 .

[0121] S802. Using mean squared error as the loss function, the target pose estimation model is pre-trained in multiple rounds based on a dual-modal virtual dataset and a hierarchical learning rate.

[0122] The Mean Squared Error (MSE) loss function directly optimizes the sum of squared Euclidean distances between the model-predicted joint coordinates and the ground truth coordinates. This aligns with the core optimization objective of the pose estimation task and enables the model to learn accurate joint localization capabilities.

[0123] For example, the pre-trained optimizer can be AdamW, with the weight decay coefficient set to 1×10. -4 The batch size was set to 32, and the training epochs were set to 3 to 5. After each epoch of training, the model's joint localization accuracy was evaluated on the validation set, and the weights of the model with the best performance on the validation set were saved.

[0124] In some implementations, data augmentation strategies are also employed during the pre-training process to simultaneously perform enhancement operations such as random cropping, random horizontal flipping, and color jittering on the RGB images and event aggregation graphs in the virtual dataset, thereby further improving the model's generalization ability and robustness.

[0125] In this embodiment, the hierarchical learning rate strategy sets differentiated training amplitudes for different network layers. The input layer adopts a higher learning rate to adapt to new modalities and quickly adapt to dual-modal inputs, while the backbone feature layer adopts a lower learning rate to retain pre-trained knowledge. This can balance the needs of modal adaptation and feature preservation, and can simultaneously improve pre-training efficiency and model accuracy. Using MSE as the loss function, the prediction error of key point coordinates is directly optimized, which is in line with the core optimization goal of the pose estimation task and ensures the accuracy of the pre-training convergence direction.

[0126] After pre-training on virtual data, the target pose estimation model has mastered the basic fusion representation of bimodal data and the basic spatial morphology of multiple vertebrae. However, virtual data and real data still exhibit certain distribution differences (domain differences) in terms of lighting conditions, texture details, noise distribution, and background complexity. Directly applying the virtual pre-trained model to real-world scenes may lead to a decrease in accuracy. Therefore, it is necessary to use a small amount of real-labeled data to perform domain-adaptive fine-tuning of the model to eliminate the distribution differences between the virtual and real domains.

[0127] like Figure 9 As shown, Figure 9 This is a flowchart illustrating the domain adaptive fine-tuning process provided in some embodiments of this application. In some embodiments, after the controller 250 performs multiple rounds of iterative pre-training on the target pose estimation model based on a bimodal virtual dataset, it is further configured to perform the following steps: S901. Obtain a real synchronous dataset with labeled key points of the spinal segments.

[0128] The controller 250 connects to the real first and second cameras through the device interface 240, and collects a certain number (e.g., 50,000) of synchronous RGB image data and event stream data in actual application scenarios (such as home fitness environments), and performs manual multi-vertebra point annotation on the collected data to obtain a real labeled dataset.

[0129] S902. Divide the real synchronous dataset into a training set and a validation set according to a preset ratio.

[0130] The training set is used for domain adaptive fine-tuning, and the validation set is used to evaluate the fine-tuning effect. For example, the preset ratio can be 8:2.

[0131] S903. A dynamic learning rate decay strategy is adopted, with mean squared error as the loss function, and the target pose estimation model is adaptively fine-tuned in the domain based on the training set and the validation set.

[0132] For example, AdamW is chosen as the optimizer for fine-tuning, and the weight decay factor is set to 1×10. -4 The initial learning rate is set to 5×10. -4 The batch size is set to 16, and the training epochs are set to 5. The dynamic learning rate decay strategy includes decaying the learning rate to its original proportion after each preset training epoch, such as decaying the learning rate to 0.8 of the original rate every 2 epochs (i.e., the learning rate for the 3rd-4th epochs is 4×10). -4 The learning rate for the 5th epoch is 3.2 × 10⁻⁶. -4 ).

[0133] In this way, the model parameters are close to the optimal parameters of the virtual pre-training in the early stage of fine-tuning, and a larger learning rate can quickly adapt to the distribution characteristics of the real data. As fine-tuning progresses, the model gradually converges to the vicinity of the optimal point in the real domain. Gradually reducing the learning rate can reduce the parameter update step size, avoid skipping the optimal point or introducing parameter oscillations due to excessive step size, and ensure the stability and convergence quality of the fine-tuning process.

[0134] In some embodiments, the fine-tuning loss function also adopts MSE to minimize the Euclidean distance between the model-predicted key coordinates and the ground truth coordinates, and to maintain the same objective function as the pre-training stage, ensuring that the optimization direction of the two stages of pre-training and fine-tuning is continuous and consistent.

[0135] In some embodiments, the training process lasts for 3-5 epochs, enabling the model to learn the general feature representation of event stream data and the basic morphology of multiple vertebrae. After training is completed, the weights of the model with the best performance on the validation set are saved.

[0136] For example, fine-tuning is performed using a small number of ground-labeled multi-vertebrae synchronized RGB-event datasets, employing a dynamic learning rate decay strategy, where the learning rate decreases to 0.8 of its original value every two epochs. The unit uses mean squared error (MSE) as the loss function, calculated as follows: ; in, This is the total training loss for the current batch, used for backpropagation to optimize model parameters; It refers to the batch size; It is the number of joints in a single sample, including at least 9 spinal key points and other human joints; It is a sample index; It is a keypoint index; It is the first The first sample The true coordinate vector of each key point; It is the first The first sample Model estimation coordinate vectors for each joint point; It represents the square of the L2 norm of a vector.

[0137] In this embodiment, the virtual data pre-trained model is migrated from the virtual domain to the real domain through domain adaptive fine-tuning of real scene data, eliminating the distribution difference between virtual and real data, and improving the model's generalization ability and actual usage accuracy in real scenes. The dynamic learning rate decay strategy can gradually reduce the weight adjustment magnitude in the later stage of training, avoid model parameter oscillation, ensure stable convergence of the fine-tuning process, and further improve model accuracy.

[0138] During domain adaptive fine-tuning, although fine-tuning was performed using real-world labeled datasets, the pose estimation accuracy of the model in these challenging scenarios—such as occlusion, large-angle twisting, and rapid motion—may be relatively low in the labeled data. However, manually labeling all unlabeled data is too costly and difficult to implement.

[0139] To solve the above problems, such as Figure 10 As shown, Figure 10 This is a flowchart illustrating the hard example mining process provided in some embodiments of this application. In some embodiments, when the controller 250 performs domain adaptive fine-tuning of the target pose estimation model, it is further configured to introduce time-consistency-based hard example mining, specifically including the following steps: S1001. Based on the target pose estimation model, inference is performed on the unlabeled real event data in the real synchronization dataset, and the predicted coordinates of the spine points in consecutive frames are output.

[0140] The controller 250, based on the target pose estimation model currently in training, infers from unlabeled real event data (along with accompanying RGB image data) in the real synchronous dataset and outputs the predicted coordinates of the spine points in consecutive frames. Here, consecutive frames refer to several temporally adjacent frames of data with a small time interval between frames (e.g., 33 milliseconds), indicating strong continuity of human movement within a short period of time.

[0141] S1002. Perform time-series smoothing on the predicted coordinates of the vertebral points to obtain the time-series smoothing result.

[0142] The controller 250 employs temporal smoothing methods such as moving average filtering or Kalman filtering to smooth the predicted coordinate sequence of the same joint point in multiple consecutive frames, eliminating high-frequency jitter and random errors in single-frame prediction, and obtaining temporally smoothed estimates of the coordinates of each joint point. The basis for this temporal smoothing is that human movement is physically constrained by the range of motion and inertia of joints, and the motion trajectories of joint points between adjacent frames are usually continuous and smooth, without abrupt jumps.

[0143] S1003. Calculate the deviation between the predicted coordinates of the vertebral point and the time-series smoothing result, and calculate the corresponding prediction confidence based on the deviation.

[0144] For each keypoint of each sample, calculate the L2 norm deviation (Euclidean distance) between the original predicted coordinates and the time-smoothed coordinates. The larger the deviation, the greater the deviation between the model prediction result of that frame and the assumption of temporal continuity, i.e., the lower the reliability of the prediction.

[0145] S1004. Add annotations to real event data whose prediction confidence is lower than a preset threshold.

[0146] The controller 250 selects samples with prediction confidence scores below a preset threshold (e.g., 0.7) from all unlabeled real data as hard examples. These hard examples are then manually labeled (or semi-automatically labeled with manual correction assistance) to obtain the ground truth coordinates of the multiple vertebrae of the hard examples. The newly labeled hard examples are added to the real training set, and the model is further trained for 1 to 2 epochs. This process can be repeated iteratively 2 to 3 times, with each iteration generating a new batch of hard examples to supplement the training data, gradually making up for the model's capability gaps in difficult scenarios and forming a self-reinforcing training loop.

[0147] In some embodiments, the preset threshold can be adjusted according to actual needs. For example, the initial threshold is set to 0.7, and as the number of iterations increases and the model accuracy improves, the threshold can be appropriately reduced (e.g., 0.65, 0.6) to filter out more challenging cases.

[0148] For example, after each training epoch, the current model is used to perform inference on unlabeled real data. The reliability of the prediction results is judged by temporal consistency, and the prediction confidence of each sample is calculated using the following formula: ; in, It is the first The prediction confidence score for each sample, with values ​​normalized to a range of 1. The closer the value is to 1, the more reliable the model prediction is; yes The joint coordinate vector obtained after time-series smoothing methods such as moving mean filtering or Kalman filtering; This represents the L2 norm of the vector. This unit will select difficult examples with prediction confidence below a preset threshold for manual annotation. After adding the newly annotated difficult examples to the training set, it will continue training for 2-3 epochs, forming a self-reinforcing training loop.

[0149] In this embodiment, based on the temporal continuity of human motion, difficult sample samples are automatically screened by the deviation between the prediction result and the temporal smoothing result. There is no need to label the entire data. The weak scenes of the model can be supplemented and optimized in a targeted manner. Furthermore, it can discover error-prone samples such as occlusion and complex motion. After supplementing the label, the model can be iteratively optimized to continuously improve the robustness and accuracy of the model's pose estimation in complex real-world scenarios.

[0150] After model training and fine-tuning, the target pose estimation model can be deployed to the controller 250 of the display device 200 for inference. During the inference phase, the controller 250 inputs the real-time acquired and preprocessed bimodal data into the target pose estimation model to obtain human pose estimation results including key points of the spinal segments.

[0151] In some embodiments, the controller 250 executes the output of human pose estimation results, specifically configured to: output, based on the target pose estimation model, a result including key points of the spinal segment and their corresponding two-dimensional coordinates, three-dimensional coordinates, and confidence scores, as the human pose estimation result. The two-dimensional coordinates are the (x, y) coordinates of the joint point in the image pixel coordinate system, in pixels; the three-dimensional coordinates are the (X, Y, Z) coordinates of the joint point in the camera coordinate system or world coordinate system with the camera as a reference, in meters; the confidence score represents the reliability of the model's prediction result for that joint point, with a value range of [0, 1], and can be used by downstream applications (such as pose assessment and motion correction) to determine whether to adopt the current prediction result.

[0152] When the controller 250 sends the human posture estimation results to the display 260, it generates a corresponding skeleton overlay image based on the key points of the spinal segments and the corresponding two-dimensional coordinates, three-dimensional coordinates and confidence values ​​of the key points of the spinal segments. The display 260 then displays the skeleton overlay image in real time.

[0153] In some embodiments, the controller 250 draws joint point markers (such as colored dots) and skeletal connection lines (such as straight line segments connecting adjacent spinal joints) on an RGB image screen based on the two-dimensional coordinates of each joint point, forming a skeleton overlay screen. Different spinal segments can be distinguished by different colors (such as red for the cervical spine, green for the thoracic spine, and blue for the lumbar spine), and joints with lower confidence can be presented as semi-transparent or dashed lines to alert the user. The skeleton overlay screen is superimposed on the user interface of the display 260 in the form of a real-time video stream, which can smoothly present the real-time posture changes of the human body.

[0154] In some embodiments, the controller 250 also supports outputting posture data files in standard formats such as JSON and CSV for further processing and analysis by upper-level applications such as fitness guidance applications and motion analysis software. Simultaneously, the controller 250 also supports uploading the estimation results to the cloud or mobile terminal via a network, enabling remote posture monitoring and guidance.

[0155] This embodiment of the disclosure outputs structured posture data containing coordinates and confidence levels, which can be directly connected to upper-layer applications such as posture assessment, health monitoring, and motion correction. The data has strong availability and scalability. The visualization display method of skeleton overlay intuitively presents the real-time posture status of the human spine, which is convenient for users to view and interact with intuitively, thereby improving the user experience of the display device 200.

[0156] Figure 11 This is an interactive schematic diagram illustrating some embodiments of the human pose estimation method provided in this application. For example... Figure 11 As shown, in some embodiments, the user triggers human posture detection-related functions through a fitness and health application on the display device 200. The controller 250 triggers the first camera and the second camera to work synchronously; the first camera acquires RGB image data of the target human body, and the second camera acquires event stream data of the target human body.

[0157] Two raw data streams are transmitted to the controller via device interface 240. The data preprocessing unit in the controller 250 normalizes and removes distortion from the RGB image data, performs polarity aggregation on the event stream data according to the time window, and completes bimodal spatial coordinate alignment to generate fused input data. The multi-vertebral point pose estimation unit in the controller 250 calls the target pose estimation model to infer the fused input data and outputs human pose estimation results containing key points of multiple spinal segments. The result output unit in the controller 250 converts the pose estimation results into a visual image and structured data, and sends them to the display 260, which displays the human pose estimation results in real time.

[0158] like Figures 12a-12d This describes the process for generating a bimodal virtual dataset. For example... Figures 12a-12d As shown, in some embodiments, Figure 12a The virtual scene shown contains a parametric human model that conforms to human anatomy, and can be configured with different body types and clothing styles. The model's skeleton includes multiple joints distributed along the spine. Figure 12b As shown, four virtual cameras are deployed around the scene, one of which is designated as the main camera (i.e., the first virtual camera). Figure 12b Camera 1 is the main camera, and the others are secondary virtual cameras. Each virtual camera captures data from a different perspective on the human model. For example... Figure 12cAs shown, each virtual camera synchronously generates an RGB image and dual-channel polarity aggregated event stream data corresponding to its viewpoint, using different colors (not shown in the figure) to distinguish between two types of event polarities: brightness increase and brightness decrease. Figure 12d As shown, the three-dimensional coordinates of all joints, including the key points of the spine, are automatically extracted from the skeletal data of the human model to generate three-dimensional skeleton annotations. By binding the RGB images and event stream data from the same viewpoint with the corresponding joint annotations, a dual-modal virtual dataset sample with multiple spine annotations can be obtained.

[0159] Based on the aforementioned display device 200, some embodiments of this application also provide a human pose estimation method, which can be applied to the display device 200. The display device 200 is connected to a data acquisition device including a first camera and a second camera via a device interface 240. The first camera is used to acquire RGB image data of the target human body, and the second camera is used to acquire event stream data of the target human body. The method includes the following steps: Obtain RGB image data and event stream data to obtain bimodal data.

[0160] The pre-built target pose estimation model is invoked. The target pose estimation model is based on the original pose estimation model pre-trained in RGB, retaining all pre-trained parameters of the original pose estimation model except for the first convolutional layer. The weights of the first convolutional layer are adapted through RGB-guided cross-modal weight transfer. The pre-processed RGB image data and event stream data are input into the target pose estimation model.

[0161] The dual-modal data is input into the target pose estimation model, and after fusion inference by the target pose estimation model, the human pose estimation result is output. The human pose estimation result includes key points of multiple spinal segments.

[0162] Send the human pose estimation results to the display and control the display to show the human pose estimation results.

[0163] Based on the above human pose estimation method, it can make full use of the texture details of RGB images and the high temporal resolution and no motion blur characteristics of event stream data to achieve high-precision multi-vertebrae pose estimation in high-speed motion scenarios such as home fitness and motion posture correction. At the same time, based on the RGB-guided cross-modal weight transfer strategy, it can quickly adapt to dual-modal input while fully preserving the mature feature extraction capabilities of the original model, thereby reducing the model training cost and data threshold.

[0164] It should be noted that the human posture estimation method described in this application embodiment can adopt the same principle and implementation as the above-mentioned display device 200 embodiment, and will not be described again here.

[0165] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the user interface; The device interface is configured to connect to a data acquisition device, which includes a first camera and a second camera. The first camera is used to acquire RGB image data of the target human body, and the second camera is used to acquire event stream data of the target human body. The controller is configured as follows: The RGB image data and the event stream data are obtained through the device interface to obtain bimodal data; The pre-built target pose estimation model is invoked; the target pose estimation model is based on the original pose estimation model pre-trained by RGB as the main network foundation, retains all pre-trained parameters of the original pose estimation model except for the first convolutional layer, and adapts the weights of the first convolutional layer through RGB-guided cross-modal weight transfer. The dual-modal data is input into the target pose estimation model, and the target pose estimation model performs fusion inference and outputs the human pose estimation result. The human posture estimation results include key points of multiple spinal segments; The human pose estimation result is sent to the display, and the display is controlled to show the human pose estimation result.

2. The display device according to claim 1, characterized in that, Before inputting the bimodal data into the target attitude estimation model, the controller is further configured to: The RGB image data is normalized and distortion-reduced to obtain RGB input data; The asynchronous event stream data is aggregated according to a preset time window to count the number of brightness increase events and the number of brightness decrease events corresponding to each pixel position within the time window. A dual-channel polarity aggregation map is generated based on the number of brightness increase events and the number of brightness decrease events, and used as event input data. The dual-channel polarity aggregation diagram includes positive polarity channels and negative polarity channels; The RGB input data and the event input data are spatially aligned so that they are in the same pixel coordinate system. The controller is configured to input the dual-modal data into the target attitude estimation model, specifically as follows: The RGB input data and the event input data in the same pixel coordinate system are input into the target pose estimation model.

3. The display device according to claim 1, characterized in that, The controller is also configured to: Samples are extracted from the target event dataset; Count the average number of pixel events in the positive and negative channels of each sample; The global information content ratio is calculated based on the average number of pixel events. The global information content ratio is the ratio of the average number of events in a single channel to the total average number of events in both channels, and the global information content ratio is used to guide the cross-modal weight fusion of the first convolutional layer.

4. The display device according to claim 3, characterized in that, The controller adapts the weights of the first convolutional layer through RGB-guided cross-modal weight transfer, specifically configured as follows: Based on the global information ratio, the original weights of the RGB three channels in the first convolutional layer are weighted and fused separately to obtain dual-channel event convolution weights; wherein, the RGB three channels include a first channel, a second channel, and a third channel, and the dual-channel event convolution weights include positive polarity event channel weights and negative polarity event channel weights; the positive polarity event channel weights are generated by weighted fusion of the weights of the first channel and the second channel, and the negative polarity event channel weights are generated by weighted fusion of the weights of the second channel and the third channel; The standard deviation of the dual-channel event convolution weights is normalized to make the output activation distribution of the dual-channel event convolution weights consistent with the pre-training output distribution of the original pose estimation model. The original weights and the dual-channel event convolution weights are concatenated along the channel dimension to form multi-channel convolution weights, thus completing the adaptation of the weights of the first convolutional layer.

5. The display device according to claim 1, characterized in that, Before training the target pose estimation model, the controller is further configured as follows: Construct a three-dimensional human body model, wherein the three-dimensional human body model contains a preset number of spinal joints; Motion capture data carrying spinal segment motion information is imported into the three-dimensional human body model to drive the three-dimensional human body model to generate motion animation sequences; A first virtual camera and a second virtual camera are deployed in a virtual scene, wherein the hardware parameters of the first virtual camera and the first camera are matched, and the hardware parameters of the second virtual camera and the second camera are matched. A virtual image sequence is generated by rendering using the first virtual camera, and a virtual event stream data that is time-synchronized with the virtual image sequence is generated using the second virtual camera. Extract the corresponding multi-vertebra point coordinate labels from the motion capture data; The coordinates of the multi-vertebra points are matched with the virtual image sequence and the virtual event stream data to generate a bimodal virtual dataset carrying the multi-vertebra point annotations. The target pose estimation model is pre-trained based on the bimodal virtual dataset.

6. The display device according to claim 5, characterized in that, The controller performs pre-training of the target pose estimation model based on the bimodal virtual dataset, specifically configured as follows: A hierarchical learning rate strategy is adopted, with gradient-decreasing learning rates set for the input layer, intermediate feature extraction layer, and output head of the target pose estimation model, respectively. Using mean squared error as the loss function, the target pose estimation model is pre-trained in multiple rounds based on the bimodal virtual dataset and the learning rate.

7. The display device according to claim 6, characterized in that, After the controller performs multiple rounds of iterative pre-training on the target pose estimation model based on the bimodal virtual dataset, it is further configured to: Obtain a real, synchronous dataset labeled with key points of the spinal segments; The real synchronous dataset is divided into a training set and a validation set according to a preset ratio; A dynamic learning rate decay strategy is adopted, with mean squared error as the loss function, and the target pose estimation model is adaptively fine-tuned in the domain based on the training set and the validation set. The dynamic learning rate decay strategy includes decaying the learning rate back to its original proportion after each preset training round.

8. The display device according to claim 7, characterized in that, When the controller performs domain adaptive fine-tuning of the target pose estimation model, it is also configured to: Based on the target pose estimation model, inference is performed on the unlabeled real event data in the real synchronization dataset to output the predicted coordinates of the spine points in consecutive frames; The predicted coordinates of the vertebral points are subjected to temporal smoothing to obtain the temporal smoothing result. Calculate the deviation between the predicted coordinates of the vertebral points and the temporal smoothing result; The prediction confidence level is calculated based on the aforementioned deviation; The real event data with a prediction confidence level lower than a preset threshold are labeled and supplemented.

9. The display device according to claim 1, characterized in that, The controller is configured to output the human pose estimation result. The target posture estimation model outputs the key points of the spinal segment and the corresponding two-dimensional coordinates, three-dimensional coordinates and confidence scores, as the human posture estimation result. The controller is configured to send the human pose estimation result to the display, specifically as follows: Based on the key points of the spinal segments and the corresponding two-dimensional coordinates, three-dimensional coordinates and confidence values ​​of the key points of the spinal segments, a corresponding skeleton overlay image is generated, and the skeleton overlay image is displayed in real time on the display.

10. A method for estimating human pose, characterized in that, include: Obtain RGB image data and event stream data to obtain bimodal data; Invoke the pre-built target pose estimation model; The target pose estimation model is based on the original pose estimation model pre-trained by RGB as the main network foundation. It retains all pre-trained parameters of the original pose estimation model except for the first convolutional layer, and adapts the weights of the first convolutional layer through cross-modal weight transfer guided by RGB. The dual-modal data is input into the target pose estimation model, and the target pose estimation model performs fusion inference and outputs the human pose estimation result. The human posture estimation results include key points of multiple spinal segments; The human pose estimation result is sent to the display, and the display is controlled to show the human pose estimation result.

Images