Court hearing public display system and method based on Internet of Things
Through IoT technology and real-time motion capture, combined with machine learning, three-dimensional models are built, which solves the problem that existing electronic whiteboards cannot effectively display the spokesperson's actions, and realizes efficient, intuitive and interactive display of trial information.
Patent Information
- Application Number
- CN202510251038.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-04
- Publication Date
- 2025-06-27
AI Technical Summary
The existing electronic whiteboards cannot effectively display the spokesperson's physical movements and hand instructions during the trial, resulting in remote participants not being able to clearly see the relationship between the graphic information and dynamic expressions.
The Internet of Things-based public trial display system is adopted, combining real-time motion capture and machine learning technology to build a three-dimensional human body and hand model, and display the spokesperson's body movements and hand movements in real time, combining graphics and text information on the electronic whiteboard.
It enhances the transparency and interactivity of the trial, improves the accuracy and intuitiveness of information expression, and ensures that all participants can view the trial information simultaneously.
Smart Images

Figure CN120223824A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of Internet of Things intelligent court systems, and particularly relates to a court trial public display system and method based on the Internet of Things. Background Art
[0002] The Internet of Things court trial public display device is a court trial information display and interaction system based on Internet of Things technology, aiming to achieve transparency, interactivity and visualization of the court trial process through remote technology. This device is mainly applied to the court trial site, combining a variety of hardware and software technologies, including functions such as electronic whiteboards, real-time video capture, motion recognition, and data synchronization, to provide a clearer and more intuitive display platform for speakers, judges, prosecutors, plaintiffs and defendants in the court trial. During the court trial process, judges, plaintiffs and defendants can conduct real-time display and interaction through Internet of Things devices, including content such as voice, body movements, hand gestures, and graphic information, providing all-round expression and display functions.
[0003] Currently, during the court trial process, a two-dimensional electronic whiteboard is usually used to display graphic information, evidence materials, and court trial-related data. The electronic whiteboard is usually marked through software tools or directly marked by contacting the electronic whiteboard with a stylus. Existing electronic whiteboards can display content such as PDFs, pictures, and videos, but they can only display two-dimensional plane information and cannot effectively display actions and dynamic processes in three-dimensional space. At the court trial site, presided over by the collegial panel, court trial information is displayed through the electronic whiteboard, and the plaintiff and defendant respectively conduct remote speeches and screen instruction explanations through Internet of Things devices. However, existing electronic whiteboards have several limitations: First, the electronic whiteboard can only display graphic information in a two-dimensional plane and cannot display the entire process from the speaker's language to body movements and then to hand gestures; Second, although remote video calls can display the overall state of the speaker, they cannot effectively synchronize the speaker's body movements, gestures, and pointing actions to the electronic whiteboard, resulting in remote participants being unable to clearly see the dynamic expressions combined with graphic information. In addition, in existing systems, the actions (such as language, gestures, body movements, etc.) of the speaker before using the whiteboard to mark information cannot be fully displayed, and remote audiences can only obtain a long-distance view from static videos and cannot effectively integrate with the information displayed on the electronic whiteboard.
[0004] Based on the above deficiencies of the existing technology, improvement is particularly necessary. Summary of the Invention
[0005] Aiming at the defects and problems existing in the prior art, the purpose of the present invention is to optimize hand and body movement modeling through Internet of Things technology, real-time motion capture, and machine learning, improve the real-time performance, intuitiveness and interactivity of court trial public display, ensure that all parties of participants can view court trial information synchronously, enhance the transparency of the court trial and optimize the information expression effect.
[0006] The solution of the present invention to solve the technical problem is: adopting a court trial public display system based on the Internet of Things, including a court trial terminal device and a terminal device, the court trial terminal device includes a background server, an electronic whiteboard and an information synchronization and model building module, the electronic whiteboard includes a main screen and multiple split screens, which are used to display graphic information and real-time dynamics of each speaker during the trial, the main screen displays the main graphic information, and the split screens display the human body model and hand movements of the speaker at each terminal in real time; the background server is responsible for data processing, information synchronization and transmission, it receives feature data from each terminal, and synchronizes these data with the display information on the electronic whiteboard; the information synchronization and model building module Construction: The backend server establishes a three-dimensional model of the speaker and his hands, maps the language features and body movement features, and displays it on the split screen of the electronic whiteboard. The hand model is superimposed on the graphic information of the main screen in real time as a "picture-in-picture"; the terminal device includes a terminal display screen, a terminal system, and front and rear cameras. The front camera is used to capture the speaker's language, facial and body movement features; the rear camera is used to capture the speaker's hand movements and record the movement trajectory and stop position of the fingers or hand-held tools in front of the terminal display screen; Data transmission: The hand feature identification information is transmitted to the backend server through the terminal system, which is used to construct a hand model and display the hand movement process in real time.
[0007] A display method based on the display system is adopted, comprising the following steps.
[0008] S1 Equipment collaboration and communication establishment: Through the Internet of Things platform, each terminal and the background server establish a connection through a wireless or wired network. The terminal establishes a communication relationship with the server through account login, authentication or role allocation. All devices use unified protocols and standards for data exchange to ensure efficient collaboration and real-time performance of the system.
[0009] S2 Display request and information transmission: When any speaker is ready to display graphic information on the electronic whiteboard, the terminal initiates a display request. After receiving the display request, the background server confirms the display authority and prepares to display the information. At this time, the terminal's camera starts working to collect the speaker's movement features and hand data.
[0010] S3 Collection and transmission of action features: The terminal collects the speaker's language, facial expressions and body movements in real time through the front camera, while the rear camera captures the speaker's hand movements and transmits the hand features and their position on the display screen to the background server in real time.
[0011] S4 Synchronization of Display Information: After the background server receives the feature information sent by all terminals, it maps the language features, body movement features, and hand features to the human model and hand model respectively, and displays the dynamics of the speakers of each terminal on the split screen of the electronic whiteboard; the main screen displays the graphic and text information that the speaker needs to show, and the split screen shows the human model and hand model of the speaker.
[0012] S5 Combination of Hand Movements and Graphic and Text Information: When the speaker uses hand movements to indicate graphic and text information, the background server maps the hand movements to the hand model according to the hand feature information, and superimposes the hand model on the graphic and text information on the main screen. In this way, all court trial personnel can not only see the speech content of the speaker, but also see their gestures and body movements, enhancing the expression effect of information and ensuring that all participants can see the real-time association between the speaker's movements and graphic and text information during the court trial process.
[0013] S6 Multi-Party Synchronization and Remote Display: The terminals of all parties synchronize data in real time through the network to ensure that court trial personnel in different locations can see the same graphic and text information, human model, and hand movements on different devices.
[0014] Advantages of the Present Invention: Enhancing the Transparency and Interaction of Court Trials: By capturing and displaying the body movements and hand gestures of the speaker in real time, the non-verbal information (such as gestures, postures, etc.) of the speaker can be synchronously displayed with the graphic and text information, improving the intuitiveness and sense of participation of court trials.
[0015] Improving the Information Expression Effect: The combination of the hand model and the graphic and text information makes the speaker's instructions more clear, enhancing the accuracy of expression. Especially when presenting complex evidence or explaining content, the guidance of gestures and movements plays an important role.
[0016] Real-Time and Low Latency: Through the cooperation of the Internet of Things platform, the data transmission uses a standardized protocol to ensure the information synchronization and efficient transmission between terminals. The terminal system models the body movements and hand movements locally, reducing the computing pressure on the server, avoiding latency, and improving the system response speed and fluency.
[0017] High-Precision Motion Capture and Modeling: Using binocular camera technology, the distance between the hand and the display screen can be accurately captured, avoiding the deviation of the hand model, thus ensuring the accurate superposition of hand movements and graphic and text information. Combined with the optimization of machine learning technology, the model can adapt to various complex gestures and body movements.
[0018] Personalized Optimization: Modeling the body and hand movements locally at the terminal can be adjusted according to the specific hardware environment (such as camera angle, display resolution), improving the accuracy of motion capture and the model rendering effect, and adapting to the needs of different devices. Brief Description of the Drawings
[0019] Figure 1 is a block diagram showing the relationship of the display system of the present invention; Figure 2 is a flowchart showing the method of the present invention; Figure 3 is a demonstration schematic diagram in which the hand models of both parties are simultaneously superimposed on the main screen.
[0020] Reference numerals in the figure: 1 - main screen; 2 - hand model. Detailed Embodiment
[0021] The present invention will be further described below in conjunction with the drawings and embodiments.
[0022] Embodiment 1: The existing electronic whiteboard display system has the following main problems. Two - dimensional display limitation: Traditional electronic whiteboards can only display planar graphic and text information and cannot effectively display actions or gestures in three - dimensional space, resulting in insufficient interactivity between the speaker and the displayed content. Lack of synchronization between actions and language: In the existing system, the language and body movements of the speaker cannot be synchronized with the displayed content on the electronic whiteboard for real - time display or guidance. During remote speaking, although the language content of the speaker can be transmitted, their body movements, gestures, or guiding actions cannot be directly combined with the graphic and text information. Poor remote interaction: During remote court trials, distant participants cannot clearly see the association between the actions of the speaker and the graphics and texts, and cannot intuitively understand the interaction process between the speaker and the displayed content, resulting in incomplete and unclear information transmission. Based on the above situation, in this embodiment, a court trial public display system based on the Internet of Things is used, as Figure 1 shown, which mainly includes court trial terminal devices and systems and terminal devices, etc. By introducing a court trial public display system based on the Internet of Things and combining real - time motion capture, three - dimensional human model construction, hand motion recognition, and synchronous display, the limitation that traditional electronic whiteboards can only display two - dimensional graphic and text information can be effectively solved. In addition, this system can also improve the transparency and interactivity of the court trial process through multi - terminal synchronous display and remote interaction functions. Especially in remote court trials, the whole process from the speaker's language to body movements and then to gesture guidance can be synchronized with the content displayed on the electronic whiteboard, realizing more intuitive and accurate information transmission and avoiding the problem of isolated information in traditional video display and graphic display. Therefore, the improved technical solution can improve the efficiency, accuracy, and transparency of court trials, providing a higher - quality display and interaction experience for court staff and remote audiences.
[0023] Specifically, the court trial terminal devices and systems include a background server, a court trial system (including the existing court trial system), and an electronic whiteboard, etc.
[0024] The electronic whiteboard includes a main screen and multiple split screens, which are used to display graphic information and real-time dynamics of each speaker during the trial. The main screen displays the main graphic information, and the split screen displays the human body model and hand movements of each terminal speaker in real time (including at least one plaintiff, only defendant and third party, etc.). If the electronic whiteboard in the trial end device is equipped with front and rear side cameras, it can be used as one of the terminals, and the human body model and hand movements of the collegial panel speaker can be displayed in one of the split screens. When displaying graphic information through the electronic whiteboard, it does not affect the normal operation of the original trial system.
[0025] Backend server: responsible for data processing, information synchronization and transmission. It receives characteristic data (language, body movements, hand movements) from each terminal and synchronizes these data with the information displayed on the electronic whiteboard. At the same time, the backend server uploads the data to the trial system for data storage.
[0026] Information synchronization and model building module: The backend server builds a three-dimensional model of the speaker and his hands, maps the language features and body movement features, and displays them on the split screen of the electronic whiteboard. The hand model is superimposed on the graphic information on the main screen as a "picture-in-picture" in real time, improving the intuitiveness of information expression.
[0027] Protocols and standards: In the IoT environment, devices need to use standardized protocols for data transmission, such as lightweight protocols such as MQTT and CoAP, to ensure real-time and bandwidth efficiency and avoid delays and freezes.
[0028] Data privacy and security: The trial involves sensitive information, so data transmission and storage must be encrypted, and technologies such as HTTPS and VPN must be used to ensure information security and avoid data leakage.
[0029] Multi-terminal adaptation: supports compatibility with different types of terminals (such as PCs, tablets, mobile phones, etc.), ensuring that all types of devices can participate in the trial and display in real time.
[0030] User interface design: In order to improve the usability of the trial system, a simple and intuitive user interface (UI) can be designed on the terminal device so that judges, plaintiffs, defendants and other personnel can easily operate and control the displayed content.
[0031] The terminal device includes the terminal host, display screen, terminal system, front and rear cameras, etc. Front camera: used to capture the speaker's language, facial and body movement features. Rear camera: used to capture the speaker's hand movements and record the movement trajectory and stop position of fingers or handheld tools in front of the terminal display screen.
[0032] Coordinate system establishment: Through the rear camera, a plane coordinate system on the terminal display screen is first established to capture the hand movements of the speaker and record the movement trajectories and staying positions of the fingers or handheld tools at the front end of the terminal display screen. The terminal system transmits each hand movement information to the background server in real time, and the background server synchronously superimposes the hand movement information (the staying position and movement trajectory information of the hand on the display screen) on the hand model.
[0033] Data transmission: Hand feature identification information (such as finger trajectories, staying positions) is transmitted to the background server through the terminal system for constructing a hand model and displaying the hand movement process in real time.
[0034] The rear camera preferably uses a binocular camera: While the binocular camera can obtain the image information of the hand features, it can also detect the distance between the hand and the terminal display. Thus, it can more accurately superimpose the hand model on the graphic information of the main screen in real time, improving the intuitiveness of information expression. This technology can not only avoid the floating deviation of the hand model but also make the hand model more conform to the actual space positioning when displayed.
[0035] Each terminal system is equipped with a remote control: The remote control can be used to adjust the displayed information, switch speakers, zoom in or out the content of the main screen and sub - screens, or project the main screen or sub - screens in full - screen mode. It provides a more intuitive control method for users, ensuring that judges and other court personnel can quickly adjust and interact. For the magnification and reduction of the main screen and sub - screens, when the hand - pointing action is not close to the graphic information on the main screen, it indicates that there is no hand - pointing action at this time, and only language and body movements are present. At this time, the human model in the sub - screen becomes larger and the main screen becomes smaller. When the hand has a pointing action towards the graphic information in the main screen, the main screen becomes larger, and the hand model is superimposed on the graphic information of the main screen. At this time, the sub - screen becomes smaller. Ensure the natural connection from the sub - screen to the main screen. It also includes data processing for increasing or decreasing the hand model. For example, when using a mobile phone as a terminal, the hand movement will cover most of the mobile phone screen. To avoid the hand movement from overly blocking the graphic information, the hand model is reduced and placed above the graphic information of the main screen, or the transparency of the hand model is increased, ensuring the guiding role of the hand while not affecting the expression of the graphic information.
[0036] Mapping of human body and hand modeling in the terminal system: In the terminal system, limb movements and hand movements are mapped to corresponding models in the terminal. The advantage of this is to reduce the latency of data transmission, reduce the system burden. Especially in an unstable network environment, it can improve real - time performance and accuracy. In addition, this modeling process provides necessary local data support for system optimization, enabling the terminal device to independently process some action recognition tasks and improving the stability and efficiency of the entire system.
[0037] In addition, a remote control is added to this court trial public display system based on the Internet of Things. Its function is to enhance the interactivity and flexibility of the system, especially to provide an intuitive and convenient operation method for all parties at the court trial site (such as judges, plaintiffs, defendants, etc.).
[0038] The remote control can be used to control the display of graphics and text. Judges, plaintiffs or defendants can quickly switch the graphics and text content displayed on the electronic whiteboard, such as displaying evidence, case-related charts, legal provisions, etc. The remote control can provide basic operations such as turning pages up and down, zooming in / out / full screen, and switching between different pages.
[0039] The remote control can be equipped with a touchpad or buttons to facilitate users to directly mark, circle or highlight specific content on the electronic whiteboard. Combined with the hand model, it can ensure that key information during the court trial is effectively highlighted.
[0040] Generally, the system can determine the switching between the main screen and the sub-screen according to the relationship between the speaker's hand and the main screen. However, the remote control can also be used to adjust the display mode switching function between the main screen and the sub-screen to help the judge or the speaker dynamically adjust the speaking mode. For example, the judge can use the remote control to enlarge (full screen) the main screen to display the speaker's gesture guidance or important evidence pictures, or switch to the sub-screen mode to view the human models of different speakers. Zooming and switching: In a court trial with multiple participants, the remote control can help adjust the size of the content displayed by each party to ensure that all participants can clearly see the required graphic information or action display.
[0041] Adjustment of hand model display: When the speaker points to graphic content with a finger, the remote control can control the display method of the hand model. For example, reduce the hand model to avoid covering important graphic information on the screen, or increase the transparency to make the hand guidance more obvious without interfering with the text display. Gesture switching: During the court trial, if it is necessary to switch between different gestures or action displays, the remote control can be used to control the switching and dynamic adjustment of the hand model.
[0042] Embodiment 2: A public display method for court trial equipment based on the Internet of Things in Embodiment 1, as Figure 2 shown, the main steps are as follows.
[0043] Device collaboration and communication establishment: Through the Internet of Things platform (based on MCU technology), each terminal and the background server establish a connection through wireless or wired networks. The terminal establishes a communication relationship with the server through account login, authentication and role assignment. All devices use a unified protocol and standard for data exchange to ensure the efficient collaboration and real-time performance of the system.
[0044] Presentation Request and Information Transmission: When any speaker (plaintiff or defendant) is ready to present graphic and text information on the electronic whiteboard, the terminal initiates a presentation request. After receiving the presentation request, the background server confirms the presentation permission and prepares the presentation information. At this time, the camera of the terminal starts to work, collecting the action features and hand data of the speaker.
[0045] Collection and Transmission of Action Features: The terminal collects the language, facial expressions, and body movements of the speaker in real time through the front camera. The rear camera captures the hand movements of the speaker and establishes a planar coordinate system on the terminal display screen. The hand features (such as finger trajectories, gestures, etc.) and their staying positions on the display screen are transmitted to the background server in real time.
[0046] Synchronization of Presentation Information: After receiving the feature information sent by all terminals, the background server maps the language features, body movement features, and hand features to the human model and hand model respectively, and displays the dynamics of the speakers of each terminal on the split screen of the electronic whiteboard.
[0047] The main screen displays the graphic and text information that the speaker needs to present, and the split screen displays the human model and hand model of the speaker. In this way, during the court trial process, all participants can see the real-time association between the speaker's actions and the graphic and text information.
[0048] Combination of Hand Movements and Graphic and Text Information: When the speaker uses hand movements to indicate graphic and text information, the background server maps the hand movements to the hand model according to the hand feature information, and superimposes the hand model on the graphic and text information on the main screen, as Figure 3 shown. In this way, all court trial personnel can not only see the speech content of the speaker, but also see their gestures and body movements, enhancing the expression effect of the information. This operation can be optimized through machine learning to make the indication of hand movements more accurate.
[0049] Multi-Party Synchronization and Remote Presentation: The terminals of all parties synchronize data in real time through the network to ensure that court trial personnel in different locations can see the same graphic and text information, human model, and hand movements on different devices. The court trial system supports multi-party remote viewing to ensure the synchronous presentation of the speaker's actions and graphic and text, avoiding information isolation and enhancing the transparency and interactivity of the court trial.
[0050] Zooming In and Out of the Main Screen and Split Screen: When the hand indication action is not close to the graphic and text information on the main screen, it indicates that there is no hand indication action at this time, and only language and body movements. At this time, the human model in the split screen becomes larger and the main screen becomes smaller; when the hand has an indication action towards the graphic and text information on the main screen, the main screen becomes larger, and the hand model is superimposed on the graphic and text information on the main screen. At this time, the split screen becomes smaller. Through this dynamic adjustment, it ensures the natural connection from the split screen to the main screen, making the court trial process smoother and more intuitive.
[0051] Data processing of the hand model: For example, when using a mobile phone as a terminal, hand movements may cover most of the phone screen. To avoid excessive occlusion of graphic and text information by hand movements, the hand model can be reduced and placed above the graphic and text information on the main screen, or the transparency of the hand model can be increased so that hand gestures do not affect the expression of graphic and text information. Through these designs, while ensuring that the hand model has a guiding role, it does not hinder the clear display of information.
[0052] In addition, the human body and hands are pre-modeled in the background server and the terminal system respectively, and limb movements and hand movements are mapped to the corresponding models. The background server does not need to transmit the model data to the terminal system, that is, only real-time dynamic data is transmitted between the background server and the terminal system, and the dynamic data of each end is respectively mapped to the models of the corresponding end, which is mainly designed for different monitor sizes, resolutions, as well as user compatibility and operation habits. This can reduce latency and improve real-time performance, and can distribute the computational part of the model from the server side to the terminal, thus reducing the burden on the background server and reducing the latency of information transmission. Especially during the real-time capture and display of complex movements, the fluency during the court trial is improved. This process can also improve the accuracy of motion capture. By pre-modeling the human body and hand movements on the terminal, the motion capture algorithm can be adjusted according to the specific environment (such as the terminal screen size, camera angle, etc.), so as to obtain more accurate limb movement and hand movement data. Through local processing on the terminal device, personalized optimization can be carried out according to different monitor sizes, resolutions, and user operation habits, thus improving the rendering effect of the hand movement and limb movement models. For example, some terminals may have high-resolution monitors or more precise cameras, which can achieve more delicate motion capture and rendering effects locally.
[0053] Based on the above solutions, to ensure the high precision and dynamic adaptation of the model, machine learning techniques can also be used to optimize the real-time motion capture algorithms for the human body and hands. By continuously training the model, it can adapt to the complex changes of various postures and gestures, thus better coping with various movements and gestures that may occur during the court trial. Using machine learning techniques to optimize the real-time motion capture algorithms for the human body and hands can significantly improve the capture accuracy and real-time performance. Through deep learning models (such as convolutional neural network CNN and recurrent neural network LSTM), the system can real-time identify the dynamic movements of the human body and hands. First, video data of the speaker is collected through a high-quality camera, and the data is labeled and preprocessed. Then, using deep learning model training, key point information of the human body and hands (such as joint positions and finger trajectories) is extracted and mapped to the 3D model in real-time. This optimization not only improves the display quality of graphic and text information during the court trial, but also enhances the interactivity during the court trial process, making the speaker's movements and verbal expressions more intuitive and understandable, and improving the transparency and efficiency of the court trial.
[0054] The above specific embodiments of the present invention are only used for exemplary illustration or explanation of the principle of the present invention, and do not constitute a limitation on the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention shall be included within the protection scope of the present invention.
Claims
1. A court trial public display system based on the Internet of Things, including a court trial terminal device and a terminal device, characterized in that: The trial terminal device includes a background server, an electronic whiteboard and an information synchronization and model building module. The electronic whiteboard includes a main screen and multiple split screens, which are used to display graphic information and real-time dynamics of each speaker during the trial. The main screen displays the main graphic information, and the split screens display the human body model and hand movements of each terminal speaker in real time; the background server is responsible for data processing, information synchronization and transmission. It receives feature data from each terminal and synchronizes these data with the display information on the electronic whiteboard; information synchronization and model building: the background server establishes a three-dimensional model of the speaker and his hand, maps language features and body movement features, and displays them on the split screen of the electronic whiteboard. The hand model is superimposed on the graphic information of the main screen as a "picture-in-picture" in real time; the terminal device includes a terminal display screen, a terminal system and front and rear side cameras. The front side camera is used to capture the speaker's language, facial and body movement features; the rear side camera is used to capture the speaker's hand movements and record the movement trajectory and stop position of the finger or handheld tool at the front end of the terminal display screen; Data transmission: Hand feature identification information is transmitted to the backend server through the terminal system to build a hand model and display the hand movement process in real time.
2. The Internet of Things-based court trial public display system according to claim 1 is characterized in that: In the IoT environment, standardized protocols are used between devices for data transmission, including MQTT or CoAP lightweight protocols, to ensure real-time and bandwidth efficiency; data transmission and storage use HTTPS or VPN technology to ensure information security and avoid data leakage.
3. The Internet of Things-based court trial public display system according to claim 1 is characterized in that: The feature data of each terminal includes language feature information, facial action features, limb feature action information and hand feature identification information.
4. The Internet of Things-based court trial public display system according to claim 1 is characterized in that: The rear camera preferably uses a binocular camera, which can obtain image information of hand features while also detecting the distance between the hand and the terminal display. Thus, the hand model can be more accurately superimposed on the graphic information on the main screen in real time, thereby improving the intuitiveness of information expression.
5. A display method based on the display system according to claim 1, characterized in that: The following steps are involved: S1 Equipment collaboration and communication establishment: Through the IoT platform, each terminal and the backend server are connected through a wireless or wired network. The terminal establishes a communication relationship with the server through account login, authentication or role allocation. All devices use unified protocols and standards for data exchange to ensure efficient collaboration and real-time performance of the system. S2 Display request and information transmission: When any speaker is ready to display graphic information on the electronic whiteboard, the terminal initiates a display request. After receiving the display request, the backend server confirms the display authority and prepares to display the information. At this time, the terminal's camera starts working to collect the speaker's motion features and hand data; S3 Collection and transmission of action features: The terminal collects the speaker's language, facial expressions and body movements in real time through the front camera, while the rear camera captures the speaker's hand movements and transmits the hand features and their position on the display screen to the backend server in real time; S4 Synchronization of display information: After receiving the feature information sent by all terminals, the backend server maps the language features, body movement features and hand features to the human body model and hand model respectively, and displays the dynamics of the speakers of each terminal on the split screen of the electronic whiteboard; the main screen displays the graphic information that the speaker needs to display, and the split screen displays the speaker's human body model and hand model; S5 Combination of hand movements and graphic information: When the speaker uses hand movements to indicate graphic information, the backend server maps the hand movements to a hand model based on the hand feature information, and superimposes the hand model onto the main screen graphic information. In this way, all court personnel can not only see the speaker's speech content, but also his gestures and body movements, which enhances the expression effect of the information and ensures that all participants can see the real-time connection between the speaker's movements and graphic information during the trial; S6 Multi-party Synchronization and Remote Display: All parties’ terminals synchronize data in real time through the network, ensuring that court personnel in different locations see the same graphic information, human body models and hand movements on different devices.
6. The display method according to claim 5, characterized in that: Through the rear camera, a plane coordinate system is first established on the terminal display screen to capture the speaker's hand movements and record the movement trajectory and stop position of the fingers or handheld tools in front of the terminal display screen. The terminal system transmits each hand movement information to the backend server in real time, and the backend server synchronously superimposes the hand movement information (the stop position and movement trajectory information of the hand on the display screen) on the hand model.
7. The display method according to claim 5 or 6, characterized in that: It also includes zooming in and out of the main screen and split screen. When the hand indication movement is not close to the graphic information on the main screen, it indicates that there is no hand indication movement at this time, only language and body movement. At this time, the character model in the split screen becomes larger and the main screen shrinks. When the hand has an indication movement towards the graphic information on the main screen, the main screen becomes larger and the hand model is superimposed on the graphic information on the main screen. At this time, the split screen shrinks to ensure the natural transition from the split screen to the main screen.
8. The display method according to claim 5 or 6, characterized in that: It also includes data processing for enlarging or reducing the hand model, reducing the hand model to the upper side of the graphic information on the main screen, or increasing the transparency of the hand model to ensure that the hand has a guiding indication function without affecting the expression of the graphic information.
9. The display method according to claim 5, characterized in that: Use machine learning technology to optimize the real-time motion capture algorithm for the human body and hands, and continuously train the model to adapt to the complex changes of various postures and gestures.
10. The display method according to claim 5, characterized in that: The human body and hand are modeled in advance in the background server and terminal system respectively, and the limb movements and hand movements are mapped to the corresponding models.
