Multi-layer dynamic interactive display system and method based on lightweight posture recognition

Abstract

The invention provides a multilayer dynamic interactive display system and method based on lightweight posture recognition, the system comprises a camera, a projector, a display screen, an image display module and a posture recognition module, the camera faces a projection image of the projector, and the image display module and the posture recognition module are arranged in the projector. The posture recognition module carries out human body posture estimation and recognition based on a lightweight human body posture estimation model MoveNet, the image display module obtains data of the posture recognition module and outputs image information to the projector, and the projector projects an image to the display screen. The visitors can obtain image change and superposition display by touching the display screen, so that the interactivity of the display screen is improved, the sense of participation of the visitors is enhanced, the visitors are more willing to watch works, and the propagation attribute is stronger.

Description

Technical Field

[0001] This invention belongs to the field of multi-layer image display technology, and particularly relates to a multi-layer dynamic interactive display system and method based on lightweight pose recognition. Background Technology

[0002] Art galleries and museums typically use screens to display artworks. The order in which the works are displayed and the duration of each piece are usually pre-programmed. In today's interactive world, this setup clearly lacks interactivity, resulting in less visitor interest and poor dissemination of the artworks. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-layer dynamic interactive display system and method based on lightweight posture recognition, which aims to solve the problem of lack of interactivity in existing displays.

[0004] This invention is implemented as follows: a multi-layer dynamic interactive display system based on lightweight posture recognition, including a camera, a projector, a display screen, an image display module, and a posture recognition module. The camera faces the projected image of the projector. The image display module and the posture recognition module are disposed within the projector. The posture recognition module performs human posture estimation and recognition based on the lightweight human posture estimation model MoveNet. The image display module acquires the data from the posture recognition module and outputs image information to the projector. The projector projects the image onto the display screen.

[0005] A further technical solution of the present invention is that the posture recognition module supports simultaneous recognition of up to 6 people and detects 17 key points, including the wrist.

[0006] A further technical solution of the present invention is: the posture recognition module calls the model to recognize the human body, and returns six sets of data after each recognition. Each set of data includes the position and confidence level of the person in the image, as well as the position and confidence level of each key point. After extracting the position and confidence level of the wrist in sequence, the relevant data is sent to the corresponding port of the image display module.

[0007] A further technical solution of the present invention is that the image display module processes the image based on the OpenCV and NumPy toolkits.

[0008] Another objective of this invention is to provide a display method for a multi-layer dynamic interactive display system based on lightweight posture recognition, comprising the following steps: Step S1: The pose recognition module loads the MoveNet model and turns on the camera. After reading the image from the camera, it infers and obtains the key point data in the image and transmits it to the image display module. Step S2: The image display module includes a first layer image, a second layer image, and a third layer image. The image display module first transmits the first layer image to the projector and projects it. After receiving the data packet sent by the posture recognition module, it extracts the wrist key points and defines the variable keeptime to represent the duration of "someone". Step S3: If at least one keypoint has a confidence level greater than or equal to 0.2, the parameter keeptime is increased, and the coordinates, confidence level, and existence time of the keypoint are recorded as an array Touch. The array Touch indicates that someone has touched the keypoint; if no keypoint has a confidence level greater than or equal to 0.2, the parameter keeptime is decreased. Step S4: Detect the value of the parameter keeptime. If the value of the parameter keeptime is 0, define the second layer image mask as completely black and the third layer image mask as completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with its corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S5: Detect the value of the parameter keeptime. If the value of the parameter keeptime is the maximum value, define the second layer image mask as completely white and the third layer image mask as completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with its corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S6: Detect the value of the parameter keeptime. If the value of the parameter keeptime is between 0 and the maximum value, define the second layer image mask to generate a Gaussian distribution image, and the third layer image mask to be completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with the corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S7: The projector receives the superimposed image and outputs the image to the display screen.

[0009] A further technical solution of the present invention is: the image display module clears all contents of the buffer each time it reads the data packet sent by the posture recognition module. In step S1, if the data buffer is not empty, then step S2 is executed; if the data buffer is empty, then data is read once.

[0010] A further technical solution of the present invention is as follows: In steps S4-S6, if it is detected that the array Touch traversal has not ended, the next Touch item is selected, the brush stroke is pasted at the coordinates of the third layer mask, and it is determined whether the existence time of Touch is greater than or equal to its life cycle. If so, the Touch is removed from the array and the detection array Touch traversal is returned to see if it has ended. If not, the detection array Touch traversal is returned to see if it has ended.

[0011] A further technical solution of the present invention is: in step S6, the center of the Gaussian distribution image generated by the second layer image mask is the image center, and the radius is proportional to keeptime.

[0012] A further technical solution of the present invention is: an exit button is provided on the display screen, and pressing the exit button returns to the function of determining whether the data buffer is empty.

[0013] The beneficial effects of this invention are: visitors can experience changes and overlays of images by touching the display screen, which enhances the interactivity of the display screen, strengthens the visitors' sense of participation, makes visitors more willing to appreciate the works, and enhances the dissemination attributes. Attached Figure Description

[0014] Figure 1 This is a flowchart of the interaction method of the present invention; Figure 2 This is a schematic diagram of a specific embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0016] This invention provides a multi-layer dynamic interactive display system based on lightweight posture recognition, including a camera, a projector, a display screen, an image display module, and a posture recognition module. The camera faces the projected image of the projector. The image display module and the posture recognition module are disposed within the projector. The posture recognition module performs human posture estimation and recognition based on the lightweight human posture estimation model MoveNet. The image display module acquires the data from the posture recognition module and outputs image information to the projector. The projector projects the image onto the display screen.

[0017] Preferably, the posture recognition module supports simultaneous recognition of up to 6 people and detects 17 key points, including the wrist.

[0018] Preferably, the posture recognition module calls the model to recognize the human body, and returns six sets of data after each recognition. Each set of data includes the position and confidence level of the person in the image, as well as the position and confidence level of each key point. After extracting the position and confidence level of the wrist in sequence, the relevant data is sent to the corresponding port of the image display module.

[0019] Preferably, the image display module processes the image based on the OpenCV and NumPy toolkit.

[0020] This invention also provides a display method for a multi-layer dynamic interactive display system based on lightweight posture recognition, comprising the following steps: Step S1: The pose recognition module loads the MoveNet model and turns on the camera. After reading the image from the camera, it infers and obtains the key point data in the image and transmits it to the image display module. Step S2: The image display module includes a first layer image, a second layer image, and a third layer image. The image display module first transmits the first layer image to the projector and projects it. After receiving the data packet sent by the posture recognition module, it extracts the wrist key points and defines the variable keeptime to represent the duration of "someone". Step S3: If at least one keypoint has a confidence level greater than or equal to 0.2, the parameter keeptime is increased, and the coordinates, confidence level, and existence time of the keypoint are recorded as an array Touch. The array Touch indicates that someone has touched the keypoint; if no keypoint has a confidence level greater than or equal to 0.2, the parameter keeptime is decreased. Step S4: Detect the value of the parameter keeptime. If the value of the parameter keeptime is 0, define the second layer image mask as completely black and the third layer image mask as completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with its corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S5: Detect the value of the parameter keeptime. If the value of the parameter keeptime is the maximum value, define the second layer image mask as completely white and the third layer image mask as completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with its corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S6: Detect the value of the parameter keeptime. If the value of the parameter keeptime is between 0 and the maximum value, define the second layer image mask to generate a Gaussian distribution image, and the third layer image mask to be completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with the corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S7: The projector receives the superimposed image and outputs the image to the display screen.

[0021] Preferably, the image display module clears all contents of the buffer each time it reads the data packet sent by the pose recognition module. In step S1, if the data buffer is not empty, step S2 is executed; if the data buffer is empty, data is read once.

[0022] Preferably, in steps S4-S6, if it is detected that the array Touch traversal has not ended, the next Touch item is selected, the brush stroke is pasted at the coordinates of the third mask, and it is determined whether the existence time of the Touch is greater than or equal to its life cycle. If so, the Touch is removed from the array and the detection array Touch traversal is returned to check if it has ended. If not, the detection array Touch traversal is returned to check if it has ended.

[0023] Preferably, in step S6, the center of the Gaussian distribution image generated by the second image mask is the image center, and the radius is proportional to keeptime.

[0024] Preferably, the display screen is provided with an exit button, which, when pressed, returns to the function of determining whether the data buffer is empty.

[0025] This invention has already been used in the installation art piece "Jiangxia Preface: Looking Back, Image, and Distant Gaze" in the Guan Shanyue Art Museum exhibition "Tracing the Origins of Handwriting—Guan Shanyue's Sketching Journey from an Aerial Perspective." This invention can be applied to art galleries and museums, or to the fields of cultural tourism and cultural heritage. It presents and integrates different layers of visuals during visitor interaction, achieving a digital reproduction of cultural IPs and historical scenes, or can be integrated into cultural experiences and study tours for visual teaching.

[0026] The system needs to include an RGB camera and a projector. The software is written in Python and includes two modules: posture recognition and image display. The two modules communicate with each other over a local network using the UDP protocol.

[0027] The pose recognition module uses the lightweight human pose estimation model MoveNet for human pose estimation and recognition. It follows the Apache 2.0 license, supports simultaneous recognition of up to six people, and can detect 17 key points, including the wrist. After loading the MoveNet model and opening the camera, the program calls the model to recognize the human body. After each recognition, it returns six sets of data. Each set includes the person's position and confidence score in the image, as well as the position and confidence score of each key point. The wrist position and confidence score are extracted sequentially and then sent to the corresponding port of the image display module. The image display module uses OpenCV and NumPy toolkits to process and present the images.

[0028] The image is divided into three layers. The first layer is displayed when no key point (wrist) is detected within the camera's range. When a key point is detected, the second layer gradually expands from the center of the image and completely becomes the second layer after a period of time. The position of the key point in the pose recognition module is mapped to the image coordinates of this module, and the corresponding position becomes the third layer for a short time. The effect of the third layer gradually disappears after the person moves away. If the camera does not detect a key point for a period of time, the image will gradually change back from the second layer to the first layer.

[0029] Each layer of the image can be a static image or a looping animation. When using animation, the video is extracted and loaded into memory frame by frame as the program starts running. The current frame to be played is calculated based on the difference between the current time and the program's start time.

[0030] After receiving the UDP data packets sent by the posture recognition module, the program extracts relevant information for 6 people, totaling 12 wrist keypoints. If at least one keypoint has a confidence score greater than 0.2, it considers "someone" to be present in the camera frame; the confidence score is the data generated by MoveNet after recognition. The variable `keeptime` represents the duration of "someone" in the frame, ranging from 0 to a set maximum value, with an initial value of 0. If the frame is empty, `keeptime` gradually decreases. To reduce latency, the buffer is cleared each time the UDP port of the posture recognition module is read; if there is no content, the process is skipped without a waiting time. Each keypoint with a confidence score greater than 0.2 generates a `Touch` and stores it in an array. Each `Touch` has a certain lifespan and records the corresponding coordinates and confidence score of the keypoint. When the current duration exceeds its lifespan, the `Touch` is removed from the array. Touch is a class created by MoveNet each time a keypoint is detected. The coordinates, confidence level, and existence time of the keypoint are recorded (initially 0). When the program enters a loop, the existence time of Touch is updated. If it exceeds its lifespan, it is deleted. Then, a full white grayscale image is taken, and all Touches are traversed. A stroke image is "pasted" with the recorded coordinates as the center. Its existence (creation) is related to the visitor, that is, it will be continuously created when the visitor's wrist is detected; its disappearance is unrelated to the visitor and depends on its own lifespan.

[0031] Define the image "paste" operation: multiply the corresponding elements of the image normalized to [0,1] in sequence.

[0032] Define the second layer mask: if keeptime is 0, it is completely black; if it is the maximum value, it is completely white; between 0 and the maximum value, a two-dimensional Gaussian distribution image is generated, with the center being the image center and the radius being proportional to keeptime, so as to create a "diffusion" effect when keeptime changes.

[0033] Define a third-layer mask: initially, it is completely white. It iterates through all existing Touches and pastes the brush stroke image at the corresponding position. The transparency of the mask is related to the confidence level of the stored key points. After the iteration is complete, the mask is inverted, meaning that the denser the key points in a region, the closer the corresponding value is to 1. The corresponding value is the normalized coordinate of the key points in the pose recognition module, which is then mapped to the image display module.

[0034] After the two masks are generated, the first mask is the difference between the pure white image and the two masks. The three images are multiplied by the masks and then added together to obtain the final display image. For the "multiplication" of the RGB image and the grayscale image, the value of each channel at the corresponding pixel is multiplied by the grayscale value.

[0035] In actual operation, the camera will take real-time pictures of the display screen and extract key points in the captured images through the posture recognition module. If a visitor touches the screen, and the confidence level of the key point is greater than or equal to 0.2, it is determined that the touch was caused by the visitor tapping the screen, and at this time it is judged that "someone is here".

[0036] The visual effects vary depending on the keeptime value. If keeptime is 0, it means no one is near the display, so the second layer image is not displayed, and the display will always show the first layer image. If keeptime is between 0 and its maximum value, a person appears and touches the display, and the second layer image gradually spreads from the center until it is completely displayed. If keeptime is at its maximum value, a third layer image appears at the corresponding position on the person's wrist. In other words, if no one is present, the display always shows the first layer image. If someone appears and touches the display, the second layer image spreads from the center, creating a superimposed image of the first and second layers until the entire screen displays the second layer image. If the visitor keeps touching the display, a third layer image appears at the point of contact, creating a superimposed image of the second and third layers. When the visitor leaves, the screen reverts to the first layer image. This interactive format makes the exhibition more interesting, stimulates visitor interest, and enhances interactivity. Specific visual demonstrations are as follows: Figure 2 As shown.

[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-layer dynamic interactive display system based on lightweight posture recognition, characterized in that, The device includes a camera, a projector, a display screen, an image display module, and a posture recognition module. The camera projects an image onto the projector. The image display module and the posture recognition module are located inside the projector. The posture recognition module estimates and recognizes human posture based on the lightweight human posture estimation model MoveNet. The image display module acquires data from the posture recognition module and outputs image information to the projector. The projector projects the image onto the display screen.

2. The multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 1, characterized in that, The posture recognition module supports simultaneous recognition of up to 6 people and detects 17 key points, including the wrist.

3. The multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 1, characterized in that, The posture recognition module calls the model to recognize the human body. After each recognition, it returns six sets of data. Each set of data includes the position and confidence level of the person in the image, as well as the position and confidence level of each key point. After extracting the position and confidence level of the wrist, the relevant data is sent to the corresponding port of the image display module.

4. The multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 1, characterized in that, The image display module processes images based on the OpenCV and NumPy toolkits.

5. A display method for a multi-layer dynamic interactive display system based on lightweight posture recognition as described in any one of claims 1-4, characterized in that, Includes the following steps: Step S1: The pose recognition module loads the MoveNet model and turns on the camera. After reading the image from the camera, it infers and obtains the key point data in the image and transmits it to the image display module. Step S2: The image display module includes a first layer image, a second layer image, and a third layer image. The image display module first transmits the first layer image to the projector and projects it. After receiving the data packet sent by the posture recognition module, it extracts the wrist key points and defines the variable keeptime to represent the duration of "someone". Step S3: If the confidence of at least one key point is greater than or equal to 0.2, the parameter keeptime is increased, and the coordinates, confidence, and existence time of the key point are recorded as an array Touch. The array Touch indicates that someone has touched the key point. If the confidence level of no key point is greater than or equal to 0.2, the parameter keeptime is reduced. Step S4: Detect the value of the parameter keeptime. If the value of the parameter keeptime is 0, define the second layer image mask as completely black and the third layer image mask as completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with its corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S5: Detect the value of the parameter keeptime. If the value of the parameter keeptime is the maximum value, define the second layer image mask as completely white and the third layer image mask as completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with its corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S6: Detect the value of the parameter keeptime. If the value of the parameter keeptime is between 0 and the maximum value, define the second layer image mask to generate a Gaussian distribution image, and the third layer image mask to be completely white. Detect whether the Touch array has finished traversing. If so, invert the third layer image mask. The first layer image is the difference between the completely white image and the second and third layer image masks. Multiply each layer image with the corresponding mask, and then superimpose the three layers of images and transmit the image to the projector. Execute step S7. Step S7: The projector receives the superimposed image and outputs the image to the display screen.

6. The display method of a multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 5, characterized in that, Each time the image display module reads a data packet sent by the pose recognition module, it clears all contents of the buffer. In step S1, if the data buffer is not empty, step S2 is executed; if the data buffer is empty, data is read once.

7. The display method of a multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 5, characterized in that, In steps S4-S6, if it is detected that the array Touch traversal has not ended, the next Touch item is selected, the brush stroke is pasted at the coordinates of the third mask, and it is determined whether the existence time of Touch is greater than or equal to its life cycle. If so, the Touch is removed from the array and the detection array Touch traversal is returned to check if it has ended. If not, the detection array Touch traversal is returned to check if it has ended.

8. The display method of a multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 5, characterized in that, In step S6, the second image mask generates a Gaussian distributed image with the image center as the center and the radius proportional to keeptime.

9. The display method of a multi-layer dynamic interactive display system based on lightweight posture recognition according to claim 6, characterized in that, An exit button is provided on the display screen. Pressing the exit button returns to the function of determining whether the data buffer is empty.

Images