Optical flow particle interaction system based on real-time attitude detection

Abstract

The invention discloses an optical flow particle interaction system based on real-time attitude detection, and belongs to the technical field of interaction systems. A neural network engine; the attitude estimation layer is connected with the heterogeneous sensing layer, and performs real-time attitude estimation of occlusion robustness through continuous time graph convolution and a sparse space-time attention mechanism; rendering the hybrid particle engine; the server is connected with the neural network engine; the cloud side end computing framework comprises an edge node, a cloud computing cluster and a digital twin scheduler, and the digital twin scheduler computes task allocation according to a map; and the application interface layer is used for providing a standard interface and a development kit, so that a particle interaction function can be conveniently integrated in a third-party application. According to the invention, the problems that the current interaction model is not high enough in human body posture detection precision and poor in adaptability, creative requirements and application scene requirements of different users cannot be met, a new interface cannot be accessed, and the cost of establishing an interaction system by a user is greatly increased are solved.

Description

Technical Field

[0001] This invention belongs to the field of interactive system technology, and in particular relates to an optical flow particle interaction system based on real-time attitude detection. Background Technology

[0002] In the field of human pose detection, some tech giants have invested heavily in the research and development of deep learning models. The Hourglass model and HRNet model based on convolutional neural networks, as well as the VisionTransformer model based on Transformer, have been widely used and continuously optimized internationally. These models have performed well on mainstream datasets such as COCO, with their average accuracy continuously improving, laying a solid technical foundation for real-time pose detection. With the rapid development of artificial intelligence technology, domestic enterprises are increasingly demanding human posture detection technology, which has driven the rapid iteration of the technology. In practical applications, some domestic enterprises have applied posture detection technology to fields such as security monitoring and sports training. However, in terms of high-precision real-time detection and integration with interactive systems, the current interactive models do not have high enough accuracy in detecting human posture, have poor adaptability, cannot meet the creative needs and application scenario requirements of different users, and cannot access new interfaces, which greatly increases the cost for users to build interactive systems. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of current interactive models in terms of high-precision real-time detection and integration with interactive systems. These models lack sufficient accuracy in detecting human posture, have poor adaptability, fail to meet the creative needs and application scenarios of different users, and cannot access new interfaces, thus significantly increasing the cost for users to build interactive systems. Therefore, this invention proposes an optical flow particle interactive system based on real-time posture detection.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: an optical flow particle interaction system based on real-time attitude detection, comprising: Heterogeneous perception layer; used for synchronously acquiring multimodal visual data from RGB image streams and event camera streams; A neural network engine, connected to the heterogeneous perception layer, is used to model human key points as dynamic spatiotemporal graph nodes and perform occlusion-robust real-time pose estimation through continuous temporal graph convolution and sparse spatiotemporal attention mechanism. A hybrid particle rendering engine is used; connected to the neural network engine, it receives attitude estimation results and drives particle physics simulation, allowing particles to interact with the virtual environment through light and shadow. Cloud-edge-device computing framework: It includes edge nodes, cloud computing clusters, and a digital twin scheduler, which allocates computing tasks based on the map; The application interface layer provides standard interfaces and development toolkits to facilitate the integration of particle interaction functionality into third-party applications.

[0005] Furthermore, the aforementioned neural network engine includes a spatial graph construction module, a temporally continuous modeling module, a sparse spatiotemporal attention calculation module, and an occlusion inference module; Among them, the spatial graph construction module is used to define human joints as points on the graph and learn the connection weights between nodes based on the laws of human movement. The time-continuous modeling module uses neural constant differential equations to perform time-series differential dynamics modeling of the motion trajectory of joints, supporting attitude interpolation and prediction under arbitrary frame rate input. Sparse spatiotemporal attention computation module: calculates attention weights between temporal neighborhoods and spatially adjacent joint nodes, reducing computational complexity from quadratic to linear. Occlusion reasoning module: Infers the location of occluded joints based on graph structure relationships.

[0006] Furthermore, the aforementioned rendering hybrid particle engine includes a physics simulator, a particle hybrid renderer, and a two-way coupling control module; The physics simulator is used to set particles to different physical properties and can also receive feedback on attitude estimation. Particle blending renderer; allows particles to participate in lighting calculations in the virtual environment, enabling light and shadow interaction; A bidirectional coupling control module is used to correlate the confidence level of attitude estimation with particle collision feedback to optimize detection and rendering.

[0007] Furthermore, a multimodal fusion module is provided between the heterogeneous perception layer and the neural network engine. The multimodal fusion module is used to align discrete event streams and frame streams to a unified coordinate system, automatically adjust the fusion weights of color images and event camera stream data and supervise optical flow estimation units according to different situations, and can fine-tune the pose estimation model in unlabeled scenarios.

[0008] Furthermore, the aforementioned multimodal fusion module can also use the luminance variance of the RGB image and the hole rate of the depth map as quality indicators.

[0009] Furthermore, the above also includes an optical flow particle interaction module, which generates a dense optical flow field based on the joint movement, driving particles to produce dynamic effects according to preset rules.

[0010] Furthermore, the above also includes a dataset construction module for collecting and labeling large-scale human pose datasets, covering various scenes, poses, lighting conditions and crowd characteristics, establishing an efficient data management and update mechanism, and continuously optimizing dataset quality to improve the training effect and generalization ability of the model.

[0011] This invention provides an optical flow particle interaction system based on real-time attitude detection. A heterogeneous perception layer is used to synchronously acquire multimodal visual data such as RGB image streams and event camera streams, providing rich visual information for subsequent processing. A subsequent multimodal fusion module aligns discrete event streams and frame streams to a unified coordinate system, automatically adjusts the fusion weights of color images and event camera stream data according to different scenarios, performs supervised optical flow estimation, and calculates RGB... The brightness variance of the image and the hole rate of the depth map are used as quality indicators to fine-tune the pose estimation model in unlabeled scenes. A neural network engine, tightly connected to the heterogeneous perception layer, includes a spatial graph construction module, a temporal continuous modeling module, a sparse spatiotemporal attention calculation module, and an occlusion inference module. The spatial graph construction module defines human joints as points on the graph and learns the connection weights between nodes. The temporal continuous modeling module uses neural network constant differential equations to perform temporal differential dynamics modeling of joint motion trajectories, supporting pose interpolation and prediction under arbitrary frame rate input. The sparse spatiotemporal attention calculation module calculates attention weights between temporally adjacent and spatially adjacent joint nodes, reducing computational complexity. The occlusion inference module infers the position of occluded joints based on graph structure relationships, thus achieving occlusion-robust real-time pose estimation. A rendering hybrid particle engine, connected to the neural network engine, consists of a physics simulator, a particle hybrid renderer, and a bidirectional coupling control module. The physics simulator sets different physical properties for particles and receives pose estimation feedback. The particle hybrid renderer allows particles to participate in virtual environment lighting calculations, achieving light and shadow interaction. The bidirectional coupling control module correlates pose estimation confidence with particle collision feedback. The system optimizes detection and rendering effects. Furthermore, it includes an optical flow particle interaction module that generates a dense optical flow field based on joint movement, driving particles to produce dynamic effects according to preset rules. The system's cloud-edge-device computing framework comprises edge nodes, a cloud computing cluster, and a digital twin scheduler. The digital twin scheduler constructs a digital map of the exhibition hall, models devices as interactive points, and generates optimal computing strategies based on actual conditions. This allows edge nodes to handle simple tasks, while the cloud handles complex calculations and tracks the same person under multiple cameras. Finally, the application interface layer provides standard interfaces and development toolkits, facilitating the integration of particle interaction functions into third-party applications to meet the needs of different scenarios. This interactive system combines human posture detection results with the particle system to achieve particle effect interaction driven by human motion. Combined with optical flow tracking technology, it implements particle effect interaction. For example, in a virtual music performance scene, when a singer waves on stage, the system recognizes the waving motion through posture detection and drives particles to spray from the singer's hand according to a preset waving particle spray template. Simultaneously, optical flow tracking technology enables the particles to smoothly follow the trajectory of the singer's arm, creating dazzling effects.When the singer starts running, the particle system uses a running particle flow template to make the particles flow in the direction the singer is running, creating a dynamic atmosphere; users can also adjust the color and spray speed of the particles according to their own creativity to create unique performance effects.

[0012] Therefore, this embodiment has the following advantages compared to the prior art: An optical flow particle interaction system based on real-time posture detection solves the problems of insufficient accuracy in human posture detection and poor adaptability in high-precision real-time detection and integration with interactive systems. These problems prevent the current model from meeting the creative needs and application scenarios of different users, and also prevent the access to new interfaces, which greatly increases the cost for users to build interactive systems. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a system block diagram of an optical flow particle interaction system based on real-time attitude detection. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0017] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0018] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0019] In the description of the embodiments of the present invention, it should be noted that the terms "upper" and "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0020] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Example 1:

[0021] Please see Figure 1 This invention provides a technical solution: an optical flow particle interaction system based on real-time attitude detection, comprising: Heterogeneous perception layer; used for synchronously acquiring multimodal visual data from RGB image streams and event camera streams; A neural network engine, connected to the heterogeneous perception layer, is used to model human key points as dynamic spatiotemporal graph nodes and perform occlusion-robust real-time pose estimation through continuous temporal graph convolution and sparse spatiotemporal attention mechanism. A hybrid particle rendering engine is used; connected to the neural network engine, it receives attitude estimation results and drives particle physics simulation, allowing particles to interact with the virtual environment through light and shadow. Cloud-edge-device computing framework: It includes edge nodes, cloud computing clusters, and a digital twin scheduler, which allocates computing tasks based on the map; The application interface layer provides standard interfaces and development toolkits to facilitate the integration of particle interaction functionality into third-party applications.

[0022] Specifically, the neural network engine includes a spatial graph construction module, a temporal continuous modeling module, a sparse spatiotemporal attention calculation module, and an occlusion inference module; Among them, the spatial graph construction module is used to define human joints as points on the graph and learn the connection weights between nodes based on the laws of human movement. The time-continuous modeling module uses neural constant differential equations to perform time-series differential dynamics modeling of the motion trajectory of joints, supporting attitude interpolation and prediction under arbitrary frame rate input. Sparse spatiotemporal attention computation module: calculates attention weights between temporal neighborhoods and spatially adjacent joint nodes, reducing computational complexity from quadratic to linear. Occlusion reasoning module: Infers the location of occluded joints based on graph structure relationships.

[0023] Specifically, the rendering hybrid particle engine includes a physics simulator, a particle hybrid renderer, and a two-way coupling control module; The physics simulator is used to set particles to different physical properties and can also receive feedback on attitude estimation. Particle blending renderer; allows particles to participate in lighting calculations in the virtual environment, enabling light and shadow interaction; A bidirectional coupling control module is used to correlate the confidence level of attitude estimation with particle collision feedback to optimize detection and rendering.

[0024] A heterogeneous perception layer is set up to synchronously acquire multimodal visual data such as RGB image streams and event camera streams, providing rich visual information for subsequent processing. The subsequent multimodal fusion module aligns the discrete event streams and frame streams to a unified coordinate system, automatically adjusts the fusion weights of color images and event camera stream data according to different scenes, performs supervised optical flow estimation, and calculates RGB... The brightness variance of the image and the hole rate of the depth map are used as quality indicators to fine-tune the pose estimation model in unlabeled scenes. A neural network engine, tightly connected to the heterogeneous perception layer, includes a spatial graph construction module, a temporal continuous modeling module, a sparse spatiotemporal attention calculation module, and an occlusion inference module. The spatial graph construction module defines human joints as points on the graph and learns the connection weights between nodes. The temporal continuous modeling module uses neural network constant differential equations to perform temporal differential dynamics modeling of joint motion trajectories, supporting pose interpolation and prediction under arbitrary frame rate input. The sparse spatiotemporal attention calculation module calculates attention weights between temporally adjacent and spatially adjacent joint nodes, reducing computational complexity. The occlusion inference module infers the position of occluded joints based on graph structure relationships, thus achieving occlusion-robust real-time pose estimation. A rendering hybrid particle engine, connected to the neural network engine, consists of a physics simulator, a particle hybrid renderer, and a bidirectional coupling control module. The physics simulator sets different physical properties for particles and receives pose estimation feedback. The particle hybrid renderer allows particles to participate in virtual environment lighting calculations, achieving light and shadow interaction. The bidirectional coupling control module correlates pose estimation confidence with particle collision feedback. The system optimizes detection and rendering effects. Furthermore, it includes an optical flow particle interaction module that generates a dense optical flow field based on joint movement, driving particles to produce dynamic effects according to preset rules. The system's cloud-edge-device computing framework comprises edge nodes, a cloud computing cluster, and a digital twin scheduler. The digital twin scheduler constructs a digital map of the exhibition hall, models devices as interactive points, and generates optimal computing strategies based on actual conditions. This allows edge nodes to handle simple tasks, while the cloud handles complex calculations and tracks the same person under multiple cameras. Finally, the application interface layer provides standard interfaces and development toolkits, facilitating the integration of particle interaction functions into third-party applications to meet the needs of different scenarios. This interactive system combines human posture detection results with the particle system to achieve particle effect interaction driven by human motion. Combined with optical flow tracking technology, it implements particle effect interaction. For example, in a virtual music performance scene, when a singer waves on stage, the system recognizes the waving motion through posture detection and drives particles to spray from the singer's hand according to a preset waving particle spray template. Simultaneously, optical flow tracking technology enables the particles to smoothly follow the trajectory of the singer's arm, creating dazzling effects. When the singer starts running, the particle system uses a running particle flow template to make the particles flow in the direction the singer is running, creating a dynamic atmosphere; users can also adjust the color and spray speed of the particles according to their own creativity to create unique performance effects.

[0025] Therefore, this embodiment has the following advantages compared to the prior art: An optical flow particle interaction system based on real-time posture detection solves the problems of insufficient accuracy in human posture detection and poor adaptability of current interaction models in terms of high-precision real-time detection and integration with interaction systems. These problems fail to meet the creative needs and application scenarios of different users and cannot access new interfaces, which greatly increases the cost for users to build interaction systems. Example 2:

[0026] See Figure 1 The figure shows an optical flow particle interaction system based on real-time attitude detection provided in Embodiment 2 of the present invention. Based on the above embodiments, the following improved technical solutions are made: a multimodal fusion module is set between the heterogeneous perception layer and the neural network engine. The multimodal fusion module is used to align discrete event streams and frame streams to a unified coordinate system, automatically adjust the fusion weights of color images and event camera stream data and supervise the optical flow estimation unit according to different situations, and can fine-tune the attitude estimation model in unlabeled scenarios. Example 3:

[0027] See Figure 1 The figure shows an optical flow particle interaction system based on real-time attitude detection provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment further makes the following improved technical solutions: the digital twin scheduler is used to construct a digital map of the exhibition hall and model the equipment as interactive points; the optimal calculation strategy is generated according to the actual situation, the edge nodes are processed simply, the cloud performs complex calculations, and the same person is tracked by multiple cameras. Example 4:

[0028] See Figure 1 The figure shows an optical flow particle interaction system based on real-time attitude detection provided in Embodiment 3 of the present invention. Based on the above embodiments, the following improved technical solutions are made: the multimodal fusion module can also use the calculated brightness variance of the RGB image and the hole rate of the depth map as quality indicators. Example 5:

[0029] See Figure 1The figure illustrates an optical flow particle interaction system based on real-time posture detection provided in Embodiment 3 of the present invention. This embodiment further improves upon the previous embodiments by including an optical flow particle interaction module. This module generates a dense optical flow field based on joint motion, driving particles to produce dynamic effects according to preset rules. By combining human posture detection results with the particle system, particle effects interaction based on human motion is achieved, and this is further enhanced by combining optical flow tracking technology. For example, in a virtual scene, a user's waving gesture can trigger a particle spray effect, and running can cause changes in the particle's flow trajectory. Multiple particle interaction effect templates are developed to meet the creative needs and application scenario requirements of different users. Example 6:

[0030] See Figure 1 The figure shows an optical flow particle interaction system based on real-time posture detection provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment further makes the following improved technical solutions: it also includes a dataset construction module for collecting and annotating large-scale human posture datasets, covering various scenes, postures, lighting conditions and crowd characteristics, establishing an efficient data management and update mechanism, and continuously optimizing the dataset quality to improve the training effect and generalization ability of the model.

[0031] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An optical flow particle interaction system based on real-time attitude detection, characterized in that, include: Heterogeneous sensing layer; Used for synchronous acquisition of multimodal visual data from RGB image streams and event camera streams; Neural network engine; Connected to the heterogeneous perception layer, it is used to model human key points as dynamic spatiotemporal graph nodes, and to perform occlusion-robust real-time pose estimation through continuous temporal graph convolution and sparse spatiotemporal attention mechanism. A hybrid particle rendering engine is used; connected to the neural network engine, it receives attitude estimation results and drives particle physics simulation, allowing particles to interact with the virtual environment through light and shadow. Cloud-edge-device computing framework: It includes edge nodes, cloud computing clusters, and a digital twin scheduler, which allocates computing tasks based on the map; The application interface layer provides standard interfaces and development toolkits to facilitate the integration of particle interaction functionality into third-party applications.

2. The optical flow particle interaction system based on real-time attitude detection according to claim 1, characterized in that, The neural network engine includes a spatial graph construction module, a temporally continuous modeling module, a sparse spatiotemporal attention calculation module, and an occlusion inference module. Among them, the spatial graph construction module is used to define human joints as points on the graph and learn the connection weights between nodes based on the laws of human movement. The time-continuous modeling module uses neural constant differential equations to perform time-series differential dynamics modeling of the motion trajectory of joints, supporting attitude interpolation and prediction under arbitrary frame rate input. Sparse spatiotemporal attention computation module: calculates attention weights between temporal neighborhoods and spatially adjacent joint nodes, reducing computational complexity from quadratic to linear. Occlusion reasoning module: Infers the location of occluded joints based on graph structure relationships.

3. The optical flow particle interaction system based on real-time attitude detection according to claim 1, characterized in that, The rendering hybrid particle engine includes a physics simulator, a particle blending renderer, and a two-way coupling control module; The physics simulator is used to set particles to different physical properties and can also receive feedback on attitude estimation. Particle blending renderer; allows particles to participate in lighting calculations in the virtual environment, enabling light and shadow interaction; A bidirectional coupling control module is used to correlate the confidence level of attitude estimation with particle collision feedback to optimize detection and rendering.

4. The optical flow particle interaction system based on real-time attitude detection according to claim 3, characterized in that, A multimodal fusion module is provided between the heterogeneous perception layer and the neural network engine. The multimodal fusion module is used to align discrete event streams and frame streams to a unified coordinate system, automatically adjust the fusion weights of color images and event camera stream data and supervise optical flow estimation units according to different situations, and can fine-tune the pose estimation model in unlabeled scenarios.

5. The optical flow particle interaction system based on real-time attitude detection according to claim 1, characterized in that, The digital twin scheduler is used to construct a digital map of the exhibition hall, modeling devices as interactive points; it generates the optimal calculation strategy based on the actual situation, performs simple processing on edge nodes, performs complex calculations in the cloud, and tracks the same person under multiple cameras.

6. The optical flow particle interaction system based on real-time attitude detection according to claim 4, characterized in that, The multimodal fusion module can also use the luminance variance of the RGB image and the hole rate of the depth map as quality indicators.

7. The optical flow particle interaction system based on real-time attitude detection according to claim 1, characterized in that, It also includes an optical flow particle interaction module, which generates a dense optical flow field based on the joint movement, driving particles to produce dynamic effects according to preset rules.

8. The optical flow particle interaction system based on real-time attitude detection according to claim 1, characterized in that, It also includes a dataset building module for collecting and annotating large-scale human pose datasets, covering various scenes, poses, lighting conditions and crowd characteristics, establishing an efficient data management and update mechanism, and continuously optimizing dataset quality to improve the training effect and generalization ability of the model.

Images