Intelligent projection method, system, medium and program product based on gesture recognition

Abstract

A method, system, medium and program product for intelligent projection based on gesture recognition, in which human behavior data is collected through a distributed multimodal sensor network; the human behavior data is preprocessed using a self-organizing data cleaning algorithm to obtain denoised human behavior data; the denoised human behavior data is input into a dynamic spatiotemporal graph network to extract human behavior feature data; based on the human behavior feature data, a hierarchical semantic parser is used to perform multi-scale parsing of human behavior to obtain parsing results, the parsing process includes action recognition, intention inference and sentiment analysis, and the parsing results include predicted intention and predicted sentiment; according to the parsing results, an environmental response generator is used to determine the target projection content from a preset dynamic knowledge graph, and the projection device is controlled to play the target projection content. The application reduces the complexity of user operations, increases user selectivity, and improves adaptability to user preferences.

Description

Technical Field

[0001] The present application belongs to the field of gesture recognition, and in particular, relates to an intelligent projection method, system, medium and program product based on gesture recognition. Background Art

[0002] With the rapid development of technology, projection technology widely used in cinemas has gradually been adopted by families. Many projectors have appeared on the market, which can project movies and TV series selected by users at home. Similarly, projectors can also project content that matches the current environment to bring users an immersive experience.

[0003] In the related art, it is usually necessary for users to manually select the projection content and scene. Users select several predetermined projection contents through a remote control or download the projection contents by themselves, and then the projector projects them. This not only increases the complexity of user operation, but also reduces the user's selectivity and reduces the adaptability to user preferences. Summary of the invention

[0004] The present application provides a method, system, medium and program product for intelligent projection based on gesture recognition, which are used to reduce the complexity of user operations, increase user selectivity, and improve adaptability to user preferences.

[0005] In a first aspect, the present application provides an intelligent projection method based on gesture recognition, wherein human behavior data is collected through a distributed multimodal sensor network, wherein the distributed multimodal sensor network includes an optical sensor, an acoustic sensor, and an electromagnetic sensor;

[0006] The human behavior data is preprocessed using a self-organizing data cleaning algorithm to obtain denoised human behavior data;

[0007] Input the denoised human behavior data into the dynamic spatiotemporal graph network to extract the human behavior feature data;

[0008] Based on human behavior feature data, a hierarchical semantic parser is used to perform multi-scale analysis of human behavior to obtain analysis results. The analysis process includes action recognition, intention inference and sentiment analysis. The analysis results include predicted intention and predicted sentiment.

[0009] According to the analysis results, the environment response generator is used to determine the target projection content from the preset dynamic knowledge graph, and the projection device is controlled to play the target projection content.

[0010] By adopting the above technical solutions, the distributed multimodal sensor network collects human behavior data and realizes comprehensive perception of human behavior. The use of multimodal sensors ensures the multidimensionality of the data and improves the accuracy of behavior recognition. The application of the self-organizing data cleaning algorithm effectively removes the noise in the original data and improves the effectiveness of subsequent processing. The multi-scale parsing process of the hierarchical semantic parser, including action recognition, intent inference and sentiment analysis, enables the system to deeply understand the user's behavior and psychological state. This multi-level understanding provides a basis for subsequent content recommendations. The environmental response generator uses a preset dynamic knowledge graph to determine the target projection content, realizing personalized and context-aware content recommendations. This technical solution reduces the complexity of user operations, increases user selectivity, and improves adaptability to user preferences.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, preprocessing the human behavior data using a self-organizing data cleaning algorithm to obtain denoised human behavior data specifically includes:

[0012] Use timestamp information to synchronize human behavior data from different sensors and generate aligned multimodal data sequences;

[0013] Based on the multimodal data sequence, an adaptive noise model is constructed, and the adaptive noise model is used to dynamically adjust the filtering parameters;

[0014] Apply the adaptive noise model to human behavior data to obtain preliminary cleaned data;

[0015] Perform multi-scale wavelet decomposition on the preliminary cleaned data to extract human behavior characteristics in different frequency domains;

[0016] Generate denoised human behavior data based on human behavior characteristics.

[0017] By adopting the above technical solution, the multimodal data is synchronized using timestamp information, which solves the time inconsistency problem between different sensor data and lays the foundation for subsequent multimodal fusion. This step ensures that the data from different sensors can be aligned in the time dimension, so as to accurately reflect the human behavior state at the same time. Constructing an adaptive noise model and dynamically adjusting the filter parameters is more flexible than the fixed parameter filter, and can adapt to the noise characteristics in different environments, thus improving the effect of noise removal. The application of multi-scale wavelet decomposition enables the system to extract human behavior features from different frequency domains. This step enhances the comprehensiveness and meticulousness of feature extraction. The quality of the final denoised human behavior data is improved, the interference of noise on subsequent analysis is reduced, the key features of the behavior data are retained, and the recognition accuracy and stability of the entire system are improved.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, constructing an adaptive noise model based on a multimodal data sequence specifically includes:

[0019] Perform spectrum analysis on historical human behavior data to identify the characteristic frequency ranges of different types of noise;

[0020] Set up a bandpass filter with adjustable parameters and use an online learning algorithm to adjust the filter parameters based on real-time data streams;

[0021] Determine the noise level evaluation index and obtain the filtering effect score of the bandpass filter;

[0022] Determine whether the filtering effect score is lower than a preset threshold;

[0023] If the filtering effect is lower than a preset threshold, a step of performing spectrum analysis on the historical human behavior data to identify characteristic frequency ranges of different types of noise is performed to reconstruct a bandpass filter;

[0024] If the filtering effect is not lower than the preset threshold, the step of applying the adaptive noise model to the human behavior data to obtain preliminary cleaned data is performed.

[0025] By adopting the above technical solutions, the historical human behavior data is spectrally analyzed to identify the characteristic frequency ranges of different types of noise. This step provides an initial reference for the noise model, enabling the system to deal with different types of noise in a targeted manner. Setting a bandpass filter with adjustable parameters and dynamically adjusting the parameters using an online learning algorithm enables the system to adapt to environmental changes and changes in data characteristics in real time, maintaining the optimality of the filtering effect. By setting noise level evaluation indicators and periodically evaluating the filtering effect, the system can objectively measure the performance of the current filter and promptly detect performance degradation. The introduction of a preset threshold mechanism triggers the reconstruction process when the filtering effect is poor, ensuring that the system maintains efficient denoising capabilities for a long time. The quality of data cleaning is improved, and the long-term availability and stability of the system are enhanced. High-quality denoising results directly affect the accuracy of subsequent feature extraction and behavior recognition, thereby improving the performance and user experience of the entire intelligent projection system.

[0026] In conjunction with some embodiments of the first aspect, in some embodiments, a hierarchical semantic parser is used to perform multi-scale parsing on human behavior to obtain a parsing result, which specifically includes:

[0027] Use convolutional neural networks to extract low-level action features, including limb positions and motion trajectories;

[0028] Based on low-level action features, recurrent neural networks are used to model temporal dependencies and identify complex action sequences;

[0029] The intention is predicted based on the complex action sequence combined with the conditional random field model;

[0030] Based on the facial expressions, voice features and physiological signals in the human behavior feature data, a multimodal fusion network is used to analyze the emotional state and obtain the predicted emotion.

[0031] Integrate predicted intent and predicted sentiment to generate parsing results.

[0032] By adopting the above technical solutions, convolutional neural networks are used to extract low-level action features, such as limb positions and motion trajectories, which lays the foundation for subsequent high-level understanding. The use of convolutional neural networks can effectively capture spatial features and improve the accuracy and robustness of underlying feature extraction. Using recurrent neural networks to model temporal dependencies and identify complex action sequences, this step enables the system to understand continuous, temporally related behaviors, enhancing the depth and breadth of behavior recognition. Integrating predicted intentions and predicted emotions to generate the final analysis results achieves a multi-dimensional and multi-level understanding of human behavior. The intelligence of the system is improved, making the projection content more in line with the actual needs and psychological state of the user.

[0033] In combination with some embodiments of the first aspect, in some embodiments, according to the parsing result, the target projection content is determined from a preset dynamic knowledge graph using an environment response generator, specifically including:

[0034] Convert the predicted intent and predicted sentiment into a query vector, which contains the intent category, sentiment polarity and intensity information;

[0035] Use the query vector to perform similarity search in the preset dynamic knowledge graph to obtain a set of content nodes;

[0036] Sort the content node set by relevance according to a preset sorting standard, where the preset sorting standard includes similarity with the query vector, historical playback feedback, and adaptability to the current environment;

[0037] Select the first several content nodes in the relevance ranking to build a candidate content pool;

[0038] According to preset content combination rules, multiple content elements are selected and combined from the candidate content pool to generate preliminary target projection content;

[0039] Obtain the physical parameters of the current environment, including ambient brightness, available projection area, and viewing distance;

[0040] The preliminary target projection content is adjusted based on the physical parameters to obtain the target projection content.

[0041] By adopting the above technical solution, the query vector contains the intent category, sentiment polarity and intensity information, ensuring that the search results can fully match the user's current state. Similarity search in the preset dynamic knowledge graph can quickly locate content nodes related to user needs, improving the efficiency of content matching. The content node set is sorted according to the preset sorting criteria, taking into account multiple factors such as similarity with the query vector, historical playback feedback and current environment adaptability. This multi-dimensional sorting mechanism can more comprehensively evaluate the applicability of the content and improve the accuracy of the recommendation. The top-ranked content nodes are selected to build a candidate content pool, and multiple content elements are selected and combined according to the preset content combination rules. This process not only ensures the relevance of the content, but also increases the diversity and richness of the content. The physical parameters of the current environment are obtained and the target projection content is adjusted accordingly, so that the final presented content can better adapt to the actual projection environment, improving the content visibility and user experience. This full-process processing from semantic understanding to content generation realizes highly personalized and context-aware content recommendation, and can dynamically adjust the projection content according to the user's real-time status and environmental conditions, improving the adaptability and user satisfaction of the intelligent projection system.

[0042] In combination with some embodiments of the first aspect, in some embodiments, after determining the target projection content from a preset dynamic knowledge graph using an environment response generator according to the parsing result and controlling the projection device to play the target projection content, the method further includes:

[0043] Get the 3D point cloud model of the current environment;

[0044] Calculate the optimal projection path and projection surface based on the 3D point cloud model;

[0045] Control the projection device to play the target projection content according to the optimal projection path and projection surface.

[0046] By adopting the above technical solutions, the three-dimensional point cloud model can accurately describe the geometric characteristics of the environment, including key information such as surface shape and spatial distribution. Based on this model, the system can fully consider the complexity of the environment by calculating the optimal projection path and projection surface, such as the irregularity of the wall and the presence of obstacles. This optimization process based on the actual environment improves the presentation effect of the projected content in a complex environment. The calculation of the optimal projection path can avoid obstacles that may be encountered during the projection process, reduce the distortion or obstruction of the picture, and ensure the integrity and clarity of the projected content. The selection of the optimal projection surface can maximize the use of the available projection space and improve the visibility and attractiveness of the content. By controlling the projection device to play the content according to the calculated optimal path and projection surface, the system can achieve high-quality projection effects in various complex environments.

[0047] In combination with some embodiments of the first aspect, in some embodiments, after determining the target projection content from a preset dynamic knowledge graph using an environment response generator according to the parsing result and controlling the projection device to play the target projection content, the method further includes:

[0048] Use eye tracking technology to capture the user's gaze data in real time, including gaze points and focus areas;

[0049] Calculate the user's interest score for different content elements based on gaze data;

[0050] Mark the content element corresponding to the highest interest score as the key preference element;

[0051] Based on the key preferred element, similar elements whose similarity to the key preferred element is greater than a preset similarity are acquired, and the similar elements are added to the target projection content.

[0052] By adopting the above technical solution, eye tracking technology is used to capture the user's gaze data in real time, including the gaze point and the focus area. This non-invasive user behavior monitoring method can accurately capture the user's focus of interest without interfering with the user's normal viewing experience. The user's interest score for different content elements is calculated based on the gaze data, and the user's potential preferences are converted into quantifiable indicators, providing an objective basis for subsequent content optimization. The content element corresponding to the highest interest score is marked as the key preference element. This method can quickly identify the content features that attract the user's attention the most. Based on the key preference elements, similar elements with high similarity are obtained and added to the target projection content. This process realizes dynamic optimization and personalized recommendation of content.

[0053] In the second aspect, an embodiment of the present application provides an intelligent projection system based on gesture recognition, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0054] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a system, causes the system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a system, the system executes the method described in any possible implementation manner in the first aspect.

[0056] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0057] 1. The present application provides an intelligent projection method based on posture recognition. A distributed multimodal sensor network collects human behavior data and realizes comprehensive perception of human behavior. The use of multimodal sensors ensures the multidimensionality of the data and improves the accuracy of behavior recognition. The application of the self-organizing data cleaning algorithm effectively removes the noise in the original data and improves the effectiveness of subsequent processing. The multi-scale parsing process of the hierarchical semantic parser, including action recognition, intent inference, and sentiment analysis, enables the system to deeply understand the user's behavior and psychological state. This multi-level understanding provides a basis for subsequent content recommendations. The environmental response generator uses a preset dynamic knowledge graph to determine the target projection content, realizing personalized and context-aware content recommendations. This technical solution reduces the complexity of user operations, increases user selectivity, and improves adaptability to user preferences.

[0058] 2. The present application provides an intelligent projection method based on gesture recognition. The query vector contains the intent category, emotion polarity and intensity information, ensuring that the search results can fully match the current state of the user. By performing similarity search in the preset dynamic knowledge graph, the content nodes related to the user's needs can be quickly located, which improves the efficiency of content matching. The content node set is sorted according to the preset sorting criteria, taking into account multiple factors such as the similarity with the query vector, historical playback feedback and the adaptability of the current environment. This multi-dimensional sorting mechanism can more comprehensively evaluate the applicability of the content and improve the accuracy of the recommendation. The top-ranked content nodes are selected to build a candidate content pool, and multiple content elements are selected and combined according to the preset content combination rules. This process not only ensures the relevance of the content, but also increases the diversity and richness of the content. The physical parameters of the current environment are obtained and the target projection content is adjusted accordingly, so that the final presented content can better adapt to the actual projection environment, improving the visibility and user experience of the content. This full-process processing from semantic understanding to content generation realizes highly personalized and context-aware content recommendation, and can dynamically adjust the projection content according to the user's real-time status and environmental conditions, improving the adaptability and user satisfaction of the intelligent projection system.

[0059] 3. The present application provides an intelligent projection method based on gesture recognition, and the three-dimensional point cloud model can accurately describe the geometric characteristics of the environment, including key information such as surface shape and spatial distribution. Based on this model, the optimal projection path and projection surface are calculated, and the system can fully consider the complexity of the environment, such as the irregularity of the wall, the presence of obstacles and other factors. This optimization process based on the actual environment improves the presentation effect of the projected content in a complex environment. The calculation of the optimal projection path can avoid obstacles that may be encountered during the projection process, reduce the distortion or obstruction of the picture, and ensure the integrity and clarity of the projected content. The selection of the optimal projection surface can maximize the use of the available projection space and improve the visibility and attractiveness of the content. By controlling the projection device to play content according to the calculated optimal path and projection surface, the system can achieve high-quality projection effects in various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of an intelligent projection method based on gesture recognition in an embodiment of the present application.

[0061] Figure 2 It is another flow chart of an intelligent projection method based on gesture recognition in an embodiment of the present application.

[0062] Figure 3 It is a schematic diagram of the physical device structure of an intelligent projection system based on gesture recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0064] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0065] Combine the following Figure 1 , an intelligent projection method based on gesture recognition in an embodiment of the present application is described:

[0066] See also Figure 1 , which is a flow chart of an intelligent projection method based on gesture recognition in an embodiment of the present application.

[0067] S101, human behavior data collected through a distributed multimodal sensor network;

[0068] The system collects human behavior data through a distributed multimodal sensor network, which includes optical sensors, acoustic sensors and electromagnetic sensors.

[0069] In this step, the system deploys and utilizes a distributed multimodal sensor network to comprehensively collect human behavior data. The network includes optical sensors, acoustic sensors, and electromagnetic sensors. This multimodal design enables the system to capture the complexity of human behavior from different angles and dimensions. Optical sensors are mainly used to capture visual information, including human posture, movements, and facial expressions. Such sensors can be high-resolution cameras or depth cameras, which can provide two-dimensional or three-dimensional visual data. Acoustic sensors are responsible for collecting sound information, including voice content, intonation changes, and environmental sounds. This can be achieved through a high-sensitivity microphone array to capture all-round sound information. Electromagnetic sensors are used to detect human physiological signals, such as heart rate, breathing rate, etc., which can be achieved through wearable devices or non-contact sensors.

[0070] The system forms an all-round perception network by distributing these sensors in different locations in the environment. Wireless communication technology is used to transmit and synchronize data between sensors to ensure that the collected data is consistent in time. The system can also dynamically adjust the working mode of the sensor according to environmental changes, such as increasing the weight of the infrared sensor when the light is insufficient, and optimizing the collection strategy of the acoustic sensor in a noisy environment. This adaptive collection method ensures the quality and reliability of the data.

[0071] In addition, the system also realizes the real-time fusion of sensor data, and performs preliminary processing at the source of data through edge computing technology, reducing transmission delays and central processing pressure. The system adopts a distributed storage and computing architecture to ensure that basic data collection functions can be maintained even if some sensors fail.

[0072] S102, preprocessing the human behavior data using a self-organizing data cleaning algorithm to obtain denoised human behavior data;

[0073] The system uses a self-organizing data cleaning algorithm to pre-process the human behavior data to obtain denoised human behavior data. Specifically: the human behavior data from different sensors are synchronized using timestamp information to generate aligned multimodal data sequences;

[0074] Based on the multimodal data sequence, an adaptive noise model is constructed, and the adaptive noise model is used to dynamically adjust the filtering parameters;

[0075] Apply the adaptive noise model to human behavior data to obtain preliminary cleaned data;

[0076] Perform multi-scale wavelet decomposition on the preliminary cleaned data to extract human behavior characteristics in different frequency domains;

[0077] Generate denoised human behavior data based on human behavior characteristics.

[0078] Among them, based on the multimodal data sequence, an adaptive noise model is constructed, which specifically includes:

[0079] Perform spectrum analysis on historical human behavior data to identify the characteristic frequency ranges of different types of noise;

[0080] Set up a bandpass filter with adjustable parameters and use an online learning algorithm to adjust the filter parameters based on real-time data streams;

[0081] Determine the noise level evaluation index and obtain the filtering effect score of the bandpass filter;

[0082] Determine whether the filtering effect score is lower than a preset threshold;

[0083] If the filtering effect is lower than a preset threshold, a step of performing spectrum analysis on the historical human behavior data to identify characteristic frequency ranges of different types of noise is performed to reconstruct a bandpass filter;

[0084] If the filtering effect is not lower than the preset threshold, the step of applying the adaptive noise model to the human behavior data to obtain preliminary cleaned data is performed.

[0085] In this step, the system uses a self-organizing data cleaning algorithm to pre-process the collected human behavior data to obtain high-quality denoised data.

[0086] First, the system uses timestamp information to synchronize human behavior data from different sensors to generate aligned multimodal data sequences. This process involves complex time series analysis and interpolation techniques to ensure that data from different modalities are accurately aligned in the time dimension.

[0087] Next, the system builds an adaptive noise model based on the multimodal data sequence. The core of this model is a bandpass filter with adjustable parameters. The system uses an online learning algorithm to dynamically adjust the filter parameters according to the real-time data stream. Specifically, the system first performs spectral analysis on historical human behavior data to identify the characteristic frequency ranges of different types of noise.

[0088] Then, the system sets the initial bandpass filter parameters and determines the noise level evaluation index. The system monitors the filtering effect in real time. If the filtering effect score is lower than the preset threshold, the system will re-perform spectrum analysis and reconstruct the bandpass filter. This process is iterative to ensure that the filter is always in the best state. When the filtering effect meets the requirements, the system applies the adaptive noise model to the human behavior data to obtain preliminary cleaned data. In order to further improve the data quality, the system performs multi-scale wavelet decomposition on the preliminary cleaned data, which allows the system to extract human behavior features in different frequency domains. In this way, the system can separate meaningful signal components and residual noise.

[0089] Finally, the system reconstructs the signal based on the extracted human behavior features to generate the final denoised human behavior data. This multi-level, adaptive data cleaning method can effectively handle complex noise types, including environmental noise, sensor noise, and human interference, greatly improving the accuracy and reliability of subsequent analysis.

[0090] S103, inputting the denoised human behavior data into a dynamic spatiotemporal graph network to extract human behavior feature data;

[0091] The system inputs the denoised human behavior data into the dynamic spatiotemporal graph network to extract human behavior feature data. In this step, the system inputs the cleaned denoised human behavior data into a specially designed dynamic spatiotemporal graph network to extract higher-level human behavior feature data. The dynamic spatiotemporal graph network is an advanced deep learning model that is specifically designed to process complex data that combines temporal and spatial information. The core idea of ​​the network is to view human behavior as a dynamically evolving graph structure, where nodes represent key points of the human body (such as joints, head, etc.), and edges represent the relationships and interactions between these key points. The network first encodes the spatial information of each time frame through a multi-layer convolutional neural network to capture the static characteristics of human posture. Then, the network uses a graph convolution layer to model the spatial relationship between key points, which allows the system to capture complex body structure information. In order to process temporal information, the system uses a recurrent neural network (such as a long short-term memory network LSTM or a gated recurrent unit GRU) to model the dependencies between consecutive frames. This spatiotemporal approach enables the system to simultaneously consider the spatial composition and temporal evolution of actions. In addition, the system introduces an attention mechanism that enables the network to automatically focus on the most relevant body parts and time periods, improving the efficiency and accuracy of feature extraction. In order to handle movements of different scales, the system adds a multi-scale feature fusion module to the network, which can capture both tiny local movements and large-scale whole-body movements. The system also uses techniques such as residual connections and dense connections to deepen the network while maintaining good gradient flow, which helps to extract more abstract and semantic features. Ultimately, through this complex network architecture, the system converts the original multimodal time series data into highly abstract feature vectors that contain both the spatial structural information of human body movements and encode the temporal evolution patterns of movements, laying a solid foundation for subsequent behavioral understanding and semantic parsing.

[0092] S104, based on the human behavior feature data, using a hierarchical semantic parser to perform multi-scale parsing on the human behavior to obtain a parsing result;

[0093] Based on human behavior feature data, the system uses a hierarchical semantic parser to perform multi-scale analysis of human behavior and obtains the analysis results. Specifically, a convolutional neural network is used to extract low-level action features, which include limb positions and motion trajectories.

[0094] Based on low-level action features, recurrent neural networks are used to model temporal dependencies and identify complex action sequences;

[0095] The intention is predicted based on the complex action sequence combined with the conditional random field model;

[0096] Based on the facial expressions, voice features and physiological signals in the human behavior feature data, a multimodal fusion network is used to analyze the emotional state and obtain the predicted emotion.

[0097] Integrate predicted intent and predicted sentiment to generate parsing results.

[0098] In this step, the system uses a hierarchical semantic parser to perform multi-scale analysis on the extracted human behavior feature data to obtain in-depth behavior understanding results. The hierarchical semantic parser adopts a bottom-up analysis strategy, gradually extracting low-level action features to high-level intention and emotion inference. First, the system uses a specially designed convolutional neural network to extract low-level action features, which mainly include limb position and motion trajectory information. The multi-layer structure of the convolutional network can effectively capture local spatial features, such as hand movements, footsteps, etc. Next, the system uses recurrent neural networks (such as bidirectional LSTM) to model temporal dependencies and identify complex action sequences. This step can connect discrete action frames into a coherent action stream and capture the contextual relationship and long-term dependency of the action. The system also introduces an attention mechanism that enables the network to automatically focus on the most relevant time periods and body parts, improving the efficiency and accuracy of sequence modeling. On this basis, the system combines the conditional random field model to further analyze the action sequence and obtain preliminary intention predictions. The conditional random field model can consider the transition probability and contextual information between actions, which helps to eliminate local ambiguity and improve the accuracy of intention recognition. At the same time, the system uses a multimodal fusion network to conduct a comprehensive analysis of facial expressions, voice features, and physiological signals in human behavior feature data to achieve accurate recognition of emotional states. This multimodal fusion network uses attention mechanisms and gating mechanisms to dynamically adjust the weights of different modal information to adapt to the characteristics of emotional expression in different scenarios. Finally, the system integrates the predicted intentions and emotional information to generate the final parsing results. This integration process is not just a simple splicing, but through an advanced semantic fusion module, it considers the mutual influence and consistency between intentions and emotions to generate more coherent and human-cognition-compliant behavioral understanding results.

[0099] S105. According to the analysis result, the target projection content is determined from the preset dynamic knowledge graph using the environment response generator, and the projection device is controlled to play the target projection content;

[0100] Based on the analysis results, the system uses an environmental response generator to determine the target projection content from a preset dynamic knowledge graph, and controls the projection device to play the target projection content. How the system uses an environmental response generator to determine the target projection content from a preset dynamic knowledge graph based on the analysis results will be described below in conjunction with another embodiment and will not be repeated here.

[0101] The following steps S106-S108 and steps S109-S112 are two solutions that do not interfere with each other and can be executed simultaneously or only one of them can be executed in no particular order.

[0102] S106, obtaining a three-dimensional point cloud model of the current environment;

[0103] In this step, the system uses advanced 3D scanning technology to obtain an accurate 3D point cloud model of the current environment. This process involves the organic combination of multiple links. First, the system starts the depth camera or lidar device arranged in the environment, which can scan the environment at high speed and capture a large amount of spatial point data. The system adopts a time-division multiplexing strategy to enable multiple scanning devices to work in different time periods to avoid mutual interference. At the same time, the system uses an inertial measurement unit (IMU) to assist in positioning and improve scanning accuracy. During the data acquisition process, the system performs real-time data preprocessing, including denoising, outlier removal, and downsampling, to reduce the computational burden of subsequent processing. Next, the system uses a simultaneous localization and mapping (SLAM) algorithm to align and fuse point cloud data from different perspectives. The SLAM algorithm accurately calculates the relative position of each scan through feature matching and pose estimation, thereby unifying all point cloud data into the same coordinate system. In order to improve the accuracy of registration, the system also introduces a loop detection mechanism to identify and correct accumulated errors. During the point cloud fusion process, the system uses a voxelization method to average the points in the overlapping area, which not only retains the detailed features of the environment, but also effectively reduces data redundancy. Finally, the system performs post-processing on the fused point cloud, including surface reconstruction and texture mapping. Surface reconstruction uses algorithms such as Poisson reconstruction or moving least squares to convert discrete point clouds into continuous surface models. Texture mapping uses synchronously collected high-definition image data to give the 3D model a realistic visual appearance.

[0104] S107, calculating the optimal projection path and projection surface based on the three-dimensional point cloud model;

[0105] In this step, the system uses the acquired high-precision 3D point cloud model to calculate the optimal projection path and projection surface through complex algorithms. First, the system segments and classifies the point cloud model to identify possible projection surfaces, such as walls, ceilings, and desktops. This process involves plane detection algorithms, such as the random sampling consensus (RANSAC) algorithm, and semantic segmentation technology based on machine learning. The system also considers the material properties of the surface and evaluates the projection width of each surface by analyzing the reflection characteristics. Next, the system generates multiple candidate projection paths. Each path is the trajectory of the projection light in space, and factors such as the position, direction, and projection angle of the projection device need to be considered. The system uses a ray tracing algorithm to simulate light propagation and calculate the obstacles and occlusions that may be encountered on each path. At the same time, the system also needs to evaluate the visibility of each projection surface, considering the position and viewing angle of the audience to ensure that the projected content can be viewed clearly. In order to optimize the projection effect, the system introduces a multi-objective optimization algorithm, which takes into account multiple factors at the same time, such as maximizing the projection area, minimizing the distortion of the picture, and optimizing the viewing comfort. This optimization process uses an evolutionary algorithm or a simulated annealing algorithm to find the global optimal solution through iterative search. During the calculation process, the system also needs to consider the hardware limitations of the projection equipment, such as projection distance, brightness attenuation, etc., to ensure that the generated solution is technically feasible. In addition, the system also implements an adaptive adjustment function, which can update the optimal projection solution in real time according to dynamic changes in the environment (such as personnel movement, furniture rearrangement). Finally, the system converts the calculated optimal projection path and projection surface information into specific control instructions, including the precise positioning parameters of the projection equipment, the projection angle adjustment value, and the image pre-distortion compensation parameters, etc., to prepare for the next step of projection control.

[0106] S108, controlling the projection device to play the target projection content according to the optimal projection path and projection surface;

[0107] In this step, the system precisely controls the projection device to ensure that the target projection content is played according to the calculated optimal projection path and projection surface. First, the system converts the optimal projection path into motion control instructions for the projection device. This involves complex motion planning algorithms that need to take into account the mechanical characteristics of the projection device, such as rotation speed, acceleration limits, etc., to ensure smooth and precise motion. The system uses methods such as quintic polynomial interpolation or Bezier curves to generate continuous motion trajectories to avoid image jitter caused by sudden changes. For projection devices with a pan-tilt structure, the system also needs to coordinate the movement of multiple motors to achieve precise spatial positioning. At the same time, the system performs real-time geometric correction and deformation processing on the projection content based on the geometric characteristics of the optimal projection surface. This includes techniques such as trapezoidal correction and surface mapping to ensure that the projection content can maintain the correct shape and proportion on irregular surfaces. The system uses advanced image processing algorithms, such as polygonal mesh deformation or physically based deformation models, to achieve accurate mapping of complex surfaces. In order to compensate for the influence of ambient light, the system also adjusts the brightness and contrast of the projection content in real time, and uses techniques such as adaptive gamma correction to ensure that the image is clearly visible under different lighting conditions. During playback, the system continuously monitors environmental changes. If a moving object is detected, the system can adjust the projection parameters in real time to avoid content being blocked or distorted. This requires fast image analysis and decision-making algorithms, such as optical flow estimation or target tracking technology. In addition, the system also realizes the coordinated control of multiple projection devices, and achieves a large-scale, seamless projection effect through precise time synchronization and content distribution. Finally, the system continuously optimizes the projection effect through a feedback mechanism, uses a camera to capture the actual projection effect, and compares it with the expected effect. The projection parameters are continuously adjusted through closed-loop control to ensure long-term stable high-quality projection.

[0108] S109, using eye tracking technology to capture user gaze data in real time;

[0109] The system uses eye tracking technology to capture the user's gaze data in real time, including the gaze point and the area of ​​interest. In this step, the system deploys advanced eye tracking technology to capture the user's gaze data in real time, including the precise gaze point position and the range of the area of ​​interest. The system uses a high-frame rate, high-resolution infrared camera, combined with precise eye center and corneal reflection detection algorithms to achieve millimeter-level eye tracking accuracy. In order to adapt to different usage scenarios, the system supports both head-mounted and remote eye tracking solutions. The head-mounted device uses a micro camera and inertial sensors to continuously track eye movements while the user is free to move; the remote device uses multiple cameras arranged in the environment to achieve non-contact eye capture. The system uses advanced computer vision algorithms, such as convolutional neural networks, to process eye images in real time and extract pupil position and eye movement characteristics. At the same time, the system also takes into account the impact of head movement, and achieves accurate positioning of the gaze point by synchronously tracking the head position and posture. In order to improve the robustness of tracking, the system uses multimodal fusion technology, combines eye movement data and head movement data, and uses algorithms such as Kalman filtering for data fusion to effectively reduce the impact of noise and outliers. The system also implements an adaptive calibration function, which can dynamically adjust the parameters of the eye movement model according to individual differences of users and environmental changes to ensure accuracy during long-term use. In terms of data processing, the system uses edge computing technology to perform preliminary processing at the data acquisition end to reduce transmission delays and central processing pressure. The system also establishes a user gaze behavior model, which identifies meaningful gaze behaviors and filters out unconscious eye movements by analyzing features such as gaze duration and scanning paths. Finally, the system transmits the processed gaze data to the central processing unit in real time, providing high-quality input data for subsequent interest analysis.

[0110] S110, calculating the user's interest scores for different content elements based on the gaze data;

[0111] In this step, the system uses advanced data analysis technology to accurately calculate the user's interest score for different elements in the projected content based on the captured user gaze data. First, the system divides the projected content into multiple semantic regions, each corresponding to a content element, such as pictures, text, videos, etc. The system uses computer vision techniques, such as regional convolutional neural networks (R-CNN) or semantic segmentation algorithms, to automatically identify and mark these content elements. Next, the system spatially aligns the user's gaze data with these semantic regions and establishes a mapping relationship between the gaze point and the content element. On this basis, the system uses a multi-dimensional interest evaluation model to comprehensively consider multiple factors to calculate the interest score. These factors include gaze duration, gaze frequency, gaze sequence, pupil dilation, etc. The system uses machine learning algorithms, such as random forests or support vector machines, to train a model that can convert these features into interest scores. In order to improve the accuracy of the evaluation, the system also considers the characteristics of the content elements themselves, such as visual salience, information density, etc., and uses the attention mechanism to weight different features. The system also implements a time series analysis function, which captures the dynamic characteristics of user interests over time through recurrent neural network (RNN) or long short-term memory network (LSTM) models. In addition, the system establishes a personalized user interest model, which adapts to the unique preferences of each user through continuous learning and updating. During the real-time calculation process, the system adopts an incremental learning strategy, which can quickly adjust the model parameters to respond to the immediate changes in user interests. In order to handle the data of a large number of concurrent users, the system adopts a distributed computing architecture to ensure efficient parallel processing capabilities. Finally, the system standardizes the calculated interest scores to generate a unified scoring standard to facilitate comparison and ranking between different content elements.

[0112] S111, marking the content element corresponding to the highest interest score as a key preference element;

[0113] In this step, the system conducts in-depth analysis and processing of the calculated interest scores to accurately identify and mark the user's key preference elements. First, the system normalizes the interest scores of all content elements to ensure that the scores of content elements of different types and scales are comparable. The system uses techniques such as Z-score normalization or Min-Max scaling to map the scores to a unified interval. Next, the system uses statistical analysis methods such as box plot analysis or density clustering to identify outliers and high-scoring groups in the score distribution. The system also considers the time series characteristics of the scores and uses techniques such as moving average or exponential smoothing to eliminate the impact of short-term fluctuations and highlight long-term stable high-scoring factors. In order to improve the robustness of the marking, the system implements a multi-round cross-validation mechanism to ensure that the marked elements have a sustained high interest through multiple independent interest evaluations and comparisons. The system also introduces a context-aware mechanism to consider the correlation between content elements and avoid judging the importance of individual elements in isolation. During the marking process, the system adopts a dynamic threshold strategy to adaptively adjust the marking criteria according to the overall interest distribution of the current content to ensure that key preference elements can be reasonably identified in different scenarios. In addition, the system implements multi-level preference tagging, which not only marks the highest-scoring elements, but also assigns different preference levels to other high-scoring elements, providing a more detailed reference for subsequent content recommendations. The system also establishes a historical record of preference elements, and through time series analysis, identifies long-term trends and cyclical changes in user interests. Finally, the system cross-validates the tagging results with the user's explicit feedback (such as ratings and likes) to further improve the accuracy and reliability of the tagging. Through this series of complex processing and analysis, the system is able to highly accurately identify and tag content elements that truly reflect the user's deep preferences.

[0114] S112: based on the key preferred element, obtaining similar elements whose similarity to the key preferred element is greater than a preset similarity and adding the similar elements to the target projection content.

[0115] In this step, the system intelligently identifies and selects similar elements with high similarity based on the marked key preferred elements, and seamlessly integrates them into the target projection content.

[0116] In the above embodiment, the distributed multimodal sensor network collects human behavior data and realizes comprehensive perception of human behavior. The use of multimodal sensors ensures the multidimensionality of the data and improves the accuracy of behavior recognition. The application of the self-organizing data cleaning algorithm effectively removes the noise in the original data and improves the effectiveness of subsequent processing. The multi-scale parsing process of the hierarchical semantic parser, including action recognition, intent inference and sentiment analysis, enables the system to deeply understand the user's behavior and psychological state. This multi-level understanding provides a basis for subsequent content recommendations. The environmental response generator uses a preset dynamic knowledge graph to determine the target projection content, realizing personalized and context-aware content recommendations. This technical solution reduces the complexity of user operations, increases user selectivity, and improves adaptability to user preferences.

[0117] In step S105 of the above embodiment, the system determines the target projection content from the preset dynamic knowledge graph using the environment response generator according to the analysis result. The following is a detailed description of how this step is implemented in conjunction with another embodiment. Figure 2 , another intelligent projection method based on gesture recognition in an embodiment of the present application is described:

[0118] See also Figure 2 , is another flow chart of an intelligent projection method based on gesture recognition in an embodiment of the present application.

[0119] S201, converting the predicted intent and predicted sentiment into a query vector;

[0120] The system converts the predicted intent and predicted sentiment into a query vector, which contains the intent category, sentiment polarity and intensity information.

[0121] In this step, the system first receives the predicted intent and predicted emotion as input. The predicted intent is usually the purpose or need implied by the user's current behavior or posture, such as wanting to relax, needing to work, etc. The predicted emotion is an estimate of the user's current emotional state, such as pleasure, anxiety, etc. The system uses natural language processing technology to convert these textual intention and emotion information into numerical vector representations. Specifically, the system uses pre-trained word embedding models (such as Word2Vec or GloVe) to convert intent and emotion words into high-dimensional vectors. Subsequently, the system applies dimensionality reduction techniques (such as principal component analysis PCA or t-SNE) to compress the high-dimensional vectors to lower dimensions to improve the efficiency of subsequent processing. In the vectorization process, the system also introduces intent category encoding, emotion polarity labels, and emotion intensity values. The intent category encoding uses one-hot encoding to map different categories of intent (such as work, entertainment, learning, etc.) to different binary bits. The emotion polarity label is represented by three values: -1 (negative), 0 (neutral), and 1 (positive). The emotional intensity is represented by a continuous value between 0 and 1, where 0 represents the weakest emotional intensity and 1 represents the strongest emotional intensity. The system concatenates these codes and numerical values ​​with the word vector to form the final query vector. This multi-dimensional vector representation enables the system to capture the user's intentions and emotional states more comprehensively and accurately, providing a basis for subsequent content matching and generation. Through this vectorization process, the system realizes the conversion of abstract intentions and emotional concepts into computable and comparable mathematical representations, laying the foundation for the next step of similarity search in the knowledge graph.

[0122] S202, using the query vector to perform similarity search in a preset dynamic knowledge graph to obtain a content node set;

[0123] In this step, the system first loads the preset dynamic knowledge graph. The knowledge graph is a complex network structure that contains a large number of content nodes and the relationships between nodes. Each content node represents a possible projected content, including text, images, videos, or interactive elements. The relationship between nodes reflects the semantic connection, topic relevance, or functional association of the content. The system uses a graph database (such as Neo4j or JanusGraph) to store and manage this knowledge graph to support efficient query and update operations. When performing similarity search, the system uses the vector space model (Vector SpaceModel) as the basic framework. Specifically, the system first represents each content node in the knowledge graph as a vector, which encodes the characteristics of the node, including content type, topic label, sentiment tendency, and other information. Subsequently, the system calculates the cosine similarity between the query vector and each content node vector. In order to improve search efficiency, the system uses an approximate nearest neighbor search algorithm (such as Locality-Sensitive Hashing or Annoy), which allows the most similar nodes to the query vector to be quickly found in large-scale graphs. The system also considers the relationship between nodes in the knowledge graph, using graph embedding technology (such as Node2Vec or TransE) to capture the location information of the nodes in the graph structure, and integrates this information into the similarity calculation. This method not only considers the characteristics of the nodes themselves, but also the semantic associations of the nodes in the entire knowledge network, so that richer and more relevant content can be discovered. The system sets a similarity threshold and collects all content nodes whose similarity exceeds the threshold to form a content node set. This set contains the potential projection content that is most relevant to the user's current intention and emotional state, providing a candidate pool for the next step of sorting and screening.

[0124] S203, sorting the content node set by relevance according to a preset sorting standard;

[0125] The system sorts the relevance of the content node set according to preset sorting criteria, which include similarity with the query vector, historical playback feedback, and adaptability to the current environment.

[0126] In this step, the system performs a refined sorting of the set of content nodes obtained in the previous step to determine the projection content that best suits the current situation. The system uses a multi-factor weighted sorting algorithm that comprehensively considers sorting criteria in multiple dimensions. First, the system calculates the similarity score between each content node and the query vector, which reflects the degree of match between the content and the user's current intention and emotion. The system uses metrics such as cosine similarity or Euclidean distance to quantify this degree of match. Second, the system considers historical playback feedback, which includes data such as the user's past viewing time, interaction frequency, and explicit ratings of similar content. The system uses a collaborative filtering algorithm to predict the current user's potential preference for each content node based on the user's historical behavior patterns. Third, the system evaluates the adaptability of each content node to the current environment. This involves analyzing whether the visual features of the content (such as brightness, color, complexity) are suitable for the current environmental conditions (such as indoor light, projection surface, etc.). The system uses computer vision techniques, such as image analysis and scene understanding algorithms, to evaluate the degree of match between content and the environment. In addition, the system also considers the timeliness, diversity, and novelty of the content. Timeliness is evaluated by checking the creation or update time of the content; diversity is ensured by comparing the differences between candidate content; novelty is achieved by analyzing the user's content consumption history and giving priority to content that the user has not been exposed to. The system assigns weights to each sorting criterion, and these weights can be dynamically adjusted based on specific application scenarios or user preferences. Using the weighted sum method, the system calculates a comprehensive score for each content node. Finally, the system sorts the set of content nodes in descending order based on this comprehensive score to obtain an ordered list of content. This multi-dimensional sorting method ensures that the system can select content that best meets the user's current needs and is most suitable for the current environment, while also taking into account the user's long-term preferences.

[0127] S204, selecting the first several content nodes in the relevance ranking to construct a candidate content pool;

[0128] In this step, the system selects the first N nodes from the sorted content node list in the previous step to build a streamlined candidate content pool. The value of N is an adjustable parameter, which is set by the system according to actual application requirements and computing resource constraints. Usually, the value of N ranges from 10 to 50, which can ensure content diversity while not increasing the complexity of subsequent processing due to too many candidate contents. In the selection process, the system not only considers the sorting position of the node, but also applies a dynamic threshold mechanism. Specifically, the system analyzes the distribution of the sorting score and sets an adaptive truncation threshold. Only nodes with scores higher than the threshold are included in the candidate pool, which ensures that the quality of the candidate content will not decrease due to simply selecting a fixed number of nodes. In the process of building the candidate content pool, the system also introduces a content type balancing mechanism. The system classifies content into different types (such as text, images, videos, interactive content, etc.) and sets a minimum and maximum selection ratio for each type. This ensures that the candidate pool contains multiple types of content, increasing the richness and flexibility of the final generated content. In addition, the system implements a dynamic update mechanism that allows new nodes to be added to the candidate pool from the original sorted list as needed during subsequent processing. This mechanism improves the robustness of the system, enabling it to cope with situations where insufficient or unsuitable content may occur. The system also attaches metadata to each content node in the candidate pool, including its position in the original ranking, score, content type, key features, and other information. These metadata will play an important role in the subsequent content combination and adjustment process, helping the system make smarter decisions. Through this step, the system successfully streamlines a large amount of potential content into a candidate set, laying the foundation for the next step of content combination.

[0129] S205, selecting and combining multiple content elements from the candidate content pool according to preset content combination rules to generate preliminary target projection content;

[0130] In this step, the system uses preset content combination rules to intelligently select and combine multiple content elements from the candidate content pool to generate a preliminary target projection content. Content combination rules are a set of predefined guidelines for determining how to select and arrange different types of content elements to create a coherent and meaningful projection content. These rules take into account multiple factors, including the semantic relevance of the content, the visual and auditory coordination, the density and complexity of the information, and the logical flow of the content. The system first applies a topic clustering algorithm, such as K-means or hierarchical clustering, to group the elements in the candidate content pool to ensure that the selected content elements are thematically consistent. Subsequently, the system uses a sequence model, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), to learn and predict the optimal ordering of content elements. This model is trained to understand the transition patterns between different types of content elements to ensure that the generated content has a natural narrative flow. During the combination process, the system also considers the temporal dynamics of the content. For projections that need to last for a period of time, the system designs a timeline planning algorithm to assign different content elements to different time periods to ensure that the overall content remains attractive and coherent over time. The system also implements an adaptive content density control mechanism to dynamically adjust the speed and complexity of information presentation according to the user's attention state and environmental conditions. In order to enhance the interactivity and attractiveness of the content, the system integrates an interactive element generator. This generator can dynamically create simple questions and answers, quizzes or animation effects based on the selected content theme to increase user engagement. At the final stage of the combination process, the system applies a content consistency check algorithm to ensure that all selected elements are consistent in visual style, color scheme and overall atmosphere. If inconsistency is detected, the system will automatically adjust or replace the corresponding content element.

[0131] S206 , obtaining physical parameters of the current environment, and adjusting the preliminary target projection content based on the physical parameters to obtain the target projection content.

[0132] The system obtains physical parameters of the current environment, including ambient brightness, available projection area, and viewing distance, and adjusts the preliminary target projection content based on the physical parameters to obtain the target projection content.

[0133] In this step, the physical parameters of the current environment are first obtained through various sensors and data acquisition devices. These parameters mainly include ambient brightness, available projection area and viewing distance. The ambient brightness is measured by a light sensor, the available projection area is obtained by scanning the environment with a depth camera or laser rangefinder, and the viewing distance can be estimated by an infrared sensor or computer vision technology. The system inputs these raw data into an environmental parameter processing module, which processes the raw data using data cleaning and standardization techniques, removes outliers and converts the data into a standardized format. Subsequently, the system starts the content adjustment engine, which performs a series of optimizations and adjustments on the preliminary target projection content based on the processed environmental parameters. For ambient brightness, the system uses image processing algorithms to dynamically adjust the brightness, contrast and color saturation of the projected content to ensure that the content is clearly visible under different lighting conditions. The system also implements an adaptive HDR (high dynamic range) algorithm that can maintain image details in high-brightness environments. For the available projection area, the system uses a responsive layout algorithm to dynamically adjust the layout and proportion of the content according to the available space. This includes adjusting text size, rearranging image positions, and cropping or scaling certain elements when necessary. The system also integrates an intelligent content prioritization mechanism to display the most important information first when the projection area is limited. For viewing distance, the system uses a visual perception model to estimate the optimal font size and image detail level. The system also implements a dynamic focus adjustment algorithm that can automatically adjust the focus of the projection according to the viewing distance to ensure that the image is always clear. In addition, the system also takes into account the material and texture of the projection surface.

[0134] In the above embodiment, the query vector contains the intent category, sentiment polarity and intensity information, ensuring that the search results can fully match the current state of the user. Performing similarity search in the preset dynamic knowledge graph can quickly locate content nodes related to user needs, thereby improving the efficiency of content matching. The content node set is sorted according to the preset sorting criteria, taking into account multiple factors such as similarity with the query vector, historical playback feedback and current environment adaptability. This multi-dimensional sorting mechanism can more comprehensively evaluate the applicability of the content and improve the accuracy of the recommendation. The top-ranked content nodes are selected to build a candidate content pool, and multiple content elements are selected and combined according to the preset content combination rules. This process not only ensures the relevance of the content, but also increases the diversity and richness of the content. The physical parameters of the current environment are obtained and the target projection content is adjusted accordingly, so that the final presented content can better adapt to the actual projection environment, improving the visibility of the content and user experience. This full-process processing from semantic understanding to content generation realizes highly personalized and context-aware content recommendation, and can dynamically adjust the projection content according to the user's real-time status and environmental conditions, thereby improving the adaptability and user satisfaction of the intelligent projection system.

[0135] The following describes the system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of the physical device structure of an intelligent projection system based on gesture recognition provided in an embodiment of the present application.

[0136] It should be noted that Figure 3 The structure of the system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0137] like Figure 3 As shown, the system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0138] The following components are connected to the I / O interface 305: an input section 306 including a camera, an infrared sensor, etc.; an output section 307 including a liquid crystal display (LCD) and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0139] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0140] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the system described in the above embodiment; or may exist independently without being assembled into the system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of a system, the system implements the method provided in the above embodiment.

[0143] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0144] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0145] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0146] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. An intelligent projection method based on gesture recognition, characterized in that: include: Human behavior data collected by a distributed multimodal sensor network, wherein the distributed multimodal sensor network includes optical sensors, acoustic sensors, and electromagnetic sensors; Preprocessing the human behavior data using a self-organizing data cleaning algorithm to obtain denoised human behavior data; Inputting the denoised human behavior data into a dynamic spatiotemporal graph network to extract human behavior feature data; Based on the human behavior feature data, a hierarchical semantic parser is used to perform multi-scale parsing of human behavior to obtain parsing results. The parsing process includes action recognition, intention inference and sentiment analysis. The parsing results include predicted intention and predicted sentiment, specifically including: Using a convolutional neural network to extract low-level motion features, the low-level motion features include limb positions and motion trajectories; Based on the low-level action features, a recurrent neural network is used to model temporal dependencies and identify complex action sequences; Obtaining a predicted intention based on the complex action sequence combined with a conditional random field model; Using a multimodal fusion network to analyze the emotional state according to the facial expressions, voice features and physiological signals in the human behavior feature data to obtain predicted emotions; Integrating the predicted intention and the predicted emotion to generate the analysis result; According to the analysis result, the target projection content is determined from the preset dynamic knowledge graph using the environment response generator, and the projection device is controlled to play the target projection content, specifically including: Converting the predicted intent and the predicted emotion into a query vector, wherein the query vector includes intent category, emotion polarity and intensity information; Using the query vector to perform similarity search in a preset dynamic knowledge graph to obtain a content node set; Sorting the content node set according to a preset sorting standard, wherein the preset sorting standard includes similarity with the query vector, historical playback feedback, and current environment adaptability; Selecting the first several content nodes in the relevance ranking to construct a candidate content pool; According to preset content combination rules, multiple content elements are selected and combined from the candidate content pool to generate preliminary target projection content; Acquire physical parameters of the current environment, including ambient brightness, available projection area, and viewing distance; The preliminary target projection content is adjusted based on the physical parameters to obtain the target projection content.

2. The method according to claim 1, characterized in that The self-organizing data cleaning algorithm is used to pre-process the human behavior data to obtain denoised human behavior data, specifically including: Use timestamp information to synchronize human behavior data from different sensors and generate corresponding multimodal data sequences; Based on the multimodal data sequence, construct an adaptive noise model, wherein the adaptive noise model is used to dynamically adjust filtering parameters; Applying the adaptive noise model to the human behavior data to obtain preliminary cleaned data; Performing multi-scale wavelet decomposition on the preliminary cleaned data to extract human behavior characteristics in different frequency domains; De-noised human behavior data is generated according to the human behavior characteristics.

3. The method according to claim 2, characterized in that The step of constructing an adaptive noise model based on the multimodal data sequence specifically includes: Perform spectrum analysis on historical human behavior data to identify the characteristic frequency ranges of different types of noise; Set up a bandpass filter with adjustable parameters and use an online learning algorithm to adjust the filter parameters based on real-time data streams; Determine a noise level evaluation index and obtain a filtering effect score of the bandpass filter; Determine whether the filtering effect score is lower than a preset threshold; If the filtering effect is lower than the preset threshold, performing the step of performing spectrum analysis on the historical human behavior data to identify characteristic frequency ranges of different types of noise to reconstruct a bandpass filter; If the filtering effect is not lower than the preset threshold, the step of applying the adaptive noise model to the human behavior data to obtain preliminary cleaned data is performed.

4. The method according to claim 1, characterized in that: After determining the target projection content from the preset dynamic knowledge graph using the environment response generator according to the analysis result, and controlling the projection device to play the target projection content, the method further includes: Get the 3D point cloud model of the current environment; Calculate the optimal projection path and projection surface based on the three-dimensional point cloud model; The projection device is controlled to play the target projection content according to the optimal projection path and the projection surface.

5. The method according to claim 1, characterized in that After determining the target projection content from the preset dynamic knowledge graph using the environment response generator according to the analysis result, and controlling the projection device to play the target projection content, the method further includes: Using eye tracking technology to capture user's gaze data in real time, the gaze data including gaze point and focus area; Calculating the user's interest scores for different content elements based on the gaze data; Mark the content element corresponding to the highest interest score as the key preference element; Based on the key favorite element, a similar element having a similarity with the key favorite element greater than a preset similarity is acquired, and the similar element is added to the target projection content.

6. An intelligent projection system based on gesture recognition, characterized in that: The system comprises: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the system to execute the method as described in any one of claims 1-5.

7. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a system, the system is caused to execute the method according to any one of claims 1 to 5.

8. A computer program product, characterized in that When the computer program product is run on a system, the system is caused to execute the method according to any one of claims 1 to 5.

Images