An ai-supported multi-modal tabletop interaction projection method and system

Abstract

The application discloses an AI-supported multi-modal desktop interaction projection method and system, comprising the following steps: synchronously collecting visual signals, tactile data and voice input in a desktop projection environment, obtaining a multi-modal information set after data fusion; according to the multi-modal information set, calculating the priority weight distribution of each mode to obtain a high-priority mode sequence; if the visual signal in the high-priority mode sequence is determined as the dominant, extracting a visual dominant feature group; cross-verification is performed on the visual dominant feature group and the tactile data to judge the information consistency after data fusion; according to the information consistency result, dynamically adjusting the transparency control parameter of the projection layer to obtain an optimized hierarchical configuration; according to the optimized hierarchical configuration, updating the projection display content to output the final projection content. The application improves the interaction real-time performance and immersion of the desktop projection, and finally achieves the technical effects of optimizing the user experience and improving the system intelligence.

Description

Technical Field

[0001] This invention belongs to the field of desktop interactive projection technology, and in particular relates to an AI-supported multimodal desktop interactive projection method and system. Background Technology

[0002] In the field of modern intelligent interaction, desktop projection technology, as a crucial bridge connecting the virtual and the real world, demonstrates irreplaceable value. It not only enhances the immersion of users in work, study, and entertainment but also enables seamless communication between people and devices in an intuitive way. However, research and application in this field are facing unprecedented challenges, urgently requiring innovative breakthroughs to drive technological progress.

[0003] Currently, although desktop projection interaction technology has made some progress, most methods fall short when dealing with complex scenarios. Especially when integrating information from multiple senses, existing solutions often struggle to balance the presentation and response of different types of information, easily leading to user confusion or information overload during interaction. This problem cannot be solved by simply piling on technology; rather, it stems from a lack of ability to dynamically coordinate multi-source information. Particularly in multi-user or multi-tasking environments, the system often fails to accurately determine user needs, resulting in a significantly diminished interactive experience.

[0004] To address the aforementioned problems in existing technologies, there is an urgent need to propose an AI-supported multimodal desktop interactive projection method. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an AI-supported multimodal desktop interactive projection method, comprising the following steps:

[0006] Simultaneously collect visual signals, tactile data, and voice input in the desktop projection environment, and obtain a multimodal information set after data fusion;

[0007] Based on the multimodal information set, the priority weight distribution of each mode is calculated to obtain the high-priority mode sequence;

[0008] If the visual signal is determined to be dominant in the high-priority modality sequence, then the dominant visual feature group is extracted;

[0009] Cross-validation is performed on the visual dominant feature group and tactile data to determine the consistency of information after data fusion;

[0010] Based on the information consistency results, the transparency control parameters of the projection layers are dynamically adjusted to obtain an optimized layer configuration;

[0011] Based on the optimized hierarchical configuration, update the projected display content, provide real-time interactive feedback, and output the final projected content.

[0012] Optionally, the process of simultaneously acquiring visual signals, tactile data, and voice input in the desktop projection environment, and then fusing the data to obtain a multimodal information set includes:

[0013] Visual signals, tactile data, and voice input are collected using cameras, sensors, and microphones to obtain an initial multimodal dataset;

[0014] The initial multimodal dataset is subjected to timestamp alignment, missing value imputation, unit unification, and synchronization correction to form a synchronous multimodal data group;

[0015] A classification model is used to identify the correlation between synchronous multimodal data groups, extract key features, and generate a multimodal information set after data fusion.

[0016] Optionally, the process of calculating the priority weight distribution of each mode based on the multimodal information set to obtain the high-priority mode sequence includes:

[0017] Extract the original features of each modality from the multimodal information set and classify and organize them;

[0018] Based on the categorized and organized modality dataset, the importance of each modality is quantified using preset evaluation criteria to obtain an initial importance score;

[0019] An attention mechanism is used to weight and sort the initial importance scores, and high-priority modalities with higher than a preset threshold are selected to generate a high-priority modality sequence.

[0020] Optionally, if the visual signal is determined to be dominant in the high-priority modality sequence, the process of extracting the visual dominant feature set includes:

[0021] Determine whether the weight ratio of visual signals in high-priority modal sequences exceeds a preset dominance threshold;

[0022] If the number of features exceeds the limit, a convolutional neural network is used to extract features from the visual signal and construct a preliminary set of visual features.

[0023] The initial set of visual features is screened, subjected to correlation analysis, and grouped and integrated to generate the dominant visual feature group.

[0024] Optionally, the process of cross-validating the visual dominant feature set and tactile data to determine the consistency of information after data fusion includes:

[0025] Visual feature sets and tactile feature sets were extracted from the dominant visual feature set and tactile data, respectively.

[0026] The matching degree between the visual feature set and the tactile feature set was analyzed by cross-validation to establish a preliminary correspondence.

[0027] Based on the initial correspondence, a multimodal fusion algorithm is used to integrate the visual feature set and the tactile feature set in terms of their correlation dimensions to form a fusion feature matrix;

[0028] Calculate the correlation between each dimension in the fusion feature matrix. If the correlation is higher than a preset threshold, the information is determined to be consistent, and a consistency determination result is generated.

[0029] Optionally, the process of dynamically adjusting the transparency control parameters of the projection layers based on the information consistency results to obtain the optimized layer configuration includes:

[0030] A threshold is used to determine the consistency of information results. If the results are lower than the preset consistency threshold, the projected hierarchical data is classified.

[0031] The initial value of the transparency parameter is obtained based on the classification results, and then dynamically adjusted according to the applicable scope.

[0032] Evaluate the matching degree between the adjusted parameters and the hierarchical configuration. If the matching degree is higher than the preset matching threshold, retain the current configuration and generate an optimized hierarchical configuration.

[0033] Optionally, the process of updating the projected display content, providing real-time interactive feedback, and outputting the final projected content according to the optimized hierarchical configuration includes:

[0034] The projection display structure is adjusted based on the optimized hierarchical configuration, and the adjusted hierarchical structure data is obtained.

[0035] The adjusted hierarchical structure data is rendered using a preset display template to generate updated projection display content.

[0036] Optionally, the process of providing real-time interactive feedback and outputting the final projected content includes:

[0037] Collect and categorize real-time user interaction data on the updated projected content.

[0038] If the interaction response status is lower than the preset response threshold, the interaction data will be filtered in layers to obtain the feedback information dataset.

[0039] The feedback information dataset is analyzed and classified to determine the depth perception priority sequence;

[0040] An enhanced output scheme is designed based on the depth perception priority sequence, mapped to the projection level, and the final projected content is output.

[0041] This invention also proposes an AI-supported multimodal desktop interactive projection system for implementing the method, comprising:

[0042] The multimodal acquisition module is used to simultaneously acquire visual signals, tactile data and voice input in the desktop projection environment, and fuse the acquired data to obtain a multimodal information set;

[0043] The priority weight calculation module is used to calculate the priority weight distribution of each mode based on the multimodal information set, and obtain the high-priority mode sequence;

[0044] The visual feature extraction module is used to extract the dominant visual feature group when the visual signal in the high-priority modality sequence is determined to be dominant.

[0045] The cross-validation module is used to cross-validate the visual dominant feature group and tactile data to determine the consistency of information after data fusion.

[0046] The transparency dynamic adjustment module is used to dynamically adjust the transparency control parameters of the projection layers based on the information consistency results, so as to obtain an optimized layer configuration.

[0047] The projection update output module is used to update the projection display content according to the optimized hierarchical configuration, provide real-time interactive feedback, and output the final projection content.

[0048] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0049] Compared with the prior art, the present invention has the following advantages and technical effects:

[0050] This invention discloses an AI-supported multimodal desktop interactive projection method. Addressing the inconsistencies in the fusion of visual, tactile, and speech data collected in desktop projection scenarios, which lead to improper priority allocation, delayed interactive feedback, and blurred depth perception, this method collects multimodal signals using cameras and sensors and fuses them into an initial dataset. An attention mechanism is employed to analyze priorities and calculate weight distributions, determining high-priority modal sequences. If visual signals dominate, a convolutional neural network is used to extract feature vectors, which are then cross-validated with tactile data. A multimodal fusion algorithm is then used to integrate the associated dimensions, assessing information consistency and dynamically adjusting the transparency parameters of the projection layers. This optimizes the layer configuration to update the display and process real-time interactive feedback, thereby achieving enhanced depth perception output. This method effectively solves the inconsistency problem in multimodal data fusion, improves the real-time interactivity and immersiveness of desktop projection, and ultimately optimizes user experience and enhances system intelligence. Attached Figure Description

[0051] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0052] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0053] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0055] Example 1

[0056] This embodiment provides an AI-supported multimodal desktop interactive projection method, including the following steps:

[0057] Simultaneously collect visual signals, tactile data, and voice input in the desktop projection environment, and obtain a multimodal information set after data fusion;

[0058] Based on the multimodal information set, the priority weight distribution of each mode is calculated to obtain the high-priority mode sequence;

[0059] If the visual signal is determined to be dominant in the high-priority modality sequence, then the dominant visual feature group is extracted;

[0060] Cross-validation is performed on the visual dominant feature group and tactile data to determine the consistency of information after data fusion;

[0061] Based on the information consistency results, the transparency control parameters of the projection layers are dynamically adjusted to obtain an optimized layer configuration;

[0062] Based on the optimized hierarchical configuration, update the projected display content, provide real-time interactive feedback, and output the final projected content.

[0063] As a possible implementation method, such as Figure 1 As shown, the AI-supported multimodal desktop interactive projection method proposed in this embodiment may include:

[0064] Step S101: Visual signals, tactile data and voice input from the desktop projection are collected by the camera and sensors and fused into an initial multimodal dataset to obtain a multimodal information set.

[0065] Visual signals, tactile feedback, and speech data are acquired through camera equipment, sensor hardware, and microphone input to form preliminary data collection results. Based on these preliminary results, timestamp alignment is performed on the visual signals, tactile feedback, and speech data to obtain a time-synchronized multimodal data set. If missing values ​​exist in the time-synchronized multimodal data set, linear interpolation is used to fill in the missing parts, determining a complete data sequence. For the complete data sequence, the dimensions of the visual signals, tactile feedback, and speech data are unified using preset standardization rules to obtain a normalized dataset. Using the normalized dataset, synchronization deviations between different modalities are detected. If the deviation exceeds a preset threshold, the deviation is corrected to obtain a corrected multimodal dataset. Based on the corrected multimodal dataset, a support vector machine algorithm is used to classify the data, determine the correlation strength between different modalities, and construct a classified information set. Using the classified information set, key correlation features between different modalities are extracted to determine the final multimodal information set.

[0066] In a feasible human-computer interaction scenario, user data is collected via cameras, sensors, and microphones to form a multimodal information processing flow. For visual signals, the camera captures the user's facial expressions at 30 frames per second, recording instantaneous images of the user smiling or frowning. Haptic feedback is acquired through pressure sensors on the handheld device, recording changes in the force of the user's pressure, such as a light press for confirmation and a heavy press for rejection, with a data sampling rate of 50 times per second. Voice data is collected via the microphone at a frequency of 16kHz, capturing the user's spoken words, such as "Hello" or "Please repeat." These initial data collection results may have timestamp inconsistencies due to device differences, thus requiring time synchronization processing.

[0067] Specifically, in the timestamp alignment process, the time base of all data is unified to a timeline measured in milliseconds. For example, if a visual signal records a smile at 1000 milliseconds, haptic feedback records a light press at 1010 milliseconds, and voice data begins recording "hello" at 990 milliseconds, the alignment algorithm adjusts the time discrepancies of these three data points to the same reference point, such as 1000 milliseconds, ensuring time synchronization of the multimodal data set. If haptic feedback data is found to be missing at 1200 milliseconds, a linear interpolation method can be used to estimate the missing value based on preceding and following data points, completing the data sequence and maintaining continuity.

[0068] In the data standardization stage, the pixel values ​​of visual signals may range from 0 to 255, the pressure values ​​of tactile feedback range from 0 to 10 Newtons, and the amplitude of speech data ranges from -1 to 1. Pre-defined rules are used to unify the dimensions of these three data points to a range of 0 to 1, facilitating subsequent processing. In the synchronization deviation detection, if the time deviation between the visual signal and the speech data is found to exceed a preset threshold of 50 milliseconds, the deviation is corrected by shifting the time axis, ensuring high consistency among the modalities in the corrected multimodal dataset.

[0069] Specifically, the support vector machine algorithm is used to classify the corrected dataset and determine the strength of the correlation between different modalities.

[0070] A visual signal of smiling and a tactile feedback of light pressing might be categorized as "positive confirmation," while a frown and a hard press would fall into the "negative feedback" category. Based on the classification results, key correlation features are extracted, such as the strong association between a smile and the spoken "hello," to construct the final multimodal information set. This not only improves the accuracy of data processing but also enhances the naturalness and intelligence of human-computer interaction.

[0071] In practical applications, this multimodal processing approach significantly improves the system's ability to understand user intent. When a user smiles and says "hello," the system combines visual and vocal features to quickly determine it as a friendly greeting and respond accordingly, rather than responding to misjudgments that might occur with a single modality. This technological effect has significant value in fields such as intelligent customer service and virtual assistants, ensuring a smoother and more human-centered interactive experience.

[0072] Step S102 involves analyzing the priority allocation of each modality based on the multimodal information set, calculating the weight distribution using an attention mechanism, and determining the high-priority modality sequence. Specifically, this includes the following steps:

[0073] Step 1: Obtain a multimodal information set, extract the raw data features of each modality, and classify the data types of different modalities to obtain a preliminary modality dataset. Step 2: Based on the preliminary modality dataset, analyze the importance of each modality, quantify the importance of each modality using pre-established evaluation criteria, and determine the initial importance score for each modality. Step 3: Using the initial importance scores, apply an attention mechanism to assign weights to each modality, calculate the weight distribution results, and obtain a weighted ranking of modality importance. Step 4: Based on the weighted ranking of modality importance, select high-priority modalities that meet a preset threshold to generate a high-priority modality list. Step 5: For the high-priority modality list, combine the weight distribution results to serialize and arrange the modalities to generate the final modality sequence. Step 6: Using the final modality sequence, verify the logical consistency between the modalities within the sequence. If the sequence arrangement is found to be inconsistent with the preset logical rules, readjust the sequence order to obtain an optimized high-priority modality sequence.

[0074] In practice, when processing multimodal information sets, raw data features can be extracted from different modalities such as vision, touch, and speech. For visual data, attention can be paid to color distribution and shape features in the image; for tactile data, changes in pressure magnitude and contact area can be extracted; and for speech data, pitch and speech rate can be considered. Through classification, these features are categorized into different modal datasets, laying the foundation for subsequent analysis.

[0075] When analyzing the importance of each modality, pre-established evaluation criteria can be used, such as quantifying importance based on data completeness and information content. Assuming visual data has 90% completeness, tactile data 70%, and speech data 80%, the initial importance score calculated using the evaluation criteria is 0.9 for visual data, 0.7 for tactile data, and 0.8 for speech data. This quantification method helps to clarify the role of each modality in the overall information.

[0076] When using attention mechanisms for weight allocation, the attention given to each modality can be adjusted based on the initial importance scores. Visual data, due to its higher score, might be allocated 50% weight, speech data 30%, and tactile data 20%. This weight distribution highlights key modalities while also taking into account the supplementary information from other modalities, ensuring more targeted subsequent processing.

[0077] When filtering high-priority modalities, an importance threshold of 0.75 can be set. Based on the weighted ranking, visual and audio data exceeding the threshold are added to the high-priority modality list, while tactile data is temporarily excluded. This filtering method focuses on modalities that contribute significantly to the overall information, improving processing efficiency.

[0078] When sequentially arranging the high-priority modal list, the weight distribution results can be used to prioritize visual data and then speech data, forming the final modal sequence. This arrangement ensures a clear hierarchy of information processing, providing a clear logical foundation for subsequent verification.

[0079] When verifying the logical consistency of modal sequences, the temporal alignment of visual and audio data can be checked. If the timestamps of the visual and audio data are inconsistent—for example, the visual data records a scene at a certain moment, while the audio data lags by 2 seconds—the sequence order needs to be readjusted to ensure temporal alignment. Optimized high-priority modal sequences better reflect the inherent connections between multimodal information, facilitating information integration in subsequent applications. Through this approach, from feature extraction to sequence optimization, each step is closely focused on multimodal data processing, ensuring the comprehensiveness and logical consistency of information processing. Furthermore, reasonable weight allocation and priority filtering significantly improve the relevance and practicality of data processing.

[0080] Step S103: If visual signals dominate in the high-priority modality sequence, feature vectors are extracted through a convolutional neural network to obtain the visually dominant feature group.

[0081] By analyzing high-priority modal sequences, initial signal classification results are obtained to determine whether visual signals are dominant, thus identifying the dominant signal category. If the dominant signal category is visual, a convolutional neural network is used to perform deep processing on the signal, extracting corresponding feature vectors to construct a preliminary visual dominant feature set. For this preliminary visual dominant feature set, a preset threshold is used to filter and remove feature vectors that do not meet the criteria, determining a refined feature subset. Based on the refined feature subset, correlation weights between features are obtained, and dependencies within the feature subset are determined to obtain a feature correlation matrix. If significant dependencies exist in the feature correlation matrix, the feature subset is grouped to construct multiple feature grouping units, determining the grouped feature structure. By integrating the grouped feature structures, the final visual dominant feature set is obtained, completing the feature construction process.

[0082] In practice, when analyzing high-priority modal sequences and obtaining initial signal classification results, we can start by examining the source and characteristics of the signals to determine whether the visual signal is dominant. For example, in a multimodal information processing scenario, the system receives image, sound, and text signals. After priority sorting, it is found that the visual signal has a weight of 70%, far exceeding the weights of other modalities. At this point, the visual signal can be preliminarily determined as the dominant category. This judgment method, based on the salience and information content of the signal, can provide a clear direction for subsequent processing.

[0083] In one possible implementation, if visual signals are confirmed to be dominant, convolutional neural networks can be used for deep processing of the image data. Assuming the input is a set of image data with a resolution of 1080x720, the system will extract features such as edges and textures layer by layer, ultimately generating corresponding feature vectors. This processing method can effectively capture detailed information in the image, laying the foundation for constructing a set of visually dominant features.

[0084] For the initial set of dominant visual features, the system can filter them using a preset threshold. Assuming the threshold is set to 0.8, feature vectors with a confidence level below this value will be discarded. After filtering, a subset of 60 eligible features may be refined from the initial 100 feature vectors. This filtering mechanism helps reduce redundant information and improves the accuracy of subsequent analysis.

[0085] In one possible implementation, for the refined feature subset, the system can further analyze the correlation weights between features and construct a feature association matrix. Suppose the analysis reveals that the correlation coefficient between two features is as high as 0.9, indicating a significant dependency between them. Identifying this dependency helps to understand the intrinsic connections between features and provides a basis for subsequent grouping.

[0086] If the feature correlation matrix shows significant dependencies, the feature subsets can be grouped. For example, 60 features can be divided into 5 groups, each containing highly correlated features. This grouping method simplifies the complex feature structure into more manageable units, facilitating subsequent integration.

[0087] In one possible implementation, a final set of dominant visual features can be generated by integrating the grouped feature structures. Assume the integration yields a set containing core features that encompass the key information in the image. This integration method ensures the integrity and consistency of the feature set, providing reliable support for subsequent applications. This workflow design not only improves the efficiency of feature extraction but also provides a more targeted data foundation for multimodal information processing, facilitating more accurate analysis and judgment in real-world scenarios.

[0088] Step S104 involves cross-validating the visually dominant feature group and tactile data, integrating the associated dimensions using a multimodal fusion algorithm, and determining the consistency of the fused information. This specifically includes the following steps:

[0089] Step 1: Obtain raw signal data from visual features and tactile data respectively. Use a pre-established feature extraction model to perform preliminary processing on the data from both modalities to obtain visual feature sets and tactile feature sets. Step 2: Analyze the matching degree of the visual and tactile feature sets using cross-validation to determine the preliminary correspondence between the two feature sets. Step 3: Based on the preliminary correspondence, use a multimodal fusion algorithm to integrate the correlation dimensions of the visual and tactile feature sets to obtain a fused feature matrix. Step 4: For the fused feature matrix, calculate the correlation between each dimension using dimensional analysis to obtain key dimension combinations. Step 5: Based on the key dimension combinations, if the correlation of each dimension in the fused feature matrix is ​​higher than a preset threshold, the information consistency is deemed to meet the requirements, and a consistency judgment result is obtained. Step 6: Based on the consistency judgment result, record the integration effect of visual features and tactile data through modal correlation analysis to determine the final multimodal data mapping structure.

[0090] In the field of multimodal data processing, the integration of visual and tactile signals can be implemented by starting with the acquisition of raw signals and gradually progressing to feature fusion and consistency analysis. Regarding the acquisition and initial processing of raw signal data, consider a smart interactive device where visual signals capture texture information of an object's surface via a high-resolution camera, while tactile signals record surface hardness data via a pressure sensor. During initial processing, the visual signals might be transformed into a two-dimensional matrix containing edge features, while the tactile signals are organized into a pressure distribution map, forming two separate feature sets. This process ensures the structuring of both modalities, laying the foundation for subsequent analysis.

[0091] In the analysis of feature set matching degree, cross-validation can be used to compare the correspondence between the texture density in the visual feature set and the pressure peak in the tactile feature set. Assuming the texture density value of a certain region in the visual feature set is 0.8, while the pressure value of the corresponding region in the tactile feature set is 0.75, a preliminary correlation can be determined by setting a matching degree threshold of 0.7. This method helps identify potential connections between two modalities of data, providing a basis for subsequent fusion.

[0092] For the application of multimodal fusion algorithms, when integrating visual and tactile feature sets, the two types of features can be mapped into a unified feature space. Assuming the dimensions of the visual feature set are color and texture, and the dimensions of the tactile feature set are pressure and friction, the fusion algorithm can generate a feature matrix containing four-dimensional information. This matrix can comprehensively reflect the characteristics of both modalities, facilitating subsequent analysis.

[0093] In dimensional correlation calculations, the correlation between the color and stress dimensions can be analyzed using the fused feature matrix. If statistical methods reveal a correlation coefficient of 0.85, higher than the preset threshold of 0.6, then the two dimensions are considered to have a strong correlation. Identifying this key dimension combination helps focus on the most representative information.

[0094] In the consistency assessment stage, if the correlation of each dimension in the fused feature matrix is ​​higher than the threshold (e.g., the correlation between color and pressure is 0.85, and the correlation between texture and friction is 0.82), then the information consistency can be determined to meet the requirements. This assessment result can provide a reliable basis for subsequent data integration.

[0095] In determining the final multimodal data mapping structure, modal correlation analysis can be used to record the performance of visual and tactile features during the fusion process.

[0096] Analysis revealed that visual features contribute 60% to the overall mapping structure in certain scenarios, while tactile features contribute 40%, thus forming a balanced mapping structure. This structure effectively supports the enhancement of the perception capabilities of smart devices in interactive scenarios.

[0097] Step S105: Obtain the consistency result of the fused information, dynamically adjust the transparency control parameters of the projection layer, and obtain the optimized layer configuration.

[0098] The process begins by acquiring the original dataset of fused information and performing a consistency analysis. A pre-established evaluation model is used to determine the consistency analysis results. Based on these results, preliminary processing is performed on the structural data of the projection layers. If the consistency analysis result is lower than a preset threshold, the projection layers are segmented to obtain a preliminary classification of the layered data. This preliminary classification yields initial values ​​for transparency parameters. The correspondence between layered data and transparency parameters is used to determine the applicable range of the transparency parameters. Based on the applicable range of the transparency parameters, a dynamic adjustment strategy is implemented. The matching degree between the adjusted parameter values ​​and the layer configuration is evaluated to determine the dynamically adjusted parameter set. This dynamically adjusted parameter set is then used to map the structural characteristics of the layer configuration. If the matching degree after mapping is higher than a preset threshold, the current layer configuration is retained, resulting in an optimized configuration scheme. Using this optimized configuration scheme, consistency verification is performed across the entire information processing flow. A support vector machine algorithm is used to classify the verification results and determine whether the final layer configuration meets the expected standards. Based on the final layer configuration, corresponding control commands are generated to update the transparency parameters of the projection layers in real time, determining the final system operating state.

[0099] In practice, when processing the raw dataset of fused information, basic data can be extracted from both visual and tactile modalities, and consistency analysis can be performed on this data. Assuming a smart interactive device, visual data comes from images captured by a camera, and tactile data comes from feedback from a pressure sensor, the evaluation model will analyze the synchronicity of the two, such as whether timestamps are aligned and whether data distribution is consistent. If the consistency is found to be below a preset threshold, such as 0.75, the projected layered data needs to be segmented into blocks, dividing the data by time period or spatial region to allow for more granular analysis of the root cause of the problem.

[0100] Specifically, for the initial processing of structural data involving projection layers, the initial value of the transparency parameter can be determined by classifying the layered data. Assuming the layered data is divided into three categories: a high-frequency interaction layer, a mid-frequency feedback layer, and a low-frequency background layer, the initial values ​​of the transparency parameter might be 0.9, 0.6, and 0.3, respectively. Based on these values, when determining the applicable range of the transparency parameter, we can analyze whether its visualization effect is clear at different layers, and thus decide whether the parameter range needs to be adjusted. This approach helps ensure the intuitiveness of information presentation.

[0101] When implementing a dynamic adjustment strategy, the adjusted parameter values ​​can be evaluated based on the matching degree of the hierarchical configuration. Assuming the adjusted transparency parameters are 0.85, 0.55, and 0.25, by comparing them with the hierarchical configuration, if the matching degree is higher than the preset threshold of 0.8, the current configuration is retained. This dynamic adjustment can better adapt to the information display needs of different scenarios and improve the system's flexibility.

[0102] Specifically, for mapping processing and hierarchical configuration optimization, the adjusted parameter set can be mapped to specific structural characteristics. For example, the transparency parameter of a high-frequency interaction layer can be linked to the frequency of user operations; if the matching degree after mapping exceeds 0.85, the configuration is considered reasonable. This method ensures that the system configuration is closely integrated with actual needs.

[0103] During consistency verification and classification, the support vector machine algorithm is used to classify the verification results into two categories: those that meet expectations and those that do not. Assuming the verification results show that 90% of the data points meet the expected standards, the hierarchical configuration can be considered reasonable. This classification method helps to quickly locate problem areas.

[0104] Specifically, when generating control commands and updating transparency parameters in real time, commands can be generated based on the final layer configuration to adjust the display effect of the projection layers. For example, the transparency of the high-frequency interaction layer can be updated to 0.88 in real time to adapt to the user's current operational needs. This real-time update can significantly improve user experience and system response efficiency.

[0105] When determining the system's operational status, monitoring the effects of updated parameters can confirm system stability. For example, if the system response time is reduced to 0.2 seconds after an update, and user satisfaction improves by 15%, then the operational status can be considered to have met expectations. This approach provides data support for subsequent optimizations, ensuring long-term system stability.

[0106] Step S106: Update the desktop projection display by optimizing the hierarchical configuration, process real-time interactive feedback, and determine the depth perception enhancement output.

[0107] The process involves acquiring the hierarchical configuration data of the desktop projection, adjusting the structure of the configuration, analyzing the distribution of display layers, and obtaining adjusted hierarchical structure data. Using this adjusted hierarchical structure data, the desktop projection display is updated, and a pre-established display template is used to determine the updated set of display content. Based on the updated set of display content, real-time interaction data records are acquired, categorized, and their response status is assessed. If the response status is below a preset threshold, the interaction data is stratified and filtered to obtain a filtered feedback information dataset. Using this filtered dataset, content parsing is performed on the feedback processing, and a support vector machine algorithm is used to classify the parsing results and determine the priority sequence for depth perception. Based on this priority sequence, an enhanced output scheme is designed, configuration parameters for the perception scheme are acquired, and the final output content format is determined. Finally, using the final output content format, parameter mapping is performed on the desktop projection display layers to obtain the enhanced depth perception display result.

[0108] In practice, when acquiring the hierarchical configuration data for desktop projection, a preliminary analysis of the projection display's hierarchical structure can be performed first. Assuming the desktop projection is divided into three layers: a foreground interaction layer, a midground information layer, and a background environment layer, the initial configuration data might define the foreground layer's proportion as 50%, the midground layer as 30%, and the background layer as 20%. By analyzing the distribution of this data, it can be found that the foreground layer may be too prominent, resulting in insufficient clarity of the midground information. Therefore, the structure needs to be adjusted, reducing the foreground layer's proportion to 40% and increasing the midground layer's proportion to 35%.

[0109] When updating the display content based on the adjusted hierarchical structure data, a pre-established display template can be used. Assuming the template specifies that the foreground layer uses primarily bright colors, the midground layer uses soft colors, and the background layer uses slightly lower saturation colors, the updated display content will re-render the projected image according to these rules, ensuring a stronger sense of depth. This approach helps improve the intuitiveness of information delivery.

[0110] When acquiring and categorizing real-time interaction data, user actions can be divided into three categories: clicks, swipes, and long presses. Assuming the records show that clicks account for 60%, if the response status detection finds that the delay of clicks exceeds a preset threshold of 0.3 seconds, then the interaction data needs to be stratified and filtered to extract the operation data with higher delays as a feedback information dataset for subsequent optimization.

[0111] For content analysis of the feedback information dataset, the analysis results can be classified using the Support Vector Machine (SVM) algorithm. Assuming the analysis reveals that 80% of latency issues are concentrated in click operations, the depth-aware priority sequence can be set to prioritize optimizing click responses. This classification method helps to accurately pinpoint the root cause of the problem.

[0112] When designing an enhanced output scheme based on depth perception priority sequences, fast response configuration parameters can be designed for click operations. Assuming the target perception delay for a click operation is set to 0.2 seconds, the output content format can be defined as high-priority dynamic refresh. This design effectively improves the smoothness of user operations.

[0113] By mapping parameters for the layers in a desktop projection display, enhanced depth perception results can be combined with the hierarchical structure. Assuming the refresh rate of the foreground layer is adjusted to 60 times per second, the midground layer to 30 times per second, and the background layer to 10 times per second, the mapped display result can better highlight key information, improving the overall projection's sense of depth and interactive efficiency. This approach optimizes the user's visual experience while ensuring information clarity.

[0114] Example 2

[0115] Based on the same general inventive concept, this invention also provides an AI-supported multimodal desktop interactive projection system. The system provided by this invention is described below, and the system described below can be referred to in conjunction with the method described above. The system includes:

[0116] The multimodal acquisition module is used to simultaneously acquire visual signals, tactile data and voice input in the desktop projection environment, and fuse the acquired data to obtain a multimodal information set;

[0117] The priority weight calculation module is used to calculate the priority weight distribution of each mode based on the multimodal information set, and obtain the high-priority mode sequence;

[0118] The visual feature extraction module is used to extract the dominant visual feature group when the visual signal in the high-priority modality sequence is determined to be dominant.

[0119] The cross-validation module is used to cross-validate the visual dominant feature group and tactile data to determine the consistency of information after data fusion.

[0120] The transparency dynamic adjustment module is used to dynamically adjust the transparency control parameters of the projection layers based on the information consistency results, so as to obtain an optimized layer configuration.

[0121] The projection update output module is used to update the projection display content according to the optimized hierarchical configuration, provide real-time interactive feedback, and output the final projection content.

[0122] As an scalable implementation, the system uses ultra-short-throw projection technology to create a 60-100 inch screen on a desktop. Based on depth-infrared fusion sensing, it captures the hand skeletal coordinates of 21 key points in real time using a 48MP RGB and infrared depth camera, with a detection latency of ≤40ms. A magnetic wireless directional sound I / O component can quickly attach to the edge of the table, featuring a built-in four-microphone array and beamforming speaker to achieve 30° narrow-directional sound pickup and emission, reducing the sound pressure level of the adjacent table by approximately 12dB. Employing an NPU+GPU heterogeneous architecture for edge computing, it completes local gesture recognition, automatic speech recognition, NLU semantic understanding, and 720p30 digital lip-reading driven inference with ≤8W power consumption. When the user speaks a wake word and simultaneously makes a gesture, the system aligns the voice and gesture through an Early-Fusion+Attention mechanism, driving the digital human to provide immediate feedback, projecting a synchronized animation, and playing the response audio in a directional manner. If the 4G / 5G network is interrupted, the system switches to an offline caching model. New digital avatars or interactive content can be differentially distributed from the cloud, with incremental updates completed on the client side within 5 seconds, eliminating the need for a full system upgrade. Typical application scenarios include: in restaurants, customers can use gestures to select dishes and voice confirmation of doneness, while the digital avatar projects coupon QR codes; in conference rooms, speakers can use gestures to turn pages and display real-time audio subtitles; and in homes, children can practice English reading aloud, with the system scoring based on GOP (Good Practice Points) and correcting pronunciation through digital lip-reading demonstrations, achieving low-interference, low-latency, and offline multi-user closed-loop interaction.

[0123] Example 3

[0124] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0125] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An AI-supported multimodal desktop interactive projection method, characterized in that, Includes the following steps: Simultaneously collect visual signals, tactile data, and voice input in the desktop projection environment, and obtain a multimodal information set after data fusion; Based on the multimodal information set, the priority weight distribution of each mode is calculated to obtain the high-priority mode sequence; If the visual signal is determined to be dominant in the high-priority modality sequence, then the dominant visual feature group is extracted; Cross-validation is performed on the visual dominant feature group and tactile data to determine the consistency of information after data fusion; Based on the information consistency results, the transparency control parameters of the projection layers are dynamically adjusted to obtain an optimized layer configuration; Based on the optimized hierarchical configuration, update the projected display content, provide real-time interactive feedback, and output the final projected content; The process of cross-validating the visual dominant feature set with tactile data to determine the consistency of information after data fusion includes: Visual feature sets and tactile feature sets were extracted from the dominant visual feature set and tactile data, respectively. The matching degree between the visual feature set and the tactile feature set was analyzed by cross-validation to establish a preliminary correspondence. Based on the initial correspondence, a multimodal fusion algorithm is used to integrate the visual feature set and the tactile feature set in terms of their correlation dimensions to form a fusion feature matrix; Calculate the correlation between each dimension in the fusion feature matrix. If the correlation is higher than a preset threshold, the information is determined to be consistent, and a consistency determination result is generated. Based on the information consistency results, the process of dynamically adjusting the transparency control parameters of the projection layers to obtain the optimized layer configuration includes: A threshold is used to determine the consistency of information results. If the results are lower than the preset consistency threshold, the projected hierarchical data is classified. The initial value of the transparency parameter is obtained based on the classification results, and then dynamically adjusted according to the applicable scope. Evaluate the matching degree between the adjusted parameters and the hierarchical configuration. If the matching degree is higher than the preset matching threshold, retain the current configuration and generate an optimized hierarchical configuration.

2. The method according to claim 1, characterized in that, The process of simultaneously acquiring visual signals, tactile data, and voice input in a desktop projection environment, and then fusing the data to obtain a multimodal information set, includes: Visual signals, tactile data, and voice input are collected using cameras, sensors, and microphones to obtain an initial multimodal dataset; The initial multimodal dataset is subjected to timestamp alignment, missing value imputation, unit unification, and synchronization correction to form a synchronous multimodal data group; A classification model is used to identify the correlation between synchronous multimodal data groups, extract key features, and generate a multimodal information set after data fusion.

3. The method according to claim 1, characterized in that, The process of calculating the priority weight distribution of each mode based on the multimodal information set to obtain the high-priority mode sequence includes: Extract the original features of each modality from the multimodal information set and classify and organize them; Based on the categorized and organized modality dataset, the importance of each modality is quantified using preset evaluation criteria to obtain an initial importance score; An attention mechanism is used to weight and sort the initial importance scores, and high-priority modalities with higher than a preset threshold are selected to generate a high-priority modality sequence.

4. The method according to claim 1, characterized in that, If the visual signal in the high-priority modality sequence is determined to be dominant, the process of extracting the dominant visual feature set includes: Determine whether the weight ratio of visual signals in high-priority modal sequences exceeds a preset dominance threshold; If the number of features exceeds the limit, a convolutional neural network is used to extract features from the visual signal and construct a preliminary set of visual features. The initial set of visual features is screened, subjected to correlation analysis, and grouped and integrated to generate the dominant visual feature group.

5. The method according to claim 1, characterized in that, The process of updating the projected display content, providing real-time interactive feedback, and outputting the final projected content based on the optimized hierarchical configuration includes: The projection display structure is adjusted based on the optimized hierarchical configuration, and the adjusted hierarchical structure data is obtained. The adjusted hierarchical structure data is rendered using a preset display template to generate updated projection display content.

6. The method according to claim 5, characterized in that, The process of providing real-time interactive feedback and outputting the final projected content includes: Collect and categorize real-time user interaction data on the updated projected content. If the interaction response status is lower than the preset response threshold, the interaction data will be filtered in layers to obtain the feedback information dataset. The feedback information dataset is analyzed and classified to determine the depth perception priority sequence; An enhanced output scheme is designed based on the depth perception priority sequence, mapped to the projection level, and the final projected content is output.

7. An AI-supported multimodal desktop interactive projection system, characterized in that, For implementing the method of any one of claims 1-6, comprising: The multimodal acquisition module is used to simultaneously acquire visual signals, tactile data and voice input in the desktop projection environment, and fuse the acquired data to obtain a multimodal information set; The priority weight calculation module is used to calculate the priority weight distribution of each mode based on the multimodal information set, and obtain the high-priority mode sequence; The visual feature extraction module is used to extract the dominant visual feature group when the visual signal in the high-priority modality sequence is determined to be dominant. The cross-validation module is used to cross-validate the visual dominant feature group and tactile data to determine the consistency of information after data fusion. The transparency dynamic adjustment module is used to dynamically adjust the transparency control parameters of the projection layers based on the information consistency results, so as to obtain an optimized layer configuration. The projection update output module is used to update the projection display content according to the optimized hierarchical configuration, provide real-time interactive feedback, and output the final projection content.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

Images