AI-supported multi-modal desktop interactive projection method and system

Abstract

The invention discloses an AI-supported multi-modal desktop interactive projection method and system, and the method comprises the following steps: synchronously collecting visual signals, tactile data and voice input in a desktop projection environment, and obtaining a multi-modal information set after data fusion; calculating priority weight distribution of each mode according to the multi-mode information set, and obtaining a high-priority mode sequence; if the visual signals in the high-priority modal sequence are judged to be dominant, extracting a visual dominant feature group; performing cross validation on the visual dominant feature group and the tactile data, and judging information consistency after data fusion; according to an information consistency result, dynamically adjusting transparency control parameters of projection layering to obtain optimized hierarchical configuration; and updating the projection display content according to the optimized hierarchical configuration, and outputting the final projection content. According to the method, the interaction real-time performance and immersion of desktop projection are improved, and finally the technical effects of optimizing user experience and improving system intelligence are achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of desktop interactive projection, and particularly relates to an AI-supported multi-modal desktop interactive projection method and system. BACKGROUND

[0002] In the field of modern intelligent interaction, desktop projection technology, as an important bridge connecting virtual and reality, shows irreplaceable value. It not only can enhance the immersion of users in work, study and entertainment, but also can realize seamless communication between people and devices through intuitive way. However, the research and application in this field are facing unprecedented challenges, and innovation breakthrough is urgently needed to promote technological progress.

[0003] At present, although desktop projection interaction technology has made certain progress, most methods are not up to the task when dealing with complex scenes. Especially in the face of the integration of multiple sensory information, the existing scheme often has difficulty in balancing the presentation and response of different types of information, which easily leads to user confusion or information overload in the interaction process. This problem cannot be solved simply by technology stacking, but is caused by the lack of dynamic coordination ability of multi-source information, especially in a multi-user or multi-task environment, the system often cannot accurately judge the user's demand, resulting in a big discount in the interactive experience.

[0004] In view of the above problems in the prior art, an AI-supported multi-modal desktop interactive projection method is urgently needed. SUMMARY

[0005] To solve the above technical problems, the present application proposes an AI-supported multi-modal desktop interactive projection method, comprising the following steps:

[0006] Synchronously collecting visual signals, tactile data and voice input in the desktop projection environment, and obtaining a multi-modal information set after data fusion;

[0007] According to the multi-modal information set, the priority weight distribution of each mode is calculated to obtain a high priority mode sequence;

[0008] If the visual signal in the high priority mode sequence is determined as the dominant, the visual dominant feature group is extracted;

[0009] Cross-validation is performed on the visual dominant feature group and tactile data to judge the consistency of the information after data fusion;

[0010] According to the information consistency result, the transparency control parameter of projection layering is dynamically adjusted to obtain an optimized hierarchical configuration;

[0011] According to the optimized hierarchical configuration, the projection display content is updated, real-time interaction feedback is carried out, and the final projection content is output.

[0012] Optionally, the process of synchronously collecting visual signals, tactile data and voice inputs in a desktop projection environment, and obtaining a multi-modal information set after data fusion includes:

[0013] The initial multi-modal data set is obtained by collecting visual signals, tactile data and voice inputs through a camera, a sensor and a microphone respectively;

[0014] The initial multi-modal data set is time-stamped, missing value filled, dimension unified and synchronized to form a synchronized multi-modal data set;

[0015] The correlation between the synchronized multi-modal data sets is identified by using a classification model, key features are extracted, and a multi-modal information set after data fusion is generated.

[0016] Optionally, according to the multi-modal information set, the process of calculating the priority weight distribution of each modality to obtain a high-priority modality sequence includes:

[0017] The original features of each modality are extracted from the multi-modal information set and classified and arranged;

[0018] Based on the classified and arranged modality data set, the importance of each modality is quantified according to a preset evaluation standard to obtain an initial importance score;

[0019] The initial importance score is weighted and sorted using an attention mechanism, and high-priority modalities higher than a preset threshold are filtered out to generate a high-priority modality sequence.

[0020] Optionally, if the visual signal in the high-priority modality sequence is determined to be dominant, the process of extracting a visual dominant feature group includes:

[0021] Determine whether the weight proportion of the visual signal in the high-priority modality sequence exceeds a preset dominant threshold;

[0022] If so, a convolutional neural network is used to extract features from the visual signal to construct a preliminary visual feature set;

[0023] The preliminary visual feature set is filtered, analyzed for correlation and integrated into groups to generate a visual dominant feature group.

[0024] Optionally, the process of cross-verifying the visual dominant feature group and the tactile data to determine the consistency of the information after data fusion includes:

[0025] Visual feature sets and tactile feature sets are extracted from the visual dominant feature group and the tactile data respectively;

[0026] The matching degree of the visual feature sets and the tactile feature sets is analyzed by a cross-validation method to establish a preliminary correspondence;

[0027] Based on the preliminary correspondence, a multi-modal fusion algorithm is used to integrate the correlation dimensions of the visual feature set and the tactile feature set, and a fusion feature matrix is formed.

[0028] The correlation between each dimension in the fusion feature matrix is calculated, and if the correlation is higher than a preset threshold, it is determined that the information is consistent, and a consistency determination result is generated.

[0029] Optionally, according to the information consistency result, the transparency control parameter of the projection layer is dynamically adjusted, and the process of obtaining the optimized hierarchical configuration includes:

[0030] The information consistency result is subjected to threshold determination, and if it is lower than a preset consistency threshold, the projection layer data is subjected to classification processing;

[0031] According to the classification result, an initial value of the transparency parameter is obtained, and dynamic adjustment is implemented in combination with the applicable range;

[0032] The matching degree of the adjusted parameter and the hierarchical configuration is evaluated, and if the matching degree is higher than a preset matching threshold, the current configuration is retained, and an optimized hierarchical configuration is generated.

[0033] Optionally, according to the optimized hierarchical configuration, the projection display content is updated, real-time interactive feedback is performed, and the process of outputting the final projection content includes:

[0034] Based on the optimized hierarchical configuration, the projection display structure is adjusted, and hierarchical structure data after adjustment is obtained;

[0035] The hierarchical structure data after adjustment is rendered by using a preset display template, and updated projection display content is generated.

[0036] Optionally, the process of real-time interactive feedback and output of the final projection content includes:

[0037] Real-time interactive data of the user on the updated projection content is collected and classified;

[0038] If the interactive response state is lower than a preset response threshold, the interactive data is subjected to hierarchical screening, and a feedback information data set is obtained;

[0039] The feedback information data set is parsed and classified to determine a depth perception priority sequence;

[0040] An enhanced output scheme is designed according to the depth perception priority sequence, mapped to the projection hierarchy, and the final projection content is output.

[0041] The application also proposes an AI-supported multi-modal desktop interactive projection system for implementing the method, comprising:

[0042] A multi-modal acquisition module is configured to synchronously acquire visual signals, tactile data and voice input in a desktop projection environment, and fuse the acquired data to obtain a multi-modal information set;

[0043] A priority weight calculation module is configured to calculate priority weight distribution of each modality according to the multi-modal information set, and obtain a high-priority modality sequence;

[0044] A visual feature extraction module is configured to extract a visual dominant feature group when the visual signal in the high-priority modality sequence is determined to be dominant.

[0045] A cross-validation module is configured to cross-validate the visual dominant feature group and the tactile data to determine information consistency after data fusion.

[0046] A transparency dynamic adjustment module is configured to dynamically adjust transparency control parameters of projection layering according to the information consistency result, and obtain an optimized hierarchical configuration.

[0047] A projection update output module is configured to update projection display content according to the optimized hierarchical configuration, perform real-time interactive feedback, and output final projection content.

[0048] The application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to realize the steps of the method.

[0049] Compared with the prior art, the application has the following advantages and technical effects:

[0050] The AI-supported multi-modal desktop interaction projection method disclosed by the application solves the inconsistency problem in multi-modal data fusion, improves the real-time interaction and immersion of desktop projection, and finally achieves the technical effects of optimizing user experience and improving system intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and of the prior art, can be best understood from the accompanying drawings, wherein:

[0052] Figure 1 A flowchart of a method according to an embodiment of the application. DETAILED DESCRIPTION

[0053] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0054] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0055] Embodiment one

[0056] An AI-supported multi-modal desktop interaction projection method is provided in this embodiment, which includes the following steps:

[0057] Synchronize the collection of visual signals, tactile data and voice input in the desktop projection environment, and obtain a multi-modal information set after data fusion;

[0058] According to the multi-modal information set, the priority weight distribution of each modality is calculated to obtain a high priority modality sequence;

[0059] If the visual signal in the high priority modality sequence is determined to be dominant, the visual dominant feature group is extracted;

[0060] Cross-validation is performed on the visual dominant feature group and tactile data to determine the consistency of the information after data fusion;

[0061] According to the information consistency result, the transparency control parameter of the projection layer is dynamically adjusted to obtain an optimized hierarchical configuration;

[0062] According to the optimized hierarchical configuration, the projection display content is updated, real-time interaction feedback is performed, and the final projection content is output.

[0063] As a feasible implementation, as shown in Figure 1 The AI-supported multi-modal desktop interaction projection method proposed in this embodiment can include:

[0064] Step S101, the visual signal, tactile data and voice input in the desktop projection are collected by the camera and the sensor, and are fused into an initial multi-modal data set to obtain a multi-modal information set.

[0065] The visual signal, tactile feedback and speech data are respectively acquired through the camera device, sensor hardware and microphone input to form preliminary acquisition results. According to the preliminary acquisition results, time stamp alignment processing is performed on the visual signal, tactile feedback and speech data to obtain a time-synchronized multi-modal data group. If there are missing values in the time-synchronized multi-modal data group, the missing part is filled by a linear interpolation method to determine a complete data sequence. For the complete data sequence, a preset standardization rule is used to unify the dimensions of the visual signal, tactile feedback and speech data to obtain a standardized data set. Through the standardized data set, the synchronization deviation between the modal data is detected, and if the deviation exceeds a preset threshold, the deviation part is corrected to obtain a corrected multi-modal data set. According to the corrected multi-modal data set, a support vector machine algorithm is used to classify the data to determine the correlation strength between the modal data and construct a classified information set. Through the classified information set, key correlation features between the modal data are extracted to determine a final multi-modal information set.

[0066] In an implementable human-computer interaction dialogue scenario, user data is collected through a camera, a sensor and a microphone to form a multi-modal information processing flow. For the visual signal, it is assumed that the camera captures the user's facial expression at a frequency of 30 frames per second, and records the momentary image when the user smiles or frowns. The tactile feedback is obtained through a pressure sensor on the handheld device, recording the force change of the user pressing the device, such as light pressing indicating confirmation and heavy pressing indicating rejection, with a data sampling rate of 50 times per second. The speech data is collected through the microphone at a frequency of 16 kHz to collect the user's speech content, such as "hello" or "repeat, please". These preliminary acquisition results may have inconsistent time stamps due to device differences, so time synchronization processing is needed.

[0067] Specifically, in the time stamp alignment processing, the time reference of all data is unified to a time axis in units of milliseconds. It is assumed that the visual signal records a smile at 1000 milliseconds, the tactile feedback records a light press at 1010 milliseconds, and the speech data starts recording "hello" at 990 milliseconds. Through the alignment algorithm, the time deviation of the three is adjusted to the same reference point, such as 1000 milliseconds, to ensure the time synchronization of the multi-modal data group. If it is found that the tactile feedback is missing data at 1200 milliseconds, a linear interpolation method can be used to estimate the missing value based on the data points before and after it to complete the data sequence and maintain continuity.

[0068] In the data standardization process, the pixel value range of visual signals may be 0 to 255, the pressure value range of tactile feedback is 0 to 10 Newton, and the amplitude range of voice data is -1 to 1. By using preset rules, the three dimensions are unified to the range of 0 to 1, facilitating subsequent processing. In the synchronization deviation detection, if it is found that the time deviation of visual signals and voice data exceeds the preset threshold of 50 milliseconds, the deviation is corrected by time axis translation to ensure that the corrected multi-modal data set is highly consistent among the modalities.

[0069] Specifically, the corrected data set is classified by using a support vector machine algorithm to determine the strength of the correlation between modalities.

[0070] Smiling visual signals and light pressing tactile feedback may be classified as the "positive confirmation" category, while frowning and heavy pressing are classified as the "negative feedback" category. By the classification results, key correlation features such as the strong correlation between smiling and "hello" voice are extracted to construct the final multi-modal information set. This not only improves the accuracy of data processing, but also enhances the naturalness and intelligence of human-computer interaction.

[0071] In practical applications, this multi-modal processing method can significantly improve the system's understanding of user intentions. When the user smiles and says "hello", the system combines visual and voice features to quickly determine it as a friendly greeting and respond accordingly, rather than a misjudgment that may be caused by a single modality. Such technical effects have important value in the fields of intelligent customer service, virtual assistants, etc., ensuring a more smooth and humanized interactive experience.

[0072] Step S102, according to the multi-modal information set, the priority allocation of each modality is analyzed, the weight distribution is calculated by using the attention mechanism, and the high priority modality sequence is determined. Specifically, the following steps are included:

[0073] Step one: obtain the multi-modal information set, extract the original data features of each modality from it, classify and process the data types of different modalities, and obtain the preliminary sorted modality data set. Step two: according to the preliminary sorted modality data set, analyze the importance of each modality, use the pre-established evaluation standard to quantify the importance of the modality, and determine the initial importance score of each modality. Step three: by using the initial importance score, the attention mechanism is used to allocate weights to each modality, calculate the weight distribution result, and obtain the weighted modality importance ranking. Step four: according to the weighted modality importance ranking, filter out the high priority modalities that meet the preset threshold, and generate a high priority modality list. Step five: for the high priority modality list, combined with the weight distribution result, the modalities are sequentially arranged to generate the final modality sequence. Step six: through the final modality sequence, the logical consistency between the modalities in the sequence is verified, if it is detected that the sequence arrangement does not meet the preset logical rules, the sequence order is adjusted to obtain an optimized high priority modality sequence.

[0074] In practice, when processing a multi-modal information set, raw data features can be extracted from different modalities such as vision, touch, and speech. For visual data, attention can be paid to color distribution and shape features in images; for touch data, pressure magnitude and contact area changes can be extracted; and for speech data, tone and speed can be focused on. Through classification processing, these features are classified into different modal data sets, laying the foundation for subsequent analysis.

[0075] When analyzing the importance of each modality, pre-established evaluation criteria can be used, such as quantifying importance based on data integrity and information size. Assuming the integrity of visual data is 90%, touch data is 70%, and speech data is 80%, the initial importance scores of visual data, touch, and speech are calculated as 0.9, 0.7, and 0.8 respectively. This quantification method helps to clarify the role of each modality in the overall information.

[0076] When using attention mechanisms for weight distribution, the focus of each modality can be adjusted based on the initial importance score. Visual data, with a higher score, may be allocated 50% of the weight, speech data 30%, and touch data 20%. Such weight distribution results can highlight key modalities while taking into account the information supplementing role of other modalities, ensuring that subsequent processing is more targeted.

[0077] When filtering high-priority modalities, an importance threshold of 0.75 can be set. According to the weighted ranking, visual data and speech data exceed the threshold and enter the high-priority modality list, while touch data is temporarily excluded. This filtering method can focus on modalities that contribute more to the overall information, improving processing efficiency.

[0078] When serializing the high-priority modality list, visual data can be placed first and speech data second based on the weight distribution results to form the final modality sequence. This arrangement ensures that the information processing is clear in terms of priority, providing a clear logical basis for subsequent verification.

[0079] In verifying the logical consistency of the modal sequence, it can be checked whether the visual data and the voice data match in the time dimension. If it is found that the timestamp of the visual data is inconsistent with the voice data, such as the visual data records a picture at a certain time, and the voice data lags behind for 2 seconds, the sequence order needs to be adjusted to ensure the time alignment of the two. The optimized high-priority modal sequence can better reflect the internal relationship of multi-modal information, which is helpful for information integration in subsequent applications. Through the above-mentioned manner, from feature extraction to sequence optimization, each link closely revolves around the processing of multi-modal data, ensuring the comprehensiveness and logicality of information processing, and through reasonable weight distribution and priority screening, the pertinence and practicality of data processing are significantly improved.

[0080] In step S103, if the visual signal is dominant in the high-priority modal sequence, a feature vector is extracted through a convolutional neural network to obtain a visual dominant feature group.

[0081] Through analysis of the high-priority modal sequence, an initial signal classification result is obtained to determine whether the visual signal is dominant, and a dominant signal category is obtained. If the dominant signal category is the visual signal, a convolutional neural network is used to perform deep processing on the signal to extract a corresponding feature vector and construct a preliminary visual dominant feature set. For the preliminary visual dominant feature set, a pre-set threshold is used for screening to eliminate feature vectors that do not meet the conditions and determine a refined feature subset. According to the refined feature subset, the correlation weight between the features is obtained to determine the dependency relationship within the feature subset and obtain a feature correlation matrix. If there is a significant dependency relationship in the feature correlation matrix, the feature subset is grouped and processed to construct multiple feature grouping units and determine the grouped feature structure. By integrating the grouped feature structure, the final visual dominant feature set is obtained to complete the feature construction process.

[0082] It can be implemented that in analyzing the high-priority modal sequence and obtaining the initial signal classification result, it can be determined whether the visual signal is dominant from the source and characteristics of the signal. Assuming that in a multi-modal information processing scene, the system receives three signals of image, sound and text, and after priority sorting, it is found that the weight proportion of the visual signal reaches 70%, which is much higher than that of other modalities, at this time, it can be preliminarily determined that the visual signal is the dominant category. This judgment method is based on the significance and information amount of the signal, which can provide a clear direction for subsequent processing.

[0083] In one possible implementation, if the visual signal is dominant, the image data can be processed deeply using a convolutional neural network. Assuming that the input is a set of image data with a resolution of 1080x720, the system will extract features such as edges and textures layer by layer, and finally generate corresponding feature vectors. This processing method can effectively capture detailed information in the image, laying the foundation for constructing the visual dominant feature set.

[0084] For the preliminary visual dominant feature set, the system can filter it through a preset threshold. Assuming that the threshold is set to 0.8, feature vectors with a confidence level below this value will be excluded. After filtering, 60 feature subsets that meet the conditions may be refined from the initial 100 feature vectors. This filtering mechanism helps to reduce redundant information and improve the accuracy of subsequent analysis.

[0085] In one possible implementation, for the refined feature subset, the system can further analyze the correlation weight between features and construct a feature correlation matrix. Assuming that in the analysis it is found that the correlation coefficient of two features is as high as 0.9, indicating that there is a significant dependent relationship between them. The identification of this dependent relationship helps to understand the internal relationship between features, providing a basis for subsequent grouping.

[0086] If the feature correlation matrix shows significant dependent relationships, the feature subset can be grouped and processed. Assuming that the 60 features are divided into 5 grouping units, each containing features with high correlation. This grouping method can simplify complex feature structures into more manageable units, facilitating subsequent integration.

[0087] In one possible implementation, by integrating the grouped feature structure, the final visual dominant feature set can be generated. Assuming that after integration, a set of core features is obtained, covering the key information in the image. This integration method can ensure the integrity and consistency of the feature set, providing reliable support for subsequent applications. Such a process design not only improves the efficiency of feature extraction, but also provides a more targeted data basis for multi-modal information processing, which helps to achieve more accurate analysis and judgment in actual scenarios.

[0088] Step S104, cross-verification is performed on the visual dominant feature group and the tactile data, and a multi-modal fusion algorithm is used to integrate the correlation dimension to judge the consistency of the integrated information. Specifically, the following steps are included:

[0089] Step 1: Obtain raw signal data from visual features and tactile data respectively. Use a pre-established feature extraction model to perform preliminary processing on the data from both modalities to obtain visual feature sets and tactile feature sets. Step 2: Analyze the matching degree of the visual and tactile feature sets using cross-validation to determine the preliminary correspondence between the two feature sets. Step 3: Based on the preliminary correspondence, use a multimodal fusion algorithm to integrate the correlation dimensions of the visual and tactile feature sets to obtain a fused feature matrix. Step 4: For the fused feature matrix, calculate the correlation between each dimension using dimensional analysis to obtain key dimension combinations. Step 5: Based on the key dimension combinations, if the correlation of each dimension in the fused feature matrix is ​​higher than a preset threshold, the information consistency is deemed to meet the requirements, and a consistency judgment result is obtained. Step 6: Based on the consistency judgment result, record the integration effect of visual features and tactile data through modal correlation analysis to determine the final multimodal data mapping structure.

[0090] In the field of multimodal data processing, the integration of visual and tactile signals can be implemented by starting with the acquisition of raw signals and gradually progressing to feature fusion and consistency analysis. Regarding the acquisition and initial processing of raw signal data, consider a smart interactive device where visual signals capture texture information of an object's surface via a high-resolution camera, while tactile signals record surface hardness data via a pressure sensor. During initial processing, the visual signals might be transformed into a two-dimensional matrix containing edge features, while the tactile signals are organized into a pressure distribution map, forming two separate feature sets. This process ensures the structuring of both modalities, laying the foundation for subsequent analysis.

[0091] In the analysis of feature set matching degree, cross-validation can be used to compare the correspondence between the texture density in the visual feature set and the pressure peak in the tactile feature set. Assuming the texture density value of a certain region in the visual feature set is 0.8, while the pressure value of the corresponding region in the tactile feature set is 0.75, a preliminary correlation can be determined by setting a matching degree threshold of 0.7. This method helps identify potential connections between two modalities of data, providing a basis for subsequent fusion.

[0092] For the application of multimodal fusion algorithms, when integrating visual and tactile feature sets, the two types of features can be mapped into a unified feature space. Assuming the dimensions of the visual feature set are color and texture, and the dimensions of the tactile feature set are pressure and friction, the fusion algorithm can generate a feature matrix containing four-dimensional information. This matrix can comprehensively reflect the characteristics of both modalities, facilitating subsequent analysis.

[0093] In dimensional correlation calculations, the correlation between the color and stress dimensions can be analyzed using the fused feature matrix. If statistical methods reveal a correlation coefficient of 0.85, higher than the preset threshold of 0.6, then the two dimensions are considered to have a strong correlation. Identifying this key dimension combination helps focus on the most representative information.

[0094] In the consistency assessment stage, if the correlation of each dimension in the fused feature matrix is ​​higher than the threshold (e.g., the correlation between color and pressure is 0.85, and the correlation between texture and friction is 0.82), then the information consistency can be determined to meet the requirements. This assessment result can provide a reliable basis for subsequent data integration.

[0095] In determining the final multimodal data mapping structure, modal correlation analysis can be used to record the performance of visual and tactile features during the fusion process.

[0096] Analysis revealed that visual features contribute 60% to the overall mapping structure in certain scenarios, while tactile features contribute 40%, thus forming a balanced mapping structure. This structure effectively supports the enhancement of the perception capabilities of smart devices in interactive scenarios.

[0097] Step S105: Obtain the consistency result of the fused information, dynamically adjust the transparency control parameters of the projection layer, and obtain the optimized layer configuration.

[0098] The process begins by acquiring the original dataset of fused information and performing a consistency analysis. A pre-established evaluation model is used to determine the consistency analysis results. Based on these results, preliminary processing is performed on the structural data of the projection layers. If the consistency analysis result is lower than a preset threshold, the projection layers are segmented to obtain a preliminary classification of the layered data. This preliminary classification yields initial values ​​for transparency parameters. The correspondence between layered data and transparency parameters is used to determine the applicable range of the transparency parameters. Based on the applicable range of the transparency parameters, a dynamic adjustment strategy is implemented. The matching degree between the adjusted parameter values ​​and the layer configuration is evaluated to determine the dynamically adjusted parameter set. This dynamically adjusted parameter set is then used to map the structural characteristics of the layer configuration. If the matching degree after mapping is higher than a preset threshold, the current layer configuration is retained, resulting in an optimized configuration scheme. Using this optimized configuration scheme, consistency verification is performed across the entire information processing flow. A support vector machine algorithm is used to classify the verification results and determine whether the final layer configuration meets the expected standards. Based on the final layer configuration, corresponding control commands are generated to update the transparency parameters of the projection layers in real time, determining the final system operating state.

[0099] In practice, when processing the raw dataset of fused information, basic data can be extracted from both visual and tactile modalities, and consistency analysis can be performed on this data. Assuming a smart interactive device, visual data comes from images captured by a camera, and tactile data comes from feedback from a pressure sensor, the evaluation model will analyze the synchronicity of the two, such as whether timestamps are aligned and whether data distribution is consistent. If the consistency is found to be below a preset threshold, such as 0.75, the projected layered data needs to be segmented into blocks, dividing the data by time period or spatial region to allow for more granular analysis of the root cause of the problem.

[0100] Specifically, for the initial processing of structural data involving projection layers, the initial value of the transparency parameter can be determined by classifying the layered data. Assuming the layered data is divided into three categories: a high-frequency interaction layer, a mid-frequency feedback layer, and a low-frequency background layer, the initial values ​​of the transparency parameter might be 0.9, 0.6, and 0.3, respectively. Based on these values, when determining the applicable range of the transparency parameter, we can analyze whether its visualization effect is clear at different layers, and thus decide whether the parameter range needs to be adjusted. This approach helps ensure the intuitiveness of information presentation.

[0101] When implementing a dynamic adjustment strategy, the adjusted parameter values ​​can be evaluated based on the matching degree of the hierarchical configuration. Assuming the adjusted transparency parameters are 0.85, 0.55, and 0.25, by comparing them with the hierarchical configuration, if the matching degree is higher than the preset threshold of 0.8, the current configuration is retained. This dynamic adjustment can better adapt to the information display needs of different scenarios and improve the system's flexibility.

[0102] Specifically, for mapping processing and hierarchical configuration optimization, the adjusted parameter set can be mapped to specific structural characteristics. For example, the transparency parameter of a high-frequency interaction layer can be linked to the frequency of user operations; if the matching degree after mapping exceeds 0.85, the configuration is considered reasonable. This method ensures that the system configuration is closely integrated with actual needs.

[0103] During consistency verification and classification, the support vector machine algorithm is used to classify the verification results into two categories: those that meet expectations and those that do not. Assuming the verification results show that 90% of the data points meet the expected standards, the hierarchical configuration can be considered reasonable. This classification method helps to quickly locate problem areas.

[0104] Specifically, when generating control commands and updating transparency parameters in real time, commands can be generated based on the final layer configuration to adjust the display effect of the projection layers. For example, the transparency of the high-frequency interaction layer can be updated to 0.88 in real time to adapt to the user's current operational needs. This real-time update can significantly improve user experience and system response efficiency.

[0105] When determining the system's operational status, monitoring the effects of updated parameters can confirm system stability. For example, if the system response time is reduced to 0.2 seconds after an update, and user satisfaction improves by 15%, then the operational status can be considered to have met expectations. This approach provides data support for subsequent optimizations, ensuring long-term system stability.

[0106] Step S106: Update the desktop projection display by optimizing the hierarchical configuration, process real-time interactive feedback, and determine the depth perception enhancement output.

[0107] The process involves acquiring the hierarchical configuration data of the desktop projection, adjusting the structure of the configuration, analyzing the distribution of display layers, and obtaining adjusted hierarchical structure data. Using this adjusted hierarchical structure data, the desktop projection display is updated, and a pre-established display template is used to determine the updated set of display content. Based on the updated set of display content, real-time interaction data records are acquired, categorized, and their response status is assessed. If the response status is below a preset threshold, the interaction data is stratified and filtered to obtain a filtered feedback information dataset. Using this filtered dataset, content parsing is performed on the feedback processing, and a support vector machine algorithm is used to classify the parsing results and determine the priority sequence for depth perception. Based on this priority sequence, an enhanced output scheme is designed, configuration parameters for the perception scheme are acquired, and the final output content format is determined. Finally, using the final output content format, parameter mapping is performed on the desktop projection display layers to obtain the enhanced depth perception display result.

[0108] In practice, when acquiring the hierarchical configuration data for desktop projection, a preliminary analysis of the projection display's hierarchical structure can be performed first. Assuming the desktop projection is divided into three layers: a foreground interaction layer, a midground information layer, and a background environment layer, the initial configuration data might define the foreground layer's proportion as 50%, the midground layer as 30%, and the background layer as 20%. By analyzing the distribution of this data, it can be found that the foreground layer may be too prominent, resulting in insufficient clarity of the midground information. Therefore, the structure needs to be adjusted, reducing the foreground layer's proportion to 40% and increasing the midground layer's proportion to 35%.

[0109] When updating the display content based on the adjusted hierarchical structure data, a pre-established display template can be used. Assuming the template specifies that the foreground layer uses primarily bright colors, the midground layer uses soft colors, and the background layer uses slightly lower saturation colors, the updated display content will re-render the projected image according to these rules, ensuring a stronger sense of depth. This approach helps improve the intuitiveness of information delivery.

[0110] When acquiring and categorizing real-time interaction data, user actions can be divided into three categories: clicks, swipes, and long presses. Assuming the records show that clicks account for 60%, if the response status detection finds that the delay of clicks exceeds a preset threshold of 0.3 seconds, then the interaction data needs to be stratified and filtered to extract the operation data with higher delays as a feedback information dataset for subsequent optimization.

[0111] For content analysis of the feedback information dataset, the analysis results can be classified using the Support Vector Machine (SVM) algorithm. Assuming the analysis reveals that 80% of latency issues are concentrated in click operations, the depth-aware priority sequence can be set to prioritize optimizing click responses. This classification method helps to accurately pinpoint the root cause of the problem.

[0112] When designing an enhanced output scheme based on depth perception priority sequences, fast response configuration parameters can be designed for click operations. Assuming the target perception delay for a click operation is set to 0.2 seconds, the output content format can be defined as high-priority dynamic refresh. This design effectively improves the smoothness of user operations.

[0113] By mapping parameters for the layers in a desktop projection display, enhanced depth perception results can be combined with the hierarchical structure. Assuming the refresh rate of the foreground layer is adjusted to 60 times per second, the midground layer to 30 times per second, and the background layer to 10 times per second, the mapped display result can better highlight key information, improving the overall projection's sense of depth and interactive efficiency. This approach optimizes the user's visual experience while ensuring information clarity.

[0114] Example 2

[0115] Based on the same general inventive concept, this invention also provides an AI-supported multimodal desktop interactive projection system. The system provided by this invention is described below, and the system described below can be referred to in conjunction with the method described above. The system includes:

[0116] The multimodal acquisition module is used to simultaneously acquire visual signals, tactile data and voice input in the desktop projection environment, and fuse the acquired data to obtain a multimodal information set;

[0117] The priority weight calculation module is used to calculate the priority weight distribution of each mode based on the multimodal information set, and obtain the high-priority mode sequence;

[0118] The visual feature extraction module is used to extract the dominant visual feature group when the visual signal in the high-priority modality sequence is determined to be dominant.

[0119] The cross-validation module is used to cross-validate the visual dominant feature group and tactile data to determine the consistency of information after data fusion.

[0120] The transparency dynamic adjustment module is used to dynamically adjust the transparency control parameters of the projection layers based on the information consistency results, so as to obtain an optimized layer configuration.

[0121] The projection update output module is used to update the projection display content according to the optimized hierarchical configuration, provide real-time interactive feedback, and output the final projection content.

[0122] As an scalable implementation, the system uses ultra-short-throw projection technology to create a 60-100 inch screen on a desktop. Based on depth-infrared fusion sensing, it captures the hand skeletal coordinates of 21 key points in real time using a 48MP RGB and infrared depth camera, with a detection latency of ≤40ms. A magnetic wireless directional sound I / O component can quickly attach to the edge of the table, featuring a built-in four-microphone array and beamforming speaker to achieve 30° narrow-directional sound pickup and emission, reducing the sound pressure level of the adjacent table by approximately 12dB. Employing an NPU+GPU heterogeneous architecture for edge computing, it completes local gesture recognition, automatic speech recognition, NLU semantic understanding, and 720p30 digital lip-reading driven inference with ≤8W power consumption. When the user speaks a wake word and simultaneously makes a gesture, the system aligns the voice and gesture through an Early-Fusion+Attention mechanism, driving the digital human to provide immediate feedback, projecting a synchronized animation, and playing the response audio in a directional manner. If the 4G / 5G network is interrupted, the system switches to an offline caching model. New digital avatars or interactive content can be differentially distributed from the cloud, with incremental updates completed on the client side within 5 seconds, eliminating the need for a full system upgrade. Typical application scenarios include: in restaurants, customers can use gestures to select dishes and voice confirmation of doneness, while the digital avatar projects coupon QR codes; in conference rooms, speakers can use gestures to turn pages and display real-time audio subtitles; and in homes, children can practice English reading aloud, with the system scoring based on GOP (Good Practice Points) and correcting pronunciation through digital lip-reading demonstrations, achieving low-interference, low-latency, and offline multi-user closed-loop interaction.

[0123] Example 3

[0124] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0125] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An AI-supported multi-modal tabletop interaction projection method, characterized in that, The method comprises the following steps: synchronously collecting visual signals, tactile data and voice inputs in a desktop projection environment, and obtaining a multi-modal information set after data fusion; calculating priority weight distribution of each modality according to the multi-modal information set, and obtaining a high-priority modality sequence; if the visual signal in the high-priority modality sequence is determined to be dominant, extracting a visual dominant feature group; cross-verifying the visual dominant feature group and the tactile data to determine information consistency after data fusion; dynamically adjusting transparency control parameters of projection layering according to the information consistency result to obtain an optimized hierarchical configuration; updating projection display content according to the optimized hierarchical configuration, performing real-time interactive feedback, and outputting final projection content.

2. The method of claim 1, wherein the process of synchronously collecting visual signals, tactile data and voice inputs in a desktop projection environment, and obtaining a multi-modal information set after data fusion comprises: collecting visual signals, tactile data and voice inputs through a camera, a sensor and a microphone respectively to obtain an initial multi-modal data set; aligning time stamps, filling in missing values, unifying dimensions and correcting synchronicity of the initial multi-modal data set to form a synchronous multi-modal data group; using a classification model to identify the correlation between the synchronous multi-modal data group, extract key features, and generate a multi-modal information set after data fusion.

3. The method of claim 1, wherein the process of calculating priority weight distribution of each modality according to the multi-modal information set, and obtaining a high-priority modality sequence comprises: extracting original features of each modality from the multi-modal information set and classifying and arranging them; quantifying the importance of each modality based on the classified and arranged modality data set according to preset evaluation criteria to obtain an initial importance score; using an attention mechanism to weight and sort the initial importance score, and filtering out high-priority modalities higher than a preset threshold to generate a high-priority modality sequence.

4. The method of claim 1, wherein the process of extracting a visual dominant feature group if the visual signal in the high-priority modality sequence is determined to be dominant comprises: determining whether the weight proportion of the visual signal in the high-priority modality sequence exceeds a preset dominant threshold; if so, using a convolutional neural network to extract features from the visual signal to construct a preliminary visual feature set; screening, correlation analyzing and grouping integrating the preliminary visual feature set to generate a visual dominant feature group.

5. The method of claim 1, wherein the process of cross-verifying the visual dominant feature group and the tactile data to determine information consistency after data fusion comprises: extracting a visual feature set and a tactile feature set from the visual dominant feature group and the tactile data respectively; analyzing the matching degree of the visual feature set and the tactile feature set by a cross-validation method to establish a preliminary correspondence relationship; based on the preliminary correspondence relationship, using a multi-modal fusion algorithm to integrate the correlation dimensions of the visual feature set and the tactile feature set to form a fusion feature matrix; calculating the correlation between each dimension in the fusion feature matrix, and if the correlation is higher than a preset threshold, determining that the information is consistent to generate a consistency determination result. ​ ​ ​ ​ 6. The method of claim 1, wherein, the process of dynamically adjusting the transparency control parameter of the projection layering according to the information consistency result to obtain the optimized hierarchical configuration comprises: performing threshold value determination on the information consistency result, and if the information consistency result is lower than a preset consistency threshold value, performing classification processing on the projection layering data; obtaining an initial value of the transparency parameter according to the classification result, and dynamically adjusting the initial value in combination with the applicable range; evaluating the matching degree of the adjusted parameter and the hierarchical configuration, and if the matching degree is higher than a preset matching threshold value, retaining the current configuration to generate the optimized hierarchical configuration.

7. The method of claim 1, wherein, the process of updating the projection display content according to the optimized hierarchical configuration, performing real-time interactive feedback, and outputting the final projection content comprises: adjusting the projection display structure based on the optimized hierarchical configuration to obtain hierarchical structure data after adjustment; rendering the hierarchical structure data after adjustment using a preset display template to generate updated projection display content.

8. The method of claim 7, wherein, the process of performing real-time interactive feedback and outputting the final projection content comprises: collecting real-time interactive data of the user on the updated projection content and performing classification and arrangement; if the interactive response state is lower than a preset response threshold value, performing hierarchical screening on the interactive data to obtain a feedback information data set; analyzing and classifying the feedback information data set to determine a depth perception priority sequence; designing an enhanced output scheme according to the depth perception priority sequence and mapping the enhanced output scheme to the projection hierarchy to output the final projection content.

9. An AI supported multi-modal tabletop interaction projection system, characterized in that, A device for implementing the method of any one of claims 1-8, comprising: a multi-modal acquisition module configured to synchronously acquire visual signals, tactile data, and voice input in a desktop projection environment, and fuse the acquired data to obtain a multi-modal information set; a priority weight calculation module configured to calculate priority weight distribution of each modality according to the multi-modal information set to obtain a high-priority modality sequence; a visual feature extraction module configured to extract a visual dominant feature group when the visual signals in the high-priority modality sequence are determined to be dominant; a cross-validation module configured to cross-validate the visual dominant feature group and the tactile data to determine information consistency after data fusion; a transparency dynamic adjustment module configured to dynamically adjust a transparency control parameter of projection layering according to the information consistency result to obtain an optimized hierarchical configuration; a projection update output module configured to update projection display content according to the optimized hierarchical configuration, perform real-time interactive feedback, and output the final projection content.

10. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 9. The processor executes the computer program to implement the steps of the method of any one of claims 1-8.

Images