An AI large model-based projection interaction method and system

Abstract

The application relates to an AI large model-based projection interaction method and system, wherein the method comprises inputting multi-modal data into an AI large model to output a current scene type; determining a user interaction data corresponding inference mode based on the user interaction data in the multi-modal data; generating response content of a projection device based on the inference mode, the current scene type and the multi-modal data through the AI large model; and presenting the response content through a presentation device. The application solves the limitations of single interaction mode and weak content processing capability in the prior art, improves the depth of interaction from instruction-response to understanding-generation, makes the content service more targeted and practical, significantly optimizes the user experience and satisfaction, and solves the problem of how to improve the depth interaction capability in the related art.

Description

Technical Field

[0001] This application relates to the field of projector interaction technology, and in particular to an intelligent projection interaction method and system that integrates a large AI model. Background Technology

[0002] Smart projection devices, as terminals integrating projection display and human-computer interaction functions, have been widely used in diverse scenarios such as education, home, office, and commerce. These devices typically possess basic voice control, gesture recognition, and image quality optimization capabilities, aiming to enhance the convenience and user experience of information display. With the rapid development of artificial intelligence technology, users are placing higher demands on the intelligence level of projection devices, expecting them to achieve deep interaction based on content understanding, multimodal information processing, and personalized services that adapt to different scenarios.

[0003] Currently, in the research and practice of smart projectors, some products have attempted to introduce artificial intelligence technology to enhance interactive functions, such as using voice assistants to achieve simple command control or leveraging computer vision to achieve basic gesture operations. However, existing smart projector interaction technologies still have significant limitations: the interaction methods are mostly a single "command-response" mode, lacking unified understanding and deep response; content processing capabilities are weak, mainly focusing on image quality adjustment, making it difficult to perform real-time analysis, expansion, or personalized generation of projected content; in different application scenarios, devices usually rely on manual settings to switch modes, lacking automatic adaptation capabilities based on environment and content perception; in addition, the AI ​​algorithms integrated into existing systems are often limited in scale and knowledge coverage, resulting in low accuracy of intelligent decision-making and poor functional scalability, making it difficult to meet users' growing demand for proactive and generative intelligent interaction.

[0004] Therefore, no effective solution has yet been proposed for improving the deep interaction capabilities of projection interaction methods based on large AI models. Summary of the Invention

[0005] This application provides a projection interaction method and system based on an AI large model, which at least solves the problem of how to improve deep interaction capabilities in related technologies.

[0006] In a first aspect, embodiments of this application provide a projection interaction method based on an AI large model, the method being applied to a projection device, the method comprising: Input multimodal data into the large AI model and output the current scene type; Based on the user interaction data in the multimodal data, determine the inference mode corresponding to the user interaction data; Based on the inference mode, the current scene type, and the multimodal data, the AI ​​big model generates the response content for the projection device, and the response content is presented through the presentation device.

[0007] In one embodiment, before inputting multimodal data into the AI ​​large model and outputting the current scene type, the method further includes: Simultaneously collect projection content data, user interaction data, environmental data, and device status data; The synchronously collected data is standardized to obtain multimodal data.

[0008] In one embodiment, the standardization process of the synchronously acquired data to obtain multimodal data includes: The voice commands in the user interaction data are converted into text data using a speech recognition algorithm; The user interaction data is identified using computer vision algorithms, and the corresponding action feature data is extracted. The image in the projected content data is subjected to denoising and sharpening processing to obtain the processed image data; The text data, motion feature data, image data, environmental data, and device status data are uniformly converted into a preset standardized data format; Abnormal data in the standardized data format is filtered out to obtain multimodal data.

[0009] In one embodiment, the step of inputting multimodal data into the AI ​​large model and outputting the current scene type includes: Pre-standardized environmental data and user interaction data are obtained from the multimodal data; The environmental data and the user interaction data are input into the AI ​​large model after scene fine-tuning; The scene recognition unit in the AI ​​model identifies the current scene type by matching the features of the environmental data and the user interaction data.

[0010] In one embodiment, the scene-fine-tuned AI large model includes: The model adaptation unit is used to fine-tune the basic multimodal large model using scene-specific datasets, wherein the basic multimodal large model includes Tongyi Qianwen and Doubao multimodal version; The inference computing unit is used to generate corresponding response content based on the type of interaction request, using either local inference or cloud-based collaborative inference modes. The scene recognition unit is used to identify the current scene type based on the environmental data and the user interaction data; The feedback optimization unit is used to collect user feedback data on the response content and to iteratively optimize the multimodal large model based on the feedback data.

[0011] In one embodiment, determining the inference mode corresponding to the user interaction data based on the user interaction data in the multimodal data includes: The type of interaction request is determined based on the user interaction data. If the interaction request is determined to be a device control command, then the local inference mode is adopted. If the interaction request is determined to be a content generation request, then the inference mode for cloud collaboration is determined.

[0012] In one embodiment, generating the response content of the projection device based on the inference mode, the current scene type, and the multimodal data using the AI ​​big model includes: If the local inference mode is adopted, the device control command corresponding to the user interaction data is generated through the local model as the response content; If a cloud-based collaborative reasoning mode is adopted, the user interaction data is initially processed through the local model to obtain key data. The key data, the current scene type, and the related multimodal data are then uploaded to the cloud-based large model for deep reasoning to generate response content corresponding to the interaction request.

[0013] In one embodiment, the key data, the current scene type, and related multimodal data are uploaded to the cloud-based large model for deep inference to generate response content corresponding to the interaction request, including: Based on the current scenario type, contextual analysis is performed on the key data and related multimodal data to obtain the deep semantic intent of the interaction request; Based on the deep semantic intent, structured information and background knowledge are extracted from the knowledge graph associated with the scene type; Based on the structured information and the background knowledge, and according to the generation logic of matching the scene type, response content including text, images and / or animations is generated. The format and content of the response content are adaptively adjusted according to the scenario type: when the scenario type is an educational scenario, explanation content including step-by-step principle explanation and dynamic demonstration is generated; when the scenario type is an office scenario, structured meeting minutes including key issues, resolutions and time nodes are generated.

[0014] In one embodiment, after generating the response content of the projection device using the AI ​​large model, the method further includes: Collect user feedback information on the response content, including positive feedback information and negative feedback information; wherein, the positive feedback information includes positive evaluation instructions or content correction operations given by the user through gestures, voice or touch, and the negative feedback information includes the user switching the response content within a preset time period; For negative feedback information, adjust the model parameters or feature weights in the AI ​​large model; for positive feedback information, strengthen the corresponding response logic.

[0015] Secondly, embodiments of this application provide a projection interaction system based on an AI large model, the system being used to execute the method described above, the system comprising: Multimodal data acquisition equipment is used to acquire data from multiple modules; A projection device is used to input the multimodal data into an AI large model and output the current scene type; based on the user interaction data in the multimodal data, determine the inference mode corresponding to the user interaction data; and based on the inference mode, the current scene type, and the multimodal data, generate the response content of the projection device through the AI ​​large model. A presentation device for presenting the response content.

[0016] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a projection interaction method based on an AI large model as described in the first aspect above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a projection interaction method based on an AI large model as described in the first aspect above.

[0018] The projection interaction method and system based on AI large model provided in this application embodiment have at least the following technical effects.

[0019] By inputting multimodal data into a large AI model and outputting the current scene type, the system can automatically and accurately identify the current application scenario (such as education, home entertainment, and office) without requiring users to manually switch modes. This solves the problems of insufficient scene adaptability and the need for manual adjustment in existing technologies, significantly improving user convenience and intelligence, while maintaining high scene recognition accuracy. Based on user interaction data in the multimodal data, the system determines the inference mode corresponding to the user interaction data. While ensuring low latency for simple interactions, it provides powerful capabilities for handling complex interactions and generative tasks (such as Q&A and courseware generation), effectively solving the problems of low AI integration, slow response, and weak functionality in existing technologies. Based on the inference mode, current scene type, and multimodal data, the system generates response content for the projection device through the large AI model and presents the response content through the presentation device. This overcomes the limitations of existing technologies in terms of single interaction methods and weak content processing capabilities, elevating interaction from command-response to understanding-generation depth, making content services more targeted and practical, thereby significantly optimizing user experience and satisfaction. This addresses the question of how to improve deep interaction capabilities in related technologies.

[0020] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a projection interaction method based on an AI large model according to an exemplary embodiment; Figure 2 This is a flowchart illustrating step S101 according to an exemplary embodiment; Figure 3 This is a flowchart illustrating step S102 according to an exemplary embodiment; Figure 4 This is a flowchart illustrating step S103 according to an exemplary embodiment; Figure 5 This is a system structure block diagram of a projection interaction system based on an AI large model, according to an exemplary embodiment. Figure 6 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0023] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0024] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0025] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0026] In this document, it should be understood that the terms used may be technical means used to implement part of the present invention or other summary technical terms. For example, the terms may include: Multimodal large-scale AI model: refers to an artificial intelligence model that has been fine-tuned with a scenario-based dataset and is capable of simultaneously processing and fusing understanding of multiple types of data (modalities) such as text, images, audio, and gestures. In this application, it specifically refers to the core algorithm engine, such as GPT and Wenxin Yiyan, after optimization, which possesses comprehensive capabilities of "content understanding - intent recognition - decision generation", including model adaptation unit, inference computing unit, scene recognition unit, and feedback optimization unit.

[0027] Cloud-based collaborative inference mode: The inference strategy dynamically allocates computational tasks based on the type or complexity of the interaction request. Specifically, simple device control requests are handled in real-time by a lightweight model deployed locally to ensure low latency; complex cognitive and generative requests (such as content Q&A and minutes generation) are uploaded to a more powerful cloud-based model for deep computation, and the results are returned to the local machine for execution.

[0028] Deep semantic intent refers to the true needs and application goals expressed by users through interactive commands (such as voice and gestures), which go beyond the surface words or actions. Its interpretation requires comprehensive inference based on the current scene, the context of the projected content, and the user's historical behavior.

[0029] Firstly, embodiments of this application provide a projection interaction method based on a large AI model. Figure 1 This is a flowchart of a projection interaction method based on a large AI model, such as... Figure 1 As shown, the method includes: Step S101: Input multimodal data into the AI ​​large model and output the current scene type.

[0030] Step S102: Based on the user interaction data in the multimodal data, determine the inference mode corresponding to the user interaction data.

[0031] Step S103: Based on the reasoning mode, the current scene type, and multimodal data, generate the response content for the projection device through the AI ​​big model, and present the response content through the presentation device.

[0032] In summary, this application provides a projection interaction method based on an AI large model. By inputting multimodal data into the AI ​​large model and outputting the current scene type, it can automatically and accurately identify the current application scene (such as education, home entertainment, and office) without requiring the user to manually switch modes. This solves the problems of insufficient scene adaptability and the need for manual adjustment in existing technologies, significantly improving user convenience and intelligence, and achieving high scene recognition accuracy. Based on user interaction data in the multimodal data, the inference mode corresponding to the user interaction data is determined. While ensuring low latency for simple interactions, it provides powerful capabilities for handling complex interactions and generative tasks (such as Q&A and courseware generation), effectively solving the problems of low AI integration, slow response, and weak functionality in existing technologies. Based on the inference mode, the current scene type, and multimodal data, the AI ​​large model generates response content for the projection device, which is then presented through the presentation device. This overcomes the limitations of existing technologies in terms of single interaction methods and weak content processing capabilities, elevating interaction from command-response to understanding-generation depth, making content services more targeted and practical, thereby significantly optimizing user experience and satisfaction, and solving the problem of how to improve deep interaction capabilities in related technologies.

[0033] In one embodiment, before step S101, inputting multimodal data into the AI ​​large model and outputting the current scene type, the method further includes the following steps: Simultaneously collect projection content data, user interaction data, environmental data, and device status data.

[0034] Optionally, the system can simultaneously collect projection content data (obtain the original content of the current projection through the internal interface of the projector host), user interaction data (capture gestures with the camera, collect voice with the microphone, and obtain touch operations with the touch module), environmental data (collect parameters such as brightness and noise with environmental sensors), and device status data (report brightness, temperature, and other statuses in real time by the projector host).

[0035] The synchronously acquired data is standardized to obtain multimodal data. Specifically, this includes: Voice commands in user interaction data are converted into text data using a speech recognition algorithm; The computer vision algorithm is used to identify gestures in user interaction data and extract corresponding action feature data. The image in the projected content data is denoised and sharpened to obtain the processed image data. Text data, motion feature data, image data, environmental data, and device status data are uniformly converted into a preset standardized data format; Filter out outlier data in the standardized data format to obtain multimodal data.

[0036] Optionally, the speech stream is converted into text in real time using ASR technology; the gesture data is used to extract feature vectors using computer vision algorithms; the projected image is denoised and its resolution is standardized; all processed data (including directly usable environmental and device status data) is encapsulated into a unified JSON structure, and outliers such as briefly triggered gestures and environmental noise fragments are removed. The final output is a set of time-aligned, formatted multimodal data packets for subsequent model inference.

[0037] It should be noted that the processed data is divided into two parts: real-time interactive data is cached in the local storage module for immediate response; historical data is periodically uploaded to cloud storage for model fine-tuning.

[0038] These features ensure that raw data from different sources and in different formats are transformed into high-quality, structured model input, fundamentally solving the problem of data-induced model comprehension bias. Secondly, they guarantee the consistency of context between different modalities. For example, when a user says "explain this," the audio clip is automatically associated with the screen coordinates pointed to by the user's gesture within the same time window, as well as the corresponding projected image content, thus clearly identifying the referent and enabling subsequent multimodal large-scale models to perform accurate fusion analysis and intent understanding. Furthermore, they achieve synchronous acquisition and standardized processing of multi-source data such as text, images, audio, gestures, and environmental parameters, providing high-quality input data for large-scale AI models.

[0039] In one embodiment, Figure 2 This is a flowchart illustrating step S101 according to an exemplary embodiment, as follows: Figure 2 As shown, step S101 involves inputting multimodal data into the AI ​​large model and outputting the current scene type. Specifically, this includes the following steps: Step S1011: Obtain pre-standardized environmental data and user interaction data from the multimodal data.

[0040] Step S1012: Input environmental data and user interaction data into the AI ​​large model after scene fine-tuning; Step S1013: The scene recognition unit in the AI ​​big model matches the features of environmental data and user interaction data to identify the current scene type.

[0041] Optionally, during scene recognition, key environmental and user interaction data are first extracted from the pre-processed and standardized multimodal data packets. Environmental data includes quantified brightness values ​​and noise levels in decibels, while user interaction data includes converted speech-text, recognized gesture classifications, and text or image features of the projected content. This data is input in real-time into a large AI model fine-tuned using scenario-specific data (such as datasets for education, home, and office scenarios). The scene recognition unit within the model then initiates a multi-dimensional feature matching analysis process to construct a temporary "scene feature vector." For example, it comprehensively evaluates features such as "moderate and stable ambient brightness," "ambient noise below the classroom threshold," "formulas or courseware structures appearing on the projected image," "real-time speech-text containing teaching-related vocabulary such as explanations, questions, and steps," and "user gestures resembling pointing or writing postures." This composite feature vector is then compared with the pre-built feature libraries of various scene prototypes within the model to calculate similarity. When the overall matching degree with the "educational scene" prototype features exceeds a preset threshold (e.g., 95%), the model determines that the current scene is an educational scene and outputs the structured recognition result. The results are synchronized to various functional modules in the application layer and the scenario-based interaction strategy library, serving as the basis for subsequent decisions on selecting interaction modes, content styles, and device parameters. For example, if the ambient brightness is low (e.g., <100 lux), the noise level is ≤30dB, and the voice command contains keywords such as "on-demand," "movie," or "animation," then it is matched as a "home entertainment scenario."

[0042] Step S101 effectively solves the problems of insufficient scene adaptability and reliance on manual user switching in existing technologies. It automatically adapts to the needs of different scenarios such as education, home, and office, eliminating the need for manual adjustments and improving user convenience. The scene recognition accuracy is ≥95%. Accurate scene recognition lays the correct contextual foundation for all subsequent interactions, enabling the system to call the most suitable interaction strategy library. This ensures that a series of responses, from content generation (e.g., generating courseware in educational scenarios), interaction methods (e.g., enabling voice-activated playback in home scenarios), to device parameters (e.g., automatically adjusting brightness based on ambient light), are highly matched to the user's current real needs and the scene atmosphere, improving personalized services.

[0043] In one embodiment, the scene-fine-tuned AI large model includes: The model adaptation unit is used to fine-tune the basic multimodal large model using scenario-specific datasets. The basic multimodal large model includes Tongyi Qianwen and Doubao multimodal version. The inference computing unit is used to generate corresponding response content based on the type of interaction request, using either local inference or cloud-based collaborative inference modes. The scene recognition unit is used to identify the current scene type based on environmental data and user interaction data; The feedback optimization unit is used to collect user feedback data on the response content and to iteratively optimize the multimodal large model based on the feedback data.

[0044] Optionally, the core intelligence of the system is implemented by four collaborative units within the AI ​​large model layer. First, the model adaptation unit performs specialized optimizations on the general-purpose basic multimodal large model (such as Tongyi Qianwen and Doubao multimodal version). By injecting and learning specific datasets for particular scenarios (such as education and family entertainment) into the basic model (e.g., an educational corpus containing courseware and teaching dialogues, or family entertainment data containing film and television preferences and children's content), the internal parameters of the model are adjusted to improve its intent understanding and content generation accuracy in the corresponding scenarios. Second, the inference and computation unit, as the scheduling and computation hub, makes dynamic decisions based on the type of real-time interaction request (such as device control commands and content generation requests) or semantic complexity (such as the simple command "increase brightness" or the complex request "generate meeting minutes"). Device control commands invoke the locally deployed lightweight model for immediate parsing and response (latency ≤ 50ms); for content generation requests, the local model performs preliminary preprocessing, uploads key data to the cloud, utilizes the powerful generation capabilities of the full cloud model for deep inference, and executes the results after they are returned. Meanwhile, the scene recognition unit continuously analyzes the input environmental data (brightness, noise) and user interaction data (voice keywords, projected content features), and determines the current scene type in real time based on preset feature matching rules (e.g., "moderate brightness + noise ≤ 40dB + voice containing 'solving problems'" matches an educational scene). Finally, the feedback optimization unit is responsible for collecting positive feedback from users on the system response (e.g., "satisfied" ratings, manual corrections) and negative feedback (e.g., content viewing time), associating this data with the original interaction records, and periodically using it to fine-tune the multimodal large model, thus forming a closed loop of continuous optimization.

[0045] By introducing a multimodal large-scale AI model that has been fine-tuned for specific scenarios, the system constructs the core logic of "content understanding - intent recognition - decision generation," supporting complex requirement parsing and intelligent response generation. This solves the fundamental problems of low AI integration and lack of intelligent decision-making in existing technologies. Specifically, through specialized fine-tuning of the model adaptation unit, the system gains the ability to delve into specific domain knowledge, making interactions more than just superficial. The inference and computation unit, while ensuring low latency, is endowed with powerful capabilities to handle complex interactions and generative tasks, effectively solving the problems of low AI integration, slow response, and weak functionality in existing technologies. The feedback optimization unit establishes an interaction-feedback-optimization closed loop, enabling the system to continuously iterate and improve with user usage, resulting in continuous growth in response accuracy and user satisfaction.

[0046] In one embodiment, Figure 3 This is a flowchart illustrating step S102 according to an exemplary embodiment, as follows: Figure 3 As shown, step S102 involves determining the inference mode corresponding to the user interaction data based on the user interaction data in the multimodal data. This specifically includes the following steps: Step S1021: Determine the type of interaction request based on user interaction data; Step S1022: If the interaction request is determined to be a device control command, then the local inference mode is adopted. Step S1023: If the interaction request is determined to be a content generation request, then the inference mode of cloud collaboration is determined.

[0047] Optionally, the preprocessed user interaction data is first analyzed in real-time, and the request type is determined according to predefined rules and classification models. If the interaction request is a direct device control command (such as "increase brightness" or "pause projection"), the intent is clear and requires no complex understanding, and the system determines to use the local inference mode. In this mode, the locally deployed lightweight model will directly parse the command, generate the corresponding device control signal, and immediately send it to the hardware layer for execution, thereby achieving low-latency control of devices such as projectors and speakers. Conversely, if the interaction request is determined to be a content generation request (such as "explain this physics experiment" or "generate meeting minutes"), such requests involve understanding multimodal content, knowledge retrieval, and creative generation, and the system determines to use the cloud-based collaborative inference mode. In this mode, the local model first performs preliminary processing and key information extraction on the request and its associated context (such as the experimental images projected or the audio recordings of the meeting), and then uploads this structured data to the cloud server. The multimodal large model deployed in the cloud completes the deep understanding, knowledge fusion, and content generation tasks, and the generated results (such as explanation animations or structured minutes) are then sent back to the local system for output.

[0048] Step S102 addresses the problem that existing single computing architectures cannot simultaneously achieve real-time response and deep intelligence. It achieves synergistic optimization of response efficiency and processing power. For high-frequency, simple device control needs, the local inference mode ensures millisecond-level (≤50ms) instantaneous feedback, guaranteeing the smoothness of basic interactions. For complex content generation and deep question-answering needs, the cloud-based collaborative inference mode fully leverages the advantages of large-scale models in knowledge breadth, logical reasoning, and generation capabilities, enabling knowledge services and content creation.

[0049] In one embodiment, Figure 4 This is a flowchart illustrating step S103 according to an exemplary embodiment, as follows: Figure 4 As shown, step S103 involves generating response content for the projection device using an AI large model based on the inference mode, current scene type, and multimodal data, and then presenting the response content through the presentation device. Specifically, this includes the following steps: Step S1031: If the local inference mode is adopted, the device control command corresponding to the user interaction data is generated through the local model as the response content. Step S1032: If a cloud-based collaborative reasoning mode is adopted, the user interaction data is initially processed through the local model to obtain key data. This key data, the current scenario type, and related multimodal data are then uploaded to the cloud-based large model for deep reasoning to generate response content corresponding to the interaction request. Specifically, this includes: Based on the current scenario type, contextual analysis is performed on key data and related multimodal data to obtain the deep semantic intent of the interaction request; Based on deep semantic intent, structured information and background knowledge are extracted from the knowledge graph associated with scene types; Based on structured information and background knowledge, and according to the generation logic of scene type matching, response content including text, images and / or animations is generated. The format and content of the response content are adaptively adjusted according to the scenario type: when the scenario type is an educational scenario, it generates explanation content including step-by-step explanation of principles and dynamic demonstrations; when the scenario type is an office scenario, it generates structured meeting minutes containing key issues, resolutions and time nodes.

[0050] Optionally, if the local inference mode is adopted, the locally deployed lightweight model will directly parse the user interaction data and map it to generate corresponding specific control commands that can directly drive the hardware layer (such as the projector optical engine and speaker module). These commands themselves serve as the final response content. If the cloud-based collaborative inference mode is adopted (corresponding to the content generation request): First, the local model preprocesses the original interaction request and related multimodal data, extracts key data (such as experimental image fragments referred to by the user, continuous audio text of the meeting), and uploads it to the cloud along with the identified current scene type and other relevant contextual data. After receiving the data, the cloud-based multimodal large model initiates deep inference, combines the scene type (such as "education") to perform contextual analysis on the key data, and interprets the user's deeper semantic intent beyond the literal meaning (such as "need to understand the experimental principle and visualize the process"). Then, based on this intent, the model accesses the domain knowledge graph strongly associated with the scene type and extracts structured principles, facts, or process information as generation material. Finally, based on pre-set generation logic templates for different scenarios (e.g., the "principle explanation - step-by-step demonstration - summary" template for educational scenarios, and the "issue - resolution - responsible person - time node" structured template for office scenarios), the model integrates and creates media response content including text, diagrams, animation sequences, or combinations thereof. The generated content is downloaded locally, and the corresponding module is called according to the scenario type, and finally output through the presentation device to complete the interaction. The presentation device includes a screen (for image and video display), an audio / speaker module (for audio output), and an auxiliary touch screen / status indicator (for interaction and status prompts), realizing multimodal output of system content.

[0051] Step S103 addresses the limitations of existing technologies in terms of their singular interaction methods and weak content processing capabilities, elevating the interaction from command-response to a deeper level of understanding-generation, making content services more targeted and practical, thereby significantly optimizing user experience and satisfaction.

[0052] In one embodiment, after generating the response content of the projection device using a large AI model, the method further includes: Collect user feedback on the response content. The feedback information includes positive feedback and negative feedback. Positive feedback includes positive evaluation instructions or content correction operations given by users through gestures, voice or touch. Negative feedback includes users switching the response content within a preset time. For negative feedback, adjust the model parameters or feature weights in the large AI model; for positive feedback, strengthen the corresponding response logic.

[0053] Optionally, after each interactive response, the user's subsequent behavior is continuously monitored through multimodal acquisition devices to collect feedback information. This information is clearly categorized into positive and negative feedback. Positive feedback includes user approval expressed through explicit gestures (such as a thumbs-up), affirmative verbal statements (such as "good"), or touch operations (such as clicking the "satisfied" virtual button), as well as constructive actions such as manual corrections and additions to system-generated content. Negative feedback mainly refers to the user's behavior of quickly switching to or closing the current response content without in-depth interaction within a preset short period (such as 3 seconds). The collected feedback data, user interaction data, and response content are uploaded to the feedback optimization unit of the AI ​​large model layer. This unit periodically (e.g., weekly) summarizes and analyzes the accumulated feedback data and historical interaction data, and uses this data to perform targeted fine-tuning of the multimodal large model. For cases associated with negative feedback, the system adjusts the feature weights or model parameters related to the intent parsing stage in the model to correct the judgment logic that leads to inaccurate responses. For cases associated with high-frequency positive feedback, the system strengthens the response logic path or knowledge association strength related to generating the successful response in the model, thereby increasing the probability of generating the same high-quality content in similar subsequent scenarios. This achieves continuous improvement in the accuracy of response content and forms a closed loop of "interaction-feedback-optimization".

[0054] It should be noted that the following are specific practical application examples: The first step is to start the system. The environmental sensor collects data showing "ambient brightness 100 lux and noise 30 dB". The user says in voice, "Project an animation suitable for a 7-year-old child". The microphone collects the voice and converts it into text. The second step is for the scene recognition unit to identify the scene as a "home entertainment scene" and load the home entertainment module. The third step involves the reasoning and calculation unit uploading the keywords "7-year-old children" and "animation" to the cloud-based big model. The big model then combines the children's age-level classification standards to recommend "Peppa Pig" and obtain playback resources. The fourth step involves the home entertainment module controlling the projector to project animations, while automatically adjusting the projection brightness to 80% based on the ambient light, and the audio module adjusting the volume to a suitable level. Fifth step: When the user gestures "pause" during viewing, the camera captures the gesture and converts it into a control command, and the system pauses playback; when the user gestures "continue" again, playback resumes, with a response latency of ≤50ms throughout the process.

[0055] In summary, this application provides a projection interaction method based on an AI large model. By inputting multimodal data into the AI ​​large model and outputting the current scene type, it can automatically and accurately identify the current application scene (such as education, home entertainment, and office) without requiring the user to manually switch modes. This solves the problems of insufficient scene adaptability and the need for manual adjustment in existing technologies, significantly improving user convenience and intelligence, and achieving high scene recognition accuracy. Based on user interaction data in the multimodal data, the inference mode corresponding to the user interaction data is determined. While ensuring low latency for simple interactions, it provides powerful capabilities for handling complex interactions and generative tasks (such as Q&A and courseware generation), effectively solving the problems of low AI integration, slow response, and weak functionality in existing technologies. Based on the inference mode, the current scene type, and multimodal data, the AI ​​large model generates response content for the projection device, which is then presented through the presentation device. This overcomes the limitations of existing technologies in terms of single interaction methods and weak content processing capabilities, elevating interaction from command-response to understanding-generation depth, making content services more targeted and practical, thereby significantly optimizing user experience and satisfaction, and solving the problem of how to improve deep interaction capabilities in related technologies.

[0056] Secondly, embodiments of this application provide a projection interaction system based on a large AI model. Figure 5 This is a system architecture block diagram illustrating a projection interaction system based on a large AI model, according to an exemplary embodiment. For example... Figure 5 As shown, the system includes: The multimodal acquisition device 510 is used to acquire data from multiple modules; Projection device 520 is used to input multimodal data into the AI ​​big model and output the current scene type; based on user interaction data in the multimodal data, it determines the inference mode corresponding to the user interaction data; based on the inference mode, the current scene type and multimodal data, it generates the response content of the projection device through the AI ​​big model; Presentation device 530 is used to present the response content.

[0057] In summary, the projection interaction system based on an AI large model provided in this application, through the multimodal acquisition device 510, projection device 520 and presentation device 530, solves the limitations of the existing technology in terms of single interaction mode and weak content processing capability, and elevates the interaction from command-response to the depth of understanding-generation, making the content service more targeted and practical, thereby significantly optimizing the user experience and satisfaction.

[0058] In one embodiment, the system includes a hardware layer, a data layer, an AI large model layer, and an application layer, wherein: The hardware layer provides the data acquisition and execution platform for the system, specifically including: Projection device: integrates a projection optical engine module (supports 1080P and above resolution, autofocus, keystone correction), a central processing module (CPU+GPU+NPU heterogeneous computing architecture to meet the needs of large model inference), a storage module (supports at least 512GB of high-speed storage for caching data and models), and a communication module (supports 5G / Wi-Fi 6 / Bluetooth 5.3 to achieve data transmission and device interconnection). Multimodal acquisition equipment includes a high-definition camera (for gesture recognition, face detection, and image acquisition), an array microphone (for voice recognition and positioning), a touch sensing module (supporting touch interaction on the projected screen), and an environmental sensor (for acquiring environmental parameters such as brightness, temperature, and noise). Presentation devices include a projection screen (supporting anti-glare properties), an audio module (supporting 3D surround sound), and auxiliary display devices (such as a small touch panel for quick operation).

[0059] The data layer is responsible for the acquisition, preprocessing, storage, and management of multi-source data, including: Data acquisition unit: used to simultaneously collect four types of data: projection content data (such as raw data of documents, pictures, and videos), user interaction data (voice commands, gestures, touch operations, and text input), environmental data (ambient brightness, noise level, and scene space size), and device status data (projection brightness, temperature, and battery level). Data preprocessing unit: used to standardize the collected data, including speech-to-text (using ASR technology, with a recognition accuracy of ≥98%), gesture recognition (extracting action features based on computer vision algorithms), image enhancement (denoising and sharpening), data format conversion (unifying to JSON format), and abnormal data filtering (removing erroneous and interfering data). Data storage unit: Used in a "local cache + cloud backup" mode, local storage of real-time interactive data and commonly used models, cloud storage of historical data and large model training data, supporting data encryption and access control.

[0060] The AI ​​large model layer is the core of the system's intelligence, including a multimodal large model that has been fine-tuned for specific scenarios and supporting modules, specifically including: Model adaptation unit: used to fine-tune the basic multimodal large model (such as Tongyi Qianwen, Doubao multimodal version) through special datasets (such as entertainment content preference data) in education, family and other scenarios, to optimize the accuracy of intent recognition and response generation in the scenario; Inference computing unit: Used in the "local inference + cloud collaboration" mode, simple interactions (such as adjusting brightness) are quickly responded to by the local lightweight model (response latency ≤50ms), while complex requirements (such as generating courseware and in-depth Q&A) are inferred by the large model in the cloud, and the results are received locally and then executed. Scene recognition unit: Used to identify scene types based on environmental data and user interaction data through a large model (such as identifying educational scenes through features like "blackboard, students, and teaching documents"), with an accuracy rate of ≥95%; Feedback optimization unit: Used to collect user feedback on response results (such as "satisfied / unsatisfied", manual correction content), and periodically upload the data to the cloud for model fine-tuning to improve the accuracy of subsequent responses.

[0061] The application layer provides contextualized interactive features, adapting to different scenarios through modular design, specifically including: The interactive education module supports Q&A sessions on projected content (e.g., when a student asks "How is this mathematical formula derived?", the system interprets and projects the derivation process in real time), personalized courseware generation (teachers input topics, and the system generates courseware containing examples and animations), and learning feedback (students answer questions via gestures, and the system automatically grades and generates analysis of incorrect answers). Family entertainment module: used to support voice-activated content (such as "project a suitable animation for children", the system recommends and projects according to the child's age), interactive games (control the projected game characters through gestures), and content expansion (when watching a movie, ask "the protagonist's backstory" by voice, and the system projects relevant information); Office collaboration module: used to support intelligent parsing of projected documents (extracting key information to generate outlines), real-time meeting minutes (speech-to-text and summarizing meeting key points), and multi-terminal interaction (scanning QR codes with mobile phones to annotate projected content, which is then displayed synchronously in the system). Commercial Display Module: Used to support user gesture interaction to browse product projection content (such as zooming in to view product details), personalized recommendations (recommending relevant product information based on user dwell time), and intelligent explanation (voice inquiry about product functions, with system projection of explanation animations).

[0062] It should be noted that the hardware selection in this application can be adjusted according to the needs of the scenario: for example, in a home scenario, a portable projector with a brightness of ≥2000 ANSI lumens can be selected, along with a 5-megapixel camera and a 4-array microphone.

[0063] Fine-tuning datasets for large AI models can be obtained through the following methods: Educational scenario datasets include primary and secondary school teaching question banks, experimental videos, and teacher teaching recordings; Family scenario datasets include film and television recommendation data, children's content rating standards, and family interactive voice samples; Office scenario datasets include meeting minutes samples, document parsing cases, and collaborative interaction logs.

[0064] The system supports both "single-machine deployment" and "multi-machine collaborative deployment": single-machine deployment is suitable for scenarios such as homes and small classrooms, where all modules are integrated into one projector host; multi-machine collaborative deployment is suitable for scenarios such as large conference rooms and commercial exhibition halls, where multiple projector hosts are networked together and uniformly scheduled by a large cloud model to achieve synchronous interaction across multiple screens.

[0065] In summary, a projection interaction system based on a large AI model has the following beneficial effects: By integrating multimodal data collection with large AI models, it enables multi-dimensional interaction such as voice, gestures, text, and images, and can respond to complex needs (such as answering questions and generating content) based on content understanding, breaking through the limitations of the shallow "command-response" interaction of existing devices, with an interaction accuracy rate of ≥95%. By using scene recognition and dynamic strategy matching, it automatically adapts to the needs of different scenarios such as education, home, and office, without the need for manual adjustments, improving user convenience, and achieving a scene recognition accuracy rate of ≥95%. Based on the generative capabilities of large models, the projected content can be interpreted, expanded, and personalized in real time, such as generating courseware in educational scenarios and generating meeting minutes in office scenarios, thereby improving the relevance and practicality of content services. It adopts a layered architecture of "hardware layer - data layer - AI large model layer - application layer", which supports large model iteration (such as replacing it with a more advanced multimodal model) and the addition of application modules (such as adding a medical display module) to adapt to future scenario needs; The local + cloud collaborative inference mode balances response speed and processing power, with simple interaction latency ≤50ms and complex interaction latency ≤1s. Furthermore, it continuously optimizes response accuracy through feedback loop, resulting in a user satisfaction improvement of ≥40%.

[0066] It should be noted that the projection interaction system based on an AI large model provided in this embodiment is used to implement the above-described implementation methods, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0067] Thirdly, embodiments of this application provide an electronic device, Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 6 As shown, the electronic device may include a processor 61 and a memory 62 storing computer program instructions.

[0068] Specifically, the processor 61 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0069] The memory 62 may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include removable or non-removable (or fixed) media. Where appropriate, the memory 62 may be internal or external to a data processing device. In a particular embodiment, the memory 62 is non-volatile memory. In a particular embodiment, the memory 62 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0070] The memory 62 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 61.

[0071] The processor 61 reads and executes computer program instructions stored in the memory 62 to implement any of the AI-based large model projection interaction methods in the above embodiments.

[0072] In one embodiment, a projection interaction device based on an AI large model may further include a communication interface 63 and a bus 60. Wherein, as Figure 6 As shown, the processor 61, memory 62, and communication interface 63 are connected through bus 60 and complete communication with each other.

[0073] The communication interface 63 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 63 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0074] Bus 60 includes hardware, software, or both, that couples together components of a projection interaction device based on an AI large model. Bus 60 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, and Local Bus. For example, and not as a limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 60 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0075] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the projection interaction method based on an AI large model provided in the first aspect.

[0076] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0077] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to perform steps implementing the projection interaction method based on an AI large model provided in the first aspect.

[0078] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A projection interaction method based on a large AI model, characterized in that, The method is applied to a projection device, and the method includes: Input multimodal data into the large AI model and output the current scene type; Based on the user interaction data in the multimodal data, determine the inference mode corresponding to the user interaction data; Based on the inference mode, the current scene type, and the multimodal data, the AI ​​big model generates the response content for the projection device, and the response content is presented through the presentation device.

2. The projection interaction method based on a large AI model according to claim 1, characterized in that, The process of generating the response content of the projection device based on the inference mode, the current scene type, and the multimodal data using the AI ​​big model includes: If the local inference mode is adopted, the device control command corresponding to the user interaction data is generated through the local model as the response content; If a cloud-based collaborative reasoning mode is adopted, the user interaction data is initially processed through the local model to obtain key data. The key data, the current scene type, and the related multimodal data are then uploaded to the cloud-based large model for deep reasoning to generate response content corresponding to the interaction request.

3. The projection interaction method based on a large AI model according to claim 2, characterized in that, The key data, the current scene type, and related multimodal data are uploaded to the cloud-based large model for deep inference to generate response content corresponding to the interaction request, including: Based on the current scenario type, contextual analysis is performed on the key data and related multimodal data to obtain the deep semantic intent of the interaction request; Based on the deep semantic intent, structured information and background knowledge are extracted from the knowledge graph associated with the scene type; Based on the structured information and the background knowledge, and according to the generation logic of matching the scene type, response content including text, images and / or animations is generated. The format and content of the response content are adaptively adjusted according to the scenario type: when the scenario type is an educational scenario, explanation content including step-by-step principle explanation and dynamic demonstration is generated; when the scenario type is an office scenario, structured meeting minutes including key issues, resolutions and time nodes are generated.

4. The projection interaction method based on a large AI model according to claim 1, characterized in that, The process of inputting multimodal data into a large AI model and outputting the current scene type includes: Pre-standardized environmental data and user interaction data are obtained from the multimodal data; The environmental data and the user interaction data are input into the AI ​​large model after scene fine-tuning; The scene recognition unit in the AI ​​model identifies the current scene type by matching the features of the environmental data and the user interaction data.

5. The projection interaction method based on a large AI model according to claim 4, characterized in that, The AI ​​model after fine-tuning for the scenario includes: The model adaptation unit is used to fine-tune the basic multimodal large model using scene-specific datasets, wherein the basic multimodal large model includes Tongyi Qianwen and Doubao multimodal version; The inference computing unit is used to generate corresponding response content based on the type of interaction request, using either local inference or cloud-based collaborative inference modes. The scene recognition unit is used to identify the current scene type based on the environmental data and the user interaction data; The feedback optimization unit is used to collect user feedback data on the response content and to iteratively optimize the basic multimodal large model based on the feedback data.

6. The projection interaction method based on a large AI model according to claim 1, characterized in that, Before inputting multimodal data into the large AI model and outputting the current scene type, the method further includes: Simultaneously collect projection content data, user interaction data, environmental data, and device status data; The synchronously collected data is standardized to obtain multimodal data.

7. The projection interaction method based on a large AI model according to claim 6, characterized in that, The standardized processing of the synchronously acquired data to obtain multimodal data includes: The voice commands in the user interaction data are converted into text data using a speech recognition algorithm; The user interaction data is identified using computer vision algorithms, and the corresponding action feature data is extracted. The image in the projected content data is subjected to denoising and sharpening processing to obtain the processed image data; The text data, motion feature data, image data, environmental data, and device status data are uniformly converted into a preset standardized data format; Abnormal data in the standardized data format is filtered out to obtain multimodal data.

8. The projection interaction method based on a large AI model according to claim 1, characterized in that, The step of determining the inference mode corresponding to the user interaction data based on the multimodal data includes: The type of interaction request is determined based on the user interaction data. If the interaction request is determined to be a device control command, then the local inference mode is adopted. If the interaction request is determined to be a content generation request, then the inference mode for cloud collaboration is determined.

9. The projection interaction method based on a large AI model according to claim 1, characterized in that, After generating the response content of the projection device using the AI ​​large model, the method further includes: Collect user feedback information on the response content, including positive feedback information and negative feedback information; wherein, the positive feedback information includes positive evaluation instructions or content correction operations given by the user through gestures, voice or touch, and the negative feedback information includes the user switching the response content within a preset time period; For negative feedback information, adjust the model parameters or feature weights in the AI ​​large model; for positive feedback information, strengthen the corresponding response logic.

10. A projection interaction system based on a large AI model, characterized in that, The system is used to perform the method according to any one of claims 1 to 9, the system comprising: Multimodal data acquisition equipment is used to acquire data from multiple modules; A projection device is used to input the multimodal data into an AI large model and output the current scene type; based on the user interaction data in the multimodal data, determine the inference mode corresponding to the user interaction data; and based on the inference mode, the current scene type, and the multimodal data, generate the response content of the projection device through the AI ​​large model. A presentation device for presenting the response content.

Images