Intelligent education projector system based on AI large model and interaction method

Abstract

The invention relates to an intelligent education projector system based on an AI large model and an interaction method. The intelligent education projector system comprises a projector body, a data acquisition unit, a core processor, a projection processing module and a voice playing module. According to the invention, the projector is integrated with AI education, so that the projector has an education function, and the teaching experience of a user is improved. According to the method, active input data such as audios, texts and pictures of a user can be collected, and the multi-modal data are coded to generate corresponding feature vectors. The vectors are input into a multi-mode large model which is trained in advance and subjected to education field data fine adjustment, so that an execution strategy aiming at the current problem is obtained. And finally, targeted teaching reply content is generated based on the strategy. According to the design, multi-modal information is fully utilized, students can be guided to solve learning problems, and the students are helped to consolidate and summarize learning contents in a targeted manner.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent projectors, and in particular to an intelligent educational projector system and an interaction method based on an AI large model. Background Art

[0002] Existing educational products (such as learning tablets and software) lack integration with the physical space of teaching environments (such as classrooms and home tutoring). Due to their technical principles, tablets or TVs shine light directly into students' eyes, and their relatively small screens can damage their eyesight and make them unsuitable for long-term use. Furthermore, these educational products are limited to text interaction and cannot fully utilize multimodal information such as audio and images, thus limiting the richness and efficiency of interaction. While traditional projectors can project images onto a screen or wall, minimizing the impact on user vision, they are limited to one-way display of image content and lack interactivity and intelligent analysis capabilities.

[0003] Based on this, it is necessary to design a projector product that can guide students to solve learning problems, help students consolidate and summarize learning content in a targeted manner, and improve students' ability to self-study and independent thinking. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to overcome the shortcomings of the existing technology and provide an intelligent educational projector system and interaction method based on AI big model.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] An intelligent educational projector system based on an AI large model, comprising:

[0007] Projector body, data acquisition unit, core processor, projection processing module and voice playback module;

[0008] The data acquisition unit includes a voice acquisition module, a picture and text acquisition module and an image acquisition module;

[0009] The voice collection module is used to collect audio data of users' questions in real time;

[0010] The image and text collection module is used to collect at least one of text data and image data for the question through a preset interactive interface;

[0011] The image acquisition module is used to capture the facial image data of the user's current learning state in real time;

[0012] The core processor is equipped with a coding module, a problem processing module, an image processing module and a result output module;

[0013] The encoding module is used to encode the audio data, text data and picture data to obtain corresponding audio feature vectors, text feature vectors and picture feature vectors, wherein the audio feature vectors, text feature vectors and picture feature vectors correspond to the same target vector space; and the audio feature vectors, text feature vectors and picture feature vectors are used as multimodal feature vectors;

[0014] The problem processing module is used to input the multimodal feature vector into a pre-trained multimodal large model to obtain an execution strategy for the problem, wherein the multimodal large model is fine-tuned based on education field data;

[0015] The image processing module is used to perform user eye movement recognition processing on the facial image data and obtain a feedback strategy for the user's current learning status;

[0016] The result output module is used to generate audio answer content and image answer content for the question based on the execution strategy, and the result output module is also used to generate audio feedback content and image feedback content for the user's current learning status based on the feedback strategy;

[0017] The projection processing module is used to receive the image reply content and the image feedback content, and project the image reply content and the image feedback content through the projection lens of the projector body;

[0018] The voice playing module is used to receive the audio response content and the audio feedback content, and play the audio response content and the audio feedback content through a speaker.

[0019] As an implementation manner, the encoding module is specifically configured to:

[0020] The text data is segmented to obtain a plurality of segmented words to be processed; the plurality of segmented words to be processed are encoded by a word embedding encoder, and mapped to the target vector space by a text projection layer to obtain the text feature vector; the image data is encoded by a CLIP visual encoder, and mapped to the target vector space by a picture projection layer to obtain the picture feature vector; the audio data is encoded by ImageBind, and mapped to the target vector space by an audio projection layer to obtain the audio feature vector.

[0021] As an implementation manner, the problem processing module is specifically configured to:

[0022] The multimodal feature vector is input into the multimodal large model to obtain initial demand information; the initial demand information is sent to the interactor to determine the retrieval strategy; the interactor searches in the memory unit based on the retrieval strategy to obtain the retrieval memory content corresponding to the multimodal feature vector; the multimodal feature vector and the retrieval memory content are input into the multimodal large model to obtain the execution strategy corresponding to the multimodal feature vector, wherein the execution strategy includes execution interface parameters.

[0023] As an implementation manner, the result output module is specifically used to:

[0024] The execution strategy is input into the scheduler, and the target education vector indicated by the execution strategy and the target display tool based on the execution interface parameter indication are obtained from the education vector database and tool library through the scheduler; the education vector database is used to store multiple school-age and multiple subject test questions and related vectors of the corresponding test questions and answers; the multimodal feature vector is input into the decoder to obtain the original data expression of the multimodal feature vector; the target education vector, target display tool and the original data expression are input into the multimodal large model to obtain the teaching response content for the question, and the teaching response content includes audio response content and image response content.

[0025] As an embodiment, the image acquisition module includes a depth camera arranged below the projector body.

[0026] As an implementation method, the picture text acquisition module includes a writing board arranged on the user's desktop.

[0027] An interactive method for an intelligent educational projector system based on an AI large model, comprising:

[0028] Collecting audio data of the user's questions in real time, and collecting at least one of text data and image data of the questions through a preset interactive interface;

[0029] Encoding the audio data, text data, and picture data to obtain corresponding audio feature vectors, text feature vectors, and picture feature vectors, wherein the audio feature vectors, text feature vectors, and picture feature vectors correspond to the same target vector space;

[0030] The audio feature vector, the text feature vector and the image feature vector are used as multimodal feature vectors;

[0031] Inputting the multimodal feature vector into a pre-trained multimodal large model to obtain an execution strategy for the problem, wherein the multimodal large model is fine-tuned based on education field data;

[0032] Capturing facial image data of the user's current learning status in real time, performing user eye movement recognition processing on the facial image data, and obtaining a feedback strategy for the user's current learning status;

[0033] generating audio answer content and image answer content for the question based on the execution strategy;

[0034] Based on the feedback strategy, generate audio feedback content and image feedback content for the user's current learning status;

[0035] Projecting the image response content and image feedback content;

[0036] The audio response content and the audio feedback content are played out.

[0037] Furthermore, encoding the audio data to obtain the audio feature vector includes encoding the audio data through ImageBind and mapping it to the target vector space through an audio projection layer to obtain the audio feature vector.

[0038] Furthermore, encoding the text data to obtain the text feature vector includes segmenting the text data to obtain a plurality of to-be-processed segmented words; encoding the plurality of to-be-processed segmented words through a word embedding encoder, and mapping them to the target vector space through a text projection layer to obtain a text content feature vector;

[0039] Furthermore, encoding the image data to obtain the image feature vector includes encoding the image data through a CLIP visual encoder and mapping it to the target vector space through an image projection layer to obtain the image feature vector.

[0040] The beneficial effects of the present invention include: integrating the projector with AI education, so that the projector has educational functions, which improves the user's teaching experience. The present invention can collect the user's active input data such as audio, text and pictures, encode these multimodal data, and generate corresponding feature vectors. These vectors are input into a multimodal large model that has been pre-trained and fine-tuned with education field data to obtain an execution strategy for the current problem. Finally, targeted teaching response content is generated based on the strategy. Such a design makes full use of multimodal information, which is conducive to guiding students to solve learning problems and helping students consolidate and summarize the learning content in a targeted manner. In addition, the user's learning status can be monitored. When it is determined that the user's learning status is not good or the attention is not focused, the corresponding feedback strategy is obtained, and the corresponding feedback content is generated based on the feedback strategy to remind the user to improve his attention.

[0041] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a communication connection diagram of the intelligent education projector system in the embodiment of the present application;

[0043] Description of reference numerals:

[0044] 11. Voice acquisition module; 12. Picture and text acquisition module; 13. Image acquisition module; 21. Encoding module; 22. Question processing module; 23. Image processing module; 24. Result output module; 31. Projection processing module; 32. Voice playback module; 311. Projection lens; 321. Speaker. DETAILED DESCRIPTION

[0045] To further illustrate various embodiments, the present invention includes accompanying drawings. These drawings form part of the present disclosure and are primarily used to illustrate the embodiments and, in conjunction with the relevant description in the specification, to explain the operating principles of the embodiments. By referring to these drawings, those skilled in the art will readily understand other possible implementations and the advantages of the present invention.

[0046] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "left", "right", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0047] See also Figure 1 This embodiment provides an intelligent educational projector system based on an AI large model, which includes: a projector body, a data acquisition unit, a core processor, a projection processing module 31, and a voice playback module 32. The projector body is provided with components such as a projection lens 311 and a speaker 321.

[0048] The data acquisition unit includes a voice acquisition module 11 , a picture and text acquisition module 12 , and an image acquisition module 13 .

[0049] The voice acquisition module 11 is used to collect audio data of the user's questions in real time. The voice acquisition module 11 can be a microphone installed on the projector body. In this embodiment, collecting audio information actively generated by the user means, for example, the user can directly verbally narrate the content that needs the system to help him complete.

[0050] The image text acquisition module 12 is used to collect at least one of the text data and image data for the question through a preset interactive interface. The image text acquisition module 12 includes a writing board set on the user's desktop. The user completes the entry of text data by handwriting the question on the writing board. The writing board also has an image loading function. The user can transfer the image to the writing board through a mobile phone or other device to complete the collection of image data. Of course, the writing board can also be a mobile device such as a tablet or a mobile phone. By recording audio data, text data and image data through a microphone and a writing board, the core processor can comprehensively collect the user's multi-modal data for the current question, including audio, text and pictures, thereby providing a rich information basis for subsequent multi-modal interaction and intelligent question and answer.

[0051] The image acquisition module 13 is used to capture the facial image data of the user's current learning state in real time. The image acquisition module 13 includes a depth camera arranged below the projector body, which is used to capture the user's facial expressions, especially the user's eye movements.

[0052] The core processor is built with a coding module 21 , a question processing module 22 , an image processing module 23 and a result output module 24 .

[0053] The encoding module 21 is used to encode the audio data, text data and picture data to obtain corresponding audio feature vectors, text feature vectors and picture feature vectors, and the audio feature vectors, text feature vectors and picture feature vectors correspond to the same target vector space; the audio feature vectors, text feature vectors and picture feature vectors are used as multimodal feature vectors.

[0054] The problem processing module 22 is used to input the multimodal feature vector into a pre-trained multimodal large model to obtain an execution strategy for the problem. The multimodal large model is obtained by fine-tuning based on education field data.

[0055] The image processing module 23 is used to perform user eye movement recognition processing on the facial image data and obtain a feedback strategy for the user's current learning status.

[0056] The result output module 24 is used to generate audio response content and image response content for the question based on the execution strategy. The result output module 24 is also used to generate audio feedback content and image feedback content for the user's current learning status based on the feedback strategy.

[0057] The projection processing module 31 is used to receive the image response content and image feedback content, and project the image response content and image feedback content through the projection lens 311 of the projector body, so that the user can directly observe the projected content, including answers to questions and reminders of their own learning status.

[0058] The voice playback module 32 is used to receive the audio answer content and the audio feedback content, and play the audio answer content and the audio feedback content through the speaker 321. Thus, the user can directly listen to the played content, including the answer to the question and the reminder of his own learning status.

[0059] In an embodiment of the present invention, the core processor illustratively receives audio data, text data, and image data related to a user's question via a microphone, tablet, or other device. For example, a middle school student asks at home, "How do I solve this quadratic equation?" Furthermore, the student uses the device's handwriting input function to write down a specific quadratic equation as text data. After receiving this raw data, the core processor begins encoding it. Encoding module 21 uses deep learning technology to convert the audio data into audio feature vectors. Encoding module 21 also performs corresponding encoding on the text and image data actively entered by the user, generating text feature vectors and image feature vectors. All these feature vectors are mapped into the same target vector space for subsequent processing and analysis. Encoding module 21 then fuses the audio, text, and image feature vectors obtained in the previous step to form a multimodal feature vector. This multimodal feature vector integrates the user's various ways of expressing the question, more comprehensively reflecting the user's problem and needs. The core processor then inputs this multimodal feature vector into a pre-trained multimodal large model through question processing module 22. This large model is fine-tuned based on a large amount of data in the field of education, so it can understand the knowledge points and common problems in the field of education. Through the deep learning and reasoning capabilities of the large model, the core processor obtains an execution strategy for the current problem through the problem processing module 22. For example, for the quadratic equation problem mentioned above, the execution strategy that the large model may give is: first explain the basic concept of the quadratic equation, then demonstrate the solution steps, and provide several similar examples for practice. Based on the execution strategy obtained in the previous step, the core processor begins to generate teaching response content for the current problem through the result output module 24. This may include text explanations, formula derivations, chart examples, etc. For the previous quadratic equation problem, the core processor may generate a detailed explanation of the solution steps, and attach several examples and exercises. These contents will be displayed to the user through the speaker 321 and the projection lens 311 to help the user understand and master the relevant knowledge points.

[0060] As can be seen from the above, the beneficial effects of the present invention include: integrating the projector with AI education, so that the projector has educational functions, which improves the user's teaching experience. The present invention can collect active input data such as audio, text, and pictures from the user, encode these multimodal data, and generate corresponding feature vectors. These vectors are input into a multimodal large model that has been pre-trained and fine-tuned with data in the education field to obtain an execution strategy for the current problem. Finally, targeted teaching response content is generated based on the strategy. Such a design makes full use of multimodal information, which is conducive to guiding students to solve learning problems and helping students consolidate and summarize the learning content in a targeted manner. In addition, the user's learning status can be monitored. When it is determined that the user's learning status is not good or the attention is not focused, the corresponding feedback strategy is obtained, and the corresponding feedback content is generated based on the feedback strategy to remind the user to improve his attention.

[0061] The encoding module 21 is specifically implemented by performing the following steps:

[0062] Segmenting the text data to obtain a plurality of to-be-processed segmented words; encoding the plurality of to-be-processed segmented words through a word embedding encoder, and mapping them to the target vector space through a text projection layer to obtain the text feature vector; encoding the image data through a CLIP visual encoder, and mapping them to the target vector space through an image projection layer to obtain the image feature vector;

[0063] Exemplarily, when a user actively inputs text data, the encoding module 21 will first perform word segmentation on the text. For example, the user inputs a mathematical problem description: "Solve the root of a quadratic equation 4x^2+7x-10=0." The encoding module 21 will use word segmentation technology to cut this sentence into multiple to-be-processed words such as "solve", "one", "quadratic equation", "4x^2", "+", "7x", "-", "10", "=", "0", "of", and "root". Next, the encoding module 21 encodes these words through a word embedding encoder. The word embedding encoder converts each word into a high-dimensional vector that can capture the semantic information of the word. Then, these high-dimensional vectors are mapped to the target vector space through the text projection layer to form text feature vectors. These feature vectors will be used for subsequent multimodal interaction processing. If the user actively inputs image data, such as uploading an image of a quadratic equation or a photo of a geometric figure, the encoding module 21 will use the CLIP visual encoder to encode these image data. The CLIP visual encoder can understand and extract visual features in images and convert these features into high-dimensional vectors. Subsequently, these high-dimensional vectors are mapped to the target vector space through the image projection layer to form image feature vectors. These feature vectors will also be used for subsequent multimodal interaction processing. Finally, the text feature vector, the image feature vector and the feature vector of other modalities (such as audio feature vectors) will be fused to support more complex multimodal interaction and teaching question-and-answer functions. For example, when the user provides a text description and a picture at the same time, the encoding module 21 will comprehensively consider these two sources of information, and through the fusion of multimodal feature vectors to more accurately understand the user's questions and needs, and generate more accurate teaching response content accordingly.

[0064] In this embodiment, the encoding module 21 may encode the audio data through ImageBind, and map it to the target vector space through an audio projection layer to obtain the audio feature vector.

[0065] For example, a user can use a microphone to record an audio clip about a mathematical problem they are confused about, or a description of a physical phenomenon. After receiving the audio data, the encoding module 21 will use ImageBind technology to encode it. ImageBind is a deep learning model that can process audio data and extract features. In this process, the audio data will be converted into a format that can be processed by the deep learning model. Taking the user's audio data as an example, suppose the user recorded an audio clip asking a question about "how to solve a quadratic equation." The encoding module 21 will input this audio data into the ImageBind model, and the model will automatically extract key information from the audio, such as the user's voice rhythm, pitch changes, and pronunciation content, and convert this information into numerical form. Next, the encoding module 21 will map the audio features extracted by the ImageBind model into the target vector space through the audio projection layer. The function of this projection layer is to convert the audio features into a vector form that is compatible with other modal data (such as text, images, etc.) to facilitate subsequent multimodal data fusion and analysis.

[0066] The problem processing module 22 is specifically used to:

[0067] The multimodal feature vector is input into the multimodal large model to obtain initial demand information; the initial demand information is sent to the interactor to determine the retrieval strategy; the interactor searches in the memory unit based on the retrieval strategy to obtain the retrieval memory content corresponding to the multimodal feature vector; the multimodal feature vector and the retrieval memory content are input into the multimodal large model to obtain the execution strategy corresponding to the multimodal feature vector, wherein the execution strategy includes execution interface parameters.

[0068] Among them, this multimodal large model has the ability to process and understand multiple modal data, and can capture the associations and complementary information between different modal data. For example, a user may describe a complex geometric problem through text, pictures, and audio. The problem processing module 22 inputs the feature vectors of these modalities into the multimodal large model, and the model will comprehensively consider this information to obtain the user's comprehensive description of the problem. After processing the input multimodal feature vectors, the multimodal large model will generate initial demand information. This information reflects the user's initial understanding and needs of the problem, and may include the type of problem, key concepts, and preliminary judgment of the solution steps.

[0069] A search strategy is a method used to locate historical data, solutions, or related knowledge related to the current problem within the core processor's memory. For example, for a problem solving the area of a geometric figure, a search strategy might include searching for area calculation formulas related to that type of geometric figure, as well as historical solutions.

[0070] The memory unit stores a large amount of historical data, knowledge, and solutions. These data can be previous users' questions and answers, teaching materials, formulas and theorems, etc.

[0071] Execution interface parameters are instructions that specifically direct the core processor how to execute the solution steps. These might include the function to call, the required input parameters, and so on. In the geometry example, the execution strategy might include using a specific area calculation formula and specifying the values of each variable in the formula. The core processor will use these execution interface parameters to accurately calculate and output the answer to the problem.

[0072] The result output module 24 is specifically used for:

[0073] The execution strategy is input into the scheduler, and the target education vector indicated by the execution strategy and the target display tool based on the execution interface parameter indication are obtained from the education vector database and tool library through the scheduler; the education vector database is used to store multiple school-age and multiple subject test questions and related vectors of the corresponding test questions and answers; the multimodal feature vector is input into the decoder to obtain the original data expression of the multimodal feature vector; the target education vector, target display tool and the original data expression are input into the multimodal large model to obtain the teaching response content for the question, and the teaching response content includes audio response content and image response content.

[0074] For example, for a junior high school geography question about day and night variations, the multimodal model might generate a textual response explaining the principles of day and night variations, along with a video response demonstrating the changes. In this way, the projection processing module 31 and the voice playback module 32 project the video and play the audio, allowing users to better understand and grasp geographical knowledge by reading and viewing these responses.

[0075] In the present invention, the image acquisition module 13 collects user facial image data in real time, and the image processing module 23 uses the facial image data to analyze the user's eye movement data and user expression data in real time, including but not limited to the user's blink frequency, user line of sight data, user pupil size, user eye saccade distance, user reading speed, distance from the user's eye corner to the mouth, user eyebrow spacing, and user mouth opening size. If the user's eye movement data and user expression data are found to be abnormal, a feedback strategy for the user's current learning status is output. The result output module 24 outputs audio feedback content and image feedback content based on the feedback strategy to guide the user to study diligently. For example, if the image processing module 23 detects that the user's pupil size continues to decrease while studying, and the duration exceeds a first defined time, it determines that the user is dozing off while studying. The core processor can use the voice playback module 32 to remind the user not to doze off and play a piece of refreshing music to encourage the user to persevere and continue studying.

[0076] When the image processing module 23 performs eye movement recognition on the user, it first uses horizontal difference and vertical integral projection to roughly locate the position of the human eye, then uses the Harris corner detection method to find the inner and outer corners of the eye, and finally determines the left and right movement posture of the eye based on the analysis method of the low gray value characteristics of the eye. Furthermore, eye movement recognition and expression recognition can also use a camera component that integrates a face recognition solution, complete eye movement recognition and expression recognition through the built-in processing module, and output corresponding data.

[0077] The present invention also provides an interactive method for an intelligent educational projector system based on an AI large model, which includes:

[0078] Collecting audio data of the user's questions in real time, and collecting at least one of text data and image data of the questions through a preset interactive interface;

[0079] Encoding the audio data, text data, and picture data to obtain corresponding audio feature vectors, text feature vectors, and picture feature vectors, wherein the audio feature vectors, text feature vectors, and picture feature vectors correspond to the same target vector space;

[0080] The audio feature vector, the text feature vector and the image feature vector are used as multimodal feature vectors;

[0081] Inputting the multimodal feature vector into a pre-trained multimodal large model to obtain an execution strategy for the problem, wherein the multimodal large model is fine-tuned based on education field data;

[0082] Capturing facial image data of the user's current learning status in real time, performing user eye movement recognition processing on the facial image data, and obtaining a feedback strategy for the user's current learning status;

[0083] generating audio answer content and image answer content for the question based on the execution strategy;

[0084] Based on the feedback strategy, generate audio feedback content and image feedback content for the user's current learning status;

[0085] Projecting the image response content and image feedback content;

[0086] The audio response content and the audio feedback content are played out.

[0087] Furthermore, the interaction method also includes encoding the audio data to obtain the audio feature vector, including encoding the audio data through ImageBind and mapping it to the target vector space through an audio projection layer to obtain the audio feature vector.

[0088] Furthermore, the interactive method further includes encoding the text data to obtain the text feature vector, including segmenting the text data to obtain a plurality of to-be-processed segmented words; encoding the plurality of to-be-processed segmented words through a word embedding encoder, and mapping them to the target vector space through a text projection layer to obtain a text content feature vector;

[0089] Furthermore, the interaction method also includes encoding the image data to obtain the image feature vector, including encoding the image data through a CLIP visual encoder and mapping it to the target vector space through an image projection layer to obtain the image feature vector.

[0090] In summary, the present invention provides an intelligent educational projector system and an interaction method based on an AI large model, which can provide users with learning guidance plans through multimodal interaction between users and the system, so that users can easily master and consolidate learning knowledge with appropriate methods, and exercise users' ability of self-study and independent thinking. At the same time, a learning status monitoring module is provided to make user learning easier.

[0091] The above-described embodiments merely illustrate several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that variations and improvements are possible without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

Claims

1. An intelligent educational projector system based on AI big model, characterized by: include: Projector body, data acquisition unit, core processor, projection processing module and voice playback module; The data acquisition unit includes a voice acquisition module, a picture and text acquisition module and an image acquisition module; The voice collection module is used to collect audio data of users' questions in real time; The image and text collection module is used to collect at least one of text data and image data for the question through a preset interactive interface; The image acquisition module is used to capture the facial image data of the user's current learning state in real time; The core processor is equipped with a coding module, a problem processing module, an image processing module and a result output module; The encoding module is used to encode the audio data, text data and picture data to obtain corresponding audio feature vectors, text feature vectors and picture feature vectors, wherein the audio feature vectors, text feature vectors and picture feature vectors correspond to the same target vector space; and the audio feature vectors, text feature vectors and picture feature vectors are used as multimodal feature vectors; The problem processing module is used to input the multimodal feature vector into a pre-trained multimodal large model to obtain an execution strategy for the problem, wherein the multimodal large model is fine-tuned based on education field data; The image processing module is used to perform user eye movement recognition processing on the facial image data and obtain a feedback strategy for the user's current learning status; The result output module is used to generate audio answer content and image answer content for the question based on the execution strategy, and the result output module is also used to generate audio feedback content and image feedback content for the user's current learning status based on the feedback strategy; The projection processing module is used to receive the image reply content and the image feedback content, and project the image reply content and the image feedback content through the projection lens of the projector body; The voice playing module is used to receive the audio response content and the audio feedback content, and play the audio response content and the audio feedback content through a speaker.

2. The intelligent educational projector system based on the AI large model according to claim 1 is characterized in that: The encoding module is specifically used for: The text data is segmented to obtain a plurality of segmented words to be processed; the plurality of segmented words to be processed are encoded by a word embedding encoder, and mapped to the target vector space by a text projection layer to obtain the text feature vector; the image data is encoded by a CLIP visual encoder, and mapped to the target vector space by a picture projection layer to obtain the picture feature vector; the audio data is encoded by ImageBind, and mapped to the target vector space by an audio projection layer to obtain the audio feature vector.

3. The intelligent educational projector system based on the AI large model according to claim 1 is characterized in that: The problem processing module is specifically used to: The multimodal feature vector is input into the multimodal large model to obtain initial demand information; the initial demand information is sent to the interactor to determine the retrieval strategy; the interactor searches in the memory unit based on the retrieval strategy to obtain the retrieval memory content corresponding to the multimodal feature vector; the multimodal feature vector and the retrieval memory content are input into the multimodal large model to obtain the execution strategy corresponding to the multimodal feature vector, wherein the execution strategy includes execution interface parameters.

4. The intelligent educational projector system based on the AI large model according to claim 3 is characterized by: The result output module is specifically used for: Inputting the execution strategy into a scheduler, and obtaining the target education vector indicated by the execution strategy and the target presentation tool indicated by the execution interface parameters from an education vector database and a tool library through the scheduler; The educational vector database is used to store relevant vectors of test questions of multiple school ages and multiple subjects and corresponding test questions and answers; the multimodal feature vector is input into a decoder to obtain the original data expression of the multimodal feature vector; The target education vector, target display tool and original data expression are input into the multimodal large model to obtain the teaching response content for the question, and the teaching response content includes audio response content and image response content.

5. The intelligent educational projector system based on the AI big model according to claim 1 is characterized in that: The image acquisition module includes a depth camera arranged below the projector body.

6. The intelligent educational projector system based on the AI large model according to claim 1 is characterized in that: The picture and text collection module includes a writing board arranged on the user's desktop.

7. An interactive method for an intelligent educational projector system based on an AI large model, characterized in that: include: Collecting audio data of the user's questions in real time, and collecting at least one of text data and image data of the questions through a preset interactive interface; Encoding the audio data, text data, and picture data to obtain corresponding audio feature vectors, text feature vectors, and picture feature vectors, wherein the audio feature vectors, text feature vectors, and picture feature vectors correspond to the same target vector space; The audio feature vector, the text feature vector and the image feature vector are used as multimodal feature vectors; Inputting the multimodal feature vector into a pre-trained multimodal large model to obtain an execution strategy for the problem, wherein the multimodal large model is fine-tuned based on education field data; Capturing facial image data of the user's current learning status in real time, performing user eye movement recognition processing on the facial image data, and obtaining a feedback strategy for the user's current learning status; generating audio answer content and image answer content for the question based on the execution strategy; Based on the feedback strategy, generate audio feedback content and image feedback content for the user's current learning status; Projecting the image response content and image feedback content; The audio response content and the audio feedback content are played out.

8. The interactive method according to claim 7, characterized in that: The audio data is encoded to obtain the audio feature vector, including encoding the audio data through ImageBind and mapping it to the target vector space through an audio projection layer to obtain the audio feature vector.

9. The interactive method according to claim 8, characterized in that: The text data is encoded to obtain the text feature vector, including segmenting the text data to obtain multiple segmented words to be processed; the multiple segmented words to be processed are encoded through a word embedding encoder, and mapped to the target vector space through a text projection layer to obtain a text content feature vector.

10. The interactive method according to claim 9, characterized in that: The image data is encoded to obtain the image feature vector, including encoding the image data through a CLIP visual encoder, and mapping it to the target vector space through a picture projection layer to obtain the image feature vector.

Images