AI model-based video AR virtual reality interactive communication method and system

Abstract

The application relates to an AI model-based video AR virtual reality interactive communication method and system, and belongs to the technical field of artificial intelligence and virtual reality interaction. The application first responds to an AR virtual reality interactive signal triggered by a user, starts an AR engine to render a virtual target in a visual field scene of AR glasses worn by the user, perceives user interest in real time through voice data fed back by the user after question and answer interactive instructions, designs the action difficulty of the user according to the user interest to cater to the user preference, and dynamically adapts interactive content through body movement data and facial expression data of the user when the user performs the action, so that personalized design is as far as possible for the user, the user can experience the interactive action of interest, and the experience of virtual reality interactive communication is improved.

Description

Technical Field

[0001] This invention relates to a video AR virtual reality interactive communication method and system based on an AI model, belonging to the field of artificial intelligence and virtual reality interactive technology. Background Technology

[0002] Augmented Reality (AR) technology provides users with entirely new interactive experiences by overlaying virtual information onto the real world. In scenarios such as fitness instruction, skills training, and interactive entertainment, AR systems often use virtual coaches or virtual characters to demonstrate movements to users and guide them in learning. The core goal of these systems is to provide personalized, immersive interactive experiences, allowing users to achieve their training or entertainment goals while enjoying the process.

[0003] However, existing AR interaction systems still have significant shortcomings in achieving this goal. First, these systems typically employ preset, fixed difficulty curves or level designs, lacking the perception and response to users' real-time interests and emotional states. This can easily lead to users feeling frustrated due to excessively difficult actions or boredom due to excessively easy actions, severely impacting the continuity of the experience and user engagement.

[0004] Therefore, there is an urgent need for an AR virtual reality interaction method that can dynamically adapt interactive content based on real-time perception of user interests. Summary of the Invention

[0005] The purpose of this invention is to provide a video AR virtual reality interactive communication method and system based on an AI model. First, the system uses voice data from the user's feedback after the virtual target gives a question-and-answer interaction command to perceive the user's interest in real time. Then, the system designs the difficulty of the user's actions according to the user's interests to cater to the user's preferences. Finally, the system dynamically adapts the interactive content by using the user's body movement data and facial expression data when performing the actions, thereby providing the most personalized design possible for the user and enabling the user to experience interactive actions that interest them, thus improving the user experience of virtual reality interactive communication.

[0006] In a first aspect, the present invention provides a video AR virtual reality interactive communication method based on an AI model, characterized in that it includes: In response to user-triggered AR virtual reality interaction signals, the AR engine is activated to render virtual targets within the field of view of the AR glasses worn by the user. Acquire voice data from the user after the virtual target gives a question-and-answer interaction instruction, wherein the question-and-answer interaction instruction includes action information designed for the user; Keyword extraction is performed on the voice data from user feedback. Each extracted keyword is assigned an interest value according to a predefined interest bias rule. The user's level of interest in the action information is determined based on the sum of the interest values ​​of all keywords. The difficulty of the action information is determined based on the level of interest, wherein the level of interest is positively correlated with the difficulty of the action; and the action interaction instructions are determined based on the difficulty of the action and transmitted to the user through a virtual target. The virtual target displays the action information according to the difficulty of the action, and the image data of the user performing the action according to the action information is obtained. The image data is then input into a pre-trained AI image recognition model to identify the user's body movement data and facial expression data. The body movement data is input into a pre-trained first AI evaluation model to obtain the user's action completion rate, and the facial expression data is input into a pre-trained second AI evaluation model to obtain the user's action acceptance rate. The user's satisfaction score for this virtual reality interaction is determined based on the action completion rate and the action acceptance rate, and the user's subsequent virtual reality interaction strategy is determined based on the satisfaction score.

[0007] In a preferred embodiment, the response to user-triggered AR virtual reality interaction signals includes a power button signal on the AR glasses or a click signal on a smart terminal.

[0008] As a preferred embodiment, activating the AR engine to render virtual targets within the field of view of the AR glasses worn by the user includes: The AR engine uses cameras and SLAM algorithms to build a 3D scene map of the user's surrounding environment in real time, and at the same time determines the pose data of the AR glasses in the map. Feature planes in the 3D scene map are identified to determine the generation location of virtual targets; Based on pre-obtained user requirements, the model data of the virtual target is loaded from the resource library, and the virtual target is loaded at the generation location according to spatial rules to render it within the field of view of the AR glasses worn by the user.

[0009] As a preferred embodiment, the voice data obtained from the user's feedback after the virtual target gives a question-and-answer interaction command includes the following methods for the virtual target to give the question-and-answer interaction command: The virtual target performs lip-sync and gesture actions through preset animations, while simultaneously providing text information for question-and-answer interaction commands within the AR glasses' field of view, or outputting voice information for question-and-answer interaction commands through the AR glasses or a pre-configured external speaker.

[0010] As a preferred embodiment, keyword extraction is performed on the user-reported voice data, and the extracted keywords are assigned interest values ​​to each keyword according to predefined interest bias rules, including: The voice data is input into an automatic speech recognition engine for decoding to obtain the user's response in text format; The response statement was processed using natural language processing and segmented to obtain multiple keywords; The keyword and the complete answer statement are input into a pre-built semantic sentiment model to obtain the interest value of the keyword in the semantic context of the current answer statement; the more the keyword is inclined to be of interest in the semantic context of the current answer statement, the greater the interest value.

[0011] As a preferred embodiment, determining the action difficulty of the action information based on the level of interest, and determining the action interaction instructions to be transmitted to the user through the virtual target based on the action difficulty, includes: The interest level is pre-divided into multiple interest level ranges, and each interest level range corresponds to a certain difficulty level of the action; The difficulty of the action information designed for the user is determined based on the interest level range to which the current interest level belongs. Multiple difficulty levels are pre-set for each action information, and each difficulty level corresponds to a specific action interaction command; the corresponding action interaction command is determined based on the difficulty level determined by the level of interest.

[0012] In a preferred embodiment, the limb movement data includes a sequence of limb key point coordinates of the user, which consists of the spatial coordinates of multiple predetermined limb key points arranged in a temporal order in a 3D scene map. The facial expression data includes a sequence of facial key point coordinates, which consists of the spatial coordinates of multiple pre-determined facial key points arranged in a temporal order on a 3D scene map. As a preferred embodiment, the training method of the first AI evaluation model includes: The body movement data of the standard movements corresponding to the action information are used as positive training samples, and the body movement data of the non-standard movements corresponding to the action information are used as negative training samples to construct the first training set. The first AI model using a temporal deep learning model is trained using the first training set until the first loss function converges to obtain the trained first AI evaluation model. The training methods for the second AI evaluation model include: Historical data of facial expressions when users perform actions is obtained, and subjective feeling labels are manually labeled on the user's facial expression data; a second training set is constructed based on the historical data of facial expression data and the manually labeled subjective feeling labels. The second AI model of the temporal deep learning model is trained using the second training set until the second loss function converges to obtain the trained second AI evaluation model.

[0013] As a preferred embodiment, determining the current user's subsequent interaction strategy based on the satisfaction score includes: If the satisfaction score is greater than the preset score threshold, it is determined that the user is satisfied with the current action information, and the user continues to receive higher difficulty action interaction instructions for the current action information through the virtual target. If the satisfaction score is not greater than the preset score threshold, it is determined that the user is not satisfied with the current action information, and the virtual target retransmits the question-and-answer interaction instructions, including other action information, to the current user.

[0014] Secondly, the present invention provides a video AR virtual reality interactive communication system based on an AI model, comprising: The startup module is used to respond to AR virtual reality interaction signals triggered by the user and start the AR engine to render virtual targets within the field of view of the AR glasses worn by the user. The first acquisition module is used to acquire voice data fed back by the user after the virtual target gives a question-and-answer interaction instruction, wherein the question-and-answer interaction instruction includes action information designed for the user; The interest level confirmation module is used to extract keywords from the voice data provided by the user, assign interest weights to each keyword according to predefined interest bias rules, and use the interest weights to perform a weighted sum of all keywords to obtain the user's interest level in the action information. The motion interaction module is used to determine the motion difficulty of the motion information based on the level of interest, wherein the level of interest is positively correlated with the motion difficulty; and to determine the motion interaction instructions to be transmitted to the user through a virtual target based on the motion difficulty. The AI ​​model recognition module is used to display the action information of the virtual target according to the difficulty of the action, obtain image data of the user when performing the action according to the action information, and input the image data into a pre-trained AI image recognition model to recognize the user's body action data and facial expression data. The AI ​​model evaluation module is used to input the body movement data into a pre-trained first AI evaluation model to obtain the user's action completion rate, and to input the facial expression data into a pre-trained second AI evaluation model to obtain the user's action acceptance rate. The virtual reality interaction module is used to determine the user's satisfaction score for the current virtual reality interaction based on the action completion rate and the action acceptance rate, and to determine the user's subsequent virtual reality interaction strategy based on the satisfaction score.

[0015] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention first responds to a user-triggered AR virtual reality interaction signal, activating the AR engine to render a virtual target within the user's AR glasses' field of view, thus providing a basic scenario for virtual reality interactive communication. Then, it acquires the user's voice data after the virtual target issues a question-and-answer interaction command. This command includes action information designed for the user, allowing the virtual target to convey question-and-answer interaction commands (e.g., fitness action instructions in a fitness application scenario or game action instructions in a game application scenario). Keywords are extracted from the user's voice data, and each extracted keyword is assigned an interest value according to a predefined interest bias rule. Finally, the sum of the interest values ​​of all keywords is used to determine the target's target. The invention assesses the user's level of interest in motion information. To cater to user experience, it analyzes user feedback voice data to pre-determine the user's level of interest in motion information. Different levels of interest indicate different levels of user acceptance of the motion information. The invention further determines the motion difficulty of the motion information based on the level of interest, and the level of interest is positively correlated with the motion difficulty. Based on the motion difficulty, the invention determines the motion interaction instructions to be transmitted to the user through a virtual target. The invention designs the motion difficulty based on the user's level of interest (e.g., the motion difficulty in a fitness scenario or the level difficulty in a game scenario) to achieve a match between the level of interest and the motion difficulty, thereby improving the user experience and avoiding poor user experience.

[0016] Subsequently, the virtual target displays the action information according to the difficulty of the action, acquiring image data of the user performing the action based on the action information. This image data is then input into a pre-trained AI image recognition model to identify the user's body movement data and facial expression data. This invention utilizes an image recognition model to accurately identify user body movement data and facial expression data for subsequent user evaluation. Finally, the body movement data is input into a pre-trained first AI evaluation model to obtain the user's action completion rate, and the facial expression data is input into a pre-trained second AI evaluation model to obtain the user's action acceptance rate. Based on the action completion rate and the action acceptance rate, the user's satisfaction score for this virtual reality interaction is determined, and the subsequent virtual reality interaction strategy for the current user is determined based on the satisfaction score. If the satisfaction score is greater than a preset scoring threshold, it is determined that the user is satisfied with the current action information, and the virtual target continues to transmit action interaction instructions with higher difficulty levels to the current user. If the satisfaction score is not greater than the preset scoring threshold, it is determined that the user is not satisfied with the current action information, and the virtual target re-transmits question-and-answer interaction instructions including other action information to the current user. The present invention, through the above method, can design the difficulty of actions by using voice data that provides feedback before the user performs an action to cater to the user's interests in advance, and determine the subsequent interaction strategy by using the user's body movement data and facial expression data when performing the action, so as to allow the user to experience interactive actions that are of interest to the user, thereby improving the user experience of virtual reality interactive communication. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart illustrating a video AR virtual reality interactive communication method based on an AI model, according to an embodiment of the present invention. Figure 2 The diagram shown is an architectural schematic of a video AR virtual reality interactive communication system based on an AI model provided in an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that: The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0019] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0020] This embodiment introduces a video AR virtual reality interactive communication method based on an AI model, which includes: Step S1: In response to the AR virtual reality interaction signal triggered by the user, start the AR engine to render the virtual target in the field of view of the AR glasses worn by the user; in this embodiment, the AR virtual reality interaction signal triggered by the user includes the power button signal on the AR glasses or the click signal on the smart terminal. The smart terminal is a mobile phone, tablet or remotely controlled computer, all of which can be used to trigger the AR virtual reality interaction signal through wireless communication.

[0021] Activating the AR engine to render virtual targets within the user's AR glasses' field of view includes: the AR engine using a camera and SLAM algorithm to construct a 3D scene map of the user's surrounding environment in real time, and simultaneously determining the AR glasses' pose data within the map; identifying feature planes (such as floors, desktops, and walls) in the 3D scene map to determine the generation location of the virtual target; loading the virtual target's model data (for example, a 3D cartoon character or a robot model) from a resource library according to pre-obtained user requirements; and, based on spatial rules (including the virtual target's standing posture and orientation, and avoiding occlusion by real objects), performing pixel-level fusion of the virtual target's image with the real-world video stream captured by the camera on the AR glasses' display module (such as an optical waveguide or MicroLED), loading the virtual target at the generation location, and rendering it within the user's AR glasses' field of view.

[0022] Step S2: Obtain the voice data of the user's feedback after the virtual target gives a question-and-answer interaction command. The question-and-answer interaction command includes action information designed for the user. The virtual target performs lip-sync and gesture movements through preset animations, and simultaneously provides text information of the question-and-answer interaction command in the AR glasses' field of view, such as various fitness movements in a fitness application, or game actions of various game levels in an entertainment application, or simultaneously outputs the voice information of the question-and-answer interaction command through the AR glasses or a pre-configured external speaker. The voice data is recorded through the microphone array built into the AR glasses. The microphone array is focused in the direction of the user's mouth, effectively suppressing ambient noise; this is a mature existing technology and will not be elaborated here.

[0023] Step S3: Extract keywords from the voice data feedback by the user, and assign an interest value to each keyword according to the predefined interest bias rule; determine the user's interest level in the action information based on the sum of the interest values of all keywords, and the sum of the interest values is normalized to obtain the user's interest level in the action information; Step S3 specifically includes inputting the voice data into an automatic speech recognition engine (such as DeepSpeech, Wav2Vec) for decoding to obtain the user's response statement in text format; performing natural language processing on the response statement (performing word segmentation,词性标注, removing stop words, such as "的", "了") on the response statement and splitting it into multiple keywords. The keywords can be divided into positive words, neutral words, and negative words. Each keyword has a different emotional meaning in the semantic context of the current response statement. For example, in a fitness application, "I have practiced this action, and I am very confident in this action", the extracted keywords are "practiced", "very", "confident", all of which have relatively high interest values. Input the keyword and the complete response statement into a pre-constructed semantic emotion model to obtain the interest value of the keyword in the semantic context of the current response statement; the more the keyword偏向于感兴趣 (more偏向于褒性词) in the semantic context of the current response statement, the greater the assigned interest value. Assume that N keywords are extracted, and the interest value of the i-th keyword is w i Interest level .

[0024] Step S4: Determine the action difficulty of the action information according to the interest level, and the interest level is positively correlated with the action difficulty; determine the action interaction instruction passed to the user through the virtual target according to the action difficulty; Step S4 specifically includes: pre-dividing multiple interest level intervals for the interest level, and each interest level interval corresponds to an action difficulty; determining the action difficulty of the action information designed for the user according to the interest level interval to which the current interest level belongs; pre-setting multiple action difficulties for each action information, and each action difficulty is correspondingly designed with a corresponding action interaction instruction; determining the corresponding action interaction instruction according to the action difficulty determined by the interest level. If the user shows a higher interest level, a higher action difficulty can be designed for the user accordingly to match the user's mood. If the user shows a lower interest level, it is not convenient to design a higher action difficulty for the user, which is likely to cause a poor user experience for the user. Therefore, a lower action difficulty needs to be designed for the user to adapt and feel.

[0025] Step S5: Display the action information to the virtual target according to the difficulty of the action. Existing systems evaluate user actions using a single dimension. Most solutions only capture the user's body movements through cameras or sensors and match them with standard action templates to determine the "completeness" or "accuracy" of the action. This evaluation method ignores the user's subjective psychological feelings when performing the action. An action may be performed correctly, but the user may have a very poor experience due to difficulty, pain, or impatience. The lack of evaluation of the user's emotional acceptance makes it impossible for the system to truly understand the user's satisfaction, and subsequent adjustments to the interaction strategy lack key basis. This embodiment acquires image data of the user performing actions based on the action information, and inputs the image data into a pre-trained AI image recognition model to identify the user's body movement data and facial expression data. The body movement data includes a sequence of key point coordinates for the user's limbs. This sequence consists of the spatial coordinates of multiple pre-determined key points (typically including the head, neck, shoulders, elbows, wrists, hips, knees, and ankles) arranged in a temporal order on a 3D scene map. The facial expression data includes a sequence of key point coordinates for the face. This sequence consists of the spatial coordinates of multiple pre-determined key points arranged in a temporal order on a 3D scene map. The sequence of key point coordinates for the face can describe the shape changes of eyebrows, eyes, nose, mouth, and facial contours.

[0026] Step S6: Input the limb movement data into a pre-trained first AI evaluation model to obtain the user's movement completion rate, and input the facial expression data into a pre-trained second AI evaluation model to obtain the user's movement acceptance rate. In step S6, the training method of the first AI evaluation model includes: acquiring limb movement data of standard movements corresponding to the action information as positive training samples, and acquiring limb movement data of non-standard movements corresponding to the action information as negative training samples to construct a first training set; using the first training set to train the first AI model using a temporal deep learning model (LSTM, Long Short-Term Memory Network in this embodiment) until the first loss function (mean squared error loss MSE or smooth L1 loss) converges to obtain the trained first AI evaluation model.

[0027] The training method for the second AI evaluation model includes: acquiring historical data of facial expression data when the user performs actions; manually labeling the user's facial expression data with subjective feelings (such as "easy to accept", "difficult but persistent", "resistant to pain"); constructing a second training set based on the historical data of the facial expression data and the manually labeled subjective feelings; and using the second training set to train the second AI model of the temporal deep learning model (LSTM, Long Short-Term Memory Network in this embodiment) until the second loss function (such as mean squared error MSE or mean absolute error MAE) converges to obtain the trained second AI evaluation model.

[0028] Step S7: Determine the user's satisfaction score for this virtual reality interaction based on the action completion rate and the action acceptance rate, and determine the user's subsequent virtual reality interaction strategy based on the satisfaction score. The user's satisfaction score for this virtual reality interaction is obtained by weighted summing of the action completion rate and the action acceptance rate. Preset action completion weight α and action acceptance weight β (α+β=1) are assigned to the action completion rate and action acceptance rate. The specific values ​​of the action completion weight α and action acceptance weight β can be adjusted according to the application scenario. For example, in fitness applications, the completion weight α can be set higher; in entertainment applications, the acceptance weight β can be set higher.

[0029] Determining the user's subsequent interaction strategy based on the satisfaction score includes: if the satisfaction score is greater than a preset score threshold, it is determined that the user is satisfied with the current action information, and the virtual target continues to transmit higher-difficulty action interaction instructions to the user regarding the current action information; if the satisfaction score is not greater than the preset score threshold, it is determined that the user is not satisfied with the current action information, and the virtual target retransmits question-and-answer interaction instructions including other action information to the user.

[0030] Another specific embodiment provides a video AR virtual reality interactive communication system based on an AI model, which includes: The startup module is used to respond to AR virtual reality interaction signals triggered by the user and start the AR engine to render virtual targets within the field of view of the AR glasses worn by the user. The first acquisition module is used to acquire voice data fed back by the user after the virtual target gives a question-and-answer interaction instruction, wherein the question-and-answer interaction instruction includes action information designed for the user; The interest level confirmation module is used to extract keywords from the voice data provided by the user, assign interest weights to each keyword according to predefined interest bias rules, and use the interest weights to perform a weighted sum of all keywords to obtain the user's interest level in the action information. The motion interaction module is used to determine the motion difficulty of the motion information based on the level of interest, wherein the level of interest is positively correlated with the motion difficulty; and to determine the motion interaction instructions to be transmitted to the user through a virtual target based on the motion difficulty. The AI ​​model recognition module is used to display the action information of the virtual target according to the difficulty of the action, obtain image data of the user when performing the action according to the action information, and input the image data into a pre-trained AI image recognition model to recognize the user's body action data and facial expression data. The AI ​​model evaluation module is used to input the body movement data into a pre-trained first AI evaluation model to obtain the user's action completion rate, and to input the facial expression data into a pre-trained second AI evaluation model to obtain the user's action acceptance rate. The virtual reality interaction module is used to determine the user's satisfaction score for the current virtual reality interaction based on the action completion rate and the action acceptance rate, and to determine the user's subsequent virtual reality interaction strategy based on the satisfaction score.

[0031] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0033] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0034] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0035] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A video AR virtual reality interactive communication method based on an AI model, characterized in that, include: In response to user-triggered AR virtual reality interaction signals, the AR engine is activated to render virtual targets within the field of view of the AR glasses worn by the user. Acquire voice data from the user after the virtual target gives a question-and-answer interaction instruction, wherein the question-and-answer interaction instruction includes action information designed for the user; Keyword extraction is performed on the voice data from user feedback. Each extracted keyword is assigned an interest value according to a predefined interest bias rule. The user's level of interest in the action information is determined based on the sum of the interest values ​​of all keywords. The difficulty of the action information is determined based on the level of interest, wherein the level of interest is positively correlated with the difficulty of the action; and the action interaction instructions are determined based on the difficulty of the action and transmitted to the user through a virtual target. The virtual target displays the action information according to the difficulty of the action, and the image data of the user performing the action according to the action information is obtained. The image data is then input into a pre-trained AI image recognition model to identify the user's body movement data and facial expression data. The body movement data is input into a pre-trained first AI evaluation model to obtain the user's action completion rate, and the facial expression data is input into a pre-trained second AI evaluation model to obtain the user's action acceptance rate. The user's satisfaction score for virtual reality interactive communication is determined based on the action completion rate and the action acceptance rate, and the user's subsequent virtual reality interaction strategy is determined based on the satisfaction score. Keyword extraction is performed on the voice data provided by users. Each extracted keyword is then assigned an interest value according to predefined interest bias rules, including: The voice data is input into an automatic speech recognition engine for decoding to obtain the user's response in text format; The response statement was processed using natural language processing and segmented to obtain multiple keywords; The keyword and the complete answer statement are input into a pre-built semantic sentiment model to obtain the interest value of the keyword in the semantic context of the current answer statement; the more the keyword is related to interest in the semantic context of the current answer statement, the greater the interest value. Based on the level of interest, the difficulty of the action information is determined, and based on the difficulty of the action, the action interaction instructions to be transmitted to the user through the virtual target are determined, including: The interest level is pre-divided into multiple interest level ranges, and each interest level range corresponds to a certain difficulty level of the action; The difficulty of the action information designed for the user is determined based on the interest level range to which the current interest level belongs. Multiple difficulty levels are pre-set for each action information, and each difficulty level corresponds to a specific action interaction command; the corresponding action interaction command is determined based on the difficulty level determined by the level of interest. Determining subsequent interaction strategies for the current user based on the satisfaction score includes: If the satisfaction score is greater than the preset score threshold, it is determined that the user is satisfied with the current action information, and the user continues to receive higher difficulty action interaction instructions for the current action information through the virtual target. If the satisfaction score is not greater than the preset score threshold, it is determined that the user is not satisfied with the current action information, and the virtual target retransmits the question-and-answer interaction instructions, including other action information, to the current user.

2. The video AR virtual reality interactive communication method based on an AI model according to claim 1, characterized in that, In response to user-triggered AR virtual reality interaction signals, such as the power button signal on AR glasses or the click signal on a smart terminal.

3. The video AR virtual reality interactive communication method based on an AI model according to claim 1, characterized in that, Starting the AR engine to render virtual targets within the user's field of view using AR glasses includes: The AR engine uses cameras and SLAM algorithms to build a 3D scene map of the user's surrounding environment in real time, and at the same time determines the pose data of the AR glasses in the map. Feature planes in the 3D scene map are identified to determine the generation location of virtual targets; Based on pre-obtained user requirements, the model data of the virtual target is loaded from the resource library, and the virtual target is loaded at the generation location according to spatial rules to render it within the field of view of the AR glasses worn by the user.

4. The video AR virtual reality interactive communication method based on an AI model according to claim 1, characterized in that, Among the voice data obtained after the virtual target gives a question-and-answer interaction command, the ways in which the virtual target gives the question-and-answer interaction command include: The virtual target performs lip-sync and gesture movements through preset animations, while simultaneously providing textual information for question-and-answer interaction commands within the AR glasses' field of view, or outputting voice information for question-and-answer interaction commands through an external speaker pre-configured on the AR glasses.

5. The video AR virtual reality interactive communication method based on an AI model according to claim 1, characterized in that, The limb movement data includes a sequence of key limb coordinates of the user, which consists of the spatial coordinates of multiple predetermined key limbs arranged in chronological order in a 3D scene map. The facial expression data includes a sequence of facial key point coordinates, which consists of the spatial coordinates of multiple pre-determined facial key points arranged in a temporal order in a 3D scene map.

6. The video AR virtual reality interactive communication method based on an AI model according to claim 1, characterized in that, The training methods for the first AI evaluation model include: The body movement data of the standard movements corresponding to the action information are used as positive training samples, and the body movement data of the non-standard movements corresponding to the action information are used as negative training samples to construct the first training set. The first AI model using a temporal deep learning model is trained using the first training set until the first loss function converges to obtain the trained first AI evaluation model. The training methods for the second AI evaluation model include: Historical data of facial expressions when users perform actions is obtained, and subjective feeling labels are manually labeled on the user's facial expression data; a second training set is constructed based on the historical data of facial expression data and the manually labeled subjective feeling labels. The second AI model of the temporal deep learning model is trained using the second training set until the second loss function converges to obtain the trained second AI evaluation model.

7. A video AR virtual reality interactive communication system based on an AI model, characterized in that, The system is used to perform the AI-based video AR virtual reality interactive communication method according to any one of claims 1-6, the system comprising: The startup module is used to respond to AR virtual reality interaction signals triggered by the user and start the AR engine to render virtual targets within the field of view of the AR glasses worn by the user. The first acquisition module is used to acquire voice data fed back by the user after the virtual target gives a question-and-answer interaction instruction, wherein the question-and-answer interaction instruction includes action information designed for the user; The interest level confirmation module is used to extract keywords from the voice data provided by the user, assign an interest value to each keyword according to a predefined interest bias rule, and use the interest values ​​to perform a weighted sum of all keywords to obtain the user's interest level in the action information. The motion interaction module is used to determine the motion difficulty of the motion information based on the level of interest, wherein the level of interest is positively correlated with the motion difficulty; and to determine the motion interaction instructions to be transmitted to the user through a virtual target based on the motion difficulty. The AI ​​model recognition module is used to display the action information of the virtual target according to the difficulty of the action, obtain image data of the user when performing the action according to the action information, and input the image data into a pre-trained AI image recognition model to recognize the user's body action data and facial expression data. The AI ​​model evaluation module is used to input the body movement data into a pre-trained first AI evaluation model to obtain the user's action completion rate, and to input the facial expression data into a pre-trained second AI evaluation model to obtain the user's action acceptance rate. The virtual reality interaction module is used to determine the user's satisfaction score for virtual reality interactive communication based on the action completion rate and the action acceptance rate, and to determine the user's subsequent virtual reality interaction strategy based on the satisfaction score.

Images