Gesture-triggered ai real-time image generation and interaction method and device

Abstract

This application discloses a method and apparatus for gesture-triggered AI real-time image generation and interaction, relating to the field of artificial intelligence technology. The method includes acquiring and establishing a feature connection link between a sensor unit and a target user; collecting and extracting features of the target user based on the sensor unit to obtain target user feature information; matching and locking the sensor unit and the target user based on the target user feature information and the feature connection link, and outputting a pre-preparation completion signal; creating a 3D user digital twin model; performing real-time detection of the target user based on the sensor unit to obtain multi-source sensor data; performing 3D reconstruction of the target user's real-time state based on the multi-source sensor data to obtain 3D user real-time dynamic information; and synchronously displaying the 3D user real-time dynamic information on the 3D user digital twin model. This application improves the efficiency of gesture-triggered AI real-time image generation and interaction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for real-time AI image generation and interaction based on gesture triggering. Background Technology

[0002] Currently, AI (Artificial Intelligence) refers to the technology of simulating human intelligence through computer systems, enabling machines to possess abilities such as perception, understanding, learning, reasoning, and creation. It is a product of the interdisciplinary integration of computer science, mathematics, statistics, neuroscience, linguistics, and other disciplines. With the progress of modern scientific society, AI real-time image generation is becoming increasingly widespread and involved in people's daily lives and work. Therefore, gesture-triggered AI real-time image generation and interaction are of paramount importance.

[0003] Existing gesture-triggered AI real-time image generation and interaction methods typically follow a closed-loop process of "perception-understanding-generation-feedback," namely gesture perception and capture, gesture recognition and parsing, AI real-time generation, and interactive feedback. However, in existing gesture-triggered AI real-time image generation and interaction methods, errors are easily identified when there are multiple users or complex scenarios, resulting in low efficiency and shortcomings. Summary of the Invention

[0004] To improve the efficiency of gesture-triggered AI real-time image generation and interaction, this application provides a gesture-triggered AI real-time image generation and interaction method and apparatus.

[0005] The gesture-triggered AI real-time image generation and interaction method provided in this application adopts the following technical solution: The first aspect is a gesture-triggered AI real-time image generation and interaction method, which includes the following steps: Step S1: Obtain the sensor unit and establish a link between the sensor unit and the target user to obtain the feature connection link. Based on the sensor unit, collect data and extract features from the target user to obtain the target user feature information. Based on the target user feature information and the feature connection link, match and lock the sensor unit and the target user, and output the preliminary preparation completion signal. Step S2: After receiving the signal that the preliminary preparation is complete, inquire about the target user's gesture interaction method to obtain the interaction option output result, create a three-dimensional user digital twin model, perform real-time detection of the target user based on the sensor unit to obtain multi-source sensor data, perform 3D reconstruction of the target user's real-time state based on the multi-source sensor data to obtain three-dimensional user real-time dynamic information, and synchronously display the three-dimensional user real-time dynamic information on the three-dimensional user digital twin model. Step S3: Create a gesture comparison database and determine the target user's corresponding gesture comparison database. Based on the three-dimensional real-time dynamic information of the user, filter out the target user's gesture actions to obtain the user's real-time gesture data, and compare and identify it with the user's corresponding gesture comparison database to obtain the user's gesture intention information. Step S4: Obtain the AI ​​model by inputting the user's gesture intent information and multi-source sensor data into the AI ​​model to generate target image generation information and display it in real time on a pre-set display screen. Step S5: Based on multi-source sensor data, feedback optimization is performed to obtain the first content information to be optimized. A feedback questionnaire is provided to the target user to collect user opinions and obtain the second content information to be optimized. Based on the first and second content information to be optimized, the gesture-triggered AI real-time image generation and interaction are optimized.

[0006] Preferably, the sensor unit includes a high-definition camera; Establishing a signal connection between the sensor unit and the target user yields a characteristic connection link; The high-definition camera is used to capture the target user's full-body image information. Facial feature information is obtained by extracting features from the target user's face based on the user's full-body image acquisition information, and limb feature information is obtained by extracting features from the target user's limbs based on the user's full-body image acquisition information. The user's facial features and the user's limb features are combined to form the target user's features.

[0007] Preferably, when there are multiple target users, the facial feature information of the user is matched with the limb feature information based on the user's full-body image acquisition information, a facial-limb link is created, and a facial-limb matching completion signal is output. Based on the target user's characteristic information and characteristic connection links, when there is only one target user, the sensor unit is used to track the target user in real time. When there are multiple target users, the information of multiple target users is collected simultaneously, and a user sensing lock completion signal is output. Upon receiving the facial and limb matching completion signal and the user sensor locking completion signal, the system outputs a preliminary preparation completion signal.

[0008] Preferably, after receiving the signal indicating completion of the preliminary preparation, the system queries whether there are any deaf or mute users among the target users. If there are deaf or mute users, the system queries whether the target user should perform basic gesture interaction or sign language gesture interaction, and outputs the interaction option result. The sensor unit also includes an inertial measurement unit, a barometer, a positioning sensor, a radio unit, and an eye tracker; Establish a signal connection link between the inertial measurement unit and the target user, and establish a signal connection link between the barometer and the target user; Based on the user's full-body image acquisition information, a digital twin of the target user in three-dimensional space is created to obtain a three-dimensional user digital twin model; The high-definition camera is used to capture the target user from multiple angles to obtain real-time shooting information of the user. The user's location information is obtained by real-time positioning of the target user based on the positioning sensor, and the user's specific location altitude is further detected based on the barometer. The user's location information is then adjusted to obtain the user's real-time location information. The motion state information of the target user is obtained by detecting the motion state of the target user using an inertial measurement unit.

[0009] Preferably, when there are non-deaf and mute users among the target users, the sound receiving unit is activated to collect sound data, obtain the historical sound feature information of the target user, match the sound collection data with the historical sound feature information of the target user, and filter out the sound of the target user in the sound collection data to obtain the real-time sound reception information of the user. Based on the real-time voice information received by the user, it is determined in real time whether there is any dissatisfaction in the target user's voice with the gesture-triggered AI real-time image generation and interaction. If so, the first content information to be optimized is recorded. Real-time eye movement information of the target user is obtained by real-time detection using an eye tracker; The user's real-time shooting information, the user's real-time location information, the user's motion status information, the user's real-time sound reception information, and the user's real-time eye movement information are combined to form multi-source sensor data; Based on real-time user shooting information, real-time user location information, and user movement status information, a real-time dynamic of the target user in three-dimensional space is created to obtain three-dimensional user real-time dynamic information, and the three-dimensional user real-time dynamic information is synchronously displayed on the three-dimensional user digital twin model.

[0010] Preferably, a comparison gesture database is created, which includes a basic comparison gesture database and a sign language comparison gesture database; If there are deaf and mute users in the output results of the interaction options, then sign language interaction will be performed. The deaf and mute users in the target users will select the sign language comparison gesture database. If there are no deaf and mute users in the output results of the interaction options, then basic gesture interaction will be performed. The target users will select the basic comparison gesture database and the corresponding gesture comparison database for the user will be output. Based on real-time dynamic information of three-dimensional users, the gestures of target users are filtered out to obtain real-time gesture data of users. The user's real-time gestures are compared and identified with the corresponding user gesture comparison database to obtain the user's gesture intent information.

[0011] Preferably, an AI model is acquired, which includes a local lightweight AI model and a cloud-based AI model. A signal connection is established between the local lightweight AI model and the target user, and a signal connection is also established between the cloud-based AI model and the target user. If the target user is a deaf or mute user, the user's gesture intention information and real-time eye movement information are sent to the local lightweight AI model to generate the content required by the deaf or mute user locally, thus obtaining the first locally generated information. Simultaneously, user gesture intent information and real-time eye movement information are sent to the cloud AI model to generate the content required by the deaf and mute user in the cloud, thus obtaining the first cloud-generated information. Based on data fusion technology, the first locally generated information and the first cloud-generated information are fused to obtain content generated by deaf and mute users; If the user is not deaf or mute, the user's gesture intention information, real-time sound reception information, and real-time eye movement information are sent to the local lightweight AI model to generate the content required by the non-deaf or mute user locally, thus obtaining the second locally generated information. Simultaneously, user gesture intent information, real-time user audio information, and real-time user eye movement information are sent to the cloud AI model to generate the content required by non-deaf and mute users in the cloud, thus obtaining the second cloud-generated information. Based on data fusion technology, the first locally generated information and the first cloud-generated information are fused to obtain content generated by non-deaf and mute users; The content generated by the deaf and mute users is combined with the content generated by the non-deaf and mute users to form target image generation information; The target image generation information is displayed in real time on a pre-set display screen.

[0012] Preferably, a feedback questionnaire is distributed to the target users, who fill it out and the data is collected. User opinions are then analyzed to obtain the second set of information to be optimized. The first information to be optimized and the second information to be optimized are combined to form a feedback optimization point; The operation of gesture-triggered AI real-time image generation and interaction is optimized based on feedback optimization points.

[0013] Secondly, this application provides a gesture-triggered AI real-time image generation and interaction device, which adopts the following technical solution: A gesture-triggered AI real-time image generation and interaction device includes: The preliminary preparation module is configured to acquire sensor units, establish a link between sensor units and target users to obtain feature connection links, collect data and extract features from target users based on sensor units to obtain target user feature information, match and lock sensor units and target users based on target user feature information and feature connection links, and output a preliminary preparation completion signal. The dynamic acquisition module is configured to, upon receiving a signal indicating completion of preliminary preparation, inquire about the target user's gesture interaction method to obtain the interaction options output result, create a three-dimensional user digital twin model, perform real-time detection of the target user based on sensor units to obtain multi-source sensor data, perform 3D reconstruction of the target user's real-time state based on multi-source sensor data to obtain three-dimensional user real-time dynamic information, and synchronously display the three-dimensional user real-time dynamic information on the three-dimensional user digital twin model. The gesture recognition module is configured to create a comparison gesture database and determine the user's corresponding gesture comparison database. Based on the real-time dynamic information of the three-dimensional user, the module filters out the gesture actions of the target user, obtains the user's real-time gesture data, and compares and recognizes it with the user's corresponding gesture comparison database to obtain the user's gesture intention information. The image generation and output module is configured to acquire an AI model, input user gesture intent information and multi-source sensor data into the AI ​​model, generate target image generation information, and display it in real time on a pre-set display screen. The feedback optimization module is configured to perform feedback optimization based on multi-source sensor data to obtain the first content information to be optimized, provide a feedback questionnaire to the target user to collect user opinions and obtain the second content information to be optimized, and optimize the AI ​​real-time image generation and interaction based on gesture triggering based on the first and second content information to be optimized.

[0014] In summary, this application includes at least one of the following beneficial technical effects: 1. By matching the facial features and limb features of multiple target users, the error rate of matching faces and limbs of different target users is reduced, and the matching degree between target users and collected gesture information is improved. This improves the efficiency of gesture-triggered AI real-time image generation and interaction. Matching and locking between sensor units and target users improves the accuracy of information collection by sensor units, improves the matching degree between target users and collected gesture information, and further improves the efficiency of gesture-triggered AI real-time image generation and interaction. 2. By creating a 3D digital twin model of the user, the system captures real-time images of the target user using a high-definition camera, locates the target user's position in real-time using a positioning sensor and adjusts the position using a barometer to obtain real-time location information, and detects the target user's motion state using an inertial measurement unit to obtain motion state information. The system then fuses the real-time image data, real-time location information, and motion state information to reconstruct the target user's real-time state in 3D. This reduces the error rate of gesture recognition when the target user's angle relative to the camera is off, improves the accuracy of gesture acquisition and recognition, and ultimately enhances the efficiency of gesture-triggered AI real-time image generation and interaction. Attached Figure Description

[0015] Figure 1 This embodiment mainly illustrates the process of AI real-time image generation and interaction method based on gesture triggering; Figure 2 This embodiment mainly illustrates the module diagram of the gesture-triggered AI real-time image generation and interaction device.

[0016] Attached image labels: 1. Preliminary preparation module; 2. Dynamic acquisition module; 3. Gesture recognition module; 4. Image generation and output module; 5. Feedback and optimization module. Detailed Implementation

[0017] The present application will be further described in detail below with reference to the accompanying drawings.

[0018] This application discloses a method for real-time AI image generation and interaction based on gesture triggering.

[0019] Reference Figure 1 A gesture-triggered AI real-time image generation and interaction method includes the following steps: Reference Figure 1 Step S1 involves acquiring a sensor unit and establishing a feature connection link between the sensor unit and the target user. Based on the sensor unit, feature data is collected from the target user and features are extracted to obtain target user feature information. Based on the target user feature information and the feature connection link, the sensor unit and the target user are matched and locked, and a pre-preparation completion signal is output. Step S1 specifically includes the following sub-steps: Step A1: Obtain the sensor unit, which includes a high-definition camera.

[0020] Step A2: Establish a signal connection between the sensor unit and the target user to obtain the characteristic connection link.

[0021] Step A3: Obtain full-body image information of the target user by taking a full-body surround shot using a high-definition camera.

[0022] Step A4: Based on the user's full-body image acquisition information, feature extraction is performed on the target user's face to obtain the user's facial feature information, and based on the user's full-body image acquisition information, feature extraction is performed on the target user's limbs to obtain the user's limb feature information.

[0023] Step A5: The user's facial features and body features are combined to form the target user's features.

[0024] Step S1 further includes the following sub-steps: Step B1: When there are multiple target users, the facial feature information of the user is matched with the limb feature information based on the user's full-body image acquisition information to create a facial-limb link and output a facial-limb matching completion signal.

[0025] In practical applications, when there is no overlap between the limbs of different target users, the facial feature information of the target user is directly matched with the limb feature information.

[0026] When there is overlap between the limbs of different target users, the distinguishing features of the limbs of different target users are determined based on the information obtained from the full-body image of the user, and the distinguishing feature information of the limbs between users is obtained. For example, the distinguishing feature information of the limbs between users includes skin color, size, appearance shape, moles and scars.

[0027] By combining the body distinguishing features between users, the facial features of different target users are matched with their body features.

[0028] Specifically, when there are multiple target users, this application embodiment reduces the error rate of matching the faces and limbs of different target users by matching the facial features and limb features of the target users, improves the matching degree between the target users and the collected gesture information, and thus improves the efficiency of AI real-time image generation and interaction based on gesture triggering.

[0029] Step B2: Based on the target user's feature information and the feature connection link, when there is only one target user, the sensor unit tracks the target user in real time. When there are multiple target users, information from multiple target users is collected simultaneously, and a user sensor lock completion signal is output. Specifically, when there are multiple target users, the focal length can be reduced to expand the shooting field, allowing for the simultaneous acquisition of image information from multiple target users.

[0030] Specifically, this application embodiment improves the accuracy of information collection by matching and locking the sensor unit with the target user, increases the matching degree between the target user and the collected gesture information, and thus improves the efficiency of gesture-triggered AI real-time image generation and interaction.

[0031] Step B3: After receiving the facial and limb matching completion signal and the user sensor locking completion signal, output the preliminary preparation completion signal.

[0032] Reference Figure 1 Step S2 involves receiving the preparatory completion signal, querying the target user for their gesture interaction method to obtain the interaction options, creating a 3D user digital twin model, performing real-time detection of the target user based on sensor units to obtain multi-source sensor data, performing 3D reconstruction of the target user's real-time state based on the multi-source sensor data to obtain 3D user real-time dynamic information, and synchronously displaying the 3D user real-time dynamic information on the 3D user digital twin model. Step S2 specifically includes the following sub-steps: Step C1: After receiving the signal that the preliminary preparation is complete, ask the target user if there is a deaf or mute user. If there is a deaf or mute user, ask the target user whether to perform basic gesture interaction or sign language gesture interaction, and output the interaction option output result.

[0033] Step C2, the sensor unit also includes an inertial measurement unit, a barometer, a positioning sensor, a radio unit, and an eye tracker.

[0034] Step C3: Establish a signal connection link between the inertial measurement unit and the target user, and establish a signal connection link between the barometer and the target user.

[0035] Step C4: Based on the user's full-body image acquisition information, create a digital twin of the target user in three-dimensional space to obtain a three-dimensional user digital twin model.

[0036] Step C5: Obtain real-time shooting information of the target user by taking multi-angle shots using a high-definition camera.

[0037] Step C6: The target user's location is located in real time using a positioning sensor to obtain the user's location information. The target user's specific location altitude is further detected using a barometer. The user's location information is then adjusted to obtain the user's real-time location information.

[0038] Step C7: The motion state of the target user is detected by the inertial measurement unit to obtain the user's motion state information.

[0039] Step S2 further includes the following sub-steps: Step D1: When there are non-deaf and mute users among the target users, start the sound receiving unit to collect sound data, obtain the historical sound feature information of the target users, match the sound collection data with the historical sound feature information of the target users, and filter out the sound of the target users in the sound collection data to obtain the real-time sound reception information of the users.

[0040] Step D2 involves using real-time user voice data to determine if the target user is dissatisfied with the gesture-triggered AI real-time video generation and interaction. If so, this is recorded as the first area for optimization, facilitating real-time user experience data collection for timely feedback and optimization. For example, if the real-time user voice data shows "inaccurate recognition" or "data delay," it indicates dissatisfaction with the gesture-triggered AI real-time video generation and interaction. Therefore, "inaccurate recognition" and "data delay" are recorded and marked as the first area for optimization.

[0041] Step D3: Real-time eye movement information of the target user is obtained by real-time detection based on the eye tracker.

[0042] Step D4: The user's real-time shooting information, user's real-time location information, user's motion status information, user's real-time sound reception information, and user's real-time eye movement information are combined to form multi-source sensor data.

[0043] Step D5: Based on the user's real-time shooting information, real-time location information, and motion status information, create the real-time dynamics of the target user in three-dimensional space, obtain the real-time dynamic information of the three-dimensional user, and display the real-time dynamic information of the three-dimensional user on the three-dimensional user digital twin model synchronously.

[0044] Reference Figure 1 Step S3 involves creating a gesture comparison database and determining the target user's corresponding gesture comparison database. Based on the real-time dynamic information of the 3D user, the target user's gesture actions are filtered out to obtain real-time user gesture data. This data is then compared and identified with the corresponding user gesture comparison database to obtain the user's gesture intent information. Step S3 specifically includes the following sub-steps: Step S31: Create a comparison gesture database, which includes a basic comparison gesture database and a sign language comparison gesture database.

[0045] Step S32: If there are deaf and mute users in the output results of the interaction options, then sign language gesture interaction is performed, and the deaf and mute users in the target users select the sign language comparison gesture database. If there are no deaf and mute users in the output results of the interaction options, then basic gesture interaction is performed, and the target users select the basic comparison gesture database. The corresponding gesture comparison database for the user is output.

[0046] If there are multiple target users, including deaf and mute users and non-deaf and mute users, then the deaf and mute users will be compared using a sign language comparison database, while the non-deaf and mute users will be compared using a basic comparison gesture database.

[0047] Step S33: Based on the real-time dynamic information of the three-dimensional user, filter out the gesture actions of the target user to obtain the real-time gesture data of the user.

[0048] Step S34: Compare and identify the user's real-time gestures with the corresponding gesture comparison database to obtain the user's gesture intent information.

[0049] Reference Figure 1 Step S4 involves acquiring the AI ​​model by inputting user gesture intent information and multi-source sensor data into the AI ​​model to generate target image generation information, which is then displayed in real-time on a pre-set display screen. Step S4 specifically includes the following sub-steps: Step S41: Obtain the AI ​​model, which includes a local lightweight AI model and a cloud AI model. Establish a signal connection between the local lightweight AI model and the target user, and establish a signal connection between the cloud AI model and the target user.

[0050] Step S42: If the target user is a deaf-mute user, the user's gesture intention information and the user's real-time eye movement information are sent to the local lightweight AI model to generate the content required by the deaf-mute user locally, and the first locally generated information is obtained.

[0051] Step S43: Simultaneously send the user's gesture intention information and the user's real-time eye movement information to the cloud AI model to generate the content required by the deaf and mute user in the cloud, and obtain the first cloud-generated information.

[0052] Step S44: Based on data fusion technology, the first locally generated information and the first cloud-generated information are fused to obtain content generated by deaf and mute users.

[0053] Step S45: If the user is not deaf or mute, the user's gesture intention information, real-time sound reception information, and real-time eye movement information are sent to the local lightweight AI model to generate the content required by the non-deaf or mute user locally, thus obtaining the second locally generated information.

[0054] Step S46: Simultaneously send the user's gesture intention information, the user's real-time sound reception information, and the user's real-time eye movement information to the cloud AI model to generate the content required by the non-deaf and mute user in the cloud, and obtain the second cloud-generated information.

[0055] Step S47: Based on data fusion technology, the first locally generated information and the first cloud-generated information are fused to obtain content generated by non-deaf and mute users.

[0056] Step S48: The content generated by deaf and mute users is combined with the content generated by non-deaf and mute users to form target image generation information.

[0057] Step S49: Display the target image generation information in real time on a pre-set display screen.

[0058] Reference Figure 1 Step S5 involves optimizing the first set of content information to be optimized based on multi-source sensor data, providing a feedback questionnaire to the target users to collect user opinions, and obtaining the second set of content information to be optimized. Based on the first and second sets of content information to be optimized, the gesture-triggered AI real-time image generation and interaction are optimized. Step S5 specifically includes the following sub-steps: Step S51: Distribute feedback questionnaires to target users, collect their feedback after they are filled out, and summarize their opinions to obtain the second set of information to be optimized.

[0059] Step S52: The first content information to be optimized and the second content information to be optimized are combined to form a feedback optimization point.

[0060] Step S53: Optimize the gesture-triggered AI real-time image generation and interaction based on feedback optimization points.

[0061] This application also discloses an AI real-time image generation and interaction device based on gesture triggering.

[0062] Reference Figure 2 The gesture-triggered AI real-time image generation and interaction device includes: The preliminary preparation module is configured to acquire sensor units, establish a link between the sensor units and the target user to obtain a feature connection link, collect data and extract features from the target user based on the sensor units to obtain target user feature information, match and lock the sensor units and the target user based on the target user feature information and the feature connection link, and output a preliminary preparation completion signal.

[0063] The dynamic acquisition module is configured to, upon receiving a signal indicating completion of preliminary preparations, inquire about the target user's gesture interaction method to obtain the interaction options and output the results. It then creates a 3D digital twin model of the user, performs real-time detection of the target user based on sensor units to obtain multi-source sensor data, performs 3D reconstruction of the target user's real-time state based on the multi-source sensor data to obtain 3D real-time dynamic information of the user, and synchronously displays the 3D real-time dynamic information of the user on the 3D digital twin model of the user.

[0064] The gesture recognition module is configured to create a comparison gesture database and determine the corresponding gesture comparison database of the target user. Based on the real-time dynamic information of the three-dimensional user, it filters out the gesture actions of the target user, obtains the user's real-time gesture data, and compares and recognizes it with the corresponding gesture comparison database to obtain the user's gesture intention information.

[0065] The image generation and output module is configured to acquire an AI model, input user gesture intent information and multi-source sensor data into the AI ​​model, generate target image generation information, and display it in real time on a pre-set display screen.

[0066] The feedback optimization module is configured to perform feedback optimization based on multi-source sensor data to obtain the first content information to be optimized, provide a feedback questionnaire to the target user to collect user opinions and obtain the second content information to be optimized, and optimize the AI ​​real-time image generation and interaction based on gesture triggering based on the first and second content information to be optimized.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or systems. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This invention is described with reference to flowchart illustrations and structural diagrams of methods and systems according to embodiments of the invention. It should be understood that the combination of each process and module in the flowchart and structural diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 One or more processes and structures Figure 1 A device for a function specified in one or more modules.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and structures Figure 1 The function specified in one or more modules.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and structures Figure 1 The steps of a specified function in one or more modules.

[0071] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0072] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A gesture-triggered AI real-time image generation and interaction method, characterized in that: Includes the following steps: Step S1: Obtain the sensor unit and establish a link between the sensor unit and the target user to obtain the feature connection link. Based on the sensor unit, collect data and extract features from the target user to obtain the target user feature information. Based on the target user feature information and the feature connection link, match and lock the sensor unit and the target user, and output the preliminary preparation completion signal. Step S2: After receiving the signal that the preliminary preparation is complete, inquire about the target user's gesture interaction method to obtain the interaction option output result, create a three-dimensional user digital twin model, perform real-time detection of the target user based on the sensor unit to obtain multi-source sensor data, perform 3D reconstruction of the target user's real-time state based on the multi-source sensor data to obtain three-dimensional user real-time dynamic information, and synchronously display the three-dimensional user real-time dynamic information on the three-dimensional user digital twin model. Step S3: Create a gesture comparison database and determine the target user's corresponding gesture comparison database. Based on the three-dimensional real-time dynamic information of the user, filter out the target user's gesture actions to obtain the user's real-time gesture data, and compare and identify it with the user's corresponding gesture comparison database to obtain the user's gesture intention information. Step S4: Obtain the AI ​​model by inputting the user's gesture intent information and multi-source sensor data into the AI ​​model to generate target image generation information and display it in real time on a pre-set display screen. Step S5: Based on multi-source sensor data, feedback optimization is performed to obtain the first content information to be optimized. A feedback questionnaire is provided to the target user to collect user opinions and obtain the second content information to be optimized. Based on the first and second content information to be optimized, the gesture-triggered AI real-time image generation and interaction are optimized.

2. The gesture-triggered AI real-time image generation and interaction method according to claim 1, characterized in that, Step S1 specifically includes: Acquisition sensor unit, the sensor unit including a high-definition camera; Establishing a signal connection between the sensor unit and the target user yields a characteristic connection link; The high-definition camera is used to capture the target user's full-body image information. Facial feature information is obtained by extracting features from the target user's face based on the user's full-body image acquisition information, and limb feature information is obtained by extracting features from the target user's limbs based on the user's full-body image acquisition information. The user's facial features and the user's limb features are combined to form the target user's features.

3. The gesture-triggered AI real-time image generation and interaction method according to claim 2, characterized in that, Step S1 further includes: When there are multiple target users, the facial feature information of the user is matched with the limb feature information based on the user's full-body image acquisition information, a facial-limb link is created, and a facial-limb matching completion signal is output. Based on the target user's characteristic information and characteristic connection links, when there is only one target user, the sensor unit is used to track the target user in real time. When there are multiple target users, the information of multiple target users is collected simultaneously, and a user sensing lock completion signal is output. Upon receiving the facial and limb matching completion signal and the user sensor locking completion signal, the system outputs a preliminary preparation completion signal.

4. The gesture-triggered AI real-time image generation and interaction method according to claim 3, characterized in that, Step S2 specifically includes: Upon receiving the signal indicating completion of the preliminary preparations, the system queries whether there are any deaf or mute users among the target users. If there are, the system queries whether the target user should perform basic gesture interaction or sign language gesture interaction, and outputs the interaction option result. The sensor unit also includes an inertial measurement unit, a barometer, a positioning sensor, a radio unit, and an eye tracker; Establish a signal connection link between the inertial measurement unit and the target user, and establish a signal connection link between the barometer and the target user; Based on the user's full-body image acquisition information, a digital twin of the target user in three-dimensional space is created to obtain a three-dimensional user digital twin model; The high-definition camera is used to capture the target user from multiple angles to obtain real-time shooting information of the user. The user's location information is obtained by real-time positioning of the target user based on the positioning sensor, and the user's specific location altitude is further detected based on the barometer. The user's location information is then adjusted to obtain the user's real-time location information. The motion state information of the target user is obtained by detecting the motion state of the target user using an inertial measurement unit.

5. The AI ​​real-time image generation and interaction method based on gesture triggering according to claim 4, characterized in that, Step S2 further includes: When there are non-deaf and mute users among the target users, the sound receiving unit is activated to collect sound data, obtain the historical sound feature information of the target users, match the sound collection data with the historical sound feature information of the target users, and filter out the sound of the target users in the sound collection data to obtain the real-time sound reception information of the users. Based on the real-time voice information received by the user, it is determined in real time whether there is any dissatisfaction in the target user's voice with the gesture-triggered AI real-time image generation and interaction. If so, the first content information to be optimized is recorded. Real-time eye movement information of the target user is obtained by real-time detection using an eye tracker; The user's real-time shooting information, the user's real-time location information, the user's motion status information, the user's real-time sound reception information, and the user's real-time eye movement information are combined to form multi-source sensor data; Based on real-time user shooting information, real-time user location information, and user movement status information, a real-time dynamic of the target user in three-dimensional space is created to obtain three-dimensional user real-time dynamic information, and the three-dimensional user real-time dynamic information is synchronously displayed on the three-dimensional user digital twin model.

6. The gesture-triggered AI real-time image generation and interaction method according to claim 5, characterized in that, Step S3 specifically includes: Create a comparison gesture database, which includes a basic comparison gesture database and a sign language comparison gesture database; If there are deaf and mute users in the output results of the interaction options, then sign language interaction will be performed. The deaf and mute users in the target users will select the sign language comparison gesture database. If there are no deaf and mute users in the output results of the interaction options, then basic gesture interaction will be performed. The target users will select the basic comparison gesture database and the corresponding gesture comparison database for the user will be output. Based on real-time dynamic information of three-dimensional users, the gestures of target users are filtered out to obtain real-time gesture data of users. The user's real-time gestures are compared and identified with the corresponding user gesture comparison database to obtain the user's gesture intent information.

7. The gesture-triggered AI real-time image generation and interaction method according to claim 6, characterized in that, Step S4 specifically includes: Acquire an AI model, which includes a local lightweight AI model and a cloud-based AI model; establish a signal connection between the local lightweight AI model and the target user; and establish a signal connection between the cloud-based AI model and the target user. If the target user is a deaf or mute user, the user's gesture intention information and real-time eye movement information are sent to the local lightweight AI model to generate the content required by the deaf or mute user locally, thus obtaining the first locally generated information. Simultaneously, user gesture intent information and real-time eye movement information are sent to the cloud AI model to generate the content required by the deaf and mute user in the cloud, thus obtaining the first cloud-generated information. Based on data fusion technology, the first locally generated information and the first cloud-generated information are fused to obtain content generated by deaf and mute users; If the user is not deaf or mute, the user's gesture intention information, real-time sound reception information, and real-time eye movement information are sent to the local lightweight AI model to generate the content required by the non-deaf or mute user locally, thus obtaining the second locally generated information. Simultaneously, user gesture intent information, real-time user audio information, and real-time user eye movement information are sent to the cloud AI model to generate the content required by non-deaf and mute users in the cloud, thus obtaining the second cloud-generated information. Based on data fusion technology, the first locally generated information and the first cloud-generated information are fused to obtain content generated by non-deaf and mute users; The content generated by the deaf and mute users is combined with the content generated by the non-deaf and mute users to form target image generation information; The target image generation information is displayed in real time on a pre-set display screen.

8. The AI ​​real-time image generation and interaction method based on gesture triggering according to claim 7, characterized in that, Step S5 specifically includes: Feedback questionnaires were distributed to target users, who filled them out and collected their feedback. User opinions were then analyzed to obtain information on the second area for optimization. The first information to be optimized and the second information to be optimized are combined to form a feedback optimization point; The operation of gesture-triggered AI real-time image generation and interaction is optimized based on feedback optimization points.

9. A gesture-triggered AI real-time image generation and interaction device, characterized in that, The gesture-triggered AI real-time image generation and interaction device is used to implement the gesture-triggered AI real-time image generation and interaction method according to any one of claims 1-8, including: The preliminary preparation module is configured to acquire sensor units, establish a link between sensor units and target users to obtain feature connection links, collect data and extract features from target users based on sensor units to obtain target user feature information, match and lock sensor units and target users based on target user feature information and feature connection links, and output a preliminary preparation completion signal. The dynamic acquisition module is configured to, upon receiving a signal indicating completion of preliminary preparation, inquire about the target user's gesture interaction method to obtain the interaction options output result, create a three-dimensional user digital twin model, perform real-time detection of the target user based on sensor units to obtain multi-source sensor data, perform 3D reconstruction of the target user's real-time state based on multi-source sensor data to obtain three-dimensional user real-time dynamic information, and synchronously display the three-dimensional user real-time dynamic information on the three-dimensional user digital twin model. The gesture recognition module is configured to create a comparison gesture database and determine the user's corresponding gesture comparison database. Based on the real-time dynamic information of the three-dimensional user, the module filters out the gesture actions of the target user, obtains the user's real-time gesture data, and compares and recognizes it with the user's corresponding gesture comparison database to obtain the user's gesture intention information. The image generation and output module is configured to acquire an AI model, input user gesture intent information and multi-source sensor data into the AI ​​model, generate target image generation information, and display it in real time on a pre-set display screen. The feedback optimization module is configured to perform feedback optimization based on multi-source sensor data to obtain the first content information to be optimized, provide a feedback questionnaire to the target user to collect user opinions and obtain the second content information to be optimized, and optimize the AI ​​real-time image generation and interaction based on gesture triggering based on the first and second content information to be optimized.

Images