Robot expression generation method and computer readable storage medium

Abstract

The application discloses a robot expression generation method and a computer readable storage medium, and the method comprises the following steps: collecting video frames of a robot interaction area, and determining candidate interaction targets; performing human body and hand position detection on the candidate interaction targets to obtain human body frames, hand frames, and position information thereof, and predicting target states; performing face detection on the video frames to obtain face frames, and extracting facial expression features; binding the human body frames and the face frames belonging to the same candidate interaction target, and determining a current interaction target from the candidate interaction targets; generating a user state result according to the position information of the current interaction target, the target state, and the facial expression features; generating text prompt content according to the user state result, selecting an eye style category, a reference image template, and eye style constraint parameters; and generating an eye image. The application can fuse interaction target semantics and facial states, and constrain the generation of a unified style of robot double-eye expressions.

Description

Technical Field

[0001] This invention relates to the field of robot interaction technology, and in particular to a method for generating robot facial expressions and a computer-readable storage medium. Background Technology

[0002] Existing methods for controlling robot facial expressions typically include: fixed switching based on a pre-set facial expression library; switching facial expressions based on the results of single facial expression recognition; matching facial expressions based on voice or text keywords; generating facial expression images by calling an image generation model based on manually input prompts; or playing pre-set binocular animation materials.

[0003] The above methods are applicable in simple scenarios, but the following technical problems are likely to occur in real-time robot interaction: single facial expression recognition is greatly affected by occlusion, side profile, backlight, distance and expression amplitude; facial expressions may come from different people in the frame; general image generation models are prone to changing the left and right eye layout, iris texture, highlight position or non-eye areas; there is a lack of regeneration and rollback mechanisms when generation fails; style drift or display incompatibility may occur during continuous output.

[0004] For the current implementation of generating eye images through raw image models, if only the general SDXL base is used, the model usually lacks the sample constraints of the robot's own eye style, and is prone to generating images with real eyes, extra facial features, misaligned highlights, changes to non-eye areas, or inconsistencies with existing eye templates. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a robot expression generation method and a computer-readable storage medium that can integrate the semantics of the interactive target and the facial state to constrain the generation of robot eye expressions with a unified style.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a robot facial expression generation method, comprising: Collect video frames of the robot's interaction area and perform human detection in the video frames to determine candidate interaction targets; Human and hand positions are detected for candidate interactive targets to obtain detection boxes and their position information. The detection boxes include human body boxes and hand boxes. The target state is predicted based on the detection boxes. Face detection is performed on the video frames to obtain face bounding boxes, and facial expression features are extracted based on the face regions within the face bounding boxes. Bind human body bounding boxes and face bounding boxes that belong to the same candidate interaction target, and determine the current interaction target from each candidate interaction target; Based on the location information of the detection box of the current interactive target, the target state, and facial expression features, generate the user state result; Based on the user status result, generate text prompt content and select eye style category, reference image template and eye style constraint parameters; Based on the text prompt content, eye style category, reference image template, and eye style constraint parameters, an eye image is generated, which is either an eye expression image or an eye animation keyframe.

[0007] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0008] The beneficial effects of this invention are as follows: By coarsely selecting the current interaction target, confirming it after feature extraction, and binding the human body bounding box and the face bounding box, the probability of incorrectly merging the actions and expressions of different users in multi-user scenarios is reduced. Through text prompts, image templates, layout masks, structural control charts, and eye style constraint parameters, the user state results are transformed into executable eye image generation conditions, reducing left-right eye misalignment, changes in non-eye regions, and style drift caused by general image generation models. This invention can fuse the semantics of the interaction target with facial states to constrain the generation of robot eye expressions with a unified style. Attached Figure Description

[0009] Figure 1 This is a flowchart of a robot facial expression generation method according to the present invention; Figure 2 This is a flowchart of the method according to Embodiment 1 of the present invention. Detailed Implementation

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] Existing robot eye expression generation processes typically lack a stable mapping from "user multimodal state semantic cues" to "robot eye style constraint generation conditions," as well as model adaptation and detection backoff mechanisms.

[0012] From the perspective of internal system processing, existing technologies often only output a coarse-grained emotion label, or simply input user images, emotion words, or artificial prompts into the image generation model, without clearly solving the following continuous processing problems: how to determine the current interaction target in a multi-person scenario; how to dynamically adjust the fusion weights based on modality quality; and how to convert user state into text prompt content and image template generation conditions.

[0013] To solve at least the above-mentioned technical problems, please refer to Figure 1 This invention provides a method for generating robot facial expressions, including: Collect video frames of the robot's interaction area and perform human detection in the video frames to determine candidate interaction targets; Human and hand positions are detected for candidate interactive targets to obtain detection boxes and their position information. The detection boxes include human body boxes and hand boxes. The target state is predicted based on the detection boxes. Face detection is performed on the video frames to obtain face bounding boxes, and facial expression features are extracted based on the face regions within the face bounding boxes. Bind human body bounding boxes and face bounding boxes that belong to the same candidate interaction target, and determine the current interaction target from each candidate interaction target; Based on the location information of the detection box of the current interactive target, the target state, and facial expression features, generate the user state result; Based on the user status result, generate text prompt content and select eye style category, reference image template and eye style constraint parameters; Based on the text prompt content, eye style category, reference image template, and eye style constraint parameters, an eye image is generated, which is either an eye expression image or an eye animation keyframe.

[0014] As can be seen from the above description, the beneficial effects of the present invention are: it can reduce the probability of incorrectly fusing the actions and expressions of different users in multi-user scenarios, and reduce left and right eye misalignment, changes in non-eye areas, and style drift caused by general image generation models.

[0015] Further, the step of performing face detection on the video frame to obtain a face bounding box, and extracting facial expression features based on the face region within the face bounding box, includes: Face detection is performed on the video frames to obtain face bounding boxes, and facial modality data is calculated; The face region within the face frame is aligned, and facial expression features are extracted based on the aligned face region.

[0016] In some embodiments, the facial expression features include facial expression category or candidate state probability, as well as the degree of eye opening, eyebrow state, corner of mouth curvature, eye direction, and facial quality score.

[0017] As described above, face alignment can improve the accuracy and stability of facial expression feature extraction.

[0018] Furthermore, the facial modal data includes face confidence, face landmark confidence, facial sharpness, illumination score, occlusion score, and head pose quality score. The step of performing face detection on the video frame to obtain face bounding boxes and calculating facial modality data includes: Face detection is performed on the video frames to obtain face bounding boxes and face confidence scores, and face landmark detection is performed on the face bounding boxes to obtain face landmarks and face landmark confidence scores. The face region within the face frame is converted to a grayscale image. Based on the grayscale image and the Laplacian operator, an edge response map is calculated, and the Laplacian variance is calculated based on the edge response map. Calculate facial sharpness based on the Laplace variance; The illumination score is calculated based on the average brightness, brightness variance, and the proportion of overexposed and underexposed pixels in the face region. The occlusion score is calculated based on the proportion of missing facial landmarks or the proportion of the visible facial area. Based on the geometric proportions of facial key points, the head pose angle is calculated, and based on the head pose angle, the head pose quality score is calculated.

[0019] As described above, calculating facial modal data facilitates the subsequent calculation of facial modal quality scores.

[0020] Furthermore, before generating the user state result based on the detection box position information of the current interaction target, the target state, and facial expression features, the process further includes: Based on the multimodal data of the current interactive target, the modal quality score of the current interactive target is calculated, and the fusion weight is calculated based on the modal quality score. The multimodal data includes human modal data and facial modal data. The human modal data is determined based on the human bounding box of the current interactive target in consecutive frames. The modal quality score includes the target semantic modal quality score and the facial modal quality score. The fusion weight includes the target semantic modal weight and the facial modal weight. The step of generating a user state result based on the detection box position information of the current interactive target, the target state, and facial expression features includes: The user status result is generated based on the location information of the detection box of the current interactive target, the target state, facial expression features, modal quality score and fusion weight.

[0021] As described above, by using the target semantic modality quality score, facial modality quality score, and fusion weights, the modality contribution is dynamically adjusted when there is face occlusion, side profile, backlighting, or missing human body detection boxes, thereby improving the stability of user state result generation.

[0022] Further, the step of calculating the modal quality score of the current interaction target based on the multimodal data of the current interaction target, and calculating the fusion weight based on the modal quality score, includes: Based on the human modality data of the current interaction target, the target semantic modality quality score is calculated. The human modality data includes the average confidence score of the human bounding box in consecutive frames, the proportion of effective frames, the stability of the human bounding box, and the stability of the trajectory across frames. Based on the facial modal data of the current interactive target, calculate the facial modal quality score. The facial modal data includes face detection confidence, facial landmark confidence, facial sharpness, illumination score, occlusion score, and head pose quality score. Calculate the target semantic modality weight and the facial modality weight based on the target semantic modality quality score and the facial modality quality score.

[0023] Furthermore, the binding of human body bounding boxes and face bounding boxes belonging to the same candidate interaction target includes: Based on the preset binding conditions, calculate the binding score between the human bounding box and the face bounding box of each candidate interactive target; If the binding score between a human body frame and a face frame is greater than a preset first score threshold, then the human body frame and the face frame are bound together. The binding conditions include upper region conditions, center distance conditions, containment relationship conditions, size ratio conditions, and cross-frame consistency conditions. The condition for the upper region is that the center point of the face frame is located in the upper region of the human body frame. The center distance condition is that the normalized distance between the center point of the face frame and the center point of the head and shoulder region of the human body frame is less than a preset distance threshold; in some embodiments, the distance threshold is 0.35 times the width of the human body frame. The inclusion relationship condition is that the face bounding box and the body bounding box have an inclusion relationship or an overlapping relationship; The size ratio condition is that the ratio of the face frame area to the body frame area is within a preset ratio range; in some embodiments, the ratio range is 0.02-0.25. The cross-frame consistency condition is that in a series of video frames with a preset number of frames, the center displacement direction and velocity change of the same human body frame trajectory and the same face frame trajectory remain consistent.

[0024] As described above, calculating the binding score to determine whether the target human body and face belong to the same person can improve the accuracy of the judgment and reduce the probability of incorrectly merging the actions and expressions of different users in multi-person scenarios. When the binding score is greater than the first score threshold (e.g., 0.6), it is considered a reliable binding; when the binding score is between the first score threshold (e.g., 0.6) and the second score threshold (e.g., 0.4), it is further confirmed by combining the previous round of interaction objects or task context.

[0025] Furthermore, before generating the user state result based on the detection box location information of the current interaction target, the target state, facial expression features, modality quality score, and fusion weight, the process further includes: If the binding scores between the human body bounding box and each face bounding box of the current interactive target are all less than the preset second score threshold, then the facial modality weight is reduced.

[0026] As described above, when the binding score is lower than the second score threshold (e.g., 0.4), it is determined that the binding cannot be reliably performed. If the binding cannot be reliably performed, the facial modality weight is reduced, or a low-intensity user state result is generated based solely on the target semantic prediction of the person's state.

[0027] Furthermore, after generating the eye image based on the text prompt content, eye style category, reference image template, and eye style constraint parameters, the process further includes: The eye image is subjected to consistency detection according to preset consistency detection conditions; If the consistency check fails, the eye image is regenerated; if the consistency check passes, the eye image is adapted for the robot display and output to the robot display. The consistency detection conditions include hard detection conditions and scoring detection conditions; The hard detection conditions include: the number of eyes is 2; the overlap between the left and right eye regions and the eye regions in the reference image template is greater than or equal to a preset overlap threshold; the generated eye region does not exceed the editable region defined by the layout mask; the change ratio of the non-eye protection region is less than or equal to a preset change ratio threshold; the screen border region and the robot's fixed appearance region are not changed; no abnormal content appears; and the resolution and clarity of the eye image meet the minimum requirements of the robot's display end. The abnormal content includes text, watermarks, extra eyes, extra faces, noses, and mouths. The consistency detection score for the scoring detection conditions is greater than or equal to a preset third score threshold. The consistency detection score is calculated based on roundness score, left-right eye symmetry score, template similarity score, color deviation score, highlight position deviation score, and sharpness score. The roundness score is calculated based on the ratio of the eye contour area to the area of ​​the smallest circumscribed circle. The left-right eye symmetry score is calculated based on the center position, area, aspect ratio, and contour similarity of the left and right eyes. The template similarity score is calculated based on the structural similarity or feature similarity between the eye image and the reference image template. The color deviation score is calculated based on the color difference between the main color of the iris region in the eye image and the main color of the iris region in the reference image template. The highlight position deviation score is calculated based on the offset ratio of the highlight center relative to the iris center. The sharpness score is calculated based on the Laplacian variance or edge intensity.

[0028] As described above, consistency detection ensures that the robot will not output obviously unqualified eye images during real-time interaction.

[0029] Furthermore, it also includes: If the number of consecutive regenerations reaches the preset upper limit threshold, or the time taken for consistency detection and image generation exceeds the preset time threshold, then the closest eye template image is selected from the preset template library.

[0030] As described above, by limiting the number of regeneration attempts / timeouts and using a rollback template library, we can ensure that the robot will not retry indefinitely during real-time interaction, nor will it output obviously unqualified eye images.

[0031] Further, predicting the target state based on the detection box includes: Based on the human body frame and hand frame, structured spatial features are calculated. The structured spatial features include the center coordinates of the human body frame, the area of ​​the human body frame, the offset of the human body center relative to the center of the image, the center coordinates of the hand frame, the position of the hand frame relative to the human body frame, and the displacement velocity of the hand frame in consecutive frames. If the center coordinates of the hand frame are located in the upper region of the human body frame, and the lateral or vertical displacement of the hand in consecutive frames exceeds a preset displacement threshold, then the predicted target state is a waving state, a raised hand state, or an active interaction state; wherein, the upper region of the human body frame is the range of the human body frame from 0% to a preset percentage in the height direction; in some embodiments, the preset percentage is 45%; If the offset of the human body center relative to the center of the screen decreases and the area of ​​the human body frame increases in consecutive frames, the predicted target state is either close to the robot or in an interactive state. If the offset of the human body center relative to the center of the screen increases and the area of ​​the human body frame decreases in consecutive frames, the predicted target state is either a state of moving away from the robot or a state of exiting interaction.

[0032] Further, determining the current interaction target from among the candidate interaction targets includes: Based on the interaction information of each candidate interaction target, the interaction score of each candidate interaction target is calculated. The interaction information includes the center offset of the screen, the area of ​​the human body frame, the target distance, the trajectory stability, whether it is the interaction object of the previous round, whether there is a gaze at the robot direction, whether there is a waving or raising action, whether it is reliably bound to the face frame, and whether the task context points to the candidate interaction target. If the highest interaction score is greater than the preset score threshold (e.g., 0.6), and the difference between the highest interaction score and the second highest interaction score is greater than the preset score difference threshold (e.g., 0.15), then the candidate interaction target corresponding to the highest interaction score is taken as the current interaction target; otherwise, the preset neutral eye template is output.

[0033] As described above, if a unique current interaction target can be identified, only the location information, state assessment, and facial expression features corresponding to that target will be input into the subsequent fusion process; if a unique target cannot be identified, a neutral safe gaze will be output or the process will revert to a preset neutral eye template.

[0034] Further, based on the user state result, text prompt content is generated, and an eye style category, reference image template, and eye style constraint parameters are selected; based on the text prompt content, eye style category, reference image template, and eye style constraint parameters, an eye image is generated, wherein the eye image is an eye expression image or an eye animation keyframe, including: Based on the user status result, a text prompt is generated, which includes the target eye status, expression intensity, editable area, and negative constraints. Based on the user status result, select an eye style category from the preset style categories, select a reference image template from the preset image templates, and select eye style constraint parameters; A layout mask is generated based on the reference image template and the robot display end coordinate system, and a structure control map is generated based on the reference image template or the eye structure map of the current interactive target. An eye image is generated based on the text prompt, eye style category, reference image template, eye style constraint parameters, layout mask, and structure control diagram.

[0035] As described above, by fusing the facial expression features of the interactive target and preset expressions in the video frames collected by the robot, a user state result is formed, and the robot's eye expression image or eye expression animation keyframe is generated based on the text prompt content, reference image template, eye style constraint parameters, layout mask and structural control information.

[0036] Furthermore, an eye image is generated using an image generation model, wherein the eye style constraint parameters are determined based on LoRA weights, which are obtained by training the image generation model.

[0037] As described above, by fusing the facial expression features of the interactive target and preset expressions in the video frames collected by the robot, a user state result is formed. Based on the user state result, text prompts, image templates, layout masks, structural control information, and eye style constraint parameters are generated. Then, the style-constrained image generation model is called to generate an eye expression image or eye expression animation keyframe that matches the semantic cues of the user state and conforms to the robot's preset eye style.

[0038] Furthermore, before generating the eye image using the image generation model, the process further includes: A robot eye style dataset is constructed, which includes a training subset corresponding to each eye style category. Each sample image in the eye style dataset is labeled, and the labeling information includes eye style category, style trigger word, eye state label and template label. The sample image is cropped or scaled according to a preset size, and the sample image is then cleaned. Based on the annotation information of the sample images, training prompts are generated; The image generation model is trained according to each training subset to obtain the LoRA weights corresponding to each eye style category; The image generation model includes a dual-text conditional encoder, an image latent space encoder, a noise-adding network, a denoising network, and a LoRA branch. The denoising network includes a downsampling module, an intermediate module, and an upsampling module, each of which includes an attention projection layer. The LoRA branch includes a LoRA dimensionality reduction matrix and a LoRA dimensionality increase matrix. The LoRA branch is connected to the attention projection layers in the downsampling, intermediate, and upsampling modules of the denoising network, respectively. The dual text conditional encoder is used to encode the training prompt content to obtain text conditional features; The latent space encoder is used to encode the sample image to obtain a target latent space representation; The noisy network is used to add random noise (such as Gaussian noise) to the target latent space representation according to a preset training time step to obtain a noisy latent space representation; The denoising network is used to output predicted noise based on the training time step, the noisy latent space representation, and the text conditional features; Specifically, when training the image generation model, backpropagation is performed using the diffusion training loss between the predicted noise and the real noise as the optimization objective to update the weight parameters of the LoRA branch.

[0039] In some embodiments, the image generation model further includes an image latent space decoder for restoring the final latent space features into a pixel image that the user can see.

[0040] As described above, by using SDXL-LoRA multi-style adaptation training, the robot's own eye style can be adapted without fully fine-tuning the basic image generation model, thereby improving the consistency of iris texture, pupil ratio, highlight position, and left and right eye layout.

[0041] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0042] Example 1 Please refer to Figure 2 Embodiment 1 of the present invention is: a robot expression generation method based on emotion perception. This method is mainly applicable to robot eye displays that only show the left and right eyes, robots that only update the eye area in the complete face screen, and robots that overlay eye patches or eye animations on the robot face display interface. It can also be used for interactive robots such as children's companions, tour guides, customer service, education, family companions, exhibition hall interaction, and shopping mall greeters.

[0043] This method is based on a robot facial expression generation system and includes: Video acquisition module: Acquires video frames from the robot's interactive area and records the frame number and timestamp.

[0044] Candidate target coarse selection module: detects candidate body instances and performs coarse selection of the current interaction target based on low-cost information.

[0045] Facial expression feature extraction module: detects and aligns the face region to extract facial expression features.

[0046] Quality scoring and fusion module: Evaluates the reliability of target semantics and facial modalities, and fuses them to form user status results.

[0047] Structured eye cue word generation module: Generates text cue content based on user status results.

[0048] Eye Style Library and Template Selection Module: Selects eye style categories and image templates based on user status results.

[0049] Layout mask and control map generation module: Generates editable areas for the left and right eyes, non-editable protected areas, eye edge maps, line art maps, or structural control maps.

[0050] Eye style constraint image generation module: Generates eye expression images or animation keyframes based on text prompts, image templates, layout masks, control charts, and style constraint parameters.

[0051] Consistency Detection and Rollback Module: Determines whether the generated result meets the robot display constraints, and regenerates or rolls back if it does not meet the requirements.

[0052] Display adaptation and optional image enhancement module: Performs display adaptation on qualified results in terms of size, brightness, color, frame rate, etc., and performs enhancement and re-inspection when necessary.

[0053] like Figure 2 As shown, this method includes the following steps: S1: Collect video frames of the robot's interaction area and perform human detection in the video frames to determine candidate interaction targets.

[0054] The robot's camera module captures video frames of the interactive area and records the frame number and timestamp. The system detects candidate body instances in the frame and obtains low-cost information such as the candidate body bounding box, body center, and body bounding box area. In this step, the system only performs coarse selection based on the body bounding box area, frame center offset, target bounding box continuity stability, target distance, or the previous round of interaction objects, without using semantic features such as "waving" or "looking at the robot," which require subsequent extraction of body and hand positions or facial features for reliable results.

[0055] S2: Perform human body and hand position detection on candidate interactive targets to obtain detection boxes and their position information. The detection boxes include human body boxes and hand boxes. Based on the detection boxes, predict the target state.

[0056] The location information extraction module performs human body and hand position detection on the candidate body or the current interactive target, and obtains the human body bounding box and its confidence score, and the hand bounding box and its confidence score.

[0057] Then, the system predicts state features based on the detection bounding boxes. Specifically, the system first calculates structured spatial features based on the human body bounding box and the hand bounding box, including the center coordinates of the human body bounding box, the area of ​​the human body bounding box (or the proportion of the human body bounding box area), the aspect ratio of the human body bounding box, the offset of the human body center relative to the center of the image, the center coordinates of the hand bounding box, the position of the hand bounding box relative to the human body bounding box, the relative height of the hand bounding box and the head region, whether the left and right hands are higher than the head, and the displacement velocity of the hand bounding box in consecutive frames.

[0058] In this embodiment, the human body bounding box feature B p =(x p ,y p ,w p ,h p ), Hand frame feature B h =(x h ,y h ,w h ,h h ), where (x p ,y p) represents the coordinates of the center of the human body bounding box, w p and h p The width and height of the human body frame; (x h ,y h ) represents the center coordinates of the hand frame, w h and h h Let be the width and height of the hand frame. At this point, the normalized lateral position r of the hand frame center relative to the human body frame center is... x (t)=(x h (t)-x p (t)) / w p (t), the normalized vertical position r of the hand frame center relative to the human body frame center. y (t)=(y h (t)-y p (t)) / h p (t). In the image coordinate system, with the downward direction as the positive Y-axis, the image's ordinate gradually increases from top to bottom. Therefore, when r y If (t) is less than or equal to the preset upper region threshold, then the center of the hand frame is determined to be located in the upper region of the human body frame. In this embodiment, the upper region of the human body is defined as the range of 0%-45% from top to bottom in the height direction of the human body frame. At this time, the upper region threshold can be set to -0.05.

[0059] Further calculate the normalized displacement of the hand in consecutive frames: Δr x (t)=|r x (t)-r x (t-1)|,Δr y (t)=|r y (t)-r y (t-1)|.

[0060] When the center coordinates of the hand bounding box are located in the upper region of the human body bounding box, and the lateral or vertical displacement of the hand in consecutive frames exceeds a preset displacement threshold, the predicted state is a waving state, a raised hand state, or an active interaction state. Specifically, when the center coordinates of the hand bounding box are located in the upper region of the human body bounding box, and the cumulative lateral displacement of the hand within a preset time window is greater than or equal to a preset lateral displacement threshold, the predicted target state is a waving state; when the normalized vertical position r of the hand bounding box center relative to the human body bounding box center is... y When the vertical displacement decreases in consecutive frames and is greater than or equal to a preset vertical displacement threshold, and eventually enters the upper region of the human body bounding box, the predicted target state is the raised hand state; based on the waving hand state, the raised hand state and their corresponding state confidence, the active interaction state can be further predicted.

[0061] If the offset of the human body center relative to the center of the screen decreases and the area of ​​the human body frame increases in consecutive frames, the predicted target state is either close to the robot or in an interactive state.

[0062] When the offset of the human body center relative to the center of the screen increases and the area of ​​the human body frame decreases in consecutive frames, the predicted target state is either a state of moving away from the robot or a state of exiting interaction.

[0063] S3: Perform face detection on the video frame to obtain face bounding boxes, calculate facial modality data, and extract facial expression features based on the face regions in the face bounding boxes.

[0064] The facial expression feature extraction module performs face detection and face region alignment on the same video frame to obtain face bounding boxes, face confidence, face key point confidence, facial sharpness, illumination score, occlusion score, and head pose angle.

[0065] The facial landmarks and their confidence scores are obtained through a facial landmark detection algorithm. Specifically, after obtaining the face bounding box, facial landmark detection is performed. Facial landmarks may include the left and right corners of the eyes, the tip of the nose, the corners of the mouth, the eyebrows, and the chin contour. The landmark detection model outputs a set of facial landmark coordinates P={(x1,y1), (x2,y2),……,(x n ,y n )}.

[0066] Facial sharpness can be obtained through the gradient of the face region, Laplacian variance, or image sharpness evaluation algorithms. Specifically, the system converts the cropped face region into a grayscale image G, and then uses the Laplacian operator to obtain the edge response map L. 2 G, and calculate its variance using the following formula:

[0067] The larger the Laplacian variance, the more obvious the edges and textures in the face region, and the clearer the image; the smaller the Laplacian variance, the more gradual the grayscale changes in the face region, which may result in defocusing, motion blur, or low-resolution issues.

[0068] Facial clarity B f The calculation formula is:

[0069] Among them, T high T represents the upper threshold of the Laplacian variance when a face region is classified as high-resolution. low T represents the lower threshold of the Laplacian variance when a face region is judged to be in a low-resolution or blurry state. high >T low When V lap ≤T low At that time, facial clarity B f Take 0; when V lap ≥Thigh At that time, facial clarity B f Take 1.

[0070] Light rating L f The brightness is obtained through the mean brightness, variance brightness, and overexposed / underexposed pixel ratio of the face region. Specifically, the system can convert the face region to grayscale, HSV, or YUV space, extract the brightness channel Y, and calculate the mean brightness μ. Y and brightness variance σ Y The calculation formula is:

[0071]

[0072] The system calculates the mean luminance suitability Sμ and the variance luminance suitability Sσ based on the preset ideal luminance center μ0, the permissible luminance deviation Δμ, the preset ideal luminance standard deviation σ0, and the permissible standard deviation Δσ. Sμ=1-min[1,|μ Y -μ0| / Δμ] Sσ=1-min[1,|σ Y -σ0| / Δσ] Overexposed pixel ratio R over =N over / N, underexposed pixel ratio R under =N under / N, where N over N represents the number of overexposed pixels (i.e., the number of pixels whose brightness value is greater than the preset overexposure threshold). under This indicates the number of underexposed pixels (i.e., the number of pixels whose brightness value is less than the preset underexposed threshold), and N represents the total number of pixels.

[0073] Calculate the illumination score L based on the above parameters. f : L f =γ1Sμ+γ2Sσ+γ3(1-R over )+γ4(1-R under ), Among them, γ1+γ2+γ3+γ4=1, γ1, γ2, γ3, γ4 and ≥ 0, therefore, L f The value of L ranges from 0 to 1. f A larger value indicates that the brightness of the face area is closer to the preset appropriate range, and the ratio of overexposed and underexposed pixels is lower.

[0074] Occlusion scores can be obtained through the proportion of missing keypoints, the proportion of visible facial areas, or occlusion detection models. Specifically, the system counts the number N keypoints among facial keypoints whose confidence level is lower than a preset threshold (e.g., 0.5). missCalculate the proportion of missing facial landmarks R. miss =N miss / N k , where N k This indicates the number of facial landmarks.

[0075] R of visible face area visible =A visible / A face , where A visible A represents the area of ​​the visible face. face This indicates the area of ​​the face region.

[0076] Occlusion score O f =1-R miss , or O f =R visible .

[0077] The head pose angle is estimated based on the geometric proportions of the two-dimensional facial key points. In this embodiment, the left and right head tilt angle roll = arctanyR - yL / xR - xL, where (xL, yL) represents the coordinates of the left eye center and (xR, yR) represents the coordinates of the right eye center.

[0078] After obtaining the head pose angles, the head pose quality score P is further calculated. f In embodiments where yaw, pitch, and roll angles can be obtained, P f It can be represented as: P f =1-clip[κ y |yaw| / θ y +κ p |pitch| / θ p +κ r |roll| / θ r [,0,1] Among them, κ y +κ p +κ r =1, and κ y κ p κ r ≥0; θ y θ p and θ r These represent the preset allowable thresholds for yaw angle, pitch angle, and roll angle, respectively.

[0079] When only the left and right tilt angles of the head can be obtained, P f It can be represented as: P f =1-clip(|roll| / θ r ,0,1).

[0080] After completing face detection, alignment, and quality evaluation, the facial expression feature extraction module further extracts facial expression features based on the aligned face regions.

[0081] Facial expression features may include facial expression category or candidate state probability, eye opening degree, eyebrow state, mouth corner curvature, eye gaze direction, and fields required for facial quality scoring. In this embodiment, the MediaPipe FaceLandmarker / Face Mesh + Face Blendshapes lightweight facial expression feature extraction algorithm is used to obtain facial expression feature fields.

[0082] S4: Bind the human body bounding box and the face bounding box that belong to the same candidate interaction target.

[0083] When multiple human bounding boxes and multiple face bounding boxes exist in the same frame, alignment by timestamp alone is insufficient to guarantee that the target and facial expression belong to the same person. In this embodiment, the system binds the face bounding box to the corresponding human bounding box based on rules such as whether the face bounding box is located in the upper region of the human bounding box, the distance between the center of the face bounding box and the center of the head and shoulder region of the human bounding box, the inclusion relationship or intersection-union ratio of the human bounding box and the face bounding box, the continuity of the target across frames, and the consistency of historical target trajectories.

[0084] Specifically, when binding the human body bounding box to the face bounding box, the following preferred judgment conditions can be followed: The upper region condition means that the center point of the face frame should be located in the upper region of the human body frame, preferably within the range of 0%-45% of the height of the human body frame. The center distance condition is that the normalized distance between the center of the face frame and the center of the head and shoulder area of ​​the human body frame is less than a preset distance threshold, preferably less than 0.35 times the width of the human body frame. The inclusion relationship condition refers to the inclusion or overlap between the face frame and the body frame. In this embodiment, the inclusion relationship condition can measure the proportion of the entire face frame falling into the body frame or the degree of overlap between frames, making it a different judgment dimension from the upper region condition. The size ratio condition, that is, the ratio of the face frame area to the body frame area is within a reasonable range, preferably 0.02 to 0.25; The cross-frame consistency condition means that in 3-5 consecutive frames, the center displacement direction and velocity change of the same human body bounding box trajectory and the same face bounding box trajectory are consistent.

[0085] The system can calculate a comprehensive binding score based on the above conditions. A binding score greater than 0.6 is considered a reliable binding; a binding score less than 0.4 is considered an unreliable binding; and a score between the two is considered a binding score further confirmed by combining the previous round of interaction objects or task context.

[0086] If reliable binding is not possible, the facial modality weight is reduced, or only a low-intensity user state result is generated based on the prediction of the person's state from the target semantics. A low-intensity user state result refers to a user state result with low confidence and weak expression intensity that is not output when facial modality is unreliable, the human body and face cannot be reliably bound, or multimodal information is insufficient. Specifically, it can be defined as follows: the user state result includes state category, state confidence, expression intensity, and conflict marker. When the system can only infer the user state based on the target semantics or low-reliability information, the expression intensity is limited to a low range.

[0087] S5: Determine the current interaction target from among the candidate interaction targets.

[0088] After obtaining the target semantic prediction person's state, facial features, and face / body binding relationship, the system combines information such as gaze direction, waving or raising hand actions, task context, previous interaction object, and target distance to confirm or correct the coarsely selected target in step S1.

[0089] The system can calculate an interaction score for each candidate human target and select the target with the highest score that exceeds a threshold as the current interaction target. Scoring factors include: human body bounding box area, image center offset, target distance (i.e., the spatial distance between the candidate interaction target and the robot or the robot's camera), trajectory stability, whether it was the object of the previous interaction, whether it is looking in the direction of the robot, whether it is waving or raising its hand, whether it is reliably bound to the face bounding box, and whether the task context points to the target. For example, the formula for calculating the interaction score can be: S i =a1C i +a2A i +a3D i +a4G i +a5H i +a6T i +a7B i +a8P i Among them, S i For the interactive score of the i-th candidate target, C i The score indicates the proximity to the center of the image, A. i D represents the score for the area of ​​the human body frame. i G represents the proximity score between the candidate interactive target and the robot. i H represents the gaze direction score of the robot. i The score is given for waving or raising a hand. (T) i B represents the trajectory continuity score. i The score indicates the reliable binding score between the human body bounding box and the face bounding box, P. iThis represents the score of the previous interaction object or task context. a1, a2, a3, a4, a5, a6, a7, and a8 are weight coefficients, all of which are greater than or equal to 0. a1+a2+a3+a4+a5+a6+a7+a8=1.

[0090] In some embodiments, when the device cannot obtain the independent target distance, A can be used. i and D i The scores are combined into a target proximity score, and one weight is reduced accordingly.

[0091] When the difference between the highest-scoring target and the second-highest-scoring target is greater than the preset score difference threshold (e.g., 0.15), and the highest score is greater than the preset score threshold (e.g., 0.6), it is determined that there is a unique current interaction target; otherwise, the target is considered uncertain, and a neutral safe gaze is output or the system reverts to the preset neutral eye template.

[0092] If a unique current interaction target can be identified, only the location information, state assessment, and facial expression features corresponding to that target will be input into the subsequent fusion process; if a unique target cannot be identified, a neutral safe gaze will be output or the process will revert to a preset neutral eye template.

[0093] S6: Calculate the modal quality score of the current interaction target based on the multimodal data of the current interaction target, and calculate the fusion weight based on the modal quality score.

[0094] The multimodal data includes human modal data and facial modal data; the modal quality score includes the target semantic modal quality score and the facial modal quality score; and the fusion weight includes the target semantic modal weight and the facial modal weight.

[0095] Assume the continuous frame window used for quality evaluation contains N frames (N=5 in this embodiment), where the set of valid frames that successfully detect and track the current interactive target human body bounding box is V, and the number of valid frames is N. v The confidence score for the human bounding box detection in frame t is c. t The human body frame is B. t The center of the human body frame is p t .

[0096] Human body frame confidence average C b for:

[0097] Effective frame ratio F b For: F b =N v / N; Human frame stability S b It can be calculated based on the average intersection-union ratio of human body bounding boxes in adjacent valid frames: S b=[1 / (N v -1)]·ΣIoU(B t B t-1 ) Cross-frame trajectory stability T b It can be calculated based on the velocity change at the center of the human body frame. Let the normalized center velocity be v. t =p t p t-1 The preset allowable average velocity change threshold is τ. a ,but: T b =1-clip{[1 / (N v -2)]·Σ||v t -v t-1 || / τ a ,0,1} Human modal quality score (i.e., target semantic modal quality score) Q b The confidence level C of the human body bounding boxes in consecutive frames can be used as the basis for this calculation. b Effective frame ratio F b Human frame stability S b and cross-frame trajectory stability T b The calculation formula is obtained from factors such as: Q b =α1C b +α2F b +α3S b +α4T b α1+α2+α3+α4=1 Where α1, α2, α3, and α4 are weighting coefficients, all of which are greater than or equal to 0. C b F b S b and T b All values ​​are preferably normalized to 0 to 1, with larger values ​​indicating higher detection or tracking quality.

[0098] The facial modality quality score is obtained from factors such as face confidence, facial landmark confidence, sharpness, illumination, head pose, and occlusion obtained in step S3. The calculation formula is as follows: Q f =β1C f +β2K f +β3B f +β4L f +β5P f +β6O f β1+β2+β3+β4+β5+β6=1 Among them, Q f C represents the facial modality quality score. f K represents the confidence level of a face.f B represents the confidence level of facial landmarks. f Indicates facial sharpness, L f P represents the illumination score. f Indicates head posture quality score, O f The occlusion score (face visibility quality score) is represented by β1, β2, β3, β4, β5, and β6, which are weighting coefficients and are all greater than or equal to 0.

[0099] The fusion weights can be obtained by normalizing the quality scores, and the calculation formula is as follows: ,

[0100] Among them, W b W represents the target semantic modality weight. f ε represents the facial modal weights, and ε is a minimal constant to prevent the denominator from being zero.

[0101] Weight adjustments can be performed automatically based on the quality score. When the face is heavily occluded, has a large profile angle, poor lighting, or low face confidence, the facial modality quality score Q... f Reduce, normalized target semantic modality weight W b Automatically improve; when human or hand detection is missing, human bounding box is unstable, or the effective frame ratio is low, the human modal quality score Q is improved. b Reduce facial modal weights W f Automatically increase. When the occlusion ratio is greater than 0.6 or the facial quality score is lower than 0.4, adjust the target semantic modality weight Q. b Increase to no less than 0.65, preferably 0.70 to 0.80; when human body and hand position information is missing or the human body mass score is less than 0.4, increase the facial modal weight W. f Increase the value to no less than 0.65, preferably 0.70 to 0.80. If both modal mass scores are high, normalize according to the relative mass score; if both modal mass scores are low, output low-intensity or neutral user state results.

[0102] S7: Generate user state results and perform conflict detection based on the detection box position information of the current interactive target, target state, facial expression features, modal quality score and fusion weight.

[0103] The system generates user status results based on location information, predicted hand status, facial expression features, modality quality score, and fusion weights.

[0104] In this embodiment, a rule mapping table or a lightweight classifier is used to map the predicted location, hand position, and facial expression semantics to a unified user state. In an optional embodiment, structured multimodal input can be fed into a visual language model or a large multimodal model to assist in generating action semantics, expression semantics, and confidence levels. When object semantics, state inference, and facial expression semantics clearly conflict, the system sets a conflict flag and reduces the intensity of eye expressions, or outputs a caring / neutral state.

[0105] The system can determine whether there is a significant conflict by using a state compatibility table or semantic polarity. Specifically, the target semantic prediction result and facial expression semantics are mapped to state category, semantic polarity, and confidence level, respectively. If the corresponding state categories are incompatible in the preset compatibility table, and both confidence levels are greater than a preset confidence threshold (i.e., Conflict=1), then if M(s) b ,s f )=0 and c b >θ b and c f >θ f If , then it is determined to be a clear conflict. Among them, s b Represents the target semantic state, s f Indicates facial expression state; M(s) b ,s f ) represents the state compatibility matrix, taking the value 1 when compatible and 0 when incompatible; c b and c f These are the target semantic confidence and facial expression confidence, respectively; θ b and θ f The preset confidence threshold is preferably 0.6.

[0106] For example, if the target semantics are "approaching, waving, and active actions," corresponding to an active interaction state, the confidence level is greater than 0.6; while the facial expression semantics are "dejected, tired, and negative expressions," the confidence level is also greater than 0.6, then it can be determined as a modal conflict.

[0107] S8: Generate text prompts based on the user's status results, i.e., structured eye prompts.

[0108] The text prompts are textual conditions that transform the user's state into a usable raw image model, including at least the target eye state, expression intensity, editable area, and negative constraints. Negative constraints are used to restrict non-eye areas, screen border areas, fixed areas of the robot character, or fixed background areas from being editable areas, and to prevent the generation of text, watermarks, extra eyes, extra faces, facial feature misalignment, or style drift.

[0109] For example, the text prompt may include: "Robot has two round crystalline eyes, a cheerful greeting look, medium to high intensity, symmetrical left and right eyes, retains a black screen background, only allows modification of the left and right eye areas, and prohibits changes to real faces, noses, mouths, text watermarks, additional eyes, and non-eye areas."

[0110] In some embodiments, the text prompt may also include style trigger words, iris texture, highlight position, and display type.

[0111] The following table shows an example of the mapping from user state to eye response:

[0112] S9: Select the eye style category, reference image template, and eye style constraint parameters.

[0113] The eye style library and template selection module selects an eye style category and a reference image template based on the user's status. The eye style category can include multiple preset categories. The reference image template is used to constrain the position, shape, and display area of ​​the eyes, and includes at least the left and right eye layout, target display ratio, and reference eye shape. It may also include fields such as template number, template style, and initial image for partial redrawing.

[0114] Eye style constraint parameters are used to maintain the consistency of the robot's free eye style. These include color / texture constraints, template constraints, or style adaptation parameters, and may also include style embedding, graph-to-graph reference constraints, LoRA / Adapter, and other style adaptation weights.

[0115] S10: Generate layout mask and structure control diagram.

[0116] The layout mask includes at least editable areas for both eyes; in a preferred embodiment, it may also include an iris area, a pupil area, a highlight area, and a non-eye protection area. The structural control diagram may include one or more of an eye edge diagram, a line drawing, or an eye structure diagram, and may also include a reference contour diagram.

[0117] The layout mask is used to limit the modification of the generated model to the eye region, while the structure control map is used to maintain the eye structure and left-right stacking. By defining the editable region and eye structure through masks and control maps, the probability of the generated model changing non-eye regions or disrupting the left-right eye layout is reduced.

[0118] In this embodiment, the layout mask is generated based on the selected reference image template and the robot's display coordinate system. The system pre-marks the left eye region, right eye region, iris region, pupil region, highlight region, non-eye protection region, screen border region, and robot fixed appearance region in the template. During online generation, the system maps the coordinates of these regions to the current generation canvas according to the template number and target display size, forming a binary or multi-value mask. The editable regions for the left and right eyes are marked as redrawable regions in the mask, while the non-eye regions, screen border, and robot fixed appearance region are marked as protected regions.

[0119] The structural control map can be generated from a template image or a target eye structure map. For example, the eye contour edge map can be extracted by an edge detection algorithm and a line drawing can be generated by line drawing processing, or the eye structural control map can be generated based on preset eye key points.

[0120] The aforementioned layout mask and control graph, along with the text prompts, image templates, and style constraint parameters, are input into the image generation model to restrict the model to generate images that conform to the binocular structure only within permitted areas.

[0121] S11: Generate an eye image based on the eye cue content, eye style category, reference image template, layout mask, and control map.

[0122] The image generation module generates robot eye expression images or eye animation keyframes based on text prompts, eye style categories, reference image templates, layout masks, structural control graphs, and eye style constraint parameters. In a preferred embodiment, the image generation model can be a diffusion model, a graph-generated model, a conditional generation model, or other models capable of generating images based on text prompts, reference image templates, masks, and structural control graphs.

[0123] This embodiment does not rely on manual input of complete prompt words. Instead, the system generates structured eye prompt content based on the user's state and combines it with image templates, layout masks, structure control charts, and eye style constraint parameters to generate images.

[0124] When the output is a static eye expression image, the system generates a single-frame image based on the current user state. When the output is an eye animation, the system can generate a starting keyframe, a target keyframe, and several intermediate control parameters based on the user state, or select a preset animation clip and adjust the pupil offset, blink frequency, flicker frequency, and brightness changes.

[0125] Furthermore, to avoid continuous output jumps, the system can smooth the user state results within a continuous time window, and only switch the animation template or generate new keyframes when the main state or intensity continuously meets the update conditions.

[0126] S12: Perform a consistency check on the eye image and determine whether the consistency check passes. If yes, proceed to step S13. If no, regenerate the eye image, i.e., proceed to step S11, or revert to the preset eye expression template.

[0127] The consistency detection module checks whether the generated result meets the robot's eye display constraints. Detection items may include the number of left and right eyes, eye region boundary violations, pupil position, iris color deviation, highlight position deviation, template similarity, whether non-eye areas have been altered, and the presence of text watermarks or extra eyes. When the detection result meets preset conditions, display adaptation begins; when the detection result does not meet preset conditions, the image is regenerated or reverts to a preset eye expression template. If the number of consecutive regenerations reaches a preset limit, or the generation and detection time exceeds a preset time threshold, the raw image model is no longer used; instead, the closest eye template is selected from the preset template library for output.

[0128] Consistency checks are used to determine whether the generated results are suitable for direct output to the robot's display. In this embodiment, consistency checks can include two types: hard checks and scoring checks.

[0129] The strict testing conditions include: the number of left and right eyes must be 2; the overlap between the left and right eye areas and the template eye area must be no less than 0.85; the generated eye area must not exceed the editable area defined by the layout mask; the change ratio of the non-eye protection area must not exceed 5%; the screen border area and the robot's fixed appearance area must not be changed; there must be no abnormal content such as text, watermarks, extra eyes, extra faces, noses, or mouths; and the resolution and clarity of the generated image should meet the minimum requirements of the display end.

[0130] The screen border and the robot's fixed appearance area are non-editable protected areas. The system determines whether these areas have been altered by comparing pixel differences, structural similarity, or template similarity before and after generation. If the change exceeds 5%, the generation is considered a failure (i.e., the consistency check fails), and a regeneration or rollback is triggered.

[0131] For scoring tests, the consistency test score can be obtained by weighting the roundness score, left-right eye symmetry score, template similarity score, color deviation score, highlight position deviation score, and sharpness score. The calculation formula is as follows: Score=λ1R+λ2Sym+λ3Sim+λ4Color+λ5H+λ6Clarity Wherein, Score represents the consistency detection score; R represents the roundness score, used to evaluate whether the left and right eyes maintain a round or preset eye shape, which can be calculated based on the ratio of the eye contour area to the area of ​​the smallest circumscribed circle; Sym represents the left and right eye symmetry score, which can be calculated based on the center position, area, aspect ratio, and contour similarity of the left and right eyes; Sim represents the template similarity score, which can be calculated using structural similarity (SSIM) or feature similarity; Color represents the color deviation score, which can be calculated based on the color difference between the main color of the iris region and the main color of the template; H represents the highlight position score, which can be calculated based on the offset ratio of the highlight center relative to the iris center; Clarity represents the sharpness score, which can be calculated based on the Laplacian variance or edge strength; λ1, λ2, λ3, λ4, λ5, and λ6 are weighting coefficients, and λ1+λ2+λ3+λ4+λ5+λ6=1.

[0132] In a preferred embodiment, a pass is determined when the consistency test score is not lower than 0.75; a fail is determined when the score is lower than 0.75 or when any hard failure condition is triggered.

[0133] Hard failure conditions include: the number of eyes is not two; the eye area exceeds the boundary; the pupil is significantly off-center from the iris; the iris or highlights are missing; non-eye areas, screen border areas, or areas where the robot's fixed appearance is altered; text watermarks, extra faces, or extra eyes appear; the image resolution or sharpness is lower than the minimum requirements of the display. When any of the above hard failure conditions occur, the system can directly determine that the generation has failed and trigger regeneration or rollback.

[0134] "Out of bounds" is defined as the generated effective eye area exceeding the editable area for the left and right eyes defined by the layout mask, or encroaching on non-eye protection areas, screen border areas, or robot fixed appearance areas. Specifically, the overlap between the generated eye area and the template eye area is calculated. If the overlap is less than 0.85, or the percentage of pixels exceeding the editable area is greater than 5%, it is determined that the eye area has exceeded the boundaries.

[0135] "Significant deviation" is defined as the offset of the pupil center or highlight center relative to the iris center exceeding a preset proportion. In this embodiment, a significant pupil deviation is determined when the distance between the pupil center and the iris center exceeds 20% of the iris radius; a significant highlight deviation is determined when the offset of the highlight center relative to the template highlight center exceeds 15% of the eye radius. If the deviation causes asymmetry between the left and right eyes or destroys the preset eye style, the generation can also be directly determined as a failure.

[0136] S13: Adapt the eye image for display and output it to the robot display terminal.

[0137] For images that pass the consistency test, perform display adaptation, including size cropping, display ratio adaptation, brightness / color adaptation, etc.

[0138] In some embodiments, the system may also perform image enhancement, such as super-resolution, noise reduction, edge enhancement, or frame rate interpolation, and after enhancement, re-detect whether the eye region exceeds the boundary, whether the left and right eye layout is maintained, whether the main color deviation exceeds the threshold, and whether the highlight position is abnormal. If the enhanced image is unsatisfactory, the system reverts to the image before enhancement or a preset template.

[0139] This embodiment reduces the probability of incorrectly fusing the actions and expressions of different users in multi-user scenarios by coarsely selecting the current interaction target, confirming it after feature extraction, and binding human / face bounding boxes. By using target semantic quality score, facial quality score, and fusion weights, modal contributions are dynamically adjusted when there is face occlusion, side profile, backlighting, or missing human detection boxes, improving the stability of user state result generation. Through text prompts, image templates, layout masks, structural control charts, and eye style constraint parameters, user state results are transformed into executable eye image generation conditions, reducing left-right eye misalignment, changes in non-eye regions, and style drift caused by general image generation models. Consistency checks, hard failure conditions, regeneration count / timeout limits, and a fallback template library ensure that the robot will not retry indefinitely during real-time interaction, nor will it output obviously unqualified eye images.

[0140] Example 2 This embodiment is a further extension of the above embodiments. The similarities will not be repeated. The difference is that the robot expression generation system also includes a LoRA style adaptation training module, which is used to train low-rank adaptation weights for various eye styles based on a self-built eye style dataset and the SDXL base.

[0141] In step S11, the robot's eye image is generated using a combination of "text prompt content + reference image template + eye style constraint parameters". The basic image generation model uses the SDXL base; the eye style constraint parameters are provided by LoRA adaptation weights trained using a self-built robot eye style dataset. This LoRA training is used to enable the raw image model to stably generate various eye styles for the robot, such as round crystalline eyes, starry eyes, natural eyes, elf eyes, anime eyes, and cute pet eyes. Details are as follows.

[0142] (a) Constructing a robot eye style dataset In this embodiment, the robot eye style dataset can consist of five eye styles: starry sky, nature, elf, anime, and cute pet. Each style can contain 2000 target eye images, of which 1800 are used as training samples and 200 are used as validation samples, for a total of 10000 target images across the five styles. Sample sources can include manually designed eye templates, paired images of the left and right eyes, keyframes of eye animation, historically generated qualified images, and robot eye materials that have been manually verified.

[0143] All samples were uniformly processed to a 1024×1024 resolution and subjected to left and right eye alignment, uniform black or preset screen background, removal of overly dark / overexposed samples, removal of watermarked text, removal of extra faces or extra eyes, and removal of samples with abnormal left and right eye counts. The amplified samples did not exceed 40% of the total samples for a single class, and the same template variant was not allowed to be included in both the training and validation sets simultaneously to avoid inflated validation results.

[0144] (ii) Text labels and training prompts Each training sample should be labeled with at least the following: eye style category, style trigger word, eye state label, structure label, template label, iris texture, pupil shape, highlight position, color style, display type, and qualification mark.

[0145] Among them, eye style categories are used to select training subsets and load corresponding weights online, such as starry sky, nature, elf, anime, and cute pet. Style trigger words are used to activate target eye styles during training and inference, such as lingdong starry eyestyle and lingdong anime eye style. Eye state labels correspond to the mapping from user state to eye response, such as cheerful greeting eyes, soft caring eyes, and curious eyes. Structure labels are used to describe at least one of the following: number of left and right eyes, eye shape, left and right eye symmetry, left and right eye layout, pupil shape, and highlight position, such as dualround crystal eyes and symmetric left and right eyes. Template labels are used to describe at least one of the following: image template number, template layout, display type, and non-editable protected area, such as left-right layouttemplate and black screen background. Negative cues are used to suppress real facial features, extra eyes, text watermarks, and non-eye anomalies, such as human face, nose, mouth, extra eyes, text, and watermark.

[0146] The system can generate negative prompts based on structural tags, template tags, qualification marks, and preset abnormal content rules. For example, positive training prompts may include the robot's dual circular crystalline eyes, specified style trigger words, target eye state, left and right eye symmetry, and a black screen background; negative prompts may include a real face, nose, mouth, extra eyes, text, watermarks, and changes in non-eye areas.

[0147] (iii) Whether the reference image template is involved in training and inference In the preferred embodiment of the current SDXL base model + LoRA, the reference image template participates in training sample construction, text label generation, mask generation, and validation, but is not used as a new model input for ordinary LoRA training. The LoRA training phase mainly uses the "target eye image + text label / training cue content" learning style; the reference image template is used to determine the left and right eye layout, editable regions, and template number, and is written into the training cue content or validation rules.

[0148] During the online inference phase, the reference image template serves as the initial template for the graph-generated image or a local redraw template, and together with the text prompts, local redraw masks, negative prompts, and LoRA weights, it constrains the generated result. If an SDXLinpainting pedestal or additional control network is subsequently used, the reference image template can also participate in training as a mask image or structural control condition.

[0149] (iv) The modules, hierarchy and connection relationship of the SDXL base In this embodiment, an existing basic image generation network model is used, namely SDXL base1.0 or a similar SDXL base. This SDXL base includes a dual text conditional encoding part, an image latent space encoding / decoding part, a latent space denoising network, and an optional Refiner refinement model. Training prompts are input into the dual text conditional encoding part to obtain text conditional features; the target eye image is input into the image latent space encoding part to obtain the target latent space representation; the system adds Gaussian noise to the target latent space representation at random training time steps to obtain a noisy latent space representation. The noisy latent space representation, training time steps, and text conditional features are then jointly input into the latent space denoising network.

[0150] The latent space denoising network comprises a downsampling module, an intermediate module, and an upsampling module, each containing a self-attention module and a cross-attention module. Text conditional feature injection into the cross-attention module constrains the denoising network to generate eye image styles that match the training cues.

[0151]

[0152] (v) Freeze module, training module and LoRA insertion level The following modules are frozen during training: dual-text conditional coding, image latent space coding, image decoding, SDXL original denoising network weights, convolutional layers, residual blocks, normalization layers, time-step embedding layers, and optional Refiner refinement model.

[0153] In this embodiment, only the LoRA low-rank adaptation branch connected to the attention projection layer of the SDXL denoising network is trained. The insertion layer preferably covers the self-attention module and cross-attention module in the downsampling module, intermediate module, and upsampling module; each connected attention module includes at least query projection, key projection, value projection, and output projection.

[0154] For any attention projection layer that is accessed, the original projection weights remain frozen. The input features are processed by the original projection weights to obtain the original projection output, and by the LoRA dimensionality reduction matrix and the LoRA dimensionality increase matrix to obtain the low-rank incremental output. The low-rank incremental output is added to the original projection output according to a preset scaling factor and online loading intensity, which is used as the final output of the projection layer and continues to enter the subsequent attention calculation. During backpropagation, only the LoRA dimensionality reduction matrix and the LoRA dimensionality increase matrix are updated.

[0155] The target layer in the engineering implementation can include the query, key, value, and output projection layers of the self-attention and cross-attention modules in the denoising network's downsampling module, intermediate module, and upsampling module.

[0156] The Refiner refinement model is an optional inference stage module and is not involved in the LoRA training in this embodiment. Its function is to perform secondary refinement on the image details after the basic generative model outputs the initial robot eye expression image, in order to improve iris texture, edge sharpness, highlight details, and overall visual quality.

[0157] The specific data processing procedure is as follows: The basic SDXL image generation or partial redrawing process first generates a preliminary eye expression image based on the text prompt content, reference image template, layout mask, structure control chart, and target style LoRA weights; if Refiner is enabled, the preliminary generated image or its latent space representation, the corresponding text prompt content, and the necessary style constraints are input into the Refiner refinement model; Refiner further denoises and enhances the texture of the image details, and outputs the refined robot eye expression image; the refinement result still needs to enter the consistency detection module. If the detection passes, the adapted output is displayed; if the detection fails, it reverts to the image before refinement and regenerates or uses a preset template.

[0158] Its effect is to improve the clarity, iris texture quality and highlight performance of the generated image, but does not change the main generation logic of "text prompt content + reference image template + layout mask + structure control chart + style constraint parameters" in this embodiment.

[0159] (vi) Training steps, loss and output 1. Establish training subsets according to eye styles such as starry sky, nature, elf, anime, and cute pet. Each training subset includes the target eye image, corresponding style trigger words, eye status labels, and template labels.

[0160] 2. Crop or scale the target eye image to 1024×1024 and remove watermarks, extra faces, incorrect left and right eye layouts, overexposure, underexposure, or duplicate samples.

[0161] 3. Generate training prompts based on the tags. The prompts include positive descriptions such as "robot has two circular crystalline eyes", specified style trigger words, target eye state, left and right eye symmetry, and clean black screen background, and also configure negative prompts.

[0162] 4. Load SDXL base 1.0 or a similar SDXL base, freeze the original weights of the dual text encoder, image latent space encoder, image decoder and denoising network, and do not train the Refiner.

[0163] 5. The attention projection layer of the downsampling module, intermediate module and upsampling module of the denoising network is bypassed to the low-rank branch of LoRA.

[0164] 6. During training, the target eye image is encoded in the latent space and denoised to obtain a noisy latent space, and the training prompts are encoded in the text to obtain text conditional features; the denoising network outputs predicted noise based on the noisy latent space, time step and text conditional features.

[0165] 7. Using the diffusion training loss between predicted noise and real noise as the optimization objective, backpropagation only updates the LoRA branch parameters.

[0166] 8. Save an independent LoRA weight file for each style, and write the style trigger words, recommended loading intensity, verification pass rate, and applicable eye conditions into the multi-style LoRA weight library.

[0167] The training output is not a fixed eye image, but multiple style LoRA weight files and their mapping relationships with style trigger words, applicable states, recommended loading intensity and validation results.

[0168] (vii) Specific training parameter configuration In one embodiment, the training method and parameter range are shown in the table below.

[0169]

[0170] (viii) Training methods for multi-style LoRA models The system trains multiple LoRA weights separately for each eye style category, instead of mixing all style samples into a single indistinguishable weight file. Specifically, starry sky style samples train starry sky style LoRA weights, natural style samples train natural style LoRA weights, elf style samples train elf style LoRA weights, anime style samples train anime style LoRA weights, and cute pet style samples train cute pet style LoRA weights. Each LoRA weight corresponds to a unique or relatively unique style trigger word, which is selected and loaded based on the user's state and the target eye style category during online inference.

[0171] The effect of style-specific training is to reduce style confusion, improve style trigger stability, and reduce the probability of iris texture, highlight position, and left-right eye layout drift during generation.

[0172] (ix) Online loading of LoRA weights and generation of text-image templates The model parameters are not updated during the online generation phase. The system generates eye text prompts based on the user's status, selects an eye image template and a local redrawing region, and chooses the corresponding LoRA weights according to the target eye style. The image generation model takes the text prompts, image template, local redrawing region, negative prompts, and target style LoRA weights as input, executes the SDXL image generation process or local redrawing process, and outputs robot eye expression images or eye animation keyframes.

[0173] In one specific embodiment, the LoRA loading strength is set to 0.75 by default; it can be increased to 0.85 for strong stylization; it can be reduced to 0.65 when template preservation is prioritized; the image-to-image denoising strength can be set to 0.42; the prompt word guidance coefficient can be set to 7.0; and the number of sampling steps can be set to 30.

[0174] (x) Image generation, display, and detection The generated image can be a 1024×1024 eye canvas, which is then cropped or scaled according to the robot's display type. For dual-eye screen scenarios, the left and right eyes are mapped to their respective display areas; for full-face screen scenarios, only the eye area is editable, while non-eye areas retain the preset screen background or the robot's fixed appearance.

[0175] The generated results undergo consistency testing: the number of left and right eyes should be 2; the overlap between the left and right eye regions and the template region should be no less than 0.85; the change ratio of the non-edited region should not exceed 5%; the template structure similarity should not be less than 0.70; and the highlight center offset should not exceed 15% of the eye radius. If two consecutive failures occur or the total time exceeds 3 seconds, the system will revert to the preset eye template. The above thresholds can be adjusted according to the display end and engineering acceptance standards.

[0176] In this embodiment, the image generation model is based on the SDXL base and uses a self-built robot eye style dataset to train various style LoRA low-rank adaptation weights to ensure that the generated results can stably maintain the robot's "agile eyes" such as round crystalline eyes, starry sky eyes, natural eyes, elf eyes, anime eyes, and cute pet eyes.

[0177] This embodiment uses SDXL-LoRA multi-style adaptation training to adapt to the robot's own eye style without fully fine-tuning the basic image generation model, thereby improving the consistency of iris texture, pupil ratio, highlight position, and left and right eye layout.

[0178] Example 3 This embodiment is a specific application scenario of the above embodiments.

[0179] 1. The user is wearing a mask but waves at the robot. The robot captures user video frames in an entrance greeting scenario. The user is wearing a mask, resulting in significant facial obscuration and low facial reliability. Simultaneously, the target semantic detection shows increased hand height, higher waving motion scores, and higher motion energy. Based on the quality evaluation, the system increases the target semantic modality weight, generates a "greeting / positive" user status result, and then generates a friendly eye-greeting text prompt. A bright and friendly eye style template and editable masks for the left and right eyes are selected. After generation and passing detection, the output is sent to the robot's display.

[0180] 2. The user is looking down and has low energy. In family-oriented scenarios, users often exhibit low energy levels, with their heads down and facial expressions revealing fatigue or dejection. The system generates a "dejected / fatigued" user state result, selects a gentle and caring eye response, reduces eye brightness and flicker intensity, and outputs a gentle and caring gaze.

[0181] 3. Multiple people appearing in the frame at the same time In a guided tour scenario, multiple users appear on screen simultaneously. The system first performs a rough selection of candidate targets based on the human body bounding box area, screen center offset, and trajectory stability. Then, it combines facial features with the prediction results obtained from the target semantics, along with gaze direction, waving gestures, and task context, to confirm the current interactive target. If a target cannot be uniquely identified, a neutral, safe gaze is output or the system reverts to a neutral template to avoid strong responses to non-interactive users.

[0182] Example 4 This embodiment is a computer-readable storage medium corresponding to the above embodiments, on which a computer program is stored. When the program is executed by a processor, it implements the various steps of the robot expression generation method in the above embodiments and can achieve the same technical effect, which will not be repeated here.

[0183] In summary, the robot facial expression generation method and computer-readable storage medium provided by this invention reduce the probability of incorrectly fusing the actions and expressions of different users in multi-user scenarios by coarsely selecting the current interaction target, confirming it after feature extraction, and binding the human body / face bounding box. By using target semantic quality score, facial quality score, and fusion weights, modal contributions are dynamically adjusted when there is face occlusion, side profile, backlighting, or missing human body detection boxes, improving the stability of user state result generation. By using text prompts, image templates, layout masks, structural control charts, and eye style constraint parameters, user state results are transformed into executable eye image generation conditions, reducing left-right eye misalignment, changes in non-eye regions, and style drift caused by general image generation models. Through consistency checks, hard failure conditions, regeneration count / timeout limits, and a fallback template library, it is ensured that the robot will not retry indefinitely during real-time interaction, nor will it output obviously unqualified eye images. Through SDXL-LoRA multi-style adaptation training, the robot's own eye style can be adapted without fully fine-tuning the base image generation model, improving the consistency of iris texture, pupil ratio, highlight position, and left and right eye layout.

[0184] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating robot facial expressions, characterized in that, include: Collect video frames of the robot's interaction area and perform human detection in the video frames to determine candidate interaction targets; Human and hand positions are detected for candidate interactive targets to obtain detection boxes and their position information. The detection boxes include human body boxes and hand boxes. The target state is predicted based on the detection boxes. Face detection is performed on the video frames to obtain face bounding boxes, and facial expression features are extracted based on the face regions within the face bounding boxes. Bind human body bounding boxes and face bounding boxes that belong to the same candidate interaction target, and determine the current interaction target from each candidate interaction target; Based on the location information of the detection box of the current interactive target, the target state, and facial expression features, generate the user state result; Based on the user status result, generate text prompt content and select eye style category, reference image template and eye style constraint parameters; Based on the text prompt content, eye style category, reference image template, and eye style constraint parameters, an eye image is generated, which is either an eye expression image or an eye animation keyframe.

2. The robot facial expression generation method according to claim 1, characterized in that, The step of performing face detection on the video frame to obtain face bounding boxes, and extracting facial expression features based on the face regions within the face bounding boxes, includes: Face detection is performed on the video frames to obtain face bounding boxes, and facial modality data is calculated; The face region within the face frame is aligned, and facial expression features are extracted based on the aligned face region.

3. The robot facial expression generation method according to claim 2, characterized in that, The facial modal data includes face confidence, face landmark confidence, facial sharpness, illumination score, occlusion score, and head pose quality score. The step of performing face detection on the video frame to obtain face bounding boxes and calculating facial modality data includes: Face detection is performed on the video frames to obtain face bounding boxes and face confidence scores, and face landmark detection is performed on the face bounding boxes to obtain face landmarks and face landmark confidence scores. The face region within the face frame is converted to a grayscale image. Based on the grayscale image and the Laplacian operator, an edge response map is calculated, and the Laplacian variance is calculated based on the edge response map. Calculate facial sharpness based on the Laplace variance; The illumination score is calculated based on the average brightness, brightness variance, and the proportion of overexposed and underexposed pixels in the face region. The occlusion score is calculated based on the proportion of missing facial landmarks or the proportion of the visible facial area. Based on the geometric proportions of facial key points, the head pose angle is calculated, and based on the head pose angle, the head pose quality score is calculated.

4. The robot facial expression generation method according to claim 2, characterized in that, Before generating the user state result based on the detection box position information of the current interaction target, the target state, and facial expression features, the process further includes: Based on the multimodal data of the current interactive target, the modal quality score of the current interactive target is calculated, and the fusion weight is calculated based on the modal quality score. The multimodal data includes human modal data and facial modal data. The human modal data is determined based on the human bounding box of the current interactive target in consecutive frames. The modal quality score includes the target semantic modal quality score and the facial modal quality score. The fusion weight includes the target semantic modal weight and the facial modal weight. The step of generating a user state result based on the detection box position information of the current interactive target, the target state, and facial expression features includes: The user status result is generated based on the location information of the detection box of the current interactive target, the target state, facial expression features, modal quality score and fusion weight.

5. The robot facial expression generation method according to claim 4, characterized in that, The step of calculating the modal quality score of the current interaction target based on its multimodal data, and calculating the fusion weight based on the modal quality score, includes: Based on the human modality data of the current interaction target, the target semantic modality quality score is calculated. The human modality data includes the average confidence score of the human bounding box in consecutive frames, the proportion of effective frames, the stability of the human bounding box, and the stability of the trajectory across frames. Based on the facial modal data of the current interactive target, calculate the facial modal quality score. The facial modal data includes face detection confidence, facial landmark confidence, facial sharpness, illumination score, occlusion score, and head pose quality score. Calculate the target semantic modality weight and the facial modality weight based on the target semantic modality quality score and the facial modality quality score.

6. The robot facial expression generation method according to claim 4, characterized in that, The binding of human body bounding boxes and face bounding boxes belonging to the same candidate interaction target includes: Based on the preset binding conditions, calculate the binding score between the human bounding box and the face bounding box of each candidate interactive target; If the binding score between a human body frame and a face frame is greater than a preset first score threshold, then the human body frame and the face frame are bound together. The binding conditions include upper region conditions, center distance conditions, containment relationship conditions, size ratio conditions, and cross-frame consistency conditions. The condition for the upper region is that the center point of the face frame is located in the upper region of the human body frame. The center distance condition is that the normalized distance between the center point of the face frame and the center point of the head and shoulder region of the human body frame is less than a preset distance threshold. The inclusion relationship condition is that the face bounding box and the body bounding box have an inclusion relationship or an overlapping relationship; The size ratio condition is that the ratio of the face frame area to the body frame area is within a preset ratio range. The cross-frame consistency condition is that, in a series of video frames with a preset number of frames, the center displacement direction and velocity change of the same human body bounding box trajectory and the same face bounding box trajectory remain consistent.

7. The robot facial expression generation method according to claim 6, characterized in that, Before generating the user state result based on the detection box position information of the current interaction target, the target state, facial expression features, modality quality score, and fusion weight, the process further includes: If the binding scores between the human body bounding box and each face bounding box of the current interactive target are all less than the preset second score threshold, then the facial modality weight is reduced.

8. The robot facial expression generation method according to claim 1, characterized in that, After generating the eye image based on the text prompt content, eye style category, reference image template, and eye style constraint parameters, the process further includes: The eye image is subjected to consistency detection according to preset consistency detection conditions; If the consistency check fails, the eye image is regenerated; If the consistency test passes, the eye image is adapted for the robot display and then output to the robot display. The consistency detection conditions include hard detection conditions and scoring detection conditions; The hard detection conditions include: the number of eyes is 2; the overlap between the left and right eye regions and the eye regions in the reference image template is greater than or equal to a preset overlap threshold; the generated eye region does not exceed the editable region defined by the layout mask; the change ratio of the non-eye protection region is less than or equal to a preset change ratio threshold; the screen border region and the robot's fixed appearance region are not changed; no abnormal content appears; and the resolution and clarity of the eye image meet the minimum requirements of the robot's display end. The abnormal content includes text, watermarks, extra eyes, extra faces, noses, and mouths. The consistency detection score for the scoring detection conditions is greater than or equal to a preset third score threshold. The consistency detection score is calculated based on roundness score, left-right eye symmetry score, template similarity score, color deviation score, highlight position deviation score, and sharpness score. The roundness score is calculated based on the ratio of the eye contour area to the area of ​​the smallest circumscribed circle. The left-right eye symmetry score is calculated based on the center position, area, aspect ratio, and contour similarity of the left and right eyes. The template similarity score is calculated based on the structural similarity or feature similarity between the eye image and the reference image template. The color deviation score is calculated based on the color difference between the main color of the iris region in the eye image and the main color of the iris region in the reference image template. The highlight position deviation score is calculated based on the offset ratio of the highlight center relative to the iris center. The sharpness score is calculated based on the Laplacian variance or edge intensity.

9. The robot facial expression generation method according to claim 8, characterized in that, Also includes: If the number of consecutive regenerations reaches the preset upper limit threshold, or the time taken for consistency detection and image generation exceeds the preset time threshold, then the closest eye template image is selected from the preset template library.

10. The robot facial expression generation method according to claim 1, characterized in that, The step of predicting the target state based on the detection box includes: Based on the human body frame and hand frame, structured spatial features are calculated. The structured spatial features include the center coordinates of the human body frame, the area of ​​the human body frame, the offset of the human body center relative to the center of the image, the center coordinates of the hand frame, the position of the hand frame relative to the human body frame, and the displacement velocity of the hand frame in consecutive frames. If the center coordinates of the hand frame are located in the upper region of the human body frame, and the lateral or vertical displacement of the hand in consecutive frames exceeds a preset displacement threshold, then the predicted target state is a waving state, a raised hand state, or an active interaction state; wherein, the upper region of the human body frame is the range of the human body frame from 0% to a preset proportion in the height direction from top to bottom. If the offset of the human body center relative to the center of the screen decreases and the area of ​​the human body frame increases in consecutive frames, the predicted target state is either close to the robot or in an interactive state. If the offset of the human body center relative to the center of the screen increases and the area of ​​the human body frame decreases in consecutive frames, the predicted target state is either a state of moving away from the robot or a state of exiting interaction.

11. The robot facial expression generation method according to claim 1, characterized in that, Determining the current interaction target from among the candidate interaction targets includes: Based on the interaction information of each candidate interaction target, the interaction score of each candidate interaction target is calculated. The interaction information includes the center offset of the screen, the area of ​​the human body frame, the target distance, the trajectory stability, whether it is the interaction object of the previous round, whether there is a gaze at the robot direction, whether there is a waving or raising action, whether it is reliably bound to the face frame, and whether the task context points to the candidate interaction target. If the highest interaction score is greater than the preset score threshold, and the difference between the highest interaction score and the second highest interaction score is greater than the preset score difference threshold, then the candidate interaction target corresponding to the highest interaction score is taken as the current interaction target; otherwise, the preset neutral eye template is output.

12. The robot facial expression generation method according to claim 1, characterized in that, The process involves generating text prompts based on the user's state, selecting an eye style category, a reference image template, and eye style constraint parameters; and generating an eye image based on the text prompts, eye style category, reference image template, and eye style constraint parameters. The eye image is either an eye expression image or an eye animation keyframe, including: Based on the user status result, a text prompt is generated, which includes the target eye status, expression intensity, editable area, and negative constraints. Based on the user status result, select an eye style category from the preset style categories, select a reference image template from the preset image templates, and select eye style constraint parameters; A layout mask is generated based on the reference image template and the robot display end coordinate system, and a structure control map is generated based on the reference image template or the eye structure map of the current interactive target. An eye image is generated based on the text prompt, eye style category, reference image template, eye style constraint parameters, layout mask, and structure control diagram.

13. The robot facial expression generation method according to claim 1, characterized in that, An eye image is generated using an image generation model, wherein the eye style constraint parameters are determined based on LoRA weights, which are obtained by training the image generation model.

14. The robot facial expression generation method according to claim 13, characterized in that, Before generating the eye image using the image generation model, the process further includes: A robot eye style dataset is constructed, which includes a training subset corresponding to each eye style category. Each sample image in the eye style dataset is labeled, and the labeling information includes eye style category, style trigger word, eye state label and template label. The sample image is cropped or scaled according to a preset size, and the sample image is then cleaned. Based on the annotation information of the sample images, training prompts are generated; The image generation model is trained according to each training subset to obtain the LoRA weights corresponding to each eye style category; The image generation model includes a dual-text conditional encoder, an image latent space encoder, a noise-adding network, a denoising network, and a LoRA branch. The denoising network includes a downsampling module, an intermediate module, and an upsampling module, each of which includes an attention projection layer. The LoRA branch includes a LoRA dimensionality reduction matrix and a LoRA dimensionality increase matrix. The LoRA branch is connected to the attention projection layers in the downsampling, intermediate, and upsampling modules of the denoising network, respectively. The dual text conditional encoder is used to encode the training prompt content to obtain text conditional features; The latent space encoder is used to encode the sample image to obtain a target latent space representation; The noisy network is used to add random noise to the target latent space representation according to a preset training time step to obtain a noisy latent space representation; The denoising network is used to output predicted noise based on the training time step, the noisy latent space representation, and the text conditional features; Specifically, when training the image generation model, backpropagation is performed using the diffusion training loss between the predicted noise and the real noise as the optimization objective to update the weight parameters of the LoRA branch.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-14.

Images