Gesture matching method, multi-modal feature matching method, medium and program product

Abstract

The embodiment of the application provides a gesture matching method, a multi-modal feature matching method, a medium and a program product. The method comprises the following steps: performing feature extraction based on a target person image and a target commodity image to obtain person feature information and commodity feature information; and matching a display posture of a target commodity and a hand posture of a target person based on the person feature information and the commodity feature information to obtain hand feature information matched with a holding type of the target commodity. The embodiment of the application can adaptively generate hand feature information matched with different holding types, and improve the naturalness and rationality of the commodity holding posture.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to gesture matching methods, multimodal feature matching methods, media and program products. Background Technology

[0002] With the continuous development of computer vision, image processing, and artificial intelligence technologies, image generation, editing, and display based on people and objects have been widely applied in e-commerce, digital content creation, and other fields. In these applications, it is often necessary to present the interaction between people and products in images or videos, such as a person holding and displaying a product, to enhance the realism and expressiveness of the content. However, in the process of displaying people and products together, problems such as unnatural visuals and illogical interaction logic can easily occur, thus affecting the display effect and user experience.

[0003] Based on this, embodiments of this application provide a gesture matching method, a multimodal feature matching method, a medium, and a program product to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a gesture matching method, a multimodal feature matching method, a medium and a program product, which adaptively generate hand feature information that matches different grip types, thereby improving the naturalness and rationality of product grip posture.

[0005] In a first aspect, embodiments of this application provide a gesture matching method, the method comprising: extracting features based on a target person image and a target product image to obtain person feature information and product feature information; and matching the display posture of the target product with the hand posture of the target person based on the person feature information and the product feature information to obtain hand feature information that is compatible with the grip type of the target product.

[0006] In some embodiments, the grip type includes at least one of fingertip pinching, palm-covering grip, thumb gripping, lifting support, hanging grip, two-hand holding, two-hand lifting, and directional display.

[0007] In some embodiments, matching the display posture of the target product with the hand posture of the target person based on the person's characteristic information and the product characteristic information to obtain hand characteristic information adapted to the grip type of the target product includes: generating product gripping parameters based on the product characteristic information; wherein the product gripping parameters include at least one of the gripping position, gripping direction, and gripping range of the target product; and matching the display posture with the hand posture based on the product gripping parameters and the person's characteristic information to generate the hand characteristic information.

[0008] In some embodiments, generating product gripping parameters based on the product feature information includes: performing structural analysis and region division on the target product based on the product feature information to determine at least one candidate gripping region of the target product; wherein the candidate gripping region is determined based on the main body region of the target product, the main body region including at least one of the upper, middle, lower and other structural functional regions of the target product; and generating the product gripping parameters based on the at least one candidate gripping region.

[0009] In some embodiments, generating the product gripping parameters based on the at least one candidate gripping region includes: performing a gripping adaptability assessment on the at least one candidate gripping region based on at least one of area information, position information, and thickness information of the candidate gripping region to obtain a gripping adaptability assessment result; determining one of the candidate gripping regions as a target gripping region based on the gripping adaptability assessment result; and determining the product gripping parameters based on the target gripping region.

[0010] In some embodiments, the product gripping parameters may further include at least one of the size information of the target gripping area and mask information.

[0011] In some embodiments, the step of evaluating the gripping adaptability of the candidate gripping region based on at least one of the area information, position information, and thickness information of the candidate gripping region to obtain a gripping adaptability evaluation result includes: mapping the area information, position information, and thickness information of the candidate gripping region to normalized feature values, and generating a gripping priority score based on the normalized feature values; the gripping priority score serves as the gripping adaptability evaluation result.

[0012] In some embodiments, generating a gripping priority score based on the normalized feature values ​​may include: generating a corresponding gripping priority score based on the normalized feature values ​​through a multilayer perceptron network or a weighted computation model.

[0013] In some embodiments, matching the display posture with the hand pose based on the product gripping parameters and the person's feature information to generate the hand feature information includes: calculating key hand points based on the product gripping parameters to obtain key hand point information; and adjusting the key hand point information based on the person's feature information to determine the hand feature information.

[0014] In some embodiments, adjusting the hand key point information based on the person's feature information to determine the hand feature information includes: performing pose estimation on the person's feature information to obtain body joint key point information; constraining the hand key point information based on the body joint key point information to obtain target hand key point information that satisfies the constraints; and determining the hand feature information based on the target hand key point information.

[0015] In some embodiments, constraining the hand keypoint information based on the body joint keypoint information to obtain target hand keypoint information that satisfies the constraints includes: constructing at least one body structure constraint condition based on the body joint keypoint information; detecting whether the hand keypoint information satisfies each body structure constraint condition based on the body joint keypoint information and the hand keypoint information; and correcting the hand keypoint information if it does not satisfy one or more body structure constraint conditions, until the hand keypoint information satisfies all body structure constraint conditions.

[0016] In some embodiments, the body structure constraints are constructed based on the body structure feature information of the target person, which includes at least one of the following: shoulder position, elbow position, hand position, and body central axis information.

[0017] In some embodiments, the body structure constraints include part bending constraints and / or body distance constraints.

[0018] In some embodiments, the part bending constraint condition may include an elbow bending constraint condition; wherein the elbow bending constraint condition is used to limit the elbow bending angle of the target person to be within a preset angle range.

[0019] In some embodiments, the elbow flexion angle is the angle between a first vector and a second vector, wherein the first vector is constructed based on the shoulder and elbow positions of the target person, and the second vector is constructed based on the elbow and hand positions of the target person.

[0020] In some embodiments, the body distance constraint can be used to limit the spatial distance of the target person's hand position relative to the body's central axis to a preset distance range.

[0021] In some embodiments, determining the hand feature information based on the target hand key point information includes: mapping the target hand key point information to two-dimensional hand position information; generating a hand feature image based on the two-dimensional hand position information; and using the hand feature image as the hand feature information.

[0022] In some embodiments, generating a hand feature image based on the two-dimensional hand position information includes: performing interpolation processing based on the two-dimensional hand position information to generate the hand feature image.

[0023] In some embodiments, the interpolation process includes bilinear interpolation.

[0024] In some embodiments, the hand feature information includes at least one of a hand feature image, a set of hand key points, and an implicit feature representation.

[0025] Secondly, embodiments of this application provide a multimodal feature matching method, the method comprising: extracting features based on a target person image, a target product image, and semantic information to obtain person feature information, product feature information, and semantic feature information; matching the display posture of the target product with the hand posture of the target person based on the person feature information and the product feature information to obtain hand feature information adapted to the holding type of the target product; and adjusting the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information and the hand feature information to generate feature matching information for characterizing the target holding state.

[0026] In some embodiments, adjusting the spatial relationships of the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information and the hand feature information to generate feature matching information characterizing the target's holding state includes: predicting and calculating the spatial offset between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information according to the semantic feature information, to obtain offset parameters and sampling weight parameters for splicing the person feature information and the product feature information, as well as corresponding splicing feature information; The offset parameters include translation parameters, rotation parameters, and scaling parameters. Based on the offset parameters and the sampling weight parameters, the spatial position of the stitched feature information is adjusted to obtain spatially adjusted stitched feature information. Based on the hand feature information, the posture of the spatially adjusted stitched feature information is adjusted to obtain posture-adjusted stitched feature information. The posture-adjusted stitched feature information corresponds to the target holding state. Based on the posture-adjusted stitched feature information, the channel dimension of the semantic feature information is mapped to match the channel dimension of the posture-adjusted stitched feature information to obtain the feature matching information.

[0027] In some embodiments, adjusting the posture of the spatially adjusted splicing feature information based on the hand feature information to obtain posture-adjusted splicing feature information includes: adjusting the posture of the hand region in the spatially adjusted splicing feature information according to the hand feature information to obtain posture-adjusted splicing feature information; performing cross-attention processing on the hand feature information and the product grasping feature information corresponding to the target grasping area in the posture-adjusted splicing feature information to obtain splicing feature information in which the target person's hand and the target product form a target grasping state, and using it as the optimized posture-adjusted splicing feature information.

[0028] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0029] Fourthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0030] This application provides a gesture matching method, a multimodal feature matching method, a medium, and a program product. By extracting feature information of the target person and product based on images of the target person and product respectively, and then matching and modeling the display posture of the target product with the hand posture of the target person, it can adaptively generate hand feature information that matches different product shapes and grip types without relying on manual annotation or preset fixed posture templates, thereby improving the naturalness and rationality of the product's grip posture. By introducing hand feature information adapted to the target product's grip type as a constraint, it can effectively reduce misalignment, overlap, or ergonomic non-compliance between the product and the hand, improving the consistency and stability of the final product display effect. Simultaneously, the automated modeling of the gesture matching process achieved through the above method helps improve the generation efficiency of product display content, meeting the comprehensive requirements of image quality and generation efficiency in product display scenarios. Attached Figure Description

[0031] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0032] Figure 1 This is a flowchart illustrating a digital human video generation method provided in an embodiment of this application.

[0033] Figure 2 This is a structural block diagram of a digital human video generation system provided in an embodiment of this application.

[0034] Figure 3 This is a flowchart illustrating a gesture matching method provided in an embodiment of this application.

[0035] Figure 4 This is a flowchart illustrating a multimodal feature matching method provided in an embodiment of this application.

[0036] Figure 5 This is a schematic flowchart of an image generation method provided in an embodiment of this application.

[0037] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0039] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0040] With the development of internet platforms and e-commerce channels, showcasing and promoting products to users through pre-recorded or live streams has become a crucial marketing method in the sales industry. Product sales videos can present the appearance, structure, and usage of products to users in a direct and vivid way, playing a significant role in increasing users' willingness to purchase. Therefore, how to efficiently and cost-effectively produce product demonstration videos has become one of the core issues that product sellers urgently need to address in their daily operations.

[0041] In recent years, digital human technology has matured, making it possible to automatically generate videos of people based on limited input data. Digital human technology generally refers to training or reasoning on a model based on user-provided image data, audio data, or other control information to generate virtual human videos that match the image of a target person. These virtual human videos can perform lip movements, head movements, and body movements under audio or command-driven prompts, thereby simulating the expression and display behavior of a real person.

[0042] In product sales applications, sellers often want to leverage digital human technology to quickly generate video content featuring a target person showcasing a product. For example, a seller can pre-acquire an image of a person holding a product; this person could be the seller themselves, a third-party model, or a virtual character used for product demonstration. This image is then input into a pre-trained digital human model, combined with corresponding audio information, to drive the target person in the image to lip-sync according to the audio content, thus generating a video of the target person explaining and demonstrating the product. Using this method, sellers can generate product sales videos and distribute them to different platforms in a short time.

[0043] However, while the relevant technical solutions and models can generate images or videos of a person holding a product, they still have significant shortcomings in terms of product display accuracy. For example, during the generation process, problems such as inaccurate spatial positioning between the person's hand and the product, and drift in product or person feature information can easily occur, resulting in the product and hand not fitting naturally in the generated result, or even abnormal product position and incorrect relationship between the person and the product. Since the image quality in the product display stage directly affects the user's intuitive perception of the product, these problems make it difficult to meet the requirements of realism and stability for actual product display.

[0044] Furthermore, in practical applications, sellers often need to generate videos for a large number of different products in a short period of time. However, the image or video generation process in related technologies is often time-consuming. For example, a single generation process may take twenty seconds or even longer, which makes it difficult to meet the needs of product sales scenarios for generation efficiency and real-time performance.

[0045] Therefore, the existing technologies are still unable to achieve efficient and rapid digital human product display video generation while ensuring a stable display relationship between the target person and the product and reliable image quality. They also struggle to simultaneously meet the dual requirements of product display accuracy and real-time generation, necessitating a new technical solution.

[0046] To address the aforementioned technical problems, this application proposes a digital human video generation method and system, and further expands the interaction methods of digital humans in different product display application scenarios. Through the technical solution provided by this application, video content of a target person showcasing a target product can be automatically generated based on limited input data without the need for manual filming of a complete product display video, thereby improving the generation efficiency and stability of digital human videos for product displays.

[0047] The digital human video generation method and system provided in this application can model the display relationship between the target person and the target product based on the acquired target person image and target product image, and generate a target display image to represent the target person holding or displaying the target product under the condition of display posture constraints. Then, the target person is driven to perform corresponding display actions on the product according to the prompt information input by the user, so as to generate a digital human product display video with continuous action effect.

[0048] In some application scenarios, users can instruct the digital human on how products are displayed by inputting text and / or audio prompts. This includes specifying the content, pace, or key points of the presentation, allowing the generated video content to adapt to different product types and marketing needs. In this way, the digital human can not only perform static product displays but also combine semantic information to achieve more expressive product explanations and demonstrations.

[0049] The application environment of the digital human video generation method and system can include a user terminal and a cloud platform. The user terminal is used to enable user interaction and display image and video results, while the deployment, training, and inference processes of the relevant models can be executed in the cloud. By placing the computationally intensive model training and inference processes in the cloud, the performance requirements of user terminal devices can be reduced, while improving the overall scalability and processing efficiency of the system.

[0050] The embodiments of this application are described in detail below.

[0051] See Figure 1 , Figure 1 This is a flowchart illustrating a digital human video generation method provided in an embodiment of this application.

[0052] This application provides a method for generating digital human videos, which includes steps S101 to S103.

[0053] Step S101: Obtain the target person image, the target product image, and prompts for indicating the product display method, wherein the prompts include text prompts and / or audio prompts.

[0054] Step S102: Based on the target person image, the target product image, and the prompt information, generate a target display image to represent the target person displaying the target product.

[0055] Step S103: Input the target display image into the digital human driving model to drive the target person to perform actions related to product display, so as to generate a target digital human video.

[0056] The prompt information is used to determine the display posture of the target product relative to the hand of the target person, and the display posture is used to constrain the generation process of the target display image.

[0057] The target person image can refer to image data of a preset person image used to generate digital human videos. The preset person can be the seller of the product, a third-party model used for product display, or a pre-defined virtual person image. In implementation, the target person image can serve as input for extracting person feature information, used to subsequently determine the person's body proportions, hand positions, and the spatial range of executable movements, thereby providing structural constraints on the person side for modeling the product display posture.

[0058] It should be noted that the target person images, digital human figures, or visual data related to person appearance involved in the embodiments of this application are all generated based on legally authorized person images or legally licensed virtual person images, and do not involve the unauthorized illegal collection, copying, or use of natural persons' portraits, identity characteristics, or protected images. The technical solutions described in the embodiments of this application focus on the technical process and implementation mechanism of digital human video generation. Its implementation is not based on infringing on the portrait rights, privacy rights, or other legitimate rights and interests of others, and in practical applications, it can be flexibly configured according to the specific authorization scope of different authorized entities to meet relevant laws, regulations, and compliance requirements.

[0059] The target product image refers to the image data corresponding to the target product to be displayed and sold in the generated video. Users can take photos of the target product and upload the image against a preset background. This embodiment does not limit the preset background; for example, it can be a green screen background or a natural background. The target product image can be obtained by photographing a real product or generated through product modeling or rendering. In practical applications, the background of the target product image can be removed, retaining only the main body area of ​​the product. In this embodiment, the target product image can be used to extract product feature information and, combined with the area of ​​a person's hand, determine the reasonable position, orientation, and display method of the product during the display process, such as the product facing the audience, being displayed at an angle, or being held by a hand.

[0060] The prompting information is used to instruct the display method of the target product in the generated image or video. In some embodiments, the prompting information can describe the positional relationship, display direction, or overall shape of the target product relative to the target person's hand, such as instructing the target product to tilt at a specific angle or specifying the direction the target product faces the viewer. Furthermore, the prompting information can also be used to describe visual attributes or display effects not fully covered in the target product image, such as providing supplementary descriptions of product transparency, color style, or visual texture, thereby providing high-level semantic constraints for the subsequent image generation process. The prompting information can include text prompts and / or audio prompts, where text prompts include phrases such as "hold the product in your right hand and display it facing the viewer" or "display the product slightly tilted," and audio prompts can be parsed to obtain corresponding semantic instructions. In implementation, the prompting information can be parsed into semantic features to participate in the determination of subsequent display posture and constraints of the image generation process, its role being to provide a controllable human-computer interaction entry point for the generation process.

[0061] The target display image refers to the intermediate image result generated in the digital human video generation process. It represents a static image of a target person displaying a target product in a predetermined pose. The target display image not only contains the visual content of the target person and the target product, but also implicitly contains the spatial and poseal relationships between them, serving as the basic input for subsequent actions executed by the digital human-driven model. By generating the target display image first, complex product display relationships can be stably modeled at a static level, thereby reducing the risk of product position drift or display distortion during subsequent video generation.

[0062] The digital human-driven model refers to a model used to drive a target person to perform actions and generate dynamic video. It combines static target display images with dynamic control information to output a continuous sequence of video frames. The digital human-driven model can generate lip movements, head movements, and / or body movements of the target person based on audio-driven, text-driven, or a combination of both. For example, it can generate lip movements based on audio-driven methods and drive hand or upper body movements based on semantic commands. Through the digital human-driven model, a digital human can complete natural and coherent product display actions while maintaining a stable product display posture.

[0063] The display posture refers to the spatial display state of the target product relative to the target person's hand, representing the product's positional relationship, orientation, and grip type during the display process. The display posture can include the product's grasping position relative to the hand, grasping direction, tilt angle, and whether it is in a hand-held, supported, or suspended display state, or one or more of these. Determining the display posture comprehensively considers the characteristics of the target person's hand area, the structural features of the target product, and the display semantics indicated in the prompts; it is a key intermediate representation connecting the information about the person and the product.

[0064] The aforementioned display posture constraint refers to a control mechanism that applies a determined display posture as a generation condition or constraint to the image generation process. By constraining the display posture, the generation process can continuously adhere to the reasonable spatial relationship between the target product and the target person's hands across multiple generation stages, improving issues such as product position misalignment, hand overlap, or unnatural posture in the generated results. In practical implementation, display posture constraints can be achieved through conditional feature injection, attention modulation, spatial constraints, or generation parameter control, thereby enhancing the stability and realism of the target display image.

[0065] The target digital human video refers to a dynamic video result generated from a target display image after processing by a digital human-driven model. It is used to present the process of a target person naturally displaying a target product in a continuous time series. Target digital human videos can be applied to e-commerce live streaming, product promotion, advertising, and other scenarios. Their characteristics include stable product display posture, coherent human movements, and clear semantic display. The method in this embodiment can quickly generate high-quality digital human product display videos without the need for actual filming, thereby improving the efficiency of product content production.

[0066] In some embodiments, such as when a target person image and a target product image are input, and the prompt message "hold the bottle in front of your right chest, place the bottle vertically, and show the front of the bottle to the audience" is input, a corresponding target display image can be generated. Based on this, a video content of the target person (such as the product seller) displaying and explaining the target product (such as the bottle) can be generated through a digital human-driven model.

[0067] In some embodiments, steps S101 and S102 can be executed by an image generation system, and step S103 can be executed by a digital human driving system. The image generation system is used to generate a target display image, and the digital human driving system is used to drive the action of the target person in the target display image. Through the coordinated work of the image generation system and the digital human driving system, efficient and stable target digital human video generation can be achieved.

[0068] The digital human video generation method provided in this embodiment introduces a product display posture constraint mechanism driven by prompt information in the digital human video generation process. The spatial position, orientation, and display method of the target product relative to the target person's hand are used as key constraints in the generation process. Specifically, firstly, images of the target person, the target product, and prompt information indicating the product display method are acquired. Based on the input, the display relationship between the target person and the target product is modeled to determine the display posture of the target product relative to the target person's hand. Furthermore, during the generation of the target display image, the display posture is continuously applied as a constraint to the image generation process, ensuring that the generated result conforms to the product display requirements in terms of spatial structure and posture relationship. Finally, the generated target display image is input into the digital human driving model, which drives the target person to perform actions corresponding to the product display, thereby generating a natural, coherent target digital human video that conforms to the semantics of product display.

[0069] This embodiment introduces a product display posture determined based on prompts during the digital human video generation process, and uses this posture to constrain the generation process of the target display image. This makes the spatial relationship between the target product and the target person's hands in the generated target display image more accurate and stable, effectively improving problems such as misalignment of product and hand positions, unnatural postures, or display distortion. By inputting the generated target display image into the digital human driving model and driving the target person to perform actions related to product display, the automatic generation from static display images to dynamic digital human videos is achieved, significantly reducing manual shooting and post-production costs and improving the efficiency of product display video generation. By unifying text prompts and / or audio prompts as the control basis for product display posture, the digital human video generation process possesses good interactivity and flexibility, thereby improving the controllability and adaptability of product display methods and meeting the application needs of different products and display scenarios.

[0070] In some embodiments, generating a target display image representing the target person displaying the target product based on the target person image, the target product image, and the prompt information may include: performing feature extraction based on the target person image, the target product image, and the prompt information to obtain person feature information, product feature information, and semantic feature information; generating display posture constraint information to constrain the display posture based on the person feature information, the product feature information, and the semantic feature information; and generating an image under the constraint of the display posture constraint information to obtain the target display image.

[0071] Feature extraction refers to the process of parsing and representing target person images, target product images, and prompt information. Its purpose is to convert the raw input data into feature representations that can be used for subsequent modeling and generation. In some embodiments, person feature information may include at least one of person appearance features, key body region features, and hand region features; product feature information may include at least one of product contour features, structural features, and spatial scale features; semantic feature information may include semantic vectors. Through feature extraction, a unified representation of multi-source heterogeneous information can be achieved for subsequent data processing.

[0072] The semantic feature information can refer to high-level semantic information (e.g., semantic vector) used to characterize the way a product is displayed. It can be generated from text prompts in the prompt information, or from audio prompts after audio recognition and semantic parsing, or directly mapped from audio prompts. It is used to indicate the display method, display posture, or display semantics of the target product.

[0073] The displayed posture constraint information can refer to conditional feature information used to characterize and constrain the displayed posture, which is jointly generated from person feature information, product feature information, and semantic feature information. In specific implementations, the displayed posture constraint information can be manifested as a constraint representation on the position, orientation, or posture parameters of the product, or it can be manifested as a conditional feature applied to the image generation process. By using displayed posture constraint information, the spatial relationship between the target product and the target person's hand can be continuously constrained during the image generation process, thereby improving the stability and consistency of the generated results.

[0074] The image generation process can refer to the process of generating a target display image under the constraints of display posture constraint information, which is used to transform feature information from multiple modalities into a visualized image result. The image generation process can adopt an iterative generation method based on a generative model, such as diffusion generation based on latent code representation or other conditional generation methods. During the generation process, the display posture constraint information acts as a conditional input or control signal, ensuring that the generated target display image conforms to the expected product display posture at multiple generation stages.

[0075] In this embodiment, the generation process of the target display image is based on a joint modeling and constraint generation mechanism using multi-source information. Specifically, firstly, feature extraction processing is performed on the target person image, the target product image, and the prompt information to obtain person feature information that characterizes the person's appearance and hand structure, product feature information that characterizes the product's shape and structural attributes, and semantic feature information that characterizes the semantics of the product display. Subsequently, based on the correlation between the person feature information, product feature information, and semantic feature information, display posture constraint information is generated to characterize the target product's display posture relative to the target person's hand. During the image generation stage, the display posture constraint information is continuously applied as a conditional constraint to the image generation process, ensuring that the generated target display image meets the product display requirements in terms of spatial structure and posture relationship, thereby obtaining a target display image that accurately reflects the target person displaying the target product in a predetermined display posture.

[0076] This embodiment extracts features from the target person image, the target product image, and the prompt information, and generates display posture constraint information based on the person's features, product features, and semantic features. This explicitly constrains the display posture of the target product relative to the target person's hand during image generation, thereby improving the accuracy and consistency of the spatial relationship between the product and the person's hand in the target display image. By generating images under the constraints of the display posture constraint information, the generation process can continuously follow the expected product display semantics, effectively improving problems such as product position drift, unnatural posture, or display distortion, and enhancing the realism and stability of the target display image. By employing a display posture constraint mechanism based on multimodal feature joint modeling, fine control over the product display method is achieved without increasing manual intervention, thereby improving the generation quality and application adaptability of digital human product display images.

[0077] In some embodiments, the display posture constraint information may be used to characterize at least one of the grasping position, grasping direction, and grip type of the target product relative to the hand of the target person.

[0078] The gripping position refers to the spatial relationship between the target product and the target person's hand, describing the specific area where the product is grasped, lifted, or supported during the display. The gripping position can include a specific contact area on the product surface, the positional relationship between the product's center and key points on the hand, or the relative spatial distribution between the product and the palm or fingers. In this embodiment, the gripping position can be used as part of the display posture constraint information to limit the placement of the product in the generated image, improving issues such as the product appearing to float, intersect, or deviate from the hand.

[0079] The gripping direction refers to the orientation of the target product relative to the target person's hand during the display process. It characterizes the directional features of the product when it is grasped or displayed. The gripping direction can include the angle between the product's main axis and the palm direction, the orientation of the product's front relative to the viewing angle, or the rotation angle of the product relative to the hand. During image generation, by constraining the gripping direction, the orientation of the product in the generated target display image can conform to the expected display semantics, such as being displayed facing the viewer, sideways, or tilted, thereby improving the product display effect.

[0080] The grip type refers to the contact method and force relationship between the target person's hand and the target product, describing whether the product is grasped, lifted, clamped, or suspended during the display process. Grip types can include various forms such as single-handed grip, two-handed lifting, and fingertip clamping, with different grip types corresponding to different hand postures and product stability methods. In this embodiment, the grip type can be used as part of the display posture constraint information to limit the contact relationship between the hand and the product when generating the target display image, thereby improving the problem of mismatch between hand posture and product structure and enhancing the naturalness and rationality of the generated result.

[0081] The display posture constraint information can be flexibly selected by combining any one or more of the grip position, grip direction, and grip type. This embodiment can configure the display posture constraint information according to actual needs in different application scenarios. For example, only the grip position can be constrained to adapt to simple display scenarios, or the grip position, grip direction, and grip type can be constrained simultaneously to adapt to high-precision product display scenarios, thereby enhancing the applicability and scalability of the technical solution.

[0082] In some embodiments, generating display posture constraint information based on the person feature information, the product feature information, and the semantic feature information to constrain the display posture may include: modulating the person feature information and the product feature information using the semantic feature information to generate the display posture constraint information. The modulation may include at least one of cross-attention processing, feature weighting, and feature mapping.

[0083] The person's characteristic information can include overall appearance features, key body area features, and hand area features. Hand area features may include, for example, the position of the palm, the distribution of key finger points, or the orientation of the hand. In this embodiment, the person's characteristic information can be used to reflect the action space and hand structure constraints that the target person can perform during product display, and is an important basis for generating a reasonable display posture.

[0084] Product feature information may include product outline features, geometric structure features, size proportion features, or features of the main body area of ​​the product, such as the long axis direction of the product, the grippable area of ​​the surface, or the position of the center of gravity. In this embodiment, product feature information can be used to describe the structural constraints of the product during the display process, and together with human feature information, participate in the modeling of the display posture to ensure the rationality and stability of the product display method.

[0085] Semantic feature information can be used to express display semantics such as "handheld display", "facing the camera", and "tilted display". In this embodiment, semantic feature information can be used as a high-level control signal to guide the modulation process of person feature information and product feature information, so that the generated display posture conforms to the expected product display semantics.

[0086] The modulation mentioned above refers to the process of guiding, adjusting, or reweighting character and product feature information using semantic feature information during the generation of display posture constraint information. Modulation can be used to ensure that character-side structural information and product-side structural information form a consistent expression under the constraints of display semantics, making the character-side and product-side information consistent with the prompt information at the semantic level. This generates display posture constraint information that meets the requirements of display semantics, improving the problem of display postures not meeting semantic requirements caused by relying solely on low-level visual features for modeling. The modulation process can be dynamic, that is, applying different influence weights to character and product feature information under different display semantic conditions.

[0087] The cross-attention processing can be a feature interaction method used to establish relationships between different modal features. It calculates the correlation between semantic feature information and character / product feature information, guiding the character and product feature information to focus on and display semantically relevant parts. In this embodiment, cross-attention processing allows character and product feature information to align under semantic guidance, thereby improving the consistency between the generated display posture and the prompt information.

[0088] Feature weighting refers to a processing method that assigns different weights to different components of person and product feature information based on semantic feature information. Through feature weighting, feature components highly relevant to the current display semantics can be enhanced, while irrelevant or weakly relevant feature components can be suppressed. For example, under the semantic condition of "displaying with one hand," the weight of person feature information related to the one-handed area can be increased, thereby generating a display posture that better meets semantic requirements.

[0089] Feature mapping refers to a processing method that transforms person feature information and product feature information into a feature space consistent with semantic feature information through a mapping function. Feature mapping can be implemented through linear mapping, nonlinear mapping, or neural network mapping, etc., to reduce the representation differences between different feature modalities, enabling multimodal features to be fused and constrained in a unified space, thereby improving the ability to express posture constraint information.

[0090] The generation of posture constraint information can be achieved by any one of cross-attention processing, feature weighting, or feature mapping, or a combination of multiple methods, which makes this embodiment more flexible and scalable under different implementation conditions and reduces excessive restrictions on specific modulation methods.

[0091] In this embodiment, the generation of display posture constraint information is achieved based on a text semantic modulation mechanism for character and product feature information. Specifically, firstly, character feature information extracted from the target character image, product feature information extracted from the target product image, and semantic feature information obtained from parsing prompt information are acquired. Subsequently, using the semantic feature information as a guiding signal, the character and product feature information are modulated to ensure semantic consistency between the character-side and product-side information. During the modulation process, cross-attention processing can be used to establish the correlation between semantic feature information and character / product feature information, or feature weighting and feature mapping can be used to dynamically adjust the character and product feature information, thereby generating display posture constraint information that simultaneously reflects the character structure, product structure, and display semantics, providing stable and controllable posture constraint conditions for subsequent image generation.

[0092] This embodiment modulates character and product feature information using semantic feature information and generates display posture constraint information based on the modulation results. This allows the determination of the display posture to simultaneously consider the semantics of product display and the structural features of the character and product, thereby improving the consistency between the display posture and the prompt information. By introducing modulation methods such as cross-attention processing, feature weighting, or feature mapping, character and product feature information are dynamically aligned under semantic guidance, thus improving the problem of inaccurate or unnatural display postures caused by relying solely on static feature modeling. By generating unified display posture constraint information and using it to constrain the image generation process, the generated results follow the expected product display semantics at multiple generation stages, thereby improving the stability, realism, and controllability of the target display image.

[0093] In some embodiments, the modulation includes cross-attention processing. Modulating the person feature information and the product feature information using the semantic feature information to generate the display posture constraint information may include: matching the display posture of the target product with the hand posture of the target person based on the person feature information and the product feature information to obtain hand feature information adapted to the grip type of the target product; adjusting the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information and the hand feature information to obtain adjusted person feature information and product feature information where the target person's hand and the target product conform to the target grip state; performing cross-attention processing on the adjusted person feature information and product feature information based on the semantic feature information to generate semantic constraint weight information for characterizing the semantic constraint strength; and using the semantic constraint weight information to perform weighted modulation on the adjusted person feature information and product feature information to obtain the display posture constraint information.

[0094] In this embodiment, the generation of the display posture constraint information can be achieved by separately modulating the person's feature information and the product's feature information. Specifically, firstly, based on the person's feature information and the product's feature information, a matching model is performed on the display posture of the target product and the hand posture of the target person to obtain hand feature information that matches the holding type of the target product. Subsequently, guided by the semantic feature information and the hand feature information, the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person's feature information and the product feature information is adjusted so that the target person's hand and the target product satisfy the target holding state in terms of spatial position, orientation, and scale. On this basis, cross-attention processing is performed on the adjusted person's feature information and product feature information respectively using semantic feature information to generate semantic constraint weight information to represent the semantic constraint strength, and the corresponding feature information is weighted and modulated using the semantic constraint weight information to obtain the display posture constraint information used to constrain the subsequent image generation process.

[0095] This embodiment performs cross-attention modulation based on person feature information and product feature information respectively, and independently weights and constrains the two types of features under the guidance of semantic feature information. This allows for precise control of the strength of semantic constraints in the person's hand area and the product area, thereby improving the accuracy and stability of the holding relationship between the target person's hand and the target product, reducing the drift of product or hand features, and helping to maintain the rationality of the person's structure and the consistency of the product's appearance under complex display semantic conditions.

[0096] In some embodiments, the modulation includes cross-attention processing. Modulating the person feature information and the product feature information using the semantic feature information to generate the display posture constraint information may include: matching the display posture of the target product with the hand posture of the target person based on the person feature information and the product feature information to obtain hand feature information adapted to the grip type of the target product; adjusting the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information and the hand feature information to generate feature matching information characterizing the target grip state; performing cross-attention processing on the feature matching information based on the semantic feature information to generate semantic constraint weight information characterizing the semantic constraint strength; and using the semantic constraint weight information to perform weighted modulation on the feature matching information to obtain the display posture constraint information.

[0097] In this embodiment, the generation of the display posture constraint information can be achieved based on joint modeling of person feature information and product feature information. Specifically, firstly, based on person feature information and product feature information, the display posture of the target product and the hand posture of the target person are matched and modeled to obtain hand feature information that matches the holding type of the target product. Subsequently, guided by semantic feature information and hand feature information, the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and product feature information is adjusted, and the adjusted features are fused to generate feature matching information that characterizes the target holding state. On this basis, cross-attention processing is performed on the feature matching information with semantic feature information to generate semantic constraint weight information that characterizes the semantic constraint strength, and the feature matching information is weighted and modulated based on the semantic constraint weight information to obtain the display posture constraint information, which is used to guide the subsequent image generation process.

[0098] This embodiment integrates the character feature information and product feature information into feature matching information after adjusting the spatial relationship, and performs cross-attention modulation based on the feature matching information. This enables the overall constraint of the holding relationship between the target character's hand and the target product within a unified feature space, thereby reducing the information inconsistency problem caused by independent modulation of multiple feature channels and improving the integrity and robustness of the display posture constraint. It is especially suitable for application scenarios with complex product structures or high semantic relevance.

[0099] In other embodiments, the generation method of the display posture constraint information is not limited to the above-mentioned implementation methods of separately modulating person feature information and product feature information, or modulating based on feature matching information. Specifically, any one or more of person feature information, product feature information, or combinations thereof can be selected for modulation processing according to actual application needs; cross-attention processing can also be combined with feature weighting, feature mapping, feature gating, or other semantic constraint mechanisms; the execution order, fusion method, and modulation level of spatial relationship adjustment can also be flexibly set according to the model structure. Any technical solution that applies constraints to the display posture between the target person's hand and the target product under the guidance of semantic feature information to generate constraint information for guiding the generation of the target display image should fall within the protection scope of this application.

[0100] In the above embodiments, the semantic feature information can be used to characterize the user's high-level semantic requirements for the product display method, and it can originate from text prompts, audio prompts, or a combination of both. In some embodiments, the semantic feature information can be generated by a text encoding model, a speech-to-text model, or a multimodal semantic encoding model, and participate in the subsequent modulation process of person feature information and product feature information in vector form, so as to introduce explicit semantic constraints in the generation process.

[0101] In some embodiments, person feature information can be extracted from a target person image, for example, represented as a person feature image or a sequence of person features. In some embodiments, product feature information can be extracted from a target product image, for example, represented as a product feature image or a sequence of product features.

[0102] The hand feature information can be used to characterize the hand posture of a target person when performing a product holding or display action. It is not limited to specific joint angles or posture parameters, but can include comprehensive features describing hand structure, orientation, opening and closing states, and contact relationship with the product. In some embodiments, the hand feature information can be obtained by matching and modeling the person's feature information with the product's feature information, reflecting the constraint relationship between the target product's holding type and the person's hand posture, thereby providing a basis for subsequent spatial relationship adjustments.

[0103] Matching the display posture of the target product with the hand gestures of the target person involves determining, based on the person's and product's characteristics, which hand gesture allows the person to naturally hold or display the product. This matching process comprehensively considers the product's size, shape, gripping area, and the structural features of the person's hand, thereby generating hand feature information that matches the target product's grip type. This matching mechanism can improve issues such as mismatched hand-product posture or unnatural hand gestures that occur during the generation process.

[0104] The grip type can be used to characterize the type of grasping or supporting method adopted by the target person when displaying the target product. It reflects the contact pattern between the hand and the product, the direction of force, and the opening and closing shape of the fingers, as well as other high-level action semantics. It can also serve as a priori constraints for hand posture matching and spatial relationship adjustment. For example, grip types can include, but are not limited to, at least one of the following: fingertip pinching (such as lipstick, card), palm-wrapping grip (such as water cup, bottle), thumb grip (such as mobile phone), lifting support (such as flat display of boxed products), hanging grip (such as bagged products with handles), holding or lifting with both hands (such as large gift boxes), and pointing / indicating display (such as emphasizing key parts of the product with a finger). In some embodiments, the grip type can be obtained from semantic descriptions in text prompts (such as "grip tightly", "support", "raise and expose the front"), mapped from a product category database, or jointly inferred from product geometric structure analysis results (such as thickness, distribution of grippable areas, whether it has a handle) and human hand structure features. In subsequent processes, these are used to select different hand key point priors, different spatial transformation ranges, or different semantic constraint weight allocation strategies, so that the generated gestures are more natural and more in line with product display requirements.

[0105] The spatial relationship adjustment can be used to adjust the spatial regions corresponding to the target person's hand and the target product in the person's feature information and product feature information, so that the two meet the target holding state in terms of spatial position, orientation, and scale. In some embodiments, the spatial relationship adjustment may include position offset, scale adjustment, rotation transformation, or local resampling, and the specific implementation is not limited. Through spatial relationship adjustment, the relative relationship between the person's hand and the product can be pre-aligned at the feature level, providing a stable foundation for subsequent semantic modulation and image generation.

[0106] The feature matching information refers to a comprehensive feature representation obtained by fusing the feature information of a person and the feature information of a product after adjusting the spatial relationship. It is used to represent the overall holding state between the target person's hand and the target product. Feature matching information can be generated through feature concatenation, feature mapping, feature overlay, or other fusion methods. It is used to unify the originally separate features of the person and the product into the same feature space, thereby facilitating the subsequent application of unified semantic constraints.

[0107] The cross-attention processing can be used to establish explicit associations between semantic feature information and person feature images or product feature images, thereby enabling semantic guidance and constraint of spatial features. In some embodiments, cross-attention processing can use semantic feature information as a query vector and person feature information, product feature information, or feature matching information as feature sequences participating in attention calculation, thereby generating weight parameters that reflect the degree of response of different spatial regions to semantic constraints. This processing method enables semantic information to have a stronger influence on key regions at the feature level.

[0108] The semantic constraint weight information can be used to characterize the constraint strength of semantic feature information on different spatial feature regions, which can be reflected in the degree to which different regions are strengthened or weakened during feature modulation. The greater the constraint strength, the more the corresponding spatial region needs to follow the display requirements indicated by the semantic feature information. By introducing semantic constraint weight information, differentiated control of semantic information over the human hand region and the product region can be achieved.

[0109] The display posture constraint information can be the result of applying semantic constraints to person feature information, product feature information, or feature matching information. It is used to constrain the display posture relationship between the target person and the target product during subsequent image generation. This display posture constraint information can guide the image generation process to gradually form a target display image that conforms to the target's holding state and semantic requirements, thereby improving the accuracy and stability of product display.

[0110] In some embodiments, before generating an image, the digital human video generation method may further include: aligning the display posture constraint information with preset time step information to constrain the image generation process of the target display image based on the aligned display posture constraint information.

[0111] The time step information refers to temporal identifiers used to characterize different generation or iteration stages of the image generation process, distinguishing different time points or generation steps. Examples of time step information include generation iteration step numbers, stage identifiers, and continuous or discrete time indices. In the case of an iterative image generation model (e.g., a diffusion generation model), time step information can indicate the noise level, feature update magnitude, or generation state corresponding to the current generation stage, thus providing a time reference for the dynamic introduction of constraints.

[0112] The alignment process refers to establishing a correspondence between display pose constraint information and time step information in the generation flow. This ensures that the display pose constraint information can participate in image generation in an adaptive manner at different time steps. The alignment process may include injecting different forms of display pose constraint information at different time steps, or adjusting the scale or weight of the display pose constraint information according to the time step. Through alignment, the effect of display pose constraint information not only in a single generation stage can be reduced, allowing it to continuously exert its constraint effect throughout the entire generation process.

[0113] Before the generation process of the target display image begins, this embodiment preprocesses or prepares the conditional information used to constrain the image generation process, such as feature alignment, conditional feature reconstruction, or generation parameter initialization. The purpose is to ensure that the display posture constraint information used to constrain the generation matches the temporal structure of the image generation process before formally entering the image generation process, thereby improving the stability and controllability of the subsequent generation process.

[0114] The image generation process can refer to the overall flow of generating a target display image under the constraints of display pose constraint information. It can be implemented using iterative generation, stepwise optimization, or conditional generation methods. The image generation process may include a diffusion generation process based on latent code representation, a layer-by-layer generation process using conditional generation networks, or other generation processes that can be used to generate the target display image. In this embodiment, the image generation process can continuously introduce display pose constraint information at multiple generation stages, ensuring that the generated result always meets the expected requirements in terms of spatial structure and display pose.

[0115] The constraint on the image generation process of the target display image based on the aligned display posture constraint information can refer to using the display posture constraint information aligned with the time step information as a conditional input or control signal to limit and guide the generation result during the image generation process. This constraint can be manifested as modulation of generation features, adjustment of generation parameters, or guidance of the generation direction. Its function is to ensure that the target display image meets the display posture requirements of the target product relative to the target person's hand at different generation stages, thereby improving the temporal consistency and overall quality of the generation result.

[0116] In this embodiment, before generating the target display image, an alignment mechanism between display posture constraint information and time step information is introduced to achieve temporal control of the image generation process. Specifically, firstly, display posture constraint information characterizing the display posture of the target product is obtained, and corresponding preset time step information during image generation is also obtained. Subsequently, the display posture constraint information and the time step information are aligned, so that the display posture constraint information participates in the image generation process in a corresponding form at different time steps. During the image generation stage, the generation process of the target display image is constrained based on the aligned display posture constraint information, ensuring that the generation process continuously follows the display posture requirements of the target product relative to the target person's hand at multiple time steps, thereby obtaining a temporally stable target display image that conforms to the display semantics.

[0117] This embodiment aligns the display pose constraint information with preset time step information before image generation, ensuring that the display pose constraint information continuously exerts its constraint effect at different time steps in the image generation process. This improves the problem of product position drift caused by display pose only being effective in local generation stages. By constraining the image generation process of the target display image based on the aligned display pose constraint information, the spatial relationship between the target product and the target person's hand remains consistent throughout the generation process, thereby improving the temporal stability and overall consistency of the target display image. By introducing a time step alignment constraint mechanism, fine control over the generation process is achieved without increasing generation complexity, thus improving the generation quality and controllability of the target display image.

[0118] In some embodiments, the step of generating an image under the constraints of the display posture constraint information to obtain the target display image may include: constraining the image generation process based on the display posture constraint information, so that the image generation process gradually forms a target display image that conforms to the display posture.

[0119] This embodiment introduces display posture constraint information, which can limit the reasonable range of product display in advance during the generation stage, rather than passively correcting it after generation.

[0120] The image generation process can refer to the process of gradually forming a target display image based on input conditions within a generative model. This process can be implemented using iterative generation, diffusion generation, conditional generation, or other generation methods based on latent code representation. In this embodiment, image generation is not a one-time output result, but rather a process of gradually refining the image content through multiple calculations, so that the structure of the person, the hand posture, and the shape of the product gradually stabilize during the generation process. This image generation process may include multiple stages, such as a coarse structure generation stage, a posture refinement stage, and a detail enhancement stage, etc., which are not limited in this embodiment.

[0121] The constraint on the image generation process based on the display pose constraint information can refer to continuously introducing the display pose constraint information as conditional information into the generation model at each stage of image generation to limit the direction of change in the generated result. This constraint method can include modulating the generated features, weighting the generated parameters, and guiding the generation path. For example, in the diffusion generation process, display pose constraint information can be injected in different iteration steps to control the direction of noise removal, ensuring that the generated result always converges around the expected display pose, thereby reducing deviations from the target display requirements.

[0122] The term "gradual formation" refers to a process where the target image is not generated directly through a single calculation, but rather gradually approaches the target state through multiple generation iterations or multi-stage generation. For example, in the gradual formation process, the early generation stage focuses on establishing the overall spatial layout of the person and the product, the mid-stage focuses on the posture coordination between the hands and the product, and the later stage focuses on detail optimization and enhanced display effects. Through this gradual formation approach, the posture constraint information continues to play a role throughout the generation process, ensuring the stability and consistency of the generated result.

[0123] The compliance with the stated display posture refers to the fact that in the final generated target display image, the display state of the target product relative to the target person's hand meets the conditions defined by the display posture constraint information. This compliance can be reflected in aspects such as reasonable relative position between the product and the hand, natural gripping method, and consistency between the display direction and the prompt information. For example, if the display posture constraint information indicates that the product should be held vertically and facing the viewer, then the spatial orientation of the product, the distribution of fingers, and the occlusion relationship in the generated result should all match this posture requirement.

[0124] In this embodiment, the image generation process is completed step-by-step under the continuous constraints of display posture constraint information. Specifically, the display posture constraint information is used to characterize the display posture requirements of the target product relative to the target person's hand, and participates in the generation calculation as conditional information during the image generation process. At each stage of image generation, the generation model guides the spatial distribution, structural relationships, and local morphology of the generated features based on the display posture constraint information, so that the generation process gradually converges towards the target state that conforms to the display posture during the iterative process, thereby obtaining a target display image of the target person displaying the target product in a reasonable and stable posture at the end of the generation.

[0125] This embodiment introduces display posture constraint information to characterize the relationship between the product and the person's hand during the image generation process, and uses this display posture constraint information to continuously constrain the image generation process. This allows the image generation process to stably form a target display image that conforms to the expected display posture through gradual iteration, thereby effectively improving the problems of posture drift, spatial misalignment, or unnatural holding between the product and the hand, improving the accuracy and consistency of product display in the generated image, and thus enhancing the visual quality of the target display image and the actual product display effect.

[0126] In some embodiments, constraining the image generation process based on the display posture constraint information, so that the image generation process gradually forms a target display image that conforms to the display posture, may include: performing attention encoding processing based on the aligned display posture constraint information to obtain global feature information; performing diffusion generation based on latent code representation based on the global feature information to obtain a target latent code; decoding the target latent code, and performing pixel-level fusion processing on the decoded first display image to obtain the target display image.

[0127] In this embodiment, the aligned display pose constraint information can refer to the conditional features obtained by matching and adapting the display pose constraint information with the time step information or generation stage information during the generation process before image generation begins. This alignment process enables the display pose constraint information to participate in the generation calculation in an appropriate form at different generation stages. For example, in the early generation stage, the focus is on constraining the overall spatial layout, while in the later generation stage, the focus is on constraining the detailed poses of the hands and the product, thereby ensuring the continuity and effectiveness of the constraints throughout the entire generation process.

[0128] The attention encoding process refers to the process of weighting and reorganizing input features through an attention mechanism. It is used to highlight key information relevant to the target task and suppress irrelevant information. In this embodiment, the attention encoding process is used to encode the aligned display posture constraint information, enabling the generative model to focus on feature dimensions or spatial regions related to the product display posture. This attention encoding process can employ self-attention, cross-attention, or multi-head attention to form a feature representation with global perception capabilities.

[0129] The global feature information refers to a high-dimensional feature sequence obtained after attention encoding, used to guide the overall image generation process. This global feature information integrates product display posture constraints and contextual information related to the person and the product, and is used as a conditional input in subsequent generation processes. Compared to local features, global feature information can reflect the overall structural relationship and display requirements between the person and the product, thus providing stable and consistent constraint guidance during the generation process.

[0130] The diffusion generation based on latent code representation refers to a generation method that progressively generates or reconstructs an image in the latent space. It involves iteratively updating the latent code over multiple steps, gradually converging the latent code from a random state towards the latent representation corresponding to the target image. In this embodiment, the diffusion generation process uses global feature information as input to guide the direction of latent code updates, ensuring that the generation result continuously conforms to the display pose constraints during progressive denoising or optimization. Compared to one-time generation, this diffusion generation method offers higher stability and controllability.

[0131] The target latent code can refer to the latent spatial representation of the target display image obtained after the diffusion generation process. This target latent code implicitly contains the display posture relationship between the target person and the target product, but has not yet been converted into a visible pixel image form. It serves as the input for subsequent decoding processing to generate the corresponding display image.

[0132] The decoding process refers to mapping the target latent code from the latent space back to the pixel space, which is used to generate a preliminary visualized display image. The decoding process can be implemented using a variational autoencoder (VAE) decoder, a generator in a generative adversarial network, or other latent code-to-image mapping networks. Through decoding, the implicit relationships of human structure, product form, and display posture contained in the latent code can be converted into specific image content.

[0133] The first displayed image may refer to the preliminary image result obtained after decoding the target latent code. It has basically presented the overall shape of the target person displaying the target product, but there may still be deficiencies in the area of ​​contact between the product and the hand, edge transitions, and consistency of lighting and shadows. The first displayed image serves as an intermediate result, providing basic input for subsequent pixel-level fusion processing.

[0134] The pixel-level fusion processing refers to a process of further optimizing the first displayed image at the pixel level. This process improves edge blending, lighting consistency, and shadow naturalness between the product, the person's hand, and the background. This fusion processing can include edge blending, lighting blending, shadow blending, and other similar methods to reduce stitching artifacts in the generated image, making the target displayed image visually more realistic and natural.

[0135] The target display image refers to the final image result obtained after latent code decoding and pixel-level fusion processing, which represents the target person displaying the target product in a manner that meets the display posture requirements. This target display image not only meets the requirements of display posture constraints in terms of structure and posture, but also has a high degree of realism and consistency in visual effects, and can serve as a reliable input for subsequent video generation by the digital human-driven model.

[0136] In this embodiment, the image generation process is completed under the continuous constraint of display posture constraint information. Specifically, firstly, attention encoding is performed on the display posture constraint information after time step alignment to enhance key information related to the target product's display posture at the feature level and form global feature information to guide the generation. Subsequently, using the global feature information as conditional input, the latent code is iteratively updated during the diffusion generation process based on latent code representation, causing the latent code to gradually converge towards the target state that satisfies the display posture constraints, thereby obtaining the target latent code. Based on this, the first display image is obtained by decoding the target latent code, and pixel-level fusion processing is further performed on the first display image to optimize the edge, lighting, and shadow effects of the contact area between the product and the person's hand, ultimately obtaining a target display image that meets the expected display posture.

[0137] This embodiment performs attention encoding on the aligned display posture constraint information to obtain global feature information for guiding generation. Based on this global feature information, it performs diffusion generation based on latent code representation, enabling the image generation process to continuously follow the preset display posture constraints during iterative progress. This effectively improves the stability and consistency of the display relationship between the product and the person's hand. By introducing pixel-level fusion processing after latent code decoding, the contact area between the product and the hand, as well as lighting and shadows, are refined and optimized. This further enhances the quality of the target display image in terms of edge transition, realism, and visual consistency, meeting the dual requirements of accuracy and visual appeal in product display scenarios.

[0138] In some embodiments, inputting the target display image into a digital human driving model to drive the target person to perform actions related to product display in order to generate a target digital human video may include: according to the prompt information, using the digital human driving model to drive the target person in the target display image to perform lip movements and / or body movements corresponding to product display in order to obtain the target digital human video.

[0139] The target display image, specifically, refers to the static image obtained in the preceding image generation step, representing the target person displaying the target product in the intended posture. This target display image not only contains visual information about the target person and the target product but also implicitly includes the established posture relationship between the person's hands and the product, such as the product's grip position, display direction, and relative spatial relationship with the hands. The target display image can serve as the visual input basis for the digital human-driven model, ensuring that the product's display posture remains stable during subsequent action-driven processes.

[0140] The digital human driving model refers to a model used to generate dynamic actions of a digital human based on input visual information (such as a target displayed image) and control information (such as prompts). This model drives the target character to perform continuous lip-sync and / or body movements. The digital human driving model can be implemented using a deep learning-based action generation model, an audio-driven model, or a multimodal driving model. Examples include an audio-driven lip-sync generation model, a semantic instruction-driven body movement generation model, or a joint driving model that simultaneously supports lip-sync and body movement generation. Through the digital human driving model, static images can be transformed into a coherent video sequence.

[0141] The prompting information may include text prompts and / or audio prompts. Text prompts can be used to describe the semantics of the display, such as "introducing product features" or "showing the front of the product." Audio prompts can be directly used as audio input to drive lip movements, or after audio-to-text processing, to drive semantically related body movements. In this embodiment, the prompting information can serve as one of the control inputs for the digital human driving model, used to determine the type, rhythm, or semantic content of the action.

[0142] The lip-syncing refers to the changes in the shape of a target person's mouth in a video in response to changes in the audio content, reflecting the target person's speaking state. Lip-syncing can be generated by a digital human-driven model based on audio signals or text rhythm information derived from audio, ensuring that the opening and closing of the mouth and changes in its shape are synchronized with the audio content in time. By generating lip-syncing, digital human videos can present a natural speaking effect, enhancing the realism and credibility of product demonstrations.

[0143] The aforementioned body movements refer to the body or hand movements performed by the target person during product display, which may include gesture changes, arm swings, body posture adjustments, and other examples. Body movements can be generated by a digital human-driven model based on semantic content in prompts or preset action templates to enhance the expressiveness of the product display. For example, a gesture pointing at the product can be generated when introducing it, and a rotating or lifting motion can be generated when showing details. During the generation of body movements, the predetermined product display posture in the target display image can be preserved.

[0144] The execution of actions corresponding to product display refers to the semantic and formal matching of the actions performed by the target character with the product display scene. This correspondence is reflected in the semantic rationality and visual consistency of the actions. For example, generating lip movements to accompany the explanation when introducing product functions, and generating auxiliary body movements to assist in the display when showcasing the product appearance. Through this correspondence control, the disconnect between actions and product display content can be reduced, improving the overall coordination of the display video.

[0145] The target digital human video refers to a video result generated through a digital human-driven model, containing continuous action frames of a target person. It is used to present the process of a target person showcasing a target product in video format. The target digital human video is continuous in the temporal dimension and maintains a stable product display posture in the spatial dimension. It combines lip movements and / or body movements to achieve an expressive and interactive product display effect. The target digital human video can be directly used in e-commerce live streaming, advertising, or other product sales scenarios.

[0146] It should be noted that, in this embodiment, the display posture constraint information can define an "allowed posture change space," within which the target product can undergo controlled movement in the target digital human video without posture drift. In other words, the display posture constraint information is used to limit the relative posture consistency of the target product with respect to the target person's hand, thereby allowing the target product to undergo overall displacement or rotation with hand movements, while reducing disruption of the gripping relationship between the product and the hand. In this embodiment, "maintaining the display posture" or "stabilizing the display posture" does not mean that the target product remains absolutely still in the video, but rather that the gripping relationship and display direction of the target product relative to the target person's hand remain consistent during video generation, thereby allowing the target product to undergo overall displacement or rotation with the target person's limb movements to achieve a natural product display effect.

[0147] In an exemplary video generation process, the target person initially displays the product by holding it in one hand. The gripping position, direction, and type of the product relative to the target person's hand are determined by the display posture constraint information. In consecutive frames of the generated video, the digital human-driven model drives the target person to slowly raise their arm, causing the product to move upwards along with the hand movement, thus placing the product in the center of the frame. Throughout this process, the product maintains a consistent relative posture with the target person's hand, without any slippage or disengagement. Subsequently, the target person performs a slight rotation around their wrist axis, causing the product to rotate while maintaining the gripping relationship, sequentially displaying the front, side, and back structures of the product, thereby achieving a multi-angle display of the product's appearance details.

[0148] In another example, the prompts include audio prompts introducing product features. Based on these audio prompts, the digital human-driven model guides the target person to perform display actions that match the audio content while maintaining predetermined display posture constraints. For example, when the audio prompt mentions a specific functional area of ​​the product, the target person adjusts their hand posture to slightly rotate or tilt the product, aligning the functional area towards the camera and highlighting the corresponding details. Throughout the video generation process, the target product maintains a stable grip with the target person's hand, and its movements are limited by the display posture constraints, reducing the possibility of product posture drift or disruption of the grip.

[0149] During the video demonstration, the target product does not remain absolutely still in the frame. Instead, within the range defined by the posture constraint information, it undergoes overall displacement and rotation in response to the body movements of the target person. This variation is used to enhance the presentation of the product, for example, by lifting, rotating, or fine-tuning the angle to showcase the product's appearance or structural details. However, because the posture constraint information restricts the relative relationship between the product and the hand, the product maintains a reasonable and natural display state throughout the video generation process, thereby improving upon common problems in related technologies such as product floating, abrupt posture changes, or distorted grasping.

[0150] In this embodiment, after obtaining a target display image that meets the display posture requirements, the target display image is provided as visual input to the digital human driving model, and the actions of the digital human are controlled in conjunction with prompting information. Specifically, the digital human driving model drives the lip movements and / or body movements of the target character based on the semantic content or rhythmic information contained in the prompting information, so that the target character completes speaking, gesturing, or displaying actions that match the product display while maintaining the product display posture in the target display image. This transforms the static target display image into a continuous video sequence, generating a digital human video showcasing the product.

[0151] This embodiment inputs a target display image that meets the display posture requirements into a digital human driving model, and drives the target character to perform lip movements and / or body movements corresponding to the product display based on prompts. This enables dynamic display of the digital human without compromising the stability of the product and the character's hand display posture. As a result, it can enhance the expressiveness and interactivity of the display video while ensuring the accuracy of the product display, reduce the reliance on manual shooting or complex motion reconstruction, improve the efficiency and consistency of product display video generation, and meet the dual requirements of display quality and generation efficiency in product sales scenarios.

[0152] See Figure 2 , Figure 2 This is a structural block diagram of a digital human video generation system provided in an embodiment of this application.

[0153] This application also provides a digital human video generation system, which includes an acquisition module, a generation module, and a driving module.

[0154] The acquisition module is used to acquire images of the target person, images of the target product, and prompts indicating the product display method. The prompts include text prompts and / or audio prompts.

[0155] The generation module is used to generate a target display image representing the target person displaying the target product based on the target person image, the target product image, and the prompt information.

[0156] The driving module is used to input the target display image into the digital human driving model, drive the target person to perform actions related to product display, and generate a target digital human video.

[0157] The prompt information is used to determine the display posture of the target product relative to the hand of the target person, and the display posture is used to constrain the generation process of the target display image.

[0158] In some embodiments, the digital human video generation system may include an image generation system and a digital human driving system. The image generation system may include the acquisition module and the generation module. The digital human driving system may include the driving module.

[0159] In some embodiments, the generation module can jointly model multimodal features through multi-attention mechanisms and other methods, and on this basis, introduce a gradient-corrected diffusion generation strategy to improve the accuracy and consistency of the generation results while ensuring the stability of the product display posture.

[0160] In some embodiments, the generation module may include a feature extraction submodule, a multimodal feature fusion submodule, and an image generation submodule. The feature extraction submodule can be used to extract features based on the target person image, the target product image, and the prompt information to obtain person feature information, product feature information, and semantic feature information. The multimodal feature fusion submodule can be used to generate display posture constraint information based on the person feature information, the product feature information, and the semantic feature information to constrain the display posture. The image generation submodule can be used to generate an image under the constraints of the display posture constraint information to obtain the target display image. The feature extraction submodule, the multimodal feature fusion submodule, and the image generation submodule can work collaboratively to complete the generation process from multimodal input to target display image output.

[0161] In some embodiments, the input of the feature extraction submodule may include a target person image, a target product image, and text prompts, and its output may include person feature information, product feature information, and semantic feature information.

[0162] In some embodiments, the feature extraction submodule may include an image feature extraction unit and a text semantic extraction unit.

[0163] In some embodiments, the image feature extraction unit can be used to perform feature extraction processing on the target person image and the target product image respectively to obtain corresponding person feature information and product feature information. The image feature extraction unit can adopt a variational autoencoder (VAE) encoder structure, with the input including the target person image and the target product image, and the output including the corresponding person feature information (e.g., person feature image) and product feature information (e.g., product feature image) respectively.

[0164] In some embodiments, the text semantic extraction unit is used to perform semantic parsing on "text prompt information" and / or "audio-to-text information obtained based on audio prompt information" to generate semantic feature information for constraining the product display method. The semantic feature information is used to characterize the high-level semantic requirements of the product display method, such as the product's display direction, display angle, visual style, or display focus. The text semantic extraction unit can employ a dual-encoder structure, which may include a first text encoder and a second text encoder. The first text encoder can be a CLIP (Contrastive Language-Image Pre-training, a pre-trained model based on contrastive text-image pairs) text encoder, used to extract semantic information related to visual attributes in the text, such as visual concepts like color, material, or appearance style, and output a first semantic vector. The second text encoder can be a T5 (Text-To-Text Transfer Transformer, a pre-trained language model for natural language processing) encoder, used to perform semantic parsing on longer or more complex text descriptions to extract semantic information related to the product's form, structure, or logical relationships, and output a second semantic vector. By introducing a dual-encoder structure, both visual and logical semantic information in the text can be considered simultaneously.

[0165] In some embodiments, the first semantic vector and the second semantic vector from the first text encoder and the second text encoder can be aligned and fused through a projection layer or a fusion layer to generate a text semantic feature vector of uniform dimension (as an example of semantic feature information). The text semantic feature vector can serve as an important input for subsequent attention encoding and display posture constraint modeling, modulating person feature information and product feature information to achieve semantic control over the product display method. As an example, the CLIP text encoder can extract the corresponding visual semantics from the text prompt information (such as the color distribution corresponding to "red"), outputting a 1×1×768-dimensional vector, and the T5 encoder can parse long text logic (such as the shape description of a "coffee cup"), outputting a 1×1×1024-dimensional vector. These two vectors are fused through a projection layer to obtain a 1×1×1280-dimensional semantic vector, which serves as the text semantic feature vector.

[0166] In some embodiments, the multimodal feature fusion submodule can be used to modulate the person feature information and the product feature information using the semantic feature information to generate the display pose constraint information. The modulation includes at least one of cross-attention processing, feature weighting, and feature mapping.

[0167] In this embodiment, the multimodal feature fusion submodule can be used to perform feature fusion processing between text and images. The input of the multimodal feature fusion submodule may include a person feature image, a product feature image, and a text semantic feature vector, and its output may include display pose constraint information.

[0168] In some embodiments, the modulation may include cross-attention processing. The multimodal feature fusion submodule may include a hand gesture adjustment unit, a dynamic offset unit, and a text conditional fusion unit. The hand gesture adjustment unit may be used to match the display posture of the target product with the hand gesture of the target person based on the person feature information and the product feature information to obtain hand feature information adapted to the grip type of the target product. The dynamic offset unit may be used to adjust the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information and the hand feature information to generate feature matching information for characterizing the target grip state. The text conditional fusion unit may be used to perform cross-attention processing on the feature matching information based on the semantic feature information to generate semantic constraint weight information for characterizing the semantic constraint strength; and use the semantic constraint weight information to perform weighted modulation on the feature matching information to obtain the display posture constraint information.

[0169] The hand gesture adjustment unit can automatically match the target product with the target person's hand, so that the hand and the target product present a natural holding posture. Specifically, the gesture adjustment unit can include a product detection subunit and a hand gesture matching subunit.

[0170] The commodity detection subunit can be used to detect the main commodity region in a commodity feature image (as an example of commodity feature information) and determine the most suitable gripping position for the target commodity based on the main commodity region.

[0171] The product detection subunit may include an object detection network, an image segmentation network, and a gripping position detection network.

[0172] The object detection network can employ a lightweight object detection network (such as YOLOv8). This network detects product boundaries and outputs product size information. This size information can include the boundary of the product's main body region in the product feature image (specifically composed of the product's center point coordinates and the coordinate range of the product's main body), as well as the product's length, width, and height. Simultaneously, the network can also determine the product's thickness information based on a pre-defined product category database (here, thickness refers to the thickness of the product as a whole, not the thickness of a specific location; for example, the thickness of a water cup refers to its horizontal diameter, not the thickness of the cup wall). For instance, if the detected product is a mobile phone, based on data in the product category database, mobile phones are typically 1cm thick, so the product size information records the product thickness as 1cm. Similarly, if the detected product is a water cup, since a water cup typically includes a cup body and a handle, with the cup body typically 10cm thick and the handle 2cm thick, the thickness information at these different locations is recorded together in the product size information.

[0173] Image segmentation networks can employ technologies such as SAM (Segment Anything Model, a deep learning model for image segmentation). These networks segment products based on their size information. Specifically, the image segmentation network takes a product feature image and its size information as input, distinguishes the product feature image according to the coordinates corresponding to the product boundaries, and outputs multiple segmented regions. Each segmented region indicates a different part of the product, such as the upper, middle, or lower part.

[0174] The grip position detection network can determine the most suitable grip position on a product by calculating scores for the multiple segmented regions. The grip position detection network can employ an MLP (Multilayer Perceptron) network, which consists of multiple fully connected layers. The network first performs calculations for each segmented region separately (this calculation process can be implemented independently of the MLP, solely through numerical computation), determining the area, position, and thickness information of each segmented region.

[0175] The area information can refer to the proportion of the area of ​​the segmented region in the total area of ​​the product body. The location information can refer to the distance between the center of the segmented region and the center of the product. The thickness information can refer to the degree of matching between the thickness of the segmented region and the standard grip range of human fingers. The grip range can vary depending on the product. For example, for thin objects like lipstick, the grip range can be 2 to 5 cm; for thicker objects like thermos cups, the grip range can be 10 to 25 cm; and for complex objects, such as water cups, different grip ranges can be set for different areas. The grip range, along with the product category database, can be pre-collected and stored in the digital human video generation system.

[0176] The area, location, and thickness information can be represented by feature values ​​ranging from 0 to 1 in the calculation results. For area information, the larger the area percentage, the larger the feature value; for location information, the closer to the center of the product, the larger the feature value; for thickness information, the higher the matching degree, the larger the feature value. Thus, each segmented region can output a three-dimensional feature vector composed of the corresponding features of area, location, and thickness information. The feature vector corresponding to each segmented region is input into an MLP network for scoring. Based on the weights of the three features for different products, a grasping priority score is obtained for each segmented region. The segmented region corresponding to the highest score is the most suitable region for grasping, and this segmented region is defined as the grasping region. Product grasping parameters are output, which may include the center coordinates and coordinate range of the grasping region, the width and thickness of the grasping region, the area information of the grasping region, and a region mask obtained based on the grasping region.

[0177] The hand pose matching subunit can be used to determine hand feature information (e.g., hand feature image) based on the product grasping parameters, to characterize the hand features of the person grasping the product in the optimal posture. The hand pose matching subunit may include a hand pose matching network, a body pose matching network, and a spatial variation network.

[0178] The hand pose matching network can use the MANO model (a parametric hand model), for example, to calculate hand key points based on the product gripping parameters and output 21 hand key points that are adapted to the product gripping parameters.

[0179] The body pose matching network can include the OpenPose model (a deep learning-based human pose estimation model), for example, used to correct hand keypoints based on the body features of a person in a feature image. Specifically, the feature image is input into the OpenPose model to calculate 17 body joint keypoints (mainly including the shoulders and elbows). Based on these, constraints are applied to the hand keypoints based on the body joint keypoints to reduce incoordination between the hand and body during hand pose adjustment. Specific constraint methods can include elbow flexion constraints and body distance constraints.

[0180] Elbow flexion constraint is calculated based on the shoulder-elbow vector from the body joint key points, and simultaneously based on the elbow-hand vector from the body joint key points and the hand key points. The angle between the two vectors (shoulder-elbow vector and elbow-hand vector) is further calculated. Generally, the elbow flexion angle is between 30° and 120°. If the calculated result is outside this angle range, it indicates that the elbow flexion is too small or too large, and the hand key points need to be corrected.

[0181] Body distance constraint calculates the distance between the hand and the body's central axis based on key points of the body joints and the hand. Generally, this distance is between 15 and 30 cm. If the calculated result is outside this range, it means that the hand is too close or too far from the body, and the hand key points need to be corrected.

[0182] After adjusting the hand key points according to the elbow flexion constraint and body distance constraint, and ensuring that the hand key points meet the constraints, the corrected hand key points can be obtained.

[0183] The spatial transformation network can employ STN (Spatial Transformer Networks) to map the corrected hand keypoints to 2D pixel coordinates, and further perform bilinear interpolation based on the 2D pixel coordinates to generate and output a hand feature image. Since the final state of the target product and the hand is also affected by text prompts, the image processing of the hand feature image can be performed in subsequent units.

[0184] The dynamic offset unit can be used to initially confirm the spatial relationship between person feature images and product feature images based on semantic feature information, so as to adapt it to subsequent product generation.

[0185] For example, a text prompt might require the product to be rotated 45° and placed in front of the target person's chest. If the generation process is executed directly, the hands in the person's feature image are often not precisely positioned in front of the chest, and the product in the product feature image is often not rotated exactly 45°. This would lead to subsequent generation failing to perfectly execute the text prompt, resulting in issues such as product angle shift, separation or overlap between the product and the fingers. To address this, this embodiment pre-calculates the required offset parameters for the hands and product relative to the original image, and uses this to correct for errors in subsequent conditional fusion, encoding, and diffusion generation processes. The dynamic offset unit may include an offset prediction layer, a feature sampling layer, a pose adjustment layer, and a projection layer.

[0186] The offset prediction layer can be used to predict and calculate the spatial offset between the original state of a person and a product in a person feature image and the expected state of the semantic description, based on the text semantic feature vector. The input of the offset prediction layer can include the person feature image and the text semantic feature vector, and the output can include offset parameters (such as translation, rotation, scaling parameters, etc.) and sampling weight parameters.

[0187] Specifically, the offset prediction layer can include a cross-attention network and an offset prediction network. A stitched feature image is obtained by pre-concatenating the person feature image and the product feature image along the channel dimension, and then mapped to a stitched feature sequence.

[0188] The cross-attention network takes the concatenated feature sequence and the text semantic feature vector as input, uses the text semantic feature vector as the query object Q, and the concatenated feature sequence as the query object (key K and value V). It calculates the attention weight of the concatenated feature sequence based on the text semantic feature vector, and assigns weights to the concatenated feature sequence based on the calculation results to obtain the text semantic-guided concatenated feature sequence, which is then mapped to the text semantic-guided concatenated feature image.

[0189] The aforementioned cross-attention network can extract position-related semantic information from text semantic feature vectors, such as spatial descriptions like "above," "left," and "middle," or local semantics like "person's hand" and "product angle." Based on these semantics, it can adjust the weights of the corresponding parts in the person feature image to give higher weights to the position-related features in the person feature image.

[0190] The offset prediction network may include a pooling layer and multiple fully connected layers (the fully connected layer can also be replaced by a lightweight convolutional network). After the text semantic-guided concatenated feature image is input into the offset prediction network, it first undergoes global pooling to compress the features into a global vector. Then, the fully connected layer maps the global vector to low-dimensional space parameters, and further converts the low-dimensional space parameters into offset parameters (the specific conversion can be completed through relevant model training).

[0191] Building upon this, after offsetting the person or product in the stitched feature image, the offset position may experience spatial continuity breaks, feature sparsity, or overlap with surrounding positions. Visually, this manifests as blank areas appearing at the edges of the product, or texture breaks between adjacent positions. The aforementioned problem arises because the surrounding positions are not optimized simultaneously after the corresponding offset processing of the person or product feature image. To address this, the offset prediction layer can calculate sampling weight parameters after calculating the offset parameters. These sampling weight parameters characterize the contribution weight of feature points surrounding the original feature point (the feature point before offset) to the offset feature point (the feature point after offset) during subsequent feature sampling.

[0192] Taking the human feature image portion of a stitched feature image as an example, suppose a feature point in the human feature image is M, and its surrounding feature points include M1, M2, M3, and M4. After offsetting, the offset position of M corresponds to M0. To improve the aforementioned issues of feature sparsity, subsequent feature sampling processes require feature fusion based on M1, M2, M3, and M4. The sampling weight parameters characterize the weights of M1, M2, M3, and M4 respectively during feature fusion. The sampling weight parameters can be calculated using bilinear interpolation, for example, by allocating weights based on the distance between the offset feature point and surrounding feature points; the closer the distance, the higher the weight. Using the example above, let's assume M... The distances between M1, M2, M3, and M4 are 1, 2, 3, and 4 respectively, so the weights assigned to M1, M2, M3, and M4 can be 0.4, 0.3, 0.2, and 0.1 respectively.

[0193] In the calculation of the above sampling weight parameters, the sampling range (i.e., the number of selected surrounding feature points) can be fixed (e.g., four). However, a fixed sampling range usually only achieves good results when the shape is simple and the offset is small. For complex shapes and large offsets, it is insufficient to achieve ideal results. To address this, this embodiment further proposes a dynamic sampling method, which dynamically adjusts the sampling range based on the offset parameter. Dynamic sampling can be determined based on preset rules, which can be as follows: The larger the translation distance, the larger the sampling range. For example, when the translation distance is less than 2 tokens, the sampling range takes 4 surrounding feature points; when the translation distance is greater than or equal to 2 tokens, the sampling range takes 8 surrounding feature points. The larger the rotation angle, the larger the sampling range. For example, when the rotation angle is less than 30°, the sampling range takes 4 surrounding feature points; when the rotation angle is greater than or equal to 30°, the sampling range takes 8 surrounding feature points. When the scaling ratio is greater than or less than a preset value (i.e., the greater the scaling degree), the larger the sampling range. For example, when the scaling ratio is greater than 1.2 (enlargement) or less than 0.8 (shrinkage), the sampling range takes 8 surrounding feature points; within the above intervals, 4 surrounding feature points are taken.

[0194] Furthermore, dynamic sampling can be performed based on the location of feature points in the image. For example, if M is located at the edge of the hand, the sampling range can be increased by 2 points to prioritize covering the original features in the edge direction and reduce edge breakage. If M is located in a solid color area of ​​the person, the sampling range can be reduced by 2 points to reduce redundant calculations. If M is located at the boundary between the hand and the background, the sampling range can be fixed to the minimum, such as 4 points, to reduce the introduction of background features.

[0195] In addition to dynamic sampling based on preset rules, a preset prediction network can also be used for calculation. The accuracy of dynamic sampling can be further optimized through pre-training, and the prediction approach is similar to the aforementioned rules. Some of the above embodiments use human feature images as an example for illustration. The processing approach for product feature images is similar and will not be repeated here.

[0196] The feature sampling layer can be used to sample features from the stitched feature image based on the offset parameter, thereby adjusting the spatial position offset.

[0197] The input to the feature sampling layer can include the stitched feature image, offset parameters, and sampling weight parameters, and the output includes the spatially adjusted stitched feature image. The feature sampling layer can include a spatial transformation network. In actual processing, the offset parameters are first converted into a bias matrix, and the spatial coordinates of the features in the stitched feature image are transformed according to this bias matrix. In this way, the stitched feature image after offset adjustment can be obtained.

[0198] Based on this, the feature sampling layer is further adjusted according to the sampling weight parameters. The sampling range selected by the offset prediction layer and the corresponding sampling weight parameters are weighted and summed to complete the feature fusion of the offset feature points and the surrounding feature points. The fused offset feature points can significantly improve spatial continuity and improve problems such as feature sparsity and overlap, and finally obtain the spliced ​​feature image after spatial position adjustment.

[0199] The pose adjustment layer can be used to further adjust the stitched feature image based on the hand feature image obtained during the hand pose matching process. The pose adjustment layer may include a pose fusion network and a product contact correction network.

[0200] Pose fusion networks can be used to fuse hand feature images with stitched feature images, adjusting the hand region in the stitched feature image to match the hand feature image. The pose fusion network first separates style and pose information in the stitched feature image. Specifically, the stitched feature image can be input into two parallel 1×1 convolutional layers, which output style and pose information respectively. Then, the pose information and hand feature image are input into an MLP network for weight allocation. Based on the dynamic weights learned during the MLP network's initial training, weights are calculated for the features representing the hand in the pose information and the features representing the hand in the hand feature image. The output is the pose information corrected based on the hand feature image, according to the weight allocation result. For example, for small or complex items that are difficult to hold, a higher weight is assigned to the hand feature image obtained from the grasping posture to enhance the accuracy of hand grasping; conversely, for regular, large items, a lower weight can be assigned to the hand feature image to preserve the consistency between the item and the hand as a whole. Finally, the corrected pose information and style information are stitched together to obtain the stitched feature image with the pose adjustment completed.

[0201] The above steps have completed the posture adjustment of the hand area, but when actually grasping the product, there may still be some details that need to be optimized, especially the contact position between the fingers and the product. The product contact correction network can further integrate the product with the contact area of ​​the hand.

[0202] The product contact correction network first obtains the corresponding product grasping mask (as an example of segmentation results) and hand feature mask based on the product grasping feature image and hand feature image corresponding to the grasping region. Then, based on the product grasping mask and hand feature mask, the product contact correction network extracts the feature images corresponding to the product grasping part and the hand part from the stitched feature image after the pose adjustment is completed (since these two parts have undergone offset prediction processing, they are different from the product and hand parts directly obtained when calculating the pose adjustment).

[0203] Based on the feature images corresponding to the product's grasping part and the hand part, product grasping feature sequences and hand feature sequences are obtained respectively. Then, through a cross-attention network, the product grasping feature sequence is used as the query object Q, and the hand feature sequence is used as the query object (key K and value V). The attention weight of the hand feature sequence is calculated according to the product grasping feature sequence, and the hand feature sequence is weighted according to the calculation result. The hand feature sequence is obtained after correction based on the product contact position. After mapping, the stitched feature image of the product and hand maintaining the best grip posture can be obtained, that is, the stitched feature image after posture adjustment (as an example of posture matching feature information).

[0204] The projection layer can be used to map the channel dimensions of the pose-adjusted stitched feature image. The input to the projection layer can include the pose-adjusted stitched feature image and the text semantic feature vector. The projection layer can be a fully connected layer, which maps the channel dimensions of the text semantic feature vector to match the channel dimensions of the stitched feature image. After completing the channel dimension mapping, the projection layer outputs the stitched feature image and the text semantic feature vector, which have undergone the aforementioned processing.

[0205] The text conditional fusion unit can use a cross-attention structure to interact the text semantic feature vector with the person feature image and the product feature image respectively, generating semantically guided person feature images and product feature images. For example, the semantic "clenching" will enhance the structural features of the corresponding area of ​​the hand, and the semantic "red" will enhance the color features of the corresponding area of ​​the product.

[0206] Specifically, the input to the text conditional fusion unit may include the stitched feature image that has undergone the above processing and the text semantic feature vector, and the output may include semantically guided person feature image and product feature image.

[0207] The text conditional fusion unit may include a cross-attention unit. The cross-attention unit first maps the input concatenated feature image into a concatenated feature sequence. Then, it performs cross-attention calculation based on the concatenated feature sequence and the text semantic feature vector. During the cross-attention calculation, using the text semantic feature vector as the query Q and the concatenated feature sequence as the key K and value V, a weight matrix (as an example of semantic constraint weight information) is calculated. This weight matrix represents the weight allocation of the concatenated feature sequence based on the text sequence vector; a higher weight indicates that the image region needs to adhere more strictly to the text constraints. After the calculation, the weights of the original concatenated feature sequence obtained by mapping the concatenated feature image are adjusted based on the weight matrix, and the weight-adjusted concatenated feature sequence is then mapped to obtain a semantically guided concatenated feature image. This semantically guided concatenated feature image represents the concatenated feature image after text conditional fusion, and it can serve as an example of displaying pose constraint information.

[0208] It's important to note that while both text conditional fusion and offset prediction involve cross-attention mechanisms to adjust the weights of image features through text, their processes, functions, and effects differ. In offset prediction, the cross-attention mechanism is used only to calculate the weights of image features, which are then used to calculate specific offset parameters and sampling weights, and to perform offset processing on the image features. In text conditional fusion, however, the cross-attention mechanism is used to fuse the conditions indicated by semantic features with image features. This process does not process the image itself; it is achieved through subsequent image generation. If text conditional fusion is performed directly without offset prediction, the aforementioned problems will inevitably arise because both person and product features are unadjusted. Therefore, offset prediction is fundamental to achieving better results in text conditional fusion. Furthermore, regarding the cross-attention mechanism itself, since offset prediction has lower accuracy requirements while text conditional fusion has higher accuracy requirements, the former can use a lightweight structure with a small number of attention layers, while the latter requires more attention layers. In other words, the number of attention layers in the cross-attention network in the offset prediction layer can be less than the number of attention layers in the cross-attention unit in the text conditional fusion unit.

[0209] In some embodiments, the multimodal feature fusion submodule may further include a time-step embedding fusion unit. The time-step embedding fusion unit is used to align the display pose constraint information (such as the semantically guided stitched feature image output by the text conditional fusion unit) with preset time-step information before image generation, so as to constrain the image generation process of the target display image based on the aligned display pose constraint information.

[0210] The temporal step embedding fusion unit can be used to convert temporal step information into a temporal step vector with the same channel dimension as the input concatenated feature image (i.e., the semantically guided concatenated feature image output by the text conditional fusion unit) through sinusoidal positional encoding. This temporal step vector is then concatenated with the input concatenated feature image along the channel dimension to generate a temporal conditional feature image. This temporal conditional feature image can serve as an example of aligned display pose constraint information.

[0211] The time step information here can include the number of diffusion steps t during the diffusion generation process performed by the diffusion generation submodule, as well as the noise scheduling parameters corresponding to different diffusion step numbers t. The noise scheduling parameters can be used to indicate the specific noise scheduling method. Depending on the diffusion framework, the value of t can be different. For example, in DDPM (Denoising Diffusion Probabilistic Models), the value of t can be any integer between 0 and 1000. The diffusion framework can also be implemented based on streams, and the value of t can be between 0 and 1, such as t=0.3, 0.5, etc.

[0212] During training and inference, the diffusion model needs to perform forward noise addition or backward noise reduction on the image according to the noise scheduling method described above, based on each diffusion step. The value of 't' guides the diffusion model on how to inject noise at different diffusion steps. A higher 't' value results in more noise injected into the image; when 't' reaches max (maximum value, such as 't=1' above), the image corresponds to pure noise. Conversely, a lower 't' value results in less noise injected into the image; when 't' reaches min (minimum value, such as 't=0' above), the image corresponds to a clear image. The diffusion steps and corresponding noise scheduling parameters can be preset for the diffusion model. As an example, 't' can focus on the overall picture in the early stages and on details in the later stages.

[0213] During the training of the diffusion model, t is a random variable. By randomly assigning t, the model learns how to adjust the noise for different t values. During the inference process of the diffusion model, t is iterated from high to low.

[0214] In some embodiments, the image generation submodule can be used to constrain the image generation process based on the display posture constraint information, so that the image generation process gradually forms a target display image that conforms to the display posture.

[0215] In some embodiments, the image generation submodule may include an attention encoding unit, a diffusion generation unit, a feature decoding unit, and a pixel fusion unit. The attention encoding unit performs attention encoding based on aligned display pose constraint information to obtain global feature information. The diffusion generation unit performs diffusion generation based on the global feature information and latent code representation to obtain a target latent code. The feature decoding unit decodes the target latent code. The pixel fusion unit performs pixel-level fusion processing on the decoded first display image to obtain the target display image.

[0216] Attention encoding units can be used to perform attention encoding on aligned display pose constraint information (such as temporal conditional feature images). The input of an attention encoding unit can include the temporal conditional feature image, and the output can include global feature information (such as global feature sequences). An attention encoding unit can include independent two-stream cross-attention subunits and single-stream self-attention subunits, corresponding to diffusion generation units, respectively. The structures of the two-stream cross-attention subunits and single-stream self-attention subunits can be identical, differing only in the number of two-stream attention block layers in the two-stream cross-attention subunit and the number of single-stream self-attention block layers in the single-stream self-attention subunit (details of the differences and their relationship with the diffusion generation unit are described later).

[0217] Attention coding units can employ a combination of two-stream and one-stream attention mechanisms. These units may include image preprocessing subunits, text preprocessing subunits, two-stream cross-attention subunits, three-dimensional rotation coding subunits, and one-stream self-attention subunits.

[0218] The image preprocessing subunit and the text preprocessing subunit can adopt a parallel structure, and are used to extract the corresponding image stream and text stream respectively based on the temporal condition feature image.

[0219] The image preprocessing subunit may include an image projection layer and an image self-attention layer. The image projection layer may consist of a 3×3 convolutional layer, a batch normalization layer, and residual connections. It can be used to perform feature processing on temporally conditional feature images, outputting a sequence of image tokens. The image self-attention layer may include a multi-head self-attention structure and a normalization layer. It can be used to ensure that each image token in the sequence (e.g., a pixel of a hand) pays attention to other spatially adjacent and temporally continuous image tokens. The dynamic relationships between image tokens are determined through the allocation of attention weights, such as the spatial change of the hand from open to closed and the corresponding temporal relationship of the action.

[0220] The text preprocessing subunit can include a text projection layer and a text self-attention layer. The text projection layer can consist of a 1×1 convolutional layer, a global average pooling layer, and residual connections. It can be used to perform feature processing on the temporally conditional feature image, outputting a text token sequence. Similar to the image self-attention layer, the text self-attention layer can include a multi-head self-attention structure and a normalization layer. It can be used to make each text token in the text token sequence pay attention to other semantically related and temporally connected text tokens. For example, "hold" will pay attention to the semantics preceding and following "raise hand" and "grab." The dynamic association between text tokens is determined through the allocation of attention weights, such as the correspondence between text descriptions and action sequences.

[0221] It should be noted that the processing approach of the aforementioned preprocessing subunits (including image preprocessing and text preprocessing subunits) is not simply to separate different features from the fused temporal conditional feature image again. Instead, it focuses on the temporal conditional feature image from different perspectives based on the allocation of different weights, thereby obtaining the corresponding image token sequences and text token sequences. For the image preprocessing subunit, its weight parameters are optimized for image information, focusing on preserving the correlation between the spatial and temporal dimensions. For the text preprocessing subunit, its weight parameters are optimized for semantic information, strengthening the correspondence between the semantic and temporal dimensions. The above processing is the foundation for the subsequent implementation of dual-stream cross-attention; therefore, this step should not be confused with simple feature segmentation.

[0222] The image token sequences and text token sequences that have undergone attention processing, output by the image preprocessing subunit and text preprocessing subunit respectively, are processed by the dual-stream cross-attention subunit. The dual-stream cross-attention subunit may include multiple layers of dual-stream attention blocks, and each layer of dual-stream attention block may include a dual-path cross-attention subunit, an image postprocessing subunit (corresponding to the image sequence), and a text postprocessing subunit (corresponding to the text sequence).

[0223] The dual-path cross-attention subunit can include two independent processing paths. Path 1 (text path) uses the image token sequence as the query object Q and the text token sequence as the query object (key K and value V), calculating a text weight matrix. This text weight matrix represents the weight allocation of the text token sequence based on the image token sequence, and the corresponding output is a weight-adjusted text token sequence. Path 2 (image path) uses the text token sequence as the query object and the image token sequence as the query object, calculating an image weight matrix. This image weight matrix represents the weight allocation of the image token sequence based on the text token sequence, and the corresponding output is a weight-adjusted image token sequence.

[0224] After the above calculations are completed, the original image token sequence is adjusted by weights to obtain the updated image token sequence (i.e., the weighted image token sequence), and the original text token sequence is adjusted by weights to obtain the updated text token sequence (i.e., the weighted text token sequence). The updated image token sequence can be input into the image post-processing subunit, and the updated text token sequence can be input into the text post-processing subunit.

[0225] The image post-processing subunit and the text post-processing subunit can have the same structure, but their parameters can be different and independent of each other. They can both include a normalization layer, a feedforward network layer, and a gated processing layer.

[0226] In the image post-processing subunit, the normalization layer can use AdaLN (Adaptive Layer Normalization) to dynamically normalize based on the distribution characteristics of image features (such as strong spatial correlation and stable numerical range), and incorporate time step information to adapt to the needs of the diffusion generation stage, such as strengthening the global structure in the early stage and preserving details in the later stage.

[0227] Feedforward network layers can be used to enhance the spatial correlation of image features (e.g., the positional dependence of a person's arm and a product, the continuity of background texture, etc.) and capture details (such as lighting gradations and material differences) through nonlinear transformations. Structurally, feedforward network layers can adopt a structure of a first fully connected layer - a nonlinear activation function (such as GELU, ReLU) - a second fully connected layer. The first fully connected layer can be used to map the normalized token sequence to a higher dimension, expanding the feature expression space; then, the nonlinearity is introduced by the nonlinear activation function, which can learn complex feature correlations; finally, the second fully connected layer maps the high-dimensional features back to the original dimension. Here, ReLU stands for Rectified Linear Unit, and GELU stands for Gaussian Error Linear Unit.

[0228] The gating processing layer can learn the gating parameters to dynamically adjust the relative weights of the initial token (the token input into the dual-stream attention block) and the token after the aforementioned series of processing (the token output from the dual-stream attention block to the gating processing layer) for each token, and output the final token of the dual-stream attention block based on the relative weights, thereby reducing the excessive influence between text features and image features during the dual-stream cross-attention calculation process.

[0229] Similar to the image post-processing subunit, the text post-processing subunit uses AdaLN for its normalization layer to optimize the consistency of text feature distribution and reduce interference from excessively large feature amplitudes of individual high-frequency words. The feedforward network layer focuses on enhancing the semantic logic of text features, capturing abstract semantics in language through non-linear transformations. The structure of the feedforward network layer and subsequent gating layers is similar to that of the image post-processing subunit and will not be elaborated further here.

[0230] After the image token sequence and text token sequence are processed by their respective image post-processing subunits and text post-processing subunits, the calculation of the current layer's two-stream attention block is completed and output. This process is then fed into the next layer's two-stream attention block, repeating the above steps to progressively deepen the cross-modal alignment of the image and text. This process is repeated iteratively through multiple layers of two-stream attention blocks, ultimately resulting in the two-stream cross-attention subunit outputting deeply interactive image and text feature sequences.

[0231] The image feature sequence output by the dual-stream cross-attention subunit can be input into the 3D rotation coding subunit. The 3D rotation coding subunit may include one or more rotational positional coding layers, which are used to embed each token in the image feature sequence with positional encoding based on RoPE (Rotary Positional Embedding).

[0232] The output of the 3D rotation coding subunit can be an image feature sequence containing positional information. In this image feature sequence, the feature vector of each position already contains positional information in 3D space, providing a feature representation with positional information for subsequent single-stream self-attention calculation.

[0233] The input to a single-stream self-attention subunit can consist only of an image feature sequence containing location information, and the output can include a global feature sequence. A single-stream self-attention subunit can comprise multiple layers of single-stream self-attention blocks, each of which can include a self-attention layer and a feedforward network layer.

[0234] The self-attention layer can employ a multi-head self-attention structure. After the image feature sequence is input into the self-attention layer, it first passes through multiple linear projection layers to convert the image feature sequence into corresponding query (Q) matrices, key (K) matrices, and value (V) matrices. Then, the query (Q) matrix, key (K) matrix, and value (V) matrix are split into multiple attention heads according to the feature dimension. Each attention head corresponds to a Q sub-matrix, K sub-matrix, and V sub-matrix, respectively. The attention weights of these multiple attention heads are calculated independently to obtain the corresponding weight matrices, thus completing the weight allocation for that attention head. After the calculation is completed, the outputs of each attention head are concatenated to obtain the first output sequence of the self-attention layer.

[0235] For the first output sequence, a residual connection is made between it and the input image feature sequence to reduce the loss of feature information in the deep network. The resulting sequence is then standardized by layer normalization to achieve a stable distribution of features, thus obtaining the second output sequence.

[0236] The second output sequence is input into a feedforward network layer for processing. This feedforward network layer may include a first fully connected layer, a second fully connected layer, and a non-linear activation function (such as GELU or ReLU). The first fully connected layer maps the input second output sequence to a higher dimension, expanding the feature representation space. Then, the non-linear activation function introduces non-linearity, enabling the learning of complex feature relationships. Finally, the second fully connected layer maps the high-dimensional features back to the original dimension, obtaining the third output sequence, which is then output.

[0237] For the third output sequence, a residual concatenation is performed between it and the second output sequence, and the resulting sequence is then subjected to layer normalization to obtain the fourth output sequence. This fourth output sequence is the output of the single-stream self-attention block in this layer.

[0238] The fourth output sequence is input into the next layer of single-stream self-attention block, and the above operation is repeated. After iterative calculation through multiple layers of single-stream self-attention blocks, the final output feature sequence is the global feature sequence (e.g., global feature vector) output by the attention encoding unit.

[0239] In the aforementioned attention calculation process, the text feature sequence output by the two-stream cross-attention subunit is no longer used as input, but is retained only as a constraint condition for semantic constraint during the attention calculation of the image feature sequence. For example, the text feature sequence is first converted into a global semantic condition vector through pooling, which represents the core indication of the text. The global semantic condition vector can supervise the single-stream self-attention calculation of the image feature sequence in the following two ways.

[0240] Method 1: Global semantic features are used to generate parameter modulation factors through a first fully connected network. These parameter modulation factors are then input into the projection parameters of the linear projection layers corresponding to Q, K, and V, respectively, for interaction. For example, the parameter modulation factors are multiplied or added to the projection parameters to conditionally inject the linear projection layers. Subsequently, when image feature sequences are input into these linear projection layers, textual semantic preferences are assigned to the generated Q / K / V results. For instance, if the text describes a product tilted 30° clockwise, the above processing will make the Q generated from the image features more focused on the relative angle of the product. This will allow for higher weighting of the angle-matching tokens during subsequent attention calculations.

[0241] Method 2: Generate a supervision matrix from the global semantic features through a second fully connected network, multiply the supervision matrix by the weight matrix output by the self-attention layer in at least one single-stream self-attention block, and use the result as the basis for the self-attention layer to assign weights to the image feature sequence.

[0242] The aforementioned supervision matrix can be used to calibrate the original weight vector of a single-stream self-attention block. For example, if the text description includes a red apple, the generated supervision matrix will indicate that the red and apple parts should be given priority. In the matrix calculation, higher values ​​(such as 0.9) are assigned to the apple region and the color region in the original weight vector, while lower values ​​(such as 0.1) are assigned to other regions not described in the text.

[0243] In the above operation, the weight matrix output by the self-attention layer in each single-stream self-attention block can be multiplied with the supervision matrix (i.e., each layer is supervised), or specific layers can be selected for supervision. As an example, two to three layers can be selected from the shallow single-stream self-attention blocks (e.g., the first 1 / 3 to 1 / 2 of the single-stream self-attention blocks) and two to three layers from the deep single-stream self-attention blocks (e.g., the last 1 / 3 to 1 / 2 of the single-stream self-attention blocks) for multiplication with the supervision matrix, while the remaining single-stream self-attention blocks retain the original weight matrix.

[0244] The two methods mentioned above can be used individually or simultaneously. The effect of both is to ensure that the region corresponding to the text description is given priority during the single-stream self-attention calculation process.

[0245] In this embodiment, the attention encoding unit works in conjunction with the dual-stream attention module and the single-stream self-attention module. The number of layers in the dual-stream attention block and the single-stream self-attention block can be described later. The global feature sequence can be used as a condition for generation in the subsequent diffusion generation process, forming a mandatory task constraint on the generation process.

[0246] The diffusion generation unit can be used to generate images based on global feature sequences using diffusion generation. The diffusion generation unit can include a first diffusion generation unit, a second diffusion generation unit, and a discriminator unit. The first diffusion generation unit can be used for model training and inference, while the second diffusion generation unit and the discriminator are only used for model training.

[0247] The first diffusion unit and the second diffusion unit have basically similar structures, both of which can include gradient correction prediction subunits and latent code update subunits.

[0248] The gradient correction prediction subunit can be used to generate gradient correction vectors. These vectors characterize the optimal path for correcting the noisy latent code at a given time step during the diffusion generation process. The gradient correction prediction subunit can include a linear projection layer, a sequence reconstruction layer, and a local fusion layer.

[0249] The input to the linear projection layer can include a global feature sequence. The linear projection layer can perform dimensionality compression on the global feature sequence according to a preset latent code channel dimension (e.g., 4 channels) to obtain the latent code feature vector after dimensionality reduction, which is then used as the output of the linear projection layer.

[0250] Sequence recombination layers can be used to recombine latent code feature vectors to obtain latent code feature images. Since the latent code feature vectors obtained after dimensionality reduction of the global feature sequence are still 1D (one-dimensional) feature sequences, subsequent processing needs to be performed according to 2D (two-dimensional) spatial feature images. Therefore, the latent code feature vectors can be recombined according to spatial arrangement to obtain 2D latent code feature images.

[0251] Local fusion layers can be used to perform local fusion of latent code feature images. For example, a local fusion layer can perform weighted fusion of feature values ​​from multiple channels corresponding to each spatial location in the latent code feature image, thereby reducing information conflicts within channels, preventing potential stitching artifacts in subsequent images, and making the overall image transition smoother. The latent code feature image after local fusion is used as the conditional latent code.

[0252] When time-step information is embedded in the global feature sequence, the conditional latent code obtained from the global feature sequence can represent the conditional latent code corresponding to different time-step information, that is, the ideal state of denoising the noisy latent code at a certain time step. Based on this, the gradient correction vector prediction model is trained based on the differential calculation between the conditional latent code and the current noisy latent code, and the gradient correction vector prediction model can be used to predict the gradient correction vector. Mathematically, the gradient correction vector can represent the rate of change of the latent code per unit time. In the application scenario of this embodiment, it can represent how to correct the current noisy latent code between two adjacent time steps to make it conform to the optimal path of the conditional latent code at the current time step.

[0253] The latent code update subunit can be used to update the noisy latent code at the current time step according to the aforementioned gradient correction vector, and generate the latent code for the next time step; the update method is as follows.

[0254]

[0255] in, The latent code indicating the next time step; Represents the noisy latent code at the current time step (as an example of the current latent code); Represents the gradient correction vector. Indicates the time step interval.

[0256] Based on the above calculations, through iterative processing of the noisy latent code, the latent code can be corrected along the gradient correction term at each step of the diffusion generation process, gradually transforming the image from noise to clear features. After iteration, the latent code update subunit outputs the target latent code, which indicates the latent code representation of the item held by the target person.

[0257] In this embodiment, the latent code is a feature representation of an image in the latent space, which typically exists in the form of a multidimensional tensor with a spatial structure. Since the latent code corresponds to the image in terms of spatial organization, it can also be called a latent code image or a latent space image representation, without causing ambiguity.

[0258] During the inference phase, the gradient correction prediction subunit no longer obtains the gradient correction vector based on explicit differential calculation between the conditional latent code and the current noisy latent code. Instead, it utilizes the gradient correction vector prediction model obtained during the training phase to perform forward inference based on the global feature sequence corresponding to the current time step and the noisy latent code at the current time step, directly predicting the gradient correction vector corresponding to the current time step. The gradient correction vector is used to indicate the optimal correction direction for updating the current noisy latent code at the current time step, thereby driving the latent code to evolve along a path that satisfies the conditional constraints between adjacent time steps.

[0259] Based on the diffusion generation method in this embodiment, the approximate time step can be 4 to 8 steps. Within this range, the time step can be manually set according to actual needs. Generally speaking, fewer time steps result in higher generation efficiency, while more time steps result in higher quality of generated details. For example, if the user expects to generate the target display image in real time, the time step can be set to 4 steps; if the user expects to improve the image generation effect, the time step can be set to 8 steps. In comparison, some deep learning-based text-to-image generation models require 20 to 30 steps, and unimproved DDPM-type models require, for example, 600 to 1000 steps. Obviously, the diffusion generation method adopted in this embodiment significantly reduces the diffusion generation time and lowers resource requirements.

[0260] Based on this, the differences between the first diffusion unit and the second diffusion unit are as follows.

[0261] The first diffusion unit and the second diffusion unit are each equipped with an independent attention encoding unit (the other units can share the same unit). In the attention encoding unit of the first diffusion unit, the dual-stream attention block can be configured with 16 to 32 layers, and the single-stream attention block can be configured with 32 to 64 layers. In the attention encoding unit of the second diffusion unit, the dual-stream attention block can be configured with 6 to 8 layers, and the single-stream attention block can be configured with 10 to 16 layers.

[0262] The time step in the first diffusion unit can be set to 60 to 100 steps, and the time step in the second diffusion unit can be set to 4 steps. (The time step setting of the first diffusion unit here is only for optimizing the model effect during training. The diffusion generation model itself can still achieve the ideal diffusion generation effect with fewer time steps, so it does not contradict the above setting of 4 to 8 steps for diffusion generation.)

[0263] During model training, the second diffusion unit is first trained using the first dataset, which contains labeled data. After training is complete (the parameters of the second diffusion unit are frozen after this training and will not be updated in reverse later), the second dataset is selected and input into the first diffusion unit and the trained second diffusion unit, respectively, to obtain the first and second generated results. The second dataset contains unlabeled data, and its data volume is usually much smaller than that of the first dataset.

[0264] After obtaining the first and second generated results, they are input into the discrimination unit for discrimination. The discrimination unit may include a lightweight CNN (Convolutional Neural Network) or MLP network, used to discriminate between the first and second generated results, determine their similarity, and output a discrimination result. This discrimination result is a probability value; the closer it is to 1, the more similar the two are, and the closer it is to 0, the greater the difference between the two.

[0265] Based on the aforementioned discrimination results, the first diffusion unit and its corresponding attention encoding unit are modified. On one hand, the discrimination results are used as input to the first diffusion unit and its corresponding attention encoding unit, with the training objective being to force the first diffusion unit to generate results that make the discrimination result of the discrimination unit approach 1, thereby completing the training of the adversarial loss function. On the other hand, the second generation result generated by the second diffusion unit is directly used as input to the first diffusion unit and its corresponding attention encoding unit, with the training objective being to drive the first diffusion unit to generate results that increasingly approach the second generation result, thereby completing the training of the distillation loss function.

[0266] Ultimately, the loss function of the first diffusion unit can include an adversarial loss function and a distillation loss function, thereby completing the training of the first diffusion unit.

[0267] In this embodiment, the global feature sequence obtained through the prior multi-attention mechanism and the denoising method based on gradient correction vectors during the diffusion generation process work together to ensure that the diffusion process strictly follows the task given by the target product and the prompt information. Furthermore, the iteration magnitude of each iteration is proportional to the overall process, allowing the image to accurately approximate the predetermined target from noise within a finite number of iterations. Therefore, this embodiment simultaneously satisfies the real-time and accuracy requirements of the diffusion generation process, making it particularly suitable for scenarios like product display where both image accuracy and real-time performance are critical.

[0268] Building on this, the first diffusion unit and the second diffusion unit work together to perform knowledge distillation, thereby making the overall model corresponding to the second diffusion unit more lightweight and enhancing its effectiveness in the multi-demand scenario of product display.

[0269] The feature decoding unit can be used to input the target latent code into the decoder (such as a VAE decoder) to obtain a preliminary target display image (as an example of a first display image). This preliminary target display image indicates a basically completed image of the target person displaying the target product, but it may have certain defects in terms of edges, lighting, and shadows, which can be further processed by the pixel fusion unit.

[0270] The pixel fusion unit can be used to perform pixel fusion on the initial target display image, making edges, lighting, and shadows more consistent with realistic grip types, thereby obtaining the final target display image. For example, the image generated by diffusion may have issues such as jagged edges, lighting and shadow distortion. To further improve the realism of the target display image, this embodiment can use the pixel fusion unit to process the aforementioned edges, lighting, and shadows separately. The pixel fusion unit may include an edge fusion subunit, a lighting fusion subunit, and a shadow fusion subunit.

[0271] The edge blending subunit can be used to perform edge blending processing on the initial target display image and output a first-order blended image. To ensure a smooth transition between the target product and the edges of other areas, a Poisson blending model can be used to process different areas. The purpose of Poisson blending is to adjust the gradient of the initial target display image (the gradient refers to the rate of change of pixel values ​​at a point in the image along the x and y axes, specifically reflecting the degree of abrupt changes in brightness, color, and contour of the corresponding area in the image) under preset constraints, so that the gradient changes between different parts of the initial target display image are continuous and smooth, resulting in a more ideal edge connection between the product and the finger.

[0272] Different markers can be used to differentiate between the finger and the product based on their different interactions. During the fusion process, gradient processing can be applied to the initial target display image based on these different markers, and the corresponding rules are as follows. For the fourth marker (corresponding to the product itself), it is necessary to ensure that the internal gradient information is completely consistent with the initial target display image, changing only in the areas near the edges of the first and second markers. For the first marker (corresponding to the edge of the occluded finger), it is necessary to strictly prevent the target product from intruding into the finger area of ​​the target person. Therefore, in actual processing, the gradient information of the finger edge at the position corresponding to the first marker is used as a constraint, and the gradient information from the target product area to the finger edge is smoothly transitioned through the calculation of the Poisson fusion model.

[0273] For the second marker (corresponding to the edge of the pressed finger), this part needs to take into account the allocation of the target product and the target person's finger. In this embodiment, gradient weight allocation calculation can be introduced for this area, as shown below.

[0274] First, a first press buffer area is formed by extending a certain distance (e.g., 1 to 5 pixels) to both sides from the edge of the area corresponding to the target product. Simultaneously, a second press buffer area is formed by extending a certain distance (e.g., 1 to 5 pixels) to both sides from the area of ​​the finger that contacts and presses the target product. The first and second press buffer areas overlap at least partially, thus forming three zones: the product area entirely within the target product, the finger area entirely within the target person's finger, and the overlapping area.

[0275] Secondly, the gradient weights for finger transparency and / or target product are adjusted in the three intervals mentioned above.

[0276] Within the overlapping area, the gradient weight of the product and the transparency of the finger gradually decrease as the finger extends towards the product. For example, the closer the finger is to the product, the higher its transparency, indicating that the finger becomes more transparent closer to the edge, resulting in lower finger realism. Simultaneously, the gradient weight of the product is higher, indicating an increased visual proportion of the product and better presentation of its details. Conversely, the closer the finger is to the inside, the lower its transparency, indicating that the finger becomes more realistic closer to the inside, resulting in higher finger realism. At the same time, the gradient weight of the product gradually decreases, indicating a reduced visual proportion of the product, ensuring that the tactile sensation of pressing the finger is preserved.

[0277] The gradient weights of target products in the product area and the transparency of fingers in the finger area still follow the above principle of change, that is, increasing from the inside (finger) to the outside (product) and decreasing from the outside to the inside.

[0278] In actual processing, the gradient of the finger can be calculated from the initial target image and the gradient of the product can be calculated from the deformed product image using preset gradient operators (such as the Sobel operator and the Prewitt operator). Through these gradient changes, the harsh edges between the target product and the hand and background can be reduced, making the pixel gradient of the target product and the gradient of the hand edge natural and continuous, reducing the sense of splicing in the contact area between the finger and the product.

[0279] The lighting blending subunit can employ a lighting model (such as the Retinex model or CycleGAN) to ensure that the lighting effect of the product matches that of the hand, based on the first-stage blended image. The input to the lighting blending unit can include the first-stage blended image, and the output can include the second-stage blended image, which refers to an image where the lighting style of the product and the hand are matched.

[0280] The shadow blending subunit can employ a shadow model (such as the CycleGAN model) to ensure that the shadow effect of the target product matches the background based on the secondary blended image. The input to the shadow blending model can include the secondary blended image, and the output can include the final target display image, which is a blended result with uniform shadow brightness and natural transition.

[0281] In some embodiments, the driving module can be used to drive the target person in the target display image to perform lip movements and / or body movements corresponding to the product display based on the prompt information and using the digital human driving model, so as to obtain the target digital human video.

[0282] In a specific application scenario, as an example rather than a limitation, the digital human driving model can be used to drive the lip movements and hand gestures of a digital human based on input audio prompts.

[0283] In this embodiment, the digital human driving model can be the EchoMimic model (an artificial intelligence portrait animation model), the Keling model, etc. By inputting the audio prompt information and the target display image into the digital human driving model, the corresponding target digital human video can be generated.

[0284] In summary, this embodiment provides the overall generation process of the target digital human video, and this embodiment can be applied to scenarios including but not limited to e-commerce live streaming and advertising sales.

[0285] This application embodiment also provides a display device, which may include a display module and a control module. The control module is used to execute any of the above-described digital human video generation methods and to display the target digital human video on the display module.

[0286] In some embodiments, the display device may further include an interaction module, which is used to receive images of a target person, images of a target product, and prompts indicating the display method of the product and provide them to the control module.

[0287] A display device can refer to a terminal device used to carry and display digital human video content, and can visualize the results of digital human video generation. In some embodiments, the display device can also provide an interactive entry point for users. In different embodiments, the display device can be a standalone hardware terminal or a functional module integrated into an existing system. For example, the display device can include, but is not limited to, smartphones, tablets, personal computers, live streaming all-in-one machines, commercial display terminals, self-service retail terminals, AR / VR (augmented reality / virtual reality) display devices, or embedded display devices, etc. This embodiment does not limit the specific hardware form of the display device.

[0288] The display module can be used to output the generated target digital human video in a visual manner, such as a component or logic unit that performs the function of presenting image or video signals. The display module can be a physical display or a display interface called by an operating system or application. In some embodiments, the display module may include a liquid crystal display, an OLED display, a projection display unit, or a head-mounted display unit; in other embodiments, the display module may also be a virtual display window or a streaming media playback interface for presenting the target digital human video on a live streaming platform, short video platform, or web page.

[0289] The control module can be used to execute digital human video generation methods. Functionally, the control module includes at least a processor and memory, and can be configured to invoke and execute steps such as image processing, feature extraction, pose matching, diffusion generation, and digital human driving. In terms of implementation, the control module can be a local computing unit, an edge computing unit, or a cloud computing node, or a combination of these forms; for example, in some embodiments, the control module can perform inference computation locally, while in other embodiments, the control module can invoke a cloud model via a network to complete the computation. In this embodiment, the control module is configured to execute any of the aforementioned digital human video generation methods.

[0290] The interaction module enables information exchange between the user and the display device. It receives user input and transmits it to the control module as input for generating digital human videos. The interaction module supports various interaction methods, such as touch input, keyboard input, voice input, file upload, or network interface calls. In some embodiments, the interaction module can be a graphical user interface, allowing users to upload images of people and products and input text prompts. In other embodiments, the interaction module can also be an API (Application Programming Interface) for receiving product information and display instructions in batches from external systems.

[0291] See Figure 3 , Figure 3 This is a flowchart illustrating a gesture matching method provided in an embodiment of this application.

[0292] This application also provides a gesture matching method, which includes steps S201 to S202.

[0293] Step S201: Perform feature extraction based on the target person image and the target product image to obtain person feature information and product feature information.

[0294] Step S202: Based on the person's characteristic information and the product's characteristic information, match the display posture of the target product with the hand posture of the target person to obtain hand characteristic information that is compatible with the holding type of the target product.

[0295] In this embodiment, feature extraction is first performed based on the target person image and the target product image to obtain person feature information representing the person's appearance, posture, and local structure, and product feature information representing the product's geometric shape, spatial distribution, and display attributes. Based on this, the person and product feature information are combined to perform joint analysis and matching of the target product's display posture and the target person's hand gestures. By modeling the correspondence between the product's posture parameters in space and the structural features of the person's hand area, hand gesture features that match the target product's grip type are determined. This generates hand feature information adapted to the target product's grip type, providing a stable and consistent gesture constraint foundation for subsequent product display image generation or digital human motion driving.

[0296] This embodiment extracts feature information about the target person and product based on their images, and then models the display posture of the target product against the hand gestures of the target person. This adaptively generates hand feature information that matches different product shapes and grip types without relying on manual annotation or preset fixed posture templates, thus improving the naturalness and rationality of the product's grip posture. By introducing hand feature information adapted to the target product's grip type as a constraint, misalignment, overlap, or ergonomic non-compliance between the product and the hand can be effectively reduced, improving the consistency and stability of the final product display effect. Furthermore, automating the gesture matching process through this method improves the efficiency of product display content generation, meeting the comprehensive requirements of image quality and generation efficiency in product display scenarios.

[0297] In some embodiments, the grip type may include at least one of fingertip pinching, palm-covering grip, thumb grip, lifting support, hanging grip, two-hand holding, two-hand lifting, and directional display.

[0298] In some embodiments, matching the display posture of the target product with the hand posture of the target person based on the person's characteristic information and the product characteristic information to obtain hand characteristic information adapted to the grip type of the target product may include: generating product gripping parameters based on the product characteristic information; wherein the product gripping parameters include at least one of the gripping position, gripping direction, and gripping range of the target product; and matching the display posture with the hand posture based on the product gripping parameters and the person's characteristic information to generate the hand characteristic information.

[0299] In this embodiment, the generation process of the hand feature information is based on the principle of collaborative matching between person feature information and product feature information. Specifically, firstly, the geometric structure, spatial distribution, and operable area of ​​the target product are analyzed based on the product feature information to generate product gripping parameters that describe the gripping method of the target product. These product gripping parameters characterize at least one of the gripping position, gripping direction, and gripping range of the target product. Subsequently, combining the product gripping parameters with person feature information, a matching model is performed between the display posture of the target product and the hand posture of the target person. By analyzing the correspondence between the spatial area indicated by the product gripping parameters and the relevant features of the person's hand, hand posture features adapted to the gripping type of the target product are determined, thereby generating hand feature information that characterizes the target person gripping the target product in a natural manner.

[0300] This embodiment generates product gripping parameters based on product feature information to characterize the gripping position, gripping direction, and gripping range of the target product. It then combines this with human feature information to match and model the display posture of the target product with the hand posture of the target person. This allows for adaptive determination of hand feature information suitable for different product forms and display requirements, thereby improving the rationality and consistency of the product gripping posture. By using product gripping parameters as an intermediate constraint for matching the product with the hand posture, it can improve issues such as misalignment, overlap, or ergonomic non-compliance between the person's hand and the product, enhancing the overall naturalness of product display images or videos. The automated generation of hand feature information through this method reduces manual intervention and improves the efficiency of product display content generation. Addressing the issues of unnatural and unstable matching between product and hand posture during product display, the collaborative modeling based on product gripping parameters and human features results in a natural, stable, and highly adaptable product gripping posture.

[0301] In some embodiments, generating product gripping parameters based on the product feature information may include: performing structural analysis and region division on the target product based on the product feature information to determine at least one candidate gripping region of the target product; wherein the candidate gripping region is determined based on the main body region of the target product, the main body region including at least one of the upper, middle, lower and other structural functional regions of the target product; and generating the product gripping parameters based on the at least one candidate gripping region.

[0302] In this embodiment, the generation of the product gripping parameters is based on the analysis of the structural features of the target product and the principle of region modeling. Specifically, firstly, the target product is structurally analyzed based on its feature information, dividing it into multiple regions with clear spatial locations and structural meanings. The main regions include the upper, middle, and lower parts of the product, or other regions related to its function or structure. Based on this, at least one main region is determined from these regions as a candidate gripping region. This candidate gripping region represents a reasonable gripping position that the target product might adopt when grasped by a person's hand. Subsequently, based on the spatial location, range, and structural features of the candidate gripping regions, product gripping parameters describing the gripping method of the target product are generated, providing structured constraint information for subsequent hand posture matching and display posture modeling.

[0303] This embodiment analyzes and divides the target product into regions based on its feature information, transforming the overall geometric shape of the product into multiple candidate grasping regions with clear structural semantics. This improves upon the problem of inaccurate grasping positions caused by relying solely on overall appearance features. By determining at least one candidate grasping region from the main body of the product and generating product grasping parameters accordingly, a unified grasping parameter expression method can be provided for products of different shapes and structural types, improving the universality and adaptability of product grasping modeling. Using product grasping parameters as the basic constraint for subsequent hand posture matching helps improve the stability and rationality of the spatial relationship modeling between the product and the human hand. Addressing the problem of complex product structures and difficulty in uniformly modeling grasping positions during product display, this embodiment, based on product structure analysis and candidate grasping region modeling, ensures accurate and highly adaptable product grasping parameter generation, facilitating subsequent posture matching.

[0304] In some embodiments, generating the product gripping parameters based on the at least one candidate gripping region may include: performing a gripping adaptability assessment on the at least one candidate gripping region based on at least one of the area information, position information, and thickness information of the candidate gripping region to obtain a gripping adaptability assessment result; determining one of the candidate gripping regions as a target gripping region based on the gripping adaptability assessment result; and determining the product gripping parameters based on the target gripping region.

[0305] In this embodiment, the generation of product gripping parameters is based on the principle of evaluating the gripping adaptability of multiple candidate gripping regions. Specifically, for at least one candidate gripping region obtained from product structure analysis, at least one of the following information—area, position, and thickness—is acquired and analyzed: the area information reflects the coverage of the candidate gripping region within the product body; the position information reflects the spatial positional relationship of the candidate gripping region relative to the overall product structure; and the thickness information reflects the degree of matching between the candidate gripping region and the natural gripping range of the human hand. Based on this, a gripping adaptability evaluation is performed on the candidate gripping regions to obtain the gripping adaptability evaluation results for each candidate gripping region. Subsequently, based on the gripping adaptability evaluation results, a candidate gripping region with optimal gripping adaptability is determined from the at least one candidate gripping region as the target gripping region. Finally, based on the spatial position, range, and structural characteristics of the target gripping region, product gripping parameters characterizing the target product gripping method are determined.

[0306] This embodiment reduces the instability caused by relying solely on a single geometric feature or empirical rule to determine the gripping position by evaluating the gripping adaptability of multiple candidate gripping areas based on area, position, and thickness information, thereby improving the rationality of gripping area selection. By automatically determining the target gripping area based on the evaluation results among multiple candidate gripping areas, it can adapt to products with different structural types and size characteristics, enhancing the universality and robustness of the product gripping parameter generation process. By generating product gripping parameters based on the target gripping area, it can provide stable and ergonomic structural constraints for subsequent human hand posture matching and display posture modeling. Addressing the problem of lack of quantitative basis for gripping area selection and difficulty in accommodating various product structural forms during product display, the gripping adaptability evaluation and optimization based on candidate gripping areas automates, improves accuracy, and enhances adaptability in the product gripping parameter generation process.

[0307] In some embodiments, the product gripping parameters may further include at least one of the size information of the target gripping area and mask information.

[0308] In this embodiment, the product gripping parameters are further extended based on the determined target gripping region. For the already determined target gripping region, the size information and / or mask information corresponding to the target gripping region are further acquired and recorded. The size information can be used to characterize the spatial scale characteristics of the target gripping region in the product image, and the mask information can be used to characterize the pixel-level spatial distribution range of the target gripping region within the product body. By outputting the size information and / or mask information along with parameters such as gripping position, gripping direction, and gripping range as part of the product gripping parameters, the product gripping parameters can not only characterize the geometric center relationship of the gripping position but also accurately describe the constraint characteristics of the gripping region in terms of spatial range and structural boundaries.

[0309] This embodiment introduces the size information and / or mask information of the target grasping area into the product grasping parameters, providing more refined region-level constraints for subsequent hand posture matching and spatial relationship adjustment. This improves problems such as interlacing, offsetting, or unnatural fitting between the hand and the product due to uncertain regions. By explicitly identifying the target grasping area using mask information, the focus on the grasping area during subsequent feature modeling is enhanced, improving the stability and consistency of grasping posture modeling. By incorporating size information and mask information as optional parameters into the product grasping parameter system, adaptability under different accuracy requirements and computational resource conditions can be considered. Addressing the problem of insufficient granularity in describing the product grasping area in related technologies, which makes it difficult to support high-quality hand posture matching, the introduction of target grasping area size information and / or mask information into the product grasping parameters allows for refined modeling of the product grasping area and improves the overall grasping matching accuracy and stability.

[0310] In some embodiments, the step of evaluating the gripping adaptability of the candidate gripping region based on at least one of the area information, position information, and thickness information of the candidate gripping region to obtain a gripping adaptability evaluation result may include: mapping the area information, position information, and thickness information of the candidate gripping region to normalized feature values, and generating a gripping priority score based on the normalized feature values; the gripping priority score serves as the gripping adaptability evaluation result.

[0311] In this embodiment, for at least one identified candidate gripping region, the gripping adaptability is evaluated by quantifying at least one of the area, position, and thickness information corresponding to the candidate gripping region. First, the area, position, and thickness information of the candidate gripping region are mapped to normalized feature values, transforming structural parameters with different dimensions and value ranges into comparable, unified feature representations. Then, based on the normalized feature values, a gripping priority score is generated according to preset evaluation rules or weighting relationships to characterize the gripping adaptability of different candidate gripping regions. This gripping priority score comprehensively reflects the merits of each candidate gripping region in terms of structural rationality, spatial accessibility, and compatibility with hand gripping ability, thus serving as the output of the gripping adaptability evaluation result.

[0312] This embodiment maps the area, location, and thickness information of candidate gripping regions to normalized feature values ​​and generates a gripping priority score based on these normalized feature values. This transforms multiple structural parameters that are difficult to compare directly into a unified quantitative evaluation index, thereby improving the problem of inaccurate gripping region selection caused by relying solely on a single geometric feature or manual rules. By using the gripping priority score as the gripping adaptability evaluation result, automatic sorting and optimization of gripping regions can be achieved when multiple candidate gripping regions exist, improving the stability and consistency of the target gripping region determination process. This method can effectively improve problems such as unnatural hand posture, product occlusion, or structural interference caused by improper gripping region selection. Addressing the lack of a unified quantitative basis for gripping region evaluation and the difficulty in adapting to different product structures in related technologies, this embodiment generates a gripping priority score based on normalized feature values, thereby providing an objective, stable, and scalable gripping adaptability evaluation for candidate gripping regions.

[0313] In some embodiments, generating a gripping priority score based on the normalized feature values ​​may include: generating a corresponding gripping priority score based on the normalized feature values ​​through a multilayer perceptron network or a weighted computation model.

[0314] In this embodiment, the normalized feature values ​​corresponding to the candidate grasping regions are used as input features and processed by a multilayer perceptron network or a weighted computation model. The multilayer perceptron network performs feature fusion and relationship modeling on the normalized feature values ​​through multilayer nonlinear mapping relationships to learn the comprehensive influence of different features on grasping adaptability. Alternatively, the weighted computation model assigns preset weights to the normalized feature values ​​and performs a weighted summation operation to generate a grasping priority score characterizing the grasping adaptability of candidate grasping regions. Thus, the grasping priority score can reflect the relative superiority or inferiority of different candidate grasping regions under the combined influence of multiple structural factors.

[0315] This embodiment uses a multilayer perceptron network or a weighted computation model to process normalized feature values ​​to generate a grasping priority score. This allows for flexible selection of the scoring method under varying implementation complexity and computational resource conditions, thus balancing system real-time performance and evaluation accuracy. By introducing a multilayer perceptron network, its nonlinear modeling capabilities can be utilized to characterize the complex relationships between various features such as area, location, and thickness information, improving the discriminative power and robustness of the grasping priority score. The use of a weighted computation model enables rapid evaluation of the grasping region without the need for complex model training, reducing system implementation costs. Addressing the problem of limited grasping region scoring methods in related technologies, which struggle to simultaneously meet accuracy and efficiency requirements, this embodiment generates a grasping priority score based on normalized feature values ​​combined with a multilayer perceptron network or a weighted computation model, thereby providing a flexible, efficient, and stable evaluation of the grasping suitability of candidate grasping regions.

[0316] In some embodiments, matching the display posture with the hand gesture based on the product gripping parameters and the person's feature information to generate the hand feature information may include: calculating key hand points based on the product gripping parameters to obtain key hand point information; and adjusting the key hand point information based on the person's feature information to determine the hand feature information.

[0317] In this embodiment, firstly, the hand structure of the target person is modeled based on the grasping position, grasping direction, and grasping range represented in the product grasping parameters, and hand keypoint calculation is performed to obtain initial hand keypoint information. This hand keypoint information can be used to describe the basic spatial configuration of the fingers, palm, and their interrelationships. Subsequently, person feature information is introduced to adjust the hand keypoint information. This person feature information represents the target person's body structure, posture, and relative hand position. By constraining and correcting the hand keypoints, the generated hand posture meets the product grasping requirements while maintaining consistency with the overall posture of the target person. Finally, based on the adjusted hand keypoint information, hand feature information for subsequent image generation is determined.

[0318] This embodiment calculates key hand points based on product gripping parameters, directly linking product structural information with the hand posture modeling process. This ensures the generated hand posture naturally conforms to the product's physical gripping characteristics. By incorporating human feature information to adjust the hand key point information, it effectively improves problems such as incoordination and motion distortion caused by generating hand postures solely based on product parameters. Through this method, the generated hand feature information satisfies both the gripping requirements of the target product and the body structure and natural movement habits of the target person. Addressing the issues of disconnect between generated hand postures and overall human posture, and unnatural product gripping in related technologies, this embodiment collaboratively constrains hand postures based on product gripping parameters and human feature information, achieving a technical effect where the generated hand posture when holding a product is natural, stable, and consistent with the overall image of the person.

[0319] In some embodiments, adjusting the hand key point information based on the person's feature information to determine the hand feature information may include: performing pose estimation on the person's feature information to obtain body joint key point information; constraining the hand key point information based on the body joint key point information to obtain target hand key point information that satisfies the constraints; and determining the hand feature information based on the target hand key point information.

[0320] In this embodiment, firstly, pose estimation processing is performed on the person's feature information to obtain key joint information representing the target person's body structure and current posture. This key joint information may include key joints highly correlated with hand movement, such as the shoulder and elbow. Subsequently, based on this key joint information, human structural constraints are applied to the aforementioned hand key point information. By constraining the relative positional, angular, or distance relationships between the hand key points and body joints, the initial hand key point information is adjusted to conform to human kinematics and structural rationality requirements. After constraint adjustment, target hand key point information satisfying the human structural constraints is obtained. Finally, based on the target hand key point information, hand feature information for subsequent image generation or pose modeling is determined, ensuring that the hand pose maintains spatial structural consistency with the overall pose of the target person.

[0321] This embodiment obtains key joint information by estimating the pose of a person's feature information, providing a stable human structural reference coordinate for hand posture adjustment. By applying human structural constraints to hand key point information based on the key joint information, problems such as incoordination with the person's body structure, abnormal joint angles, or spatial distortions during hand posture generation can be effectively improved. By determining hand feature information based on the target hand key point information that meets the constraints, it can be ensured that the generated hand posture meets both the product grasping requirements and conforms to the natural movement patterns of the target person. Addressing the problem in related technologies where hand posture generation lacks human structural constraints and easily leads to inconsistencies between hand and body postures, this embodiment uses human structural constraints based on person pose estimation to adjust hand posture, resulting in a natural, reliable hand posture that is highly consistent with the overall image of the person.

[0322] In some embodiments, constraining the hand keypoint information based on the body joint keypoint information to obtain target hand keypoint information that satisfies the constraints may include: constructing at least one body structure constraint condition based on the body joint keypoint information; detecting whether the hand keypoint information satisfies each body structure constraint condition based on the body joint keypoint information and the hand keypoint information; and correcting the hand keypoint information if it does not satisfy one or more body structure constraint conditions, until the hand keypoint information satisfies all body structure constraint conditions.

[0323] In this embodiment, firstly, at least one body structure constraint is constructed based on the body joint key point information. This constraint characterizes the spatial or kinematic relationships that the target person's hand and various body joints should satisfy in a natural movement state. Subsequently, based on the body joint key point information and hand key point information, the hand key point information is checked one by one to see if it meets each body structure constraint. When it is detected that the hand key point information does not meet one or more body structure constraints, the corresponding hand key point information is corrected, gradually adjusting it towards meeting the constraints. Through this iterative process of detection and correction, the hand key point information simultaneously meets all body structure constraints, thereby obtaining target hand key point information that satisfies the rationality of human body structure, providing a reliable basis for the subsequent determination of hand feature information.

[0324] This embodiment constructs body structure constraints based on key joint information, explicitly incorporating human anatomy and kinematics into the hand posture adjustment process. By detecting whether the hand key point information meets the various body structure constraints, anomalies where hand posture does not match the human body structure can be identified in a timely manner. By iteratively correcting the hand key point information when constraints are not met, hand posture deviations can be gradually reduced, improving issues such as hands intersecting the body, abnormal joint angles, or unnatural movements. Addressing the problem in related technologies where hand posture generation lacks human structure constraints, easily leading to distorted and unrealistic human movements, this embodiment constructs constraints based on key joint information and iteratively corrects hand key points, ensuring that hand posture is highly consistent with the overall human structure and that movements are natural and believable.

[0325] In some embodiments, the body structure constraints can be constructed based on the body structure feature information of the target person, which may include at least one of the following: shoulder position, elbow position, hand position, and body central axis information.

[0326] In this embodiment, by performing pose analysis on the target person's image, body structure feature information representing the overall body structure of the target person can be obtained. This body structure feature information may include one or more of the following: shoulder position, elbow position, hand position, and body central axis information. Based on this, body structure constraints are constructed using the aforementioned body structure feature information to limit the reasonable range of motion, direction of movement, and relative relationship between the hand and other parts of the body in space. By applying these body structure constraints to the constraint and correction process of hand key point information, the hand key points are always positioned in a reasonable manner that conforms to the target person's body structure features, thereby ensuring that the subsequently generated hand pose is coordinated with the overall body structure of the target person.

[0327] This embodiment constructs body structure constraints based on the target person's body structural features, directly incorporating individual differences into the hand posture adjustment process. By structurally defining hand posture using shoulder, elbow, hand, and body axis information, it effectively improves the incoordination issues caused by uniform template processing in hand posture generation for individuals of different body types and postures. Introducing body structure constraints during hand key point correction significantly enhances the spatial consistency and naturalness between hand movements and the overall posture of the person. Addressing the problem in related technologies where hand posture generation does not adequately consider differences in body structure, leading to distorted movements or violations of human movement principles, this embodiment constructs body structure constraints based on the target person's body structural features, ensuring a high degree of matching between hand posture and body structure, resulting in natural and believable movements.

[0328] In some embodiments, the body structure constraints may include part bending constraints and / or body distance constraints.

[0329] In this embodiment, body structure constraints may include part bending constraints and / or body distance constraints. In practical applications, part bending constraints can be constructed based on the spatial relationships between key parts such as the shoulder, elbow, and hand by analyzing the body structure features of the target person. This limits the range of bending angles of the hand relative to the upper limb joints. Simultaneously, body distance constraints are constructed based on the relative distance between the hand position and the central axis or torso region of the person's body. This limits the reasonable range of hand movement around the body. When constraining and correcting key hand points, adjustments are made to key hand points that do not conform to human structure and kinematics, based on the part bending constraints and / or body distance constraints, thereby gradually converging the hand posture to a reasonable state that conforms to the body structure features of the target person.

[0330] This embodiment effectively improves the problem of abnormal elbow or upper limb joint bending angles during hand pose generation by introducing part bending constraints. By introducing body distance constraints, it reduces motion distortion or unnaturalness caused by the hand pose being too close or too far from the central axis of the human body. By synergistically applying these multiple body structure constraints to the hand keypoint correction process, the rationality and stability of the hand pose can be ensured simultaneously from both angular and spatial distance constraints. Addressing the problem in related technologies where hand pose generation lacks human structure constraints and easily results in movements that violate human movement principles, this embodiment, based on part bending constraints and / or body distance constraints, ensures a high degree of coordination between the hand pose and the overall human movement, resulting in a natural and believable display effect.

[0331] In some embodiments, the part bending constraint condition may include an elbow bending constraint condition; wherein the elbow bending constraint condition is used to limit the elbow bending angle of the target person to be within a preset angle range.

[0332] In this embodiment, the bending constraint conditions may include elbow bending constraints. For example, by analyzing the structural features of the target person, the spatial connection relationship between the shoulder, elbow, and hand in the upper limb is determined, and the joint bending angle corresponding to the elbow is calculated based on the spatial connection relationship. During the generation or adjustment of hand key point information, it is detected in real time whether the elbow bending angle is within a preset angle range. When the elbow bending angle is detected to exceed the preset angle range, the hand key point information associated with the elbow is corrected, thereby guiding the hand posture to meet the display requirements of the target product while always conforming to the natural range of motion of the target person's elbow.

[0333] This embodiment effectively reduces abnormal postures that violate human movement principles, such as excessive elbow extension or bending, during hand posture generation by introducing elbow flexion constraints. Real-time constraint and correction of the elbow flexion angle improves the coordination between hand posture and overall upper limb movement, resulting in a more natural, stable, and realistic performance when displaying products. Addressing the issue of unnatural hand posture generation due to a lack of upper limb joint constraints in related technologies, this embodiment, based on elbow flexion angle constraints, ensures that the movements of the person displaying products conform to human anatomy and exhibit a natural and believable visual appearance.

[0334] In some embodiments, the elbow flexion angle can be the angle between a first vector and a second vector, wherein the first vector is constructed based on the shoulder and elbow positions of the target person, and the second vector is constructed based on the elbow and hand positions of the target person.

[0335] In this embodiment, the elbow flexion angle is characterized by the angle between a first vector and a second vector. The first vector is constructed from the shoulder and elbow positions of the target person, and the second vector is constructed from the elbow and hand positions of the target person. This method transforms the three-point spatial relationship of the target person's upper limb into a calculable geometric vector relationship, thereby obtaining the elbow flexion angle reflecting the elbow flexion state. During the generation or adjustment of hand key point information, the angle between the first and second vectors is calculated in real time and compared with a preset angle range to determine whether the current elbow posture conforms to human structural constraints. When the elbow flexion angle does not meet the preset conditions, the hand key point information associated with the elbow is corrected to bring the elbow flexion state back to a reasonable range, thus introducing clear and quantifiable elbow structural constraints during the hand posture generation process.

[0336] This embodiment defines the elbow flexion angle as a geometric quantity based on the angle between the shoulder-elbow vector and the elbow-hand vector. This transforms the human joint state, which originally relied on experience or implicit constraints, into a clear mathematical expression, thereby improving the accuracy and stability of elbow posture judgment and adjustment. By introducing elbow flexion constraints based on vector angles during hand posture generation, the chain reaction of abnormal elbow postures on the overall display action can be effectively reduced, making the upper limb movements of the person holding the product more natural, coherent, and in line with the laws of human movement. Therefore, addressing the problems of unreasonable elbow flexion and motion distortion in the hand posture of the person during product display, an elbow flexion angle constraint mechanism based on key joint vector relationships is adopted, achieving an overall improvement in the structural rationality and visual realism of product display actions.

[0337] In some embodiments, the body distance constraint can be used to limit the spatial distance of the target person's hand position relative to the body's central axis to be within a preset distance range.

[0338] In this embodiment, the body distance constraint condition is used to limit the spatial distance of the target person's hand position relative to the body's central axis to within a preset distance range. The body central axis can be constructed based on the target person's torso key point information, which may include, for example, the midpoint of the shoulder, the midpoint of the hip, or a central axis defined by the line connecting the shoulder and hip. For example, during the generation or adjustment of hand key point information, the vertical or radial distance from the hand key point to the body central axis is calculated, and the calculated distance is compared with the preset distance range. When the spatial distance of the hand position relative to the body central axis exceeds the preset range, the corresponding hand key point information is corrected to bring the hand position back into a reasonable spatial area that satisfies the body distance constraint condition, thereby introducing a spatial constraint on the overall body structure during the hand posture generation process.

[0339] This embodiment introduces a body distance constraint during hand pose generation to limit the spatial distance between the hand position and the body's central axis. This restricts upper limb movements within a spatial area that conforms to the natural range of human movement, reducing motion distortion or visual incoordination caused by excessive hand position deviation from the body's center. By constraining the hand position based on the body's central axis, the spatial consistency between hand pose and overall body pose can be effectively improved without relying on complex dynamic modeling, resulting in more stable and natural movements when displaying products. Therefore, addressing the issue of hand poses drifting and detaching from the overall body structure during product display or digital human generation, the hand spatial distance constraint mechanism based on the body's central axis achieves an overall improvement in the rationality of the character's movement structure and the realism of the display.

[0340] In some embodiments, determining the hand feature information based on the target hand key point information may include: mapping the target hand key point information to two-dimensional hand position information; generating a hand feature image based on the two-dimensional hand position information; and using the hand feature image as the hand feature information.

[0341] In this embodiment, the constrained and corrected key point information of the target hand is mapped to a two-dimensional space to obtain the corresponding two-dimensional hand position information. This two-dimensional hand position information represents the coordinate distribution of each key point in the target image plane. Based on this, a hand feature image is constructed using the two-dimensional hand position information. For example, this can be achieved by marking each key point in two-dimensional space, modeling its thermal distribution, or encoding local regions to form a feature map that reflects the overall contour of the hand, the distribution of fingers, and their relative structural relationships. Finally, the generated hand feature image is used as hand feature information for subsequent human pose modeling, product display constraints, or image generation processes.

[0342] This embodiment maps the key point information of the target hand into two-dimensional hand position information and further generates a hand feature image. This transforms the originally discrete key point data into a spatially continuous feature representation, facilitating its fusion and processing with person and product feature information within a unified image feature space. By using the hand feature image as the hand feature information, the expressive power of the hand region in subsequent generation or rendering processes can be enhanced while maintaining the constraints of the hand structure, making the hand posture more stable in terms of spatial position, morphological distribution, and visual coherence. Thus, addressing the problem that relying solely on key point information is insufficient to fully represent the overall shape of the hand and easily leads to discontinuous posture expression, the mapping method from key points to feature maps achieves an overall improvement in the accuracy and visual consistency of hand posture expression.

[0343] In some embodiments, generating a hand feature image based on the two-dimensional hand position information may include: performing interpolation processing based on the two-dimensional hand position information to generate the hand feature image.

[0344] In this embodiment, the two-dimensional hand position information is used as discrete spatial sampling points to model the distribution of each key hand position in the image plane. Based on the two-dimensional hand position information, interpolation processing is performed on the discrete points at a preset image resolution to transform the spatial relationship between each key hand point into a continuous two-dimensional feature distribution. Through the interpolation processing, a hand feature image with a continuous numerical distribution is generated in two-dimensional space, enabling this hand feature image to characterize the relative positional relationship between key hand points, the coherence of local structures, and the overall spatial morphology.

[0345] This embodiment generates hand feature images through interpolation processing based on two-dimensional hand position information. This transforms the originally sparse and discrete hand position information into a continuous and smooth two-dimensional feature representation, thereby improving the structural breaks or spatial discontinuities caused by relying solely on discrete key points. By constructing continuous feature regions between hand key points through interpolation, the spatial distribution of the hand feature image more closely matches the real hand shape, facilitating subsequent joint modeling and fusion processing with human or product feature information. Therefore, addressing the problem of highly discrete hand position information and its difficulty in directly participating in image-level modeling, this embodiment employs an interpolation-based feature map generation concept, achieving continuity and stability in hand pose expression, and comprehensively improving the naturalness of the digital human hand display and the generated results.

[0346] In some embodiments, the interpolation process may include bilinear interpolation.

[0347] In this embodiment, the interpolation process may include bilinear interpolation. For example, during the generation of a hand feature image, using the discrete pixel coordinates corresponding to the two-dimensional hand position information as reference points, at the target pixel position, based on multiple known pixels adjacent to the target pixel position in the horizontal and vertical directions, the feature values ​​of each known pixel are linearly weighted according to their spatial distance. By performing linear interpolation in both directions and combining the interpolation results, the feature value corresponding to the target pixel position is obtained, thereby forming a continuously changing hand feature distribution in the entire two-dimensional space, completing the construction of the hand feature image.

[0348] This embodiment employs bilinear interpolation to generate features from 2D hand position information. This allows for the smooth mapping of discrete hand position information into a continuous 2D hand feature image without significantly increasing computational complexity, thereby reducing abrupt changes or discontinuities between hand key points caused by discrete sampling. By weighted fusion of adjacent pixel features in the horizontal and vertical directions, the generated hand feature image exhibits a smoother and more stable spatial distribution, which is beneficial for subsequent feature map-based pose constraints, semantic modulation, or diffusion generation. Therefore, addressing the issue of discrete hand key point information and the potential for unnatural structures when directly mapped, this embodiment utilizes a feature continuity concept based on bilinear interpolation to achieve a smooth transition and robust modeling of hand pose feature representation, ultimately improving the naturalness and consistency of the digital human hand display.

[0349] In some embodiments, the hand feature information may include at least one of a hand feature image, a set of hand key points, and an implicit feature representation.

[0350] In this embodiment, the hand feature information may include at least one of hand feature images, hand keypoint sets, and implicit feature representations. For example, in the process of gesture matching and posture generation, hand information is represented using different levels and forms of feature representation according to different processing stages and computational requirements. Specifically, hand feature images are used to describe the structural morphology of the hand in two-dimensional or latent space from a spatial distribution perspective; hand keypoint sets are used to characterize the geometric relationships and topological constraints between the joints of the hand in a structured form; and implicit feature representations are used to encode hand posture, deformation trends, and their potential correlation with product grasping parameters through high-dimensional vector forms. By introducing one or more of the above hand feature information within the same technical framework, multi-level modeling and unified description of hand posture are achieved to support subsequent posture constraints, semantic modulation, or generation processes.

[0351] This embodiment designs hand feature information to include at least one of hand feature images, hand keypoint sets, and implicit feature representations. This allows for flexible selection or combination of hand feature representations based on the input requirements of different algorithm modules, thereby reducing the limitations of single feature forms in terms of accuracy, stability, or expressive power. Introducing hand keypoint sets provides explicit constraints on hand structural relationships, improving the controllability of posture rationality during gesture matching. Introducing hand feature images or implicit feature representations enables continuous modeling of hand spatial distribution characteristics and high-dimensional semantic information, enhancing the naturalness and consistency of the generated results. Therefore, addressing the problem of balancing structural constraints and generation quality when relying solely on a single hand feature representation, this embodiment employs a multi-morphological hand feature collaborative expression concept, achieving a balance between the flexibility of hand posture modeling and the stability of display effects. This comprehensively improves the reliability and realism of gesture matching and generation results in digital human product display scenarios.

[0352] See Figure 4 , Figure 4 This is a flowchart illustrating a multimodal feature matching method provided in an embodiment of this application.

[0353] This application embodiment also provides a multimodal feature matching method, the method including steps S301~S303.

[0354] Step S301: Extract features based on the target person image, the target product image, and semantic information to obtain person feature information, product feature information, and semantic feature information.

[0355] Step S302: Based on the person's characteristic information and the product's characteristic information, match the display posture of the target product with the hand posture of the target person to obtain hand characteristic information that is compatible with the holding type of the target product.

[0356] Step S303: Based on the semantic feature information and the hand feature information, adjust the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information to generate feature matching information for characterizing the target's holding state.

[0357] In this embodiment, firstly, feature extraction is performed based on the target person image, the target product image, and semantic information to obtain person feature information, product feature information, and semantic feature information, respectively, to construct a multimodal feature foundation for representing the person's appearance, product structure, and display intention. Based on this, the display posture of the target product is matched with the hand posture of the target person based on the person and product feature information. By modeling the geometric and semantic relationship between the product's grip structure and the person's hand shape, hand feature information adapted to the target product's grip type is obtained. Further, based on the semantic feature information and hand feature information, the spatial relationships of the spatial regions corresponding to the target person's hand and the target product in the person and product feature information are adjusted so that the relevant features satisfy the constraints of the target grip state in terms of spatial position, orientation, and interrelationship, thereby generating feature matching information to represent the target grip state.

[0358] This embodiment extracts multimodal features based on the target person image, target product image, and semantic information. This enables a comprehensive representation of the person's appearance, product structure, and display semantics within a unified feature space, improving upon the semantic gaps or structural mismatches caused by relying solely on single-modal information. By matching display postures with hand gestures based on person and product feature information, the hand features are adapted to the target product's grip type, enhancing the rationality and naturalness of the product's grasping posture. Furthermore, semantic feature information is introduced to adjust the spatial relationship between the corresponding spatial regions of the person and the product, ensuring that the final generated feature matching information simultaneously satisfies structural and semantic constraints. Therefore, addressing the issues of fragmented multimodal information and inconsistencies between hand gestures and product structure during product display or digital human generation, this embodiment employs a concept based on multimodal feature collaborative modeling and semantic guidance. This achieves unified matching of the person's hand gestures and product display postures at both structural and semantic levels, comprehensively improving the accuracy, stability, and visual consistency of the target gripping state expression.

[0359] In some embodiments, adjusting the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information and the hand feature information to generate feature matching information for characterizing the target's holding state may include: predicting and calculating the spatial offset between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information based on the semantic feature information, so as to obtain offset parameters and sampling weight parameters for splicing the person feature information and the product feature information, as well as the corresponding splicing feature information; The offset parameters include translation parameters, rotation parameters, and scaling parameters. Based on the offset parameters and the sampling weight parameters, the spatial position of the stitched feature information is adjusted to obtain spatially adjusted stitched feature information. Based on the hand feature information, the posture of the spatially adjusted stitched feature information is adjusted to obtain posture-adjusted stitched feature information. The posture-adjusted stitched feature information corresponds to the target holding state. Based on the posture-adjusted stitched feature information, the channel dimension of the semantic feature information is mapped to match the channel dimension of the posture-adjusted stitched feature information to obtain the feature matching information.

[0360] In this embodiment, firstly, based on semantic feature information, the relative relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and product feature information is modeled. By predicting and calculating the spatial offset between the two, offset parameters and sampling weight parameters for splicing the person feature information and product feature information are obtained, and initial splicing feature information is generated accordingly. The offset parameters are used to characterize the translation, rotation, and scaling relationships of the person and product in space. Subsequently, according to the offset parameters and sampling weight parameters, the spatial position of the splicing feature information is adjusted to ensure that the person's hand region and the product region form a relative layout that conforms to the target holding relationship. Based on this, the posture of the spatially adjusted splicing feature information is further adjusted by combining hand feature information, so that the splicing feature information is consistent with the target holding state in both local structure and overall posture. Finally, the semantic feature information is mapped in the channel dimension to a channel dimension that matches the posture-adjusted splicing feature information, thereby achieving the fusion of semantic constraints and spatial structure to obtain feature matching information representing the target holding state.

[0361] This embodiment predicts the spatial offset parameters and sampling weight parameters between a person's hand and a product based on semantic feature information, achieving precise alignment of the person's and product's feature information in spatial position. This improves the posture distortion problem caused by relying solely on fixed rules or static splicing. By introducing hand feature information for posture adjustment after spatial offset adjustment, the spliced ​​feature information is not only reasonable in overall positional relationship but also conforms to human anatomy and actual gripping habits in local holding posture. By mapping and fusing semantic feature information to the channel dimension of spliced ​​feature information, the generated feature matching information is subject to both semantic and spatial structural constraints, thereby enhancing the consistency and stability of the target holding state expression. Thus, addressing the difficulties in accurately modeling the spatial relationship between a person's hand and a product and the ineffective implementation of semantic constraints in the process of multimodal feature fusion, this embodiment adopts a concept centered on semantically guided spatial offset prediction and posture collaborative adjustment. This achieves unified modeling of the target holding state in terms of spatial layout, posture, and semantic expression, comprehensively improving the naturalness, accuracy, and controllability of feature matching results.

[0362] In some embodiments, adjusting the posture of the spatially adjusted splicing feature information based on the hand feature information to obtain posture-adjusted splicing feature information may include: adjusting the posture of the hand region in the spatially adjusted splicing feature information according to the hand feature information to obtain posture-adjusted splicing feature information; performing cross-attention processing on the hand feature information and the product grasping feature information corresponding to the target grasping area in the posture-adjusted splicing feature information to obtain splicing feature information in which the target person's hand and the target product form a target grasping state, and using it as the optimized posture-adjusted splicing feature information.

[0363] In this embodiment, firstly, for the spliced ​​feature information that has already undergone spatial positioning adjustment, the hand region corresponding to the target person's hand is identified and located. Based on the hand feature information, the posture of this hand region is adjusted so that its local structure conforms to the hand shape and movement characteristics of the target person, thus obtaining the spliced ​​feature information after preliminary posture adjustment. Subsequently, a cross-attention mechanism is introduced to interactively model the hand feature information and the product grasping feature information corresponding to the target grasping area in the posture-adjusted spliced ​​feature information. By calculating the attention weight between the two, the matching relationship between the hand posture and the product grasping area is characterized. Based on this, the spliced ​​feature information is further optimized according to the attention weight, so that the target person's hand posture and the target product form a semantically consistent and structurally coordinated target gripping state at the grasping area, ultimately obtaining the optimized posture-adjusted spliced ​​feature information.

[0364] This embodiment adjusts the hand region in the spatially adjusted stitched feature information based on hand feature information to ensure that the human hand maintains consistency with the real hand features of the target person in terms of local structure and overall posture. This improves the problem of stiff or distorted hand posture caused by relying solely on spatial offset adjustment. By performing cross-attention processing on the hand feature information and the corresponding product grasping feature information of the target grasping area, the hand posture adjustment process can perceive the structural features and semantic information of the product grasping area, thereby establishing a fine correspondence between the hand and the product. By optimizing the stitched feature information based on the cross-attention results, the final target holding state is consistent in spatial position, hand posture, and grasping semantics. Thus, addressing the problem of difficulty in accurately coordinating the human hand posture and the product grasping area, and the resulting unnatural posture or grasping mismatch, this embodiment adopts a concept of joint optimization using hand feature-driven posture adjustment combined with a cross-attention mechanism. This achieves a high consistency match between the human hand and the product grasping state, significantly improving the naturalness, stability, and credibility of the target grasping state expression.

[0365] See Figure 5 , Figure 5 This is a schematic flowchart of an image generation method provided in an embodiment of this application.

[0366] This application also provides an image generation method, which includes steps S401 to S402.

[0367] Step S401: Based on the target person image, the target product image, and the prompt information, generate display posture constraint information to constrain the display posture of the target product; wherein, the prompt information is used to indicate the product display method, and the display posture constraint information is used to characterize the constraint on the relative relationship between the target person image and the target product image at least according to the prompt information.

[0368] Step S402: Under the constraint of the display posture constraint information, perform image diffusion generation to obtain a target display image representing the target person displaying the target product; wherein, the target display image is used to characterize the target person displaying the target product according to the display posture constraint information.

[0369] In this embodiment, firstly, based on the target person image, the target product image, and prompts indicating the product display method, the display relationship between the person and the product is modeled. By integrating the person's appearance features, the product's structural features, and the display semantics expressed by the prompts, display posture constraint information is generated to constrain the target product's display posture. This display posture constraint information clarifies the relative relationships such as the target person's posture, hand position, and product orientation when displaying the target product. Subsequently, during the diffusion generation process, this display posture constraint information is introduced as a conditional constraint to guide the potential representations at each time step of the diffusion generation, causing the image generation process to evolve gradually in the direction that satisfies the display posture constraint information. Through this method, while ensuring the stability of the diffusion generation, the final generated image conforms to the expected constraints in terms of the person's posture and the product display method, thereby obtaining a target display image representing the target person displaying the target product.

[0370] This embodiment generates display posture constraint information based on the target person image, the target product image, and prompt information. This ensures that the image generation process has clear display semantics and posture constraints from the initial stage, thereby improving the randomness and instability of the relative relationship between people and products in related diffusion generation processes. By performing image diffusion generation under the constraints of the display posture constraint information, the diffusion process is continuously guided by posture and semantics at each time step, thereby improving the consistency and rationality of the posture when people display products. Thus, addressing the problems of difficult control of product display posture and unstable relationship between people and products in related image generation methods, this embodiment adopts the concept of first constructing display posture constraint information and then performing diffusion generation under this constraint. This achieves controllable generation of the display relationship between the target person and the target product, significantly improving the naturalness of the target display image's posture, semantic consistency, and application usability.

[0371] In some embodiments, generating display posture constraint information for constraining the display posture of the target product based on the target person image, the target product image, and the prompt information may include: performing feature extraction based on the target person image, the target product image, and the prompt information to obtain person feature information, product feature information, and semantic feature information; and generating display posture constraint information for constraining the display posture based on the person feature information, the product feature information, and the semantic feature information.

[0372] In this embodiment, feature extraction is performed on the target person image, the target product image, and the prompt information to form person feature information representing the person's appearance and posture attributes, product feature information representing the product's structure and shape attributes, and semantic feature information representing the semantic constraints of the product's display method. Based on this, the above multi-source features are jointly modeled and fused to establish a correspondence between the person feature information, product feature information, and semantic feature information within a unified feature space, thereby generating display posture constraint information to constrain the target product's display posture. This display posture constraint information comprehensively represents the relative positional relationship, posture relationship, and semantic constraints between the target person and the target product during the display process, providing a controllable posture guidance basis for the subsequent image generation process.

[0373] This embodiment extracts features based on the target person image, target product image, and prompt information, obtaining person feature information, product feature information, and semantic feature information respectively. This allows for unified modeling of information from different modalities, thus avoiding the problem that a single image feature cannot accurately depict the product display intent. Furthermore, by generating display posture constraint information based on person feature information, product feature information, and semantic feature information, the display posture between the target person and the target product is explicitly constrained before image generation, thereby improving the stability of the subsequent generation results in terms of posture consistency and semantic conformity. Therefore, addressing the problem of accurately controlling the display relationship between people and products during the generation of related product display images, the concept of using multimodal feature joint modeling to generate display posture constraint information achieves controllable generation of product display posture and semantically consistent expression, significantly improving the display rationality and application adaptability of the target display image.

[0374] In some embodiments, the step of generating a diffusion image under the constraint of the display posture constraint information to obtain a target display image representing a target person displaying the target product may include: aligning the display posture constraint information with preset time step information; and generating a diffusion image based on the aligned display posture constraint information to obtain the target display image.

[0375] In this embodiment, by aligning the generated display posture constraint information with preset time step information, the display posture constraint information can be matched one-to-one with the generation stage corresponding to each time step in the diffusion generation process. This provides staged posture and semantic constraints to the generation process at different stages of diffusion generation. Based on this, in each time step, the current generation state is guided by the aligned display posture constraint information, ensuring that the diffusion generation process is always controlled by the target product display posture requirements during the gradual denoising process, ultimately generating a target display image representing the target person displaying the target product.

[0376] This embodiment aligns the display posture constraint information with preset time step information, enabling the display posture constraint to participate in the diffusion generation process in stages. This addresses the problem that constraint information only takes effect in the initial stage and gradually becomes ineffective in subsequent generation processes. By performing diffusion generation under the constraints of the aligned display posture constraint information, the relative posture relationship between the target person and the target product remains consistent throughout the generation process, guiding the image to gradually converge towards the target state that meets the display requirements. Therefore, addressing the issues of unstable and uncontrollable product display posture and easy posture drift in related diffusion generation methods, this embodiment adopts the concept of aligning display posture constraint information with the time step and participating in diffusion generation. While ensuring the quality of diffusion generation, it constrains the product display posture throughout the entire process, significantly improving the overall performance of the target display image in terms of posture accuracy and generation stability.

[0377] In some embodiments, the time step information may include the number of diffusion steps and noise scheduling parameters corresponding to at least some of the diffusion steps, the noise scheduling parameters being configured to inject noise from high to low order based on the diffusion steps.

[0378] In this embodiment, the diffusion generation process is discretized into several diffusion steps, and corresponding noise scheduling parameters are configured for at least some of the diffusion steps, enabling the diffusion generation to have a controllable noise evolution mechanism in the time dimension. For example, by incorporating the number of diffusion steps as part of the time step information, the iteration length of the diffusion generation process is clearly defined. Simultaneously, noise scheduling parameters are configured for different diffusion steps, causing the noise injection intensity to gradually decrease in descending order of diffusion step intensity. This injects more noise into the latent code in the initial diffusion step to enhance the global structure exploration capability, while gradually reducing noise injection in subsequent diffusion steps to strengthen detail recovery and pose convergence, allowing the diffusion generation process to gradually complete the generation of the target image in a coarse-to-fine direction.

[0379] This embodiment improves upon the instability caused by fixed or uncontrollable noise injection methods by setting the time step information to include the number of diffusion steps and corresponding noise scheduling parameters, thus enabling clear phase division of the diffusion generation process in the time dimension. By injecting noise in descending order of diffusion steps, the diffusion generation achieves stronger global exploration capabilities in the early stages and higher detail restoration accuracy in the later stages, balancing generation efficiency and quality. Therefore, addressing the issues of coarse noise scheduling and difficulty in balancing global structure and local details in related diffusion generation processes, this embodiment employs a phased noise scheduling concept based on time step information, resulting in a stable and controllable diffusion generation process with a reasonable structure and clear details in the generated results.

[0380] In some embodiments, the diffusion generation based on aligned display pose constraint information to obtain the target display image may include: performing attention encoding processing based on aligned display pose constraint information to obtain global feature information; performing diffusion generation based on latent code representation based on the global feature information to obtain target latent code; and decoding the target latent code to obtain the target display image.

[0381] In this embodiment, attentional encoding is first performed on the display pose constraint information, which has already been aligned with the time step information. This fuses the person's features, product features, and semantic constraints within a unified attention modeling framework, thereby obtaining global feature information that can represent global structural relationships and pose constraint relationships. Subsequently, using this global feature information as conditional constraints, the diffusion generation process iteratively updates the noisy latent code in the latent code space, obtaining a target latent code that conforms to the display pose constraint information. Finally, by decoding the target latent code, the latent code representation in the latent space is restored to the image space, thereby generating a target display image representing the target person displaying the target product.

[0382] This embodiment introduces attention encoding based on display posture constraints before diffusion generation, ensuring that character features, product features, and posture constraints are uniformly incorporated into the global feature modeling process from the initial generation stage. This addresses the issue of delayed or ineffective constraint information during diffusion generation. By performing diffusion generation based on global feature information in the latent code space, the generation process can progressively refine local display details while maintaining global structural consistency, improving the stability and rationality of the relative relationship between the target product and the target character. Decoding the target latent code after diffusion generation ensures that the generated result can be accurately mapped to a high-quality display image. Therefore, addressing the problem of difficulty in effectively integrating display posture constraints into the diffusion process and the tendency for posture deviations or structural inconsistencies in the generated results, this embodiment combines attention encoding based on posture constraints with latent code diffusion generation, thereby improving the structural consistency, posture rationality, and visual quality of the target display image while ensuring generation efficiency.

[0383] In some embodiments, the attention encoding process based on the aligned display pose constraint information to obtain global feature information may include: performing image feature extraction and text feature extraction based on the aligned display pose constraint information to obtain image feature information and text feature information; performing dual-path cross-attention processing based on the image feature information and the text feature information to obtain interactive image feature information and interactive text feature information; performing position encoding embedding processing based on the interactive image feature information to obtain image feature information containing position information; and performing multi-head self-attention processing based on the image feature information containing position information under the semantic constraints of the interactive text feature information to obtain the global feature information.

[0384] In this embodiment, firstly, image feature extraction and text feature extraction are performed on the display pose constraint information that has been aligned to time steps, respectively, to obtain image feature information representing the visual structure of the person and the product, and text feature information representing the display pose and semantic constraints. Subsequently, a dual-path cross-attention structure is constructed based on the image and text feature information, enabling the two types of features to perceive and constrain each other in the feature dimension, resulting in interactive image feature information and interactive text feature information. On this basis, positional encoding embedding processing is introduced into the interactive image feature information to explicitly represent the relative spatial position information of the person and the product. Finally, under the semantic constraints provided by the interactive text feature information, multi-head self-attention processing is performed on the image feature information containing positional information, thereby modeling the structural relationship between the person, the product, and the display pose globally, obtaining global feature information for subsequent diffusion generation.

[0385] This embodiment extracts features from both image and text information within the posture constraint information and introduces a dual-path cross-attention mechanism to achieve bidirectional interaction between image and text features, enabling semantic constraints to directly influence image structure modeling at the feature level. By embedding positional encoding into the interactive image features, the global feature information simultaneously possesses the ability to express spatial positional information and structural relationships, thereby improving the stability of the relative layout of people and goods. By performing multi-head self-attention processing under the semantic constraints of the interactive text features, the generation process can comprehensively consider the consistency of local details and the overall display posture globally. Thus, addressing the problem of difficulty in co-modeling semantic constraints and spatial structure during the fusion of related multimodal features, leading to unstable posture or semantic deviation in the generated results, this embodiment adopts a concept based on a combination of dual-path cross-attention and position-aware self-attention to achieve refined modeling of display posture constraint information, thereby effectively improving the structural consistency, semantic accuracy, and overall generation quality of the target display image during subsequent diffusion generation.

[0386] In some embodiments, the dual-path cross-attention processing based on the image feature information and the text feature information to obtain interactive image feature information and interactive text feature information may include: S1: using the image feature information as the query object and the text feature information as the query object, adjusting the weight of the text feature information to obtain weight-adjusted text feature information; S2: using the text feature information as the query object and the image feature information as the query object, adjusting the weight of the image feature information to obtain weight-adjusted image feature information; S3: based on the weight-adjusted image feature information and text feature information, re-executing S1 and S2 until a preset first end-of-iteration condition is met; using the image feature information at the end of the iteration as the interactive image feature information, and using the text feature information at the end of the iteration as the interactive text feature information.

[0387] In this embodiment, cross-attention calculation is performed using image feature information as the query object and text feature information as the query object. The semantic dimensions related to image content in the text feature information are weighted and adjusted to obtain weighted text feature information. Similarly, cross-attention calculation is performed using text feature information as the query object and image feature information as the query object. The spatial regions in the image feature information related to text semantics are weighted and adjusted to obtain weighted image feature information. Based on this, the aforementioned cross-attention processing is repeatedly performed using the weighted image feature information and text feature information in a preset manner, gradually aligning the image and text features through multiple rounds of interaction. When a preset first termination iteration condition is met, the interaction processing stops, and the final obtained image feature information is used as the interactive image feature information, and the final obtained text feature information is used as the interactive text feature information for subsequent global feature modeling.

[0388] This embodiment achieves bidirectional constraint and information complementarity between image and text features by weighting text semantics using image features as the query and text features as the query. Furthermore, it weights image spatial regions using text features as the query and image features as the query, thus realizing bidirectional constraint and information complementarity between image and text features. By repeatedly executing dual-path feature interaction in multiple rounds of cross-attention processing, semantic information and spatial structure gradually converge and form a stable correspondence, thereby reducing semantic bias or spatial mismatch caused by single feature fusion. Therefore, addressing the problem of unidirectional constraint between image and text information and the difficulty in forming a consistent interactive expression during multimodal feature fusion, this embodiment adopts a concept based on iterative interaction of dual-path cross-attention, achieving collaborative alignment of image and text features at the semantic and spatial levels. This improves semantic consistency, structural stability, and overall generation effect in subsequent display pose modeling and image generation processes.

[0389] In some embodiments, the multi-head self-attention processing based on the image feature information containing location information to obtain the global feature information may include: R1: splitting the current image feature information into multiple attention heads and assigning weights to the multiple attention heads to obtain first output information; R2: performing a residual concatenation between the first output information and the current image feature information to obtain second output information; R3: performing feedforward processing based on the second output information to obtain third output information; R4: performing a residual concatenation between the third output information and the second output information and performing layer normalization processing based on the residual concatenation result to obtain fourth output information; R5: using the fourth output information as the updated current image feature information, re-executing R1 until a preset second end-of-iteration condition is met; and using the current image feature information at the end of the iteration as the global feature information.

[0390] In this embodiment, firstly, the current image feature information is split into multiple attention heads, and weights are assigned to each attention head to obtain first output information reflecting the relationships between different subspaces. Then, the first output information is residually concatenated with the current image feature information to obtain second output information, which introduces contextual information while maintaining the stability of the original features. Based on this, feedforward processing is performed on the second output information to obtain third output information, used for nonlinear enhancement of feature representation. The third output information is residually concatenated with the second output information, and layer normalization is performed on the residual concatenation result to obtain fourth output information, thereby suppressing feature distribution shift and improving training and inference stability. Finally, the fourth output information is used as the updated current image feature information, and the above multi-head self-attention processing process is repeated according to a preset method until a preset second iteration termination condition is met. The current image feature information after the iteration ends is determined as the global feature information.

[0391] This embodiment achieves parallel modeling of different spatial locations and semantic subspace relationships within image features by splitting image feature information containing location information into multiple attention heads and performing self-attention modeling on each head. By introducing residual connections and layer normalization mechanisms in each processing round, it maintains the stable representation of the original image features while effectively suppressing feature degradation and gradient vanishing problems. Iterative updates of the multi-head self-attention processing enable image features to gradually aggregate global contextual information and form a unified global representation. Therefore, addressing the issues of strong local dependencies and insufficient global semantic associations in related image feature modeling, this embodiment employs a multi-head self-attention iterative modeling concept based on location information, effectively characterizing the global structure and long-range dependencies of the image. This provides more stable, complete, and semantically consistent global feature information for subsequent diffusion generation and display pose control.

[0392] In some embodiments, the step of performing diffusion generation based on latent code representation based on the global feature information to obtain a target latent code may include: performing gradient correction prediction based on the global feature information to obtain gradient correction information; and performing latent code update based on the gradient correction information to obtain the target latent code.

[0393] In this embodiment, firstly, using the global feature information as input, gradient correction prediction is performed on the latent code state at the current diffusion time step to generate gradient correction information characterizing the latent code's evolution toward the target distribution at the current time step. This gradient correction information indicates the direction and magnitude of correction for the current latent code while satisfying global feature constraints. Subsequently, based on the gradient correction information, a latent code update operation is performed, causing the latent code to gradually evolve along a preset diffusion path in the latent space. By repeatedly executing the gradient correction prediction and latent code update process across multiple diffusion time steps, the latent code gradually transitions from a high-noise state to a low-noise state matching the global feature information, thereby obtaining the target latent code characterizing the target display image.

[0394] This embodiment utilizes gradient correction prediction based on global feature information to ensure that the latent code update process is jointly guided by display pose, semantic constraints, and spatial relationship information at each diffusion time step, thereby reducing the deviation of the latent code from the target display structure during diffusion. By using gradient correction information to gradually update the latent code, fine control over the latent code evolution path is achieved, enabling the diffusion generation process to converge to the target latent code that satisfies the conditional constraints within a shorter number of time steps. Therefore, addressing the issues of low convergence efficiency, weak conditional constraints, and unstable generation results in related diffusion generation processes, this embodiment adopts a concept that uses global feature information as a condition and gradient correction prediction to drive latent code evolution. This significantly improves diffusion generation efficiency and stability while maintaining generation accuracy, thus providing a reliable latent code foundation for high-quality, real-time target display image generation.

[0395] In some embodiments, the step of performing gradient correction prediction based on the global feature information to obtain gradient correction information may include: mapping the global feature information of the current time step to the latent code space according to a preset latent code channel dimension to obtain latent code feature information; spatially reconstructing the latent code feature information and performing local fusion based on spatial location on the spatially reconstructed latent code feature information to obtain the conditional latent code of the current time step; and performing gradient correction prediction based on the conditional latent code and the current latent code to determine the gradient correction information of the current time step.

[0396] In this embodiment, firstly, based on a preset latent code channel dimension, the global feature information corresponding to the current time step is mapped to the latent code space, thereby obtaining latent code feature information that matches the latent code representation of the diffusion generation process. Subsequently, the latent code feature information undergoes spatial reconstruction processing, restoring it from a sequence form to a latent code representation with spatial topological relationships. Furthermore, based on the spatial location of the latent code, the spatially reconstructed latent code feature information undergoes local fusion processing to obtain a conditional latent code representing the ideal denoising state of the current time step. On this basis, the conditional latent code and the current latent code at the current diffusion time step are used together as input, and a gradient correction prediction calculation is performed through a gradient correction prediction model to determine gradient correction information used to indicate the direction and magnitude of the latent code update at the current time step.

[0397] This embodiment maps global feature information to the latent code space and constructs a conditional latent code corresponding to the current time step, enabling the gradient correction prediction process to explicitly characterize the ideal evolution state of the latent code under the current conditional constraints. Through local fusion processing based on spatial location, the problem of discontinuous or conflicting features in different regions of the latent code space is effectively alleviated, thereby improving the stability and accuracy of gradient correction prediction. By combining the conditional latent code with the current latent code for gradient correction prediction, the latent code update direction is guided by global feature constraints at each diffusion time step. Therefore, addressing the technical problems of conditional information's ineffective participation in latent code updates and unstable latent code correction paths during diffusion generation, the concept of constructing a conditional latent code with global feature information and driving gradient correction prediction achieves precise control over the latent code evolution path, improving the convergence efficiency and generation quality of the diffusion generation process.

[0398] In some embodiments, updating the latent code based on the gradient correction information to obtain the target latent code may include: updating the corresponding current latent code according to the gradient correction information of each current time step and a preset time step interval, so as to gradually obtain the target latent code.

[0399] In this embodiment, at each current time step of diffusion generation, an update operation is performed on the current latent code based on the gradient correction information obtained from gradient correction prediction and a preset time step interval. For example, by using the gradient correction information as an indicator of the correction direction and magnitude of the latent code at the current time step, and combining it with the time step interval to incrementally update the current latent code, the latent code gradually changes along a predetermined evolution path between adjacent time steps. By iteratively executing the above latent code update process across multiple time steps, the target latent code representing the target person displaying the target product is finally obtained.

[0400] This embodiment introduces a latent code update mechanism based on gradient correction information in each diffusion time step, giving the latent code update process a clear direction and controllability, thus improving the generation instability problem caused by the accumulation of random perturbations during latent code evolution. By adjusting the latent code update amplitude in conjunction with a preset time step interval, the latent code update process ensures both convergence stability and generation efficiency. Therefore, addressing the problems of difficult-to-control latent code update path and long generation time in the correlated diffusion generation process, the concept of gradually updating the latent code based on gradient correction information is adopted, thereby stably obtaining high-quality target latent codes within a few diffusion time steps.

[0401] In some embodiments, the attention encoding process based on the aligned display pose constraint information to obtain global feature information may include: performing attention encoding processing using an attention encoding model based on the aligned display pose constraint information to obtain global feature information.

[0402] In some embodiments, the step of performing diffusion generation based on latent code representation based on the global feature information to obtain the target latent code may include: performing diffusion generation based on latent code representation using a first diffusion model based on the global feature information to obtain the target latent code.

[0403] In some embodiments, the training process of the attention encoding model and the first diffusion model may include: training a second diffusion model using a first dataset to obtain a trained second diffusion model; inputting a second dataset into the first diffusion model and the trained second diffusion model respectively to obtain a first generation result and a second generation result; using a discriminant model to determine the similarity between the first generation result and the second generation result to obtain a similarity discrimination result; calculating a loss based on the similarity discrimination result, and training the attention encoding model and the first diffusion model based on the loss calculation result.

[0404] In this embodiment, firstly, the aligned display posture constraint information is input into an attention encoding model. The attention encoding model performs attention encoding on the display posture constraint information, giving higher attention weights to features highly correlated with the target person's hands, the target product, and the display posture constraints. This forms global feature information in the feature space that has global consistency with the overall display semantics. Based on this global feature information, a diffusion generation process based on latent code representation is performed in the latent code space using a first diffusion model. Noise is gradually removed through multi-timestep latent code evolution, ultimately obtaining the target latent code used to represent the target person displaying the target product.

[0405] During the model training phase, the second diffusion model is trained using the first dataset to obtain a trained second diffusion model. Then, the second dataset is input into both the first and trained second diffusion models to obtain a first generated result and a second generated result. These two generated results are then input into a discriminative model to determine their similarity, yielding a similarity judgment result. Finally, loss is calculated based on the similarity judgment result, and the attention encoding model and the first diffusion model are jointly trained according to the loss calculation result. This allows the first diffusion model to gradually approach the generation effect of the second diffusion model while maintaining low computational complexity.

[0406] This embodiment combines display posture constraint information with an attention encoding model, explicitly strengthening features closely related to the product display posture during the generation process at the encoding stage. This improves the problem of posture constraint information being overwhelmed or weakened by noise during diffusion generation. By performing diffusion generation based on latent code representation under the guidance of global feature information, the evolution of latent codes is always globally guided by display posture constraint information, thereby improving the consistency and stability of the target display image in terms of the relative posture of the person and product, hand gripping relationship, etc.

[0407] Simultaneously, by introducing a training mechanism that combines the first and second diffusion models, and utilizing a discriminative model to constrain the similarity of their generated results, the first diffusion model receives effective guidance from the high-quality diffusion model during training. This allows it to achieve near-high-quality diffusion model generation results with less computational resources during the inference phase. Therefore, addressing the issues of unstable pose constraints and the difficulty in balancing generation efficiency and quality in product display image generation in related technologies, a concept of "attention encoding constraint + latent code diffusion generation + discriminative distillation training" is adopted. This significantly improves image generation efficiency and overall generation quality while ensuring the accuracy of the display pose.

[0408] In some embodiments, the loss calculation based on the similarity discrimination result may include: calculating the loss using a preset target loss function based on the similarity discrimination result. The target loss function includes an adversarial loss function and a distillation loss function, wherein the adversarial loss function is constructed based on the similarity discrimination result, and the distillation loss function is constructed based on the first generated result and the second generated result.

[0409] In this embodiment, firstly, based on the similarity discrimination result output by the discriminant model, an error measurement of the generation process is performed using a preset target loss function. The target loss function includes an adversarial loss function and a distillation loss function. Specifically, the adversarial loss function is constructed based on the similarity discrimination result and is used to measure the degree of difference between the first generated result and the target generated distribution as determined by the discriminant model, driving the first generated result to optimize in a direction that is more difficult for the discriminant model to distinguish during training. The distillation loss function is constructed based on the first generated result and the second generated result and is used to constrain the first generated result to approximate the second generated result in the feature space or output space. By jointly calculating the adversarial loss function and the distillation loss function, a total loss value for backpropagation is obtained, and the parameters of the attention encoding model and the first diffusion model are updated based on the total loss value.

[0410] This embodiment incorporates similarity discrimination results into the loss calculation process and simultaneously introduces adversarial loss and distillation loss functions into the same objective loss function. This allows the model training to utilize the discriminative model's global judgment capability on generation quality while fully inheriting the structural and detailed information contained in the generation results of high-quality diffusion models. This overcomes the limitation of a single loss function in simultaneously ensuring generation realism and stability. Therefore, addressing the issues of insufficient generation quality, unstable convergence, or excessive reliance on high-quality models in lightweight generation models during training, this embodiment adopts the concept of "adversarial constraints based on similarity discrimination + consistency distillation based on generation results." This significantly improves the quality of generation results and training stability while controlling model complexity and computational overhead.

[0411] In some embodiments, decoding the target latent code to obtain the target display image may include: decoding the target latent code to obtain a first display image; and performing a fusion process on the first display image to obtain the target display image.

[0412] In this embodiment, firstly, the target latent code obtained through diffusion generation is input into the decoding model to decode the target latent code, restoring the latent code representation in the latent space to an image representation in the visible space, thereby obtaining a first display image. Subsequently, a fusion process is performed on the first display image to optimize areas that may arise during the decoding process, such as boundary discontinuities, inconsistent local details, or unnatural transitions between people and products. This ensures that the decoded image maintains consistency in spatial structure, texture details, and overall visual effect, ultimately obtaining a target display image representing a target person displaying a target product.

[0413] This embodiment decodes the target latent code generated by diffusion and then introduces fusion processing for the first displayed image. This further coordinates and optimizes the result of mapping from the latent space to the image space in the connection area, edge area, and detail area between the person and the product, effectively improving the problems of image fragmentation, artifacts, or detail distortion caused by relying solely on latent code decoding. Therefore, addressing the issue of visual inconsistency and insufficient overall visual experience in latent code decoding results in scenarios combining people and products, this embodiment adopts the concept of "decoding and restoring the image first, then performing image fusion optimization," thereby significantly improving the overall consistency and visual quality of the target displayed image while ensuring diffusion generation efficiency.

[0414] In some embodiments, the fusion process can be a pixel-level fusion process.

[0415] In this embodiment, after decoding the target latent code and obtaining the first display image, pixel information corresponding to different sources, different generation stages, or different semantic regions in the first display image is analyzed and fused pixel by pixel. By comprehensively processing color values, brightness information, texture details, or boundary transitions at the pixel level, the person area and the product area achieve a continuous transition at the pixel level, thereby generating a target display image with overall visual consistency.

[0416] This embodiment employs pixel-level fusion processing to finely integrate the decoded first display image at the pixel level, resulting in smoother boundaries and more natural transitions between the person and product areas. This effectively reduces splicing artifacts and visual discontinuities caused by decoding errors or regional differences. Therefore, addressing the issue of discontinuous edges and inconsistent textures that often occur when combining people and products in the image generation process, this pixel-level fusion concept improves the overall consistency, visual realism, and display quality of the target image.

[0417] In some embodiments, the pixel-level blending process may include at least one of edge blending, lighting blending, and shadow blending.

[0418] In this embodiment, the pixel-level fusion processing may include at least one of edge fusion, illumination fusion, and shadow fusion. After obtaining the first display image, edge fusion processing is performed on the pixels at the boundary between the person area and the product area to smooth out pixel abrupt changes between different areas. Simultaneously, illumination fusion is used to perform pixel-level consistency adjustments to the brightness, color temperature, or illumination direction of different areas. Shadow fusion is used to perform pixel-level reconstruction and overlay of the shadow information corresponding to the person's hand, the product surface, and their contact areas, so that the fused image achieves harmony and unity in terms of edge continuity, illumination consistency, and spatial hierarchy.

[0419] This embodiment introduces at least one pixel-level blending method among edge blending, lighting blending, and shadow blending to finely coordinate the adjustment of the person and product areas at the pixel level. This effectively reduces the boundary and splicing artifacts between different generated areas, enhancing the consistency of the person and product in terms of lighting and spatial occlusion relationships. Therefore, addressing the issues of harsh edges, inconsistent lighting, or unrealistic shadows that often occur when people and products are combined in related display images, this embodiment employs a multi-dimensional pixel-level blending concept to improve the overall realism, visual naturalness, and product display effect of the target display image.

[0420] In some embodiments, the prompt information may include text prompt information and / or audio prompt information.

[0421] In this embodiment, the prompting information may include text prompting information and / or audio prompting information. By acquiring text prompting information describing the product display method, display intention, or display context, and / or acquiring audio prompting information containing tone, rhythm, or emotional tendency, the prompting information is input as a high-level semantic constraint into the feature extraction and pose constraint generation process. This provides clear semantic guidance for the relative relationship between the target person and the target product when generating display pose constraint information, enabling subsequent pose matching and image generation processes to be performed under unified semantic conditions.

[0422] This embodiment introduces text and / or audio prompts as a high-level semantic constraint source for the display generation process. This allows for the simultaneous consideration of display semantics, style, and interactive intent when generating display posture constraints, making the posture and manner in which the person displays the product more consistent with the intended expression. Therefore, addressing the issues of limited semantic instruction expression and difficulty in flexibly controlling display methods during the generation of product display images, this embodiment adopts a concept based on multimodal prompt information collaborative constraints. This improves the controllability of display posture generation, semantic consistency, and the adaptability of the target display image to different display needs.

[0423] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods.

[0424] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the methods.

[0425] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0426] See Figure 6 , Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0427] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods.

[0428] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0429] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0430] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0431] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0432] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described.

[0433] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0434] In implementation, each step of the method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can reside in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 110, and the processor 120 reads information from memory 110 and, in conjunction with its hardware, completes the steps of the method. To avoid repetition, detailed descriptions are omitted here.

[0435] In some implementations, in addition to the hardware components described above, computer devices may also include software modules, such as operating systems, Basic Input Output System (BIOS), and application software.

[0436] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0437] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0438] Application software, also known as an application program, can be understood as software written for a specific user application purpose, and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0439] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0440] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0441] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0442] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “described,” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0443] Those skilled in the art will recognize that the examples and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0444] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the various embodiments described herein can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0445] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of parts is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple parts or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection of apparatuses or parts, and may be electrical, mechanical, or other forms.

[0446] The parts described as separate components may or may not be physically separate. The parts shown as partial components may or may not be physical parts; that is, they may be located in one place or distributed across multiple network parts. Some or all of the parts can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0447] In addition, the functional parts in the various embodiments of this application can be integrated into one processing part, or each part can exist physically separately, or two or more parts can be integrated into one part.

[0448] If a function is implemented as a software component and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0449] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gesture matching method, characterized in that, The method includes: Feature extraction is performed based on the target person image and the target product image to obtain the person feature information and the product feature information; Based on the person's characteristic information and the product's characteristic information, the display posture of the target product is matched with the hand posture of the target person to obtain hand characteristic information that is compatible with the holding type of the target product. The step of matching the display posture of the target product with the hand posture of the target person based on the person's feature information and the product feature information to obtain hand feature information that is compatible with the grip type of the target product includes: generating product grip parameters based on the product feature information; wherein the product grip parameters include at least one of the grip position, grip direction and grip range of the target product; and matching the display posture with the hand posture based on the product grip parameters and the person's feature information to generate the hand feature information. The step of generating product gripping parameters based on the product feature information includes: performing structural analysis and region division on the target product based on the product feature information to determine at least one candidate gripping region of the target product; wherein the candidate gripping region is determined based on the main body region of the target product, and the main body region includes at least one of the upper, middle, lower and other structural functional regions of the target product; and generating the product gripping parameters based on the at least one candidate gripping region. The step of generating the product gripping parameters based on the at least one candidate gripping region includes: for the at least one candidate gripping region, performing a gripping adaptability evaluation on the candidate gripping region based on at least one of the area information, position information, and thickness information of the candidate gripping region to obtain a gripping adaptability evaluation result; determining one of the candidate gripping regions as a target gripping region based on the gripping adaptability evaluation result; and determining the product gripping parameters based on the target gripping region. The product gripping parameters also include at least one of the size information and mask information of the target gripping area; The step of evaluating the gripping adaptability of the candidate gripping region based on at least one of the area information, position information, and thickness information of the candidate gripping region to obtain a gripping adaptability evaluation result includes: mapping the area information, position information, and thickness information of the candidate gripping region to normalized feature values, and generating a gripping priority score based on the normalized feature values; the gripping priority score serves as the gripping adaptability evaluation result. The step of generating a grasping priority score based on the normalized feature values ​​includes: generating a corresponding grasping priority score based on the normalized feature values ​​through a multilayer perceptron network or a weighted computation model.

2. The gesture matching method according to claim 1, characterized in that, The grip types include at least one of the following: fingertip pinching, palm-covering grip, thumb grip, lifting support, hanging grip, two-hand holding, two-hand lifting, and directional display.

3. The gesture matching method according to claim 1, characterized in that, The step of matching the display posture with the hand gesture based on the product gripping parameters and the person's feature information to generate the hand feature information includes: Based on the product gripping parameters, key hand points are calculated to obtain key hand point information; The hand key point information is adjusted based on the person's characteristic information to determine the hand characteristic information.

4. The gesture matching method according to claim 3, characterized in that, The step of adjusting the key hand points based on the person's feature information to determine the hand feature information includes: Pose estimation is performed on the aforementioned character feature information to obtain key body joint information; The hand key point information is constrained based on the body joint key point information to obtain target hand key point information that satisfies the constraints. Based on the key point information of the target hand, the hand feature information is determined.

5. The gesture matching method according to claim 4, characterized in that, The step of constraining the hand key point information based on the body joint key point information to obtain target hand key point information that satisfies the constraints includes: Based on the aforementioned key point information of the body joints, at least one body structural constraint condition is constructed. Based on the key point information of the body joints and the key point information of the hand, it is detected whether the key point information of the hand satisfies the constraints of each body structure. If the hand key point information does not meet one or more body structure constraints, the hand key point information is corrected until all hand key point information meets all body structure constraints.

6. The gesture matching method according to claim 5, characterized in that, The body structure constraints are constructed based on the body structure feature information of the target person, which includes at least one of the following: shoulder position, elbow position, hand position, and body central axis information.

7. The gesture matching method according to claim 5, characterized in that, The body structure constraints include part bending constraints and / or body distance constraints.

8. The gesture matching method according to claim 7, characterized in that, The bending constraint conditions include elbow bending constraints; wherein, the elbow bending constraint conditions are used to limit the elbow bending angle of the target person to be within a preset angle range.

9. The gesture matching method according to claim 8, characterized in that, The elbow flexion angle is the angle between a first vector and a second vector. The first vector is constructed based on the target person's shoulder and elbow positions, and the second vector is constructed based on the target person's elbow and hand positions.

10. The gesture matching method according to claim 8, characterized in that, The body distance constraint condition is used to limit the spatial distance between the target person's hand position and the body's central axis to a preset distance range.

11. The gesture matching method according to claim 4, characterized in that, The step of determining the hand feature information based on the target hand key point information includes: The key point information of the target hand is mapped into two-dimensional hand position information; Based on the two-dimensional hand position information, a hand feature image is generated; the hand feature image serves as the hand feature information.

12. The gesture matching method according to claim 11, characterized in that, The step of generating a hand feature image based on the two-dimensional hand position information includes: Based on the two-dimensional hand position information, interpolation processing is performed to generate the hand feature image.

13. The gesture matching method according to claim 12, characterized in that, The interpolation process includes bilinear interpolation.

14. The gesture matching method according to any one of claims 1 to 13, characterized in that, The hand feature information includes at least one of hand feature image, hand key point set, and implicit feature representation.

15. A multimodal feature matching method, characterized in that, Including the gesture matching method according to any one of claims 1 to 14, the multimodal feature matching method includes: Feature extraction is performed based on the target person image, target product image, and semantic information to obtain person feature information, product feature information, and semantic feature information; Based on the person's characteristic information and the product's characteristic information, the display posture of the target product is matched with the hand posture of the target person to obtain hand characteristic information that is compatible with the holding type of the target product. Based on the semantic feature information and the hand feature information, the spatial relationship between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information is adjusted to generate feature matching information to characterize the target's holding state.

16. The multimodal feature matching method according to claim 15, characterized in that, The step of adjusting the spatial relationships of the spatial regions corresponding to the target person's hand and the target product in the person feature information and product feature information based on the semantic feature information and the hand feature information to generate feature matching information for characterizing the target's holding state includes: Based on the semantic feature information, the spatial offset between the spatial regions corresponding to the target person's hand and the target product in the person feature information and the product feature information is predicted and calculated to obtain the offset parameters and sampling weight parameters used to splice the person feature information and the product feature information, as well as the corresponding splicing feature information; the offset parameters include translation parameters, rotation parameters and scaling parameters; Based on the offset parameter and the sampling weight parameter, the spatial position of the splicing feature information is adjusted to obtain the spatially adjusted splicing feature information; Based on the hand feature information, the spatially adjusted splicing feature information is subjected to posture adjustment to obtain posture-adjusted splicing feature information; the posture-adjusted splicing feature information corresponds to the target gripping state; Based on the stitched feature information after the pose adjustment, the channel dimension of the semantic feature information is mapped to match the channel dimension of the stitched feature information after the pose adjustment, so as to obtain the feature matching information.

17. The multimodal feature matching method according to claim 16, characterized in that, The step of adjusting the posture of the spatially adjusted splicing feature information based on the hand feature information to obtain posture-adjusted splicing feature information includes: The hand region in the spatially adjusted splicing feature information is then adjusted in posture according to the hand feature information to obtain the posture-adjusted splicing feature information. Cross-attention processing is performed on the hand feature information and the product gripping feature information corresponding to the target gripping area in the stitched feature information after posture adjustment, so as to obtain the stitched feature information in which the target person's hand and the target product form the target gripping state, and use it as the optimized stitched feature information after posture adjustment.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

Images