Image driving method and apparatus

Abstract

The application provides an image driving method and device, the image driving method comprises: obtaining a first image containing a first commodity and a preset character and a second image containing a second commodity, performing image segmentation on the first image to obtain a first mask corresponding to the first commodity and the preset character, performing image segmentation on the first image based on a commodity prompt, a hand key point of the preset character and the first mask to obtain a second mask corresponding to the first commodity, performing image fusion based on the first mask, the second mask, the first image and the second image to obtain a third image, performing driving processing on the third image according to a driving instruction to obtain a driving image, and the preset character in the driving image presents a driving action corresponding to the driving instruction, thereby improving the quality of image fusion, improving the image generation efficiency and enhancing the application scene flexibility.

Description

[0001] This application claims application number 202610199608.4, filed on February 11, 2026, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to an image driving method and device. Background Technology

[0003] With the popularization of online sales, it has become the main sales method for sellers to provide sales videos of the products to be sold to the audience on different platforms through pre-recorded or live broadcasts. However, when sellers need to sell different products, they need to shoot images of themselves holding different products as material. With the wide variety of product categories, shooting each product one by one will undoubtedly affect the user experience. With technological advancements, the rise of digital humans, or virtual human technology, has made the automatic generation of human videos feasible. Digital human technology involves training a model using user-provided image and audio data to generate a digital human that matches the user's image. This digital human can move its lip, head, and other body parts in response to audio or other commands. Therefore, in the context of product sales, how to combine virtual human technology to alleviate the pressure of shooting product footage and improve sales efficiency has become a pressing issue. Summary of the Invention

[0004] This application provides an image-driven method and device that can improve the quality of image fusion, increase image generation efficiency, and enhance the flexibility of application scenarios.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: In a first aspect, embodiments of this application provide an image-driven method, the method comprising: acquiring a first image and a second image; the first image comprising a first product and a preset person, and the second image comprising a second product; performing image segmentation on the first image to obtain a first mask corresponding to the first product and the preset person; performing image segmentation on the first image based on product prompts, key points of the preset person's hands, and the first mask to obtain a second mask corresponding to the first product; performing image fusion based on the first mask, the second mask, the first image, and the second image to obtain a third image; performing driving processing on the third image according to a driving instruction to obtain a driving image; and having the preset person in the driving image perform a driving action corresponding to the driving instruction.

[0006] Based on this solution, cross-image fusion of the first product with a preset character and the second product is achieved. A precise image mask is obtained through secondary image segmentation, solving the problem of single image content limitations. By combining hand key point constraints and driving instructions, the fused character can present a specified action, improving the image's dynamism and scene adaptability. The entire process is automated and requires no manual intervention, improving image generation efficiency and adapting to the needs of multiple scenarios such as e-commerce display and content creation.

[0007] In another possible implementation, the above-mentioned image segmentation of the first image to obtain the first mask corresponding to the first product and the preset person includes: performing image segmentation on the first image based on the first product and the preset person using a first image segmentation model to obtain the first mask corresponding to the hands of the first product and the preset person; the first mask contains at least a partially overlapping area between the hands of the first product and the preset person.

[0008] Based on this solution, the product and the person's hand are segmented to avoid interference from irrelevant areas and improve the targeting and accuracy of the first mask; the segmentation priority of the person's hand area is clarified to lay the foundation for the natural integration of the product and the hand in the future and reduce the sense of incongruity in the integration.

[0009] In another possible implementation, the above-mentioned image segmentation of the first image based on product prompts, preset hand key points of the person, and a first mask to obtain a second mask includes: performing key point recognition on the first image using a hand recognition model to obtain hand key points; performing image segmentation on the first image using a second image segmentation model based on product prompts, hand key points, and the first mask to obtain a second mask corresponding to the first product; the second mask includes an image mask obtained by completing the partially overlapping area according to the product prompts.

[0010] Based on this solution, by introducing constraints of hand key points and product prompts, problems such as blurred product edges and adhesion to the hand that may be caused by a single segmentation model are solved, thereby improving the segmentation accuracy of the second mask. By using dedicated models to realize key point recognition and secondary segmentation respectively, the hierarchical sense of segmentation logic is optimized, and the robustness of segmentation results is improved.

[0011] In another possible implementation, the above-mentioned method of segmenting the first image using a second image segmentation model based on the product prompt, the hand key points, and the first mask to obtain a second mask corresponding to the first product includes: performing feature transformation on the product prompt and the first image respectively using the second image segmentation model to obtain prompt features and image features; calculating the feature similarity between the image features and the prompt features based on the hand key points; determining the position information of the first product in the first mask according to the feature similarity; and completing the overlapping area according to the prompt features and the position information to obtain the second mask.

[0012] Based on this solution, through dual guidance of product prompts and key hand points, combined with feature similarity matching, the product area is located and the overlapping parts of the mask are filled in, thereby achieving accurate segmentation of the first product and efficient generation of the second mask.

[0013] In another possible implementation, the above-mentioned image fusion based on the first mask, the second mask, the first image, and the second image to obtain the third image includes: performing mask fusion on the first mask and the second mask to obtain the third mask; performing shape adjustment on the second image based on the third mask and hand key points to obtain a deformed image; and performing image fusion based on the deformed image, the first image, and the third mask to obtain the third image.

[0014] Based on this scheme, a unified third mask is generated through mask fusion to achieve precise positioning of segmented regions and provide a unified constraint standard for subsequent fusion. The shape of the second product is adjusted first, and then the image is fused to ensure the adaptability of the second product with the person's hand posture and the scene of the first image, so as to avoid the product's position, shape and scene being out of place after fusion. The mask fusion, shape adjustment and image fusion are completed step by step, and the fusion effect is optimized layer by layer to improve the naturalness and harmony of the third image.

[0015] In another possible implementation, the above-mentioned morphological adjustment of the second image based on the third mask and hand key points to obtain a deformed image includes: extracting control points from the third mask based on the hand key points to obtain a first control point, and extracting control points from the second image to obtain a second control point; determining control point adjustment parameters based on the second image, the first control point, and the second control point, and performing morphological adjustment on the second image according to the control point adjustment parameters to obtain a deformed image.

[0016] Based on this solution, control points are extracted using key hand points as a benchmark, which strongly correlates the shape adjustment of the second product with the hand movements of the person, ensuring precise connection between the product and the hand (such as matching the angle and position of the hand holding the product); the shape adjustment range is quantified by adjusting the parameters of the control points to avoid over- or under-adjustment, thereby improving the accuracy and stability of the deformed image; based on the adjustment logic of the control points, the shape of the second product is made to fit the actual physical scene, enhancing the realism of the fused image.

[0017] In another possible implementation, the above-mentioned determination of control point adjustment parameters based on the second image, the first control point, and the second control point, and the morphological adjustment of the second image according to the control point adjustment parameters to obtain a deformed image, includes: calculating the first adjustment parameter based on the distance between the first control point and the second control point, and moving the second control point in the second image based on the first adjustment parameter to obtain an initial deformed image; calculating the second adjustment parameter and the second adjustment weight based on the distance between the second control point in the initial deformed image and other control points; and moving the second control point in the initial deformed image based on the second adjustment parameter and the second adjustment weight to obtain the deformed image.

[0018] Based on this scheme, a two-stage control point adjustment strategy is adopted. First, the initial alignment between the product and the hand is achieved through the first adjustment parameter. Then, the local details are optimized and the deformation accuracy is improved through the second adjustment parameter and weight. The adjustment parameters are calculated by combining the distance relationship between control points, so that the deformation logic conforms to the geometric space law and avoids distortion and deformation after the product is deformed. The introduction of adjustment weight can flexibly adapt to the morphological characteristics of different products and enhance the adaptability of the method to diverse products.

[0019] In another possible implementation, the above-mentioned extraction of control points from the third mask based on the hand key points to obtain the first control point includes: extracting contour points from the third mask based on the hand key points through the contour processing layer of the contour processing model to obtain contour points; performing convex hull calculation on the contour points through the convex hull processing layer of the contour processing model to obtain convex hull boundary points; and performing boundary optimization on the convex hull boundary points through the boundary optimization layer of the contour processing model to obtain the first control point.

[0020] In another possible implementation, the above-mentioned convex hull calculation of the contour points through the convex hull processing layer of the contour processing model to obtain the convex hull boundary points includes: sorting the contour points based on the coordinate values ​​of the contour point sequence to obtain a contour point sequence; determining the current contour point and the previous two contour points in the contour point sequence according to a preset traversal order; constructing a first point vector and a second point vector based on the current contour point and the previous two contour points, and calculating the product of the first point vector and the second point vector; if the product value is positive, taking the current contour point and the previous two contour points as the first convex hull boundary points, and continuing traversal until the entire contour point sequence has been traversed to obtain the convex hull boundary points; if the first product value is negative, deleting the migrated contour points of the current contour point, and continuing traversal until the entire contour point sequence has been traversed to obtain the convex hull boundary points.

[0021] In another possible implementation, the above-mentioned extraction of control points from the second image to obtain the second control points includes: determining a target feature detection model based on the commodity form of the second commodity corresponding to the second image, and extracting features from the second image through the target feature detection model to obtain the second control points.

[0022] In another possible implementation, the above-mentioned image fusion based on the deformed image, the first image, and the third mask to obtain the third image includes: fusing the deformed image and the first image at their edges according to the third mask to obtain a first fused image; fusing the first fused image and the background area of ​​the person in the first image excluding the first product to obtain a second fused image; and fusing the second fused image and the background area of ​​the first image excluding the first product and the preset person to obtain the third image.

[0023] Based on this solution, edge blending and background blending are performed in layers. First, the connection problem of the core area (product + person's hand) is solved, and then the background transition is gradually optimized to improve the overall coordination of the blended image. Focusing on edge blending, the solution avoids problems such as jagged edges and blurring at the edges of products, people, and backgrounds, thereby enhancing the visual coherence of the image. The solution preserves the original background features of the first image, making the scene of the blended third image more natural and reducing the traces of artificial synthesis.

[0024] In another possible implementation, the above-mentioned image edge fusion of the deformed image and the first image according to the third mask to obtain a first fused image includes: determining each region to be fused contained in the third mask; the region to be fused includes an occluded edge region, an occluded inner region, a pressed edge region, a pressed inner region, and / or a partially overlapping region; and performing image edge fusion of the deformed image and the first image according to the fusion strategy corresponding to each region to be fused to obtain the first fused image.

[0025] Based on this solution, a region-based matching exclusive fusion strategy is used to achieve precise fusion in different regions, avoiding poor results caused by using a uniform fusion method for different regions (such as product edges and the edges of people's hands); the fusion priority and parameters of each region are clearly defined to improve the targeting of edge fusion and further optimize the naturalness of the connection between products, people, and scenes; the complexity of the fusion logic is simplified, and the fusion strategy can be flexibly switched through region marking to enhance the scalability of the method.

[0026] In another possible implementation, the above-described method of performing image edge fusion on the deformed image and the first image according to the fusion strategy corresponding to each region to be fused to obtain the first fused image includes: performing edge protection processing on the occluded edge region in the region to be fused, performing transparency processing on the pressed edge region in the region to be fused, performing pixel retention processing on the occluded inner region and the pressed inner region in the region to be fused, and performing pixel replacement processing on some overlapping regions in the region to be fused based on the deformed image to obtain the first fused image.

[0027] Based on this scheme, by performing differentiated processing on different sub-regions of the image to be fused (edge ​​protection of occluded edge areas, transparency of pressed edge areas, pixel retention of internal areas, and pixel replacement of overlapping areas), targeted edge fusion of the deformed image and the first image is achieved, ensuring the hierarchical differentiation and overall visual harmony of the fused area.

[0028] In another possible implementation, the above-mentioned edge protection processing of the occluded edge region in the region to be fused includes: spreading the region edge of the occluded edge region to obtain an edge buffer; calculating edge distance field information based on the distance between each pixel in the edge buffer and the region edge; and performing Gaussian blur processing on the pixels in the edge buffer based on the edge distance field information to obtain the edge-protected occluded edge region.

[0029] Based on this scheme, an edge diffusion buffer is generated in the occluded edge area, the edge distance field information is calculated by combining the distance from the pixel to the edge of the area, and Gaussian blur processing is performed to achieve a smooth transition of the occluded edge area and eliminate the abruptness of the edge when the deformed image is fused with the first image.

[0030] In another possible implementation, the above-mentioned process of making the pressed edge area in the region to be merged transparent includes: spreading the regional edge of the pressed edge area to obtain a pressed buffer; calculating the pressed distance field information based on the distance between the key points of the fingertip in the pressed buffer; calculating the transparency weight based on the pressed distance field information and a preset transparency calculation model; and performing the transparency processing based on the transparency weight to obtain the transparent pressed edge area.

[0031] Based on this solution, a pressing buffer is generated by edge diffusion of the pressing edge area. The pressing distance field information is calculated based on the distance between the fingertip key points. Then, the transparency weight is configured and the transparency processing is completed by combining the transparency calculation model, so as to achieve natural transparency fusion of the pressing edge area and simulate the visual effect of real pressing interaction.

[0032] In another possible implementation, the method further includes determining the hand state of the preset character in the first mask; dividing the hand of the preset character into regions according to the hand state to obtain each hand region, and taking each hand region and the partially overlapping region as the region to be merged.

[0033] Based on this solution, by dividing the hand region according to the hand's state, the contact and non-contact areas between the hand and the product are accurately identified, providing more detailed regional constraints for subsequent fusion. The product completion area is determined, which can fill in the product edges and details that may be missing during the segmentation process, improving the integrity of the product image. By assigning exclusive fusion strategies to each region through region marking, a clear basis is provided for subsequent edge fusion and background fusion, further improving the detail quality of the fused image.

[0034] In another possible implementation, determining the hand state of the preset figure in the first mask includes: detecting the distance between the hand of the preset figure in the first mask and the edge of the product; if the edge distance is greater than or equal to a preset threshold, determining that the hand is in an occluded state; and / or, if the edge distance is less than the preset threshold, determining that the hand is in a pressing state. Based on this solution, by detecting the distance between the hand and the edge of the product, and combining it with a preset threshold, it is possible to quickly and quantitatively distinguish between the occlusion and pressing states of the hands of a preset person. The judgment logic is simple, efficient and easy to implement in engineering.

[0035] In another possible implementation, determining the hand state of the preset character in the first mask includes: detecting whether the hand of the preset character in the first mask has epidermal diffusion characteristics; if so, determining that the hand is in an occluded state; if not, determining that the hand is in a pressing state; and / or, detecting whether the hand of the preset character in the first mask has squeezing characteristics; if so, determining that the hand is in a pressing state; if not, determining that the hand is in an occluded state.

[0036] Based on this solution, by recognizing visual morphological features such as skin diffusion and squeezing characteristics of the hand, it can accurately identify the occlusion and pressing states of the preset person's hand, adapting to the need to distinguish the state of the hand under different interactive postures with the product.

[0037] In another possible implementation, determining the hand state of the preset character in the first mask includes: calculating the pressure confidence of the hand of the preset character in the first mask; determining that the hand is in a covered state when the pressure confidence is less than a first pressure confidence; and determining that the hand is in a pressed state when the pressure confidence is greater than a second pressure confidence.

[0038] Based on this solution, by calculating the pressure confidence of the hand and setting a dual confidence threshold for interval division, a refined and gradient determination of the occlusion and pressing states of the preset character's hand is achieved, thereby improving the robustness and accuracy of state differentiation.

[0039] Secondly, embodiments of this application provide an image driving device, comprising: an image acquisition module for acquiring a first image and a second image; the first image includes a first product and a preset person, and the second image includes a second product; a first image segmentation module for segmenting the first image to obtain a first mask corresponding to the first product and the preset person; a second image segmentation module for segmenting the first image based on product prompts, key hand points of the preset person, and the first mask to obtain a second mask corresponding to the first product; an image fusion module for performing image fusion based on the first mask, the second mask, the first image, and the second image to obtain a third image; and a driving module for driving the third image according to a driving instruction to obtain a driving image; wherein the preset person in the driving image presents a driving action corresponding to the driving instruction.

[0040] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program for executing the image-driving method provided in the first aspect above.

[0041] Fourthly, embodiments of this application also provide a computing device, including: one or more processors; and a memory configured to store one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the image driving method as described in any of the first aspects above.

[0042] Fifthly, embodiments of this application provide a computer program product that, when instructions in the computer program product are executed by a processor, performs the image driving method provided in the first aspect above. Attached Figure Description

[0043] Figure 1 This is a flowchart of an image-driven method provided in an embodiment of this application.

[0044] Figure 2 This is a schematic diagram of a first image provided for an embodiment of this application.

[0045] Figure 3 This is a schematic diagram of an image replacement provided in an embodiment of this application.

[0046] Figure 4 This is a flowchart illustrating an image segmentation method provided in an embodiment of this application.

[0047] Figure 5 This is a schematic diagram of an image fusion process provided in an embodiment of this application.

[0048] Figure 6 This is a schematic diagram of an image morphology adjustment process provided in an embodiment of this application.

[0049] Figure 7 This is a schematic diagram of a control point movement process provided in an embodiment of this application.

[0050] Figure 8 This is a schematic diagram of an image segmentation process provided in an embodiment of this application.

[0051] Figure 9 This is a schematic diagram of an image driving device provided in an embodiment of this application.

[0052] Figure 10 This is a schematic diagram of an image driving device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. To facilitate a clear description of the technical solutions of the embodiments of this application, the use of terms such as "first," "second," etc., in the embodiments of this application is merely for illustration and to distinguish the objects being described. There is no particular order between them, nor does it indicate a specific limitation on the number of devices in the embodiments of this application, and they cannot constitute any limitation on the embodiments of this application.

[0054] To address the issues of unnatural integration between products and people, low segmentation accuracy, and poor scene adaptability in image fusion, this application provides an image-driven method. This method acquires a first image containing a first product and a preset person, and a second image containing a second product. The first image is segmented to obtain a first mask corresponding to the first product and the preset person. Based on product cues, key hand points of the preset person, and the first mask, the first image is segmented again to obtain a second mask corresponding to the first product. Image fusion is then performed based on the first mask, the second mask, the first image, and the second image to obtain a third image. The third image is then processed according to a driving instruction to obtain a driving image. In this driving image, the preset person performs the driving action corresponding to the driving instruction. By obtaining a precise image mask through two image segmentations and combining hand key point constraints, a natural integration of products and people is achieved. Finally, a dynamic motion image is generated using the driving instruction. This method solves the problems of unnatural integration between products and people, low segmentation accuracy, and poor scene adaptability in image fusion, achieving the technical effects of improving the quality of fused images, increasing generation efficiency, and enhancing scene flexibility.

[0055] Figure 1 This is a flowchart illustrating an image-driven method provided in an embodiment of this application. Figure 1 As shown, the method includes steps 101 to 105.

[0056] Step 101: Obtain the first image and the second image.

[0057] In some examples, the first image includes a first product and a pre-defined person, where the pre-defined person may be holding the first product. For example, as shown... Figure 2 As shown in the first image illustration, a pre-set person is holding a mobile phone to sell goods. The first image can be a photo uploaded by the user, or any frame extracted from a sales video of the first product; this embodiment does not limit this. The pre-set person can be the seller, a third party selling the goods, or a virtual character.

[0058] The first product can be a replacement product. By replacing the first product in the first image with the second product, image utilization efficiency can be improved. When selling the second product, there is no need to repeatedly invest in shooting resources, thus reducing the cost for the seller. For example, Figure 3As shown in the image replacement diagram, the second product can be a beverage. By replacing the mobile phone held by the preset person in the first image with a beverage, the sale of the product can be completed while reducing shooting costs.

[0059] The second image includes an image of a second product used for display or sale. For example, an initial second image can be obtained by pre-photographing the second product against a preset background (e.g., a green screen background or other natural background), and the background of the initial second image can be removed to obtain a second image of the second product.

[0060] It should be noted that all preset characters (including but not limited to real person portraits, virtual character images, digital human images, etc.) involved in this embodiment and the accompanying drawings have been authorized or permitted by the user.

[0061] Step 102: Perform image segmentation on the first image to obtain the first mask corresponding to the first product and the preset character.

[0062] In some examples, based on a first product and a preset person, the first image is segmented using a first image segmentation model to obtain a first mask corresponding to the hands of the first product and the preset person; the first mask contains at least a partially overlapping area between the hands of the first product and the preset person.

[0063] In this embodiment, a first image segmentation model (e.g., U) can be used. 2 Image segmentation is performed using a first image segmentation model (-Net). The first mask output by the first image segmentation model is a pixel-level classification result mapping of the image, meaning each pixel uniquely corresponds to a label (such as background, product, hand, etc.), essentially assigning a classification label to each pixel in the image. For example, the first mask can be a binary mask, which only distinguishes between the target and the background. Pixels of the target are labeled as 1 (or 255, white), and pixels of the background are labeled as 0 (black). For instance, the target could be the first product and a preset person in the first image. To segment the first product and the preset person in the image, the white areas in the mask represent the product and the preset person, and the black areas represent the background.

[0064] For example, the first mask may include the hand of a first product and a preset character. The hand is a whole region including the outline of the hand and the internal area of ​​the hand. Because U 2 -Net performs saliency detection based on pre-training, which cannot effectively distinguish between hands and arms. Therefore, it uses the hands of the first product and the preset character as the first mask.

[0065] Step 103: Based on the product prompts, the key points of the person's hand, and the first mask, perform image segmentation on the first image to obtain the second mask corresponding to the first product.

[0066] Figure 4 This is a flowchart illustrating an image segmentation method provided in an embodiment of this application. Figure 4 As shown, step 102 includes steps 401 to 402.

[0067] Step 401: Use a hand recognition model to identify key points in the first image to obtain hand key points.

[0068] The key points of a person's hand refer to the two-dimensional (2D) or three-dimensional (3D) coordinate points of the bones, joints, and key parts of the hand obtained by extracting key points from the first image using a model. These coordinate points completely cover the core motion nodes of the palm and fingers, and are the basic data for reconstructing hand posture and judging gestures. For example, the key points of the hand can be obtained by performing key point recognition on the first image using a gesture recognition model (such as MediaPipe Hands).

[0069] For example, key points of the hand may include: the base of the palm (wrist) numbered 0, which includes the center of the wrist joint; the thumbs numbered 1-4, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and thumb tip; the index fingers numbered 5-8, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and index finger tip; the middle fingers numbered 9-12, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and middle finger tip; the ring fingers numbered 13-16, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and ring finger tip; and the little fingers numbered 17-20, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and little finger tip.

[0070] These hand key points can reconstruct hand postures. For example, by changing the coordinates of 21 points, it is possible to capture hand movements such as bending, extending, and rotating (e.g., clenching a fist, extending fingers, and wrist rotation) in real time, accurately reconstructing the hand's posture in space. In addition, it can support gesture recognition. For example, based on the relative position, distance, and angle relationship of hand key points, specific gestures can be defined (e.g., "OK gesture" corresponds to the thumb and index finger tips being close together and other fingers being extended; "thumbs up" corresponds to the thumb being extended and other fingers being bent). Alternatively, it can enable human-computer interaction: the coordinates of key points can be directly used as control signals and applied to gesture control scenarios (e.g., controlling screen scrolling by sliding fingers across key points, switching functions by clenching fists / extending fingers through key points). Furthermore, it can provide 3D information: some models (e.g., the 3D mode of MediaPipe Hands) can output the depth coordinates (Z-axis) of key points, which can distinguish the overlapping of hands and their distance, adapting to more complex spatial interaction scenarios (e.g., virtual button clicks, 3D modeling gesture control).

[0071] Step 402: Based on the product prompts, hand key points, and the first mask, the first image is segmented using the second image segmentation model to obtain the second mask corresponding to the first product.

[0072] In this embodiment, the definition of the second mask is the same as described above, and will not be repeated here. The second mask includes removing pixels of the hand and other pixels that do not belong to the first product (e.g., other items that may accidentally appear in the frame when the digital human is holding the product for display) based on the first mask. The second mask includes an image mask obtained by completing the partially overlapping areas according to the product prompts.

[0073] For example, a product prompt refers to descriptive text about the first product, or it can be a reference image of the first product. The product prompt is used to determine the similarity of product regions. Specifically, the product prompt is encoded into a conditional vector using the FiLM mechanism to guide the image segmentation process. After the product prompt is input into the image segmentation model, the model can perform more accurate image segmentation of the first product in the first image based on the product prompt, obtaining the second mask corresponding to the first product.

[0074] For example, if the first item is a red coffee cup, the corresponding product description could be "red coffee cup" or a similar wording, or an image of the first item could be used as a reference image. Generally speaking, for items with simple shapes and patterns, text descriptions are preferred, while for items with complex shapes and patterns, reference images are preferred.

[0075] It should be noted that in this embodiment, the user can choose to input product prompts separately during the product replacement process, or the text description of the first product can be used as part of the image tag during the initial upload of the original display image, or a separate image of the first product can be uploaded as a reference image. In this way, the system can automatically obtain product prompts for the first product when the user selects the original display image. Similarly, during the subsequent generation of the target display image, the description of the second product can be used as part of the image tag, or the target product image of the second product can be directly used as a reference image. In this way, when replacing products based on the target display image (for example, replacing the second product with a third product), the system can automatically obtain product prompts for the second product.

[0076] In some examples, the product prompt and the first image are respectively transformed by the second image segmentation model to obtain prompt features and image features; based on the hand key points, the feature similarity between the image features and the prompt features is calculated, the position information of the first product in the first mask is determined according to the feature similarity, and the overlapping area is filled in according to the prompt features and position information to obtain the second mask.

[0077] For example, the above is via U 2 -Net performs image segmentation on the first image to obtain a first mask corresponding to the hands of the first product and the preset person. On the one hand, because the range covered by the first mask is too large, it may lead to problems such as excessive computation and long response time. On the other hand, U 2 -Net image segmentation can accurately identify the edge positions of the hand and the product, but it cannot handle the part of the product that is occluded by the hand. Therefore, based on the first mask, a second image segmentation model is used to perform secondary image segmentation to obtain a second mask that corresponds only to the first product.

[0078] For example, the second image segmentation model can be a cross-modal cue segmentation model (CLIP Segmentation, CLIPSeg), which segments the image by inputting the first image, product cue, hand key points and the first mask into CLIPSeg to obtain the second mask corresponding to the first product.

[0079] The second image segmentation model locates the image within the area corresponding to the first mask. Based on the product prompts, it removes the hand and other pixels that do not belong to the first product (such as other items that may accidentally appear in the frame when the digital human is holding the product for display). At the same time, the second image segmentation model can complete the parts of the hand that are covered by the product prompts. Therefore, the cooperation between the first image segmentation model and the second image segmentation model can completely separate the first product from the preset person and accurately preserve the third mask of the preset person's hand, especially the edge shape.

[0080] If the second image segmentation model is not used to perform secondary segmentation on the first product (i.e., segmentation only on the visible part of the first product), geometric discontinuity problems will occur between the first and second products during the subsequent image fusion process, such as missing or floating parts, misaligned patterns, and jagged edges. Therefore, the secondary segmentation operation of the second image segmentation model is necessary.

[0081] It should be noted that the execution order of steps 401 and 102 can be changed according to actual needs. That is, step 401 can be executed first, followed by step 102, or steps 401 and 102 can be executed simultaneously. For example, while performing image segmentation on the first image to obtain the first mask corresponding to the first product and the preset person, key point recognition is performed on the first image through a hand recognition model to obtain hand key points. In this embodiment, the execution order of steps 401 and 102 is not limited.

[0082] Step 104: Perform image fusion based on the first mask, the second mask, the first image, and the second image to obtain the third image.

[0083] For example, image fusion refers to merging the second product in the second image with the preset person in the first image to obtain a third image in which the preset person is holding the second product.

[0084] Figure 5 This application provides a schematic diagram of an image fusion process, such as... Figure 5 As shown, step 104 above includes steps 501 to 503.

[0085] Step 501: Perform mask fusion on the first mask and the second mask to obtain the third mask.

[0086] In some examples, after obtaining the first mask and the second mask, it is necessary to fuse the first mask and the second mask to obtain a high-quality third mask corresponding to the first product. However, if the mask is directly fused, boundary conflicts, ghosting, or even the disappearance of some areas will occur. The reason for the above problems is that the hand area in the first mask and the area occluded by the hand in the second mask are treated as the same type of area.

[0087] In response to this, this embodiment divides the hand area in the first mask into an occlusion edge area, an occlusion inner area, a pressing edge area, and a pressing inner area based on the two states that exist when the hand is in contact with the product: occlusion and pressing.

[0088] In some examples, the hand state of the preset character in the first mask is determined; based on the hand state, the hand of the preset character is divided into regions to obtain each hand region, and each hand region and the partially overlapping region are taken as the region to be merged.

[0089] For example, determining the hand state of the preset figure in the first mask includes: measuring the distance between the hand of the preset figure in the first mask and the edge of the product; if the edge distance is greater than or equal to a preset threshold, determining that the hand is in an obstructed state; and / or, if the edge distance is less than the preset threshold, determining that the hand is in a pressed state. And / or, detecting whether the hand of the preset figure in the first mask has epidermal diffusion characteristics; if yes, determining that the hand is in an obstructed state; if no, determining that the hand is in a pressed state; and / or, detecting whether the hand of the preset figure in the first mask has squeezing characteristics; if yes, determining that the hand is in a pressed state; if no, determining that the hand is in an obstructed state. And / or, calculating the pressure confidence level of the hand of the preset figure in the first mask; if the pressure confidence level is less than a first pressure confidence level, determining that the hand is in an obstructed state; if the pressure confidence level is greater than a second pressure confidence level, determining that the hand is in a pressed state.

[0090] For example, the "covering state" means that the hand merely covers the product without making any actual contact with the product, or the contact force is so small that it does not cause any compression or deformation of the contact area between the hand and the product. The "pressing state" means that there is actual contact between the hand and the product, causing the hand to be compressed and deformed.

[0091] For the hand in the occluded state, its outer contour is the occlusion edge area, and the area inside the outer contour is the occlusion inner area; for the hand in the pressing state, its outer contour is the pressing edge area, and the area inside the outer contour is the pressing inner area. The detection of the occlusion and pressing states can be directly completed by image detection.

[0092] For example, if the distance between the finger and the edge of the first product is greater than or equal to a preset value (usually 2 pixels), it is considered an occlusion state; if it is less than the preset value, it is considered a pressing state. If there are epidermal diffusion features on the finger surface, it is considered an occlusion state; if white squeezing features are detected on the finger surface, it is considered a pressing state; otherwise, it is considered an occlusion state. The pressure confidence score is calculated; if it is greater than 0.7, it is considered a pressing state; if it is less than 0.3, it is considered an occlusion state. If any of the above three conditions are met, it can be determined to be an occlusion state (actual detection conditions are not limited to these).

[0093] In some examples, algorithmic models can be used to calculate stress confidence, such as deep learning-based stress estimation models. By inputting stress-annotated hand images into the stress estimation model to be trained, a stress estimation model that can accurately output stress confidence based on the input hand images or hand image features can be obtained.

[0094] After completing the above-described area division of the hand in the first mask, the first mark can be applied to the occluded edge area of ​​the hand in the first mask, the second mark can be applied to the pressed edge area, and the third mark can be applied to the occluded inner area and the pressed inner area. At the same time, the completed first product in the second mask can be set as the fourth mark. The following example illustrates how a pre-defined character holds a coffee cup with one hand. Assume that the character's thumb and index finger do not fully contact the cup, but only the middle, ring, and little fingers. The marking rules for the first and second masks obtained using the aforementioned method are as follows: In the first mask, the edges of the thumb, index finger, and palm are the obscured edge areas, marked in red; the edges of the middle, ring, and little fingers are the pressed edge areas, marked in yellow; the interiors of each finger and the palm are the obscured interior areas, not marked; and in the second mask, the completed edge of the coffee cup, along with its interior, is uniformly marked in blue.

[0095] Based on the processing of the different markers mentioned above, the third mask of the first product obtained by the final mask fusion is a four-channel third mask (red, yellow, blue, and no marker).

[0096] If the masked edge area is not properly addressed, it may result in jagged edges, rough edges, or excessive smoothing. During subsequent product replacement, inaccurate mask edges can easily lead to color intrusion between the second product and the preset character's hand, or obvious splicing marks. Therefore, an edge protection strategy is adopted for this area to create a strict distinction between it and the product area.

[0097] For the pressing edge area, unlike the aforementioned obscured edge area, since a certain degree of squeezing and overlap occurs between the finger and the product during the pressing process, it is necessary to comprehensively consider the interaction between the finger and the product in order to simulate a more realistic pressing state.

[0098] Based on the above, in the subsequent image fusion process, the blue markers are completely replaced, the red markers are edge protected, the yellow markers are made transparent, and the unmarked parts (i.e., the occluded internal areas and other background areas) can be directly preserved as original pixels.

[0099] It should be noted that the edge protection processing performed on the red area as described above specifically includes: 1) spreading a certain distance (usually 1 to 5 pixels) to both sides of the edge corresponding to the red mark to form an edge buffer area; 2) obtaining edge distance field information based on the distance between each pixel in the buffer area and the edge of the area, and performing Gaussian blur processing on the pixels in the buffer area based on the edge distance field information to obtain the feathered red mark part.

[0100] The transparency processing performed on the yellow area specifically includes: 1) spreading a certain distance (usually 1 to 5 pixels) to both sides of the edge corresponding to the yellow mark to form a first press buffer area; 2) taking the fingertip key point inside this area as the origin, calculating the distance from each pixel to the neighboring fingertip key point to obtain the press distance field information; 3) calculating the transparency weight corresponding to different pixels based on the above press distance field and the preset transparency calculation model; the above transparency calculation model can refer to the following formula: α = AB × exp(-d² / (2σ²)) The aforementioned α is the transparency weight, which is inversely proportional to the visual effect of transparency; σ is the diffusion coefficient, i.e. the aforementioned diffusion distance; A and B are constants, for example, A is 0.9 and B is 0.6.

[0101] Based on the above calculations, the closer a pixel is to the fingertip, the larger its α value, resulting in a more realistic pixel appearance; conversely, the farther a pixel is from the fingertip, the smaller its α value, resulting in a more transparent pixel appearance. This creates a gradual increase in transparency from the center area pressed by the fingertip outwards. This avoids affecting the presentation of the core features of the second product during subsequent fusion, and in the scenario of a finger holding the product, it allows the overall effect to more closely resemble the deformation of the finger during actual pressing, achieving a more realistic grip and finger-wrapping effect.

[0102] Step 502: Based on the third mask and key points of the hand, perform morphological adjustments on the second image to obtain a deformed image.

[0103] For example, morphological adjustment of the second image refers to geometrically deforming the second image to adapt the second product to the hand of a preset person. The deformed image includes an image of the second product that matches the hand-holding posture of the preset person.

[0104] Because the second product may differ significantly in size from the first product—for example, the first product is a mobile phone and the second product is a tablet—directly replacing the first product with the second would result in a mismatch between the replaced hand and the product due to the size difference, leading to issues such as the product appearing to float or the hand clipping through the image. Therefore, by morphologically adjusting the second image, a deformed image is obtained that matches the hand's gripping posture of the preset character.

[0105] Figure 6 This is a schematic diagram of an image morphology adjustment process provided in an embodiment of this application, such as... Figure 6 As shown, step 502 above includes steps 601 to 602.

[0106] Step 601: Extract control points from the third mask based on the key points of the hand to obtain the first control point, and extract control points from the second image to obtain the second control point.

[0107] The first control point can be the target control point, and the second control point can be the source control point.

[0108] In some examples, based on hand key points, contour points are extracted from the third mask through the contour processing layer of the contour processing model to obtain contour points; convex hull calculation is performed on the contour points through the convex hull processing layer of the contour processing model to obtain convex hull boundary points; and boundary optimization is performed on the convex hull boundary points through the boundary optimization layer of the contour processing model to obtain the first control points.

[0109] For example, control points can be obtained by extracting control points from a third mask based on hand key points using a first extraction model. The first extraction model can be a contour processing model. The first control points are obtained by inputting the hand key points and the third mask into the contour processing model for control point extraction. These first control points represent the contour of the first product. By incorporating hand control points during feature point extraction in the contour processing model, the extracted first control points can conform to the contour of the first product.

[0110] The first extraction model may include a contour processing layer and a convex hull processing layer. The contour processing layer is used to extract the contour point sequence of the third mask corresponding to the first product. The input of the contour processing layer includes the third mask corresponding to the first product, and the output is the contour point sequence.

[0111] The processing steps for the contour processing layer are as follows: First, input the third mask and randomly preset a pixel matrix. Set the pixel value of the first pixel in the pixel matrix to 1, and set at least one pixel with a value of 0 in the adjacent pixels as the starting point. Use the starting point as the starting detection point for the contour extraction process. Secondly, starting from the starting point, detect the surrounding 8 pixels in a fixed manner. Pixels that have a value of 1 and at least one of their adjacent pixels with a value of 0 are taken as the next pixel of the contour. This process is iterated until the starting point is included in the contour again. At this point, the contour point sequence of the third mask can be obtained based on the coordinates of the obtained multiple pixels.

[0112] In some examples, the convex hull processing layer of the contour processing model is used to calculate the convex hull of the contour points to obtain the convex hull boundary points. This includes: sorting the contour points based on the coordinate values ​​of the contour point sequence to obtain the contour point sequence; determining the current contour point and the two previous contour points in the contour point sequence according to a preset traversal order; constructing a first point vector and a second point vector based on the current contour point and the two previous contour points, and calculating the product of the first point vector and the second point vector; if the product value is positive, taking the current contour point and the two previous contour points as the first convex hull boundary points, and continuing to traverse until the entire contour point sequence has been traversed to obtain the convex hull boundary points; if the first product value is negative, deleting the migrated contour points of the current contour point, and continuing to traverse until the entire contour point sequence has been traversed to obtain the convex hull boundary points.

[0113] For example, the convex hull processing layer is used to further calculate the convex hull boundary points based on the contour point sequence of the first code. Here, the convex hull represents the smallest convex polygon without any indentations that can completely enclose the contour of the first product; that is, the convex hull is the outermost contour of the first product, while some indented details in the first product are ignored, thereby highlighting the overall geometric shape of the first product and providing a simpler data foundation for subsequent key points. The input of the convex hull processing layer is the contour point sequence, and the output is the convex hull boundary point sequence.

[0114] The processing steps for the convex hull layer are as follows: First, sort the contour point sequence according to the x-coordinate (or y-coordinate if the x-coordinates are the same); Secondly, construct the first convex hull: traverse the sorted contour points from left to right, and take the contour points that meet the requirements as the first convex hull points; The above requirement means that, based on the currently detected contour point and the two previously adjacent contour points (a total of three points), two vectors are constructed, and the cross product of these two vectors is calculated. If it is positive, it means that the position of the three contour points is not concave, and the first two contour points can be directly used as the first convex hull point, and the last contour point is included in the next round of detection; if it is negative, it means that the position of the three contour points is concave, and the contour point in the middle of the three points is deleted, and the remaining two contour points are included in the next round of detection. Next, construct the second convex hull: traverse the sorted contour points from right to left, and use the contour points that meet the requirements as the second convex hull points. The specific calculation is the same as for the first convex hull. Finally, the first convex hull and the second convex hull are merged, and duplicate convex hull points are removed. The remaining first convex hull points and the second convex hull points together constitute the final sequence of convex hull boundary points.

[0115] In addition, the first extraction model may also include a contour simplification layer. The contour simplification layer simplifies the convex hull boundary point sequence to obtain a final target control point sequence that facilitates subsequent calculations. The input to the contour simplification layer is the aforementioned convex hull boundary point sequence, and the output is the final target control point sequence.

[0116] The processing steps for the contour simplification layer are as follows: First, sort the sequence of convex hull boundary points, and connect the starting point A and the ending point B of the sequence with a straight line AB. Next, calculate the perpendicular distance from other convex hull boundary points in the sequence to AB, and take the maximum value P1. Compare P1 with a preset tolerance value. If P1 is greater than the tolerance value, then the convex hull boundary point C corresponding to P1 is the core convex hull point. At this time, construct two straight lines AC and BC, and calculate the perpendicular distance between the convex hull boundary point between point A and point C and AC. Again, take the maximum value P2. Based on the comparison result of P2 and the tolerance value, if P2 is greater than the tolerance value, then the convex hull boundary point D corresponding to P2 is the core convex hull point, and then construct two straight lines AD and CD; otherwise, if P2 is less than the tolerance value, all convex hull boundary points in segment AC are redundant. The processing of segment BC is the same as above (it should be noted that the aforementioned P1 may also be less than the tolerance value, similar to the first product itself being an approximately straight object such as chopsticks or a pen). Iterate in this way until P in any segment is less than the tolerance value. At this time, all core convex hull points can be obtained, and this sequence of core convex hull points is the final target control point (i.e., the first control point).

[0117] In some examples, a target feature detection model is determined based on the product form of the second product corresponding to the second image. The target feature detection model is then used to extract features from the second image to obtain the second control point.

[0118] For example, control points can be extracted from the second image using a second extraction model to obtain second control points. The second extraction model can be a feature detection model, such as the SuperPoint model or the MeshLab model. If the product is a regular object, the SuperPoint model can be used; if the product is an irregular object, the MeshLab model can be used.

[0119] The extraction rules followed by the second extraction model when extracting source control points can be determined according to the shape of the second product. For example, for rigid objects, the boundary corner points and edge midpoints are used as source control points (usually 8 to 12 points). For objects with handles, the source control points are sampled more densely in the handle area (usually 30 points) on the basis of the above. For flexible objects, the surface mesh vertices are used as source control points (usually 50 to 200 points).

[0120] It should be noted that if the first and second products are different types of products, such as the first product being a mobile phone and the second product being a water cup, then the extraction of source control points and target control points must be processed using different models (the first extraction model and the second extraction model) as described above. If the first and second products are of the same type of product, such as both being mobile phones, due to their similar structures, the same model can be used, such as the Siamese model, to extract source control points and target control points.

[0121] Step 602: Based on the second image, the first control point, and the second control point, determine the control point adjustment parameters, and perform morphological adjustments on the second image according to the control point adjustment parameters to obtain a deformed image.

[0122] In some examples, a deformed image can be obtained by adjusting the image based on a second image, a first control point, and a second control point using an interpolation model (Thin Plate Spline, TPS). The inputs to the interpolation model are the first control point, the second control point, and the second image, and the output is the deformed image. For example, the deformed image is obtained by adjusting the second image.

[0123] It should be noted that other methods such as triangular mesh deformation or spatial transformation network can also be used to adjust the image to obtain deformed images. The choice can be made according to the application. For example, affine transformation can be used directly for simple-shaped items such as books; STN model can be used for irregularly shaped products such as musical instruments and handicrafts.

[0124] The control point adjustment parameters include a first adjustment parameter, a second adjustment parameter, and / or a second adjustment weight. The first adjustment parameter can be an affine transformation coefficient of the second control point relative to the first control point, calculated based on the distance (or positional difference) between the second and first control points.

[0125] The second adjustment parameter can be the local influence intensity of the second control point on other pixels, calculated based on the second control point and other pixels. This local influence intensity characterizes the intensity or amplitude of the pixel during the deformation process, determined by the distance between each pixel and a certain second control point.

[0126] The second adjustment weight can be the local influence weight of the second control point on other pixels. This local influence weight represents the specific magnitude of deformation of each pixel caused by the simultaneous constraint process of multiple different second control points.

[0127] Figure 7 This application provides a schematic diagram of a control point movement process, as shown in the embodiment of the present application. Figure 7 As shown, step 602 above includes steps 701 to 703.

[0128] Step 701: Calculate the first adjustment parameter based on the distance between the first control point and the second control point, and move the second control point in the second image based on the first adjustment parameter to obtain the initial deformed image.

[0129] For example, based on the positional difference between the first and second control points, the affine transformation coefficients of the first control point relative to the second control point can be calculated. The second control point is then moved based on these affine transformation coefficients to obtain an initial deformed image. This initial deformed image can also be called a global deformed image. The affine transformation coefficients may include translation, rotation, and scaling factors. It should be noted that during this process, other pixels in the second image are only adjusted according to the aforementioned affine transformation coefficients and do not participate in the calculation.

[0130] Step 702: Calculate the second adjustment parameter and the second adjustment weight based on the distance between the second control point and other control points in the initial deformed image.

[0131] Based on the initial deformed image, the local influence intensity of each second control point on other pixels (including both second and non-second control points) in the initial deformed image is calculated according to the distance between the moved second control point and other pixels in the initial deformed image. The local influence intensity characterizes the strength or magnitude of the deformation process of each pixel, determined by the distance between it and a given second control point.

[0132] Based on the calculated local influence intensity, and using the second control point and the first control point, the local influence weight of each second control point on other pixels is calculated. The local influence weight represents the specific magnitude of deformation caused by the simultaneous constraint process of multiple different second control points on each pixel.

[0133] For example, the local influence weight can be calculated using the following formula: Second control point coordinates + local influence intensity Local influence weight = coordinates of the first control point It should be noted that if the initial deformed image is used directly as the output, on the one hand, it cannot achieve a precise match between the second control point and the first control point, which may result in a certain deviation. On the other hand, since the non-second control points in the image often cannot be directly adjusted by linear transformation, distortion and other distortion phenomena will inevitably occur. This embodiment uses a combination of global deformation and local deformation to process the image, making the obtained deformed image more realistic.

[0134] Step 703: Based on the second adjustment parameter and the second adjustment weight, move the second control point in the initial deformation image to obtain the deformation image.

[0135] For example, the second adjustment parameter (i.e., local influence intensity) and the second adjustment weight (i.e., local influence weight) together constitute the local deformation parameter of a certain pixel in the initial deformed image (i.e., global deformed image). Each pixel in the global deformed image is adjusted according to the local deformation parameter to obtain the output deformed image.

[0136] Step 503: Perform image fusion based on the deformed image, the first image, and the third mask to obtain the third image.

[0137] For example, the third image is an image of a preset person holding the second product. Based on the aforementioned deformed image, although the second product in the deformed image already matches the shape of the preset person's hand, there may still be issues such as jagged edges, and distortion in lighting and shadows. To further improve the realism of the third image, this embodiment processes the edges, lighting, and shadows of the deformed image to obtain a more realistic and aesthetically pleasing third image.

[0138] Figure 8 This application provides a schematic diagram of an image segmentation process, such as... Figure 8 As shown, step 503 above includes steps 801 to 803.

[0139] Step 801: Based on the third mask, perform image edge fusion between the deformed image and the first image to obtain the first fused image.

[0140] For example, image edge blending includes performing Poisson blending on the image edges. The first blended image may be an image of a pre-defined person holding a second product, with the edges of the second product and the hand smoothly transitioning.

[0141] In some examples, the regions to be fused contained in the third mask are determined; each region to be fused includes an occlusion edge region, an occlusion interior region, a pressed edge region, a pressed interior region, and / or a partially overlapping region; according to the fusion strategy corresponding to each region to be fused, the deformed image and the first image are fused at the image edges to obtain the first fused image.

[0142] In some examples, image edge fusion is performed on the deformed image and the first image according to the fusion strategy corresponding to each region to be fused to obtain a first fused image. This includes: performing edge protection processing on the occluded edge areas in each region to be fused, making the pressed edge areas in the region to be fused transparent, performing pixel retention processing on the occluded inner areas and pressed inner areas in the region to be fused, and performing pixel replacement processing on some overlapping areas in the region to be fused based on the deformed image to obtain the first fused image.

[0143] For example, edge protection processing is performed on the occluded edge area in the region to be fused, including: spreading the region edge of the occluded edge area to obtain an edge buffer; calculating edge distance field information based on the distance between each pixel in the edge buffer and the region edge; and performing Gaussian blur processing on the pixels in the edge buffer based on the edge distance field information to obtain the edge-protected occluded edge area.

[0144] For example, the process of making the pressed edge area in the area to be merged transparent includes: spreading the edge of the pressed edge area to obtain a pressed buffer; calculating the pressed distance field information based on the distance between the key points of the fingertip in the pressed buffer; calculating the transparency weight based on the pressed distance field information and a preset transparency calculation model; and performing transparency processing based on the transparency weight to obtain the pressed edge area after transparency processing.

[0145] For example, during the process of image edge fusion of the deformed image and the first image according to the third mask, the image area to be replaced in the first image can be determined based on the coordinates of each pixel in the third mask, and the image replacement of the second product can be performed in the image area.

[0146] In the image replacement process, a Poisson fusion model is used to process different regions to ensure a smooth transition between the second product and other areas. The purpose of Poisson fusion is to adjust the gradient of the image (the gradient refers to the rate of change of pixel values ​​at a point in the image along the x and y axes, specifically reflecting the degree of abrupt changes in brightness, color, and contour of the corresponding area in the image) under preset constraints, so that the gradient changes between different parts of the image are continuous and smooth, resulting in a better edge connection between the second product and the preset person's finger.

[0147] Since the aforementioned third mask has already used different markers to distinguish different interactions between the finger and the product during the generation process, the image of the product can be gradient-processed according to the above different markers during the fusion process, and the corresponding rules are as follows: For the fourth marker (corresponding to the product body part), it is necessary to ensure that the internal gradient information is completely consistent with the deformed product image, and only change in the area near the edge of the first and second markers.

[0148] For the first marker (corresponding to the edge of the occluded finger), it is necessary to strictly prevent the second product from intruding into the finger area. Therefore, in actual processing, the gradient information of the finger edge at the position corresponding to the first marker is used as a constraint, and the gradient information from the area corresponding to the second product to the finger edge is smoothly transitioned through the calculation of the Poisson fusion model.

[0149] For the second marker (corresponding to the edge of the pressing finger), this part needs to comprehensively consider the allocation of the product and the finger. In this invention, gradient weight allocation calculation is introduced for this area, as follows: the second product is spread out to both sides of the edge position of the corresponding area by a certain distance (usually 1 to 5 pixels) to form a second pressing buffer area; the second pressing buffer area and the first pressing buffer area in the first mask mentioned above overlap at least partially. Thus, the first pressing buffer area and the second pressing buffer area together constitute three intervals: the product area completely located in the second product, the finger area completely located in the finger, and the overlapping area that overlaps with each other.

[0150] The gradient weights of the finger transparency and / or the second product are adjusted within the aforementioned range. Specifically, in the overlapping area, the gradient weight of the second product and the transparency of the finger gradually decrease as the finger extends towards the second product. This is reflected in the following ways: Closer to the second product, the higher the finger transparency, indicating that the finger becomes more transparent closer to the edge, resulting in lower finger realism. Simultaneously, the gradient weight of the second product is higher, indicating an increased visual proportion of the second product and better presentation of product details. Conversely, closer to the finger, the lower the finger transparency, indicating that the finger becomes more realistic closer to the inside, resulting in higher finger realism. Simultaneously, the gradient weight of the second product gradually decreases, indicating a reduced visual proportion of the second product, ensuring that the pressure sensation of the finger is preserved. The gradient weight of the second product in the product area and the finger transparency in the finger area still follow the above principle, i.e., increasing from the inside (finger) to the outside (product) and decreasing from the outside to the inside.

[0151] In actual processing, the gradient of the finger can be calculated from the original display image using preset gradient operators (such as the Sobel operator and the Prewitt operator), the gradient of the second product can be calculated from the deformed image, and different markers can be obtained from the third mask to complete the above calculations. Through these gradient changes, the harsh edges between the second product and the hand and background can be eliminated, making the pixel gradient of the second product and the gradient of the hand edge natural and continuous, thus solving the problem of the splicing effect in the contact area between the finger and the product.

[0152] Step 802: Perform image background fusion based on the first fused image and the background area of ​​the person in the first image excluding the first product to obtain the second fused image.

[0153] In some examples, lighting models (such as the Retinex model, CycleGAN, etc.) can be used to ensure that the lighting effect of the second product matches that of the hand, based on the first blended image. The input to the lighting model includes the first blended image mentioned above, and the first image with the background image of the person from the first product removed, and the output is the second blended image, which refers to the image where the lighting style of the second product and the hand are matched.

[0154] Step 803: Perform image background fusion based on the background areas of the second fused image and the first image, excluding the first product and the preset person, to obtain the third image.

[0155] In some examples, a shadow model (such as the CycleGAN model) can be used to ensure that the shadow effect of the second product matches the background, based on the second fused image. The input of the shadow model includes the second fused image mentioned above, and a pure background image from which the first product and a preset person (including hands) are removed from the first image (to prevent shadows from overlapping the hands). The output is the final target display image (i.e., the third image), in which the shadow brightness is uniform and the transition is natural.

[0156] Step 105: Perform driving processing on the third image according to the driving instructions to obtain a driving image.

[0157] In some examples, the driving instructions can be input audio, video, etc. A driving model (e.g., the EchoMimic model, the Keling model) can then process the third image according to the driving instructions to obtain a driving image. In the driving image, a preset character performs the driving action corresponding to the driving instruction; the driving action can be the preset character's lip movements or hand gestures. It should be noted that the driving image can be a sequence of driving images, i.e., a driving video.

[0158] The image-driven method provided in this application involves acquiring a first image containing a first product and a preset person, and a second image containing a second product. The first image is segmented to obtain a first mask corresponding to the first product and the preset person. Based on product prompts, key hand points of the preset person, and the first mask, the first image is segmented again to obtain a second mask corresponding to the first product. Image fusion is performed based on the first mask, the second mask, the first image, and the second image to obtain a third image. The third image is then processed according to a driving instruction to obtain a driving image. In the driving image, the preset person performs a driving action corresponding to the driving instruction. By obtaining a precise image mask through two image segmentations and combining hand key point constraints, a natural fusion of the product and person is achieved. Finally, a dynamic motion image is generated through the driving instruction. This method solves the problems of unnatural connection between the product and person, low segmentation accuracy, and poor scene adaptability in image fusion, achieving the technical effects of improving the quality of the fused image, increasing generation efficiency, and enhancing scene flexibility.

[0159] Corresponding to the aforementioned embodiments of the image driving method, this application also provides embodiments of the image driving device. Figure 9 A task processing apparatus provided in the embodiments of this application, such as Figure 9As shown, the task processing device 900 includes a distance image acquisition module 901, a first image segmentation module 902, a second image segmentation module 903, an image fusion module 904, and a driving module 905.

[0160] Image acquisition module 901 is used to acquire a first image and a second image; the first image contains a first product and a preset person, and the second image contains a second product; The first image segmentation module 902 is used to segment the first image to obtain a first mask corresponding to the first product and the preset person; The second image segmentation module 903 is used to segment the first image based on the product prompt, the key points of the preset person's hand and the first mask, and obtain the second mask corresponding to the first product. Image fusion module 904 is used to perform image fusion based on the first mask, the second mask, the first image, and the second image to obtain a third image; The driving module 905 is used to perform driving processing on the third image according to the driving instruction to obtain a driving image; the preset character in the driving image presents the driving action corresponding to the driving instruction.

[0161] In another possible implementation, the first image segmentation module 902 is specifically used to perform image segmentation on the first image based on the first product and the preset person using a first image segmentation model to obtain a first mask corresponding to the hands of the first product and the preset person; the first mask contains at least a partially overlapping area between the hands of the first product and the preset person.

[0162] In another possible implementation, the second image segmentation module 903 is specifically used to perform key point recognition on the first image using a hand recognition model to obtain hand key points; based on the product prompt, the hand key points, and the first mask, the second image segmentation model is used to perform image segmentation on the first image to obtain a second mask corresponding to the first product; the second mask includes an image mask obtained by completing the partially overlapping area according to the product prompt.

[0163] In another possible implementation, the second image segmentation module 903 is further configured to perform feature transformation on the product prompt and the first image respectively through the second image segmentation model to obtain prompt features and image features; calculate the feature similarity between the image features and the prompt features based on the hand key points; determine the position information of the first product in the first mask according to the feature similarity; and complete the overlapping area according to the prompt features and the position information to obtain the second mask.

[0164] In another possible implementation, the image fusion module 904 is specifically used to perform mask fusion on the first mask and the second mask to obtain a third mask; to perform shape adjustment on the second image based on the third mask and the hand key points to obtain a deformed image; and to perform image fusion on the deformed image, the first image and the third mask to obtain the third image.

[0165] In another possible implementation, when the image fusion module 904 performs the action of "adjusting the shape of the second image based on the third mask and the hand key points to obtain a deformed image", it is specifically used to extract control points from the third mask based on the hand key points to obtain a first control point, and to extract control points from the second image to obtain a second control point; based on the second image, the first control point, and the second control point, it determines control point adjustment parameters, and adjusts the shape of the second image according to the control point adjustment parameters to obtain the deformed image.

[0166] In another possible implementation, the image fusion module 904, when performing the process of determining control point adjustment parameters based on the second image, the first control point, and the second control point, and adjusting the shape of the second image according to the control point adjustment parameters to obtain the deformed image, specifically calculates a first adjustment parameter based on the distance between the first control point and the second control point, and moves the second control point in the second image based on the first adjustment parameter to obtain an initial deformed image; calculates a second adjustment parameter and a second adjustment weight based on the distance between the second control point in the initial deformed image and other control points; and moves the second control point in the initial deformed image based on the second adjustment parameter and the second adjustment weight to obtain the deformed image.

[0167] In another possible implementation, when the image fusion module 904 performs the action of "extracting control points from the third mask based on the hand key points to obtain the first control point", it specifically performs the following steps: extracting contour points from the third mask based on the hand key points through the contour processing layer of the contour processing model to obtain contour points; performing convex hull calculation on the contour points through the convex hull processing layer of the contour processing model to obtain convex hull boundary points; and performing boundary optimization on the convex hull boundary points through the boundary optimization layer of the contour processing model to obtain the first control point.

[0168] In another possible implementation, the image fusion module 904, when performing convex hull calculation on the contour points through the convex hull processing layer of the contour processing model to obtain convex hull boundary points, specifically sorts the contour points based on their coordinate values ​​to obtain a contour point sequence; determines the current contour point and the previous two contour points in the contour point sequence according to a preset traversal order; constructs a first point vector and a second point vector based on the current contour point and the previous two contour points, and calculates the product of the first point vector and the second point vector; if the product value is positive, uses the current contour point and the previous two contour points as convex hull boundary points, and continues traversing until the entire contour point sequence has been traversed to obtain the convex hull boundary points; if the product value is negative, deletes the current contour point, and continues traversing until the entire contour point sequence has been traversed to obtain the convex hull boundary points.

[0169] In another possible implementation, when the image fusion module 904 performs control point extraction on the second image to obtain the second control point, it is specifically used to determine the target feature detection model based on the commodity form of the second commodity corresponding to the second image, and to extract features from the second image through the target feature detection model to obtain the second control point.

[0170] In another possible implementation, when the image fusion module 904 performs image fusion based on the deformed image, the first image, and the third mask to obtain the third image, it is specifically used to perform image edge fusion between the deformed image and the first image according to the third mask to obtain a first fused image; perform image background fusion based on the first fused image and the background area of ​​the person in the first image excluding the first product to obtain a second fused image; and perform image background fusion based on the second fused image and the background area of ​​the first image excluding the first product and the preset person to obtain the third image.

[0171] In another possible implementation, the image fusion module 904, when performing image edge fusion of the deformed image and the first image according to the third mask to obtain a first fused image, is specifically used to determine each region to be fused contained in the third mask; the regions to be fused include occlusion edge regions, occlusion interior regions, pressed edge regions, pressed interior regions and / or partially overlapping regions; and performs image edge fusion of the deformed image and the first image according to the fusion strategy corresponding to each region to be fused to obtain the first fused image.

[0172] In another possible implementation, the image fusion module 904, when performing image edge fusion of the deformed image and the first image according to the fusion strategy corresponding to each region to be fused to obtain the first fused image, specifically performs edge protection processing on the occluded edge area in the region to be fused, performs transparency processing on the pressed edge area in the region to be fused, performs pixel retention processing on the occluded inner area and the pressed inner area in the region to be fused, and performs pixel replacement processing on some overlapping areas in the region to be fused based on the deformed image to obtain the first fused image.

[0173] In another possible implementation, the image fusion module 904, when performing edge protection processing on the occluded edge region in the region to be fused, specifically diffuses the region edge of the occluded edge region to obtain an edge buffer; calculates edge distance field information based on the distance between each pixel in the edge buffer and the region edge; and performs Gaussian blur processing on the pixels in the edge buffer based on the edge distance field information to obtain the edge-protected occluded edge region.

[0174] In another possible implementation, when the image fusion module 904 performs transparency processing on the pressed edge area in the region to be fused, it specifically diffuses the regional edge of the pressed edge area to obtain a pressed buffer; calculates pressed distance field information based on the distance between the fingertip key points in the pressed buffer; calculates transparency weight based on the pressed distance field information and a preset transparency calculation model; and performs transparency processing based on the transparency weight to obtain the transparent pressed edge area.

[0175] In another possible implementation, the image driving device 900 further includes a hand state determination and region division module 906, used to determine the hand state of the preset character in the first mask; according to the hand state, the hand of the preset character is divided into regions to obtain each hand region, and each hand region and the partially overlapping region are used as the region to be fused.

[0176] In another possible implementation, the hand state determination and region division module 906, when performing the determination of the hand state of the preset person in the first mask, is specifically used to detect the edge distance between the hand of the preset person in the first mask and the edge of the product. If the edge distance is greater than or equal to a preset threshold, the hand is determined to be in an occluded state; and / or, if the edge distance is less than the preset threshold, the hand is determined to be in a pressing state.

[0177] In another possible implementation, the hand state determination and region segmentation module 906, when determining the hand state of the preset character in the first mask, specifically detects whether the hand of the preset character in the first mask has epidermal diffusion characteristics. If so, it determines that the hand is in an occluded state; if not, it determines that the hand is in a pressing state. And / or, it detects whether the hand of the preset character in the first mask has squeezing characteristics. If so, it determines that the hand is in a pressing state; if not, it determines that the hand is in an occluded state.

[0178] In another possible implementation, the hand state determination and region division module 906, when performing "determining the hand state of the preset character in the first mask", is specifically used to calculate the pressure confidence of the hand of the preset character in the first mask; if the pressure confidence is less than the first pressure confidence, the hand is determined to be in an obstructed state; if the pressure confidence is greater than the second pressure confidence, the hand is determined to be in a pressing state.

[0179] The beneficial technical effects corresponding to the exemplary embodiment of the image driving device 900 described above can be found in the corresponding beneficial technical effects in the above method embodiment section, and will not be repeated here.

[0180] Figure 10 is a schematic diagram of an image driving device provided in some embodiments of this application. In some embodiments, the computing device may be a server, a terminal device, etc. The computing device includes a multi-core processor and a memory. The multi-core processor includes multiple processor cores; the memory is configured to store one or more programs. Wherein, when the one or more programs are executed by the multi-core processor, the multi-core processor implements the master station communication method or the master station module deployment method in the above embodiments.

[0181] As shown in Figure 10, device 1000 includes a multi-core processor 1001 and a memory 1002. Exemplarily, computing device 1000 may also include a communications interface 1003 and a communications bus 1004.

[0182] The multi-core processor 1001, memory 1002, and communication interface 1003 communicate with each other via communication bus 1004. Communication interface 1003 is used to communicate with other network elements such as clients or other servers.

[0183] In some examples, the multi-core processor 1001 is used to execute program 1005, specifically performing the relevant steps in the above-described master station communication method or master station module deployment method embodiments. Specifically, program 1005 may include program code, which includes computer-executable instructions.

[0184] For example, the multi-core processor 1001 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of this application. The computing device 1000 may include multi-core processors, which may be of the same type, such as one or more CPUs; or they may be multi-core processors of different types, such as one or more CPUs and one or more ASICs.

[0185] In some examples, memory 1002 is used to store program 1005. Memory 1002 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage.

[0186] Specifically, program 1005 can be called by multi-core processor 1001 to enable computing device 1000 to execute master station communication methods or master station module deployment methods.

[0187] Some embodiments of this application provide a computer-readable storage medium storing at least one executable instruction that, when executed on a computing device 1000, causes the computing device 1000 to perform the master station communication method or the master station module deployment method described above.

[0188] Specifically, the executable instructions can be used to enable the computing device 1000 to perform master station communication methods or master station module deployment methods.

[0189] For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device.

[0190] The beneficial effects that the readable storage medium provided in some embodiments of this application can achieve can be referred to the beneficial effects in the corresponding master station communication method or master station module deployment method provided above, and will not be repeated here.

[0191] In addition to the methods, apparatus, and devices described above, embodiments of this application may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the image driving methods of the various embodiments of this application described in the above method embodiment section.

[0192] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0193] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the image driving methods of the various embodiments of this application described in the above-described method embodiment section.

[0194] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0195] The basic principles of this application have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details of the above embodiments are for illustrative and facilitative purposes only, and are not limitations. These details do not restrict this application from being implemented using the aforementioned specific details.

[0196] Those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0197] Furthermore, the embodiments described above are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solution of this application should be included within the scope of protection of this application.

Claims

1. An image-driven method, characterized in that, The method includes: Acquire a first image and a second image; the first image contains a first product and a preset person, and the second image contains a second product; Perform image segmentation on the first image to obtain a first mask corresponding to the first product and the preset character; Based on the product prompts, the key points of the hand of the preset person, and the first mask, the first image is segmented to obtain the second mask corresponding to the first product. A third image is obtained by fusing the first mask, the second mask, the first image, and the second image. The third image is processed according to the driving instruction to obtain a driving image; the preset character in the driving image presents the driving action corresponding to the driving instruction.

2. The image driving method according to claim 1, characterized in that, The step of segmenting the first image to obtain a first mask corresponding to the first product and the preset person includes: Based on the first product and the preset person, the first image is segmented using a first image segmentation model to obtain a first mask corresponding to the hands of the first product and the preset person; the first mask contains at least a partially overlapping area between the hands of the first product and the preset person.

3. The image driving method according to claim 2, characterized in that, The step of segmenting the first image based on product prompts, key hand points of the preset person, and the first mask to obtain a second mask corresponding to the first product includes: The first image is used to identify key points using a hand recognition model to obtain key hand points; Based on the product prompt, the hand key points, and the first mask, the first image is segmented using a second image segmentation model to obtain a second mask corresponding to the first product; the second mask includes an image mask obtained by filling in the partially overlapping area according to the product prompt.

4. The image driving method according to claim 3, characterized in that, The step of segmenting the first image using a second image segmentation model based on the product prompt, the hand key points, and the first mask to obtain a second mask corresponding to the first product includes: The product prompt and the first image are respectively transformed using the second image segmentation model to obtain prompt features and image features; Based on the key hand points, the feature similarity between the image features and the prompt features is calculated. The position information of the first item in the first mask is determined according to the feature similarity. The overlapping area is then filled in according to the prompt features and the position information to obtain the second mask.

5. The image driving method according to claim 1, characterized in that, The step of fusing images based on the first mask, the second mask, the first image, and the second image to obtain a third image includes: The first mask and the second mask are merged to obtain the third mask; Based on the third mask and the key points of the hand, the second image is morphologically adjusted to obtain a deformed image; The third image is obtained by fusing the deformed image, the first image, and the third mask.

6. The image driving method according to claim 5, characterized in that, The step of performing morphological adjustments on the second image based on the third mask and the key points of the hand to obtain a deformed image includes: Based on the key points of the hand, control points are extracted from the third mask to obtain the first control point, and control points are extracted from the second image to obtain the second control point; Based on the second image, the first control point, and the second control point, control point adjustment parameters are determined, and the second image is morphologically adjusted according to the control point adjustment parameters to obtain the deformed image.

7. The image driving method according to claim 6, characterized in that, The step of determining control point adjustment parameters based on the second image, the first control point, and the second control point, and then performing morphological adjustments on the second image according to the control point adjustment parameters to obtain the deformed image, includes: Based on the distance between the first control point and the second control point, a first adjustment parameter is calculated, and the second control point in the second image is moved based on the first adjustment parameter to obtain an initial deformed image; Based on the distance between the second control point and other control points in the initial deformation image, calculate the second adjustment parameter and the second adjustment weight; Based on the second adjustment parameter and the second adjustment weight, the second control point in the initial deformation image is moved to obtain the deformation image.

8. The image driving method according to claim 6, characterized in that, The step of extracting control points from the third mask based on the key hand points to obtain the first control points includes: Based on the aforementioned key hand points, contour points are extracted from the third mask through the contour processing layer of the contour processing model to obtain the contour points. The convex hull processing layer of the contour processing model is used to perform convex hull calculation on the contour points to obtain the convex hull boundary points. The first control point is obtained by performing boundary optimization on the convex hull boundary points through the boundary optimization layer of the contour processing model.

9. The image driving method according to claim 8, characterized in that, The step of calculating the convex hull of the contour points through the convex hull processing layer of the contour processing model to obtain the convex hull boundary points includes: The contour points are sorted based on their coordinate values ​​to obtain the contour point sequence. The current contour point and the previous two contour points are determined in the contour point sequence according to the preset traversal order; Based on the current contour point and the previous two contour points, construct a first point vector and a second point vector, and calculate the product of the first point vector and the second point vector; If the product value is positive, the current contour point and the previous two contour points are taken as the first convex hull boundary point, and the traversal continues until the entire sequence of contour points has been traversed to obtain the convex hull boundary point. If the first product value is negative, the migrated contour point of the current contour point is deleted, and the traversal continues until the entire contour point sequence has been traversed to obtain the convex hull boundary point.

10. The image driving method according to claim 6, characterized in that, The step of extracting control points from the second image to obtain the second control points includes: Based on the product form of the second product corresponding to the second image, a target feature detection model is determined, and the second image is used to extract features through the target feature detection model to obtain the second control point.

11. The image driving method according to claim 5, characterized in that, The step of fusing the image based on the deformed image, the first image, and the third mask to obtain the third image includes: Based on the third mask, the deformed image and the first image are fused at the image edges to obtain a first fused image; A second fused image is obtained by performing image background fusion based on the first fused image and the background area of ​​the person in the first image excluding the first product; The third image is obtained by performing image background fusion between the second fused image and the background areas of the first image excluding the first product and the preset person.

12. The image driving method according to claim 11, characterized in that, The step of fusing the deformed image with the first image according to the third mask to obtain a first fused image includes: Determine each region to be merged contained in the third mask; each region to be merged includes an occlusion edge region, an occlusion interior region, a pressed edge region, a pressed interior region, and / or a partially overlapping region; Based on the fusion strategy corresponding to each region to be fused, the deformed image and the first image are fused at the image edges to obtain the first fused image.

13. The image driving method according to claim 12, characterized in that, The step of performing image edge fusion on the deformed image and the first image according to the fusion strategy corresponding to each region to be fused, to obtain the first fused image, includes: Edge protection processing is performed on the occluded edge area in the region to be merged, transparency processing is performed on the pressed edge area in the region to be merged, pixel retention processing is performed on the occluded inner area and the pressed inner area in the region to be merged, and pixel replacement processing is performed on the partially overlapping area in the region to be merged based on the deformed image to obtain the first merged image.

14. The image driving method according to claim 13, characterized in that, The edge protection processing of the occluded edge areas in the region to be merged includes: The edge of the occluded edge area is diffused to obtain an edge buffer zone; The edge distance field information is calculated based on the distance between each pixel in the edge buffer and the edge of the region. Based on the edge distance field information, the pixels in the edge buffer are subjected to Gaussian blurring to obtain the occluded edge area after edge protection.

15. The image driving method according to claim 13, characterized in that, The process of making the pressed edge area of ​​the region to be merged transparent includes: The edge of the pressed edge area is diffused to obtain a pressed buffer zone; Calculate the pressing distance field information based on the distance between the key points of the fingertip in the pressing buffer; Based on the pressing distance field information and the preset transparency calculation model, the transparency weight is calculated, and the transparency is processed according to the transparency weight to obtain the transparent pressing edge area.

16. The image driving method according to claim 12, characterized in that, The method further includes: Determine the hand state of the preset character in the first mask; Based on the hand state, the hand of the preset character is divided into regions to obtain each hand region, and each hand region and the partially overlapping region are taken as each region to be merged.

17. The image driving method according to claim 16, characterized in that, Determining the hand state of the preset character in the first mask includes: The distance between the hand of the preset figure in the first mask and the edge of the product is detected. If the edge distance is greater than or equal to a preset threshold, the hand is determined to be in an obscured state; and / or, if the edge distance is less than the preset threshold, the hand is determined to be in a pressing state.

18. The image driving method according to claim 16, characterized in that, Determining the hand state of the preset character in the first mask includes: Detect whether the hand of the preset character in the first mask has epidermal diffusion characteristics. If yes, determine that the hand is in an obscured state; if no, determine that the hand is in a pressed state. And / or, detect whether the hand of the preset character in the first mask has squeezing characteristics. If yes, determine that the hand is in a pressed state; if no, determine that the hand is in an obscured state.

19. The image driving method according to claim 16, characterized in that, Determining the hand state of the preset character in the first mask includes: Calculate the pressure confidence of the hand of the preset character in the first mask; if the pressure confidence is less than the first pressure confidence, determine that the hand is in a covered state; if the pressure confidence is greater than the second pressure confidence, determine that the hand is in a pressed state.

20. An electronic device, characterized in that, include: One or more processors; and The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image driving method according to any one of claims 1-19.

21. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image driving method according to any one of claims 1-19.

Images