Image fusion system

Abstract

Embodiments of the present application relate to the field of image processing. The present application provides an image fusion system, which comprises an image adjustment module and a pixel fusion module; the image adjustment module is configured to perform morphological adjustment on a second commodity image based on hand key points of a preset person in a first commodity image and an image mask of the first commodity to obtain a morphing image; the pixel fusion module is configured to perform image fusion based on the morphing image, the first commodity image and the image mask to obtain a target commodity image, thereby improving the quality of image fusion.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image fusion system. Background Technology

[0002] With the popularization of online sales, it has become the main sales method for sellers to provide sales videos of the products to be sold to the audience on different platforms through pre-recorded or live broadcasts. However, when sellers need to sell different products, they need to shoot images of themselves holding different products as material. With the wide variety of product categories, shooting each product one by one will undoubtedly affect the user experience.

[0003] To improve the efficiency of online sales, it is often necessary to fuse the visual characteristics of different products with images of specific scenes to generate display images that combine product features with a sense of scene realism. For example, merging images of newly launched beverages with existing images of people holding cups, or replacing images of new electronic products with promotional images of users holding older products, can reduce the time and cost of reshooting and quickly produce diverse product display materials. Traditional image fusion solutions often use direct pixel overlay, simple scaling, or global deformation to achieve product replacement, but these methods have many technical limitations.

[0004] Therefore, how to achieve accurate adaptation between product images and hand holding postures in the scene, while realizing efficient and natural image fusion, has become an urgent problem to be solved in this field. Summary of the Invention

[0005] This application provides an image fusion system that can achieve efficient and natural image fusion and improve image fusion efficiency.

[0006] In one embodiment of this application, an image fusion system is provided, the system comprising: an image adjustment module and a pixel fusion module; the image adjustment module is configured to perform morphological adjustment on a second product image based on preset key points of a person's hand in a first product image and an image mask of the first product, to obtain a deformed image; wherein, the deformed image includes an image obtained by adjusting the position of the outer contour of the second product image according to the outer contour of the first product in the first product image, and the deformed image and the second product image represent the same product; the pixel fusion module is configured to perform image fusion based on the deformed image, the first product image and the image mask to obtain a target product image.

[0007] In another possible implementation, the image adjustment module includes: a control point extraction unit configured to extract control points from the image mask based on the hand key points to obtain multiple target control points, and to extract control points from the second product image to obtain multiple source control points; wherein the target control points are used to characterize the outer contour of the first product, and the source control points are used to characterize the outer contour of the second product image; and a pixel adjustment unit configured to perform morphological adjustment on the second product image based on the multiple target control points and the multiple source control points to obtain the deformed image.

[0008] In another possible implementation, the control point extraction unit includes: a target control point extraction unit configured to perform contour calculation based on the hand key points and the image mask using a contour processing model to obtain the target control points; and a source control point extraction unit configured to determine a target feature detection model based on the product shape of the second product in the second product image, and to extract features from the second product image using the target feature detection model to obtain the source control points.

[0009] In another possible implementation, the target control point extraction unit includes: a contour point extraction subunit, configured to extract contour points from the image mask based on the hand key points through the contour processing layer of the contour processing model to obtain contour points; a convex hull boundary point extraction subunit, configured to perform convex hull calculation on the contour points through the convex hull processing layer of the contour processing model to obtain convex hull boundary points; and a boundary optimization subunit, configured to perform boundary optimization on the convex hull boundary points through the boundary optimization layer of the contour processing model to obtain the target control points.

[0010] In another possible implementation, the convex hull boundary point extraction subunit is configured to: sort the contour points based on the coordinate values ​​of the contour point sequence to obtain a contour point sequence; traverse the contour point sequence based on a first traversal rule to obtain a first convex hull boundary point; traverse the contour point sequence based on a second traversal rule to obtain a second convex hull boundary point; and merge the first convex hull boundary point and the second convex hull boundary point to obtain the convex hull boundary point.

[0011] In another possible implementation, the first traversal rule includes: determining the current contour point and the two preceding contour points in a left-to-right order; constructing a first point vector and a second point vector based on the current contour point and the two preceding contour points, and calculating a first product value of the first point vector and the second point vector; if the first product value is positive, using the current contour point and the two preceding contour points as the first convex hull boundary points, and continuing traversal until the entire contour point sequence has been traversed; if the first product value is negative, deleting the previous contour point of the current contour point, and continuing traversal until the entire contour point sequence has been traversed.

[0012] In another possible implementation, the second traversal rule includes: determining the current contour point and the two preceding contour points in a right-to-left order; constructing a third point vector and a fourth point vector based on the current contour point and the two preceding contour points, and calculating the second product value of the third point vector and the fourth point vector; if the second product value is positive, using the current contour point and the two preceding contour points as the second convex hull boundary points, and continuing traversal until the entire contour point sequence has been traversed; if the second product value is negative, deleting the previous contour point of the current contour point, and continuing traversal until the entire contour point sequence has been traversed.

[0013] In another possible implementation, the pixel adjustment unit includes: a global pixel adjustment unit configured to calculate a global pixel adjustment coefficient based on the coordinate values ​​between the target control point and the source control point, and to perform global shape adjustment on the second product image based on the pixel adjustment coefficient to obtain a global deformed image; and a local pixel adjustment unit configured to calculate local pixel adjustment parameters based on the coordinate values ​​of the source control point and other pixels in the global deformed image, and to perform local shape adjustment on the second product image based on the local pixel adjustment parameters to obtain the deformed image; the local pixel adjustment parameters include local pixel adjustment coefficients and local pixel adjustment weights.

[0014] In another possible implementation, the local pixel adjustment unit is configured to: calculate the local pixel adjustment coefficient corresponding to each of the source control points based on the distance between the pixel and each of the source control points in the global deformation image, wherein the local pixel adjustment coefficient is used to characterize the deformation amplitude of the pixel corresponding to the source control points at different distances; calculate the local pixel adjustment weight corresponding to each of the source control points in the global deformation image based on the local pixel adjustment coefficient, the coordinate values ​​of the source control points, and the coordinate values ​​of the target control points; wherein the local pixel adjustment weight is used to characterize the constraint relationship of the pixel corresponding to the source control points at different distances.

[0015] In another possible implementation, the system further includes a state recognition module, a region segmentation module, and a mask fusion module; the state recognition module is configured to perform state recognition on the hand of the preset person in the first image mask to obtain the hand state; the first image mask is obtained based on image segmentation of the first product image; the first product and the hand of the preset person in the first image mask at least partially overlap; the region segmentation module is configured to perform region segmentation on the first image mask according to the hand state to obtain multiple hand regions; the hand regions include occlusion edge regions, occlusion inner regions, pressing edge regions, and / or pressing inner regions; the mask fusion module is configured to perform mask fusion on the first image mask and the second image mask based on the multiple hand regions and the partially overlapping regions to obtain an image mask of the first product; the second image mask is obtained based on product hints, key points of the preset person's hand, and the first image mask through image segmentation; the product hints are used to characterize the reference type of the product; the second image mask includes a product mask obtained by completing the partially overlapping regions according to the product hints.

[0016] In another possible implementation, the state recognition module is configured to: detect the edge distance between the hand of the preset person in the first image mask and the edge of the first product; if the edge distance is greater than or equal to a preset threshold, determine that the hand is in an occluded state; and / or, if the edge distance is less than the preset threshold, determine that the hand is in a pressing state.

[0017] In another possible implementation, the region division module is configured to: for a hand in an occluded state, use the outer contour of the hand as the occlusion edge area and the inside of the outer contour of the hand as the occlusion inner area; or, for a hand in a pressing state, use the outer contour of the hand as the pressing edge area and the inside of the outer contour of the hand as the pressing inner area.

[0018] In another possible implementation, the pixel fusion module includes: an edge fusion unit configured to perform image edge fusion with the deformed image and the first product image according to the image mask to obtain a first fused image; an illumination fusion unit configured to perform image background fusion based on the first fused image and the background area of ​​the person in the first product image excluding the first product to obtain a second fused image; and a shadow fusion unit configured to perform image background fusion based on the second fused image and the background area of ​​the first product image excluding the first product and the preset person to obtain the target product image.

[0019] In another possible implementation, the edge blending unit is configured to: determine each region to be blended contained in the image mask; each region to be blended includes the occluded edge region, the occluded inner region, the pressed edge region, the pressed inner region, and / or the partially overlapping region; perform image edge blending on the deformed image and the first product image according to the blending strategy corresponding to each region to be blended, to obtain the first blended image; wherein the blending strategy includes: performing edge protection processing on the occluded edge region, performing transparency processing on the pressed edge region, performing pixel retention processing on the occluded inner region and the pressed inner region, and performing pixel replacement processing on the partially overlapping region.

[0020] In another possible implementation, the illumination fusion unit is configured to perform image background fusion based on the first fused image and the background area of ​​the person in the first product image, excluding the first product, to obtain a second fused image.

[0021] In another possible implementation, the shadow blending unit is configured to: perform image background blending based on the background areas of the second blended image and the first product image, excluding the first product and the preset person, to obtain the target product image.

[0022] The image fusion system provided by the embodiments of this application includes an image adjustment module and a pixel fusion module. The image adjustment module is configured to adjust the shape of a second product image based on the preset key points of a person's hand in the first product image and the image mask of the first product to obtain a deformed image. The pixel fusion module is configured to perform image fusion based on the deformed image, the first product image and the image mask to obtain a target product image, thereby improving the quality of image fusion. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of an image fusion system provided in an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of an image adjustment module provided in an embodiment of this application.

[0025] Figure 3 This is a schematic diagram of a control point extraction unit provided in an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of a target control point extraction unit provided in an embodiment of this application.

[0027] Figure 5 This is a schematic diagram of a pixel adjustment unit provided in an embodiment of this application.

[0028] Figure 6This is a schematic diagram of a pixel fusion module provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. To facilitate a clear description of the technical solutions of the embodiments of this application, the use of terms such as "first," "second," etc., in the embodiments of this application is merely for illustration and to distinguish the objects being described. There is no particular order between them, nor does it indicate a specific limitation on the number of devices in the embodiments of this application, and they cannot constitute any limitation on the embodiments of this application.

[0030] To address the issue of unnatural transitions between products and people in image fusion, this application provides an image fusion system comprising: an image adjustment module and a pixel fusion module. The image adjustment module is configured to adjust the shape of a second product image based on preset key points of a person's hand in a first product image and an image mask of the first product, thereby obtaining a deformed image. The pixel fusion module is configured to perform image fusion based on the deformed image, the first product image, and the image mask, thereby obtaining a target product image. This improves the quality of image fusion, solves the problem of unnatural transitions between products and people in image fusion, and enhances the quality of the fused image.

[0031] Based on the above problems, this application provides the following implementation method.

[0032] Figure 1 This is a schematic diagram of an image fusion system provided in an embodiment of this application. The system includes: an image adjustment module 10 and a pixel fusion module 20.

[0033] The image adjustment module 10 is configured to adjust the shape of the second product image based on the preset key points of the person's hand in the first product image and the image mask of the first product, so as to obtain a deformed image.

[0034] In some examples, the distorted image includes an image obtained by adjusting the position of the outer contour of the second product image based on the outer contour of the first product image, and the distorted image and the second product image represent the same product.

[0035] For example, the first product image includes a first product and a preset person, wherein the preset person may be holding the first product. The first product image can be obtained from a user terminal, for example, by obtaining a photo uploaded by the user and using it as the first product image, or by obtaining a sales video uploaded by the user and extracting any frame from the sales video as the first product image. The preset person can be the seller, a third party selling the product, or a virtual person (digital human).

[0036] The first product can be a replacement product. By replacing the first product in the first product image with the second product, image utilization efficiency can be improved. When selling the second product, there is no need to repeatedly invest in shooting resources, thus reducing the cost for the seller. For example, the first product can be a mobile phone, and the second product can be a beverage. By replacing the mobile phone held by the preset person in the first product image with a beverage, the sale can be completed while reducing shooting costs.

[0037] It should be noted that all preset characters (including but not limited to real person portraits, virtual character images, digital human images, etc.) involved in this embodiment and the accompanying drawings have been authorized or permitted by the user.

[0038] For example, image segmentation refers to splitting an image into multiple non-overlapping, semantically independent sets of pixels according to its semantic categories (such as objects, backgrounds, region functions, etc.). In this embodiment, image segmentation can be performed using an image segmentation model. For instance, the prompt word and a first product image are input into the image segmentation model, which then segments the first product image based on the prompt word to obtain an image mask.

[0039] An image mask maps pixel-level classification results of an image, meaning each pixel uniquely corresponds to a label (such as background, product, hand, etc.), essentially assigning a classification label to each pixel in the image. An image mask can be a binary mask, which distinguishes only between the target and the background. Target pixels are labeled with 1 (or 255, white), and background pixels are labeled with 0 (black). For example, the target could be the first product and a preset person in a first product image. To segment out the first product and the preset person in the image, the white area in the mask represents the first product and the preset person, and the black area represents the background. In this embodiment, the image mask can be the image region corresponding to the first product.

[0040] The second product image includes an image of a second product that will be used for display or sale. For example, an initial second product image may be obtained by pre-shooting the second product against a preset background (e.g., a green screen background or other natural background), and the background of the initial second product image may be removed to obtain a second product image of the second product.

[0041] For example, the first product to be replaced in the first product image corresponding to the image mask is replaced with the second product in the second product image, and edge, lighting and shadow fusion is performed to generate the target product image.

[0042] The key points of a person's hand refer to the two-dimensional (2D) or three-dimensional (3D) coordinates of the bones, joints, and key parts of the hand obtained by extracting key points from the first product image using a model. These coordinates completely cover the core motion nodes of the palm and fingers, and are the basic data for reconstructing hand posture and judging gestures. For example, the key points of the hand can be obtained by performing key point recognition on the first product image using a gesture recognition model (such as MediaPipe Hands).

[0043] For example, key points of the hand may include: the base of the palm (wrist) numbered 0, which includes the center of the wrist joint; the thumbs numbered 1-4, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and thumb tip; the index fingers numbered 5-8, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and index finger tip; the middle fingers numbered 9-12, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and middle finger tip; the ring fingers numbered 13-16, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and ring finger tip; and the little fingers numbered 17-20, which include the metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint, and little finger tip.

[0044] These hand key points can reconstruct hand postures. For example, by changing the coordinates of 21 points, it is possible to capture hand movements such as bending, extending, and rotating (e.g., clenching a fist, extending fingers, and wrist rotation) in real time, accurately reconstructing the hand's posture in space. In addition, it can support gesture recognition. For example, based on the relative position, distance, and angle relationship of hand key points, specific gestures can be defined (e.g., "OK gesture" corresponds to the thumb and index finger tips being close together and other fingers being extended; "thumbs up" corresponds to the thumb being extended and other fingers being bent). Alternatively, it can enable human-computer interaction: the coordinates of key points can be directly used as control signals and applied to gesture control scenarios (e.g., controlling screen scrolling by sliding fingers across key points, switching functions by clenching fists / extending fingers through key points). Furthermore, it can provide 3D information: some models (e.g., the 3D mode of MediaPipe Hands) can output the depth coordinates (Z-axis) of key points, which can distinguish the overlapping of hands and their distance, adapting to more complex spatial interaction scenarios (e.g., virtual button clicks, 3D modeling gesture control).

[0045] In some examples, the image mask of the first product can be obtained as follows: perform image segmentation on the first product image to obtain a first image mask corresponding to the first product and the preset person in the first product image; perform image segmentation on the first product image based on the product prompt, the key points of the preset person's hand, and the first image mask to obtain a second image mask corresponding to the first product; and perform mask fusion on the first image mask and the second image mask to obtain the image mask.

[0046] For example, morphological adjustment of the second product image refers to geometrically deforming the second product image to adapt it to the hand of a preset person. The deformed image includes an image of the second product that matches the hand-holding posture of the preset person. Product prompts include text prompts or reference images, used to provide the image segmentation model with reference information such as the product's category and shape, assisting in completing partially overlapping areas.

[0047] Because the second product may differ significantly in size from the first product—for example, the first product is a mobile phone and the second product is a tablet—directly replacing the first product with the second would result in a mismatch between the replaced hand and the product due to the size difference, leading to issues such as the product appearing to float or the hand clipping through the image. Therefore, by morphologically adjusting the image of the second product, a deformed image is obtained that matches the hand's gripping posture of the preset character.

[0048] See Figure 2 It can be seen that, further, in some examples, the image adjustment module 10 includes: a control point extraction unit 101 and a pixel adjustment unit 102.

[0049] The control point extraction unit 101 is configured to extract control points from an image mask based on hand key points to obtain multiple target control points, and to extract control points from a second product image to obtain multiple source control points.

[0050] In some examples, target control points are used to characterize the outer contour of the first product image, and source control points are used to characterize the outer contour of the second product image.

[0051] See Figure 3 It can be seen that, further, in some examples, the control point extraction unit 101 includes: target control point extraction unit 101-1 and source control point extraction unit 101-2.

[0052] The target control point extraction unit 101-1 is configured to obtain target control points by performing contour calculations through a contour processing model based on hand key points and image masks.

[0053] See Figure 4 It can be seen that, further, in some examples, the target control point extraction unit 101-1 includes: contour point extraction subunit 101-1a, convex hull boundary point extraction subunit 101-1b, and boundary optimization subunit 101-1c.

[0054] The contour point extraction subunit 101-1a is configured to extract contour points from the image mask based on hand key points through the contour processing layer of the contour processing model.

[0055] The convex hull boundary point extraction subunit 101-1b is configured to perform convex hull calculation on the contour points through the convex hull processing layer of the contour processing model to obtain the convex hull boundary points.

[0056] In some examples, the convex hull boundary point extraction subunit is configured to: sort the contour points based on the coordinate values ​​of the contour point sequence to obtain the contour point sequence; traverse the contour point sequence based on a first traversal rule to obtain the first convex hull boundary point; traverse the contour point sequence based on a second traversal rule to obtain the second convex hull boundary point; and merge the first and second convex hull boundary points to obtain the convex hull boundary point.

[0057] The first traversal rule includes: determining the current contour point and the two preceding contour points in a left-to-right order; constructing a first point vector and a second point vector based on the current contour point and the two preceding contour points, and calculating the first product value of the first point vector and the second point vector; if the first product value is positive, using the current contour point and the two preceding contour points as the first convex hull boundary points, and continuing traversal until the entire contour point sequence has been traversed; if the first product value is negative, deleting the previous contour point of the current contour point, and continuing traversal until the entire contour point sequence has been traversed.

[0058] The second traversal rules include: determining the current contour point and the two preceding contour points in a right-to-left order; constructing the third and fourth point vectors based on the current contour point and the two preceding contour points, and calculating the second product of the third and fourth point vectors; if the second product is positive, using the current contour point and the two preceding contour points as the boundary points of the second convex hull, and continuing the traversal until the entire contour point sequence has been traversed; if the second product is negative, deleting the previous contour point of the current contour point, and continuing the traversal until the entire contour point sequence has been traversed.

[0059] The boundary optimization subunit 101-1c is configured to perform boundary optimization on the convex hull boundary points through the boundary optimization layer of the contour processing model to obtain the target control points.

[0060] The source control point extraction unit 101-2 is configured to determine the target feature detection model based on the commodity shape of the second commodity in the second commodity image, and to extract features from the second commodity image through the target feature detection model to obtain source control points.

[0061] For example, target control points can be obtained by extracting control points from an image mask based on hand key points using a first extraction model. The first extraction model can be a contour processing model. By inputting the hand key points and the image mask into the contour processing model for control point extraction, the target control points are obtained. The target control points can represent the contour of the first product. During the feature point extraction process of the contour processing model, by incorporating hand control points, the extracted target control points can conform to the contour of the first product.

[0062] The first extraction model may include a contour processing layer and a convex hull processing layer. The contour processing layer is used to extract the contour point sequence of the image mask corresponding to the first product. The input of the contour processing layer includes the image mask corresponding to the first product, and the output is the contour point sequence.

[0063] The processing steps for the contour processing layer are as follows:

[0064] First, input an image mask and randomly preset a pixel matrix. Set the pixel value of the first pixel in the pixel matrix to 1, and set at least one pixel with a value of 0 in its adjacent pixels as the starting point. Use the starting point as the starting detection point for the contour extraction process.

[0065] Secondly, starting from the starting point, detect the surrounding 8 pixels in a fixed manner. Pixels that have a value of 1 and at least one of their adjacent pixels with a value of 0 are taken as the next pixel of the contour. This process is iterated until the starting point is included in the contour again. At this point, the contour point sequence of the image mask can be obtained based on the coordinates of the obtained multiple pixels.

[0066] The convex hull processing layer is used to further calculate the convex hull boundary points based on the contour point sequence of the first product. The convex hull represents the smallest convex polygon that completely encloses the contour of the first product without any concavities; that is, the convex hull is the outermost contour of the first product, while ignoring some concave details, thus highlighting the overall geometric shape of the first product and providing a simpler data foundation for subsequent key points. The input to the convex hull processing layer is the contour point sequence, and the output is the convex hull boundary point sequence.

[0067] The processing steps for the convex hull layer are as follows:

[0068] First, sort the contour point sequence according to the x-coordinate (or y-coordinate if the x-coordinates are the same);

[0069] Secondly, construct the first convex hull: traverse the sorted contour points from left to right, and take the contour points that meet the requirements as the first convex hull points;

[0070] The above requirement means that, based on the currently detected contour point and the two previously adjacent contour points (a total of three points), two vectors are constructed, and the cross product of these two vectors is calculated. If it is positive, it means that the position of the three contour points is not concave, and the first two contour points can be directly used as the first convex hull point, and the last contour point is included in the next round of detection; if it is negative, it means that the position of the three contour points is concave, and the contour point in the middle of the three points is deleted, and the remaining two contour points are included in the next round of detection.

[0071] Next, construct the second convex hull: traverse the sorted contour points from right to left, and use the contour points that meet the requirements as the second convex hull points. The specific calculation is the same as for the first convex hull.

[0072] Finally, the first convex hull and the second convex hull are merged, and duplicate convex hull points are removed. The remaining first convex hull points and the second convex hull points together constitute the final sequence of convex hull boundary points.

[0073] In addition, the first extraction model may also include a contour simplification layer. The contour simplification layer simplifies the convex hull boundary point sequence to obtain a final target control point sequence that facilitates subsequent calculations. The input to the contour simplification layer is the aforementioned convex hull boundary point sequence, and the output is the final target control point sequence.

[0074] The processing steps for the contour simplification layer are as follows:

[0075] First, sort the sequence of convex hull boundary points, and connect the starting point A and the ending point B of the sequence with a straight line AB.

[0076] Next, calculate the perpendicular distance from other convex hull boundary points in the sequence to AB, and take the maximum value P1. Compare P1 with a preset tolerance value. If P1 is greater than the tolerance value, then the convex hull boundary point C corresponding to P1 is the core convex hull point. At this time, construct two straight lines AC and BC, and calculate the perpendicular distance between the convex hull boundary point between point A and point C and AC. Again, take the maximum value P2. Based on the comparison result of P2 and the tolerance value, if P2 is greater than the tolerance value, then the convex hull boundary point D corresponding to P2 is the core convex hull point, and then construct two straight lines AD and CD; otherwise, if P2 is less than the tolerance value, all convex hull boundary points in segment AC are redundant. The processing of segment BC is the same as above (it should be noted that the aforementioned P1 may also be less than the tolerance value, similar to the first product itself being an approximately straight object such as chopsticks or a pen). Iterate in this way until P in any segment is less than the tolerance value. At this time, all core convex hull points can be obtained, and this sequence of core convex hull points is the final target control point (i.e., target control point).

[0077] For example, control points can be extracted from the second product image using a second extraction model to obtain source control points. The second extraction model can be a feature detection model, such as the SuperPoint model or the MeshLab model. If the product is a regular object, the SuperPoint model can be used; if the product is an irregular object, the MeshLab model can be used.

[0078] The extraction rules followed by the second extraction model when extracting source control points can be determined according to the shape of the second product. For example, for rigid objects, the boundary corner points and edge midpoints are used as source control points (usually 8 to 12 points). For objects with handles, the source control points are sampled more densely in the handle area (usually 30 points) on the basis of the above. For flexible objects, the surface mesh vertices are used as source control points (usually 50 to 200 points).

[0079] It should be noted that if the first and second products are different types of products, such as the first product being a mobile phone and the second product being a water cup, then the extraction of source control points and target control points must be processed using different models (the first extraction model and the second extraction model) as described above. If the first and second products are of the same type of product, such as both being mobile phones, due to their similar structures, the same model can be used, such as the Siamese model, to extract source control points and target control points.

[0080] The pixel adjustment unit 102 is configured to perform morphological adjustment on the second commodity image based on multiple target control points and multiple source control points to obtain a deformed image.

[0081] In some examples, a deformed image can be obtained by adjusting the image based on a second product image, target control points, and source control points using an interpolation model (Thin Plate Spline, TPS). The input to the interpolation model is the target control points, source control points, and the second product image, and the output is the deformed image. For example, the deformed image is obtained by adjusting the second product image.

[0082] It should be noted that other methods such as triangular mesh deformation or spatial transformation network can also be used to adjust the image to obtain deformed images. The choice can be made according to the application. For example, affine transformation can be used directly for simple-shaped items such as books; STN model can be used for irregularly shaped products such as musical instruments and handicrafts.

[0083] Control point adjustment parameters include global pixel adjustment coefficients, local pixel adjustment parameters, and / or local pixel adjustment weights. The global pixel adjustment coefficients can be affine transformation coefficients of the source control point relative to the target control point, calculated based on the distance (or positional difference) between the source and target control points.

[0084] The local pixel adjustment parameter can be the intensity of the local influence of the source control point on other pixels, calculated based on the source control point and other pixels. This local influence intensity characterizes the intensity or amplitude of the pixel during the deformation process, which is determined by the distance between each pixel and a certain source control point.

[0085] Local pixel adjustment weights can be the local influence weights of source control points on other pixels. These local influence weights characterize the specific magnitude of deformation of each pixel caused by the simultaneous constraint process of multiple different source control points.

[0086] After determining the global pixel adjustment coefficient, local pixel adjustment parameters, and local pixel adjustment weights, the second product image is morphologically adjusted based on the global pixel adjustment coefficient, local pixel adjustment parameters, and local pixel adjustment weights to obtain a deformed image.

[0087] See Figure 5 It can be seen that, further, in some examples, the pixel adjustment unit 102 includes: a global pixel adjustment unit 102-1 and a local pixel adjustment unit 102-2.

[0088] The global pixel adjustment unit 102-1 is configured to calculate the global pixel adjustment coefficient based on the coordinate values ​​between the target control point and the source control point, and to perform global morphological adjustment on the second commodity image based on the pixel adjustment coefficient to obtain a global deformed image.

[0089] For example, based on the positional difference between the target control point and the source control point, the affine transformation coefficients of the target control point relative to the source control point can be calculated. The source control point is then moved based on these affine transformation coefficients to obtain a globally deformed image. These affine transformation coefficients may include translation, rotation, and scaling factors. It should be noted that in this process, other pixels in the second product image are only adjusted according to the aforementioned affine transformation coefficients and do not participate in the calculation.

[0090] The local pixel adjustment unit 102-2 is configured to calculate local pixel adjustment parameters based on the coordinate values ​​of the source control point and other pixels in the global deformed image, and to perform local shape adjustment on the second product image based on the local pixel adjustment parameters to obtain a deformed image.

[0091] For example, local pixel adjustment parameters include local pixel adjustment coefficients and local pixel adjustment weights.

[0092] In some examples, based on the distance between a pixel and each source control point in the global deformation image, a local pixel adjustment coefficient corresponding to each source control point is calculated. The local pixel adjustment coefficient is used to characterize the deformation amplitude of the pixel corresponding to source control points at different distances. Based on the local pixel adjustment coefficient, the coordinate values ​​of the source control points, and the coordinate values ​​of the target control points, a local pixel adjustment weight corresponding to each source control point in the global deformation image is calculated. The local pixel adjustment weight is used to characterize the constraint relationship between the pixel and source control points at different distances.

[0093] Based on the global deformation image, the local influence intensity of each source control point on other pixels (including source control points and non-source control points) in the global deformation image is calculated according to the distance between the moved source control points in the global deformation image and other pixels in the global deformation image. The local influence intensity characterizes the intensity or magnitude of the deformation process of each pixel, determined by the distance between each pixel and a certain source control point.

[0094] Based on the calculated local influence intensity, and using the source control point and target control point, the local influence weight of each source control point on other pixels is calculated. The local influence weight characterizes the specific magnitude of deformation caused by the simultaneous constraint process of multiple different source control points on each pixel.

[0095] For example, the local influence weight can be calculated using the following formula:

[0096] Source control point coordinates + local influence intensity Local influence weight = coordinates of the target control point

[0097] It should be noted that if a global deformation image is directly used as the output, on the one hand, it cannot achieve a precise match between the source control points and the target control points, which may result in a certain deviation. On the other hand, since non-source control points in the image often cannot be directly adjusted through linear transformation, distortion and other distortion phenomena will inevitably occur. This embodiment uses a combination of global deformation and local deformation to process the image, making the obtained deformation image more realistic.

[0098] For example, the second adjustment parameter (i.e., local influence intensity) and the second adjustment weight (i.e., local influence weight) together constitute the local deformation parameter of a certain pixel in the global deformation image (i.e., global deformation image). Each pixel in the global deformation image is adjusted according to the local deformation parameter to obtain the output deformation image.

[0099] The pixel fusion module 20 is configured to perform image fusion based on the deformed image, the first product image, and the image mask to obtain the target product image.

[0100] For example, image edge blending includes performing Poisson blending on the image edges. The first blended image may be an image of a pre-defined person holding a second product, with the edges of the second product and the hand smoothly transitioning.

[0101] See Figure 6 Furthermore, in some examples, the pixel blending module 20 includes an edge blending unit 201, a lighting blending unit 202, and a shadow blending unit 203.

[0102] The edge blending unit 201 is configured to perform image edge blending between the deformed image and the first product image based on the image mask to obtain a first blended image.

[0103] In some examples, the edge blending unit is configured to: determine each region to be blended contained in the image mask; and perform image edge blending on the deformed image and the first product image according to the blending strategy corresponding to each region to be blended, to obtain the first blended image.

[0104] For example, each region to be merged includes an occluded edge region, an occluded inner region, a pressed edge region, a pressed inner region, and / or a partially overlapping region. The merging strategy includes: edge protection processing for the occluded edge region, transparency processing for the pressed edge region, pixel preservation processing for the occluded inner region and the pressed inner region, and pixel replacement processing for the partially overlapping region.

[0105] In some examples, the system further includes a state recognition module, a region segmentation module, and a mask fusion module; the state recognition module is configured to perform state recognition on the hand of a preset person in the first image mask to obtain the hand state; the first image mask is obtained based on image segmentation of a first product image; the first product and the hand of the preset person in the first image mask at least partially overlap; the region segmentation module is configured to segment the first image mask according to the hand state to obtain multiple hand regions; the hand regions include occlusion edge regions, occlusion inner regions, pressing edge regions, and / or pressing inner regions; the mask fusion module is configured to perform mask fusion on the first image mask and the second image mask based on the multiple hand regions and the partially overlapping regions to obtain an image mask of the first product; the second image mask is obtained based on product hints, key points of the preset person's hand, and the first image mask through image segmentation; the product hints are used to represent the reference type of the product; the second image mask includes a product mask obtained by completing the partially overlapping regions according to the product hints.

[0106] In some examples, the state recognition module is configured to: detect the edge distance between the hand of a preset person in the first image mask and the edge of the first product; determine that the hand is in an occluded state if the edge distance is greater than or equal to a preset threshold; and / or determine that the hand is in a pressing state if the edge distance is less than the preset threshold.

[0107] In some examples, the region segmentation module is configured to: for a hand in an occluded state, use the outer contour of the hand as the occlusion edge area and the inside of the outer contour of the hand as the occlusion inner area; or for a hand in a pressing state, use the outer contour of the hand as the pressing edge area and the inside of the outer contour of the hand as the pressing inner area.

[0108] The region marking can be obtained in the following way: based on the hand state of the preset person in the image mask, the hand of the preset person in the first image mask is divided into regions to obtain multiple first hand regions, and the completion region of the first product in the second image mask is determined; the multiple first hand regions and the completion region are marked respectively to obtain each region marking.

[0109] For example, during the process of image edge fusion of the deformed image and the first product image based on the image mask, the image area to be replaced in the first product image can be determined based on the coordinates of each pixel in the image mask, and the image of the second product can be replaced within the image area.

[0110] In the image replacement process, a Poisson fusion model is used to process different regions to ensure a smooth transition between the second product and other areas. The purpose of Poisson fusion is to adjust the gradient of the image (the gradient refers to the rate of change of pixel values ​​at a point in the image along the x and y axes, specifically reflecting the degree of abrupt changes in brightness, color, and contour of the corresponding area in the image) under preset constraints, so that the gradient changes between different parts of the image are continuous and smooth, resulting in a better edge connection between the second product and the preset person's finger.

[0111] Since the aforementioned image mask has already used different markers to distinguish different interactions between the finger and the product during the generation process, the deformed image can be subjected to gradient processing based on these different markers during the fusion process, with the following corresponding rules:

[0112] When the first marker corresponds to the occlusion edge area, the second marker corresponds to the pressing edge area, the third marker corresponds to the occlusion interior area and / or the pressing interior area, and the fourth marker corresponds to the partially overlapping area, for the fourth marker (corresponding to the product body, i.e., the partially overlapping area), it is necessary to ensure that the internal gradient information is completely consistent with the deformed image, and only changes occur in the area near the edges of the first and second markers. For the first marker (corresponding to the edge of the occluded finger, i.e., the occlusion edge area), it is necessary to strictly prevent other products from intruding into the finger area. Therefore, in actual processing, the gradient information of the finger edge at the position corresponding to the first marker is used as a constraint condition, and the gradient information from the corresponding area of ​​other products to the finger edge is smoothly transitioned through the calculation of the Poisson fusion model.

[0113] For the second marker (corresponding to the edge of the pressing finger, i.e., the pressing edge area), this part needs to comprehensively consider the allocation of the product and the finger. In this invention, gradient weight allocation calculation is introduced for this area, as follows:

[0114] The second press buffer area is formed by spreading a certain distance (usually 1 to 5 pixels) to both sides of the edge of the corresponding area of ​​other products. The second press buffer area overlaps with the first press buffer area at least partially. Thus, the first press buffer area and the second press buffer area together constitute three areas: the product area completely located in other products, the finger area completely located in the finger, and the overlapping area that overlaps with each other.

[0115] The gradient weights of the finger transparency and / or the second product are adjusted within the aforementioned range. Specifically, in the overlapping area, the gradient weight of the second product and the transparency of the finger gradually decrease as the finger extends towards the second product. This is reflected in the following ways: Closer to the second product, the higher the finger transparency, indicating that the finger becomes more transparent closer to the edge, resulting in lower finger realism. Simultaneously, the gradient weight of the second product is higher, indicating an increased visual proportion of the second product and better presentation of product details. Conversely, closer to the finger, the lower the finger transparency, indicating that the finger becomes more realistic closer to the inside, resulting in higher finger realism. Simultaneously, the gradient weight of the second product gradually decreases, indicating a reduced visual proportion of the second product, ensuring that the pressure sensation of the finger is preserved. The gradient weight of the second product in the product area and the finger transparency in the finger area still follow the above principle, i.e., increasing from the inside (finger) to the outside (product) and decreasing from the outside to the inside.

[0116] In actual processing, the gradient of the finger can be calculated from the original display image using preset gradient operators (such as the Sobel operator and the Prewitt operator), the gradient of the second product can be calculated from the distorted image, and different markers can be obtained from the image mask to complete the above calculations. Through these gradient changes, the harsh edges between the second product and the hand and background can be eliminated, making the pixel gradient of the second product and the gradient of the hand edge natural and continuous, thus solving the problem of the splicing effect in the contact area between the finger and the product.

[0117] The illumination fusion unit 202 is configured to perform image background fusion based on the first fused image and the background area of ​​the person in the first product image excluding the first product, to obtain a second fused image.

[0118] In some examples, the illumination fusion unit is configured to perform image background fusion based on the first fused image and the background area of ​​the person in the first product image, excluding the first product, to obtain a second fused image.

[0119] In some examples, lighting models (such as the Retinex model, CycleGAN, etc.) can be used to ensure that the lighting effect of the second product matches that of the hand, based on the first blended image. The input to the lighting model includes the first blended image mentioned above, and the image of the person from the first product image after removing the background image of the first product. The output is the second blended image, which is an image in which the lighting style of the second product matches that of the hand.

[0120] The shadow blending unit 203 is configured to perform image background blending based on the background areas of the second blended image and the first product image, excluding the first product and the preset person, to obtain the target product image.

[0121] In some examples, the shadow blending unit is configured to perform image background blending based on the background areas of the second blended image and the first product image, excluding the first product and the preset person, to obtain the target product image.

[0122] In some examples, a shadow model (such as the CycleGAN model) can be used to ensure that the shadow effect of the second product matches the background, based on the second fused image. The input to the shadow model includes the aforementioned second fused image, and a pure background image from which the first product image is stripped of the first product and a preset figure (including hands) (to prevent shadows from overlapping the hands). The output is the final target display image (i.e., the target product image), in which the shadow brightness is uniform and the transition is natural.

[0123] It should be noted that the above embodiments of this application do not constitute a limitation on the scope of protection of this application. Those skilled in the art can combine, separate, or reorganize the embodiments provided in this application to obtain other embodiments, none of which exceed the scope of protection of this application.

[0124] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. An image fusion system, characterized in that, The system includes an image adjustment module and a pixel fusion module; The image adjustment module is configured to perform shape adjustment on the second product image based on the preset key points of a person's hand in the first product image and the image mask of the first product, to obtain a deformed image; wherein, the deformed image includes an image obtained by adjusting the position of the outer contour of the second product image according to the outer contour of the first product in the first product image, and the deformed image and the second product image represent the same product; The pixel fusion module is configured to perform image fusion based on the deformed image, the first product image, and the image mask to obtain the target product image; The image adjustment module includes: The control point extraction unit is configured to extract control points from the image mask based on the hand key points to obtain multiple target control points, and to extract control points from the second product image to obtain multiple source control points; wherein the target control points are used to characterize the outer contour of the first product, and the source control points are used to characterize the outer contour of the second product image. The pixel adjustment unit is configured to perform morphological adjustment on the second product image based on the plurality of target control points and the plurality of source control points to obtain the deformed image.

2. The image fusion system according to claim 1, characterized in that, The control point extraction unit includes: The target control point extraction unit is configured to perform contour calculation based on the hand key points and the image mask, using a contour processing model, to obtain the target control points. The source control point extraction unit is configured to determine a target feature detection model based on the product shape of the second product in the second product image, and to extract features from the second product image through the target feature detection model to obtain the source control points.

3. The image fusion system according to claim 2, characterized in that, The target control point extraction unit includes: The contour point extraction subunit is configured to extract contour points from the image mask based on the hand key points through the contour processing layer of the contour processing model to obtain contour points. The convex hull boundary point extraction subunit is configured to perform convex hull calculation on the contour points through the convex hull processing layer of the contour processing model to obtain the convex hull boundary points. The boundary optimization subunit is configured to perform boundary optimization on the convex hull boundary points through the boundary optimization layer of the contour processing model to obtain the target control point.

4. The image fusion system according to claim 3, characterized in that, The convex hull boundary point extraction subunit is configured as follows: The contour points are sorted based on their coordinate values ​​to obtain the contour point sequence. The contour point sequence is traversed based on the first traversal rule to obtain the first convex hull boundary point; The contour point sequence is traversed based on the second traversal rule to obtain the second convex hull boundary points; The first convex hull boundary point and the second convex hull boundary point are merged to obtain the convex hull boundary point.

5. The image fusion system according to claim 4, characterized in that, The first traversal rule includes: Determine the current contour point and the two previous contour points in order from left to right; Based on the current contour point and the previous two contour points, construct a first point vector and a second point vector, and calculate the first product value of the first point vector and the second point vector; If the first product value is positive, the current contour point and the previous two contour points are taken as the first convex hull boundary points, and the traversal continues until the entire sequence of contour points has been traversed. If the first product value is negative, the previous contour point of the current contour point is deleted, and the traversal continues until the entire contour point sequence has been traversed.

6. The image fusion system according to claim 4, characterized in that, The second traversal rule includes: Determine the current contour point and the two previous contour points in order from right to left; Based on the current contour point and the first two contour points, construct a third point vector and a fourth point vector, and calculate the second product value of the third point vector and the fourth point vector; If the second product value is positive, the current contour point and the previous two contour points are taken as the second convex hull boundary points, and the traversal continues until the entire sequence of contour points has been traversed. If the second product value is negative, the previous contour point of the current contour point is deleted, and the traversal continues until the entire contour point sequence has been traversed.

7. The image fusion system according to claim 1, characterized in that, The pixel adjustment unit includes: The global pixel adjustment unit is configured to calculate a global pixel adjustment coefficient based on the coordinate values ​​between the target control point and the source control point, and to perform global shape adjustment on the second product image based on the pixel adjustment coefficient to obtain a global deformation image; The local pixel adjustment unit is configured to calculate local pixel adjustment parameters based on the coordinate values ​​of the source control point and other pixels in the global deformed image, and to perform local shape adjustment on the second product image based on the local pixel adjustment parameters to obtain the deformed image; the local pixel adjustment parameters include local pixel adjustment coefficients and local pixel adjustment weights.

8. The image fusion system according to claim 7, characterized in that, The local pixel adjustment unit is configured as follows: Based on the distance between the pixel and each source control point in the global deformation image, the local pixel adjustment coefficient corresponding to each source control point is calculated, wherein the local pixel adjustment coefficient is used to characterize the deformation amplitude of the pixel corresponding to the source control points at different distances; Based on the local pixel adjustment coefficient, the coordinates of the source control point, and the coordinates of the target control point, the local pixel adjustment weight corresponding to each source control point in the global deformation image is calculated; wherein, the local pixel adjustment weight is used to characterize the constraint relationship between the pixel and the source control points at different distances; The second product image is locally shaped according to the local pixel adjustment coefficient and the local pixel adjustment weight to obtain the deformed image.

9. The image fusion system according to claim 1, characterized in that, The system also includes a state recognition module, a region division module, and a mask fusion module; The state recognition module is configured to perform state recognition on the hand of the preset person in the first image mask to obtain the hand state; the first image mask is obtained based on image segmentation of the first product image; the first product and the hand of the preset person in the first image mask at least partially overlap. The region segmentation module is configured to segment the first image mask into regions based on the hand state to obtain multiple hand regions; the hand regions include occlusion edge regions, occlusion inner regions, pressing edge regions, and / or pressing inner regions; The mask fusion module is configured to perform mask fusion on the first image mask and the second image mask based on the plurality of hand regions and the partially overlapping regions to obtain the image mask of the first product. The second image mask is obtained by image segmentation based on the product prompt, the key points of the hand of the preset person, and the first image mask; the product prompt is used to represent the reference type of the product; The second image mask includes a product mask obtained by completing the partially overlapping area according to the product prompt.

10. The image fusion system according to claim 9, characterized in that, The status recognition module is configured as follows: The distance between the hand of the preset person in the first image mask and the edge of the first product is detected. If the edge distance is greater than or equal to a preset threshold, the hand is determined to be occluded. And / or, if the edge spacing is less than a preset threshold, the hand is determined to be in a pressing state.

11. The image fusion system according to claim 9, characterized in that, The region division module is configured as follows: For a hand in a covered state, the outer contour of the hand is defined as the covered edge area and the inside of the outer contour of the hand is defined as the covered inner area; or, for a hand in a pressed state, the outer contour of the hand is defined as the pressed edge area and the inside of the outer contour of the hand is defined as the pressed inner area.

12. The image fusion system according to claim 1, characterized in that, The pixel fusion module includes: An edge blending unit is configured to perform image edge blending between the deformed image and the first product image based on the image mask to obtain a first blended image; The illumination fusion unit is configured to perform image background fusion based on the first fused image and the background area of ​​the person in the first product image, excluding the first product, to obtain a second fused image; The shadow blending unit is configured to perform image background blending based on the background areas of the second blended image and the first product image, excluding the first product and the preset person, to obtain the target product image.

13. The image fusion system according to claim 12, characterized in that, The edge fusion unit is configured as follows: Determine each region to be merged contained in the image mask; each region to be merged includes an occlusion edge region, an occlusion interior region, a pressed edge region, a pressed interior region, and / or a partially overlapping region; Based on the fusion strategy corresponding to each region to be fused, image edge fusion is performed on the deformed image and the first product image to obtain the first fused image; The fusion strategy includes: performing edge protection processing on the occluded edge area, making the pressed edge area transparent, performing pixel retention processing on the occluded inner area and the pressed inner area, and performing pixel replacement processing on the partially overlapping area.

Images