Garment image segmentation method and device based on human skeleton joint points and storage medium

Abstract

The invention discloses a clothing image segmentation method and equipment based on human skeleton joint points, and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: respectively inputting a target image into a pre-trained human body posture estimation model and a semantic segmentation model, obtaining a skeleton joint graph of an image character in the target image output by the human body posture estimation model, and obtaining a segmentation mask of a garment worn by the image character output by the semantic segmentation model; determining the body shape characteristics of the image figure according to the skeleton joint point graph; taking a connecting line of left and right hip joint points in the skeleton joint point graph as a segmentation reference line, and adjusting the segmentation reference line according to an adjustment rule corresponding to the body shape characteristics to obtain a cutting path; and cutting the segmentation mask according to the cutting path so as to segment the wearing clothes, thereby obtaining the upper and lower parts of the one-piece dress conforming to the human body structure, and improving the segmentation visualization effect of the virtual character integrated clothes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a garment image segmentation method based on human skeleton joints, a device and a storage medium. BACKGROUND

[0002] In the current field of electronic commerce, fashion design, virtual reality, etc., virtual fitting technology has become the key to improving user experience. Users often expect to intuitively view the matching effect of different lower garment designs, such as skirt styles, while keeping the upper garment design unchanged. However, when the image character is wearing a one-piece garment such as a dress, the current semantic segmentation model usually segments the "dress" as a whole category, cannot divide the upper body and lower body masks, and forcibly cutting the whole segmentation result will lead to unnatural segmentation boundaries and poor overall effect.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a garment image segmentation method based on human skeleton joints, a device and a storage medium, aiming to solve the technical problem of how to improve the segmentation visualization effect of virtual characters wearing one-piece garments.

[0005] To achieve the above purpose, the present application provides a garment image segmentation method based on human skeleton joints, which comprises: inputting a target image into a pre-trained human pose estimation model and a semantic segmentation model respectively, obtaining a skeleton joint map of an image character in the target image output by the human pose estimation model, and obtaining a segmentation mask of a worn garment of the image character output by the semantic segmentation model; determining the body shape feature of the image character according to the skeleton joint map; taking the left and right hip joint connection line in the skeleton joint map as a segmentation reference line, and adjusting the segmentation reference line to obtain a cutting path according to the adjustment rule corresponding to the body shape feature; cutting the segmentation mask according to the cutting path to segment the worn garment.

[0006] In an embodiment, the step of determining the body shape feature of the image character according to the skeleton joint map comprises: based on a preset body shape feature node pair, according to the proportion value between the distances of the body shape feature node pair in the skeleton joint map, the body shape feature node pair at least includes a left and right shoulder joint pair and a left and right hip joint pair; taking the proportion value between the distances of the body shape feature node pair as the body shape feature.

[0007] In an embodiment, the step of adjusting the segmentation reference line according to the adjustment rule corresponding to the body feature to obtain a cutting path comprises: taking a ratio value between the distances of the body feature node pair as an input parameter of a preset adjustment formula, and calculating a vertical offset of the segmentation reference line corresponding to the ratio value through the preset adjustment formula; offsetting the segmentation reference line based on the vertical offset to obtain the cutting path.

[0008] In an embodiment, the step of adjusting the segmentation reference line according to the adjustment rule corresponding to the body feature to obtain a cutting path comprises: determining a body type of the image character based on a ratio value between the distances of the body feature node pair; determining a vertical offset of the segmentation reference line corresponding to the body type in a preset adjustment mapping table based on the body type; offsetting the segmentation reference line based on the vertical offset to obtain the cutting path.

[0009] In an embodiment, after the step of offsetting the segmentation reference line based on the vertical offset, the method further comprises: taking a ratio value between the distances of the body feature node pair as an input of a preset curvature calculation formula, and determining a curvature of the segmentation reference line according to the preset curvature calculation formula; fitting the offset segmentation reference line into a curve according to the curvature to obtain the cutting path.

[0010] In an embodiment, after the step of offsetting the segmentation reference line based on the vertical offset, the method further comprises: obtaining garment boundary texture feature points of the worn garment; determining distances of each garment boundary texture feature point to the offset segmentation reference line; determining alignment degrees of each garment boundary texture feature point according to products of type weights corresponding to the garment boundary texture feature points and the distances, wherein the distance is inversely proportional to the alignment degree; if the alignment degree of the garment boundary texture feature point is greater than a preset threshold, taking the garment boundary texture feature point as a target fitting point, and fitting a curve according to the target fitting point as the cutting path.

[0011] In an embodiment, after the step of fitting a curve according to the target fitting point as the cutting path, the method further comprises: displaying the garment boundary texture feature points and the cutting path on the segmentation mask; In response to a selection operation on the garment boundary texture feature points on the segmentation mask, re-fitting a curve according to the selected garment boundary texture feature points as a new cutting path.

[0012] In an embodiment, after the step of segmenting the worn garment by cutting the segmentation mask according to the cutting path, the method further comprises: In response to a selection operation on the segmented sub-masks after cutting, determining and marking a replacement region in the target image based on the pixel coordinate boundary of the selected segmented sub-mask; obtaining a replacement garment picture, determining a scaling ratio of the replacement garment picture based on the spatial position of the joint corresponding to the replacement region in the joint graph; After scaling the replacement garment picture according to the scaling ratio, inputting the replacement garment picture and the marked target image into a pre-trained image inpainting model to obtain a refitted image output by the image inpainting model.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a garment image segmentation device based on human body joint, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the garment image segmentation method based on human body joint as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the garment image segmentation method based on human body joint as described above.

[0015] The present application provides a garment image segmentation method based on human body joint, which comprises the following steps: inputting a target image into a pre-trained human pose estimation model and a semantic segmentation model respectively, obtaining a joint graph of an image character in the target image output by the human pose estimation model, and obtaining a segmentation mask of a worn garment of the image character output by the semantic segmentation model; determining a body shape feature of the image character according to the joint graph; taking the left and right hip joint connection line in the joint graph as a segmentation reference line, and adjusting the segmentation reference line to obtain a cutting path according to the adjustment rule corresponding to the body shape feature; cutting the segmentation mask according to the cutting path to segment the worn garment, so as to obtain the upper and lower parts of a dress that conforms to the human body structure, and improve the segmentation and visualization effect of the virtual character one-piece garment. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or the prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings from these drawings without creative effort.

[0018] Figure 1 A flowchart provided by the first embodiment of the garment image segmentation method based on human skeleton joints of the present application; Figure 2 A flowchart provided by the second embodiment of the garment image segmentation method based on human skeleton joints of the present application; Figure 3 A flowchart provided by the third embodiment of the garment image segmentation method based on human skeleton joints of the present application; Figure 4 A flowchart provided by the fourth embodiment of the garment image segmentation method based on human skeleton joints of the present application; Figure 5 A flowchart provided by the fifth embodiment of the garment image segmentation method based on human skeleton joints of the present application; Figure 6 A device structure diagram of a hardware running environment related to the garment image segmentation method based on human skeleton joints in the embodiments of the present application.

[0019] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0021] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings in the specification and specific embodiments. It should be noted that all the actions of obtaining signals, information or data in the present application are carried out under the premise of complying with the corresponding data protection regulations and policies of the place, and with the authorization of the corresponding device owner.

[0022] In the current field of e-commerce, fashion design, virtual reality, etc., virtual fitting technology has become a key to improving user experience. Users often expect to intuitively view the matching effect of different lower garment designs such as A-line styles while keeping the upper garment design unchanged. However, when the image character is wearing a one-piece dress, the current semantic segmentation model usually segments the "dress" as a whole category, cannot divide the mask of the upper body and the lower body, and forcibly cutting the whole segmentation result will lead to unnatural segmentation boundary and poor overall effect.

[0023] In view of the above problems, the present application provides a garment image segmentation method based on human skeleton joints. The target image is input into a pre-trained human pose estimation model and a semantic segmentation model respectively, the skeleton joint map of the image character in the target image output by the human pose estimation model is obtained, and the segmentation mask of the worn garment of the image character output by the semantic segmentation model is obtained. The body shape feature of the image character is determined according to the skeleton joint map. The left and right hip joint connecting lines in the skeleton joint map are taken as the segmentation reference line, and the segmentation reference line is adjusted to obtain the cutting path according to the adjustment rule corresponding to the body shape feature. The segmentation mask is cut according to the cutting path to segment the worn garment, so as to obtain the upper and lower parts of the dress conforming to the human body structure, and improve the segmentation visualization effect of the one-piece dress of the virtual character.

[0024] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a virtual garment fitting system, etc. The present embodiment and the following embodiments will be described below taking the virtual garment fitting system as an example.

[0025] Based on this, the first embodiment of the present application provides a garment image segmentation method based on human skeleton joints, which is described in detail as follows. Figure 1 In the present embodiment, the garment image segmentation method based on human skeleton joints includes steps S10-S40: Step S10, the target image is input into a pre-trained human pose estimation model and a semantic segmentation model respectively, the skeleton joint map of the image character in the target image output by the human pose estimation model is obtained, and the segmentation mask of the worn garment of the image character output by the semantic segmentation model is obtained.

[0026] The skeleton joint refers to the coordinate point of the key part of the character in the image, which can reflect the position boundary of the human body posture and each part of the body; the segmentation mask is the image data generated by the semantic segmentation model, which outlines the contour of the worn garment with a specific identifier such as a preset pixel value, and can realize the division of the garment area and the background area.

[0027] For example, a virtual clothing try-on system receives a target image containing a target person and their clothing, and calls a trained human pose estimation model such as OpenPose, AlphaPose, or YOLOv8 Pose to analyze the target image. The human pose estimation model identifies features of key parts of the person's body in the target image and outputs the specific coordinate data of each skeletal joint, forming a complete set of human skeletal joints. Simultaneously, a trained semantic segmentation model such as SegNet, U-Net, or DeepLab series models is called to perform semantic classification processing on the target image. The semantic segmentation model divides the pixels in the image into different categories such as clothing, human skin, and background, filters out the pixel regions corresponding to the clothing category, and generates segmentation mask data that marks the outline of the clothing with specific pixel values. The initial state of this segmentation mask data is the overall outline mask of a one-piece garment.

[0028] Step S20: Determine the body shape characteristics of the person in the image based on the skeletal joint map.

[0029] Optionally, step S20 includes steps S21 to S22: Step S21: Based on preset body feature node pairs, determine the ratio between the spacing of at least two sets of body feature node pairs in the skeletal joint map. The body feature node pairs include at least left and right shoulder joint pairs and left and right hip joint pairs.

[0030] Step S22: The ratio between the spacing of the body shape feature node pairs is taken as the body shape feature.

[0031] Preferably, the body shape features include the ratio between any two or three pairs of body shape feature nodes, such as the shoulder-hip ratio, waist-hip ratio, shoulder-waist ratio, or shoulder-waist-hip ratio mentioned above.

[0032] Optionally, in other embodiments, the body shape feature node pairs may also include left and right hip feature joint pairs, upper arm mid-section node pairs, lower rib edge node pairs, anterior superior iliac spine node pairs, etc.; based on at least two of the above body shape feature nodes, after arbitrary combination, the ratio between the spacing of each group of body shape feature node pairs after combination is calculated as the body shape feature characterizing the human torso morphology.

[0033] For example, the virtual clothing try-on system extracts joint pairs related to torso shape from the skeletal joint map, such as left and right shoulder joint pairs, left and right hip joint pairs, and left and right waist feature joint pairs. It calculates body shape feature parameters in the following ways: shoulder width is calculated by the distance between the left and right shoulder joints; hip width is calculated by the distance between the left and right hip joints, and the ratio of shoulder width to hip width is taken to obtain the shoulder-hip ratio; waist width is calculated by the distance between the left and right waist feature joints, and the ratio of waist width to hip width is taken to obtain the waist-hip ratio; the ratio of shoulder width to hip width is taken to obtain the shoulder-hip ratio; and torso posture parameters are obtained by the slope of the line connecting the coordinates of the shoulder joints, waist feature joints, and hip joints. These parameters are then integrated to form a set of body shape features describing the human body shape in the image.

[0034] In some specific embodiments, in order to save computing resources and improve response speed, a simpler human pose estimation model is used to identify skeletal joints. Such models may not be able to identify lumbar feature joints very well.

[0035] Based on this, the distance between the lines connecting the left and right shoulder joints can be taken as the shoulder width; the midpoint of the line connecting the left and right shoulder joints and the midpoint of the line connecting the left and right hip joints can be taken as the trunk midline; based on a preset proportional parameter, the center point of the waist is determined on the trunk midline, and a horizontal line is drawn based on the center point of the waist, intersecting with the segmentation mask, and the distance between the intersection points is taken as the waist width. The preset proportional parameter is the ratio of the length of the line segment from the midpoint of the shoulder joint to the center point of the waist on the trunk midline to the total length of the entire trunk midline. This ratio is a pre-set proportional threshold based on the conventional anatomical structure proportions of the human trunk, used to locate the center point of the waist on the trunk midline. Using this preset ratio, the lateral reference position of the waist, i.e., the center point of the waist, can be accurately located on the trunk midline even in the absence of a waist joint, providing a basis for subsequent waist width calculations.

[0036] Step S30: Use the line connecting the left and right hip joints in the skeletal joint diagram as the segmentation baseline, and adjust the segmentation baseline according to the adjustment rules corresponding to the body shape characteristics to obtain the clipping path.

[0037] Understandably, when human pose estimation models identify skeletal joints, they classify and label these joints. Virtual clothing try-on systems can directly locate the left and right hip joints using these category labels. A straight line connects the two points to form a dividing baseline.

[0038] The left and right hip joints are key dividing points between the upper and lower body skeletons. The connection between the torso and lower limbs corresponds precisely to the natural dividing line of the upper and lower parts of a one-piece garment. In different postures, such as standing, slightly turned to the side, or slightly bent over, the relative positions of the left and right hip joints remain relatively stable, and the line connecting them always accurately corresponds to the dividing area between the upper and lower body of the garment. Therefore, the line connecting the left and right hip joints can be used as a dividing baseline for segmenting the clothing worn by a person in an image. By adjusting this baseline based on the person's body shape, a cropping path that better conforms to human visual perception can be obtained.

[0039] Alternatively, as a first feasible implementation, the aforementioned segmentation baseline can be directly used as the cutting path. This method only requires selecting two skeletal joints and connecting them; the calculation process is simple, requiring no complex coordinate fitting algorithms or multi-dimensional feature analysis, resulting in low computational consumption for the virtual clothing try-on system. This concise logical design enables it to quickly respond to processing needs, adapting to scenarios with real-time requirements, such as virtual try-on where users can switch bottom styles online in real time, thus improving the user experience.

[0040] Alternatively, as a second feasible implementation, step S30 includes steps S31-S32: Step S31: The ratio between the spacing of the body feature node pairs is used as the input parameter of the preset adjustment formula, and the vertical offset of the segmentation baseline corresponding to the ratio is calculated by the preset adjustment formula.

[0041] The vertical offset refers to the distance the dividing baseline needs to be adjusted in the vertical direction, calculated by a preset adjustment formula based on a proportional value. A positive value indicates that the baseline moves upward, and a negative value indicates that it moves downward.

[0042] The virtual clothing try-on system obtains the proportional values ​​between the spacing of body shape feature node pairs and uses these proportional values ​​as input parameters, substituting them into a preset adjustment formula. The preset adjustment formula is a predefined mathematical expression used to map the proportional values ​​to the vertical offset of the segmentation baseline. This preset adjustment formula is typically designed based on human anatomy; for example, it can be expressed as: Vertical Offset = k × (Proportional Value - Baseline Proportion), where k is a scaling factor, and the baseline proportion is a preset reference ratio for a standard body shape, such as the average shoulder-hip ratio. The virtual clothing try-on system processes the proportional values ​​through mathematical calculations to obtain the specific vertical offset value. The specific form of the preset adjustment formula may vary depending on the system design, but its core logic is that when the proportional value deviates from the baseline proportion, the offset changes accordingly to adapt to different body shapes.

[0043] Step S32: Offset the segmentation baseline based on the vertical offset to obtain the clipping path.

[0044] After obtaining the vertical offset, the virtual clothing try-on system performs a vertical offset on the segmentation baseline. Since the segmentation baseline is a straight line, the virtual clothing try-on system can adjust the vertical coordinate values ​​of the left and right hip joint points, adding these vertical coordinate values ​​to the vertical offset to obtain new coordinate points. Then, connecting these two new coordinate points yields the cutting path.

[0045] Alternatively, as a third feasible implementation, step S30 further includes steps S33 to S35: Step S33: Determine the body type of the person in the image based on the ratio between the spacing of the body feature node pairs.

[0046] Optionally, a set of body shape features, including the ratio values ​​between the spacing of different body shape feature node pairs, is input into a pre-trained body shape recognition model, and the body shape recognition model is used to infer the body shape type of the person in the image.

[0047] For example, first, collect human images and corresponding skeletal joint point data samples of different body types, define standardized body type labels such as hourglass, pear, apple, and rectangle, and determine the proportional range of the spacing between different body feature node pairs corresponding to each body type. Professional annotators or semi-automated tools then label each data sample with the corresponding body type label. The labeled data samples are randomly divided into training, validation, and test sets according to a preset ratio, and the body type recognition model is trained and optimized accordingly to obtain a pre-trained body type recognition model. Then, the aforementioned feature set acquired in real-time during the application is input to the input layer of the body type recognition model, initiating the forward propagation calculation. The body type recognition model outputs a vector containing the probability values ​​corresponding to each body type; a higher probability value indicates a greater likelihood that the person in the image belongs to that body type.

[0048] Step S34: Based on the body type, determine the vertical offset of the segmentation baseline corresponding to the body type in a preset adjustment mapping table.

[0049] Step S35: Offset the segmentation baseline based on the vertical offset to obtain the clipping path.

[0050] Understandably, in certain application scenarios, virtual clothing try-on systems allow users to generate virtual avatars to try on clothing based on their own image. For different body types, the different waistline cutting positions can create different visual effects. For example, the core characteristics of an hourglass figure are a close shoulder-to-hip ratio and a noticeably constricted waist, naturally resulting in a waistline higher than the hip joint line. Shifting the dividing baseline upwards ensures the cutting path falls precisely below the narrowest point of the waist, highlighting the natural boundary of the waistline while allowing the upper garment to cover the core torso and the lower garment to correspond to the lower limbs, conforming to the visual logic of clothing segmentation and avoiding an overly long upper garment or an abrupt starting position of the lower garment due to a low baseline. Similarly, apple-shaped figures have more fat accumulation around the waist, with less pronounced constriction and an outward-expanding appearance, resulting in a blurred and lower-positioned waistline. Offsetting the dividing baseline downwards allows the cutting path to avoid the protruding abdominal area and fall at the transition point above the hip joint below the line connecting the hip joints. This avoids the dividing line getting stuck on the abdomen due to the baseline being too high, resulting in unnatural boundaries and a disconnect between the garment's division and the human body shape. At the same time, it ensures that the lower garment area can naturally connect to the hips.

[0051] Therefore, the vertical offset corresponding to different body types can be preset. After determining the body type of the person in the image, the corresponding vertical offset is found in the preset adjustment mapping table according to the body type, and the segmentation baseline is offset. The offset segmentation baseline is then used as the cropping path.

[0052] Step S40: Cut the segmentation mask according to the cutting path to segment the clothing.

[0053] After determining the cutting path, the overall garment segmentation mask is processed pixel-level. Following the cutting path, the pixel region corresponding to the overall mask is divided into two parts. Simultaneously, the edges of the segmented mask are optimized, such as eliminating jagged pixels and adjusting the consistency of edge pixel identifiers, to avoid unnatural segmentation boundaries.

[0054] Understandably, current image segmentation methods typically use image segmentation models to divide image pixels into different semantic categories to separate the target area from the background. This approach can only separate the clothing area as a single semantic category, but cannot further separate the upper and lower garments from the overall clothing area. Therefore, it fails to meet the need for virtual try-on while keeping the upper body unchanged and viewing different lower garment designs. Thus, the embodiments described in this application use skeletal joint point positioning to cut the clothing mask, which can fit a cutting path that conforms to the natural curves of the human body, achieving precise separation of the upper and lower body of a one-piece garment.

[0055] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 After the step of offsetting the segmentation baseline based on the vertical offset, the method further includes steps S36-S37: Step S36: Based on the ratio between the spacing of the body feature node pairs as the input of the preset curvature calculation formula, determine the curvature of the segmentation baseline according to the preset curvature calculation formula.

[0056] Step S37: Fit the offset segmentation baseline into a curve according to the curvature to obtain the trimming path.

[0057] For example, the proportional values ​​between the spacing of body feature node pairs, such as the shoulder-hip ratio or waist-hip ratio, are used as input parameters and substituted into a preset curvature calculation formula. This preset curvature calculation formula is a pre-defined functional relationship based on statistical laws of human anatomy. Then, based on the curvature values ​​calculated by the preset curvature formula, the current straight path is fitted into a smooth curve. This process can be implemented using curve fitting algorithms such as quadratic Bézier curves or spline interpolation algorithms.

[0058] Taking the quadratic Bézier curve algorithm as an example, the algorithm first sets a control point at the midpoint of the dividing baseline. Based on the calculated curvature, it determines the vertical offset of the control point. The sign of the curvature determines the direction of the control point's offset, and the magnitude of the absolute value of the curvature determines the amount of offset in that direction. A larger absolute value of curvature results in a larger offset, making the curve more curved. For example, suppose the curve fitting formula is: Where B(t) represents the coordinates of the curve; P0 and P2 represent the two endpoints of the dividing baseline; and P1 represents the control point. The virtual clothing try-on system iterates from 0 to 1 with a preset step size t, calculating a series of dense, continuous point coordinates according to the above fitting formula. Connecting these points forms a smooth cutting path.

[0059] Optionally, the horizontal width parameter, vertical curve parameter, and pose compensation parameter of the person in the image are extracted from the above body shape features, and the curve is fitted based on the control points selected by the user on the segmentation baseline as the cropping path.

[0060] Among them, the lateral width parameter is a quantitative description of the width distribution characteristics of the human torso in the horizontal direction, reflecting the relative width relationship of different horizontal sections of the torso, such as the shoulders, waist, and hips; the longitudinal curve parameter is a quantitative description of the bending characteristics of the human torso in the vertical direction, reflecting the degree of undulation and curvature of the torso lines; the posture compensation parameter is a quantitative description of the deviation of the human torso and pelvis from non-standard upright posture, reflecting the posture tilt and rotation state of the human body in space. For example, the lateral width parameter includes the ratio of shoulder width or hip width, the ratio of the narrowest point of the waist to the hip width, etc.; the longitudinal curve parameter includes the vertical offset of the waist joint points relative to the segmentation baseline, the curvature of the torso midline, etc.; the posture compensation parameter includes the left and right tilt angles of the torso that can be calculated based on the slope of the line connecting the shoulder, waist, and hip joint points, and the pelvic rotation angle that can be calculated based on the height difference between the left and right hip joint points, etc.

[0061] Then, three key control points are evenly selected on the segmentation baseline. Based on these key control points, a third-order Bézier curve is used to generate a curve as the trimming path. The curvature of this curve is determined by the aforementioned parameters. For example, the aforementioned parameters are used as input, and the parameter values ​​are converted into the vertical offset of the key control points according to a preset proportional formula to determine the curvature of the curve. It should be noted that the number of control points selected above is only an example. In other embodiments, the user can set the number of control points to be evenly selected, or the user can randomly select control points on the segmentation baseline.

[0062] In this embodiment, a pre-trained model is first used to extract the joint point map and clothing mask of the figure. Then, body shape features are quantified by calculating key proportions such as shoulder width and hip width. Based on this, not only is the initial segmentation line based on the hip joint point intelligently adjusted vertically according to the body shape, but curvature calculation is further introduced to fit the straight path into a smooth curve, thereby accurately simulating the realistic contour of the human waist and abdomen. This method effectively achieves a natural and accurate separation of the upper and lower body of a one-piece garment, significantly improving the visual realism of the segmentation results and the user experience.

[0063] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After the step of offsetting the segmentation baseline based on the vertical offset, the method further includes steps S38 to S311: Step S38: Obtain the garment boundary texture feature points of the worn clothing.

[0064] Textile feature points on clothing boundaries refer to pixels on the surface of clothing formed by texture changes such as belts, seams, pattern edges, fabric seams, folds, and turns, which can be used as references for the boundaries between the upper and lower parts of the clothing.

[0065] For example, the virtual clothing try-on system locates the clothing area based on segmentation mask, extracts the texture information of the clothing area from the target image, including pixel grayscale changes, edge gradients, color distribution differences, etc., and identifies the clothing boundary texture feature points in the clothing area through texture feature detection algorithms such as edge detection, corner detection, texture gradient analysis, etc., and records the coordinates of these feature points and classifies and labels them according to texture type.

[0066] Step S39: Determine the distance from each of the garment boundary texture feature points to the offset segmentation baseline.

[0067] Step S310: Determine the alignment degree of each of the garment boundary texture feature points based on the product of the type weight corresponding to the garment boundary texture feature point and the distance, wherein the distance is inversely proportional to the alignment degree.

[0068] Alignment is an indicator that measures the degree of matching between the garment boundary texture feature points and the offset segmentation baseline. The closer the garment boundary texture feature points are to the segmentation baseline, the higher the type weight, and the higher the alignment.

[0069] For example, the vertical distance from each garment boundary texture feature point to the offset segmentation baseline is calculated. The alignment H is calculated using a preset formula: H = c × (1 / (1+d)). Here, c is the type weight, and d is the distance from the garment boundary texture feature point to the segmentation baseline.

[0070] Step S311: If the alignment of the garment boundary texture feature points is greater than a preset threshold, the garment boundary texture feature points are used as target fitting points, and the fitting curve based on the target fitting points is used as the cutting path.

[0071] It should be noted that the target fitting point is a garment boundary texture feature point with an alignment degree greater than a preset threshold, that is, an effective feature point with a high degree of matching with the offset segmentation baseline and which can be used as a reference for the cutting path.

[0072] For example, feature points with alignment values ​​greater than a preset threshold are retained, while low-reliability feature points with substandard alignment are removed to obtain target fitting points. The target fitting points are then sorted by coordinates in the horizontal direction, and curve fitting algorithms such as Bézier curves and polynomial curve fitting are used to fit the sorted target fitting points. The curve fitting algorithm generates a smooth trajectory line that fits most target fitting points by minimizing the overall deviation between the feature points and the fitted curve. This results in a smooth cutting path based on the garment's own boundary texture features and matching the offset segmentation baseline. This cutting path conforms to both the human skeletal boundaries and the garment's own structural texture. Specifically, the fitting method can refer to Embodiment 2 above, where the target fitting points are used as control points, and a curve is fitted using the Bézier curve algorithm.

[0073] Optionally, the pose features of the person in the image are determined based on the skeletal joint map; the curvature of the line connecting the target fitting points is determined based on the pose features; and the fitting curve of each target fitting point and curvature is used as the cropping path.

[0074] For example, key joints related to posture, such as the left and right shoulder joints, left and right hip joints, and left and right knee joints, are selected from the set of skeletal joints. The left and right tilt angles of the torso are calculated by the slope of the line connecting the midpoints of the left and right shoulder joints and the midpoints of the left and right hip joints. The pelvic rotation angle is determined based on the height difference and horizontal offset of the left and right hip joints. The curvature of the line connecting the coordinates of the neck, waist, and hip joints is used to determine whether the spine is straight, slightly curved, or laterally curved. Integrating the above parameters forms a set of posture features describing the current posture of the person in the image.

[0075] Optionally, the curvature direction of the line connecting the target fitting points is determined based on the left and right tilt angles of the torso. For example, if the left tilt angle of the torso is greater than 0, meaning the torso tilts to the left, the curvature on the left side of the curve is controlled to be greater than that on the right side to fit the tilted torso contour. Optionally, the curvature of the line connecting the target fitting points is determined based on the pelvic rotation angle, where the pelvic rotation angle is proportional to the curvature. For example, if the posture features show a large pelvic rotation angle, the curvature in the middle section of the curve is increased to conform to the transition between the abdomen and hips. If the pelvic rotation angle is small, the curvature is decreased to maintain a smooth path.

[0076] Optionally, a mapping relationship between each pose feature and the above curvature can be preset, and the curvature of the line connecting the target fitting points corresponding to the pose features of the person in the image can be found according to the mapping relationship.

[0077] In this embodiment, after obtaining the offset segmentation baseline based on skeletal joints and body shape features, the alignment degree between the garment boundary texture feature points on the garment surface and the baseline is calculated and effective target fitting points are selected. Then, the cutting path is fitted based on these target fitting points. Furthermore, the curvature of the curve can be determined by combining human posture features, so that the cutting path is more adapted to dynamic posture. This allows the cutting path to not only conform to the human skeleton and body shape, but also to the garment's own structural texture, further improving the accuracy and naturalness of garment segmentation.

[0078] Based on the above embodiments of this application, in the fourth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 After step S311, steps S312-S313 are also included: Step S312: Display the garment boundary texture feature points and the cutting path on the segmentation mask.

[0079] Step S313: In response to the selection action of the garment boundary texture feature points on the segmentation mask, the curve is refitted based on the selected garment boundary texture feature points as the new cutting path.

[0080] For example, the virtual clothing try-on system loads a segmentation mask as the underlying image onto the interactive display interface, and overlays clothing boundary texture feature points onto the segmentation mask. For instance, it assigns differentiated visual identifiers to different types of clothing boundary texture feature points and marks their coordinate positions. Simultaneously, it overlays a cutting path onto the segmentation mask. The line width and color of the cutting path must be distinct from the clothing boundary texture feature points and the segmentation mask to ensure clear visibility.

[0081] Optionally, interface control elements such as zoom and pan buttons can be displayed simultaneously to allow users to adjust the viewing range and clearly observe the relative positional relationship between the garment boundary texture feature points and the cutting path.

[0082] The virtual clothing try-on system captures the user's selection action on the clothing boundary texture feature points on the segmentation mask through input devices such as mice and touch screens, and records the coordinates and type information of the selected clothing boundary texture feature points.

[0083] The user-selected garment boundary texture feature points are used as new control points. The cutting path is regenerated using the curve fitting method described in Example 3 above. The newly generated cutting path is then overlaid on the segmentation mask, replacing the original cutting path, for the user to confirm whether it meets expectations.

[0084] In this embodiment, by adding a user interaction optimization step, the garment boundary texture feature points and the generated cutting path are visualized and overlaid on the segmentation mask, providing viewing and control functions such as zooming and panning. At the same time, users can select garment boundary texture feature points through input devices to refit and generate new cutting paths, meeting the subjective needs of different scenarios, improving the accuracy and adaptability of cutting paths, and significantly enhancing the interactivity and user experience of the virtual try-on scenario.

[0085] Based on the above embodiments of this application, in the fifth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 After step S40, steps S50 to S70 are also included: Step S50: In response to the selection operation of the sub-segmentation mask obtained after cropping, a replacement region is determined and marked in the target image based on the pixel coordinate boundaries of the selected sub-segmentation mask.

[0086] The virtual clothing try-on system responds to user selections, such as clicking to select a sub-segmentation mask for "top" or "bottom" on the graphical interface. It then acquires the selected sub-segmentation mask and performs contour detection on it. For example, it uses algorithms like the findContours class in the OpenCV library to traverse the segmentation mask image, resulting in a closed polygon defined by the pixel coordinates of the boundary points of the segmentation mask. This polygonal region, defined by the boundary point coordinates, is the precise region in the target image that needs to be replaced. The virtual clothing try-on system assigns a specific identifier to this region, either by generating a new mask corresponding to the polygonal region, such as a replacement region mask, or by internally marking the pixels of this region as the area to be edited.

[0087] Step S60: Obtain a replacement clothing image, and determine the scaling ratio of the replacement clothing image based on the spatial position of the joint points corresponding to the replacement area in the skeletal joint map.

[0088] For example, if the replacement area is a cropped upper garment area, its corresponding joint points may include shoulder joint points, neck joint points, etc., and the scaling ratio of the replacement garment image can be determined based on the distance from the left shoulder joint point to the right shoulder joint point, or the distance from the neck joint point to the shoulder joint point. If the replacement area is a cropped lower garment area, its corresponding joint points may include hip joint points, knee joint points, etc., and the scaling ratio of the replacement garment image can be determined based on the distance from the left hip joint point to the right hip joint point, or the distance from the hip joint point to the knee joint point.

[0089] Specifically, the scaling ratio can be determined by the ratio of the distance between the joints corresponding to the replacement area in the selected skeletal joint map and the size of the replacement clothing image, ensuring that the replacement clothing image matches the target person in size.

[0090] Step S70: After scaling the replacement clothing image according to the scaling ratio, use the replacement clothing image and the labeled target image as input to the pre-trained image restoration model to obtain the clothing replacement image output by the image restoration model.

[0091] For example, the virtual clothing try-on system takes a scaled-down image of the replacement clothing, overlays it onto the replacement area of ​​the target image based on the joints corresponding to the replacement area in the selected skeletal joint map, and then inputs it along with the target image into a pre-trained image inpainting model, such as LaMa (Large Mask Inpainting) or AOT (All-Of-Transformer for Image Inpainting) deep learning models. The image inpainting model first erases the original clothing pixels in the marked replacement area of ​​the target image, and then generates the content of the new replacement clothing image in a way that matches the lighting and shadow parameters of the surrounding area of ​​the replacement area in the target image, and outputs a complete changed clothing image.

[0092] In this embodiment, by responding to user operations, the replacement region in the image is precisely defined based on the contour of the sub-segmentation mask. Then, the human skeletal joints are intelligently used as a spatial scale to dynamically calculate the scaling ratio required for the replacement clothing, ensuring a natural match with the target person's body shape and posture. Finally, using an advanced image inpainting model, the scaled clothing image is deeply fused with the target image, automatically handling lighting coordination and edge transitions, ultimately generating a realistic, seamless image of the changed clothing. This solution seamlessly integrates clothing segmentation, size adaptation, and image generation technologies, significantly improving the automation level and visual realism of virtual try-on.

[0093] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the clothing image segmentation method based on human skeletal joints in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0094] This application provides a clothing image segmentation device based on human skeletal joints. The clothing image segmentation device based on human skeletal joints includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the clothing image segmentation method based on human skeletal joints in the above embodiment 1.

[0095] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of a clothing image segmentation device based on human skeletal joints suitable for implementing embodiments of this application. The clothing image segmentation device based on human skeletal joints in these embodiments may include, but is not limited to, mobile terminals such as laptops and tablets (PADs), and fixed terminals such as digital TVs and desktop computers. Figure 6 The clothing image segmentation device based on human skeletal joints shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] like Figure 6As shown, the clothing image segmentation device based on human skeletal joints may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the clothing image segmentation device based on human skeletal joints. The processing unit 1001, the read-only memory 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the clothing image segmentation device based on human skeletal joints to exchange data with other devices wirelessly or via wired communication. Although the figure shows clothing image segmentation devices based on human skeletal joints with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0097] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0098] The clothing image segmentation device based on human skeletal joints provided in this application, employing the clothing image segmentation method based on human skeletal joints in the above embodiments, can solve the technical problem of how to improve the segmentation visualization effect of virtual character integrated clothing. Compared with the prior art, the beneficial effects of the clothing image segmentation device based on human skeletal joints provided in this application are the same as the beneficial effects of the clothing image segmentation method based on human skeletal joints provided in the above embodiments, and other technical features in the clothing image segmentation device based on human skeletal joints are the same as the features disclosed in the previous embodiment method, and will not be repeated here.

[0099] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0101] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the clothing image segmentation method based on human skeletal joints in the above embodiments.

[0102] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0103] The aforementioned computer-readable storage medium may be included in a clothing image segmentation device based on human skeletal joints; or it may exist independently and not be assembled into a clothing image segmentation device based on human skeletal joints.

[0104] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a clothing image segmentation device based on human skeletal joints, enable the device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0107] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described clothing image segmentation method based on human skeletal joints. This addresses the technical problem of improving the segmentation visualization effect of virtual character one-piece clothing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the clothing image segmentation method based on human skeletal joints provided in the above embodiments, and will not be repeated here.

[0108] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the clothing image segmentation method based on human skeletal joints as described above.

[0109] The computer program product provided in this application solves the technical problem of how to improve the segmentation and visualization effect of virtual character integrated clothing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the clothing image segmentation method based on human skeletal joints provided in the above embodiments, and will not be repeated here.

[0110] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A garment image segmentation method based on human skeleton joints, characterized in that, The garment image segmentation method based on human body skeleton joints comprises: inputting a target image into a pre-trained human pose estimation model and a semantic segmentation model respectively, obtaining a skeleton joint map of an image character in the target image output by the human pose estimation model, and obtaining a segmentation mask of a worn garment of the image character output by the semantic segmentation model; determining a body shape feature of the image character according to the skeleton joint map; taking a left-right hip joint connection line in the skeleton joint map as a segmentation reference line, and adjusting the segmentation reference line according to an adjustment rule corresponding to the body shape feature to obtain a cutting path; cutting the segmentation mask according to the cutting path to segment the worn garment.

2. The human pose based garment image segmentation method of claim 1, wherein, The step of determining the body shape feature of the image character according to the skeleton joint map comprises: determining a proportion value between the distances of at least two groups of body shape feature node pairs in the skeleton joint map based on a pre-set body shape feature node pair, wherein the body shape feature node pair at least includes a left-right shoulder joint pair and a left-right hip joint pair; taking the proportion value between the distances of the body shape feature node pair as the body shape feature. 3.The human body pose based garment image segmentation method of claim 1, wherein, The body shape feature includes a proportion value between the distances of a body shape feature node pair, and the step of adjusting the segmentation reference line to obtain a cutting path according to an adjustment rule corresponding to the body shape feature comprises: taking the proportion value between the distances of the body shape feature node pair as an input parameter of a pre-set adjustment formula, and calculating a vertical offset of the segmentation reference line corresponding to the proportion value through the pre-set adjustment formula; offsetting the segmentation reference line based on the vertical offset to obtain the cutting path.

4. The human pose based garment image segmentation method of claim 1, wherein, The body shape feature includes a proportion value between the distances of a body shape feature node pair, and the step of adjusting the segmentation reference line to obtain a cutting path according to an adjustment rule corresponding to the body shape feature comprises: determining a body shape type of the image character based on the proportion value between the distances of the body shape feature node pair; determining a vertical offset of the segmentation reference line corresponding to the body shape type in a pre-set adjustment mapping table based on the body shape type; offsetting the segmentation reference line based on the vertical offset to obtain the cutting path.

5. The human-skeleton-joint-based garment image segmentation method according to claim 3 or 4, characterized in that, After the step of offsetting the segmentation reference line based on the vertical offset, the method further comprises: taking the proportion value between the distances of the body shape feature node pair as an input of a pre-set curvature calculation formula, and determining a curvature of the segmentation reference line according to the pre-set curvature calculation formula; fitting the offset segmentation reference line into a curve according to the curvature to obtain the cutting path.

6. The human-skeleton-joint-based garment image segmentation method according to claim 3 or 4, characterized in that, After the step of offsetting the segmentation reference line based on the vertical offset, the method further comprises: obtaining garment boundary texture feature points of the worn garment; determining distances from each garment boundary texture feature point to the offset segmentation reference line; determining an alignment degree of each garment boundary texture feature point according to a product of a type weight corresponding to the garment boundary texture feature point and the distance, wherein the distance is inversely proportional to the alignment degree. If the alignment degree of the clothing boundary texture feature point is greater than a preset threshold, the clothing boundary texture feature point is taken as a target fitting point, and a curve is fitted according to the target fitting point as the cutting path.

7. The human-skeleton-joint-based garment image segmentation method of claim 6, wherein, The step of fitting the curve according to the target fitting point as the cutting path further comprises: displaying the clothing boundary texture feature point and the cutting path on the segmentation mask; In response to a selection action of the clothing boundary texture feature point on the segmentation mask, a curve is refitted according to the selected clothing boundary texture feature point as a new cutting path.

8. The human pose based garment image segmentation method of claim 1, wherein, The step of cutting the segmentation mask according to the cutting path to segment the worn clothing further comprises: In response to a selection operation of the sub-segmentation mask obtained after cutting, a replacement area is determined and marked in the target image based on the pixel coordinate boundary of the selected sub-segmentation mask; An image of a replacement garment is obtained, and a scaling ratio of the image of the replacement garment is determined based on the spatial position of the joint corresponding to the replacement area in the joint graph; After scaling the image of the replacement garment according to the scaling ratio, the image of the replacement garment and the marked target image are taken as inputs of a pre-trained image inpainting model to obtain a changed image output by the image inpainting model.

9. A human body bone joint-based garment image segmentation device, characterized by, The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the garment image segmentation method based on human body joint as claimed in any one of claims 1 to 8.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the garment image segmentation method based on human body joint as claimed in any one of claims 1 to 8.

Images