Method, device and storage medium for clothing image segmentation based on human body skeletal joints

Abstract

The application discloses a garment image segmentation method and device based on human skeleton joints and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: inputting a target image into a pre-trained human pose estimation model and a semantic segmentation model respectively, obtaining a skeleton joint graph of an image character in the target image output by the human pose estimation model, and obtaining a segmentation mask of a worn garment of the image character output by the semantic segmentation model; determining a body shape feature of the image character according to the skeleton joint graph; taking a left-right hip joint connection line in the skeleton joint graph as a segmentation reference line, and adjusting the segmentation reference line to obtain a cutting path according to an adjustment rule corresponding to the body shape feature; and cutting the segmentation mask according to the cutting path to segment the worn garment, so that the upper and lower parts of a dress conforming to a human structure are obtained, and the segmentation visualization effect of an integrated garment of a virtual character is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, device and storage medium for clothing image segmentation based on human skeletal joints. Background Technology

[0002] In e-commerce, fashion design, virtual reality, and other fields, virtual try-on technology has become crucial for enhancing user experience. Users often expect to visually see how different bottom garment designs, such as hem styles, would look when the upper body remains unchanged. However, when the image features a person wearing a dress or other one-piece garment, current semantic segmentation models typically segment "dress" as a single category, failing to differentiate between upper and lower body masks. Furthermore, simply cropping the overall segmentation result leads to unnatural segmentation boundaries and poor overall quality.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device and storage medium for clothing image segmentation based on human skeletal joints, aiming to solve the technical problem of how to improve the segmentation and visualization effect of virtual character one-piece clothing.

[0005] To achieve the above objectives, this application proposes a method for clothing image segmentation based on human skeletal joints, the method comprising:

[0006] The target image is input into a pre-trained human pose estimation model and a semantic segmentation model, respectively. The skeletal joint map of the person in the target image is obtained from the output of the human pose estimation model, and the segmentation mask of the clothing of the person in the image is obtained from the output of the semantic segmentation model.

[0007] The body shape features of the person in the image are determined based on the skeletal joint diagram;

[0008] The line connecting the left and right hip joints in the skeletal joint diagram is used as the segmentation baseline, and the segmentation baseline is adjusted according to the adjustment rules corresponding to the body shape characteristics to obtain the clipping path.

[0009] The segmentation mask is cropped according to the cropping path to segment the clothing.

[0010] In one embodiment, the step of determining the body shape features of the person in the image based on the skeletal joint map includes:

[0011] Based on preset body feature node pairs, and according to the ratio between the spacing of the body feature node pairs in the skeletal joint diagram, the body feature node pairs include at least left and right shoulder joint point pairs and left and right hip joint point pairs.

[0012] The ratio between the spacing of the pairs of body shape feature nodes is taken as the body shape feature.

[0013] In one embodiment, the step of adjusting the segmentation baseline to obtain the clipping path according to the adjustment rules corresponding to the body shape features includes:

[0014] The ratio between the spacing of the body feature node pairs is used as the input parameter of the preset adjustment formula, and the vertical offset of the segmentation baseline corresponding to the ratio is calculated by the preset adjustment formula.

[0015] The cutting path is obtained by offsetting the segmentation baseline based on the vertical offset.

[0016] In one embodiment, the step of adjusting the segmentation baseline to obtain the clipping path according to the adjustment rules corresponding to the body shape features includes:

[0017] The body type of the person in the image is determined based on the ratio between the spacing of the pairs of body feature nodes.

[0018] Based on the body type, the vertical offset of the segmentation baseline corresponding to the body type is determined in a preset adjustment mapping table;

[0019] The cutting path is obtained by offsetting the segmentation baseline based on the vertical offset.

[0020] In one embodiment, after the step of offsetting the segmentation baseline based on the vertical offset, the method further includes:

[0021] The curvature of the segmentation baseline is determined based on the ratio between the spacing of the body feature node pairs as input to the preset curvature calculation formula.

[0022] The offset segmentation baseline is fitted into a curve based on the curvature to obtain the trimming path.

[0023] In one embodiment, after the step of offsetting the segmentation baseline based on the vertical offset, the method further includes:

[0024] Obtain the garment boundary texture feature points of the worn clothing;

[0025] Determine the distance from each of the garment boundary texture feature points to the offset segmentation baseline;

[0026] The alignment degree of each of the garment boundary texture feature points is determined by multiplying the type weight corresponding to the garment boundary texture feature point with the distance, wherein the distance is inversely proportional to the alignment degree.

[0027] If the alignment of the garment boundary texture feature point is greater than a preset threshold, the garment boundary texture feature point is used as the target fitting point, and the fitting curve based on the target fitting point is used as the cutting path.

[0028] In one embodiment, after the step of fitting a curve based on the target fitting point as the clipping path, the method further includes:

[0029] Display the garment boundary texture feature points and the cutting path in the segmentation mask;

[0030] In response to the selection action of the garment boundary texture feature points on the segmentation mask, a curve is refitted based on the selected garment boundary texture feature points as the new cutting path.

[0031] In one embodiment, after the step of cropping the segmentation mask according to the cropping path to segment the clothing, the method further includes:

[0032] In response to the selection operation of the sub-segmentation mask obtained after cropping, a replacement region is determined and marked in the target image based on the pixel coordinate boundaries of the selected sub-segmentation mask;

[0033] Obtain a replacement clothing image, and determine the scaling ratio of the replacement clothing image based on the spatial position of the joint points corresponding to the replacement area in the skeletal joint point map;

[0034] After scaling the replacement clothing image according to the scaling ratio, the replacement clothing image and the labeled target image are used as input to a pre-trained image restoration model to obtain the clothing replacement image output by the image restoration model.

[0035] Furthermore, to achieve the above objectives, this application also proposes a clothing image segmentation device based on human skeletal joints. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the clothing image segmentation method based on human skeletal joints as described above.

[0036] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the clothing image segmentation method based on human skeletal joints as described above.

[0037] This application provides a method for clothing image segmentation based on human skeletal joints. The method involves inputting a target image into a pre-trained human pose estimation model and a semantic segmentation model, respectively. The human pose estimation model outputs a skeletal joint map of the person in the target image, while the semantic segmentation model outputs a segmentation mask of the clothing worn by the person. The method then determines the body shape features of the person based on the skeletal joint map. The line connecting the left and right hip joints in the skeletal joint map is used as a segmentation baseline. The baseline is adjusted according to the adjustment rules corresponding to the body shape features to obtain a clipping path. The segmentation mask is then clipped according to the clipping path to segment the clothing, thereby obtaining the upper and lower parts of a dress that conform to the human body structure, improving the visualization effect of segmenting a virtual character's one-piece clothing. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating an embodiment of the clothing image segmentation method based on human skeletal joints provided in this application.

[0041] Figure 2 This is a flowchart illustrating Embodiment 2 of the clothing image segmentation method based on human skeletal joints provided in this application;

[0042] Figure 3 This is a flowchart illustrating Embodiment 3 of the clothing image segmentation method based on human skeletal joints provided in this application;

[0043] Figure 4 This is a flowchart illustrating Embodiment 4 of the clothing image segmentation method based on human skeletal joints in this application.

[0044] Figure 5 This is a flowchart illustrating Embodiment 5 of the clothing image segmentation method based on human skeletal joints in this application;

[0045] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the clothing image segmentation method based on human skeletal joints in the embodiments of this application.

[0046] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0048] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. It should be noted that all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and regulations of the country where the application is located, and with authorization from the owner of the corresponding device.

[0049] In e-commerce, fashion design, virtual reality, and other fields, virtual try-on technology has become crucial for enhancing user experience. Users often expect to visually see how different bottom garment designs, such as hem styles, would look when the upper body remains unchanged. However, when the image features a person wearing a dress or other one-piece garment, current semantic segmentation models typically segment "dress" as a single category, failing to differentiate between upper and lower body masks. Furthermore, simply cropping the overall segmentation result leads to unnatural segmentation boundaries and poor overall quality.

[0050] In view of the above problems, this application proposes a clothing image segmentation method based on human skeletal joints. The method involves inputting the target image into a pre-trained human pose estimation model and a semantic segmentation model, respectively, to obtain the skeletal joint map of the person in the target image output by the human pose estimation model, and the segmentation mask of the clothing worn by the person in the image output by the semantic segmentation model. The method then determines the body shape features of the person based on the skeletal joint map. The line connecting the left and right hip joints in the skeletal joint map is used as the segmentation baseline, and the baseline is adjusted according to the adjustment rules corresponding to the body shape features to obtain a clipping path. Finally, the segmentation mask is clipped according to the clipping path to segment the clothing, thereby obtaining the upper and lower parts of a dress that conform to the human body structure, improving the visualization effect of segmenting the one-piece clothing of the virtual character.

[0051] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or virtual clothing try-on system capable of the above functions. The following description uses a virtual clothing try-on system as an example to illustrate this embodiment and the subsequent embodiments.

[0052] Based on this, the first embodiment of this application provides a method for clothing image segmentation based on human skeletal joints, referring to... Figure 1In this embodiment, the clothing image segmentation method based on human skeletal joints includes steps S10 to S40:

[0053] Step S10: Input the target image into the pre-trained human pose estimation model and semantic segmentation model respectively, obtain the skeletal joint map of the person in the target image output by the human pose estimation model, and obtain the segmentation mask of the clothing of the person in the image output by the semantic segmentation model.

[0054] Skeletal joints refer to the coordinates of key parts of a person's body in an image, reflecting the posture of the human body and the positional boundaries of various body parts; segmentation masks are image data generated by semantic segmentation models, outlining the clothing with specific identifiers such as preset pixel values, enabling the distinction between clothing areas and background areas.

[0055] For example, a virtual clothing try-on system receives a target image containing a target person and their clothing, and calls a trained human pose estimation model such as OpenPose, AlphaPose, or YOLOv8 Pose to analyze the target image. The human pose estimation model identifies features of key parts of the person's body in the target image and outputs the specific coordinate data of each skeletal joint, forming a complete set of human skeletal joints. Simultaneously, a trained semantic segmentation model such as SegNet, U-Net, or DeepLab series models is called to perform semantic classification processing on the target image. The semantic segmentation model divides the pixels in the image into different categories such as clothing, human skin, and background, filters out the pixel regions corresponding to the clothing category, and generates segmentation mask data that marks the outline of the clothing with specific pixel values. The initial state of this segmentation mask data is the overall outline mask of a one-piece garment.

[0056] Step S20: Determine the body shape characteristics of the person in the image based on the skeletal joint map.

[0057] Optionally, step S20 includes steps S21 to S22:

[0058] Step S21: Based on preset body feature node pairs, determine the ratio between the spacing of at least two sets of body feature node pairs in the skeletal joint map. The body feature node pairs include at least left and right shoulder joint pairs and left and right hip joint pairs.

[0059] Step S22: The ratio between the spacing of the body shape feature node pairs is taken as the body shape feature.

[0060] Preferably, the body shape features include the ratio between any two or three pairs of body shape feature nodes, such as the shoulder-hip ratio, waist-hip ratio, shoulder-waist ratio, or shoulder-waist-hip ratio mentioned above.

[0061] Optionally, in other embodiments, the body shape feature node pairs may also include left and right hip feature joint pairs, upper arm mid-section node pairs, lower rib edge node pairs, anterior superior iliac spine node pairs, etc.; based on at least two of the above body shape feature nodes, after arbitrary combination, the ratio between the spacing of each group of body shape feature node pairs after combination is calculated as the body shape feature characterizing the human torso morphology.

[0062] For example, the virtual clothing try-on system extracts joint pairs related to torso shape from the skeletal joint map, such as left and right shoulder joint pairs, left and right hip joint pairs, and left and right waist feature joint pairs. It calculates body shape feature parameters in the following ways: shoulder width is calculated by the distance between the left and right shoulder joints; hip width is calculated by the distance between the left and right hip joints, and the ratio of shoulder width to hip width is taken to obtain the shoulder-hip ratio; waist width is calculated by the distance between the left and right waist feature joints, and the ratio of waist width to hip width is taken to obtain the waist-hip ratio; the ratio of shoulder width to hip width is taken to obtain the shoulder-hip ratio; and torso posture parameters are obtained by the slope of the line connecting the coordinates of the shoulder joints, waist feature joints, and hip joints. These parameters are then integrated to form a set of body shape features describing the human body shape in the image.

[0063] In some specific embodiments, in order to save computing resources and improve response speed, a simpler human pose estimation model is used to identify skeletal joints. Such models may not be able to identify lumbar feature joints very well.

[0064] Based on this, the distance between the lines connecting the left and right shoulder joints can be taken as the shoulder width; the midpoint of the line connecting the left and right shoulder joints and the midpoint of the line connecting the left and right hip joints can be taken as the trunk midline; based on a preset proportional parameter, the center point of the waist is determined on the trunk midline, and a horizontal line is drawn based on the center point of the waist, intersecting with the segmentation mask, and the distance between the intersection points is taken as the waist width. The preset proportional parameter is the ratio of the length of the line segment from the midpoint of the shoulder joint to the center point of the waist on the trunk midline to the total length of the entire trunk midline. This ratio is a pre-set proportional threshold based on the conventional anatomical structure proportions of the human trunk, used to locate the center point of the waist on the trunk midline. Using this preset ratio, the lateral reference position of the waist, i.e., the center point of the waist, can be accurately located on the trunk midline even in the absence of a waist joint, providing a basis for subsequent waist width calculations.

[0065] Step S30: Use the line connecting the left and right hip joints in the skeletal joint diagram as the segmentation baseline, and adjust the segmentation baseline according to the adjustment rules corresponding to the body shape characteristics to obtain the clipping path.

[0066] Understandably, when human pose estimation models identify skeletal joints, they classify and label these joints. Virtual clothing try-on systems can directly locate the left and right hip joints using these category labels. A straight line connects the two points to form a dividing baseline.

[0067] The left and right hip joints are key dividing points between the upper and lower body skeletons. The connection between the torso and lower limbs corresponds precisely to the natural dividing line of the upper and lower parts of a one-piece garment. In different postures, such as standing, slightly turned to the side, or slightly bent over, the relative positions of the left and right hip joints remain relatively stable, and the line connecting them always accurately corresponds to the dividing area between the upper and lower body of the garment. Therefore, the line connecting the left and right hip joints can be used as a dividing baseline for segmenting the clothing worn by a person in an image. By adjusting this baseline based on the person's body shape, a cropping path that better conforms to human visual perception can be obtained.

[0068] Alternatively, as a first feasible implementation, the aforementioned segmentation baseline can be directly used as the cutting path. This method only requires selecting two skeletal joints and connecting them; the calculation process is simple, requiring no complex coordinate fitting algorithms or multi-dimensional feature analysis, resulting in low computational consumption for the virtual clothing try-on system. This concise logical design enables it to quickly respond to processing needs, adapting to scenarios with real-time requirements, such as virtual try-on where users can switch bottom styles online in real time, thus improving the user experience.

[0069] Alternatively, as a second feasible implementation, step S30 includes steps S31-S32:

[0070] Step S31: The ratio between the spacing of the body feature node pairs is used as the input parameter of the preset adjustment formula, and the vertical offset of the segmentation baseline corresponding to the ratio is calculated by the preset adjustment formula.

[0071] The vertical offset refers to the distance the dividing baseline needs to be adjusted in the vertical direction, calculated by a preset adjustment formula based on a proportional value. A positive value indicates that the baseline moves upward, and a negative value indicates that it moves downward.

[0072] The virtual clothing try-on system obtains the proportional values ​​between the spacing of body shape feature node pairs and uses these proportional values ​​as input parameters, substituting them into a preset adjustment formula. The preset adjustment formula is a predefined mathematical expression used to map the proportional values ​​to the vertical offset of the segmentation baseline. This preset adjustment formula is typically designed based on human anatomy; for example, it can be expressed as: Vertical Offset = k × (Proportional Value - Baseline Proportion), where k is a scaling factor, and the baseline proportion is a preset reference ratio for a standard body shape, such as the average shoulder-hip ratio. The virtual clothing try-on system processes the proportional values ​​through mathematical calculations to obtain the specific vertical offset value. The specific form of the preset adjustment formula may vary depending on the system design, but its core logic is that when the proportional value deviates from the baseline proportion, the offset changes accordingly to adapt to different body shapes.

[0073] Step S32: Offset the segmentation baseline based on the vertical offset to obtain the clipping path.

[0074] After obtaining the vertical offset, the virtual clothing try-on system performs a vertical offset on the segmentation baseline. Since the segmentation baseline is a straight line, the virtual clothing try-on system can adjust the vertical coordinate values ​​of the left and right hip joint points, adding these vertical coordinate values ​​to the vertical offset to obtain new coordinate points. Then, connecting these two new coordinate points yields the cutting path.

[0075] Alternatively, as a third feasible implementation, step S30 further includes steps S33 to S35:

[0076] Step S33: Determine the body type of the person in the image based on the ratio between the spacing of the body feature node pairs.

[0077] Optionally, a set of body shape features, including the ratio values ​​between the spacing of different body shape feature node pairs, is input into a pre-trained body shape recognition model, and the body shape recognition model is used to infer the body shape type of the person in the image.

[0078] For example, first, collect human images and corresponding skeletal joint point data samples of different body types, define standardized body type labels such as hourglass, pear, apple, and rectangle, and determine the proportional range of the spacing between different body feature node pairs corresponding to each body type. Professional annotators or semi-automated tools then label each data sample with the corresponding body type label. The labeled data samples are randomly divided into training, validation, and test sets according to a preset ratio, and the body type recognition model is trained and optimized accordingly to obtain a pre-trained body type recognition model. Then, the aforementioned feature set acquired in real-time during the application is input to the input layer of the body type recognition model, initiating the forward propagation calculation. The body type recognition model outputs a vector containing the probability values ​​corresponding to each body type; a higher probability value indicates a greater likelihood that the person in the image belongs to that body type.

[0079] Step S34: Based on the body type, determine the vertical offset of the segmentation baseline corresponding to the body type in a preset adjustment mapping table.

[0080] Step S35: Offset the segmentation baseline based on the vertical offset to obtain the clipping path.

[0081] Understandably, in certain application scenarios, virtual clothing try-on systems allow users to generate virtual avatars to try on clothing based on their own image. For different body types, the different waistline cutting positions can create different visual effects. For example, the core characteristics of an hourglass figure are a close shoulder-to-hip ratio and a noticeably constricted waist, naturally resulting in a waistline higher than the hip joint line. Shifting the dividing baseline upwards ensures the cutting path falls precisely below the narrowest point of the waist, highlighting the natural boundary of the waistline while allowing the upper garment to cover the core torso and the lower garment to correspond to the lower limbs, conforming to the visual logic of clothing segmentation and avoiding an overly long upper garment or an abrupt starting position of the lower garment due to a low baseline. Similarly, apple-shaped figures have more fat accumulation around the waist, with less pronounced constriction and an outward-expanding appearance, resulting in a blurred and lower-positioned waistline. Offsetting the dividing baseline downwards allows the cutting path to avoid the protruding abdominal area and fall at the transition point above the hip joint below the line connecting the hip joints. This avoids the dividing line getting stuck on the abdomen due to the baseline being too high, resulting in unnatural boundaries and a disconnect between the garment's division and the human body shape. At the same time, it ensures that the lower garment area can naturally connect to the hips.

[0082] Therefore, the vertical offset corresponding to different body types can be preset. After determining the body type of the person in the image, the corresponding vertical offset is found in the preset adjustment mapping table according to the body type, and the segmentation baseline is offset. The offset segmentation baseline is then used as the cropping path.

[0083] Step S40: Cut the segmentation mask according to the cutting path to segment the clothing.

[0084] After determining the cutting path, the overall garment segmentation mask is processed pixel-level. Following the cutting path, the pixel region corresponding to the overall mask is divided into two parts. Simultaneously, the edges of the segmented mask are optimized, such as eliminating jagged pixels and adjusting the consistency of edge pixel identifiers, to avoid unnatural segmentation boundaries.

[0085] Understandably, current image segmentation methods typically use image segmentation models to divide image pixels into different semantic categories to separate the target area from the background. This approach can only separate the clothing area as a single semantic category, but cannot further separate the upper and lower garments from the overall clothing area. Therefore, it fails to meet the need for virtual try-on while keeping the upper body unchanged and viewing different lower garment designs. Thus, the embodiments described in this application use skeletal joint point positioning to cut the clothing mask, which can fit a cutting path that conforms to the natural curves of the human body, achieving precise separation of the upper and lower body of a one-piece garment.

[0086] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 After the step of offsetting the segmentation baseline based on the vertical offset, the method further includes steps S36-S37:

[0087] Step S36: Based on the ratio between the spacing of the body feature node pairs as the input of the preset curvature calculation formula, determine the curvature of the segmentation baseline according to the preset curvature calculation formula.

[0088] Step S37: Fit the offset segmentation baseline into a curve according to the curvature to obtain the trimming path.

[0089] For example, the proportional values ​​between the spacing of body feature node pairs, such as the shoulder-hip ratio or waist-hip ratio, are used as input parameters and substituted into a preset curvature calculation formula. This preset curvature calculation formula is a pre-defined functional relationship based on statistical laws of human anatomy. Then, based on the curvature values ​​calculated by the preset curvature formula, the current straight path is fitted into a smooth curve. This process can be implemented using curve fitting algorithms such as quadratic Bézier curves or spline interpolation algorithms.

[0090] Taking the quadratic Bézier curve algorithm as an example, the algorithm first sets a control point at the midpoint of the dividing baseline. Based on the calculated curvature, it determines the vertical offset of the control point. The sign of the curvature determines the direction of the control point's offset, and the magnitude of the absolute value of the curvature determines the amount of offset in that direction. A larger absolute value of curvature results in a larger offset, making the curve more curved. For example, suppose the curve fitting formula is: Where B(t) represents the coordinates of the curve; P0 and P2 represent the two endpoints of the dividing baseline; and P1 represents the control point. The virtual clothing try-on system iterates from 0 to 1 with a preset step size t, calculating a series of dense, continuous point coordinates according to the above fitting formula. Connecting these points forms a smooth cutting path.

[0091] Optionally, the horizontal width parameter, vertical curve parameter, and pose compensation parameter of the person in the image are extracted from the above body shape features, and the curve is fitted based on the control points selected by the user on the segmentation baseline as the cropping path.

[0092] Among them, the lateral width parameter is a quantitative description of the width distribution characteristics of the human torso in the horizontal direction, reflecting the relative width relationship of different horizontal sections of the torso, such as the shoulders, waist, and hips; the longitudinal curve parameter is a quantitative description of the bending characteristics of the human torso in the vertical direction, reflecting the degree of undulation and curvature of the torso lines; the posture compensation parameter is a quantitative description of the deviation of the human torso and pelvis from non-standard upright posture, reflecting the posture tilt and rotation state of the human body in space. For example, the lateral width parameter includes the ratio of shoulder width or hip width, the ratio of the narrowest point of the waist to the hip width, etc.; the longitudinal curve parameter includes the vertical offset of the waist joint points relative to the segmentation baseline, the curvature of the torso midline, etc.; the posture compensation parameter includes the left and right tilt angles of the torso that can be calculated based on the slope of the line connecting the shoulder, waist, and hip joint points, and the pelvic rotation angle that can be calculated based on the height difference between the left and right hip joint points, etc.

[0093] Then, three key control points are evenly selected on the segmentation baseline. Based on these key control points, a third-order Bézier curve is used to generate a curve as the trimming path. The curvature of this curve is determined by the aforementioned parameters. For example, the aforementioned parameters are used as input, and the parameter values ​​are converted into the vertical offset of the key control points according to a preset proportional formula to determine the curvature of the curve. It should be noted that the number of control points selected above is only an example. In other embodiments, the user can set the number of control points to be evenly selected, or the user can randomly select control points on the segmentation baseline.

[0094] In this embodiment, a pre-trained model is first used to extract the joint point map and clothing mask of the figure. Then, body shape features are quantified by calculating key proportions such as shoulder width and hip width. Based on this, not only is the initial segmentation line based on the hip joint point intelligently adjusted vertically according to the body shape, but curvature calculation is further introduced to fit the straight path into a smooth curve, thereby accurately simulating the realistic contour of the human waist and abdomen. This method effectively achieves a natural and accurate separation of the upper and lower body of a one-piece garment, significantly improving the visual realism of the segmentation results and the user experience.

[0095] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After the step of offsetting the segmentation baseline based on the vertical offset, the method further includes steps S38 to S311:

[0096] Step S38: Obtain the garment boundary texture feature points of the worn clothing.

[0097] Textile feature points on clothing boundaries refer to pixels on the surface of clothing formed by texture changes such as belts, seams, pattern edges, fabric seams, folds, and turns, which can be used as references for the boundaries between the upper and lower parts of the clothing.

[0098] For example, the virtual clothing try-on system locates the clothing area based on segmentation mask, extracts the texture information of the clothing area from the target image, including pixel grayscale changes, edge gradients, color distribution differences, etc., and identifies the clothing boundary texture feature points in the clothing area through texture feature detection algorithms such as edge detection, corner detection, texture gradient analysis, etc., and records the coordinates of these feature points and classifies and labels them according to texture type.

[0099] Step S39: Determine the distance from each of the garment boundary texture feature points to the offset segmentation baseline.

[0100] Step S310: Determine the alignment degree of each of the garment boundary texture feature points based on the product of the type weight corresponding to the garment boundary texture feature point and the distance, wherein the distance is inversely proportional to the alignment degree.

[0101] Alignment is an indicator that measures the degree of matching between the garment boundary texture feature points and the offset segmentation baseline. The closer the garment boundary texture feature points are to the segmentation baseline, the higher the type weight, and the higher the alignment.

[0102] For example, the vertical distance from each garment boundary texture feature point to the offset segmentation baseline is calculated. The alignment H is calculated using a preset formula: H = c × (1 / (1+d)). Here, c is the type weight, and d is the distance from the garment boundary texture feature point to the segmentation baseline.

[0103] Step S311: If the alignment of the garment boundary texture feature points is greater than a preset threshold, the garment boundary texture feature points are used as target fitting points, and the fitting curve based on the target fitting points is used as the cutting path.

[0104] It should be noted that the target fitting point is a garment boundary texture feature point with an alignment degree greater than a preset threshold, that is, an effective feature point with a high degree of matching with the offset segmentation baseline and which can be used as a reference for the cutting path.

[0105] For example, feature points with alignment values ​​greater than a preset threshold are retained, while low-reliability feature points with substandard alignment are removed to obtain target fitting points. The target fitting points are then sorted by coordinates in the horizontal direction, and curve fitting algorithms such as Bézier curves and polynomial curve fitting are used to fit the sorted target fitting points. The curve fitting algorithm generates a smooth trajectory line that fits most target fitting points by minimizing the overall deviation between the feature points and the fitted curve. This results in a smooth cutting path based on the garment's own boundary texture features and matching the offset segmentation baseline. This cutting path conforms to both the human skeletal boundaries and the garment's own structural texture. Specifically, the fitting method can refer to Embodiment 2 above, where the target fitting points are used as control points, and a curve is fitted using the Bézier curve algorithm.

[0106] Optionally, the pose features of the person in the image are determined based on the skeletal joint map; the curvature of the line connecting the target fitting points is determined based on the pose features; and the fitting curve of each target fitting point and curvature is used as the cropping path.

[0107] For example, key joints related to posture, such as the left and right shoulder joints, left and right hip joints, and left and right knee joints, are selected from the set of skeletal joints. The left and right tilt angles of the torso are calculated by the slope of the line connecting the midpoints of the left and right shoulder joints and the midpoints of the left and right hip joints. The pelvic rotation angle is determined based on the height difference and horizontal offset of the left and right hip joints. The curvature of the line connecting the coordinates of the neck, waist, and hip joints is used to determine whether the spine is straight, slightly curved, or laterally curved. Integrating the above parameters forms a set of posture features describing the current posture of the person in the image.

[0108] Optionally, the curvature direction of the line connecting the target fitting points is determined based on the left and right tilt angles of the torso. For example, if the left tilt angle of the torso is greater than 0, meaning the torso tilts to the left, the curvature on the left side of the curve is controlled to be greater than that on the right side to fit the tilted torso contour. Optionally, the curvature of the line connecting the target fitting points is determined based on the pelvic rotation angle, where the pelvic rotation angle is proportional to the curvature. For example, if the posture features show a large pelvic rotation angle, the curvature in the middle section of the curve is increased to conform to the transition between the abdomen and hips. If the pelvic rotation angle is small, the curvature is decreased to maintain a smooth path.

[0109] Optionally, a mapping relationship between each pose feature and the above curvature can be preset, and the curvature of the line connecting the target fitting points corresponding to the pose features of the person in the image can be found according to the mapping relationship.

[0110] In this embodiment, after obtaining the offset segmentation baseline based on skeletal joints and body shape features, the alignment degree between the garment boundary texture feature points on the garment surface and the baseline is calculated and effective target fitting points are selected. Then, the cutting path is fitted based on these target fitting points. Furthermore, the curvature of the curve can be determined by combining human posture features, so that the cutting path is more adapted to dynamic posture. This allows the cutting path to not only conform to the human skeleton and body shape, but also to the garment's own structural texture, further improving the accuracy and naturalness of garment segmentation.

[0111] Based on the above embodiments of this application, in the fourth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 After step S311, steps S312-S313 are also included:

[0112] Step S312: Display the garment boundary texture feature points and the cutting path on the segmentation mask.

[0113] Step S313: In response to the selection action of the garment boundary texture feature points on the segmentation mask, the curve is refitted based on the selected garment boundary texture feature points as the new cutting path.

[0114] For example, the virtual clothing try-on system loads a segmentation mask as the underlying image onto the interactive display interface, and overlays clothing boundary texture feature points onto the segmentation mask. For instance, it assigns differentiated visual identifiers to different types of clothing boundary texture feature points and marks their coordinate positions. Simultaneously, it overlays a cutting path onto the segmentation mask. The line width and color of the cutting path must be distinct from the clothing boundary texture feature points and the segmentation mask to ensure clear visibility.

[0115] Optionally, interface control elements such as zoom and pan buttons can be displayed simultaneously to allow users to adjust the viewing range and clearly observe the relative positional relationship between the garment boundary texture feature points and the cutting path.

[0116] The virtual clothing try-on system captures the user's selection action on the clothing boundary texture feature points on the segmentation mask through input devices such as mice and touch screens, and records the coordinates and type information of the selected clothing boundary texture feature points.

[0117] The user-selected garment boundary texture feature points are used as new control points. The cutting path is regenerated using the curve fitting method described in Example 3 above. The newly generated cutting path is then overlaid on the segmentation mask, replacing the original cutting path, for the user to confirm whether it meets expectations.

[0118] In this embodiment, by adding a user interaction optimization step, the garment boundary texture feature points and the generated cutting path are visualized and overlaid on the segmentation mask, providing viewing and control functions such as zooming and panning. At the same time, users can select garment boundary texture feature points through input devices to refit and generate new cutting paths, meeting the subjective needs of different scenarios, improving the accuracy and adaptability of cutting paths, and significantly enhancing the interactivity and user experience of the virtual try-on scenario.

[0119] Based on the above embodiments of this application, in the fifth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 After step S40, steps S50 to S70 are also included:

[0120] Step S50: In response to the selection operation of the sub-segmentation mask obtained after cropping, a replacement region is determined and marked in the target image based on the pixel coordinate boundaries of the selected sub-segmentation mask.

[0121] The virtual clothing try-on system responds to user selections, such as clicking to select a sub-segmentation mask for "top" or "bottom" on the graphical interface. It then acquires the selected sub-segmentation mask, performs contour detection on it (e.g., using an algorithm like the findContours class in the OpenCV library to traverse the segmentation mask image), and obtains a closed polygon composed of the pixel coordinates of the boundary points of the segmentation mask. This polygonal region defined by the boundary point coordinates is the precise region in the target image that needs to be replaced. The virtual clothing try-on system assigns a specific identifier to this region, either by generating a new mask corresponding to the polygonal region (e.g., a replacement region mask) or by internally marking the pixels of this region as the area to be edited.

[0122] Step S60: Obtain a replacement clothing image, and determine the scaling ratio of the replacement clothing image based on the spatial position of the joint points corresponding to the replacement area in the skeletal joint map.

[0123] For example, if the replacement area is a cropped upper garment area, its corresponding joint points may include shoulder joint points, neck joint points, etc., and the scaling ratio of the replacement garment image can be determined based on the distance from the left shoulder joint point to the right shoulder joint point, or the distance from the neck joint point to the shoulder joint point. If the replacement area is a cropped lower garment area, its corresponding joint points may include hip joint points, knee joint points, etc., and the scaling ratio of the replacement garment image can be determined based on the distance from the left hip joint point to the right hip joint point, or the distance from the hip joint point to the knee joint point.

[0124] Specifically, the scaling ratio can be determined by the ratio of the distance between the joints corresponding to the replacement area in the selected skeletal joint map and the size of the replacement clothing image, ensuring that the replacement clothing image matches the target person in size.

[0125] Step S70: After scaling the replacement clothing image according to the scaling ratio, use the replacement clothing image and the labeled target image as input to the pre-trained image restoration model to obtain the clothing replacement image output by the image restoration model.

[0126] For example, the virtual clothing try-on system takes a scaled-down image of the replacement clothing, overlays it onto the replacement area of ​​the target image based on the joints corresponding to the replacement area in the selected skeletal joint map, and then inputs it along with the target image into a pre-trained image inpainting model, such as LaMa (Large Mask Inpainting) or AOT (All-Of-Transformer for Image Inpainting) deep learning models. The image inpainting model first erases the original clothing pixels in the marked replacement area of ​​the target image, and then generates the content of the new replacement clothing image in a way that matches the lighting and shadow parameters of the surrounding area of ​​the replacement area in the target image, and outputs a complete changed clothing image.

[0127] In this embodiment, by responding to user operations, the replacement region in the image is precisely defined based on the contour of the sub-segmentation mask. Then, the human skeletal joints are intelligently used as a spatial scale to dynamically calculate the scaling ratio required for the replacement clothing, ensuring a natural match with the target person's body shape and posture. Finally, using an advanced image inpainting model, the scaled clothing image is deeply fused with the target image, automatically handling lighting coordination and edge transitions, ultimately generating a realistic, seamless image of the changed clothing. This solution seamlessly integrates clothing segmentation, size adaptation, and image generation technologies, significantly improving the automation level and visual realism of virtual try-on.

[0128] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the clothing image segmentation method based on human skeletal joints in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0129] This application provides a clothing image segmentation device based on human skeletal joints. The clothing image segmentation device based on human skeletal joints includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the clothing image segmentation method based on human skeletal joints in the above embodiment 1.

[0130] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of a clothing image segmentation device based on human skeletal joints suitable for implementing embodiments of this application. The clothing image segmentation device based on human skeletal joints in these embodiments may include, but is not limited to, mobile terminals such as laptops and tablets (PADs), and fixed terminals such as digital TVs and desktop computers. Figure 6 The clothing image segmentation device based on human skeletal joints shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 6As shown, the clothing image segmentation device based on human skeletal joints may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the clothing image segmentation device based on human skeletal joints. The processing unit 1001, the read-only memory 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the clothing image segmentation device based on human skeletal joints to exchange data with other devices wirelessly or via wired communication. Although the figure shows clothing image segmentation devices based on human skeletal joints with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0132] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0133] The clothing image segmentation device based on human skeletal joints provided in this application, employing the clothing image segmentation method based on human skeletal joints in the above embodiments, can solve the technical problem of how to improve the segmentation visualization effect of virtual character integrated clothing. Compared with the prior art, the beneficial effects of the clothing image segmentation device based on human skeletal joints provided in this application are the same as the beneficial effects of the clothing image segmentation method based on human skeletal joints provided in the above embodiments, and other technical features in the clothing image segmentation device based on human skeletal joints are the same as the features disclosed in the previous embodiment method, and will not be repeated here.

[0134] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0136] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the clothing image segmentation method based on human skeletal joints in the above embodiments.

[0137] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0138] The aforementioned computer-readable storage medium may be included in a clothing image segmentation device based on human skeletal joints; or it may exist independently and not be assembled into a clothing image segmentation device based on human skeletal joints.

[0139] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a clothing image segmentation device based on human skeletal joints, enable the device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, or as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0142] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described clothing image segmentation method based on human skeletal joints. This addresses the technical problem of improving the segmentation visualization effect of virtual character one-piece clothing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the clothing image segmentation method based on human skeletal joints provided in the above embodiments, and will not be repeated here.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the clothing image segmentation method based on human skeletal joints as described above.

[0144] The computer program product provided in this application solves the technical problem of how to improve the segmentation and visualization effect of virtual character integrated clothing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the clothing image segmentation method based on human skeletal joints provided in the above embodiments, and will not be repeated here.

[0145] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A garment image segmentation method based on human skeleton joints, characterized in that, The garment image segmentation method based on human body skeleton joints comprises: inputting a target image into a pre-trained human pose estimation model and a semantic segmentation model respectively, obtaining a skeleton joint map of an image character in the target image output by the human pose estimation model, and obtaining a segmentation mask of a worn garment of the image character output by the semantic segmentation model; determining a body shape feature of the image character according to the skeleton joint map, wherein the body shape feature comprises a proportional value between intervals of a body shape feature node pair; taking a left-right hip joint connection line in the skeleton joint map as a segmentation reference line, taking the proportional value between the intervals of the body shape feature node pair as an input parameter of a preset adjustment formula, calculating a vertical offset of the segmentation reference line corresponding to the proportional value through the preset adjustment formula, or determining a body shape type of the image character based on the proportional value between the intervals of the body shape feature node pair, and determining a vertical offset of the segmentation reference line corresponding to the body shape type in a preset adjustment mapping table based on the body shape type; wherein the vertical offset is a distance value that the segmentation reference line needs to be adjusted in the vertical direction; offsetting the segmentation reference line based on the vertical offset to obtain a cropping path; cropping the segmentation mask according to the cropping path to segment the worn garment.

2. The human pose based garment image segmentation method of claim 1, wherein, The step of determining a body shape feature of the image character according to the skeleton joint map comprises: determining a proportional value between intervals of at least two groups of body shape feature node pairs in the skeleton joint map based on a preset body shape feature node pair, wherein the body shape feature node pair at least comprises a left-right shoulder joint pair and a left-right hip joint pair; taking the proportional value between the intervals of the body shape feature node pair as the body shape feature.

3. The human pose based garment image segmentation method of claim 1, wherein, After the step of offsetting the segmentation reference line based on the vertical offset, the method further comprises: determining a curvature of the segmentation reference line according to a preset curvature calculation formula based on the proportional value between the intervals of the body shape feature node pair as an input of the preset curvature calculation formula; fitting the offset segmentation reference line into a curve according to the curvature to obtain the cropping path.

4. The human pose based garment image segmentation method of claim 1, wherein, After the step of offsetting the segmentation reference line based on the vertical offset, the method further comprises: obtaining garment boundary texture feature points of the worn garment; determining distances from each garment boundary texture feature point to the offset segmentation reference line; determining an alignment degree of each garment boundary texture feature point according to a product of a type weight corresponding to the garment boundary texture feature point and the distance, wherein the distance is inversely proportional to the alignment degree; if the alignment degree of the garment boundary texture feature point is greater than a preset threshold, taking the garment boundary texture feature point as a target fitting point, and fitting a curve according to the target fitting point as the cropping path.

5. The human pose based garment image segmentation method of claim 4, wherein, After the step of fitting a curve according to the target fitting point as the cropping path, the method further comprises: displaying the garment boundary texture feature points and the cropping path on the segmentation mask. In response to the selecting action of the garment boundary texture feature points on the segmentation mask, a curve is refitted according to the selected garment boundary texture feature points as a new cutting path.

6. The human-skeleton-joint-based garment image segmentation method of claim 1, wherein, After the step of segmenting the worn garment according to the cutting path, the method further comprises: In response to a selection operation on the sub-segmentation mask obtained after cutting, a replacement region is determined and marked in the target image based on the pixel coordinate boundary of the selected sub-segmentation mask; An image of a replacement garment is obtained, and a scaling ratio of the image of the replacement garment is determined based on the spatial positions of the joints corresponding to the replacement region in the joint graph; After scaling the image of the replacement garment according to the scaling ratio, the image of the replacement garment and the target image after marking are input into a pre-trained image inpainting model to obtain a refitted image.

7. A human body bone joint-based garment image segmentation device, characterized by, The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the garment image segmentation method based on human body joints according to any one of claims 1 to 6.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the garment image segmentation method based on human body joints according to any one of claims 1 to 6.

Images