Jewelry try-on method and device, electronic equipment and storage medium

Abstract

The invention discloses a jewelry try-on method and device, electronic equipment and a storage medium. The method comprises the following steps: firstly, acquiring an original image containing a hand of a user; carrying out image segmentation on the hand in the original image, and extracting the contour of the hand in the original image to obtain a hand image; then, key point detection is carried out on the original image, and a plurality of hand key points are determined; according to the position information of the hand key points and the hand image, determining the wearing direction and the wearing point position of the jewelry and the inner diameter length of the jewelry; according to the inner diameter length of the jewelry, zooming the image of the jewelry to be worn to obtain a zoomed jewelry image; and finally, according to the wearing direction and the wearing point position, synthesizing the zoomed jewelry image with the original image to generate an effect picture of jewelry try-on. According to the invention, the effect picture of the jewelry try-on can be generated, so that the user can realize the try-on effect without going to a physical store.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and particularly to a method, device, electronic device and storage medium for trying on jewelry. Background Art

[0002] In recent years, with the rapid progress of society and the continuous enhancement of national strength, online shopping has become a new shopping mode and its development is getting hotter. When shopping for jewelry online, users cannot try it on at the scene and cannot intuitively feel the wearing effect. Relying solely on a few pre-beautified jewelry pictures obviously cannot meet the needs of users. Therefore, how to enable users to intuitively feel the wearing effect of jewelry when shopping online is an urgent problem to be solved. Summary of the Invention

[0003] Embodiments of the present invention provide a method, device, electronic device and storage medium for trying on jewelry, which can realize the try-on of jewelry based on image processing, enabling users to intuitively feel the wearing effect of jewelry without going to a physical store.

[0004] An embodiment of the present invention provides a method for trying on jewelry, including:

[0005] Obtaining an original image containing a user's hand; wherein, the original image includes: a first original image containing a finger region, or a second original image containing a wrist region;

[0006] Performing image segmentation on the hand in the original image, extracting the hand contour in the original image to obtain a hand image; wherein, when the original image is the first original image, the hand image is a first hand image containing a finger region; when the original image is the second original image, the hand image is a second hand image containing a wrist region;

[0007] Performing key point detection on the original image to determine a plurality of hand key points; wherein, when the original image is the first original image, the hand key points include: finger key points of each finger; each finger key point includes: a metacarpophalangeal joint key point and a proximal interphalangeal joint key point; when the original image is the second original image, the hand key points include: the metacarpophalangeal joint key point of the middle finger and the wrist joint key point;

[0008] When the jewelry wearing area is a finger, determining the wearing direction, wearing point position and inner diameter length of the jewelry according to the position information of the metacarpophalangeal joint key point and the proximal interphalangeal joint key point of the finger to be worn and the first hand image;

[0009] When the jewelry wearing area is the wrist, determine the wearing direction, wearing point position and inner diameter length of the jewelry according to the key points of the metacarpophalangeal joint of the middle finger, the key points of the wrist joint and the second hand image;

[0010] According to the inner diameter length of the jewelry, scale the image of the jewelry to be worn to obtain a scaled jewelry image;

[0011] According to the wearing direction and wearing point position, synthesize the scaled jewelry image with the original image to generate an effect picture of trying on the jewelry.

[0012] Furthermore, when the jewelry wearing area is a finger, determine the wearing direction of the jewelry in the following way;

[0013] Connect the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn to obtain a first connecting line segment;

[0014] According to the positions of the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn, calculate the slope of the first connecting line segment in the image coordinate system of the first original image;

[0015] Trisect the first connecting line segment, and take the trisection point adjacent to the key point of the proximal interphalangeal joint of the finger to be worn as the target trisection point;

[0016] Draw a perpendicular line to the first connecting line segment through the target trisection point to obtain a first perpendicular line;

[0017] According to the slope of the first connecting line segment in the image coordinate system of the first original image, calculate the slope of the first perpendicular line in the image coordinate system of the first original image;

[0018] Determine the wearing direction of the jewelry according to the slope of the first perpendicular line in the image coordinate system of the first original image.

[0019] Furthermore, when the jewelry wearing area is a finger, determine the wearing point position of the jewelry in the following way;

[0020] Intersect the first hand image with the first perpendicular line, and intercept the line segment of the first perpendicular line within the area of the finger to be worn to obtain a first line segment;

[0021] Take the position of the midpoint of the first line segment as the wearing point position of the jewelry.

[0022] Furthermore, when the jewelry wearing area is a finger, determine the inner diameter length of the jewelry in the following way;

[0023] Calculate the line segment length of the first line segment, and take the line segment length of the first line segment as the inner diameter length of the jewelry.

[0024] Further, when the jewelry wearing area is the wrist, the wearing direction of the jewelry is determined by the following method;

[0025] Connect the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint to obtain a second connecting line segment;

[0026] Extend the second connecting line segment a preset distance along the direction from the key points of the metacarpophalangeal joint of the middle finger to the key points of the wrist joint to obtain a first extension line; wherein the preset distance is two-thirds of the length of the second connecting line segment;

[0027] According to the positions of the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint, calculate the slope of the first extension line in the image coordinate system of the second original image;

[0028] Determine the endpoint of the first extension line; draw a perpendicular line to the first extension line through the endpoint of the first extension line to obtain a second perpendicular line;

[0029] According to the slope of the first extension line in the image coordinate system of the second original image, calculate the slope of the second perpendicular line in the image coordinate system of the second original image;

[0030] Determine the wearing direction of the jewelry according to the slope of the second perpendicular line in the image coordinate system of the second original image.

[0031] Further, when the jewelry wearing area is the wrist, the wearing point position of the jewelry is determined by the following method:

[0032] Intersect the second hand image with the second perpendicular line and intercept the line segment of the second perpendicular line within the wrist area to obtain a second line segment;

[0033] Take the position of the midpoint of the second line segment as the wearing point position of the jewelry.

[0034] Further, when the jewelry wearing area is the wrist, the inner diameter length of the jewelry is determined by the following method:

[0035] Calculate the line segment length of the second line segment, and take the line segment length of the second line segment as the inner diameter length of the jewelry.

[0036] Based on the above method item embodiments, the present invention correspondingly provides device item embodiments:

[0037] An embodiment of the present invention provides a jewelry trial wearing device, including: an image acquisition module, an image segmentation module, a key point detection module, a first wearing information determination module, a second wearing information determination module, a jewelry image scaling module, and an image synthesis module;

[0038] The image acquisition module is used to acquire the original image containing the user's hand; wherein, the original image includes: a first original image containing a finger region, or a second original image containing a wrist region;

[0039] The image segmentation module is used to perform image segmentation on the hand in the original image, extract the hand contour in the original image, and obtain a hand image; wherein, when the original image is the first original image, the hand image is a first hand image containing a finger region; when the original image is the second original image, the hand image is a second hand image containing a wrist region;

[0040] The key point detection module is used to perform key point detection on the original image to determine several hand key points; wherein, when the original image is the first original image, the hand key points include: the finger key points of each finger; each finger key point includes: a metacarpophalangeal joint key point and a proximal interphalangeal joint key point; when the original image is the second original image, the hand key points include: the metacarpophalangeal joint key point of the middle finger and the wrist joint key point;

[0041] The first wearing information determination module is used to, when the jewelry wearing area is a finger, determine the wearing direction, wearing point position, and inner diameter length of the jewelry according to the position information of the metacarpophalangeal joint key point and the proximal interphalangeal joint key point of the finger to be worn and the first hand image;

[0042] The second wearing information determination module is used to, when the jewelry wearing area is the wrist, determine the wearing direction, wearing point position, and inner diameter length of the jewelry according to the metacarpophalangeal joint key point of the middle finger, the wrist joint key point, and the second hand image;

[0043] The jewelry image scaling module is used to scale the image of the jewelry to be worn according to the inner diameter length of the jewelry to obtain a scaled jewelry image;

[0044] The image synthesis module is used to synthesize the scaled jewelry image with the original image according to the wearing direction and wearing point position to generate an effect picture of trying on the jewelry.

[0045] Based on the above method item embodiments, the present invention correspondingly provides an electronic device item embodiment;

[0046] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the jewelry try-on method described in any one of the present invention.

[0047] Based on the above method embodiment, the present invention correspondingly provides a storage medium embodiment;

[0048] An embodiment of the present invention provides a storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the storage medium is located to execute the jewelry try-on method described in any one of the present invention.

[0049] By implementing the embodiments of the present invention, the following beneficial effects are achieved:

[0050] The embodiments of the present invention provide a jewelry try-on method, device, electronic device and storage medium. The method first obtains an original image containing the user's hand; the original image includes: a first original image containing a finger area or a second original image containing a wrist area; then, it performs image segmentation on the hand in the original image, extracts the hand contour in the original image to obtain a hand image; when the original image is the first original image, the hand image is a first hand image containing a finger area; when the original image is the second original image, the hand image is a second hand image containing a wrist area; then, it performs key point detection on the original image to determine several hand key points; when the hand image is the first hand image, the hand key points include: the finger key points of each finger; each finger key point includes: a metacarpophalangeal joint key point and a proximal interphalangeal joint key point; when the hand image is the second hand image, the hand key points include: the metacarpophalangeal joint key point of the middle finger and the wrist joint key point; then, when the jewelry wearing area is the finger, according to the position information of the metacarpophalangeal joint key point and the proximal interphalangeal joint key point of the finger to be worn and the first hand image, determine the wearing direction, wearing point position and inner diameter length of the jewelry; when the jewelry wearing area is the wrist, according to the metacarpophalangeal joint key point of the middle finger, the wrist joint key point and the second hand image, determine the wearing direction, wearing point position and inner diameter length of the jewelry; according to the inner diameter length of the jewelry, scale the image of the jewelry to be worn to obtain a scaled jewelry image; finally, according to the wearing direction and wearing point position, synthesize the scaled jewelry image with the original image to generate an effect diagram of jewelry try-on. That is, based on the original image containing the user's hand, the present invention determines the wearing direction, wearing point position and inner diameter length of the jewelry in the original image, and then synthesizes the image of the jewelry into the original image according to the above wearing direction, wearing point position and inner diameter length of the jewelry, so as to generate an effect diagram of jewelry try-on, enabling the user to achieve the try-on effect without going to a physical store. Description of the Drawings

[0051] Figure 1 It is a schematic flowchart of the jewelry try-on method provided by an embodiment of the present invention.

[0052] Figure 2It is a schematic diagram of the first original image provided by an embodiment of the present invention.

[0053] Figure 3 It is a schematic diagram of the second original image provided by an embodiment of the present invention.

[0054] Figure 4 It is a schematic diagram of the first hand image provided by an embodiment of the present invention.

[0055] Figure 5 It is a schematic diagram of the second hand image provided by an embodiment of the present invention.

[0056] Figure 6 It is a schematic diagram of the hand key points corresponding to the first hand image provided by an embodiment of the present invention.

[0057] Figure 7 It is a schematic diagram of the hand key points corresponding to the second hand image provided by an embodiment of the present invention.

[0058] Figure 8 It is a schematic diagram of the principle for determining the wearing direction of the jewelry in an embodiment of the present invention.

[0059] Figure 9 It is a schematic diagram of the principle for determining the wearing point position and the inner diameter length of the jewelry in an embodiment of the present invention.

[0060] Figure 10 It is a schematic diagram of the principle for determining the wearing direction of the jewelry in another embodiment of the present invention.

[0061] Figure 11 It is a schematic diagram of the principle for determining the wearing point position and the inner diameter length of the jewelry in another embodiment of the present invention.

[0062] Figure 12 It is a schematic diagram of the image of the jewelry provided by an embodiment of the present invention.

[0063] Figure 13 It is an effect diagram of the trial wearing of the jewelry provided by an embodiment of the present invention.

[0064] Figure 14 It is an effect diagram of the trial wearing of the jewelry provided by another embodiment of the present invention.

[0065] Figure 15 It is a schematic structural diagram of the jewelry trial wearing device provided by an embodiment of the present invention. Detailed implementation manners

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0067] As Figure 1 shown, the embodiment of the present invention provides a method for trying on jewelry, which at least includes:

[0068] Step S1: Obtain an original image containing the user's hand; wherein, the original image includes a first original image containing a finger region or a second original image containing a wrist region.

[0069] Specifically, the jewelry to be worn in the present invention can be divided into two types. One is the jewelry worn on the finger, such as a ring; the other is the jewelry worn on the wrist, such as a bracelet, a bangle, and a bracelet, etc. To achieve the try-on effect of the above two types of jewelry, it is necessary to obtain the original images containing different hand regions according to the actual situation. If the try-on effect of the jewelry worn on the finger needs to be achieved, then at this time, it is necessary to obtain the first original image containing the finger region. Schematically, the first original image is as Figure 2 shown; if the try-on effect of the jewelry worn on the finger needs to be achieved, then at this time, it is necessary to obtain the second original image containing the wrist region. Schematically, the second original image is as Figure 3 shown.

[0070] Step S2: Segment the hand in the original image, extract the hand contour in the original image, and obtain a hand image; wherein, when the original image is the first original image, the hand image is the first hand image containing the finger region; when the original image is the second original image, the hand image is the second hand image containing the wrist region.

[0071] Specifically, the existing fast segmentation model FastSAM can be used to quickly segment the original image uploaded by the user, and segment the hand region from the original image to obtain a hand image.

[0072] Schematically, when the original image is the above-mentioned first original image, the obtained first hand image is as Figure 4 shown. When the original image is the above-mentioned second original image, the obtained second hand image is as Figure 5 shown.

[0073] Step S3: Perform key point detection on the original image to determine a number of hand key points. Among them, when the original image is the first original image, the hand key points include: the finger key points of each finger; each finger key point includes: the metacarpophalangeal joint key point and the proximal interphalangeal joint key point; when the original image is the second original image, the hand key points include: the metacarpophalangeal joint key point of the middle finger and the wrist joint key point.

[0074] Preferably, a hand vision model, such as midiapipe, can be used to perform key point detection on the original image and determine a number of hand key points.

[0075] Such as Figure 6 and Figure 7 As shown, the following hand key points can be recognized by the hand vision model: 0 is the wrist joint key point, 1 is the metacarpophalangeal joint key point of the thumb, 2 is the proximal interphalangeal joint key point of the thumb, 3 is the distal interphalangeal joint key point of the thumb, 4 is the fingertip key point of the thumb, 5 is the metacarpophalangeal joint key point of the index finger, 6 is the proximal interphalangeal joint key point of the index finger, 7 is the distal interphalangeal joint key point of the index finger, 8 is the fingertip key point of the index finger, 9 is the metacarpophalangeal joint key point of the middle finger, 10 is the proximal interphalangeal joint key point of the middle finger, 11 is the distal interphalangeal joint key point of the middle finger, 12 is the fingertip key point of the middle finger, 13 is the metacarpophalangeal joint key point of the ring finger, 14 is the proximal interphalangeal joint key point of the ring finger, 15 is the distal interphalangeal joint key point of the ring finger, 16 is the fingertip key point of the ring finger, 17 is the metacarpophalangeal joint key point of the little finger, 18 is the proximal interphalangeal joint key point of the little finger, 19 is the distal interphalangeal joint key point of the little finger, 20 is the fingertip key point of the little finger.

[0076] Preferably, when the original image is the first original image, only the metacarpophalangeal joint key point and the proximal interphalangeal joint key point can be selected, that is, only select the key points: 1, 2, 5, 6, 9, 10, 13, 14, 17 and 18. When the original image is the second original image, only the metacarpophalangeal joint key point of the middle finger and the wrist joint key point can be selected, that is, only select the key points: 9 and 0.

[0077] Step S4: When the jewelry wearing area is a finger, determine the wearing direction, wearing point position and inner diameter length of the jewelry according to the position information of the metacarpophalangeal joint key point and the proximal interphalangeal joint key point of the finger to be worn and the first hand image.

[0078] In a preferred embodiment, when the jewelry wearing area is a finger, the wearing direction of the jewelry is determined as follows: Connect the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn to obtain a first connection line segment; Calculate the slope of the first connection line segment in the image coordinate system of the first original image according to the positions of the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn; Trisect the first connection line segment, and use the trisection point adjacent to the key point of the proximal interphalangeal joint of the finger to be worn as the target trisection point; Draw a perpendicular line to the first connection line segment through the target trisection point to obtain a first perpendicular line; Calculate the slope of the first perpendicular line in the image coordinate system of the first original image according to the slope of the first connection line segment in the image coordinate system of the first original image; Determine the wearing direction of the jewelry according to the slope of the first perpendicular line in the image coordinate system of the first original image.

[0079] Schematically, as Figure 8 shown, taking the example of wearing a ring on the index finger, connect the key point 5 of the metacarpophalangeal joint of the index finger and the key point 6 of the proximal interphalangeal joint to obtain a first connection line segment;

[0080] Immediately afterwards, according to the position coordinates of the key point 5 of the metacarpophalangeal joint of the index finger and the key point 6 of the proximal interphalangeal joint of the index finger, calculate the slope m of the first connection line segment in the image coordinate system of the first original image; It should be noted that for the image coordinate system of the first original image, the X-axis is constructed in the horizontal direction of the first original image, and the y-axis is constructed in the vertical direction of the first original image. Immediately afterwards, trisect the first connection line segment, use the trisection point adjacent to the key point 6 of the proximal interphalangeal joint of the index finger as the target trisection point, and draw a perpendicular line to the first connection line segment through the target trisection point to obtain the above-mentioned first perpendicular line. At this time, the slope m of the first perpendicular line ⊥ is

[0081] At this time, the angle θ formed by the first perpendicular line and the X-axis is: θ = atan(m ⊥ ) * (180 / π);

[0082] Take the calculated angle θ as the wearing direction of the jewelry, that is, the angle by which the ring needs to be rotated when wearing the ring on the index finger subsequently.

[0083] In a preferred embodiment, when the jewelry wearing area is a finger, the position of the wearing point of the jewelry is determined as follows: Intersect the first hand image with the first perpendicular line, and intercept the line segment of the first perpendicular line within the finger area to be worn to obtain a first line segment; Take the position of the midpoint of the first line segment as the position of the wearing point of the jewelry.

[0084] Schematically, as Figure 9As shown, still taking wearing a ring on the index finger as an example, the first hand image is intersected with the first vertical line, and then the line segment of the first vertical line within the index finger area is intercepted to obtain the above-mentioned first line segment, that is, the line segment AB in the figure. The midpoint of the line segment AB is taken to obtain the wearing point position of the jewelry. The above-mentioned wearing point position is the position where the center point of the ring is located when the ring is worn on the index finger. This can make the center point of the ring located in the middle of the finger and prevent the ring from being worn crooked.

[0085] In a preferred embodiment, further, when the jewelry wearing area is a finger, the inner diameter length of the jewelry is determined by the following method; calculate the line segment length of the first line segment, and use the line segment length of the first line segment as the inner diameter length of the jewelry.

[0086] Schematically, that is, the length of the line segment AB is used as the inner diameter length of the ring when the ring is worn on the index finger.

[0087] Step S5: When the jewelry wearing area is the wrist, determine the wearing direction, wearing point position and inner diameter length of the jewelry according to the key points of the metacarpophalangeal joint of the middle finger, the key points of the wrist joint and the second hand image.

[0088] In a preferred embodiment, when the jewelry wearing area is the wrist, the wearing direction of the jewelry is determined by the following method; connect the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint to obtain a second connecting line segment; extend the second connecting line segment a preset distance along the direction from the key points of the metacarpophalangeal joint of the middle finger to the key points of the wrist joint to obtain a first extension line; wherein the preset distance is two-thirds of the length of the second connecting line segment; according to the positions of the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint, calculate the slope of the first extension line in the image coordinate system of the second original image; determine the end point of the first extension line; draw a perpendicular line to the first extension line through the end point of the first extension line to obtain a second perpendicular line; calculate the slope of the second perpendicular line in the image coordinate system of the second original image according to the slope of the first extension line in the image coordinate system of the second original image; determine the wearing direction of the jewelry according to the slope of the second perpendicular line in the image coordinate system of the second original image.

[0089] Schematically, as Figure 10 shown, connect the key point 9 of the metacarpophalangeal joint of the middle finger and the key point 0 of the wrist joint to obtain the above-mentioned second connecting line segment, and then extend the second connecting line segment along the direction from the key point of the metacarpophalangeal joint of the middle finger to the key point of the wrist joint, that is, extend two-thirds of its own length along the wrist direction to obtain a first extension line;

[0090] Next, calculate the slope of the first extension line in the image coordinate system of the second original image. Similarly, for the image coordinate system of the second original image, the X-axis is constructed in the horizontal direction of the second original image, and the y-axis is constructed in the vertical direction of the second original image. Next, determine the endpoint of the first extension line, that is, Figure 10 point C in, and then draw a perpendicular line to the first extension line through the endpoint C to obtain the second perpendicular line. Similarly, based on the slope of the first extension line, determine the slope of the second perpendicular line in the image coordinate system of the second original image, and then determine the angle of the second perpendicular line relative to the positive direction of the X-axis in the image coordinate system of the second original image according to the slope of the second perpendicular line in the image coordinate system of the second original image. Take the calculated angle as the wearing direction of the jewelry, that is, when the bracelet (or other accessories worn on the wrist) is worn on the wrist later, the angle by which the bracelet needs to be rotated. The calculation principle of the specific angle is the same as that in the previous step when calculating the angle by which the ring needs to be rotated when worn on the index finger, and will not be elaborated here.

[0091] In a preferred embodiment, when the jewelry wearing area is the wrist, the position of the jewelry wearing point is determined in the following way: intersect the second hand image with the second perpendicular line, and intercept the line segment of the second perpendicular line within the wrist area to obtain the second line segment; take the position of the midpoint of the second line segment as the position of the jewelry wearing point.

[0092] Schematically, as Figure 11 shown, taking the example of wearing a bracelet on the wrist, intersect the second hand image and the second perpendicular line, and then intercept the line segment of the second perpendicular line within the wrist area to obtain the above-mentioned second line segment, that is, the line segment DE in the figure. Take the midpoint of the line segment DE to obtain the wearing point position of the bracelet. The above-mentioned wearing point position is the position where the center point of the bracelet is located when the bracelet is worn on the wrist. This can make the center point of the bracelet located in the middle of the wrist and prevent the bracelet from being worn crooked.

[0093] In a preferred embodiment, when the jewelry wearing area is the wrist, the inner diameter length of the jewelry is determined in the following way: calculate the line segment length of the second line segment, and take the line segment length of the second line segment as the inner diameter length of the jewelry. Schematically, take the length of the above-mentioned line segment DE as the inner diameter length of the bracelet when the bracelet is worn on the wrist.

[0094] Step S6: Scale the image of the jewelry to be worn according to the inner diameter length of the jewelry to obtain a scaled jewelry image.

[0095] Specifically, after obtaining the inner diameter length of the jewelry through the above steps, after obtaining the corresponding jewelry image from the preset database, then scale the jewelry image so that the inner diameter of the jewelry in the scaled jewelry image is consistent with the inner diameter length calculated in the above steps to obtain a scaled jewelry image.

[0096] Step S7: Synthesize the scaled jewelry image with the original image according to the wearing direction and the position of the wearing point to generate an effect diagram of trying on the jewelry.

[0097] Specifically, after rotating the scaled jewelry image according to the wearing direction, the center of the jewelry in the rotated jewelry image is corresponding to the wearing point, and then it is synthesized into the original image to generate the final effect diagram of trying on the jewelry.

[0098] Schematically, when the jewelry is a ring, the image of the jewelry to be worn corresponding to it is as Figure 12 shown. It should be noted that initially, the images of all the jewelry to be worn are placed horizontally. After synthesis, the generated effect diagram of trying on the jewelry is as Figure 13 shown.

[0099] When the jewelry is a bracelet, after synthesis, the generated effect diagram of trying on the jewelry is as Figure 14 shown.

[0100] By implementing the present invention, based on the original image including the user's hand, the wearing direction, the position of the wearing point and the inner diameter length of the jewelry in the original image are determined, and then the image of the jewelry is synthesized into the original image according to the above-mentioned wearing direction, the position of the wearing point and the inner diameter length of the jewelry, so as to generate an effect diagram of trying on the jewelry, enabling the user to achieve the effect of trying on without going to a physical store.

[0101] In an optional embodiment, a model can be used for training. By collecting sufficient hand pictures and manually marking the wearing position coordinates and the planar rotation angles of the jewelry (including fingers and wrists) for each picture, a model that can be used to identify the wearing position of the jewelry in the picture is trained by means of pre-training or fine-tuning based on an existing model.

[0102] In an optional embodiment, a model is used for training. Sufficient hand pictures are collected, key point detection is performed on each hand picture, and all key point arrays are used as inputs, and the wearing position and rotation angle are used as outputs. By means of pre-training or fine-tuning based on an existing model, a model that can be used to identify the wearing position from the picture or hand key point data is trained.

[0103] Based on the above method embodiment, the present invention correspondingly provides an apparatus embodiment;

[0104] As Figure 14 shown, an embodiment of the present invention provides a jewelry try-on apparatus, including: an image acquisition module, an image segmentation module, a key point detection module, a first wearing information determination module, a second wearing information determination module, a jewelry image scaling module, and an image synthesis module;

[0105] The image acquisition module is used to acquire the original image containing the user's hand; wherein, the original image includes: a first original image containing a finger region, or a second original image containing a wrist region;

[0106] The image segmentation module is used to perform image segmentation on the hand in the original image, extract the hand contour in the original image, and obtain a hand image; wherein, when the original image is the first original image, the hand image is a first hand image containing a finger region; when the original image is the second original image, the hand image is a second hand image containing a wrist region;

[0107] The key point detection module is used to perform key point detection on the original image to determine a number of hand key points; wherein, when the original image is the first original image, the hand key points include: the finger key points of each finger; each finger key point includes: a metacarpophalangeal joint key point and a proximal interphalangeal joint key point; when the original image is the second original image, the hand key points include: the metacarpophalangeal joint key point of the middle finger and the wrist joint key point;

[0108] The first wearing information determination module is used to, when the jewelry wearing area is a finger, determine the wearing direction, wearing point position, and inner diameter length of the jewelry according to the position information of the metacarpophalangeal joint key point and the proximal interphalangeal joint key point of the finger to be worn and the first hand image;

[0109] The second wearing information determination module is used to, when the jewelry wearing area is the wrist, determine the wearing direction, wearing point position, and inner diameter length of the jewelry according to the metacarpophalangeal joint key point of the middle finger and the wrist joint key point and the second hand image;

[0110] The jewelry image scaling module is used to scale the image of the jewelry to be worn according to the inner diameter length of the jewelry to obtain a scaled jewelry image;

[0111] The image synthesis module is used to synthesize the scaled jewelry image with the original image according to the wearing direction and wearing point position to generate an effect picture of trying on the jewelry.

[0112] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity, the specific working process of the above-described device can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0113] Based on the foregoing embodiments of the method items, the present invention correspondingly provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the above-described jewelry try-on method of the present invention is implemented.

[0114] The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the electronic device and connects various parts of the entire electronic device through various interfaces and circuits.

[0115] The memory can be used to store the computer program. By running or executing the computer program stored in the memory and invoking the data stored in the memory, the processor realizes various functions of the electronic device. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, memory, plug-in hard disks, Smart Media Cards (SMCs), Secure Digital (SD) cards, Flash Cards, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

[0116] Based on the embodiments of the above method items, the present invention correspondingly provides a storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the storage medium is located to execute the jewelry try-on method according to any one of the present invention.

[0117] The storage medium mentioned here is a computer-readable storage medium. If the modules / units integrated in the jewelry try-on method are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of the present invention, it can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be realized. Among them, the computer program includes computer program code. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0118] The above is the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A jewelry trying-on method, characterized in that: include: Acquire an original image containing a user's hand; wherein the original image includes: a first original image containing a finger area, or a second original image containing a wrist area; Performing image segmentation on the hand in the original image, extracting the hand contour in the original image, and obtaining a hand image; wherein, when the original image is a first original image, the hand image is a first hand image containing a finger area; and when the original image is a second original image, the hand image is a second hand image containing a wrist area; Perform key point detection on the original image to determine a number of hand key points; wherein, when the original image is a first original image, the hand key points include: finger key points of each finger; each finger key point includes: a metacarpophalangeal joint key point and a proximal interphalangeal joint key point; when the original image is a second original image, the hand key points include: a metacarpophalangeal joint key point of the middle finger and a wrist joint key point; When the jewelry wearing area is a finger, the wearing direction, the wearing point position and the inner diameter length of the jewelry are determined according to the position information of the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn and the first hand image; When the jewelry wearing area is the wrist, the wearing direction, the wearing point position and the inner diameter length of the jewelry are determined according to the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint and the second hand image; Scaling the image of the jewelry to be worn according to the inner diameter length of the jewelry to obtain a scaled jewelry image; According to the wearing direction and the wearing point position, the scaled jewelry image is synthesized with the original image to generate a jewelry try-on effect image.

2. The jewelry trying-on method according to claim 1, characterized in that: When the jewelry wearing area is a finger, the jewelry wearing direction is determined by the following method; Connecting the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn to obtain a first connecting line segment; Calculate the slope of the first connecting line segment in the image coordinate system of the first original image according to the positions of the key points of the metacarpophalangeal joint and the key points of the proximal interphalangeal joint of the finger to be worn; Divide the first connecting line segment into three equal parts, and use the three-division point adjacent to the key point of the proximal interphalangeal joint of the finger to be worn as the target three-division point; Draw a perpendicular line through the target trisection point to the first connecting line segment to obtain a first perpendicular line; Calculating the slope of the first vertical line in the image coordinate system of the first original image according to the slope of the first connecting line segment in the image coordinate system of the first original image; The wearing direction of the jewelry is determined according to the slope of the first vertical line in the image coordinate system of the first original image.

3. The jewelry trying-on method according to claim 2, characterized in that: When the jewelry wearing area is a finger, the jewelry wearing point position is determined by the following method; Intersecting the first hand image with the first vertical line, and intercepting a line segment of the first vertical line in the finger region to be worn, to obtain a first line segment; The midpoint of the first line segment is used as the wearing point of the jewelry.

4. The jewelry trying-on method according to claim 3, characterized in that: When the jewelry is worn on the finger, the inner diameter of the jewelry is determined by the following method; The length of the first line segment is calculated, and the length of the first line segment is used as the inner diameter length of the jewelry.

5. The jewelry trying-on method according to claim 4, characterized in that: When the jewelry is worn on the wrist, the jewelry wearing direction is determined by the following method; Connect the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint to obtain a second connecting line segment; Extending the second connecting line segment by a preset distance in the direction from the key point of the metacarpophalangeal joint of the middle finger to the key point of the wrist joint to obtain a first extension line; wherein the preset distance is two-thirds of the length of the second connecting line segment; Calculating the slope of the first extension line in the image coordinate system of the second original image according to the positions of the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint; determining endpoints of the first extension line; Draw a perpendicular line through the endpoint of the first extension line to obtain a second perpendicular line; Calculating the slope of the second vertical line in the image coordinate system of the second original image according to the slope of the first extended line in the image coordinate system of the second original image; The wearing direction of the jewelry is determined according to the slope of the second vertical line in the image coordinate system of the second original image.

6. The jewelry trying-on method according to claim 5, characterized in that: When the jewelry is worn on the wrist, the jewelry wearing point is determined by the following method: Intersecting the second hand image with the second vertical line, and intercepting a line segment of the second vertical line in the wrist area to obtain a second line segment; The midpoint of the second line segment is used as the wearing point of the jewelry.

7. The jewelry trying-on method according to claim 6, characterized in that: When the jewelry is worn on the wrist, determine the inner diameter of the jewelry as follows: The length of the second line segment is calculated, and the length of the second line segment is used as the inner diameter length of the jewelry.

8. A jewelry fitting device, characterized in that: include: An image acquisition module, an image segmentation module, a key point detection module, a first wearing information determination module, a second wearing information determination module, a jewelry image scaling module, and an image synthesis module; The image acquisition module is used to acquire an original image containing a user's hand; wherein the original image includes: a first original image containing a finger area, or a second original image containing a wrist area; The image segmentation module is used to perform image segmentation on the hand in the original image, extract the hand contour in the original image, and obtain a hand image; wherein, when the original image is a first original image, the hand image is a first hand image containing a finger area; when the original image is a second original image, the hand image is a second hand image containing a wrist area; The key point detection module is used to perform key point detection on the original image to determine a number of hand key points; wherein, when the original image is a first original image, the hand key points include: finger key points of each finger; each finger key point includes: a metacarpophalangeal joint key point and a proximal interphalangeal joint key point; when the original image is a second original image, the hand key points include: a metacarpophalangeal joint key point of the middle finger and a wrist joint key point; The first wearing information determination module is used to determine the wearing direction, wearing point position and inner diameter length of the jewelry according to the position information of the key points of the metacarpophalangeal joint and the proximal interphalangeal joint of the finger to be worn and the first hand image when the jewelry wearing area is a finger; The second wearing information determination module is used to determine the wearing direction, wearing point position and inner diameter length of the jewelry according to the key points of the metacarpophalangeal joint of the middle finger and the key points of the wrist joint and the second hand image when the jewelry wearing area is the wrist; The jewelry image scaling module is used to scale the image of the jewelry to be worn according to the inner diameter length of the jewelry to obtain a scaled jewelry image; The image synthesis module is used to synthesize the scaled jewelry image with the original image according to the wearing direction and the wearing point position to generate a jewelry try-on effect image.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the jewelry trying-on method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is executed, the device where the storage medium is located is controlled to execute the jewelry try-on method according to any one of claims 1 to 7.

Images