Method, device and medium for segmenting and optimizing jewelry images in test certificates

Abstract

The embodiment of the present invention relates to the field of computer vision and information technology, and discloses a method, device and medium for segmenting and optimizing jewelry pictures in a detection certificate. The method comprises: collecting jewelry pictures, framing part of the jewelry pictures and attaching labels of corresponding types to obtain pictures to be trained; training a target detection model based on the pictures to be trained to obtain a trained target detection model; detecting the jewelry pictures using the trained target detection model to obtain pictures to be segmented; inputting the pictures to be segmented into an image segmentation model, identifying the picture type of the pictures to be segmented using the image segmentation model, and marking the pictures according to the picture types; performing image segmentation based on the marked pictures to obtain segmented pictures; optimizing the segmented pictures using a predetermined optimization algorithm, and uploading the optimized pictures to the detection certificate. Through the above-mentioned manner, the embodiment of the present invention can improve the accuracy and efficiency of jewelry picture segmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision and information technology, and specifically to a method, device and medium for segmenting and optimizing jewelry images in a detection certificate. Background Art

[0002] In the field of jewelry testing, there are high requirements for jewelry images on the test certificate. Generally, due to factors such as background and angle, the pictures taken cannot be used directly on the test certificate. The pictures need to be segmented and beautified before they can be used. Image segmentation and beautification can be achieved using PS, but it takes a lot of manpower and time. It usually takes 1-6 hours for front-line jewelry testers to complete the fine-tuning of a picture. Faced with tens of thousands of pictures every day, manual PS processing is inefficient.

[0003] At present, graphic segmentation models have been applied in many fields, but basically these models are proprietary models, which have good performance in segmenting images in this field, but often have poor performance in segmenting images in other fields. This is because traditional image segmentation models usually focus on specific fields, use images in their specific fields to train models, and do not use images from other fields for training, resulting in limited generalization ability of the model. Especially in the field of jewelry, the inventors found in the research and development that the uses and complexity of jewelry pictures collected from various data sources vary greatly, and the pixel differences and scale differences of the pictures are also large. Most of them are not suitable for jewelry detection and cannot meet the requirements of the generalization ability of image segmentation models. As a result, the accuracy of directly using these models to segment images is not high and cannot meet the requirements of jewelry detection. In terms of picture beautification, the inventors tried image / video generation models such as SORA and ComfyUI in the early stage, but directly using these models cannot accurately meet the requirements. Even the function of generating pictures through semantic graphs may cause the generated images to be not the required images, and the generated images may greatly reduce the pixels of the image output, and the accuracy of segmentation is also low, about 70%-80%.

[0004] Therefore, there is an urgent need for an automated image segmentation and beautification processing technology to solve the above-mentioned problems of low accuracy and low efficiency in jewelry image segmentation. Summary of the invention

[0005] In view of the above problems, the embodiments of the present invention provide a method, device and medium for segmenting and optimizing jewelry images in detection certificates, which are used to solve the above-mentioned problems of low accuracy and low efficiency in segmenting jewelry images.

[0006] According to one aspect of an embodiment of the present invention, a method for segmenting a jewelry image in a certificate is provided, the method comprising:

[0007] Collecting a variety of jewelry pictures, using a predetermined frame model to frame part of the jewelry pictures and attaching corresponding types of labels to obtain training pictures with marked frames and labels;

[0008] Training a predetermined target detection model based on the image to be trained to obtain a trained target detection model;

[0009] Detect the jewelry image using the trained object detection model to obtain the image to be segmented with a marked frame and a label;

[0010] Inputting the image to be segmented into a predetermined image segmentation model, using the image segmentation model to identify the image type of the image to be segmented, and marking the image according to the image type;

[0011] Perform image segmentation based on the dotted image to obtain a segmented image;

[0012] The segmented image is optimized using a predetermined optimization algorithm, and the optimized image is uploaded to the detection certificate.

[0013] In an optional manner, the step of training a predetermined target detection model based on the image to be trained to obtain a trained target detection model includes:

[0014] Inputting the image to be trained into the target detection model for training, and predicting a prediction box corresponding to the image to be trained;

[0015] When the intersection-and-union ratio of the annotation box corresponding to the jewelry and the prediction box is greater than or equal to a preset threshold, the training is stopped to obtain a trained object detection model.

[0016] In an optional manner, the picture type includes a non-ornament type and an ornament type, the inputting of the picture to be segmented into a predetermined image segmentation model, using the image segmentation model to identify the picture type of the picture to be segmented, and marking according to the picture type, including:

[0017] If it is a non-jewelry picture, dotting is performed according to the annotation frame and label of the non-jewelry picture to obtain the coordinates of the first type of points after dotting;

[0018] If it is a jewelry picture, switch to the manual dotting mode and obtain the coordinates of the second type of points after the jewelry picture is dotted.

[0019] In an optional manner, performing image segmentation based on the dotted picture to obtain a segmented picture includes:

[0020] Input the first type of point coordinates, annotation boxes and labels of the non-jewelry pictures into the prompt encoder of the image segmentation model for encoding to obtain a first code, and input the second type of point coordinates, annotation boxes and labels of the jewelry pictures into the prompt encoder for encoding to obtain a second code;

[0021] Inputting the first code into the mask interpreter of the image segmentation model for mask processing to obtain a first mask, and performing image segmentation based on the first mask to obtain a segmented image of the non-jewelry picture;

[0022] The second code is input into the mask interpreter for mask processing to obtain a second mask, and image segmentation is performed based on the second mask to obtain a segmented image of the jewelry picture.

[0023] In an optional manner, inputting the first code into a mask interpreter of the image segmentation model for mask processing to obtain a first mask, and performing image segmentation based on the first mask to obtain a segmented image of the non-jewelry picture includes:

[0024] Inputting the first code into the mask interpreter of the image segmentation model, determining K categories of the non-jewelry pictures, and obtaining K trained tokens based on the K categories, the first code and a global-local feature fusion function;

[0025] Based on the token, the mask interpreter is prompted with the category of segmentation, and image segmentation is performed based on the prompted category to obtain a segmented image of the non-jewelry picture.

[0026] In an optional manner, after performing image segmentation based on the dotted image to obtain the segmented image, the method further includes:

[0027] The segmented pictures with poor segmentation effects are screened, and the pictures to be segmented corresponding to the segmented pictures with poor segmentation effects are input into the image segmentation model again, and the manual dotting mode is switched to perform dotting to re-segment the image.

[0028] In an optional manner, the optimizing process of the segmented image by using a predetermined optimization algorithm and uploading the optimized image to the detection certificate includes:

[0029] Performing Gaussian blur processing on the segmented image, and replacing the background of the segmented image after the Gaussian blur processing with a solid color background to obtain a solid color background image;

[0030] Generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm;

[0031] The pure color background image and the shadow are superimposed to obtain the optimized image, and the optimized image is uploaded to the inspection certificate.

[0032] In an optional manner, generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm includes:

[0033] Obtaining a calculated view transformation matrix and a calculated projection transformation matrix of the jewelry image;

[0034] Based on the calculated view transformation matrix and the calculated projection transformation matrix, transforming points of the camera coordinate system to a camera normalized device coordinate space;

[0035] The distance between the image pixels of the segmented image and the light source is calculated based on the camera normalized device coordinate space, and the shadow of the jewelry is generated based on the distance.

[0036] According to another aspect of an embodiment of the present invention, there is provided a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method as described above.

[0038] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and when the executable instruction is executed on a computer device, the computer device executes the method described above.

[0039] The embodiment of the present invention collects a variety of jewelry pictures, establishes a data set in the field of jewelry detection, uses a predetermined framing model to frame part of the jewelry pictures and affixes corresponding types of labels, obtains a to-be-trained picture with a labeled frame and label, trains a predetermined target detection model based on the to-be-trained picture, enables the target detection model to accurately frame and label blind samples, inputs the to-be-segmented picture into a predetermined image segmentation model, uses the image segmentation model to identify the picture type of the to-be-segmented picture, performs type marking according to the characteristics of the diversity of jewelry types and styles, performs image segmentation based on the marked picture, obtains a segmented picture, uses an optimization algorithm to optimize the segmented picture, realizes beautification of the segmented picture, and finally obtains a picture that meets the requirements of the jewelry detection certificate. This embodiment constructs a visual large model for the field of jewelry detection, which includes target detection, image and semantic segmentation, and image optimization of the picture, and basically forms an automated processing flow, which can accurately and efficiently realize the segmentation and beautification of jewelry pictures, and meet the high requirements for jewelry pictures in the detection certificate.

[0040] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0042] Figure 1 A schematic diagram of a process for detecting jewelry image segmentation in a certificate provided by an embodiment of the present invention is shown;

[0043] Figure 2 A schematic diagram of the structure of a device for segmenting jewelry images in a certificate provided by an embodiment of the present invention is shown;

[0044] Figure 3 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0046] Figure 1 FIG. 1 is a flow chart of a method for segmenting jewelry images in a certificate provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0047] Step 101: Collect a variety of jewelry pictures, use a predetermined frame model to frame part of the jewelry pictures and attach corresponding types of labels to obtain training pictures with marked frames and labels;

[0048] Collect various types of jewelry pictures and establish a jewelry picture data set. Among them, jewelry pictures are mainly pictures retained during daily jewelry inspection. They are generally taken with conventional cameras, and jewelry of different styles and cuts are collected into a picture set. The larger the amount of collected data, the better. For example, more than 1 million pictures can cover the jewelry styles submitted for inspection in recent years, and have a unified placement and arrangement method. The picture resolution is greater than or equal to 2000*1600. The types of jewelry pictures include at least earrings, pendants, pendants, rings, ornaments, bracelets, bangles and necklace pendants. Summarize jewelry pictures of different types and styles, clean the pictures, and delete duplicate pictures.

[0049] In this embodiment, the predetermined framing model is preferably an unsupervised-self-supervised learning model, such as the AnyLabeling model. Of course, it can also be other framing models, such as the ‌AutoLabelImg model and the X-Anylabeling model, etc., which are not limited here. When using the framing model to frame the target object in the jewelry picture, that is, the jewelry, in order to reduce the impact of the deviation, all objects except the accessories are framed with a rectangular frame and labeled with the corresponding type. The types of accessories are very wide, and they can be framed in a suitable shape (such as a round frame). In this embodiment, the AnyLabeling model can be used to frame some jewelry pictures, for example, 100,000 pictures, as training samples for the following target detection model.

[0050] Step 102: training a predetermined target detection model based on the image to be trained to obtain a trained target detection model;

[0051] In this embodiment, the target detection model includes the classification of the training images and the determination of the precise position of the jewelry in the images, i.e., target positioning. The predetermined target detection model is preferably YOLO (You Only Look Once), and of course it can also be other target detection models, such as SSD (Single Shot Detector) model and Faster R-CNN model, etc., which are not limited here. The target detection model is trained by training the training images with marked boxes and labels to obtain a trained target detection model.

[0052] In an optional embodiment, a predetermined target detection model is trained based on the image to be trained to obtain a trained target detection model, including: inputting the image to be trained into the target detection model for training, and predicting a prediction box corresponding to the image to be trained; when the intersection and union ratio of the annotation box corresponding to the jewelry and the prediction box is greater than or equal to a preset threshold, stopping the training to obtain a trained target detection model.

[0053] In this embodiment, a predetermined computer language (such as Python) can be used to write mask files of different jewelry types for label prediction during the training process. The image to be trained can be processed into a suitable size for training. First, the image to be trained is divided into multiple n×n grids (for example, 3*3=9 grids). For each grid, m frames are predicted, and n×n×m frames can be predicted. Each frame predicts its corresponding category e (one of 8 categories). For each frame vector y=[pc, bx, by, bw, bh, e], pc is the probability that the target (i.e., jewelry) exists in the grid, bx and by are the x and y coordinates of the center point of the jewelry in the grid, the upper left vertex of the grid is the coordinate origin, and the coordinates of the lower right vertex of the grid are (1, 1). The values ​​of bx and by are always between 0-1, bw is the ratio of the width of the frame to the width of the grid, and bh is the ratio of the height of the frame to the height of the grid.

[0054] Assume that C1-C8 represent the eight types of earrings, pendants, pendants, rings, accessories, bracelets, bangles and necklace pendants, respectively. For each grid, a fully connected layer is used to predict one or more bounding boxes, as well as the possible categories and confidence scores of each bounding box.

[0055] For each grid in the image, select the border with the highest confidence, remove the duplicate borders according to the non-maximum suppression algorithm and select the final border, and repeat until a border with the largest IoU value remains, where the IoU value refers to the intersection-over-union ratio between the annotation box and the prediction box. In order to improve the accuracy of the target detection model, the IoU value of this embodiment is as close to 1 as possible, that is, the prediction box and the annotation box can basically overlap. Optionally, the preset threshold of IoU is 0.95. When the intersection-over-union ratio is greater than or equal to the preset threshold, stop training to obtain a trained target detection model. In this embodiment, during the training process, if there are a continuous preset number of images to be trained whose corresponding IoU values ​​are greater than or equal to the preset threshold, then the training can be stopped to obtain a trained target detection model.

[0056] In an optional embodiment, in order to solve the problem that part of the frame is inaccurate and the predicted frame does not completely frame the jewelry, a bounding box regression algorithm (BBR) can be used. Through the bounding box regression algorithm, a mapping function f is calculated. Through the mapping function f, the mapping of the predicted frame is made infinitely close to the real target frame. The mapping function f includes a translation transformation and a scale transformation. The translation and scale transformation are used to adjust the position and size of the predicted frame to make it closer to the real target frame.

[0057] Optionally, the loss function of the bounding box regression algorithm has a greater impact on its performance, that is, the existing IOU-based loss function is affected by unreasonable penalty factors, which causes the anchor box to expand during the regression process, significantly slowing down the convergence speed. In order to solve this problem, the present embodiment also provides a strong IOU loss function, which is a loss function that combines a target size adaptive penalty factor and a gradient adjustment function based on the quality of the anchor box. The strong IOU loss function loss guides the anchor frame to regress along a valid path, thereby converging faster than the IOU-based loss. In addition, for the focusing mechanism, a non-monotonic attention layer can be used and combined with the strong IOU loss function to obtain a new loss function v2. The loss function v2 loss enhances the ability to focus on medium-quality anchor boxes. By integrating the loss function v2 into the target detection model, an increase in average accuracy and an improvement in detection performance are achieved.

[0058] In this embodiment, the trained target detection model can accurately frame and label eight types of items, including earrings, pendants, pendants, rings, accessories, bracelets, bangles and necklace pendants, and its accuracy rate has been verified to be greater than 0.95.

[0059] Step 103: Detect the jewelry image using the trained object detection model to obtain the image to be segmented with a marked frame and a label;

[0060] In this embodiment, the trained target detection model can accurately frame and label blind samples, and the trained target detection model is used to detect jewelry images that have not participated in training, so as to accurately frame and label each jewelry image for subsequent image segmentation.

[0061] Step 104: input the image to be segmented into a predetermined image segmentation model, use the image segmentation model to identify the image type of the image to be segmented, and perform dot marking according to the image type;

[0062] In this embodiment, the predetermined image segmentation model is preferably a high-quality segmentation model (SegmentAnything in High Quality, HQ-SAM). Of course, it can also be other image segmentation models, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), etc., which are not limited here. When a rectangular frame is used as a labeling frame, its boundary sharpness, edges and corners are very prominent. If the boundary is not adjusted, it cannot be directly placed in the test certificate. Therefore, it is necessary to dot the picture to assist in finding the accurate boundary.

[0063] Optionally, the image types include non-ornaments and ornaments. The ornaments are very diverse and are divided into one category. The non-ornaments include 7 categories: earrings, pendants, pendants, rings, bracelets, bangles and necklace pendants. The image to be segmented is input into a predetermined image segmentation model, the image segmentation model is used to identify the image type of the image to be segmented, and the marking is performed according to the image type, including:

[0064] If it is a non-jewelry picture, dot it according to the annotation box and label of the non-jewelry picture to obtain the first type of point coordinates after dotting; if it is a jewelry picture, switch to manual dotting mode and obtain the second type of point coordinates after dotting the jewelry picture.

[0065] In this embodiment, for non-jewelry pictures, dotting is performed automatically according to the marking box and label. Dotting means marking coordinate points in the picture. The actual part of the jewelry is marked with positive points, which represents the part to be selected, and the gap and blank position are marked with negative points, which represent the part to be deleted. After the picture to be segmented is input into the predetermined image segmentation model, the image segmentation model identifies whether it is a non-jewelry picture or a jewelry picture according to the label. In order to achieve fast and automatic dotting, the non-jewelry pictures in this embodiment are dotted according to the preset dotting rules. The dotting rules are as follows:

[0066] Earrings: Use a vertical line and two horizontal lines to divide the marked frame (i.e. rectangular frame) into 6 equal parts. From left to right and from top to bottom, take the center point of each equal part as the positive point, and the middle part as the negative point;

[0067] Hanging parts: Use a vertical line and two horizontal lines to divide the marking frame into 6 equal parts. From left to right and from top to bottom, take the center points of the 1st, 3rd and 5th parts as positive points, and the center points of the 4th and 6th parts as negative points;

[0068] Pendant: The marking frame is not divided, the coordinates of the center point of the frame are taken as the positive point, and the points at the four corners of the marking frame are taken as the negative points;

[0069] Ring: The annotation frame is not divided, the coordinates of the center point of the frame are taken as the positive point, and the points on both sides of the center point are also taken as positive points by moving 50 pixels each. The points at the four corners of the annotation frame are taken as negative points.

[0070] Bracelet: The marking frame is not divided, the coordinates of the center point of the frame are taken as the positive point, and the points on both sides of the center point are also moved 50 pixels as positive points, and the points at the four corners of the marking frame are taken as negative points;

[0071] Bracelet: The marking frame is divided into equal parts in a cross, the center point of each equal part is taken as the positive point, and the coordinates of the center point of the frame are taken as the negative point;

[0072] Necklace pendant: The two horizontal lines of the marking frame are divided into three equal parts. The center point of the third equal part is taken as the positive point, and the coordinates of the center point of the frame are taken as the negative point;

[0073] For pictures of jewelry, since there are a wide variety of jewelry types, various appearances and sizes, but in general, jewelry pictures account for a small proportion of jewelry, basically accounting for 5%, so they can be dotted manually, and the image segmentation model will automatically switch to manual dotting mode for the dotting personnel to manually dot carefully. This embodiment adopts a combination of automatic dotting and manual dotting, which can improve the accuracy of image segmentation.

[0074] In practical applications, the more dots are added, the more accurate the segmentation will not necessarily be. Instead, it will cause a mottled cutout. In fact, in many cases, negative dots have no effect. Adding too many dots not only requires a huge workload but also fails to meet the requirements. This embodiment takes comprehensive considerations and, based on different style types, the number of dots added to each image is at least 3-6 coordinate points to achieve the segmentation prompt function.

[0075] Step 105: performing image segmentation based on the dotted image to obtain a segmented image;

[0076] In this embodiment, the dotted image will form a corresponding mask, that is, a mask, and the image segmentation model performs image segmentation according to the mask and the annotation box.

[0077] Optionally, image segmentation is performed based on the dotted image to obtain a segmented image, including:

[0078] The first type of point coordinates, annotation boxes and labels of the non-jewelry pictures are input into the prompt encoder of the image segmentation model for encoding to obtain a first code, and the second type of point coordinates, annotation boxes and labels of the jewelry pictures are input into the prompt encoder for encoding to obtain a second code; the first code is input into the mask interpreter of the image segmentation model for mask processing to obtain a first mask, and image segmentation is performed based on the first mask to obtain a segmented image of the non-jewelry pictures; the second code is input into the mask interpreter for mask processing to obtain a second mask, and image segmentation is performed based on the second mask to obtain a segmented image of the jewelry pictures.

[0079] In this embodiment, the architecture of the image segmentation model includes an image encoder, a prompt encoder and a mask interpreter. Among them, the image encoder generates a one-time image embedding, which can be applied before the prompt encoder. The prompt encoder encodes various types of input prompts, including encoding point coordinates, annotation boxes, text and low-resolution mask inputs, which helps to inform the model of the location and context of the target in the picture. The mask interpreter masks the image embedding and encoding and converts it into a final mask, which is used to define which areas in the picture should be merged and which areas should remain independent. In this embodiment, the annotation box and the first type of point coordinates of non-jewelry pictures are encoded, and the annotation box and the second type of point coordinates of jewelry pictures are encoded. The mask interpreter performs mask processing according to the encoding to obtain a mask, and the pixel values ​​in the mask area are merged through the mask to achieve image segmentation. In this embodiment, the annotation box can be specified using the coordinates [x1, y1, x2, y2] to inform the model of the location and size of the annotation box. The first type of point coordinates and the second type of point coordinates can help define the corresponding boundaries.

[0080] In an optional embodiment, the first code is input into the mask interpreter of the image segmentation model for mask processing to obtain a first mask, and image segmentation is performed based on the first mask to obtain a segmented image of the non-jewelry picture, including:

[0081] The first code is input into the mask interpreter of the image segmentation model to determine K categories of the non-jewelry pictures, and K trained tokens are obtained based on the K categories, the first code and the global-local feature fusion function; based on the tokens, the mask interpreter is prompted with the category of segmentation, and image segmentation is performed based on the prompted category to obtain a segmented image of the non-jewelry picture.

[0082] In this embodiment, the token is a prompt of the corresponding category, which is obtained through training based on the global-local feature fusion function. The global-local feature fusion function is a function that fuses global features and local features. In the field of image segmentation, global features focus on the overall structure and content of the image, while local features focus on the detailed information of the image. By fusing global and local features, the complementary relationship between the two can be fully utilized to understand the image scene more comprehensively and accurately, and the fusion method can better cope with noise and interference in the image, and improve the accuracy and robustness of segmentation.

[0083] Among them, K=7 of the K categories of non-jewelry images include earrings, pendants, pendants, rings, bracelets, bangles and necklace pendants. The global and local feature fusion function G(h 256 , P) = { <iou i , ŷ i >|i=1, 2, 3, 4, 5, 6, 7}, h 256is the encoded representation of the input image, i.e., the first encoding, P is the category prompt set, i.e., the token set, iou i and i They represent the intersection of the original labeled box and the predicted box of category i and the predicted value of the mask decoder respectively.

[0084] After training, we get token P={p i |i=1, 2, 3, 4, 5, 6, 7 (“ear ornaments”, “pendant”, “pendant”, “ring”, “bracelet”, “bracelet”, “necklace pendant”)}, each p i is a category cue for category i, suggesting that the mask interpreter should segment category i.

[0085] This embodiment improves the mask interpreter through a global-local feature fusion function, so that the mask interpreter automatically generates the intersection-over-union ratio of the original marked box and the predicted box of each category of the image. For the image segmentation task of the seven categories other than accessories, a set of category prompts is provided to prompt the mask interpreter of the category that should be segmented, which helps to improve the accuracy of image segmentation.

[0086] In an optional embodiment, the loss function of the image segmentation model is the sum of the Jaccard loss function and the Tversky loss function, that is, L=L1+L2, where L1 is the Jaccard loss function, which is based on the Jaccard index, that is, the IoU index, and is used to evaluate the similarity between the segmentation result of the model and the true label in the image segmentation task. L2 is the Tversky loss function, which is based on the Tversky index and is used to measure the overlap between two segmented images. The above-mentioned Jaccard loss function and Tversky loss function can improve the accuracy of image segmentation.

[0087] Step 106: Optimize the segmented image using a predetermined optimization algorithm, and upload the optimized image to the detection certificate.

[0088] In the prior art, image optimization generally uses contrast adjustment, filtering, wavelet transform and other noise reduction processing and uniform lighting methods, but these methods are generally used in the live broadcast industry and artistic image creation field, and are not suitable for the field of jewelry pictures, especially the field of jewelry detection. In this embodiment, the segmented picture is optimized using a predetermined optimization algorithm, including one or more of the following: segmented picture edge optimization, shadow processing and brightness and gloss optimization to meet the requirements of the detection certificate.

[0089] In one embodiment, the segmented image is optimized using a predetermined optimization algorithm, and the optimized image is uploaded to the detection certificate, including:

[0090] Performing Gaussian blur processing on the segmented image, and replacing the background of the segmented image after Gaussian blur processing with a solid color background to obtain a solid color background image; generating the shadow of the jewelry based on the segmented image and a soft shadow generation algorithm; superimposing the solid color background image and the shadow to obtain the optimized image, and uploading the optimized image to the inspection certificate.

[0091] In this embodiment, Gaussian blur processing is used to optimize the image edge. Of course, other algorithms can also be used to optimize the edge, such as the Sobel operator or the Laplace operator method, which is not limited here. In this embodiment, Gaussian blur processing is performed on the segmented image, which can smooth the image edge, eliminate details and textures, reduce noise, and help improve the accuracy of subsequent processing. Gaussian blur processing is performed on the segmented image, including:

[0092] 1. Based on two-dimensional Gaussian distribution, the Gaussian convolution kernel is 3*3. The size and shape of the convolution kernel can be adjusted according to the image pixels. The value of each element in the convolution kernel is calculated according to the Gaussian function, and the sum of all elements is 1;

[0093] 2. Perform a convolution operation and apply the defined Gaussian convolution kernel to each pixel of the image, where:

[0094] out=(k*(x-1)+k*x+k*(x+1)) / 9;

[0095] Among them, out is the new pixel value after Gaussian blur processing, k is the weight value in the convolution kernel, (x-1) and (x+1) are the values ​​of adjacent pixels, and x is the value of the current pixel.

[0096] 3. Mask image edge processing, using the following formula to extend the image edge to more pixel positions:

[0097] out[i,j]=(in[i-1,j-1]*0.5+in[i-1,j]*0.5)*0.5;

[0098] out[i,j]=(in[i,j-1]*0.5+in[i,j]*0.5)*0.5;

[0099] Where i and j represent the coordinates of the pixel.

[0100] In this embodiment, after the segmented image is subjected to Gaussian blur processing, it is automatically placed in a fixed background, that is, the background is replaced with a pure color background, for example, a pure black or pure white background, to facilitate subsequent shadow processing.

[0101] In an optional embodiment, generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm includes:

[0102] Obtain a calculated view transformation matrix and a calculated projection transformation matrix of the jewelry image; based on the calculated view transformation matrix and the calculated projection transformation matrix, transform the points of the camera coordinate system to the camera normalized device coordinate space; based on the camera normalized device coordinate space, calculate the distance between the image pixels of the segmented image and the light source, and generate the shadow of the jewelry based on the distance.

[0103] In this embodiment, ray Marching is used to simulate the propagation of light in space, and the filter kernel size is automatically adjusted according to the light source and the distance from the shading point to the obstruction to obtain a realistic shadow effect of the jewelry.

[0104] Among them, the calculated view transformation matrix View is the calculated view transformation from the camera space to the naked eye space, and the calculated view transformation matrix View=R*(Tgem-Rtgem), R is the rotation matrix, Tgem is the translation vector, and Rtgem is the rotation quaternion;

[0105] Calculate the projection transformation matrix Pgem. The matrix is ​​the projection transformation from the naked eye space to the screen space. The projection transformation matrix Pgem=P*(A*F) / S, where P is the perspective or orthogonal projection matrix, A is the aspect ratio of the screen, F is the focal length of the camera, and S is the screen size.

[0106] The camera normalized device coordinate space involves the coordinate transformation from model coordinates to screen display. It is a cubic space with the three axes of xyz ranging from -1 to 1. The camera normalized device coordinate space provides a standardized way to process depth values ​​in screen space, so that objects at different distances can be compared and calculated on the same scale. Transforming the scene of jewelry pictures to the camera normalized device coordinate space includes calculating the ST matrix, ST=View*Pgem. The role of the ST matrix is ​​to transform the points in the camera coordinate system through perspective division or homogeneous division to obtain the coordinates of the camera normalized device coordinate space.

[0107] Based on the camera normalized device coordinate space, the distance SD between the image pixel of the segmented image and the light source is calculated as L*(glF-LSP), where SD is the distance between each pixel of the segmented image and the light source, L is used to calculate the Euclidean distance between two points, glF is the coordinate of the current segment or pixel, and LSP is the position of the light source in the world space. In this embodiment, by calculating the distance SD, the shadow of the jewelry can be projected according to the distance SD.

[0108] In an optional embodiment, the gloss or brightness of the cut jewelry may be adjusted by contrast, enhancement and lighting adjustment, especially for manually dotted pictures, to further optimize the pictures.

[0109] In this embodiment, the above-mentioned solid color background image and shadow are superimposed to achieve accurate segmentation and optimization of jewelry images, and obtain an electronic version of jewelry images that can be uploaded to the inspection certificate, and can also be printed to obtain a paper version of the inspection certificate. In practical applications, the segmentation accuracy of the jewelry images after the above segmentation and optimization reaches more than 0.95, and for these accurately segmented images, there is no pixel loss when output, and the overall segmentation effect is much better than the existing automatic image segmentation software. Using the NVIDIA GeForce RTX 4090 graphics card for processing, about 15,000 jewelry images can be processed within 3 hours, which is much more efficient than the existing technology of processing about 500 jewelry images in 3 hours.

[0110] In an optional embodiment, image segmentation is performed based on the dotted image, and after the segmented image is obtained, it also includes: screening the segmented images with poor segmentation effect, inputting the to-be-segmented images corresponding to the segmented images with poor segmentation effect into the image segmentation model again, and switching to the manual dotting mode for dotting to re-segment the image.

[0111] In this embodiment, for segmented images with poor segmentation effects, such as segmented images with blurred segmentation edges, inaccurate segmentation lines or reduced sizes, the corresponding images to be segmented are input into the image segmentation model again, and switched to the manual dotting mode for careful dotting before re-segmentation, which improves the accuracy of segmentation overall.

[0112] Compared with the prior art, the embodiment of the present invention collects a variety of jewelry pictures, establishes a data set in the field of jewelry detection, uses a predetermined framing model to frame part of the jewelry pictures and affixes corresponding types of labels, obtains a training picture with a labeled frame and label, trains a predetermined target detection model based on the training picture, so that the target detection model has the ability to accurately frame and label blind samples, inputs the picture to be segmented into a predetermined image segmentation model, uses the image segmentation model to identify the picture type of the picture to be segmented, performs type marking according to the characteristics of the diversity of jewelry types and styles, performs image segmentation based on the marked picture, obtains a segmented picture, uses an optimization algorithm to optimize the segmented picture, realizes beautification of the segmented picture, and finally obtains a picture that meets the requirements of the jewelry detection certificate. This embodiment constructs a visual large model for the field of jewelry detection, which includes target detection, image and semantic segmentation, and image optimization of the picture, and basically forms an automated processing flow, which can accurately and efficiently realize the segmentation and beautification of jewelry pictures, and meet the high requirements for jewelry pictures in the detection certificate.

[0113] Figure 2FIG. 1 is a schematic diagram showing a structure of a device for segmenting jewelry images in a certificate according to an embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0114] The framing module 201 is used to collect a variety of jewelry pictures, frame some of the jewelry pictures using a predetermined framing model and attach corresponding types of labels to obtain training pictures with marked frames and labels;

[0115] A training module 202 is used to train a predetermined object detection model based on the image to be trained to obtain a trained object detection model;

[0116] The detection module 203 is used to detect the jewelry image using the trained object detection model to obtain the image to be segmented with a marked frame and a label;

[0117] A dotting module 204 is used to input the image to be segmented into a predetermined image segmentation model, identify the image type of the image to be segmented by using the image segmentation model, and perform dotting according to the image type;

[0118] The segmentation module 205 is used to perform image segmentation based on the dotted image to obtain a segmented image;

[0119] The optimization module 206 is used to optimize the segmented image using a predetermined optimization algorithm and upload the optimized image to the detection certificate.

[0120] The embodiment of the device for segmenting jewelry images in the detection certificate is basically the same as the above-mentioned method embodiment, and will not be repeated here.

[0121] Figure 3 The schematic diagram of the structure of the computer device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0122] like Figure 3 As shown, the computer device may include: a processor (processor) 302 , a communication interface (Communications Interface) 304 , a memory (memory) 306 , and a communication bus 308 .

[0123] The processor 302, the communication interface 304, and the memory 306 communicate with each other via the communication bus 308. The communication interface 304 is used to communicate with other computer devices such as client or other server network elements. The processor 302 is used to execute the program 310, which can specifically execute the relevant steps in the above-mentioned computer device embodiment.

[0124] Specifically, the program 310 may include program code including computer executable instructions.

[0125] The processor 302 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0126] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0127] The program 310 can be specifically called by the processor 302 to enable the computer device to perform the following operations:

[0128] Collecting a variety of jewelry pictures, using a predetermined frame model to frame part of the jewelry pictures and attaching corresponding types of labels to obtain training pictures with marked frames and labels;

[0129] Training a predetermined target detection model based on the image to be trained to obtain a trained target detection model;

[0130] Detect the jewelry image using the trained object detection model to obtain the image to be segmented with a marked frame and a label;

[0131] Inputting the image to be segmented into a predetermined image segmentation model, using the image segmentation model to identify the image type of the image to be segmented, and marking the image according to the image type;

[0132] Perform image segmentation based on the dotted image to obtain a segmented image;

[0133] The segmented image is optimized using a predetermined optimization algorithm, and the optimized image is uploaded to the detection certificate.

[0134] In an optional manner, the step of training a predetermined target detection model based on the image to be trained to obtain a trained target detection model includes:

[0135] Inputting the image to be trained into the target detection model for training, and predicting a prediction box corresponding to the image to be trained;

[0136] When the intersection-and-union ratio of the annotation box corresponding to the jewelry and the prediction box is greater than or equal to a preset threshold, the training is stopped to obtain a trained object detection model.

[0137] In an optional manner, the picture type includes a non-ornament type and an ornament type, the inputting of the picture to be segmented into a predetermined image segmentation model, using the image segmentation model to identify the picture type of the picture to be segmented, and marking according to the picture type, including:

[0138] If it is a non-jewelry picture, dotting is performed according to the annotation frame and label of the non-jewelry picture to obtain the coordinates of the first type of points after dotting;

[0139] If it is a jewelry picture, switch to the manual dotting mode and obtain the coordinates of the second type of points after the jewelry picture is dotted.

[0140] In an optional manner, performing image segmentation based on the dotted picture to obtain a segmented picture includes:

[0141] Input the first type of point coordinates, annotation boxes and labels of the non-jewelry pictures into the prompt encoder of the image segmentation model for encoding to obtain a first code, and input the second type of point coordinates, annotation boxes and labels of the jewelry pictures into the prompt encoder for encoding to obtain a second code;

[0142] Inputting the first code into the mask interpreter of the image segmentation model for mask processing to obtain a first mask, and performing image segmentation based on the first mask to obtain a segmented image of the non-jewelry picture;

[0143] The second code is input into the mask interpreter for mask processing to obtain a second mask, and image segmentation is performed based on the second mask to obtain a segmented image of the jewelry picture.

[0144] In an optional manner, inputting the first code into a mask interpreter of the image segmentation model for mask processing to obtain a first mask, and performing image segmentation based on the first mask to obtain a segmented image of the non-jewelry picture includes:

[0145] Inputting the first code into the mask interpreter of the image segmentation model, determining K categories of the non-jewelry pictures, and obtaining K trained tokens based on the K categories, the first code and a global-local feature fusion function;

[0146] Based on the token, the mask interpreter is prompted with the category of segmentation, and image segmentation is performed based on the prompted category to obtain a segmented image of the non-jewelry picture.

[0147] In an optional manner, after performing image segmentation based on the dotted image to obtain the segmented image, the method further includes:

[0148] The segmented pictures with poor segmentation effects are screened, and the pictures to be segmented corresponding to the segmented pictures with poor segmentation effects are input into the image segmentation model again, and the manual dotting mode is switched to perform dotting to re-segment the image.

[0149] In an optional manner, the optimizing process of the segmented image by using a predetermined optimization algorithm and uploading the optimized image to the detection certificate includes:

[0150] Performing Gaussian blur processing on the segmented image, and replacing the background of the segmented image after the Gaussian blur processing with a solid color background to obtain a solid color background image;

[0151] Generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm;

[0152] The pure color background image and the shadow are superimposed to obtain the optimized image, and the optimized image is uploaded to the inspection certificate.

[0153] In an optional manner, generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm includes:

[0154] Obtaining a calculated view transformation matrix and a calculated projection transformation matrix of the jewelry image;

[0155] Based on the calculated view transformation matrix and the calculated projection transformation matrix, transforming points of the camera coordinate system to a camera normalized device coordinate space;

[0156] The distance between the image pixels of the segmented image and the light source is calculated based on the camera normalized device coordinate space, and the shadow of the jewelry is generated based on the distance.

[0157] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device, the computer device executes any of the above method embodiments.

[0158] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computer device to execute any of the above method embodiments.

[0159] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer is enabled to execute any of the above method embodiments.

[0160] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description made to specific languages ​​above is for disclosing the best mode of the present invention.

[0161] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0162] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly recited in each claim.

[0163] Those skilled in the art will appreciate that the modules in the computer device in the embodiment may be adaptively changed and set in one or more computer devices different from the embodiment. The modules or units or components in the embodiment may be combined into one module or unit or component, and may be divided into multiple sub-modules or sub-units or sub-components. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or computer device disclosed in this manner may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0164] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A method for segmenting jewelry images in certificates, characterized in that: The method comprises: Collecting a variety of jewelry pictures, using a predetermined frame model to frame part of the jewelry pictures and attaching corresponding types of labels to obtain training pictures with marked frames and labels; Training a predetermined target detection model based on the image to be trained to obtain a trained target detection model; Detect the jewelry image using the trained object detection model to obtain the image to be segmented with a marked frame and a label; Input the image to be segmented into a predetermined image segmentation model, use the image segmentation model to identify the image type of the image to be segmented, and dot the image according to the image type to assist in finding the accurate boundary, wherein the number of dotted points for each image is at least 3-6 coordinate points; Perform image segmentation based on the dotted image to obtain a segmented image; The segmented image is optimized using a predetermined optimization algorithm, and the optimized image is uploaded to the test certificate, including: Performing Gaussian blur processing on the segmented image, and replacing the background of the segmented image after the Gaussian blur processing with a solid color background to obtain a solid color background image; Generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm; Superimposing the pure color background image and the shadow to obtain the optimized image, and uploading the optimized image to the test certificate; The step of generating the shadow of the jewelry based on the segmented image and the soft shadow generation algorithm includes: Obtaining a calculated view transformation matrix and a calculated projection transformation matrix of the jewelry image; Based on the calculated view transformation matrix and the calculated projection transformation matrix, transforming points of the camera coordinate system to a camera normalized device coordinate space; The distance between the image pixels of the segmented image and the light source is calculated based on the camera normalized device coordinate space, and the shadow of the jewelry is generated based on the distance.

2. The method according to claim 1, characterized in that The step of training a predetermined target detection model based on the image to be trained to obtain a trained target detection model includes: Inputting the image to be trained into the target detection model for training, and predicting a prediction box corresponding to the image to be trained; When the intersection-and-union ratio of the annotation box corresponding to the jewelry and the prediction box is greater than or equal to a preset threshold, the training is stopped to obtain a trained object detection model.

3. The method according to claim 1, characterized in that The picture types include non-ornaments and ornaments, the picture to be segmented is input into a predetermined image segmentation model, the picture type of the picture to be segmented is identified by using the image segmentation model, and the marking is performed according to the picture type, including: If it is a non-jewelry picture, dotting is performed according to the annotation frame and label of the non-jewelry picture to obtain the coordinates of the first type of points after dotting; If it is a jewelry picture, switch to the manual dotting mode and obtain the coordinates of the second type of points after the jewelry picture is dotted.

4. The method according to claim 3, characterized in that The image segmentation is performed based on the dotted picture to obtain the segmented picture, including: Input the first type of point coordinates, annotation boxes and labels of the non-jewelry pictures into the prompt encoder of the image segmentation model for encoding to obtain a first code, and input the second type of point coordinates, annotation boxes and labels of the jewelry pictures into the prompt encoder for encoding to obtain a second code; Inputting the first code into the mask interpreter of the image segmentation model for mask processing to obtain a first mask, and performing image segmentation based on the first mask to obtain a segmented image of the non-jewelry picture; The second code is input into the mask interpreter for mask processing to obtain a second mask, and image segmentation is performed based on the second mask to obtain a segmented image of the jewelry picture.

5. The method according to claim 4, characterized in that The step of inputting the first code into the mask interpreter of the image segmentation model for mask processing to obtain a first mask, and performing image segmentation based on the first mask to obtain a segmented image of the non-jewelry picture includes: Inputting the first code into the mask interpreter of the image segmentation model, determining K categories of the non-jewelry pictures, and obtaining K trained tokens based on the K categories, the first code and a global-local feature fusion function; Based on the token, the mask interpreter is prompted with the category of segmentation, and image segmentation is performed based on the prompted category to obtain a segmented image of the non-jewelry picture.

6. The method according to claim 3, characterized in that: Image segmentation is performed based on the dotted image. After obtaining the segmented image, the following steps are also included: The segmented pictures with poor segmentation effects are screened, and the pictures to be segmented corresponding to the segmented pictures with poor segmentation effects are input into the image segmentation model again, and the manual dotting mode is switched to perform dotting to re-segment the image.

7. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device, the computer device executes the method according to any one of claims 1 to 6.

Images