Distillation data acquisition method and device of part segmentation model, equipment, medium and product

Abstract

The invention provides a distillation data acquisition method and device of a part segmentation model, equipment, a medium and a product. The method comprises the following steps: acquiring an initial image and an intercepted image of a part; matching the initial image with the intercepted image to obtain position information of the part in the initial image; inputting the position information and the initial image into an initial segmentation model for part segmentation to obtain a segmentation mask and an intersection-union ratio score corresponding to the segmentation mask; and screening and segmenting the mask by adopting an intersection-to-union ratio score to obtain distillation data. According to the embodiment of the invention, the technical problems that the parts in the picture cannot be accurately segmented and the segmentation time is long in the prior art can be solved.

Description

Technical Field

[0001] The present application relates to the fields of deep learning, computers, and industrial vision technology, and in particular to a method, device, equipment, medium, and product for acquiring distillation data of a part segmentation model. Background Art

[0002] In industrial vision systems, part segmentation is one of the important basic links. For example, in the process of defect detection of parts, we only want to focus on the defects of the parts themselves, so we need to segment the parts from the image background.

[0003] In related technologies, traditional visual segmentation methods require manual feature extraction and are highly dependent on the environment, resulting in poor image segmentation results. Although the visual segmentation model based on deep learning has better image segmentation results, it takes a long time to complete the image segmentation due to the large number of parameters, which can easily cause computer freezes and affect machine operation, making it difficult to actually apply it in industrial vision systems.

[0004] Therefore, there is an urgent need for an image segmentation method that can accurately segment parts in images and takes a short time to segment. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment, storage medium and computer program product for obtaining distilled data of a part segmentation model to solve the technical problem in the related art that part image segmentation cannot simultaneously meet the requirements of short segmentation time and high segmentation accuracy.

[0006] In a first aspect, an embodiment of the present application provides a method for acquiring distilled data of a part segmentation model, the method comprising: acquiring an initial image and a cropped image of a part, wherein the cropped image is a partial image obtained by cropping the initial image according to the position of the part in the initial image; matching the initial image and the cropped image to obtain position information of the part in the initial image; inputting the position information and the initial image into an initial segmentation model to perform part segmentation, and obtaining a segmentation mask and an intersection-over-union (IoU) score corresponding to the segmentation mask; and using the IoU score to filter the segmentation mask to obtain distilled data.

[0007] In the second aspect, an embodiment of the present application provides a distillation data acquisition device for a part segmentation model, the device comprising: an acquisition module, used to acquire an initial image and a cropped image of a part, wherein the cropped image is a partial image obtained by cropping the initial image according to the position of the part in the initial image; a matching module, used to match the initial image and the cropped image to obtain the position information of the part in the initial image; a segmentation module, used to input the position information and the initial image into the initial segmentation model to perform part segmentation, and obtain a segmentation mask and an intersection-and-union ratio score corresponding to the segmentation mask; a screening module, used to use the intersection-and-union ratio score to screen the segmentation mask to obtain distilled data.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing programs or instructions; the processor implements the above method when executing the program or instructions.

[0009] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium, on which a program or instruction is stored, and the program or instruction implements the above method when executed by a processor.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the above method.

[0011] The distillation data acquisition method of the part segmentation model provided in the embodiment of the present application can obtain the initial image and the cropped image of the part; match the initial image and the cropped image to obtain the position information of the part in the initial image; input the position information and the initial image into the initial segmentation model to perform part segmentation, and obtain the segmentation mask and the intersection-and-union ratio score corresponding to the segmentation mask; use the intersection-and-union ratio score to filter the segmentation mask to obtain the distillation data.

[0012] This application uses the initial segmentation model as the teacher model and the preset lightweight model as the student model. First, the data is input into the initial segmentation model to obtain the distilled data, and then the preset lightweight model is distilled according to the distilled data. The obtained part segmentation model has both the lightweight characteristics of the preset lightweight model and the good segmentation effect of the initial segmentation model. Therefore, the part segmentation model shortens the time of part segmentation while also improving the accuracy of part segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 is a flow chart of a method for acquiring distillation data of a part segmentation model according to an embodiment of the present invention;

[0015] Figure 2 is a schematic diagram of an application flow of a method for acquiring distillation data of a part segmentation model according to an embodiment of the present invention;

[0016] Figure 3 It is a schematic diagram of the overall process of a method for obtaining distillation data of a part segmentation model according to an embodiment of the present invention;

[0017] Figure 4 is a schematic diagram of an encoding process of a first encoder according to an embodiment of the present invention;

[0018] Figure 5 is a schematic diagram of the structure of a Transformer module of a first encoder according to an embodiment of the present invention;

[0019] Figure 6 is a schematic diagram of a decoding process of a mask decoder according to an embodiment of the present invention;

[0020] Figure 7 is a schematic structural diagram of a preset lightweight model according to an embodiment of the present invention;

[0021] Figure 8 is a schematic diagram of a CNN skeleton network structure of a preset lightweight model according to an embodiment of the present invention;

[0022] Fig. 9 is a schematic diagram of the structure of a feature pyramid network and a detection head of a preset lightweight model according to an embodiment of the present invention;

[0023] Fig.10 is a structural block diagram of a distillation data acquisition device of a part segmentation model according to an embodiment of the present invention;

[0024] Fig.11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0026] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0027] In addition, it should be noted that the acquisition, storage, use, and processing of data in the embodiments of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0028] In order to solve the problems of the prior art, an embodiment of the present application provides a method for acquiring distilled data of a part segmentation model. The method can be applied in an industrial vision system, specifically in a scenario of visual-based defect detection of parts. Figure 1 is a flow chart of a method for obtaining distillation data of a part segmentation model according to an embodiment of the present invention. Figure 1 As shown, the distillation data acquisition method of the part segmentation model may include the following steps:

[0029] S110, acquiring an initial image and a captured image of the part.

[0030] In S110, the initial image may be a two-dimensional image or a 3D point cloud image obtained by photographing the part with a 2D camera or a 3D camera. The intercepted image may be a partial image obtained by manually or automatically intercepting the initial image according to the position of the part in the initial image.

[0031] S120, matching the initial image and the captured image to obtain position information of the part in the initial image.

[0032] In S120, after the image of the part is acquired, in order to acquire the approximate position information of the part in the initial image, image matching may be performed on the initial image and the captured image. For example, a normalized correlation coefficient method may be used for image matching.

[0033] In an example, the initial image is I, the intercepted image is T, the width and length of the intercepted image are (w, h) respectively, and the normalized correlation coefficient method is used to match the initial image and the intercepted image, including: for each local window W with a width and length of (w, h) in I, the normalized correlation coefficient NCC(x, y, I, T) between W and T is calculated respectively, and the position information B(I, T) of the part in the initial image is calculated based on NCC(x, y, I, T);

[0034] For example, the normalized correlation coefficient NCC(x, y, I, T) can be calculated using formula (1):

[0035]

[0036] The position information B(I,T) can be calculated using formula (2):

[0037] B(I,T)=argmax (x,y) NCC(x,y,I,T),argmax (x,y) NCC(x,y,I,T)+(w,h) (2)

[0038] Among them, (x′, y′) represents the position coordinates of the pixel element of T, and (x′, y′) represents the position coordinates of the pixel element of I.

[0039] In this way, by matching the initial image and the captured image, the position information of the part in the initial image can be obtained, so that the part can be segmented more finely later.

[0040] S130, inputting the position information and the initial image into an initial segmentation model to perform part segmentation, and obtaining a segmentation mask and an intersection-over-union ratio score corresponding to the segmentation mask.

[0041] In S130, the initial segmentation model may be a large visual model, which is used to perform image segmentation on the parts in the initial image based on the position information to obtain the segmentation mask of the parts. For example, the initial segmentation model may adopt a model architecture of a segmentation anything model (SAM).

[0042] Furthermore, the initial segmentation model using the SAM architecture can obtain the segmentation result of the part by encoding and decoding, wherein the segmentation result of the part can be represented in the form of a mask. It can be understood that in a machine vision system, the mask of the part has more detailed information than the captured image, which is convenient for machine recognition and grasping.

[0043] S140, using the intersection-over-union ratio score to filter the segmentation mask to obtain distilled data.

[0044] In S140, the intersection over union (IoU) score is an indicator for measuring the degree of overlap between the predicted mask and the true mask. The IoU value can represent the accuracy of the segmentation mask segmentation and can also be used as an indicator for measuring the confidence of the initial segmentation model. Therefore, the IoU value can be used to filter out segmentation masks with segmentation accuracy lower than a preset value, and the obtained segmentation masks higher than the preset value are used as distillation data.

[0045] In this way, by screening out segmentation masks with low segmentation accuracy, the obtained distilled data has a higher segmentation accuracy, which can provide more accurate distillation data for subsequent model distillation.

[0046] S150, performing model distillation on the preset lightweight model according to the distillation data to obtain a target segmentation model.

[0047] In S150, since the model parameters of the initial segmentation model are large, the model calculation time is relatively long. In order to shorten the model calculation time, the preset lightweight model can be distilled using distillation data to obtain a lightweight target segmentation model.

[0048] In an example, the model size of the initial segmentation model is 2445.75 MegaBytes, and the model size of the target segmentation model is 6.73 MegaBytes. Under the same experimental environment, the calculation time of the initial segmentation model is 1581.72 ms, and the calculation time of the target segmentation model is 4.2 ms.

[0049] Figure 2 FIG. 1 is a schematic diagram of an application flow of a method for obtaining distilled data of a part segmentation model according to an embodiment of the present invention. Figure 2 As shown, the present application uses the initial segmentation model as the teacher model and the preset lightweight model as the student model. After data processing of the initial image to be processed and the intercepted image obtained by intercepting the initial image, the initial segmentation model is input to generate a segmentation mask, and the preset lightweight model is distilled according to the distillation data formed by the segmentation mask. The obtained part segmentation model has both the lightweight features of the preset lightweight model and the good segmentation effect of the initial segmentation model. In this way, the part segmentation model is obtained, and the user can obtain the segmentation mask based on the part segmentation model to complete the part segmentation.

[0050] In this way, the part segmentation model not only shortens the time of part segmentation, but also improves the accuracy of part segmentation.

[0051] In some embodiments, the initial segmentation model includes a first encoder, a second encoder, and a mask decoder; the position information and the initial image are input into the initial segmentation model to perform part segmentation, and a segmentation mask and an intersection-over-union score corresponding to the segmentation mask are obtained, including:

[0052] By using a first encoder, extracting features from the initial image according to the position information to obtain feature data;

[0053] Performing data enhancement on the position information through a second encoder to obtain enhanced position information;

[0054] The feature data and enhanced position information are input into the mask decoder for decoding to obtain the segmentation mask and the intersection-over-union score.

[0055] In this embodiment, the feature data may be a feature map with rich features obtained by first encoder using deep learning technology to determine a partial image based on position information in the initial image, extracting and transforming features of the partial image.

[0056] In this embodiment, enhancing the position information may be performing position enhancement on a specific position of the position information. For example, the specific position may be determined according to the user's need to segment the image, and this application does not limit this.

[0057] Figure 3 FIG. 1 is a schematic diagram of the overall process of a method for obtaining distillation data of a part segmentation model according to an embodiment of the present invention. Figure 3 As shown, the initial segmentation model includes a first encoder, a second encoder and a mask decoder. The position information and the initial image are input into the initial segmentation model for part segmentation to obtain a segmentation mask and an intersection-over-union score corresponding to the segmentation mask. Specifically, the initial image is input into the first encoder to obtain feature data, and then the matched position information is input into the second encoder to obtain enhanced position information. Finally, the feature data and the enhanced position information are input into the mask interpreter for decoding to obtain a mask result (segmentation mask).

[0058] In this way, the mask decoder receives the outputs from the two encoders (feature data and enhanced position information) and decodes the two parts of information to generate a segmentation mask. This allows the decoder to comprehensively consider image content and position information, thereby generating more accurate and detailed segmentation results.

[0059] In some embodiments, feature extraction is performed on the initial image to obtain feature data, including:

[0060] Inputting the initial image into the first encoder to pre-process the initial image based on a preset format;

[0061] Marking position information in the preprocessed data to obtain target format data;

[0062] A preset visual converter model is used to extract features of the target format data to obtain feature data, and the feature data is used to characterize a feature image obtained after encoding the initial image.

[0063] In this embodiment, the first encoder may be an image encoder Φe, which maps the initial image to a feature space so as to subsequently extract features from the initial image. The initial image is preprocessed based on a preset format, and the initial image may be converted into a plurality of vectors of fixed dimensions by image segmentation and linear projection.

[0064] In one example, Figure 4 is a schematic diagram of an encoding process of a first encoder according to an embodiment of the present invention; Figure 4 As shown, in the process of encoding the initial image by the image encoder Φe, the initial image can be first divided into multiple fixed-size image blocks (patches), and the size of each block is usually 16x16 pixels. Assuming that the size of the initial image is 224x224 pixels, the number of patches after division is (224 / 16)x(224 / 16)=196.

[0065] After the initial image is segmented, each image block can be input into a linear layer and converted into a vector representation of a fixed dimension, so that the pixel value of the image block can be converted into a higher-level feature representation, thereby realizing the preprocessing of the initial image based on a preset format.

[0066] Since the Transformer module itself does not have the ability to process the position information of elements in the sequence, it is necessary to mark the position information in the preprocessed data to obtain the target format data. For example, a position encoding vector can be added to each linearly projected image block. In this way, after adding the position encoding, the target format data contains the feature information and position information of the image block.

[0067] After obtaining the target format data, the preset visual converter model can be used to extract features from the target format data to obtain feature data. For example, Figure 5 is a schematic diagram of the Transformer module structure of a first encoder according to an embodiment of the present invention, and the structure of the preset visual converter model is as follows Figure 5 As shown, the multi-head attention mechanism in the preset visual transformer model can effectively capture the global context and long-range dependencies in the target format data, and enrich the understanding of the target format data by focusing on different aspects of the relationship between tags of each head. Finally, the output of the Transformer module is compressed through the convolutional dimensionality reduction module to obtain the image embedding (corresponding to the feature data of this application).

[0068] In this way, by using the first encoder to extract features from the initial image according to the position information, the initial image is encoded into a feature map, which provides reliable feature data for subsequent decoding to generate a segmentation mask.

[0069] In some embodiments, data enhancement is performed on the position information by a second encoder to obtain enhanced position information, including:

[0070] Inputting the position information into a second encoder to position encode the position information using a prompt loose encoding to obtain position encoding data;

[0071] Boundary information in the position information is extracted, and data enhancement is performed on the position encoding information based on the boundary information to obtain enhanced position information.

[0072] In this embodiment, the second encoder may be a hint encoder Φp, which is used to position-code the position information B using hint sparse coding to obtain position-coded data. For example, the hint sparse coding may first transform the position information B into a coordinate system, then normalize the transformed position information, and finally normalize the normalized position information B. embeddings Gaussian random matrix and sinusoidal periodic coding are used to obtain position coding data.

[0073] In one example, the position encoding data Φp(B) can be expressed as formula (3):

[0074] Φp(B)=concat[sin(2πB embeddings *N(0,1) 2x64 ][sin(2πB embeddings *N(0,1) 2x64 ] (3)

[0075] In this embodiment, the boundary information may include the boundary coordinates of the bounding box of the local window in the initial image. In order to further highlight the position information of the bounding box in the initial image, the starting point and the end point of the bounding box may be weighted to achieve data enhancement, thereby obtaining enhanced position information.

[0076] It is understandable that by establishing a semantic association between the position information and the feature data, the model can obtain a better image segmentation effect than performing image segmentation based solely on the feature data. In this embodiment, by performing data enhancement on the position information, the enhanced position information obtained can be better semantically associated with the feature data, thereby obtaining a more accurate image segmentation result.

[0077] In some embodiments, the feature data and the enhanced position information are input into a mask decoder for decoding to obtain a segmentation mask and an intersection-over-union score, including:

[0078] Use attention mechanism to process feature data and enhance position information;

[0079] The processed data is iteratively operated using a multi-layer perceptron to obtain segmentation masks and intersection-over-union scores.

[0080] In this embodiment, by using the attention mechanism to process the feature data and the enhanced position information, it helps the mask decoder to find relevant features in the feature data according to the enhanced position information, thereby establishing a semantic association between the feature data and the enhanced position information.

[0081] After establishing the semantic association between the feature data and the enhanced position information, a multi-layer perceptron operation is performed on the feature data and the enhanced position information to update their representation. The multi-layer perceptron can introduce nonlinear transformations to help the model capture more complex feature combinations.

[0082] In one example, Figure 6 is a schematic diagram of a decoding process of a mask decoder according to an embodiment of the present invention, such as Figure 6 As shown, in order to better fuse feature data and enhance location information, a multi-layer perceptron can be used to iterate multiple rounds of data operations.

[0083] After all decoding layers are processed, the fused features are input into the final prediction layer. The prediction layer is responsible for generating segmentation masks and corresponding IoU values ​​based on these features.

[0084] In this way, through the decoding of the mask decoder, the segmentation results of the parts in the initial image of the initial segmentation model and the corresponding IoU values ​​are obtained.

[0085] In some embodiments, an attention mechanism is used to process feature data and enhanced position information, including:

[0086] The self-attention mechanism is used to process the enhanced position information;

[0087] A cross-attention mechanism is used to process data to enhance position information and feature data.

[0088] In this embodiment, a self-attention mechanism can be first applied to the enhanced position information, which helps the model understand the internal relationships of the enhanced position information, such as the spatial or semantic relationship between multiple position information. Then, the enhanced position information can be used as a query (Query) to perform a cross-attention operation with the feature data. In this way, by searching for relevant features in the feature data through the enhanced position information, a semantic association between the enhanced position information and the feature data can be established.

[0089] By using the attention mechanism to process feature data and enhanced position information, the model’s understanding of the internal relationships of the enhanced position information is enhanced, and the semantic association between the enhanced position information and the feature data is established, which can effectively improve the model’s understanding ability and obtain more accurate image segmentation results.

[0090] In some embodiments, the segmentation mask is filtered using an intersection-over-union ratio score to obtain distilled data, including:

[0091] Obtaining an evaluation result input by a user, wherein the evaluation result is used to characterize the accuracy of the segmentation mask segmentation parts and the initial image;

[0092] When both the evaluation result and the intersection-over-union ratio score exceed the preset threshold, the segmentation mask is added to the distillation dataset to obtain the distilled data.

[0093] In this embodiment, before determining a high-quality mask, it is necessary to obtain the evaluation result input by the user, and when both the evaluation result and the intersection-over-union score exceed a preset threshold, the segmentation mask is added to the distillation data set to ensure the quality of the segmentation mask. The preset threshold is obtained based on the user's comprehensive evaluation of the accuracy of the segmented parts and the segmentation calculation time, and this application does not limit this.

[0094] If the evaluation result and the intersection-over-union score both exceed the preset threshold, the segmentation mask can be determined to be a high-quality mask. After that, the segmentation mask can be batch-screened separately with the intersection-over-union score of the segmentation mask as the preset threshold, thereby speeding up the screening speed. Specifically, formula (4) and formula (5) can be used to batch-screen the segmentation mask and add it to the distillation dataset ImageSet new In the distillation data, we get

[0095] ImageSet new ={I|I∈ImageSet∧Conf(I)>=c-δ} (4)

[0096] MaskSet new ={Φd(Φe(I),Φp(B(I,T))|I∈ImageSet∧Conf(I)>=c-δ,T} (5)

[0097] Among them, Conf(I) is the IoU value of the initial image, c is the IoU of the high-quality mask confirmed by the user, and δ is the fluctuation coefficient of the confidence.

[0098] It can be understood that the segmentation mask obtained by screening based on the user evaluation result and the intersection-over-union score has higher quality. At the same time, by using the intersection-over-union score of the segmentation mask determined by the user evaluation result as a preset threshold, the segmentation mask is batch screened as a screening condition alone, which can speed up the screening process without continuous user evaluation.

[0099] In some embodiments, a method for training a part segmentation model is provided, the method comprising:

[0100] Get the preset lightweight model and distillation data;

[0101] Perform data enhancement on the distilled data to obtain enhanced distilled data;

[0102] The preset lightweight model is trained according to the enhanced distillation data to obtain the part segmentation model.

[0103] In this embodiment, since the model parameters of the initial segmentation model are large, the model calculation time is relatively long. In order to shorten the model calculation time, distillation data can be used to perform model distillation on the preset lightweight model to obtain a lightweight part segmentation model.

[0104] In an example, the model size of the initial segmentation model is 2445.75 MegaBytes, and the model size of the part segmentation model is 6.73 MegaBytes. Under the same experimental environment, the calculation time of the initial segmentation model is 1581.72 ms, and the calculation time of the part segmentation model is 4.2 ms.

[0105] In this embodiment, since the distilled data has been screened by the user's evaluation results and the intersection-over-union ratio score, the distilled data can be enhanced to avoid the technical problem that the part segmentation model has poor image segmentation accuracy corresponding to this part of the data due to the partial loss of the distilled data. For example, the distilled data can be enhanced by mosaic enhancement, illumination enhancement, size scaling, rotation, cropping, etc. to obtain the enhanced distilled data ImageSet aug .

[0106] In an example, formula (6) can be used to perform data augmentation on the distilled data:

[0107] ImageSet aug ={f crop-aug ·f rotate-aug ·f scale-aug ·f light-aug ·f mosaic-aug (I) | I∈ImageSet new} (6)

[0108] In order to understand this solution more clearly, the preset lightweight model is introduced below. Figure 7 is a schematic diagram of a structure of a preset lightweight model according to an embodiment of the present invention; Figure 7 As shown, the preset lightweight model can be a model of a convolutional neural network (CNN) structure, which can specifically include a CNN skeleton network, a feature pyramid network, and a detection head.

[0109] Among them, the CNN skeleton network structure can be CSP (Cross Stage Partial, cross-stage partial connection) Darknet or EfficientNet, Resnet, WideResnet and other network structures, which are mainly used for feature extraction. Figure 8 is a schematic diagram of a CNN skeleton network structure of a preset lightweight model according to an embodiment of the present invention, such as Figure 8 As shown in the figure, when the CNN skeleton network structure is the network structure of CSPDarknet, it specifically includes a residual CSP structure, a convolution module, and feature pooling. Based on the above structure, the input can be converted into multi-scale features.

[0110] Among them, the feature pyramid network can also adopt the CSP structure, which is mainly used to fuse the multi-scale features extracted by the CNN skeleton network. Fig. 9 is a structural schematic diagram of a feature pyramid network and a detection head of a preset lightweight model according to an embodiment of the present invention; Fig. 9 As shown in the figure, when the feature pyramid network adopts the CSP structure, it specifically includes the residual CSP structure, convolution module, concat connection, upsampling, etc. Based on the above structure, the multi-scale features can be converted into fused multi-scale features.

[0111] It can be understood that by inputting the enhanced distillation data into the preset lightweight model, the preset lightweight model can extract the features of the enhanced distillation data through the CNN skeleton network, integrate features of different scales through the feature pyramid network, and finally output it to the detection head of the segmentation network to complete the model training of the preset lightweight model and obtain the part segmentation model.

[0112] After completing the model training of the preset lightweight model, the weight coefficient λ1 of the preset lightweight model can be optimized by using a weighted loss function. For example, the loss functions including binary cross entropy loss, distribution focal loss, complete IoU loss, mask edge box loss, etc. can be used for optimization. In an example, the loss function calculation formula of the preset lightweight model is as follows: L total =λ1L DFL +λ2L CIoU +λ3L seg +λ4LBCE Among them, DFL (Distributional Feature Loss, data average fidelity loss function), CIOU (Complete Intersection over Union, complete intersection over union loss function), seg (Segmentation, image segmentation loss function), BCE (Binary Cross-Entropy, binary cross entropy loss function).

[0113] In some embodiments, a part defect detection method is provided, the method comprising: acquiring a part image; inputting the part image into a part segmentation model, and obtaining a part defect detection result output by the part segmentation model and corresponding to the part image.

[0114] Fig.10 A schematic diagram of the structure of a distillation data acquisition device for a part segmentation model provided in another embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0115] Reference Fig.10 , the distillation data acquisition device 1000 of the part segmentation model may include:

[0116] An acquisition module 1001 is used to acquire an initial image and a captured image of a part, wherein the captured image is a partial image obtained by capturing the initial image according to the position of the part in the initial image;

[0117] A matching module 1002 is used to match the initial image and the captured image to obtain the position information of the part in the initial image;

[0118] The segmentation module 1003 is used to input the position information and the initial image into the initial segmentation model to perform part segmentation, and obtain a segmentation mask and an intersection-over-union ratio score corresponding to the segmentation mask;

[0119] The screening module 1004 is used to screen the segmentation mask using the intersection-over-union ratio score to obtain distilled data.

[0120] In some embodiments, the initial segmentation model includes a first encoder, a second encoder and a mask decoder; the segmentation module 1003 may include the following units:

[0121] An extraction unit, configured to extract features from the initial image according to the position information through a first encoder to obtain feature data;

[0122] An enhancement unit, configured to perform data enhancement on the position information through a second encoder to obtain enhanced position information;

[0123] The decoding unit is used to input the feature data and the enhanced position information into the mask decoder for decoding to obtain the segmentation mask and the intersection-and-union ratio score.

[0124] In some embodiments, the extraction unit may include the following subunits:

[0125] A processing subunit, configured to input the initial image into a first encoder to pre-process the initial image based on a preset format;

[0126] A marking subunit, used for marking position information in the preprocessed data to obtain target format data;

[0127] The extraction subunit is used to extract features from the target format data using a preset visual converter model to obtain feature data, and the feature data is used to characterize a feature image obtained after encoding the initial image.

[0128] In some embodiments, the enhancement unit may include the following subunits:

[0129] The encoding subunit is used to input the position information into the second encoder to perform position encoding on the position information by using the hint loose encoding to obtain position encoding data;

[0130] The enhancement subunit is used to extract boundary information from the position information, and perform data enhancement on the position encoding information based on the boundary information to obtain enhanced position information.

[0131] In some embodiments, the decoding unit may include the following subunits:

[0132] A processing subunit, for performing data processing on feature data and enhanced position information by using an attention mechanism;

[0133] The operator unit is used to iteratively perform data operations on the processed data using a multi-layer perceptron to obtain a segmentation mask and an intersection-over-union ratio score.

[0134] In some embodiments, the above-mentioned processing subunit is also used to adopt a self-attention mechanism to perform data processing on the enhanced position information; and adopt a cross-attention mechanism to perform data processing on the enhanced position information and feature data.

[0135] In some embodiments, the screening module 1004 may include the following units:

[0136] An acquisition unit, used to acquire an evaluation result input by a user, wherein the evaluation result is used to characterize the accuracy of the segmentation mask segmentation parts and the initial image;

[0137] The matching unit is used to add the segmentation mask to the distillation data set to obtain the distilled data when both the evaluation result and the intersection-over-union ratio score exceed a preset threshold.

[0138] In some embodiments, the distillation module 1005 may include the following units:

[0139] An enhancement unit, used for performing data enhancement on the distillation data to obtain enhanced distillation data;

[0140] The training unit is used to perform model training on a preset lightweight model according to enhanced distillation data to obtain a part segmentation model.

[0141] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application, and are devices corresponding to the distillation data acquisition method of the above-mentioned part segmentation model. All implementation methods in the above-mentioned method embodiment are applicable to the embodiment of the device. Its specific functions and technical effects can be found in the method embodiment part, which will not be repeated here.

[0142] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0143] Fig.11 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Fig.11 As shown:

[0144] The device may include a processor 111 and a memory 112 storing programs or instructions.

[0145] When the processor 111 executes the program, the steps in any of the above method embodiments are implemented.

[0146] Exemplarily, the program may be divided into one or more modules / units, one or more modules / units are stored in the memory 112, and executed by the processor 111 to complete the present application. One or more modules / units may be a series of program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the program in the device.

[0147] By way of example, the processor 111 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0148] The memory 112 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 112 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 112 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 112 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 112 is a non-volatile solid-state memory.

[0149] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) machine-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0150] The processor 111 implements any one of the methods in the above embodiments by reading and executing the program or instruction stored in the memory 112 .

[0151] In one example, the electronic device may further include a communication interface 113 and a bus 114. The processor 111, the memory 112, and the communication interface 113 are connected via the bus 114 and communicate with each other.

[0152] The communication interface 113 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0153] Bus 114 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 114 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0154] In addition, in combination with the method in the above embodiment, the embodiment of the present application may provide a machine-readable storage medium for implementation. The machine-readable storage medium stores a program or instruction; when the program or instruction is executed by a processor, any one of the methods in the above embodiment is implemented. The machine-readable storage medium can be read by a machine such as a computer.

[0155] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0156] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0157] An embodiment of the present application provides a computer program product, which is stored in a machine-readable storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0158] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0159] The functional modules shown in the above block diagram can be implemented as hardware, software, firmware or their combination. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer grids such as the Internet, intranets, etc.

[0160] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0161] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by a computer program or instruction. These programs or instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0162] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for acquiring distillation data of a part segmentation model, characterized in that: The method comprises: Acquire an initial image and a captured image of a part, wherein the captured image is a partial image obtained by capturing the initial image according to a position of the part in the initial image; Matching the initial image and the captured image to obtain position information of the part in the initial image; Inputting the position information and the initial image into an initial segmentation model to perform part segmentation, and obtaining a segmentation mask and an intersection-over-union ratio score corresponding to the segmentation mask; The segmentation mask is filtered using the intersection-over-union ratio score to obtain distilled data.

2. The method according to claim 1, characterized in that The initial segmentation model includes a first encoder, a second encoder and a mask decoder; The step of inputting the position information and the initial image into an initial segmentation model to perform part segmentation, and obtaining a segmentation mask and an intersection-over-union ratio score corresponding to the segmentation mask, comprises: Extracting features from the initial image using the first encoder according to the position information to obtain feature data; Performing data enhancement on the position information by the second encoder to obtain enhanced position information; The feature data and the enhanced position information are input into the mask decoder for decoding to obtain the segmentation mask and the intersection-over-union score.

3. The method according to claim 2, characterized in that The extracting features of the initial image to obtain feature data includes: Inputting the initial image into the first encoder to pre-process the initial image based on a preset format; Marking the position information in the preprocessed data to obtain target format data; A preset visual converter model is used to perform feature extraction on the target format data to obtain the feature data, and the feature data is used to characterize a feature image obtained after encoding the initial image.

4. The method according to claim 2, characterized in that: The step of performing data enhancement on the position information by the second encoder to obtain enhanced position information includes: Inputting the position information into the second encoder to position encode the position information using hint loose coding to obtain position encoding data; Boundary information in the position information is extracted, and data enhancement is performed on the position encoding information based on the boundary information to obtain the enhanced position information.

5. The method according to claim 2, characterized in that: The step of inputting the feature data and the enhanced position information into the mask decoder for decoding to obtain the segmentation mask and the intersection-over-union score includes: Using an attention mechanism to perform data processing on the feature data and the enhanced position information; The processed data is iteratively operated using a multi-layer perceptron to obtain the segmentation mask and the intersection-over-union ratio score.

6. The method according to claim 5, characterized in that The adopting the attention mechanism to perform data processing on the feature data and the enhanced position information includes: Using a self-attention mechanism to perform data processing on the enhanced position information; A cross-attention mechanism is used to perform data processing on the enhanced position information and the feature data.

7. The method according to claim 1, characterized in that The using the intersection-over-union ratio score to filter the segmentation mask to obtain distilled data includes: Obtaining an evaluation result input by a user, wherein the evaluation result is used to characterize the accuracy of the segmentation mask in segmenting the part and the initial image; When both the evaluation result and the intersection-over-union ratio score exceed a preset threshold, the segmentation mask is added to the distillation data set to obtain the distillation data.

8. A method for training a part segmentation model, characterized in that: The method comprises: Obtain a preset lightweight model and distillation data, wherein the distillation data is obtained according to the method described in any one of claims 1 to 7; Performing data enhancement on the distilled data to obtain enhanced distilled data; The preset lightweight model is trained according to the enhanced distillation data to obtain a part segmentation model.

9. A method for detecting part defects, characterized in that: The method comprises: Get part images; The part image is input into a part segmentation model to obtain a part defect detection result corresponding to the part image output by the part segmentation model, wherein the part segmentation model is obtained according to the method of claim 8.

10. A distillation data acquisition device for a part segmentation model, characterized in that: The device comprises: An acquisition module, used for acquiring an initial image and a captured image of a part, wherein the captured image is a partial image obtained by capturing the initial image according to a position of the part in the initial image; A matching module, used for matching the initial image with the captured image to obtain position information of the part in the initial image; A segmentation module, used for inputting the position information and the initial image into an initial segmentation model to perform part segmentation, and obtaining a segmentation mask and an intersection-over-union ratio score corresponding to the segmentation mask; A screening module is used to screen the segmentation mask using the intersection-over-union ratio score to obtain distilled data.

11. An electronic device, characterized in that: The device comprises: a processor and a memory storing a computer program or instruction; When the processor executes the computer program instructions, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

Images