Instance segmentation method and device based on deep learning in high-precision scene

By combining Faster R-CNN and Sam models, using high-precision microscope to capture data, the precise example segmentation of conductive particles on the gold finger is achieved, and the problems of difficulty in segmenting small and medium-sized targets and strong data dependence in the existing technology are solved, and measurement is improved Efficiency and precision, suitable for automated measurement of high-precision scenarios.

CN119091437APending Publication Date: 2024-12-06GUANGDONG UNIV OF TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411140625.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

On high-precision measurement production lines, existing object detection models are difficult to achieve accurate instance segmentation of small goals, and most deep learning models are highly dependent on data and require a large amount of annotated data for training, resulting in high data acquisition and labeling costs.

Method used

High-magnitude high-precision microscope is used to shoot data sets, Faster R-CNN is used as the preprocessing model to train a small number of annotated data sets, and combined with Sam as the postprocessing model to extract the size and feature information of the measured object from the image to achieve Accurate example segmentation of conductive particles on gold fingers.

Benefits of technology

It realizes efficient and high-precision instance segmentation, reduces manual operation costs, improves measurement efficiency, reduces training time of deep learning models, overcomes the impact of traditional measurement instruments on ambient light and complex shapes, and is suitable for automation of complex scenarios. Measurement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091437A_ABST
    Figure CN119091437A_ABST
Patent Text Reader

Abstract

The invention relates to an instance segmentation method and device based on deep learning in a high-precision scene, and the method comprises the steps: firstly obtaining an image data set which is the sum of surface images of a to-be-trained workpiece; annotating the image data set by using an annotation plug-in to generate an annotated information graph; then, the image data set and the graph with the annotated information are sent into a trainer, and a training result weight file is generated; according to the method, two deep learning models of Faster Rcnn and Sam are combined, the problem that a Faster Rcnn target detection model is difficult to segment a small target is solved, the problem that Sam has no label information output and cannot perform instance segmentation is also solved, manual processing is not needed in the segmentation process, conductive particles on a golden finger can be segmented, contour information can be obtained for measurement, and the method is suitable for large-scale popularization and application. The optimal implementation effect of the instance segmentation task on small target segmentation in the field of high-precision measurement is achieved, and an efficient, high-quality, automatic and intelligent technology is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to an instance segmentation method and device in a high-precision scenario. Background Art

[0002] In the field of precision component measurement, traditional measurement tools (such as coordinate measuring machines (CMMs) and laser interferometers), although they can provide high-precision measurement results, have certain limitations in processing complex geometries and improving measurement speed.

[0003] Then people introduced deep learning methods, and through automation and intelligent technology, they solved the problem of slow measurement speed in large-scale or highly complex objects, and have achieved good results in applications in many fields. For example, an automatic flaw detection system for metal plate surfaces uses machine vision technology to automatically inspect metal surface defects, and performs high-speed and accurate detection during the production process. At the same time, the use of non-contact measurement avoids the possibility of new scratches. Another example is a defect detection method based on yolov8, which can locate and classify ceramic surface defects by training a large number of data sets.

[0004] In this field, the present invention specifically aims at the problem that high-precision measurement production lines require manual visual positioning and data processing after annotation to obtain measurement data, and provides an instance segmentation method based on deep learning in high-precision scenarios. The method aims to achieve the segmentation task of precision components through deep learning technology, obtain the size information of the segmented objects, replace manual operations, and improve measurement efficiency.

[0005] But the reality is that most target detection models do not have the ability to segment instances. Even if they have the ability to segment, the segmented objects are not ideal, especially in the detection and segmentation tasks of small targets. For example, when segmenting the conductive particles on the gold finger, the yolo target detection model is used. Although this model has the segmentation function, after testing, it is found that the segmentation contours obtained by yolo will have incorrect phenomena such as squares and sharp corners. In addition, most deep learning models used as pre-processing are too dependent on data and require a large amount of labeled data for training, with high data acquisition and labeling costs.

[0006] Therefore, the present invention uses a high-magnification and high-precision microscope to shoot the data set before segmentation, uses Faster Rcnn as pre-processing, trains a small amount of labeled data set, and then uses Sam as post-processing to extract the size and feature information of the measured object from the image; that is, the present invention combines the two deep learning models of Faster Rcnn and Sam, which solves the problem that the Faster Rcnn target detection model is difficult to segment small targets; and solves the problem that Sam has no label information output and cannot perform instance segmentation, thereby achieving efficient and high-precision instance segmentation. Summary of the invention

[0007] To solve the above problems, the present invention provides an instance segmentation method and device based on deep learning in a high-precision scenario, which can accurately segment small targets such as conductive particles on gold fingers.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] According to an embodiment of the first aspect of the present invention, a method for instance segmentation based on deep learning in a high-precision scenario comprises:

[0010] S1 acquires an image data set, where the image data set is the sum of surface images of a workpiece to be trained;

[0011] S2 uses an annotation plug-in to annotate the image data set to generate an annotated information graph;

[0012] S3 sends the image data set and the labeled information graph to the trainer to generate a training result weight file;

[0013] S4 divides the surface image of the workpiece to be segmented from the image data set, and sends it and the training result weight file to the inference engine, and the training result weight file performs inference on the image to generate segmentation result information;

[0014] S5 maps the segmentation result information onto the original inference image to segment the contour information of the object.

[0015] Preferably, the method of sending the data set to a trainer to obtain a training result weight file comprises:

[0016] S301 uses the ImageNet pre-trained weights to initialize the shared convolutional layer of the RPN network, and randomly initializes the RPN-specific layer, and then trains the RPN network to update the weights of the shared convolutional layer and the RPN-specific layer;

[0017] S302 uses the trained RPN network to generate candidate regions;

[0018] S303 uses the ImageNet pre-trained weights to initialize the shared convolutional layer of the Faster RCNN network again, and randomly initializes the Faster RCNN-specific layer, and then trains the Faster RCNN network to update the weights of the shared convolutional layer and the Faster RCNN-specific layer;

[0019] S304 uses the trained shared convolutional layer and RPN-specific layer to reinitialize the RPN network;

[0020] S305 uses the trained RPN network to generate candidate regions again;

[0021] S306 uses the trained shared convolutional layer and Faster RCNN-specific layer to reinitialize the Faster RCNN network and train the Faster RCNN network for the second time.

[0022] Preferably, the surface image of the workpiece to be segmented is divided from the image data set, and the surface image and the training result weight file are sent to the inference engine, and the training result weight file is used to infer the image to generate segmentation result information, and the method includes:

[0023] S401 inputs the image into the convolutional network to obtain the corresponding feature map ①;

[0024] S402 takes the feature map ① and generates a candidate box using the RPN structure;

[0025] S403 takes the candidate box and projects it onto the feature map ① to obtain a corresponding feature matrix;

[0026] S404 takes the feature matrix and scales it to a feature map of size 7*7 through a ROI pooling layer②;

[0027] S405: flatten the feature map ② and pass it through a series of fully connected layers to obtain a prediction result;

[0028] S406 sends the image to the SAM network, and the obtained prediction result assists in segmenting the target object to obtain mask information after SAM segmentation;

[0029] S407 maps the obtained mask information after SAM segmentation onto the original inference image to obtain the contour information of the segmented object.

[0030] Preferably, during the training process of S304, the weights of the shared convolutional layer need to be fixed.

[0031] Preferably, the data format of the annotated information graph generated in S2 needs to be converted into the input format of Faster RCNN.

[0032] Preferably, the number of images in the image data set is 42 to 56.

[0033] According to an embodiment of the second aspect of the present invention, a device for instance segmentation based on deep learning in a high-precision scenario comprises: a microscope, a storage, a processor, and a computer program stored in the storage and executable on the processor. When the computer program is executed by the processor, a method for instance segmentation based on deep learning in a high-precision scenario is implemented as described in any one of claims 1 to 6.

[0034] Preferably, the device is equipped with an RTX4060 graphics card.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The method based on deep learning in the present invention is used to replace manual labor, avoiding the problem that most traditional measurement technologies require skilled workers to operate, have large errors, and are inefficient in large-scale measurements, effectively reducing labor costs and improving measurement efficiency;

[0037] 2. The present invention uses high-precision measuring equipment to obtain high-pixel, high-magnification pictures, and then the annotation plug-in automatically completes the annotation of the pictures. The complex scenes are processed by the model self-learning method, avoiding the influence of ambient light, surface reflection, etc. on traditional measuring instruments. It can also easily measure complex shapes or difficult-to-reach areas, thereby improving the applicability and measurement accuracy of the measurement technology in different scenarios;

[0038] 3. The segmentation task pre-processing in the present invention uses the Faster RCNN model, which overcomes the strong dependence of most learning models on data, and can achieve good training results by inputting only about 50 pictures. Under given hardware conditions, it only takes 10 minutes to train the model for about 20 epochs, which greatly reduces the training time of the deep learning model, reduces the dependence on data, and reduces the time required to build a data set;

[0039] 4. The present invention uses Sam as post-processing, which overcomes the fact that most target detection models (Faster Rcnn) in the prior art do not have the ability to segment instances. Through its own powerful segmentation ability, the size and feature information of the measured object are extracted from the image, solving the problem of difficult measurement of complex scenes and low generalization ability;

[0040] 5. The present invention combines the two deep learning models, Faster Rcnn and Sam, which solves the problem that the FasterRcnn target detection model is difficult to segment small targets, and solves the problem that Sam cannot perform instance segmentation because it has no label information output. The segmentation process does not require manual processing, and the conductive particles on the gold finger can be segmented and the contour information can be obtained for measurement, achieving the best implementation effect of the instance segmentation task for small target segmentation in the field of high-precision measurement, and realizing efficient, high-quality, automated and intelligent technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall flow chart of the segmentation task in the present invention;

[0042] Figure 2 is a flow chart of the training task in the present invention;

[0043] Figure 3 is a flow chart of the reasoning task in the present invention;

[0044] Figure 4 The original image provided for model training in the embodiment of the present invention;

[0045] Figure 5 It is a labeled information graph provided for model training in an embodiment of the present invention;

[0046] Figure 6 The original image to be segmented according to an embodiment of the present invention;

[0047] Figure 7 This is a contour display diagram obtained after segmentation of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Example 1

[0050] refer to Figure 1 , an example segmentation method based on deep learning in a high-precision scenario provided by an embodiment of the present application includes the following steps:

[0051] Step S1, obtaining an image data set, wherein the image data set is the sum of surface images of a workpiece to be trained.

[0052] In an optional embodiment, a high-magnification and high-precision microscope (such as a scanning electron microscope or a high-magnification digital microscope) is used to capture images of the surface of the gold finger under the illumination of a coaxial light source. The high-magnification and high-precision microscope has extremely high resolution and can observe the fine details of tiny sample structures and surface morphology. For example, the secondary electron imaging resolution of a scanning electron microscope has reached 3 to 4 nm, which can clearly capture nanometer-level structural features and is highly adaptable to precise instance segmentation of small targets such as conductive particles.

[0053] Step S2: annotate the image data set using an annotation plug-in to generate an annotated information graph.

[0054] In an optional embodiment, annotation plug-ins such as Specees 2, Measure, and Design Annotation Plugin are used to annotate images in the image data set, and the annotation content is the conductive particles on the surface of the gold finger. After annotation, an annotated information graph is generated, and the annotated information graph is output to obtain a json file in ISAT format, which is stored and used for subsequent model training.

[0055] Step S3, sending the image data set and the annotated information graph to a trainer to generate a training result weight file.

[0056] In an optional embodiment, any target detection framework such as R-CNN, Faster R-CNN, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector) is used to achieve rapid generation of candidate regions, and based on the generated candidate regions, the image data set and the annotated information graph are trained multiple times to obtain a training result weight file.

[0057] Step S4, dividing the surface image of the workpiece to be segmented in the image data set, sending it and the training result weight file to the inference engine, and the training result weight file infers the image to generate segmentation result information.

[0058] In an optional embodiment, any model such as Sam, Seem, fasterSam, etc. is used to infer the image to be segmented based on the training result weight file to generate segmentation result information.

[0059] Step S5, mapping the segmentation result information onto the original inference image to segment the contour information of the object.

[0060] In this embodiment, a high-magnification and high-precision microscope is first used to collect images of the surface of the gold finger to form an image data set, and then a labeling plug-in is used to label the images in the image data set, and the labeling content is the conductive particles on the surface of the gold finger. After labeling, a labeled information map is generated, and then the image data set and the labeled information map are sent to the trainer to realize the rapid generation of candidate areas, and based on the candidate areas generated above, the image data set and the labeled information map are trained multiple times to obtain a training result weight file, and then based on the training result weight file, the image to be segmented is inferred to generate segmentation result information, and then the segmentation result information is mapped to the inference original image to segment the contour information of the object (conductive particles). In the segmentation process, there is no need for manual visual positioning and data labeling processing. The deep learning technology is used to realize the segmentation task of precision zero components, and the size information of the segmented object can be obtained, replacing manual operation and improving measurement efficiency.

[0061] Example 2

[0062] refer to Figure 1-Figure 2 In a preferred embodiment, step S3, the image data set and the annotated information graph are sent to the trainer to generate a training result weight file. Among them, any target detection framework such as R-CNN, Faster R-CNN, YOLO, SSD, etc. can be used to achieve rapid generation of candidate regions. R-CNN is an earlier target detection framework based on regional convolutional neural network. Faster R-CNN is improved on the basis of R-CNN, and realizes one-time feature extraction of the entire image, and then maps the candidate region to the feature map to avoid repeated calculations; SSD combines the speed of YOLO and the accuracy based on the candidate region framework, uses multiple convolutional layers on feature maps of different scales to predict the category and bounding box of the target, and uses the anchor mechanism to generate candidate boxes of different sizes and aspect ratios, and the detection accuracy is also significantly improved.

[0063] Both Faster R-CNN and SSD have low dependence on data sets, and only require a small amount of image annotation data for training, effectively reducing the high cost of data acquisition and annotation. Further subdividing the two, SSD is a single-detection object detection algorithm, which has higher efficiency when processing large amounts of data; while Faster R-CNN is a two-stage object detection algorithm. The first stage generates candidate areas, and the second stage classifies and accurately locates these candidate areas. In theory, it has higher requirements for detailed information in the data (such as the accuracy of the bounding box). Although it is not as efficient as SSD, its accuracy is higher than SSD. Considering that this embodiment is used for accurate instance segmentation of small targets such as conductive particles, the Faster R-CNN model is selected as the training model.

[0064] Furthermore, after the annotated information graph data generated in step S2 is output as a json file in ISAT format, it needs to be converted into the input format of Faster RCNN; when using the SSD model, it needs to be converted into the corresponding NCHW (Num_batch, Channels, Height, Width) or NHWC (Num_batch, Height, Width, Channels) input format.

[0065] In a specific embodiment, referring to Figure 4-Figure 5 , the step S3 comprises:

[0066] S301 uses the ImageNet pre-trained weights to initialize the shared convolutional layer of the RPN network, and randomly initializes the RPN-specific layer, and then trains the RPN network to update the weights of the shared convolutional layer and the RPN-specific layer;

[0067] S302 uses the trained RPN network to generate candidate regions (proposals);

[0068] S303 uses the ImageNet pre-trained weights to initialize the shared convolutional layer of the Faster RCNN network again, and randomly initializes the Faster RCNN-specific layer, and then trains the Faster RCNN network to update the weights of the shared convolutional layer and the Faster RCNN-specific layer;

[0069] S304 uses the trained shared convolutional layer and RPN-specific layer to reinitialize the RPN network②;

[0070] S305 uses the trained RPN network to generate candidate regions (proposals) again;

[0071] S306 uses the trained shared convolutional layer and Faster RCNN-specific layer to reinitialize the Faster RCNN network and train the Faster RCNN network for the second time.

[0072] In this embodiment, the acquired image data set is used with a trained RPN network to generate candidate regions. Candidate regions refer to a series of regions that may contain objects (conductive particles) generated on the input image (image data set) by some method (deep learning). These regions serve as preliminary estimated positions for subsequent detection or recognition tasks, which help to improve the efficiency and accuracy of target detection; then the candidate regions are input into the Faster RCNN neural network for training to generate a training result weight file. After the training result weight file is trained, the embodiment provided by the present invention only needs to input the image to be cut and the training result weight file into the inference engine, and the training result weight file can automatically generate the segmentation result information, and then map the segmentation result information to the original inference image to segment the contour information of the object. No manual processing is required throughout the process, and multiple training of candidate regions is achieved to improve the accuracy of segmentation.

[0073] It should be noted that in step S304, the trained shared convolutional layer and RPN-specific layer are used to reinitialize the RPN network. The weights of the shared convolutional layer need to be fixed to ensure that they remain unchanged during training, and only the weights of the RPN-specific layer are trained. It is understandable that at this time the shared convolutional layer has been optimized through pre-training, and image features have been learned. These features are universal in most cases. Therefore, fixed weights can not only avoid overfitting when training RPN, but also significantly reduce the amount of calculation and increase the training speed. Furthermore, this step enables RPN and Fast R-CNN to share convolutional layers to form a unified network architecture. This unified network architecture not only simplifies the model structure, but also improves the overall performance of the model.

[0074] Moreover, due to the low dependence of Faster RCNN on the data set, the number of images required to be annotated is small. After training experiments with input of various numbers of images, it is found that this embodiment can achieve good training results by inputting only about 50 images. Under given hardware conditions, it only takes 10 minutes to train the model for about 20 epochs. "Epochs" refers to the number of times the entire training data set is traversed (i.e., used for training). In other words, one epoch represents that the model has learned all the samples in the training set once. In this embodiment, about 20 epochs of training time are required, and the entire training process only takes about 10 minutes, which is very short, greatly improving the efficiency of pre-processing in segmentation technology.

[0075] Example 3

[0076] refer to Figure 1In a preferred embodiment, in step S4, the surface image of the workpiece to be segmented is divided from the image data set, and the surface image and the training result weight file are sent to the inference engine, and the training result weight file is used to infer the image to generate segmentation result information. Among them, any model such as Sam, Seem, fasterSam, etc. can be used to infer the image to be segmented based on the training result weight file to generate segmentation result information.

[0077] In a specific embodiment, referring to Figure 3 , Figure 6 , Figure 7 , the Sam model is selected as the inference model, and step S4 includes: S401 inputting the image into the convolutional network to obtain the corresponding feature map ①;

[0078] S402 takes the feature map ① and generates a candidate box using the RPN structure;

[0079] S403 takes the candidate box and projects it onto the feature map ① to obtain a corresponding feature matrix;

[0080] S404 takes the feature matrix and scales it to a feature map of size 7*7 through a ROI pooling layer②;

[0081] S405: flatten the feature map ② and pass it through a series of fully connected layers to obtain a prediction result;

[0082] S406 sends the image to the SAM network, and the obtained prediction result assists in segmenting the target object to obtain mask information after SAM segmentation;

[0083] S407 maps the obtained mask information after SAM segmentation onto the original inference image to obtain the contour information of the segmented object.

[0084] In this embodiment, the corresponding feature map obtained is a high-dimensional feature representation extracted from the image after a series of convolution operations. These feature maps capture the details, textures, edges, and higher-level semantic information in the image, which is crucial for subsequent image segmentation. During the convolution process, the feature map is usually downsampled through a pooling layer to reduce the spatial dimension (i.e., height and width) of the data while retaining important information. Downsampling helps reduce the amount of model calculations and increase the model's receptive field.

[0085] Next, the feature map refers to the feature matrix generated after network extraction. The feature matrix is ​​scaled to a 7*7 feature map through the ROI pooling layer, which is used to classify the receptive field. The larger the receptive field value, the larger the image range it can contact, and the smaller the receptive field value, the more local and detailed the features it contains. Understandably, given the segmented image area of ​​the golden finger, the sampling size of the 7*7 feature map will be smaller than 3*3, and then the receptive field value of the 7*7 feature map will be smaller, more local and detailed, effectively improving its reasoning accuracy.

[0086] The fully connected layer is mainly used in the convolutional neural network (CNN) to integrate and map the features extracted by the previous layers (the convolution layer and the pooling layer in this embodiment) to finally obtain the prediction result (detection result). Its essence is a classifier or regressor. As shown in Figure *, the integration operation of the fully connected layer specifically includes but is not limited to the following methods:

[0087] Softmax is used in the output layer of multi-classification problems to convert the original output of the convolutional neural network (CNN) into a probability distribution, which makes it easier to make classification decisions based on the probability size. The calculation formula is:

[0088]

[0089] Where zi is the i-th element of the input vector z, e is the base of the natural logarithm, and ∑jezj is the sum of all input elements after exponential operation.

[0090] bbox_pre refers to the bounding box parameters predicted by the model in the target detection task. These parameters usually include the center point coordinates (x, y), width (w) and height (h) of the bounding box, or the upper left and lower right corner coordinates of the bounding box (x1, y1, x2, y2). Its essence is the model's prediction result for the target (conductive particle) position.

[0091] cls_prob (Classification Probability) refers to the model's predicted probability that the input sample belongs to each category. It is used to provide the model's prediction confidence for each category, and classification decisions can be made based on these probability values.

[0092] The sum of the three functions indirectly realizes the integration of all the features extracted by the pooling layer by the fully connected layer, integrates them into a more representative feature vector, and finally obtains the prediction result.

[0093] Based on the contents of Examples 1 and 2, the present invention first uses high-precision measurement equipment to obtain high-pixel, high-magnification images, greatly reducing the amount of data sets required for model training, and then combines the two deep learning models Faster Rcnn and Sam. Faster Rcnn has high training and learning efficiency, which solves the problem that most deep learning model training requires a lot of time and data, and at the same time improves the accuracy of the model for target detection; Sam has good segmentation capabilities, and extracts the size and feature information of the measured object from the image through segmentation, solving the problem that complex scenes are difficult to measure and have low generalization capabilities. After the two are combined, the problem that the Faster Rcnn target detection model is difficult to segment small targets is solved, and the problem that Sam cannot instance segment without label information output is solved. The segmentation process does not require manual processing, and the conductive particles on the gold finger can be segmented and the contour information can be obtained for measurement, achieving the best implementation effect of instance segmentation tasks for small target segmentation in the field of high-precision measurement, and realizing efficient, high-quality, automated and intelligent segmentation technology.

[0094] Example 4

[0095] According to the second aspect of the present application, the present invention also provides an instance segmentation device based on deep learning in a high-precision scenario, the device comprising: a microscope, a storage, a processor, and a computer program stored in the storage and executable on the processor, the device can be run in computing devices such as desktop computers, notebooks, PDAs, and cloud servers, and the executable devices include but are not limited to microscopes, storage, and processors. Those skilled in the art will appreciate that the examples are merely examples of instance segmentation devices based on deep learning in high-precision scenarios, and do not constitute a limitation on instance segmentation devices based on deep learning in high-precision scenarios, and may include more or fewer components than the examples, or a combination of certain components, or different components.

[0096] The microscope can be any type of imaging lens with extremely high resolution, such as a scanning electron microscope or a high-magnification digital microscope. For example, the secondary electron imaging resolution of a scanning electron microscope has reached 3 to 4 nm, which can clearly capture nanoscale structural features and is highly adaptable to precise instance segmentation of small targets such as conductive particles.

[0097] The processor can be any one of a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. It can use various interfaces and lines to connect the various executable parts in the deep learning-based instance segmentation device in the entire high-precision scenario and act as a control center.

[0098] The memory can be used to store the computer program and / or module, mainly including a program storage area and a data storage area, and can realize various functions of the instance segmentation device based on deep learning in the high-precision scenario by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. Specifically, it is a non-volatile memory, such as at least one storage device in a hard disk, a memory, a plug-in hard disk, a smart memory card, a secure digital card, and a flash memory card.

[0099] Example 5

[0100] A deep learning-based instance segmentation device in high-precision scenarios, with an RTX4060 graphics card as the hardware condition for processing tools, not only has powerful graphics processing capabilities, but can also support high-resolution rendering of images captured by high-precision microscopes; it itself supports NVIDIA technologies such as CUDA (Compute Unified Device Architecture) and TensorRT, which can significantly accelerate the training and reasoning process of deep learning models, and has powerful computing capabilities.

[0101] Based on the contents of Example 4, the RTX 4060 graphics card not only supports collaborative work with a processor (such as a CPU), the processor is responsible for executing logic control and complex algorithms, while the graphics card focuses on graphics processing and large-scale parallel computing, thereby maximizing system performance; it can also interact with memory (such as RAM) through the system bus, frequently read image data or calculation results from the memory, and store them in its own video memory for fast access.

[0102] In summary, the RTX 4060 graphics card is mainly responsible for graphics processing and computing acceleration in the device. It can use its powerful computing power to provide high-resolution rendering and real-time processing functions for images captured by high-precision microscopes, while supporting the accelerated execution of complex computing tasks such as deep learning. In addition, the graphics card also works with system components such as processors and memory to jointly achieve overall performance optimization of the device.

[0103] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0104] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A method and device for instance segmentation based on deep learning in high-precision scenarios, It is characterized in that The method comprises: S1 acquires an image data set, where the image data set is the sum of surface images of a workpiece to be trained; S2 uses an annotation plug-in to annotate the image data set to generate an annotated information graph; S3 sends the image data set and the labeled information graph to the trainer to generate a training result weight file; S4 divides the surface image of the workpiece to be segmented from the image data set, and sends it and the training result weight file to the inference engine, and the training result weight file performs inference on the image to generate segmentation result information; S5 maps the segmentation result information onto the original inference image to segment the contour information of the object.

2. The instance segmentation method based on deep learning in a high-precision scenario as claimed in claim 1, It is characterized in that The method of sending the data set to a trainer to obtain a training result weight file includes: S301 uses the ImageNet pre-trained weights to initialize the shared convolutional layer of the RPN network, and randomly initializes the RPN-specific layer, and then trains the RPN network to update the weights of the shared convolutional layer and the RPN-specific layer; S302 uses the trained RPN network to generate candidate regions; S303 uses the ImageNet pre-trained weights to initialize the shared convolutional layer of the Faster RCNN network again, and randomly initializes the Faster RCNN-specific layer, and then trains the Faster RCNN network to update the weights of the shared convolutional layer and the Faster RCNN-specific layer; S304 uses the trained shared convolutional layer and RPN-specific layer to reinitialize the RPN network; S305 uses the trained RPN network to generate candidate regions again; S306 uses the trained shared convolutional layer and Faster RCNN-specific layer to reinitialize the Faster RCNN network and train the Faster RCNN network for the second time.

3. The instance segmentation method based on deep learning in a high-precision scenario as claimed in claim 1, It is characterized in that The method includes dividing a surface image of a workpiece to be segmented from an image data set, sending the surface image and the training result weight file to a reasoner, and the training result weight file performs reasoning on the image to generate segmentation result information. S401 inputs the image into the convolutional network to obtain the corresponding feature map ①; S402 takes the feature map ① and generates a candidate box using the RPN structure; S403 takes the candidate box and projects it onto the feature map ① to obtain a corresponding feature matrix; S404 takes the feature matrix and scales it to a feature map of size 7*7 through a ROI pooling layer②; S405: flatten the feature map ② and pass it through a series of fully connected layers to obtain a prediction result; S406 sends the image to the SAM network, and the obtained prediction result assists in segmenting the target object to obtain mask information after SAM segmentation; S407 maps the obtained mask information after SAM segmentation onto the original inference image to obtain the contour information of the segmented object.

4. The instance segmentation method based on deep learning in a high-precision scenario as claimed in claim 3, It is characterized in that During the training process of S304, the weights of the shared convolutional layer need to be fixed.

5. The instance segmentation method based on deep learning in a high-precision scenario as claimed in claim 1, It is characterized in that The data format of the annotated information graph generated in S2 needs to be converted into the input format of FasterRCNN.

6. The instance segmentation method based on deep learning in a high-precision scenario as claimed in claim 5, It is characterized in that The number of images in the image data set is 42 to 56.

7. The instance segmentation device based on deep learning in a high-precision scenario as claimed in claim 1, It is characterized in that The device includes: a microscope, a storage, a processor, and a computer program stored in the storage and executable on the processor. When the computer program is executed by the processor, an instance segmentation method based on deep learning in a high-precision scenario is implemented as described in any one of claims 1-6.

8. The instance segmentation device based on deep learning in a high-precision scenario as claimed in claim 7, It is characterized in that The device is equipped with an RTX4060 graphics card.