Microscope lens collision early warning method and device, storage medium and program product

Abstract

The invention discloses a microscope lens collision early warning method and device, a storage medium and a program product, and belongs to the field of microscopes. According to the method, the double image segmentation models, namely the instance segmentation model and the semantic segmentation model, are set, detection and recognition of different detection targets and effective areas are achieved respectively, the two models interact and verify each other, and the accuracy of a final collision detection result is jointly improved. Especially for the detection of an effective area, the problem that the position of the detected object colliding with the lens first cannot be predicted through standard image straight line operation due to view angle difference is solved.

Description

Microscope lens collision warning methods, devices, storage media and software products Technical Field

[0001] This invention belongs to the field of microscopes, and specifically relates to a microscope lens anti-collision technology. Background Technology

[0002] When observing an object using an industrial microscope, the operator moves the lens up and down to adjust the distance between the lens and the object, ensuring a clear image. During this process, if the distance is inappropriate, the image will be blurry. Furthermore, during the operator's movement of the lens using the controller, there is a possibility of collision between the lens and the object, especially if the object is at a certain height. If the operator lowers the lens too quickly, a collision can easily occur, damaging both the object and the lens.

[0003] Existing methods to prevent microscope lenses from colliding with the object being measured include: 1. Using a distance sensor to measure the distance between the lens and the object, triggering a collision warning when the distance is too small; 2. Using a contact sensor installed next to the lens, triggering a collision warning when the sensor comes into contact with the object. The first method is a non-contact measurement, which is effective and does not affect the object, but it is more expensive, as laser distance sensors are generally costly, and fixing devices need to be added to both ends of the lens, increasing the overall lens weight. The second method is less expensive, but the installed sensor will come into contact with the object, potentially affecting it. Furthermore, this contact sensor has a limited lifespan and requires external circuitry to function properly, all of which limit the reliability of this method. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention proposes a microscope lens collision early warning method, device, storage medium, and program product.

[0005] To achieve the above technical objectives, the technical solution of this invention is as follows: a microscope lens collision warning method based on a dual segmentation model. A side camera is deployed on the microscope support to capture images of the microscope during operation from the side. The captured images are then input into a trained instance segmentation model and a semantic segmentation model, respectively. The instance segmentation model detects the current position information of the microscope lens and the object under test based on the input images. The semantic segmentation model detects whether a valid region exists based on the input images. The valid region refers to the region where the microscope lens and the object under test are most likely to collide first. If the semantic segmentation model does not detect the valid region, it is determined that there is no risk of collision, and the side camera is controlled to perform the next round of shooting. If the semantic segmentation model detects the valid region, it is determined that there is a risk of collision. Based on the detection results of the instance segmentation model, the vertical distance between the lower edge of the microscope lens and the upper edge of the object under test is calculated. If this distance exceeds a set safety threshold, a collision warning is triggered; otherwise, it is determined that there is no risk of collision, and the side camera is controlled to perform the next round of shooting.

[0006] Furthermore, the categories output by the instance segmentation model also include the stage. The detection results of the stage are used to verify the results of the effective region detected by the semantic segmentation model. The effective region should be completely contained within the range of the stage.

[0007] Furthermore, during the training of the instance segmentation model, by replacing different test objects, changing the position of the test objects on the stage, and adjusting the height of the microscope lens, the side camera acquires different images, and the targets are labeled on these images, thereby obtaining the training data of the instance segmentation model.

[0008] Furthermore, when preparing the training data for the instance segmentation model, the target is cut out from the image based on the annotation results, and the cut-out target image is pasted onto other different background images in the form of a texture to simulate different scenes.

[0009] Furthermore, when training the semantic segmentation model, the object to be tested is placed on a stage, and the microscope lens is adjusted so that it is about to contact the object. At this time, the side camera takes a picture and records the point where the microscope lens is about to contact the object. This point is marked in the image, and an effective area is delineated in the image with this point as the center and the area is N times the cross-sectional area of ​​the microscope lens. This yields the training data for the semantic segmentation model, where N is a real number greater than 1.

[0010] Secondly, the present invention also provides a computer device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the above-described microscope lens collision warning method based on a dual segmentation model.

[0011] Furthermore, the present invention also provides a computer storage medium storing a computer program, which, when executed on a processor, implements the microscope lens collision warning method based on the dual segmentation model described above.

[0012] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the microscope lens collision warning method based on the dual segmentation model described above.

[0013] The beneficial effects of adopting the above technical solution are as follows: This invention uses machine deep learning to detect the distance between the microscope lens and the object being measured, thereby achieving the purpose of early warning and collision avoidance. This avoids the drawbacks of traditional contact or non-contact sensor solutions, such as increased costs, interference with the object being measured, additional peripheral circuit design, and reduced reliability. Specifically, this invention sets up a dual image segmentation model—an instance segmentation model and a semantic segmentation model—to detect and identify different detection targets and effective regions, respectively. The two models interact and verify each other, jointly improving the accuracy of the final collision detection results. In particular, the detection of the effective region overcomes the problem that, due to differences in viewing angle, it is impossible to predict the location of the object being measured from the lens first using standard image straight-line calculations. Attached Figure Description

[0014] Figure 1 is a schematic diagram of the side camera installation; Figure 2 is a schematic diagram of the side camera shooting; Figure 3 is a schematic diagram of the instance segmentation model segmenting three types of targets; Figure 4 is a schematic diagram of the synthesis of target and different background images; Figure 5 is a schematic diagram of the effective area; Figure 6 is a schematic diagram of the vertical distance between the microscope lens and the object being measured; Figure 7 is a flowchart of one implementation scheme. Detailed Implementation

[0015] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0016] The basic idea of ​​this embodiment is to deploy a side camera on the microscope's support, which captures images of the lens and the object under test from the side. A deep learning model is used to identify and detect the lens and the object in the captured images, determining the current distance between them to provide a collision avoidance warning. The installation position of the side camera is shown in Figure 1. The captured images are shown in Figure 2, showing the lens, the object under test, and the stage from the side.

[0017] In this embodiment, an instance segmentation model is constructed to accurately detect the edges of the lens, the object under test, and the stage. Typically, the objects to be identified are the lens and the object under test, but the stage is also an important feature in the image. To facilitate better model training, the stage is also labeled as a separate object to be identified. Therefore, the instance segmentation model outputs three categories: lens, object under test, and stage. As shown in Figure 3, blue represents the edge of the microscope lens, green represents the edge of the object under test, and red represents the stage area.

[0018] The instance segmentation model structure used in this embodiment includes a feature extraction backbone and an instance segmentation head connected sequentially. The feature extraction backbone uses a commonly used high-speed model backbone, such as EfficientV2 or MobileNet. The instance segmentation head uses an instance segmentation head from the YOLO series.

[0019] Training a generalizable instance segmentation network model requires collecting a large amount of data and various targets for training. Commonly used objects to be detected are collected, such as metal blocks, colloidal blocks, resistors, capacitors, inductors, semiconductor materials like chips, plastic products, wooden items, nanomaterials, ceramic materials, polymer materials like rubber products, rocks like granite / limestone, minerals like quartz / mica, soil particles, and various other items. After preparing these items, they are placed at different positions on the microscope stage, and the lens is adjusted at different heights (approximately four height positions can be set: the highest position, the middle position, the position close to the object being measured, and the position where it will almost touch the object), and corresponding images are acquired by the side camera.

[0020] The collected data were labeled with categories: lens, object under test, and stage. All objects were uniformly labeled as "object under test," without distinguishing specific object categories. One issue here is that different backgrounds are seen from the side camera, making it impossible to fully simulate the backgrounds used in different scenarios. To address this, a texture mapping method was used to continuously change the background. The three labeled target categories were cut out from the original image and then textured onto other background images to simulate different backgrounds and improve the scene diversity of the dataset. Here, the selected "other background images" can come from the InteriorNet dataset, which contains over 20 million indoor scene images. Randomly selected indoor scene images were used as background images. An example of the texture mapping effect is shown in Figure 4.

[0021] Images captured by the side camera are fed into a trained instance segmentation model for instance segmentation and detection. The model detects the position of the current camera relative to the object being measured. A valid region is defined, located directly below the camera. If the bottom edge of the object is placed within this region, it is considered to be below the camera, and a collision warning is issued. Otherwise, no collision warning is issued. As shown in Figure 5, the area enclosed by the cyan lines is the valid region. The bottom edge of the object is calculated and marked, and it is determined whether the bottom edge of the object is within the defined valid region. The bottom edge of the camera and the top edge of the object are calculated. Based on the positions of these two edges, the vertical distance between the camera and the object is determined, and a collision warning is issued. As shown in Figure 6, the red line represents the bottom edge of the camera, and the blue line represents the top edge of the object.

[0022] The effective region indicates the part of the object most likely to first contact the lens (the point of collision). This effective region is related not only to the lens position but also to the shape and position of the object being measured. The effective region is defined centered on the point where the object is most likely to first contact the lens, and is then divided into regions N (N>1) times the lens size range. This effective region is determined by fully considering the relative positional relationship between the lens and the object. Theoretically, the lowest point of the lens is most likely to first contact the object, but due to the object's shape and its position on the stage, it may contact different points on the lens, resulting in a collision. Due to differences in perspective (not perfectly equal perspectives), standard image line calculations cannot predict the initial collision point. Therefore, a semantic segmentation model is constructed, allowing the model to actively learn which point on the lens is most likely to collide with the object in the current state, and using this to define a potential collision region, called the effective collision detection region.

[0023] The semantic segmentation model takes the image to be detected as input and outputs a mask of the effective region. The model structure consists of a feature extraction module and a semantic segmentation head connected sequentially. For feature extraction, mainstream CNN architectures such as EfficientNetv2 and MobileNetV3 can be used; lightweight networks are recommended to improve computational speed. The semantic segmentation head uses upsampling and convolution to output a binary mask image. To prepare training data for this semantic segmentation model, the object to be tested is placed on a platform, and the lens is lowered until it is about to touch the object. The point at which the two are about to touch is recorded and marked on the image. The effective region is then defined in the image with this point as the center and the lens cross-section is N (N>1) times larger.

[0024] This embodiment can be implemented according to the flowchart shown in Figure 7.

[0025] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A microscope lens collision warning method based on a dual segmentation model, characterized in that: A side camera is mounted on the microscope support to capture images of the microscope during operation from the side. These images are then input into a trained instance segmentation model and a semantic segmentation model. The instance segmentation model detects the current position information of the microscope lens and the object under test based on the input images. The semantic segmentation model detects whether a valid region exists, defined as the area where the microscope lens and the object are most likely to collide. If the semantic segmentation model does not detect a valid region, it determines that there is no risk of collision, and the side camera proceeds to the next capture. If the semantic segmentation model detects a valid region, it determines that there is a risk of collision. Based on the detection results of the instance segmentation model, the vertical distance between the lower edge of the microscope lens and the upper edge of the object under test is calculated. If this distance exceeds a set safety threshold, a collision warning is triggered; otherwise, it determines that there is no risk of collision, and the side camera proceeds to the next capture.

2. The microscope lens collision warning method based on a dual segmentation model according to claim 1, characterized in that: The category output by the instance segmentation model also includes the stage. The detection result of the stage is used to verify the result of the effective region detected by the semantic segmentation model. The effective region should be completely contained within the range of the stage.

3. The microscope lens collision warning method based on a dual segmentation model according to claim 1, characterized in that: When training the instance segmentation model, the side camera acquires different images by replacing different test objects, changing the position of the test objects on the stage, and adjusting the height of the microscope lens. The target is labeled on these images to obtain the training data for the instance segmentation model.

4. The microscope lens collision warning method based on a dual segmentation model according to claim 3, characterized in that: When preparing the training data for the instance segmentation model, the target is cut out from the image based on the annotation results, and the cut-out target image is pasted onto other different background images as a texture to simulate different scenes.

5. The microscope lens collision warning method based on a dual segmentation model according to claim 1, characterized in that: When training the semantic segmentation model, the object to be tested is placed on a stage, and the microscope lens is adjusted so that it is about to contact the object. At this time, a side camera takes a picture and records the point where the microscope lens is about to contact the object. This point is marked in the image. The effective area is delineated in the image with this point as the center and the area is N times the cross-sectional area of ​​the microscope lens. This is how the training data of the semantic segmentation model is obtained, where N is a real number greater than 1.

6. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the microscope lens collision warning method based on the dual segmentation model according to any one of claims 1-5.

7. A computer storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the microscope lens collision warning method based on a dual segmentation model according to any one of claims 1-5.

8. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by the processor, they implement the microscope lens collision warning method based on the dual segmentation model as described in any one of claims 1-5.

Images