Microscope-based biological experiment operation evaluation method, device and readable medium

Abstract

The application discloses a microscope-based biological experiment operation evaluation method and device and readable medium, wherein the method comprises the following steps: acquiring and processing the light-focusing image of the microscope in the biological experiment operation, performing edge detection on the processed light-focusing image to obtain edge information and determine the state of the positioning circle in the light-focusing image; acquiring the component state detected by the light-focusing operation component detection sensor, determining the light-focusing step operation result according to the state of the positioning circle and the component state; acquiring the specimen observation image of the microscope at different times in the biological experiment operation and processing the specimen observation image, obtaining the black-and-white pixel point proportion and Gaussian blur difference value according to the processed specimen observation image, and calculating the clarity factor; acquiring the trigger state detected by the focusing operation component trigger sensor, and determining the observation step operation result according to the change process of the clarity factor corresponding to the specimen observation image at different times and the trigger state. The method can reduce the computer computing power requirement and reduce the deployment cost.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically to a microscope-based method, apparatus, and readable medium for evaluating biological experimental operations. Background Technology

[0002] Currently, the assessment of microscope operation techniques in biological experiments mainly relies on multiple cameras to achieve "unmanned supervision." Five cameras are currently in use to monitor the experimenters, and then the proctor remotely grades the results.

[0003] With the advancement and maturation of artificial intelligence, it is no longer solely reliant on the role of proctors. Instead, it provides auxiliary judgment tools. For example, facial recognition can assess the participant's demeanor, such as whether they appear anxious or confident; gesture detection can evaluate their actions, such as whether they follow a pattern or act arbitrarily; and image recognition can determine whether the final image meets the required standards. By combining these recognition methods, the proctor can determine whether the participant's results are satisfactory, whether their operational steps are up to standard, and whether the results require further review by the proctor.

[0004] First, the above methods cannot truly solve students' anxiety. As it is currently implemented, monitoring is carried out through additional cameras and remote monitoring of the experiment content by the teacher. However, a teacher or technician needs to be present on-site to ensure that "unmanned proctoring" can be achieved.

[0005] Secondly, the data throughput and computational load are enormous, including data from multiple cameras and the computing power required for inference from multiple different deep learning models. Achieving real-time performance requires more than just additional hardware; it incurs significant additional costs. The black-box reasoning of artificial intelligence is prone to probabilistic misjudgments: the student's operation may be correct, but the judgment may be flawed, or vice versa. Furthermore, preparing the on-site environment is difficult, requiring the deployment of additional observation cameras and computers. Summary of the Invention

[0006] In view of the aforementioned problems such as large data volume, high computational load, difficulty in reducing exam anxiety, high cost, and the possibility of misjudgment, the purpose of this application is to propose a microscope-based biological experiment operation evaluation method, device, and readable medium to solve the technical problems mentioned in the background section.

[0007] In a first aspect, embodiments of this application provide a microscope-based method for evaluating biological experimental operations, comprising the following steps:

[0008] S1, acquire the illumination image of the microscope during biological experiment, process the illumination image to obtain the processed illumination image, perform edge detection on the processed illumination image to obtain edge information, and determine the state of the positioning circle in the illumination image based on the edge information.

[0009] S2, acquire the component status detected by the light-operating component detection sensor, and determine the light-operating step operation result based on the status of the positioning circle and the component status;

[0010] S3, acquire specimen observation images under a microscope at different times during biological experiments, process the specimen observation images to obtain processed specimen observation images, and obtain the black and white pixel ratio and Gaussian blur difference value based on the processed specimen observation images.

[0011] S4, the sharpness factor is obtained based on the ratio of black and white pixels and the difference in Gaussian blur;

[0012] S5, acquire the trigger state detected by the trigger sensor of the focusing operation component, and determine the operation result of the observation step based on the change process of the sharpness factor corresponding to the sample observation image at different times and the trigger state.

[0013] Preferably, step S1 specifically includes:

[0014] The light image is processed to obtain a grayscale image of the light image;

[0015] Gaussian filtering is applied to the grayscale image of the illumination image to obtain the noise-reduced illumination image;

[0016] The Sobel operator is used to extract edge information from the noise-reduced illumination image to obtain edge information;

[0017] The positioning circle is determined by Hough circle transform based on edge information. If the positioning circle can be determined, the positioning circle is in the state of "existence"; otherwise, the positioning circle is in the state of "non-existence".

[0018] Preferably, the light operation component detection sensor includes an objective lens conversion detection sensor and a diaphragm conversion detection sensor. The component status includes the objective lens magnification detected by the objective lens conversion detection sensor and the diaphragm position information detected by the diaphragm conversion detection sensor.

[0019] Preferably, step S2 specifically includes:

[0020] If the positioning circle is in a certain state, then the brightness of all pixels within the positioning circle is counted, and the average brightness of all pixels is calculated.

[0021] If the average brightness is greater than the first threshold, then the operator is operating the objective lens, diaphragm, and mirror correctly.

[0022] If the average brightness is greater than the second threshold and less than the third threshold, then the operator is operating the reflector correctly, or the operator is operating the shackle or reflector incorrectly. The following methods will be used to determine whether the shackle or reflector is being operated incorrectly.

[0023] Obtain the operating magnification and compare it with the preset magnification. If the comparison matches, the objective lens operation is correct; otherwise, the objective lens operation is incorrect.

[0024] Obtain the position information. If the position information shows that the maximum aperture is aligned with the light-transmitting hole, the shutter is operating correctly. If the position information does not show that the maximum aperture is aligned with the light-transmitting hole, the shutter is operating incorrectly.

[0025] If the positioning circle is in a state of absence or the average brightness is less than the fourth threshold, then the operator has erroneously operated the reflector.

[0026] Preferably, the Gaussian blur difference value includes the difference value of the image mean and the difference value of the image standard deviation. Step S3 specifically includes:

[0027] The specimen observation image is blacked out to obtain a blacked-out specimen observation image. The number of black pixels and white pixels in the blacked-out specimen observation image is counted, and the ratio of the number of black pixels to the number of white pixels is calculated to obtain the black-and-white pixel ratio.

[0028] Gaussian filtering was applied to the specimen observation images to obtain Gaussian-filtered specimen observation images, and the mean and standard deviation of the Gaussian-filtered specimen observation images were calculated.

[0029] The specimen observation images are converted to grayscale to obtain grayscale images of the specimen observation images, and the mean and standard deviation of the grayscale images of the specimen observation images are calculated.

[0030] The difference between the mean values ​​of the Gaussian-filtered specimen observation image and the mean value of the grayscale image of the specimen observation image is obtained.

[0031] The difference in standard deviations between the standard deviations of the Gaussian-filtered specimen observation image and the standard deviations of the grayscale image of the specimen observation image is obtained.

[0032] Preferably, step S4 specifically includes: weighting the difference between the proportion of black and white pixels, the difference between the image mean and the difference between the image standard deviation to obtain the sharpness factor, as shown in the following formula:

[0033] δ=ω1α+ω2β+ω3γ;

[0034] Where δ is the sharpness factor, α is the ratio of black and white pixels, β is the difference in the image mean, γ is the difference in the image standard deviation, and ω1, ω2, and ω3 are the weight values ​​corresponding to the differences in the ratio of black and white pixels, the difference in the image mean, and the difference in the image standard deviation, respectively, and ω1+ω2+ω3=1.

[0035] Preferably, step S5 specifically includes:

[0036] If the difference between the sharpness factor and the first threshold at the first moment is within the first range, or if a trigger state is detected that rotating the coarse focus knob moves the lens barrel to the preset minimum distance, then rotate the coarse focus knob to make the lens barrel reach the lowest position.

[0037] If the difference between the sharpness factor and the second threshold at the second time after the first time is within the second range, and a trigger state of counterclockwise rotation of the coarse focus knob is detected; and the difference between the sharpness factor and the third threshold at the third time after the second time is within the third range, and a trigger state of rotation of the fine focus knob is detected, then the observation procedure is correct.

[0038] If the difference between the sharpness factor and the second threshold at the second moment is within the second range, and a trigger state of counterclockwise rotation of the coarse focus screw is detected, and the difference between the sharpness factor and the third threshold at the third moment after the second moment exceeds the third range, or no trigger state of rotation of the fine focus screw is detected, then only the coarse focus screw operation is correct, and the fine focus screw operation is incorrect.

[0039] If the difference between the sharpness factor and the second threshold at the second moment exceeds the second range, or if the trigger state of the coarse focus screw rotating counterclockwise is not detected, and the difference between the sharpness factor and the third threshold at the third moment after the second moment exceeds the third range, or if the trigger state of the fine focus screw rotating is not detected, then both the coarse and fine focus screws are operating incorrectly.

[0040] Secondly, embodiments of this application provide a microscope-based biological experiment operation evaluation device, comprising:

[0041] The illumination image processing module is configured to acquire illumination images of the microscope during biological experiments, process the illumination images to obtain processed illumination images, perform edge detection on the processed illumination images to obtain edge information, and determine the state of the positioning circle in the illumination image based on the edge information.

[0042] The light-adjusting step detection module is configured to acquire the component status detected by the light-adjusting operation component detection sensor, and determine the light-adjusting step operation result based on the status of the positioning circle and the component status.

[0043] The specimen observation image processing module is configured to acquire specimen observation images from the microscope at different times during biological experiments, process the specimen observation images to obtain processed specimen observation images, and obtain the black and white pixel ratio and Gaussian blur difference value based on the processed specimen observation images.

[0044] The sharpness factor acquisition module is configured to obtain the sharpness factor based on the ratio of black and white pixels and the Gaussian blur difference value.

[0045] The observation step detection module is configured to acquire the trigger state detected by the trigger sensor of the focusing operation component, and determine the observation step operation result based on the change process of the sharpness factor corresponding to the sample observation image at different times and the trigger state.

[0046] Thirdly, embodiments of this application provide an electronic device including one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) This invention processes the acquired illumination images and specimen observation images and analyzes whether the operation process of illumination step and observation step is correct. This can reduce data throughput, reduce the number of cameras, increase the number of sensors for setting up the microscope, thereby reducing the operator's potential tension. At the same time, it does not require the introduction of deep learning models, which also reduces the computing power requirement of the computer and reduces the misjudgment rate.

[0050] (2) The present invention improves the accuracy of operation judgment and reduces the amount of calculation by combining the detection results of the sensor integrated on the microscope.

[0051] (3) The present invention is simple to deploy on site, does not require multiple cameras, does not need to consider the data transmission stability of multiple cameras and computer host, and can effectively reduce costs. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an exemplary device architecture diagram in which an embodiment of this application can be applied;

[0054] Figure 2 This is a schematic flowchart of a microscope-based biological experiment operation evaluation method according to an embodiment of this application.

[0055] Figure 3 The image shows a light-sensitive image of a microscope-based biological experimental operation evaluation method according to an embodiment of this application.

[0056] Figure 4 The image shown is a grayscale image of the light-sensitive image of the microscope-based biological experimental operation evaluation method according to an embodiment of this application.

[0057] Figure 5 The image shows the edge detection results of the light image in the microscope-based biological experiment operation evaluation method according to an embodiment of this application.

[0058] Figure 6 This is a schematic diagram of the positioning circle in the light image of the microscope-based biological experiment operation evaluation method according to an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the brightness distribution of a light-sensitive image for an embodiment of the microscope-based biological experimental operation evaluation method of this application.

[0060] Figure 8 The images show specimen observations and black-and-white images of specimens under different examples of the microscope-based biological experimental operation evaluation method for embodiments of this application.

[0061] Figure 9 The images shown are specimen observation images, grayscale images of specimen observation images, and Gaussian filtered specimen observation images, representing different examples of the microscope-based biological experiment operation evaluation method in embodiments of this application.

[0062] Figure 10 This is a schematic diagram of a microscope-based biological experiment operation evaluation device according to an embodiment of this application;

[0063] Figure 11 This is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the present application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0065] Figure 1 An exemplary device architecture 100 is shown that can be applied to the microscope-based biological experiment operation evaluation method or microscope-based biological experiment operation evaluation device according to the embodiments of this application.

[0066] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, and 103.

[0068] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0069] Server 105 can be a server that provides various services, such as a background data processing server that processes files or data uploaded by terminal devices 101, 102, and 103. The background data processing server can process the acquired files or data and generate processing results.

[0070] It should be noted that the microscope-based biological experiment operation evaluation method provided in this application embodiment can be executed by server 105 or by terminal devices 101, 102, and 103. Correspondingly, the microscope-based biological experiment operation evaluation device can be set in server 105 or in terminal devices 101, 102, and 103.

[0071] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. If the data being processed does not need to be retrieved remotely, the above architecture may not include a network, requiring only servers or terminal devices.

[0072] Figure 2 An embodiment of this application illustrates a microscope-based biological experiment operation evaluation method, comprising the following steps:

[0073] S1. Acquire the illumination image of the microscope during biological experiment. Process the illumination image to obtain the processed illumination image. Perform edge detection on the processed illumination image to obtain edge information. Determine the state of the positioning circle in the illumination image based on the edge information.

[0074] In a specific embodiment, step S1 specifically includes:

[0075] The light image is processed to obtain a grayscale image of the light image;

[0076] Gaussian filtering is applied to the grayscale image of the illumination image to obtain the noise-reduced illumination image;

[0077] The Sobel operator is used to extract edge information from the noise-reduced illumination image to obtain edge information;

[0078] The positioning circle is determined by Hough circle transform based on edge information. If the positioning circle can be determined, the positioning circle is in the state of "existence"; otherwise, the positioning circle is in the state of "non-existence".

[0079] Specifically, in the light adjustment step, the following three points will be verified:

[0080] a) Operate the converter, i.e., objective lens operation or objective lens selection, to determine whether the objective lens has been switched to low magnification;

[0081] b) Operate the light-blocking device on the rotating disk, i.e., select the light-blocking device, and determine whether the larger aperture of the light-blocking device is aligned with the light-transmitting hole.

[0082] c) Operate the reflector on the rotating dial, i.e., select the reflector, to determine whether a bright circular field of view can be seen through the eyepiece, such as... Figure 3 As shown.

[0083] Different light-operating components correspond to the following different characteristics:

[0084] 1) Characteristics of objective lenses: Low magnification lenses have a relatively bright field of view; high magnification lenses have a relatively dark field of view;

[0085] 2) Characteristics of light-blocking devices: Larger ones allow more light in and produce a brighter field of view, while smaller ones allow less light in and produce a darker field of view;

[0086] 3) Characteristics of the reflector: The function of the reflector is to reflect the light from the light source onto the condenser, and then illuminate the specimen through the light aperture.

[0087] Therefore, if the objective lens or diaphragm is not selected properly, a dark image will be obtained. If the mirror is used incorrectly, no content will be captured, and the brightness will be extremely low. Therefore, the ability to find the positioning circle can be used as an evaluation indicator of the mirror operation result.

[0088] First of all, Figure 3 The light image is processed to grayscale to obtain, as shown below. Figure 4 The grayscale image shown is used for noise reduction processing based on the grayscale image, using the following two-dimensional Gaussian filtering formula:

[0089]

[0090] After denoising, a denoised illumination image is obtained. Edge detection is then performed using the Canny algorithm based on this denoised illumination image. Specifically, the Sobel operator is first used to extract edge information in the horizontal and vertical directions of the denoised illumination image, resulting in the image shown in Figure 5.

[0091] The formula for horizontal calculation is as follows:

[0092] d x =[f(i-1,j-1)+2f(i-1,j)+f(i-1,j+1)]-[f(i+1,j)+f(i+1,j+1)];

[0093] The formula for vertical calculation is as follows:

[0094] d y =[f(i-1,j+1)+2f(i,j+1)+f(i+1,j+1)]-[f(i-1,j-1)+2f(i,j-1)+f(i+1,j-1)];

[0095] Where (i, j) correspond to the coordinates. The horizontal calculation formula uses a 3*3 window to perform a horizontal search on the denoised lighting image; the vertical calculation formula uses a 3*3 window to perform a vertical search on the denoised lighting image. Based on the horizontal image (d x) and longitudinal graph (d y Extract the edge gradient (G) and pixel orientation (θ):

[0096]

[0097]

[0098] Specifically, the Hough Transform can be used to find the positioning circle in the illumination image. The Hough Transform converts a circle in two-dimensional image space into a point in three-dimensional parameter space determined by the radius and center coordinates of that circle. Therefore, the circle defined by any three points on the circumference should correspond to a single point in three-dimensional parameter space after the Hough Transform. This process is similar to an election voting process, where any three points on the circumference are voters, and the circle defined by these three points is a candidate (hereinafter referred to as the candidate circle). The process iterates through all points on the circumference, voting on the candidate circles defined by any three points. After the iteration, the circle defined by the point with the highest number of votes (theoretically, the circle defined by any three points on the circumference corresponds to the same point in three-dimensional parameter space after the Hough Transform) is the circle defined by the vast majority of points on that circumference (hereinafter referred to as the elected circle). In other words, the vast majority of points are on the circumference of the elected circle, and this circle is thus identified as the positioning circle. Figure 6 As shown.

[0099] S2, acquire the component status detected by the light-operating component detection sensor, and determine the light-operating step operation result based on the status of the positioning circle and the component status.

[0100] In a specific embodiment, the light operation component detection sensor includes an objective lens conversion detection sensor and a shackle conversion detection sensor. The component status includes the objective lens magnification detected by the objective lens conversion detection sensor and the shackle position information detected by the shackle conversion detection sensor.

[0101] In a specific embodiment, step S2 specifically includes:

[0102] If the positioning circle is in a certain state, then the brightness of all pixels within the positioning circle is counted, and the average brightness of all pixels is calculated.

[0103] If the average brightness is greater than the first threshold, then the operator is operating the objective lens, diaphragm, and mirror correctly.

[0104] If the average brightness is greater than the second threshold and less than the third threshold, then the operator is operating the reflector correctly, or the operator is operating the shackle or reflector incorrectly. The following methods will be used to determine whether the shackle or reflector is being operated incorrectly.

[0105] Obtain the operating magnification and compare it with the preset magnification. If the comparison matches, the objective lens operation is correct; otherwise, the objective lens operation is incorrect.

[0106] Obtain the position information. If the position information shows that the maximum aperture is aligned with the light-transmitting hole, the shutter is operating correctly. If the position information does not show that the maximum aperture is aligned with the light-transmitting hole, the shutter is operating incorrectly.

[0107] If the positioning circle is in a state of absence or the average brightness is less than the fourth threshold, then the operator has erroneously operated the reflector.

[0108] Specifically, when the positioning circle is in a certain state, the positioning circle can be determined using the Hough circle transform. Then, the brightness of all pixels within that positioning circle is counted. For details, refer to... Figure 7 By reading the illumination image, the brightness of pixels is counted and a list is formed. A set of pixels within a designated circle is defined as those with a brightness greater than a preset threshold. Specifically, pixels with a brightness greater than 20 are considered within the designated circle. Only the brightness of all pixels within the designated circle is counted, and then the average brightness is calculated. In one embodiment, the first threshold is set to 200, the second threshold to 100, the third threshold to 199, and the fourth threshold to 50. If the average brightness is greater than 200, the operator is using the objective lens, diaphragm, and mirror correctly. If the average brightness is between 100 and 199, the operator is using the mirror correctly but the objective lens and diaphragm incorrectly. In this case, the data detected by the light operation component detection sensor can be used to further determine whether the incorrect objective lens or diaphragm is being used incorrectly. If the magnification detected by the objective lens conversion detection sensor matches the preset magnification, the objective lens operation is correct; if the magnification does not match the preset magnification, the objective lens operation is incorrect. If the position information detected by the diaphragm conversion detection sensor is the maximum aperture corresponding to the light-transmitting hole of the diaphragm, the diaphragm operation is correct; if the position information detected by the diaphragm conversion detection sensor is not the maximum aperture corresponding to the light-transmitting hole, the diaphragm operation is incorrect. If the average brightness is less than 50, the operator is using the mirror incorrectly. In this case, regardless of the operation of the objective lens and diaphragm, the positioning circle cannot be obtained, and the brightness cannot be increased. The results of the operator's light-adjusting steps are determined based on the above methods, and this is used as the basis for evaluating microscope operation in biological experiments.

[0109] S3: Obtain specimen observation images from the microscope at different times during biological experiments; process the specimen observation images to obtain processed specimen observation images; and obtain the black and white pixel ratio and Gaussian blur difference value based on the processed specimen observation images.

[0110] In a specific embodiment, the Gaussian blur difference value includes the difference value of the image mean and the difference value of the image standard deviation. Step S3 specifically includes:

[0111] The specimen observation image is blacked out to obtain a blacked-out specimen observation image. The number of black pixels and white pixels in the blacked-out specimen observation image is counted, and the ratio of the number of black pixels to the number of white pixels is calculated to obtain the black-and-white pixel ratio.

[0112] Gaussian filtering was applied to the specimen observation images to obtain Gaussian-filtered specimen observation images, and the mean and standard deviation of the Gaussian-filtered specimen observation images were calculated.

[0113] The specimen observation images are converted to grayscale to obtain grayscale images of the specimen observation images, and the mean and standard deviation of the grayscale images of the specimen observation images are calculated.

[0114] The difference between the mean values ​​of the Gaussian-filtered specimen observation image and the mean value of the grayscale image of the specimen observation image is obtained.

[0115] The difference in standard deviations between the standard deviations of the Gaussian-filtered specimen observation image and the standard deviations of the grayscale image of the specimen observation image is obtained.

[0116] In a specific embodiment, the observation steps include specimen placement and inspection. Specifically, the slide specimen to be observed is placed on the stage and clamped with the slide clips, with the specimen aligned with the center of the light aperture. The coarse adjustment knob is rotated to slowly lower the microscope tube to its lowest position, and then the coarse adjustment knob is rotated counterclockwise until the image is clear. The fine adjustment knob is then slightly rotated to make the image even clearer. This step is to assess whether the operator is using the coarse and fine adjustment knobs correctly. The coarse adjustment knob has a larger change in focal plane, meaning a larger change in sharpness. The fine adjustment knob has a smaller change in focal plane, meaning a smaller change in sharpness. Based on this characteristic, the images captured by the digital camera are analyzed to generate time-correlated log files. Therefore, the sharpness of the specimen observation image needs to be evaluated, specifically by calculating the black-and-white pixel ratio and Gaussian blur difference value of the specimen observation image.

[0117] Specifically, the calculation process for the ratio of black and white pixels in the specimen observation image is as follows:

[0118] (1) Perform grayscale processing on the specimen observation images to obtain grayscale images;

[0119] (2) Thresholding is then applied to the grayscale image to convert it to black and white, resulting in bipolar data of 0 and 255. The original image and the black and white image are shown in the reference. Figure 8 ;

[0120] (3) Count the number of black pixels;

[0121] (4) Count the number of white pixels;

[0122] (5) Calculate the ratio of black pixels to white pixels to obtain the black-and-white pixel ratio.

[0123] Specifically, the calculation process for the Gaussian blur difference value of the specimen observation image is as follows:

[0124] Noise information is extracted from the specimen observation images using the following two-dimensional Gaussian filtering formula to obtain the Gaussian-filtered specimen observation images:

[0125]

[0126] Using the open-source (OPENCV) image reading method, the mode of the specimen observation image was modified to:

IMREAD_GRAYSCALE

[0127] The formula for calculating the annotation difference is as follows:

[0128]

[0129] The method for calculating the mean is as follows:

[0130]

[0131] Finally, the mean of the Gaussian-filtered specimen observation image is subtracted from the mean of the grayscale image to obtain the difference in image means; the standard deviation of the Gaussian-filtered specimen observation image is subtracted from the standard deviation of the grayscale image to obtain the difference in image standard deviation. The Gaussian blur difference value includes both the difference in image means and the difference in image standard deviation.

[0132] S4, the sharpness factor is obtained based on the ratio of black and white pixels and the difference in Gaussian blur.

[0133] In a specific embodiment, step S4 specifically includes: weighting the difference between the proportion of black and white pixels, the difference between the image mean and the difference between the image standard deviation to obtain a sharpness factor, as shown in the following formula:

[0134] δ=ω1α+ω2β+ω3γ;

[0135] Where δ is the sharpness factor, α is the ratio of black and white pixels, β is the difference in the image mean, γ is the difference in the image standard deviation, and ω1, ω2, and ω3 are the weight values ​​corresponding to the differences in the ratio of black and white pixels, the difference in the image mean, and the difference in the image standard deviation, respectively, and ω1+ω2+ω3=1.

[0136] Specifically, the sharpness factor can be calculated based on the difference in the ratio of black and white pixels, the difference in the image mean, and the difference in the image standard deviation. The sharpness factor is used to evaluate the sharpness of the specimen observation image. The higher the ratio of black and white pixels, the sharper the specimen observation image; the larger the difference in Gaussian blur, the sharper the specimen observation image.

[0137] S5, acquire the trigger state detected by the trigger sensor of the focusing operation component, and determine the operation result of the observation step based on the change process of the sharpness factor corresponding to the sample observation image at different times and the trigger state.

[0138] In a specific embodiment, step S5 specifically includes:

[0139] If the difference between the sharpness factor and the first threshold at the first moment is within the first range, or if a trigger state is detected that rotating the coarse focus knob moves the lens barrel to the preset minimum distance, then rotate the coarse focus knob to make the lens barrel reach the lowest position.

[0140] If the difference between the sharpness factor and the second threshold at the second time after the first time is within the second range, and a trigger state of counterclockwise rotation of the coarse focus knob is detected; and the difference between the sharpness factor and the third threshold at the third time after the second time is within the third range, and a trigger state of rotation of the fine focus knob is detected, then the observation procedure is correct.

[0141] If the difference between the sharpness factor and the second threshold at the second moment is within the second range, and a trigger state of counterclockwise rotation of the coarse focus screw is detected, and the difference between the sharpness factor and the third threshold at the third moment after the second moment exceeds the third range, or no trigger state of rotation of the fine focus screw is detected, then only the coarse focus screw operation is correct, and the fine focus screw operation is incorrect.

[0142] If the difference between the sharpness factor and the second threshold at the second moment exceeds the second range, or if the trigger state of the coarse focus screw rotating counterclockwise is not detected, and the difference between the sharpness factor and the third threshold at the third moment after the second moment exceeds the third range, or if the trigger state of the fine focus screw rotating is not detected, then both the coarse and fine focus screws are operating incorrectly.

[0143] Specifically, during the operation, the differences in the ratio of black and white pixels, the image mean, and the image standard deviation at different times are recorded, and the sharpness factor at different times is calculated to form a data list. The correctness of the focusing operation is determined based on the changes in the sharpness factor within that time period. The data list is as follows:

[0144]

[0145] If the sharpness factor extracted from a set of data at the first moment in the data list matches the preset first threshold for rotating the coarse adjustment knob to bring the lens barrel to its lowest position, i.e., the difference between the sharpness factor at the first moment and the first threshold is within a first range (preferably within ±5%), then it is confirmed that rotating the coarse adjustment knob has moved the lens barrel to its lowest position. After the first moment, a search is performed to check if the sharpness factor exhibits either of the following two changes. If so, it indicates that the operator correctly completed the focusing operation, i.e., using the coarse adjustment knob first and then the fine adjustment knob.

[0146] First, at the second moment, the sharpness factor changes significantly, that is, the difference between the sharpness factor and the second threshold is within the second range. Preferably, the second range is within ±5 to 7%, that is, the operator used the coarse focus knob.

[0147] Secondly, at the third moment, the sharpness factor changes slightly, that is, the difference between the sharpness factor and the third threshold exceeds the third range. Preferably, the third range is within ±2 to 3%, that is, the operator uses a fine focus knob.

[0148] Since the experimental slices are fixed, the sharpness factor corresponds to specific values ​​at specific positions of the coarse and fine adjustment knobs. The first, second, and third thresholds can be pre-determined, and due to the comprehensive score, four decimal places will be used. Specifically: a higher proportion of black and white pixels results in a sharper image; a higher Gaussian blur difference also results in a sharper image. When using the coarse adjustment knob, the sharpness factor varies more significantly; when using the fine adjustment knob, the variation is smaller. Both methods attempt to find the sharpest area, and all sharpness factors will gradually increase.

[0149] Alternatively, the data collected by the sensor can be analyzed using the focusing mechanism on the microscope to determine if the coarse and fine adjustment knobs were used correctly. Specifically, a sensor is added to the microscope to detect whether the microscope tube has reached the preset minimum distance. If a signal is detected at the minimum distance, it indicates that the coarse adjustment knob has been rotated to move the microscope tube to its lowest position. Then, it is checked whether the coarse adjustment knob is rotated counterclockwise and the fine adjustment knob is rotated to check if the distance between the objective lens and the slide specimen is appropriate. Finally, the imaging results are checked. This process is not the focus of this application and will not be described in detail here.

[0150] Further reference Figure 10 As an implementation of the methods shown in the above figures, this application provides an embodiment of a microscope-based biological experiment operation evaluation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0151] This application provides a microscope-based biological experiment operation evaluation device, including:

[0152] The illumination image processing module 1 is configured to acquire illumination images of the microscope during biological experiments, process the illumination images to obtain processed illumination images, perform edge detection on the processed illumination images to obtain edge information, and determine the state of the positioning circle in the illumination image based on the edge information.

[0153] The light-alignment step detection module 2 is configured to acquire the component status detected by the light-alignment operation component detection sensor, and determine the light-alignment step operation result based on the status of the positioning circle and the component status.

[0154] The specimen observation image processing module 3 is configured to acquire specimen observation images from the microscope at different times during biological experiments, process the specimen observation images to obtain processed specimen observation images, and obtain the black and white pixel ratio and Gaussian blur difference value based on the processed specimen observation images.

[0155] The sharpness factor acquisition module 4 is configured to obtain the sharpness factor based on the ratio of black and white pixels and the Gaussian blur difference value.

[0156] The observation step detection module 5 is configured to acquire the trigger state detected by the trigger sensor of the focusing operation component, and determine the observation step operation result based on the change process of the sharpness factor corresponding to the sample observation image at different times and the trigger state.

[0157] The following is for reference. Figure 11 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., Figure 1A schematic diagram of the structure of a computer device 1100 (shown as a server or terminal device). Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0158] like Figure 11 As shown, the computer device 1100 includes a central processing unit (CPU) 1101 and a graphics processing unit (GPU) 1102, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1103 or programs loaded from storage portion 1109 into random access memory (RAM) 1104. The RAM 1104 also stores various programs and data required for the operation of the device 1100. The CPU 1101, GPU 1102, ROM 1103, and RAM 1104 are interconnected via a bus 1105. An input / output (I / O) interface 1106 is also connected to the bus 1105.

[0159] The following components are connected to I / O interface 1106: an input section 1107 including a keyboard, mouse, etc.; an output section 1108 including an LCD, speakers, etc.; a storage section 1109 including a hard disk, etc.; and a communication section 1110 including a network interface card, such as a LAN card, modem, etc. The communication section 1110 performs communication processing via a network such as the Internet. A drive 1111 may also be connected to I / O interface 1106 as needed. Removable media 1112, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1111 as needed so that computer programs read from them can be installed into storage section 1109 as needed.

[0160] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1110, and / or installed from removable medium 1112. When the computer program is executed by central processing unit (CPU) 1101 and graphics processing unit (GPU) 1102, the functions defined in the methods of this application are performed.

[0161] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0162] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based means to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0165] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a microscope illumination image during a biological experiment; process the illumination image to obtain a processed illumination image; perform edge detection on the processed illumination image to obtain edge information; determine the state of the positioning circle in the illumination image based on the edge information; acquire the state of the component detected by the illumination operation component detection sensor; determine the illumination step operation result based on the state of the positioning circle and the component state; acquire specimen observation images of the microscope at different times during the biological experiment; process the specimen observation images to obtain a processed specimen observation image; obtain the black-and-white pixel ratio and Gaussian blur difference value based on the processed specimen observation image; obtain a sharpness factor based on the black-and-white pixel ratio and Gaussian blur difference value; acquire the trigger state detected by the focusing operation component trigger sensor; and determine the observation step operation result based on the change process of the sharpness factor corresponding to the sample observation images at different times and the trigger state.

[0166] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A microscope-based method for evaluating biological experimental procedures, characterized in that, Includes the following steps: S1, acquire the illumination image of the microscope during biological experiment, process the illumination image to obtain a processed illumination image, perform edge detection on the processed illumination image to obtain edge information, and determine the state of the positioning circle in the illumination image based on the edge information; S2, acquire the component state detected by the light operation component detection sensor, and determine the light operation result based on the state of the positioning circle and the component state; S3, acquire microscope images of specimens at different times during the biological experiment, process the specimen images to obtain processed specimen images, and obtain the black-and-white pixel ratio and Gaussian blur difference value based on the processed specimen images. The Gaussian blur difference value includes the difference in image mean and the difference in image standard deviation. Step S3 specifically includes: The specimen observation image is blacked out to obtain a blacked-out specimen observation image. The number of black pixels and white pixels in the blacked-out specimen observation image is counted, and the ratio of the number of black pixels to the number of white pixels is calculated to obtain the black-out pixel ratio. The specimen observation image is subjected to Gaussian filtering to obtain a Gaussian-filtered specimen observation image, and the mean and standard deviation of the Gaussian-filtered specimen observation image are calculated. The specimen observation image is converted to grayscale to obtain a grayscale image of the specimen observation image, and the mean and standard deviation of the grayscale image of the specimen observation image are calculated. The difference value of the image means is obtained by subtracting the mean of the Gaussian filtered specimen observation image from the mean of the grayscale image of the specimen observation image. The difference value of the standard deviation of the Gaussian filtered specimen observation image is obtained by subtracting the standard deviation of the grayscale image of the specimen observation image; S4, the sharpness factor is obtained based on the black-and-white pixel ratio and the Gaussian blur difference value. Specifically, this includes: weighting the difference between the black-and-white pixel ratio, the difference between the image mean and the difference between the image standard deviation to obtain the sharpness factor, as shown in the following formula: ； in, For the clarity factor, The ratio of the black and white pixels. The difference value of the mean of the image. The difference value of the standard deviation of the image. These are the weight values ​​corresponding to the differences in the proportion of black and white pixels, the difference in the image mean, and the difference in the image standard deviation, respectively. ; S5, acquire the trigger state detected by the trigger sensor of the focusing operation component, and determine the operation result of the observation step based on the change process of the sharpness factor corresponding to the sample observation image at different times and the trigger state.

2. The microscope-based biological experiment operation evaluation method according to claim 1, characterized in that, Step S1 specifically includes: The light image is processed to obtain a grayscale image of the light image; Gaussian filtering is applied to the grayscale image of the illumination image to obtain a noise-reduced illumination image; The edge information is obtained by extracting edge information from the noise-reduced illumination image using the Sobel operator; The positioning circle is determined by Hough circle transform based on the edge information. If the positioning circle can be determined, the positioning circle is in a state of existence; otherwise, the positioning circle is in a state of non-existence.

3. The microscope-based biological experiment operation evaluation method according to claim 1, characterized in that, The light-operating component detection sensor includes an objective lens conversion detection sensor and a diaphragm conversion detection sensor. The component status includes the objective lens magnification detected by the objective lens conversion detection sensor and the diaphragm position information detected by the diaphragm conversion detection sensor.

4. The microscope-based biological experiment operation evaluation method according to claim 3, characterized in that, Step S2 specifically includes: If the positioning circle is in a certain state, then the brightness of all pixels within the positioning circle is counted, and the average brightness of all pixels is calculated. If the average brightness is greater than the first threshold, then the operator is operating the objective lens, diaphragm, and reflector correctly. If the average brightness is greater than the second threshold and less than the third threshold, then the operator is operating the reflector correctly or incorrectly, and the operator is determining whether the reflector or the shade is being operated incorrectly by means of the following methods; Obtain the operation magnification and compare it with the preset magnification. If the comparison matches, the objective lens operation is correct; if the comparison does not match, the objective lens operation is incorrect. If the position information indicates that the maximum aperture is aligned with the light-transmitting hole, the operation of the light-blocking device is correct; otherwise, the operation of the light-blocking device is incorrect. If the positioning circle is in a state of absence or the average brightness is less than the fourth threshold, then the operator is operating the reflector incorrectly.

5. The microscope-based biological experiment operation evaluation method according to claim 1, characterized in that, Step S5 specifically includes: If the difference between the sharpness factor and the first threshold at the first moment is within the first range, or if a trigger state is detected in which rotating the coarse focus knob moves the lens barrel to a preset minimum distance, then rotating the coarse focus knob will move the lens barrel to the lowest position. If the difference between the sharpness factor and the second threshold at the second time after the first time is within the second range, and a trigger state of counterclockwise rotation of the coarse focus knob is detected; and the difference between the sharpness factor and the third threshold at the third time after the second time is within the third range, and a trigger state of rotation of the fine focus knob is detected, then the observation step is correct. If the difference between the sharpness factor and the second threshold at the second moment is within the second range, and a trigger state of counterclockwise rotation of the coarse focus screw is detected, and the difference between the sharpness factor and the third threshold at the third moment after the second moment exceeds the third range, or no trigger state of rotation of the fine focus screw is detected, then only the coarse focus screw is operated correctly, and the fine focus screw is operated incorrectly. If the difference between the sharpness factor and the second threshold at the second moment exceeds the second range, or if no trigger state of counterclockwise rotation of the coarse focus screw is detected, and the difference between the sharpness factor and the third threshold at the third moment after the second moment exceeds the third range, or if no trigger state of rotation of the fine focus screw is detected, then both the coarse and fine focus screws are operating incorrectly.

6. A microscope-based biological experiment operation evaluation device, characterized in that, include: The illumination image processing module is configured to acquire illumination images of a microscope during biological experiments, process the illumination images to obtain processed illumination images, perform edge detection on the processed illumination images to obtain edge information, and determine the state of the positioning circle in the illumination images based on the edge information. The light-adjusting step detection module is configured to acquire the component state detected by the light-adjusting operation component detection sensor, and determine the light-adjusting step operation result based on the state of the positioning circle and the component state; A specimen observation image processing module is configured to acquire microscope specimen observation images at different times during biological experiments, process the specimen observation images to obtain processed specimen observation images, and obtain the black-and-white pixel ratio and Gaussian blur difference value based on the processed specimen observation images. The Gaussian blur difference value includes the difference in image mean and the difference in image standard deviation. The specimen observation image processing module specifically includes: The specimen observation image is blacked out to obtain a blacked-out specimen observation image. The number of black pixels and white pixels in the blacked-out specimen observation image is counted, and the ratio of the number of black pixels to the number of white pixels is calculated to obtain the black-out pixel ratio. The specimen observation image is subjected to Gaussian filtering to obtain a Gaussian-filtered specimen observation image, and the mean and standard deviation of the Gaussian-filtered specimen observation image are calculated. The specimen observation image is converted to grayscale to obtain a grayscale image of the specimen observation image, and the mean and standard deviation of the grayscale image of the specimen observation image are calculated. The difference value of the image means is obtained by subtracting the mean of the Gaussian filtered specimen observation image from the mean of the grayscale image of the specimen observation image. The difference value of the standard deviation of the Gaussian filtered specimen observation image is obtained by subtracting the standard deviation of the grayscale image of the specimen observation image; The sharpness factor acquisition module is configured to obtain a sharpness factor based on the black-and-white pixel ratio and the Gaussian blur difference value. Specifically, it includes: weighting the difference between the black-and-white pixel ratio, the difference between the image mean and the difference between the image standard deviation to obtain the sharpness factor, as shown in the following formula: ； in, For the clarity factor, The ratio of the black and white pixels. The difference value of the mean of the image. The difference value of the standard deviation of the image. These are the weight values ​​corresponding to the differences in the proportion of black and white pixels, the difference in the image mean, and the difference in the image standard deviation, respectively. ; The observation step detection module is configured to acquire the trigger state detected by the focusing operation component trigger sensor, and determine the observation step operation result based on the change process of the sharpness factor corresponding to the sample observation image at different times and the trigger state.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images