A method and system for recognizing experimental operations of a rubber-tipped dropper

Abstract

The application provides a recognition method and system for experimental operation of a rubber head dropper, and relates to the technical field of experimental scoring, which comprises the following steps: collecting continuous frame images of experimental operation, recognizing the dropper and the container, extracting the region of interest, and recognizing the first distance from the liquid surface in the dropper to the dropper tip; adopting a YOLO target detection model to detect the positional relationship between the dropper tip and the container mouth, and when the relationship meets the preset condition, recognizing liquid pixels and distinguishing continuous liquid flow from isolated liquid drops; fitting a dropping trajectory line to determine whether the liquid falls into the container; counting the isolated liquid drops and calculating the first dropping amount; after the dropper is away from the container by a preset distance, calculating the second distance from the liquid surface to the dropper tip; calculating the second dropping amount according to the first distance and the second distance; and performing weighted fusion on the first dropping amount and the second dropping amount to obtain a comprehensive dropping amount. The recognition method has good robustness for transparent liquid and complex background, and can realize automatic scoring of middle school chemistry experiment operation tests.

Description

Technical Field

[0001] This invention belongs to the field of experimental scoring technology, specifically relating to a method and system for identifying dropper experimental operations. Background Technology

[0002] In chemistry lab exams, the correct use of a dropper is a crucial component of the basic skills assessment. Grading criteria typically include: whether the dropper is applied vertically, whether the droplet falls accurately into the container, and the accuracy of the amount added. Traditional grading methods rely on invigilators' real-time observation or post-exam video review, which suffers from issues such as subjectivity, inconsistent grading standards, and high labor costs.

[0003] Most existing methods can only detect "whether there is liquid movement," but cannot accurately determine whether the movement is a continuous liquid flow or an isolated droplet, making it impossible to correctly assess whether the drop rate of the examinee is appropriate. Furthermore, in the specific scenario of a chemistry experiment exam, their ability to detect transparent liquids is insufficient.

[0004] Therefore, there is an urgent need for a purely visual droplet detection method that requires no additional hardware, is effective for transparent liquids, and can distinguish between droplets and liquid streams, in order to meet the actual needs of an automatic scoring system for chemistry experiment exams. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for identifying dropper experimental operations, thereby improving the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0006] Firstly, this application provides a method for identifying dropper experimental operations, including...

[0007] Acquire continuous frame images of the experimental operation;

[0008] The dropper and container are identified using an image recognition algorithm, the region of interest is extracted, the liquid level inside the dropper is identified, and the first distance from the liquid level to the tip of the dropper is calculated.

[0009] A pre-trained YOLO object detection model was used to detect the positional relationship between the dropper tip and the container opening;

[0010] When the positional relationship meets the preset conditions, differential operation is performed on adjacent frames to extract the pixel change features of the region of interest. Based on the pixel change features, liquid pixels are identified and continuous liquid flow and isolated droplets are distinguished.

[0011] The dripping trajectory is fitted based on the movement path of the liquid pixels in consecutive frames, and the liquid is determined to fall into the container based on the positional relationship between the dripping trajectory and the container.

[0012] The number of isolated droplets identified is counted, and the amount of the first droplet added is calculated based on the preset single droplet volume;

[0013] After the dropper is moved away from the container by a preset distance, calculate the second distance from the liquid surface to the tip of the dropper;

[0014] Calculate the second drop volume based on the first and second distances;

[0015] The first and second drops are weighted and combined to obtain the overall drop volume.

[0016] A system for recognizing dropper experimental operations includes:

[0017] The first module is used to acquire continuous frame images of experimental operations;

[0018] The second module is used to identify the dropper and container through image recognition algorithms, extract the region of interest, identify the liquid surface inside the dropper, and calculate the first distance from the liquid surface to the tip of the dropper.

[0019] The third module is used to detect the positional relationship between the dropper tip and the container opening using a pre-trained YOLO object detection model;

[0020] The fourth module is used to perform differential operations on adjacent frames when the positional relationship meets the preset conditions, extract the pixel change features of the region of interest, identify liquid pixels based on the pixel change features, and distinguish between continuous liquid flow and isolated droplets.

[0021] The fifth module is used to fit a dripping trajectory line based on the movement path of the liquid pixels in consecutive frames, and to determine whether the liquid has fallen into the container based on the positional relationship between the dripping trajectory line and the container.

[0022] The sixth module is used to count the isolated droplets identified and calculate the amount of the first drop to be added based on the preset single drop volume;

[0023] The seventh module is used to calculate the second distance from the liquid surface to the tip of the dropper after the dropper has moved away from the container by a preset distance;

[0024] The eighth module is used to calculate the second drop volume based on the first and second distances;

[0025] The ninth module is used to weight and fuse the first and second drop amounts to obtain the comprehensive drop amount.

[0026] The beneficial effects of this invention are as follows:

[0027] This method achieves pure visual detection entirely based on ordinary cameras, requiring no additional hardware. It integrates multiple techniques such as inter-frame difference, entropy difference pulse, and YOLO spatial localization to enhance the detection effect on transparent liquids. By utilizing the entropy difference pulse feature, it can accurately distinguish between isolated droplets and continuous liquid flow. Through cross-validation of two independent information sources, droplet counting and liquid surface change, it can accurately assess the amount of liquid added. This provides a practical and feasible technical solution for large-scale, low-cost, and highly reliable automated scoring of middle school chemistry experiment operation examinations.

[0028] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the identification method for dropper experimental operations in an embodiment of this application.

[0031] Figure 2 This is a structural diagram of the identification device for the dropper experimental operation in an embodiment of this application.

[0032] Symbol explanation: 800 - Identification device for dropper experimental operation; 801 - Processor; 802 - Memory; 803 - Multimedia component; 804 - I / O interface; 805 - Communication component. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0034] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0035] High school chemistry experiment operation exams typically include three core sections: laboratory preparation and properties of gases, preparation and identification of substances, and basic experimental operations. Among these, the use of droppers is a major operational skill being assessed. Operational requirements include that the dropper must not be inverted, the dropper must be suspended directly above the container opening during addition without touching the inner wall of the container, one drop must be squeezed at a time, and the droplet must be completely separated; the amount added must meet the experimental requirements. However, transparent reagents commonly used in chemical experiments (such as water, dilute hydrochloric acid, and alcohol) are difficult to accurately identify in conventional visible light images. To address this problem, this application aims to provide a purely visual droplet detection method that is effective for transparent liquids and can distinguish between droplets and liquid streams, in order to meet the actual needs of an automatic scoring system for chemistry experiment exams.

[0036] Example 1:

[0037] See Figure 1 This embodiment provides a method for identifying dropper experimental operations, including steps S100, S200, S300, S400, S500, S600, S700, S800 and S900.

[0038] S100: Acquire continuous frame images of experimental operations;

[0039] A camera at a fixed position is used to continuously capture experimental operation videos, and each frame of the image is processed for noise reduction and brightness normalization.

[0040] S200: Identify the dropper and container using an image recognition algorithm, extract the region of interest, identify the liquid surface inside the dropper, and calculate the first distance from the liquid surface to the tip of the dropper.

[0041] The first distance needs to be calculated before the drop is added and when the liquid surface is relatively stable. Therefore, it is necessary to first calculate whether the direction of the dropper's movement is towards the container through consecutive frames. If so, the distance from the dropper to the container and the perpendicularity of the dropper are calculated. If both the distance and the perpendicularity meet the preset conditions, the first distance from the liquid surface to the tip of the dropper is calculated based on the first frame image that meets the conditions.

[0042] S300: A pre-trained YOLO object detection model is used to detect the positional relationship between the dropper tip and the container opening;

[0043] Using images with labeled dropper tips and test tube / measuring cylinder openings, train the YOLO object detection model. The YOLO object detection model can locate key points on the dropper tip and container openings. During training, use operation videos of different candidates under different lighting conditions as samples.

[0044] S400: When the positional relationship meets the preset conditions, perform differential operation on adjacent frames, extract the pixel change features of the region of interest, identify liquid pixels based on the pixel change features, and distinguish between continuous liquid flow and isolated droplets.

[0045] The preset condition is that the tip of the dropper is above the container opening and the distance between the dropper and the container opening is within a preset range;

[0046] S410. Convert the image frame into a grayscale image, calculate the absolute value of the grayscale difference between adjacent frames pixel by pixel, and obtain the difference image.

[0047] The images of frame t and frame (t-1) are obtained from the video stream and converted to grayscale to reduce computation. The absolute value of the grayscale difference between corresponding pixels in the two frames is calculated to obtain the difference image D.

[0048] S420. Convert the difference image into a binary image according to the preset grayscale threshold, and perform connected component analysis on the foreground pixels in the binary image to obtain at least one connected component.

[0049] The grayscale threshold is set according to the ambient lighting conditions, for example, 30. If the grayscale difference between pixels in the differential image is greater than the grayscale threshold, they are classified as foreground pixels; otherwise, they are classified as background pixels.

[0050] Optionally, morphological filtering can be performed on the binary image in this step to remove isolated noise points and fill small holes inside liquid regions through opening and closing operations.

[0051] S430. Calculate the area, aspect ratio, shape factor, motion direction, and velocity for each connected component to obtain pixel change features;

[0052] The area is obtained by counting the number of pixels in the connected components. The ratio of the border width to the height and the ratio of the area to the square of the perimeter (roundness) are calculated. The direction of motion and speed are calculated by tracking the centroid position between adjacent frames.

[0053] By tracking the same connected component across multiple consecutive frames (e.g., 5-10 frames), recording its area change rate, velocity fluctuations, and the number of frames in which it appears and disappears, pixel change characteristics can be obtained.

[0054] S440. Based on the pixel change characteristics, the connected components are filtered to obtain liquid pixels;

[0055] First, exclude connected regions with unreasonable area, border width-to-height ratio, and roundness, i.e., regions that clearly do not conform to the shape of liquid flow / droplet;

[0056] Then determine whether the direction of motion is close to the direction of gravity (mainly downward) and whether the speed is reasonable; only retain reasonable areas to obtain liquid pixels.

[0057] Optionally, the selected area can be further confirmed by combining color or reflective features (such as red or highlights for common liquids). If the RGB values ​​in the target area match the preset liquid color model, it is determined to be a liquid pixel, and the position mask image of the liquid pixel is output.

[0058] S450. Based on the spatial morphology and temporal continuity of the connected domains of liquid pixels, distinguish between continuous liquid flow and isolated droplets.

[0059] For each selected independent region, analyze its spatial morphology. If the aspect ratio of the connected region is significantly greater than 1, it indicates that it is elongated and strip-shaped, and is more likely to be a liquid flow. If the aspect ratio is close to 1, such as 0.7~1.3, it indicates that it is approximately circular or elliptical, and is more likely to be a droplet.

[0060] Analyzing temporal continuity, if a connected component in the previous frame overlaps with a connected component in the current frame, or the inter-frame distance is less than a threshold, and the shape remains slender, it indicates continuous liquid flow; if the regions of the same droplets do not overlap between frames, but the centroid position and motion trend are consistent, it indicates isolated droplets.

[0061] When a droplet (even a transparent one) enters the region of interest, the refraction, reflection, and introduction of its edge contour alter the statistical characteristics of the local grayscale distribution, causing a change in entropy. This change exhibits different temporal characteristics from slow disturbances such as background noise and gradual changes in illumination—the entropy value rises rapidly and then falls rapidly as the droplet passes through, forming an entropy difference pulse. As another implementation method, liquid flow and droplets can be distinguished based on this entropy difference pulse, as follows:

[0062] Cut the area of ​​interest between the dropper tip and the container opening according to the preset width;

[0063] For each frame, calculate the entropy within a preset sliding window in the region of interest;

[0064] The region of interest is divided into multiple small windows (e.g., 16×16 pixels). The entropy of each window is calculated independently to obtain an entropy map. The maximum or average value of the entropy map is taken as the representative entropy of the frame.

[0065] ;

[0066] in, The entropy within a single window, The probability of a pixel with a grayscale value of g within the window;

[0067] Calculate the entropy difference between adjacent frames, detect entropy difference pulses, extract pulse features, and distinguish between continuous liquid flow and isolated droplets based on pulse features;

[0068] Calculate the entropy difference between adjacent frames frame by frame. ,in Represents the entropy of frame t;

[0069] Set adaptive threshold:

[0070] ;

[0071] in, For adaptive threshold, The first part is the average entropy difference over 100 frames (without any operation). This represents the standard deviation of the entropy difference in the first 100 frames.

[0072] when Greater than or equal to And the first two frames are both smaller than If so, then mark the frame as the pulse start point; when Less than And the last two frames are both smaller than If the interval between the start points of two pulses is less than 3 frames, then the frame is marked as the end of the pulse; if the interval between the start points of two pulses is less than 3 frames, then they are merged into one pulse.

[0073] For each detected pulse, calculate the pulse width, waveform, and energy integral. If the pulse width is between 3 and 8 frames, the waveform is a symmetrical single peak, and the energy integral is moderate, it is an isolated droplet. If the pulse width is greater than 10 frames, the waveform is plateau-shaped or multi-peaked (fluid flow disturbance), and the energy integral is large, it is a continuous liquid flow.

[0074] S500: Fit a dripping trajectory line based on the movement path of the liquid pixels in consecutive frames, and determine whether the liquid has fallen into the container based on the positional relationship between the dripping trajectory line and the container; specifically as follows:

[0075] S510. Identify the container opening, obtain the container opening plane, and calculate the effective receiving range based on the container opening diameter and container wall thickness.

[0076] The bounding box of the container opening (such as the opening of a test tube or graduated cylinder) is detected using the YOLO algorithm. The upper left corner of the bounding box is (x min y min The bottom left corner is (x) max y max );

[0077] The horizontal range of the container opening is [x min x max Considering the container wall thickness and edge tolerance, the effective receiving range of the container is defined as [x]. min +q, x max -q], where q is the edge safety distance.

[0078] S520. Calculate the centroid coordinates (x, y) of the liquid pixels in each frame. c y cUsing Kalman filtering, the centroids of the same liquid clump are matched between adjacent frames to construct a centroid trajectory sequence;

[0079] S530. Based on the centroid trajectory sequence, the least squares method is used to fit a straight line;

[0080] Select a stable falling section of the liquid flow after it leaves the tip of the dropper (avoiding the acceleration section near the tip and the splash zone near the container opening), and fit a straight line based on the centroid trajectory sequence (a straight line is sufficient for short-distance dropping).

[0081] S540. Extend the fitted straight line towards the container opening to predict the landing point of the liquid on the container opening plane.

[0082] For the fitted linear model, extrapolate it to the y=y plane of the container opening. r At that location, we obtain the corresponding x-coordinate. pred ; where y r This is the ordinate of the container opening, usually taken as the top edge of the bounding box or slightly offset downwards.

[0083] S550: Compare whether the landing point is within the valid reception range to obtain the judgment result.

[0084] Determine x pred Is it in [x] min +q, x max If the value is within the range of -q], it means that the liquid has fallen into the container; otherwise, it has not fallen into the container.

[0085] S600: Count the identified isolated droplets and calculate the first drop volume based on the preset single drop volume;

[0086] The average single drop volume of the dropper is determined through preliminary experiments; then the total number of isolated droplets is multiplied by the single drop volume to obtain the first drop volume, which is the drop volume calculated based on the droplets.

[0087] S700: After the dropper moves away from the container at a preset distance, calculate the second distance from the liquid surface to the tip of the dropper;

[0088] The second distance needs to be calculated after the dripping is completed and when the liquid surface is relatively stable. Therefore, it is necessary to first calculate whether the direction of the dropper's movement is away from the container through consecutive frames. If so, the second distance is calculated after the dropper moves away from the container by a preset distance.

[0089] S800, Calculate the second drop volume based on the first distance and the second distance;

[0090] The difference between the first and second distances is the height at which the liquid is squeezed out of the dropper. The volume of liquid squeezed out can be calculated based on the specifications of the dropper.

[0091] S900. Weighted fusion of the first and second drop amounts to obtain the comprehensive drop amount. This step requires considering the confidence level of the drop amount calculated above for fusion, as detailed below:

[0092] S910. Extract the geometric features of the liquid pixels and calculate the morphological confidence based on the geometric features;

[0093] The roundness of the droplet is calculated based on the area and perimeter of the region. An ideal droplet is approximately circular, and the closer the roundness is to 1, the higher the score. It is determined whether the area falls within the preset range of a single droplet. If it does, the score is high, and if it is too large or too small, the score is low. The aspect ratio is calculated. A aspect ratio between 0.7 and 1.3 scores high, and if it exceeds this range, the score decreases linearly. The scores of these geometric features are weighted and averaged to obtain the morphological confidence score.

[0094] S920: Extract the motion features of the liquid pixels and calculate the motion confidence based on the motion features;

[0095] The smaller the angle between the droplet velocity direction and the direction of gravity, the higher the score; a high score is given if the velocity is within the theoretical free fall velocity range, and a low score is given if the velocity is too fast or too slow; the smaller the deviation from the gravitational acceleration g, the higher the score; the scores of these motion characteristics are weighted and averaged to obtain the motion confidence score.

[0096] S930. Extract the color and highlight features of the liquid pixels, calculate the matching degree between the color and highlight features and the standard liquid features, and obtain the appearance confidence score.

[0097] S940. Calculate the droplet detection confidence level based on morphological confidence level, motion confidence level, and appearance confidence level.

[0098] The droplet detection confidence level indicates whether the detected droplet is reliable, and directly determines the fusion weight of the subsequent droplet addition.

[0099] S950. Calculate the fusion weight based on the droplet detection confidence level, and then weight and fuse the first and second droplet amounts according to the fusion weight to obtain the comprehensive droplet amount.

[0100] The lower the droplet detection confidence, the lower the fusion weight of the first droplet addition. The overall droplet addition is calculated according to the fusion weight.

[0101] The above steps can identify whether students have inserted the dropper into the container, whether the liquid has been accurately added to the container, whether the liquid has been squeezed out as a drop instead of a continuous flow, and whether the amount added is correct. It can also identify whether the dropper is inverted. Based on the identification results, the operation can be scored directly.

[0102] Optionally, to avoid disputes arising from complete reliance on machine scoring, a human arbitration channel can be retained to calculate confidence levels for each testing process:

[0103] The consistency confidence level is calculated based on the difference between the first and second drops; if the difference is too large, the confidence level of the overall drops is low.

[0104] The illumination uniformity, occlusion ratio, and overall inter-frame displacement are calculated based on consecutive frame images. The video quality confidence level is then calculated based on the illumination uniformity, occlusion ratio, and overall inter-frame displacement.

[0105] The target detection confidence score is obtained based on the YOLO target detection model;

[0106] The overall confidence level is calculated based on the consistency confidence level, video quality confidence level, target detection confidence level, and droplet detection confidence level. The video is then labeled with a manual review tag based on the overall confidence level. For example, if the confidence level is less than 0.6, it is marked as requiring manual review, and if the confidence level is between 0.6 and 0.8, it can be marked as recommending manual sampling.

[0107] Example 2:

[0108] This embodiment provides a system for recognizing dropper experimental operations, including:

[0109] The first module is used to acquire continuous frame images of experimental operations;

[0110] The second module is used to identify the dropper and container through image recognition algorithms, extract the region of interest, identify the liquid surface inside the dropper, and calculate the first distance from the liquid surface to the tip of the dropper.

[0111] The third module is used to detect the positional relationship between the dropper tip and the container opening using a pre-trained YOLO object detection model;

[0112] The fourth module is used to perform differential operations on adjacent frames when the positional relationship meets the preset conditions, extract the pixel change features of the region of interest, identify liquid pixels based on the pixel change features, and distinguish between continuous liquid flow and isolated droplets.

[0113] The fifth module is used to fit a dripping trajectory line based on the movement path of the liquid pixels in consecutive frames, and to determine whether the liquid has fallen into the container based on the positional relationship between the dripping trajectory line and the container.

[0114] The sixth module is used to count the isolated droplets identified and calculate the amount of the first drop to be added based on the preset single drop volume;

[0115] The seventh module is used to calculate the second distance from the liquid surface to the tip of the dropper after the dropper has moved away from the container by a preset distance;

[0116] The eighth module is used to calculate the second drop volume based on the first and second distances;

[0117] The ninth module is used to weight and fuse the first and second drop amounts to obtain the comprehensive drop amount.

[0118] As an optional implementation, the fourth module includes:

[0119] The first unit is used to convert image frames into grayscale images, calculate the absolute value of the grayscale difference between adjacent frames pixel by pixel, and obtain the difference image.

[0120] The second unit is used to convert the difference map into a binary image according to a preset grayscale threshold, and to perform connected component analysis on the foreground pixels in the binary image to obtain at least one connected component.

[0121] The third unit is used to calculate the area, aspect ratio, shape factor, motion direction, and speed of each connected component to obtain pixel change features;

[0122] The fourth unit is used to filter connected components based on pixel change characteristics to obtain liquid pixels;

[0123] The fifth unit is used to distinguish between continuous liquid flow and isolated droplets based on the spatial morphology and temporal continuity of the liquid pixel connected domain.

[0124] As an optional implementation, the fifth module includes:

[0125] The sixth unit is used to identify the container opening, obtain the container opening plane, and calculate the effective receiving range based on the container opening diameter and container wall thickness.

[0126] The seventh unit is used to calculate the centroid coordinates of the liquid pixels in each frame, and uses Kalman filtering to match the centroids of the same liquid clump between adjacent frames to construct a centroid trajectory sequence.

[0127] Unit 8 is used to fit straight lines using the least squares method based on centroid trajectory sequences;

[0128] The ninth unit is used to extend the fitted straight line towards the container opening to predict the landing point of the liquid on the container opening plane.

[0129] The tenth unit is used to compare whether the landing point is within the valid reception range and obtain the judgment result.

[0130] As an optional implementation, the ninth module includes:

[0131] The eleventh unit is used to extract the geometric features of liquid pixels and calculate the morphological confidence based on the geometric features;

[0132] The twelfth unit is used to extract motion features from liquid pixels and calculate motion confidence based on the motion features;

[0133] The thirteenth unit is used to extract the color and highlight features of liquid pixels, calculate the matching degree between the color and highlight features and the standard liquid features, and obtain the appearance confidence score.

[0134] Unit 14 is used to calculate the droplet detection confidence level based on morphological confidence level, motion confidence level, and appearance confidence level.

[0135] The fifteenth unit is used to calculate the fusion weight based on the droplet detection confidence level, and to perform weighted fusion of the first and second droplet amounts according to the fusion weight to obtain the comprehensive droplet amount.

[0136] Example 3:

[0137] Corresponding to the above method embodiments, this embodiment also provides a device for identifying dropper experimental operations. The device for identifying dropper experimental operations described below and the method for identifying dropper experimental operations described above can be referred to in correspondence.

[0138] Figure 2 This is a block diagram illustrating an identification device 800 for a dropper experimental operation according to an exemplary embodiment. Figure 2As shown, the dropper experiment operation recognition device 800 includes a processor 801 and a memory 802. The dropper experiment operation recognition device 800 may also include one or more of the following: a multimedia component 803, an input / output (I / O) interface 804, and a communication component 805. The processor 801 controls the overall operation of the dropper experiment operation recognition device 800 to complete all or part of the steps in the dropper experiment operation recognition method described above. The memory 802 stores various types of data to support the operation of the dropper experiment operation recognition device 800. This data may include, for example, commands for any application or method operating on the dropper experiment operation recognition device 800, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0139] Multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals.

[0140] The received audio signal can be further stored in memory 802 or transmitted via communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the identification device 800 for the dropper experimental operation and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof, is used. Therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0141] Example 4:

[0142] Corresponding to the above embodiment of the method for identifying dropper experimental operations, this embodiment also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the above-described method for identifying dropper experimental operations.

[0143] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for identifying dropper experimental operations.

[0144] Specifically, the readable storage medium can be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0145] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0146] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying experimental operations using a dropper, characterized in that, include: Acquire continuous frame images of the experimental operation; The dropper and container are identified using an image recognition algorithm, the region of interest is extracted, the liquid level inside the dropper is identified, and the first distance from the liquid level to the tip of the dropper is calculated. A pre-trained YOLO object detection model was used to detect the positional relationship between the dropper tip and the container opening; When the positional relationship meets preset conditions, differential operations are performed on adjacent frames to extract pixel change features of the region of interest. Based on these pixel change features, liquid pixels are identified, and continuous liquid flow and isolated droplets are distinguished. This includes: The image frames are converted into grayscale images, and the absolute value of the grayscale difference between adjacent frames is calculated pixel by pixel to obtain the difference image; The difference map is converted into a binary image according to a preset grayscale threshold. Connectivity analysis is performed on the foreground pixels in the binary image to obtain at least one connected component. For each connected component, calculate its area, aspect ratio, shape factor, direction of motion, and velocity to obtain pixel change features; Liquid pixels are obtained by filtering connected components based on pixel change characteristics; Continuous liquid flow and isolated droplets can be distinguished based on the spatial morphology and temporal continuity of the connected domains of liquid pixels; The dripping trajectory is fitted based on the movement path of the liquid pixels in consecutive frames, and the liquid is determined to fall into the container based on the positional relationship between the dripping trajectory and the container. The number of isolated droplets identified is counted, and the amount of the first droplet added is calculated based on the preset single droplet volume; After the dropper is moved away from the container by a preset distance, calculate the second distance from the liquid surface to the tip of the dropper; Calculate the second drop volume based on the first and second distances; The first and second drops are weighted and combined to obtain the overall dosage, which includes: Extract the geometric features of the liquid pixels and calculate the morphological confidence score based on the geometric features; Motion features of liquid pixels are extracted, and motion confidence is calculated based on these motion features. Extract color and highlight features from liquid pixels, calculate the matching degree between color and highlight features and standard liquid features, and obtain appearance confidence score; The droplet detection confidence level is calculated based on morphological confidence level, motion confidence level, and appearance confidence level. The fusion weight is calculated based on the droplet detection confidence level. The first and second droplet amounts are then weighted and fused according to the fusion weight to obtain the comprehensive droplet amount.

2. The method for identifying dropper experimental operations according to claim 1, characterized in that, The process involves fitting a drop trajectory line based on the movement path of liquid pixels in consecutive frames, and determining whether liquid has fallen into the container based on the positional relationship between the drop trajectory line and the container, including: Identify the container opening, obtain the container opening plane, and calculate the effective receiving range based on the container opening diameter and container wall thickness; For each liquid pixel in a frame, calculate its centroid coordinates, use Kalman filtering to match the centroids of the same liquid clump between adjacent frames, and construct a centroid trajectory sequence. Based on the centroid trajectory sequence, the least squares method is used to fit a straight line; Extend the fitted straight line towards the container opening to predict the landing point of the liquid on the container opening plane. The judgment result is obtained by comparing whether the landing point is within the effective reception range.

3. The method for identifying dropper experimental operations according to claim 1, characterized in that, The method of identifying liquid pixels and distinguishing between continuous liquid flow and isolated droplets based on pixel change features includes: Cut the area of ​​interest between the dropper tip and the container opening according to the preset width; For each frame, calculate the entropy within a preset sliding window in the region of interest; The entropy difference between adjacent frames is calculated, and entropy difference pulses are detected. Pulse features are extracted, and continuous liquid flow and isolated droplets are distinguished based on the pulse features.

4. The method for identifying dropper experimental operations according to claim 1, characterized in that, The method further includes: Calculate the consistency confidence level based on the difference between the first and second drops; The illumination uniformity, occlusion ratio, and overall inter-frame displacement are calculated based on consecutive frame images. The video quality confidence level is then calculated based on the illumination uniformity, occlusion ratio, and overall inter-frame displacement. The target detection confidence score is obtained based on the YOLO target detection model; The overall confidence score is calculated based on the consistency confidence score, video quality confidence score, target detection confidence score, and droplet detection confidence score. The video is then labeled with a manual review tag based on the overall confidence score.

5. A system for recognizing dropper experimental operations, characterized in that, include: The first module is used to acquire continuous frame images of experimental operations; The second module is used to identify the dropper and container through image recognition algorithms, extract the region of interest, identify the liquid surface inside the dropper, and calculate the first distance from the liquid surface to the tip of the dropper. The third module is used to detect the positional relationship between the dropper tip and the container opening using a pre-trained YOLO object detection model; The fourth module is used to perform differential operations on adjacent frames when the positional relationship meets the preset conditions, extract the pixel change features of the region of interest, identify liquid pixels based on the pixel change features, and distinguish between continuous liquid flow and isolated droplets. The fourth module includes: The first unit is used to convert image frames into grayscale images, calculate the absolute value of the grayscale difference between adjacent frames pixel by pixel, and obtain the difference image. The second unit is used to convert the difference map into a binary image according to a preset grayscale threshold, and to perform connected component analysis on the foreground pixels in the binary image to obtain at least one connected component. The third unit is used to calculate the area, aspect ratio, shape factor, motion direction, and speed of each connected component to obtain pixel change features; The fourth unit is used to filter connected components based on pixel change characteristics to obtain liquid pixels; The fifth unit is used to distinguish between continuous liquid flow and isolated droplets based on the spatial morphology and temporal continuity of the liquid pixel connected domains. The fifth module is used to fit a dripping trajectory line based on the movement path of the liquid pixels in consecutive frames, and to determine whether the liquid has fallen into the container based on the positional relationship between the dripping trajectory line and the container. The sixth module is used to count the isolated droplets identified and calculate the amount of the first drop to be added based on the preset single drop volume; The seventh module is used to calculate the second distance from the liquid surface to the tip of the dropper after the dropper has moved away from the container by a preset distance; The eighth module is used to calculate the second drop volume based on the first and second distances; The ninth module is used to weight and fuse the first and second drop amounts to obtain the comprehensive drop amount; The ninth module includes: The eleventh unit is used to extract the geometric features of liquid pixels and calculate the morphological confidence based on the geometric features; The twelfth unit is used to extract motion features from liquid pixels and calculate motion confidence based on the motion features; The thirteenth unit is used to extract the color and highlight features of liquid pixels, calculate the matching degree between the color and highlight features and the standard liquid features, and obtain the appearance confidence score. Unit 14 is used to calculate the droplet detection confidence level based on morphological confidence level, motion confidence level, and appearance confidence level. The fifteenth unit is used to calculate the fusion weight based on the droplet detection confidence level, and to perform weighted fusion of the first and second droplet amounts according to the fusion weight to obtain the comprehensive droplet amount.

6. The identification system for dropper experimental operations according to claim 5, characterized in that, The fifth module includes: The sixth unit is used to identify the container opening, obtain the container opening plane, and calculate the effective receiving range based on the container opening diameter and container wall thickness. The seventh unit is used to calculate the centroid coordinates of the liquid pixels in each frame, and uses Kalman filtering to match the centroids of the same liquid clump between adjacent frames to construct a centroid trajectory sequence. Unit 8 is used to fit straight lines using the least squares method based on centroid trajectory sequences; The ninth unit is used to extend the fitted straight line towards the container opening to predict the landing point of the liquid on the container opening plane. The tenth unit is used to compare whether the landing point is within the valid reception range and obtain the judgment result.

Images