A funnel spout and beaker position detection method, system, terminal and medium

Abstract

The application provides a funnel spout and beaker position detection method, system, terminal and medium, comprising: acquiring a video of experimental operation; extracting any image in the video and inputting the image into a ResNet18 model to obtain the position relationship of the funnel spout and the beaker on both sides; inputting the key frame of the video into a CRNN model to obtain the drop state of the funnel spout; and judging whether the funnel spout and the beaker are attached to the wall according to the position relationship of the funnel spout and the beaker on both sides and / or the drop state of the funnel spout. The application performs multi-directional detection and judgment, avoids misjudgment, greatly improves the test accuracy, and saves a large amount of manpower cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a funnel spout and beaker position detection method, system, terminal and medium. BACKGROUND

[0002] With the continuous expansion of junior high school students in Shanghai, the teacher resources needed for physics and chemistry experiments are also increasing every year. Therefore, it is necessary to use artificial intelligence methods to alleviate this problem. For the "filtration" experiment in chemical experiments, a model method for judging whether the funnel spout is close to the inner wall of the beaker is proposed in the field of computer vision.

[0003] In the prior art, a convolutional neural network is used for binary classification to determine whether it is close to the inner wall. However, since the experimental camera is a monocular camera, it cannot obtain the depth information of the picture. Therefore, the following situations may occur: Figure 1 When viewed from the front, the funnel spout is not close to the edges on both sides, but in fact, the funnel spout is close to the front and rear edge sides of the beaker. If the convolutional neural network is used for binary classification, it will cause misjudgment, which will indirectly affect the fairness of the examination and delay the future of the students.

[0004] After searching, the Chinese patent with application number 202010203774.X discloses a chemical funnel device building experiment operation device and intelligent scoring method, which includes an experiment table, an operation assembly, a student cloud terminal display screen and a network camera. The operation assembly is placed on the experiment table, and the student cloud terminal display screen is installed directly above the experiment table. The network camera is also installed on the experiment table; the operation assembly includes an iron stand, a long-neck funnel, a beaker and filter paper, and the long-neck funnel is placed on the iron stand. The present application uses a front-view camera and a top-view camera to collect the video of the student's experiment operation process, uses a deep learning target detection algorithm to identify the target objects in the collected video, scores the student's funnel device building experiment according to the current scoring rules of the funnel building experiment, and obtains the final operation score. It judges whether the funnel is close to the wall by measuring the distance between the funnel and the beaker in the top view. This method will still cause misjudgment in some experimental situations. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a funnel spout and beaker position detection method, system, terminal and medium.

[0006] According to one aspect of the present application, a funnel spout and beaker position detection method in a physics and chemistry experiment is provided, comprising:

[0007] obtaining a video of an experiment operation;

[0008] extracting any one image in the video to input a ResNet18 model to obtain the position relationship of the funnel spout and the two sides of the beaker;

[0009] inputting the key frame of the video into a CRNN model to obtain the drop state of the funnel spout;

[0010] judging whether the funnel spout and the beaker are attached to the wall according to the position relationship of the funnel spout and the two sides of the beaker and / or the drop state of the funnel spout.

[0011] Preferably, the extracting any one image in the video to input a ResNet18 model to obtain the position relationship of the funnel spout and the two sides of the beaker comprises:

[0012] inputting the any one image into a convolutional layer;

[0013] inputting the result output after the convolutional layer into 8 residual modules;

[0014] inputting the result output by the residual module into a pooling layer;

[0015] inputting the result output by the pooling layer into a fully connected layer;

[0016] performing SoftMax operation on the result output by the fully connected layer to obtain the position relationship of the funnel spout and the two sides of the beaker, and judging whether the funnel spout is attached to the wall.

[0017] Preferably, each residual module comprises two convolutional layers.

[0018] Preferably, the inputting the key frame image of the video into a CRNN model to obtain the drop state of the funnel spout comprises:

[0019] selecting each two frames of the left and right of the key frame to form five frames of images;

[0020] inputting the five frames of images into a light flow network FlowNet to obtain light flow features;

[0021] performing Grayscale on the five frames of images to obtain grayscale features;

[0022] splicing the light flow features and the grayscale features to obtain spliced features;

[0023] inputting the spliced features into a ResNet18 network to obtain a feature vector;

[0024] inputting the feature vectors obtained by all key frames into an RNN network to obtain the drop state of the funnel spout.

[0025] Preferably, the key frame is every frame from the third frame to the third last frame in the video.

[0026] Preferably, the feature vector obtained by all the key frames is input into the CRNN model to obtain the drop state of the funnel spout, including:

[0027] The video includes X key frames, and X feature vectors are obtained correspondingly;

[0028] The X feature vectors are input into the CRNN network to obtain X results;

[0029] Among the X results, a are liquid drops sliding along the beaker wall, b are liquid drops falling, and c are no drops, and a+b+c=X;

[0030] total is set to be a-b;

[0031] When |total|>2, if total is positive, it indicates that the liquid drops are sliding along the beaker wall, and the wall sticking state is determined; if total is negative, it indicates that the liquid drops are falling, and the non-wall sticking state is determined;

[0032] When |total|≤2, manual intervention is determined.

[0033] Preferably, the determination of whether the funnel spout and the beaker are wall sticking according to the positional relationship between the funnel spout and the beaker, the existence of the drop in the funnel spout, and / or the state of the drop falling, includes:

[0034] When at least one of the positional relationship between the funnel spout and the beaker and / or the drop state is determined to be wall sticking, the final result is determined to be wall sticking.

[0035] According to a second aspect of the present application, a detection system for the position of the funnel spout and the beaker in a physical and chemical experiment is provided, including:

[0036] A position module for the funnel spout and the beaker on both sides, which obtains the position of the funnel spout and the beaker;

[0037] A drop state module, which obtains the drop state of the funnel spout;

[0038] A determination module, which determines whether the funnel spout and the beaker are wall sticking according to the positional relationship between the funnel spout and the beaker and / or the drop state of the funnel spout.

[0039] According to a third aspect of the present application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to execute any of the methods or runs the system.

[0040] According to a fourth aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, is adapted to perform any of the methods described, or to operate the system described.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] The funnel spout and beaker position detection method and system in the physical and chemical experiment provided by the embodiment of the present application can detect and judge the experimental operation video from multiple directions, avoid misjudgment, greatly improve the test accuracy, and save a large amount of manpower cost. BRIEF DESCRIPTION OF DRAWINGS

[0043] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0044] Figure 1 In actual operation, the funnel spout and the inner wall of the beaker are in close contact, which may cause misjudgment;

[0045] Figure 2 The ResNet18 model structure diagram in the preferred embodiment of the present application;

[0046] Figure 3 The CRNN model structure diagram in another preferred embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These all belong to the protection scope of the present application.

[0048] The final purpose of the present application is to judge whether the funnel spout is in contact with the wall. In view of the misjudgment in the prior art, the problem needs to be converted. After conversion, the judgment process is:

[0049] Step 1: First, judge whether the spout funnel is in close contact with the left and right sides of the beaker. If it is in close contact, it is considered to be in contact with the wall; if it is not in close contact, further judgment is needed, and go to step 2.

[0050] Step 2: In the case that the funnel spout is not in close contact, it may be in contact with the wall or not, such as Figure 1Therefore, the problem of determining whether the liquid adheres to the beaker wall needs to be transformed into determining whether the droplet in the video is dripping, sliding down the beaker wall, or not dripping at all. If it is dripping, it is considered not adhering to the beaker wall, and the experiment receives no points; if it is sliding down the beaker wall, it is considered adhering to the beaker wall; if there is no dripping, it is considered to have no effect.

[0051] Based on the above conceptual steps, applied to, for example Figure 1 The model is shown. Generally, the input content is the main view. Figure (a) is close to both sides of the beaker wall and is directly judged as scoring. Figures (b) and (c) need to be further distinguished. The case without droplets is considered to be attached to the wall. The case with droplets needs to be further distinguished to determine whether the droplets flow down the beaker wall or drip down drop by drop.

[0052] In view of the above-mentioned inventive concept, the present invention provides an embodiment of a method for detecting the position of the funnel tip relative to the beaker in a physicochemical experiment, comprising:

[0053] S1, Obtain the experimental video;

[0054] S2, extract several images from the video of the experimental operation and input them into the ResNet18 model to obtain the positional relationship between the funnel tip and the two sides of the beaker;

[0055] S3, input the keyframes of the video in S1 into the CRNN model to obtain the dripping state of the funnel tip;

[0056] S4. Based on the positional relationship between the funnel tip and the two sides of the beaker obtained in S2 and / or the dripping state of the funnel tip obtained in S3, determine whether the funnel tip is attached to the wall of the beaker.

[0057] like Figure 2 The diagram shown is a schematic diagram of the ResNet18 model structure in S2 of a preferred embodiment of the present invention.

[0058] In this embodiment, before S2 is executed, images need to be acquired and labeled, and training is performed. Figure 2 The ResNet18 model is used for classification. The specific process includes:

[0059] All frames from multiple experimental operation videos were saved. Ten images were randomly selected from each video, observed, and labeled to obtain a training set and a validation set. The labeled categories were divided into two groups: images with the funnel tip close to the left and right sides were considered one group, while other cases were considered far away. In this embodiment, the training set contained 1110 images with the tip close to the left and right sides, and 3330 images with the tip far away. The validation set contained 280 images with the tip close to the left and right sides, and 840 images with the tip far away.

[0060] Using the above training and validation sets, train Figure 2The ResNet18 model in the S1 is used to make a close or far binary classification. If the model considers it close, it is considered that the operation step of the experiment is correct; if the model considers it far, it needs to enter S2 for further judgment.

[0061] Further, the trained ResNet18 model is used to execute S2, including:

[0062] S201, an image in the experimental operation video is selected as an input image, the image size is (224, 224, 3), and a convolutional layer is used.

[0063] S202, 8 residual modules are used, each residual module includes two convolutional layers.

[0064] S203, a pooling layer is further used, and then a fully connected layer is used, and finally a SoftMax operation is performed to obtain a classification result.

[0065] In a preferred embodiment of the present application, the training Figure 3 The CRNN model in the S1 is used to judge whether the droplet is dropped, slides along the beaker wall or there is no droplet. The meaning of "no event" can be understood as follows: because the water droplets are not very fast and continuous, there may be some video frames without water droplets, therefore, a "no event" is needed as a result

[0066] Based on the CRNN model, when the result of S2 is that the funnel tip is close to the beaker inner wall, S3 is implemented, specifically including:

[0067] S301, the input image size is (224, 224, 3), a key frame in the video is selected, and two frames on the left and right thereof are further selected to form five frames of images.

[0068] S302, the five frames of images are input into a flow network (FlowNet) to obtain flow features.

[0069] S303, the five frames of images are grayscale (Grayscale) to obtain grayscale features.

[0070] S304, the grayscale features and the flow features are spliced and input into a ResNet18 network to obtain a 512-dimensional vector.

[0071] S305, each frame from the third frame to the third last frame in the video is taken as a key frame, and the steps of S301-S304 are executed, and the obtained features are input into an RNN network to obtain a final classification result.

[0072] Further, X 512-dimensional feature vectors are finally obtained through S305 and input into the RNN network to obtain X results. Among the X results, a are the results of the liquid drops sliding along the beaker wall, b are the results of the liquid drops falling, and c are the results of no liquid drops, and a+b+c=X.

[0073] total=a-b is set.

[0074] When |total|>2, if total is positive, it indicates that the liquid drops slide along the beaker wall, and the wall sticking state is determined; if total is negative, it indicates that the liquid drops fall, and the non-wall sticking state is determined.

[0075] When |total|≤2, manual intervention is determined.

[0076] The embodiment is aimed at the difficulty of monocular camera in estimating the depth information of the object in the picture, which leads to the inability to determine whether the funnel spout sticks to the wall. A deep learning method is proposed to classify and convert the problem into an approximately equivalent learnable problem. Therefore, considering the difference between the funnel spout sticking to the wall and not sticking to the wall, intuitively, the flowing liquid is in the form of liquid drops or water flow. Since the water drops only occupy a small part of the whole picture, and most of the areas in the video do not change, the changes between two pictures can be extracted through the optical flow model. In this way, whether the water drops fall can be focused on. The embodiment improves the detection accuracy and saves a lot of manpower.

[0077] It is worth noting that the execution order of each step of S1 and S2 is not limited; and the use of single or multiple S1 and S2 is not limited. If the determination result is wall sticking, the judgment process of the embodiment ends; if it is temporarily unable to determine whether it is wall sticking, another determination step can be selected for the next step of judgment until the determination result is obtained.

[0078] Based on the same inventive concept, other embodiments of the present application also provide a detection system for the position of the funnel spout and the beaker in a physical and chemical experiment, which comprises a position module on both sides of the funnel spout and the beaker, a liquid drop state module, and a judgment module; the position module obtains the position of the funnel spout and the beaker; the liquid drop state module obtains the liquid drop state; and the judgment module determines whether the funnel spout and the beaker stick to each other according to the position relationship between the funnel spout and the beaker and / or the liquid drop state of the funnel spout.

[0079] Based on the same inventive concept, in other embodiments of the present application, a terminal is also provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to execute any of the methods or runs the system.

[0080] Based on the same inventive concept, in other embodiments of the present application, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, is adapted to perform any of the methods described, or to operate the system described.

[0081] It should be noted that the steps in the method provided by the present application can be implemented by using the corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solutions of the system to implement the step flow of the method, i.e., the embodiments in the system can be understood as preferred examples of implementing the method, and thus will not be described here.

[0082] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device thereof in a pure computer readable program code manner, the same functions can also be achieved by logically programming the method steps to make the system provided by the present application and each device thereof in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, the system provided by the present application and each device thereof can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing methods and structures within hardware components.

[0083] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which do not affect the essential content of the present application. The above preferred features are combinable in any combination as long as they are not in conflict with each other.

Claims

1. A method for detecting the position of a funnel spout and a beaker in a physical and chemical experiment, characterized in that, The method comprises the following steps: acquiring a video of experimental operation; extracting any one image in the video to input a ResNet18 model to acquire a position relationship between a funnel spout and two sides of a beaker; inputting key frames of the video into a CRNN model to acquire a drop state of the funnel spout; judging whether the funnel spout and the beaker are wall-adhered according to the position relationship between the funnel spout and the two sides of the beaker and / or the drop state of the funnel spout; the step of inputting the video key frame image into the CRNN model to acquire the drop state of the funnel spout comprises the following steps: selecting each two frames on the left and right of the key frame to form five frame images; inputting the five frame images into a flow network FlowNet to obtain flow features; grayscale processing the five frame images to obtain grayscale features; splicing the flow features and the grayscale features to obtain spliced features; inputting the spliced features into a ResNet18 network to obtain a feature vector; inputting the feature vectors obtained by all the key frames into an RNN network to obtain the drop state of the funnel spout; the step of inputting the feature vectors obtained by all the key frames into the RNN network to obtain the drop state of the funnel spout comprises the following steps: the video comprises X key frames, and X feature vectors are obtained correspondingly; inputting the X feature vectors into an RNN model to obtain X results; in the X results, a liquid drop slides along the wall of the beaker, b liquid drops drop, and c has no drop, and a+b+c=X; setting total = a-b; when |total|>2, if total is positive, it indicates that the liquid drop slides along the wall of the beaker, and it is judged as a wall-adhered state; if total is negative, it indicates that the liquid drop drops, and it is judged as a non-wall-adhered state; when |total|≤2, manual intervention is required. the step of extracting any one image in the video to input the ResNet18 model to acquire the position relationship between the funnel spout and the two sides of the beaker comprises the following steps:

2. The method according to claim 1, wherein, inputting the any one image into a convolution layer; inputting the output result of the convolution layer into eight residual modules; inputting the output result of the residual modules into a pooling layer; inputting the output result of the pooling layer into a full connection layer; performing a SoftMax operation on the output result of the full connection layer to obtain the position relationship between the funnel spout and the two sides of the beaker, and judging whether the funnel spout is wall-adhered. Each residual module comprises two convolution layers.

3. The method according to claim 2, wherein, The key frame is each frame from the third frame to the last third frame in the video.

4. The method according to any one of claim 3, characterized in that, the step of judging whether the funnel spout and the beaker are wall-adhered according to the position relationship between the funnel spout and the two sides of the beaker, whether the funnel spout has a drop, and / or a drop falling state comprises the following steps:

5. The method according to any one of claims 1-4, wherein, when at least one of the position relationship between the funnel spout and the two sides of the beaker and / or the drop state is judged as wall-adhered, the final result is judged as wall-adhered. The method comprises the following steps:

6. A detection system for implementing the detection method of the position of the funnel spout and the beaker in a physico-chemical experiment according to claim 1, characterized in that, a position module for acquiring the position of the funnel spout and the beaker; a drop state module for acquiring the drop state of the funnel spout. ​ A judging module is configured to judge whether the beaker is in contact with the funnel spout according to the position relationship between the funnel spout and the beaker and / or the state of the droplet of the funnel spout.

7. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, can be configured to execute the method of any one of claims 1-5, or run the system of claim 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, can be configured to execute the method of any one of claims 1-5, or run the system of claim 6. The program, when executed by the processor, can be configured to execute the method of any one of claims 1-5, or run the system of claim 6.

Images