A method and system for determining whether a filter paper experiment is scored

Abstract

The application provides a method and system for judging whether a filter paper experiment is scored, comprising: detecting experiment operations in a video stream of the filter paper experiment to obtain a picture set of a time period of interest; training a full convolutional neural network (FCN) model for obtaining an image label based on the picture set of the time period of interest; performing strong supervision target classification on the picture set of the time period of interest frame by frame based on the trained FCN model to obtain a classification result of whether a lower end of a glass rod is close to three layers of filter paper; and performing multi-frame voting based on the classification result of each frame in the picture set of the time period of interest to determine whether the experiment operations in the video stream are scored. The data required by the application is greatly reduced, and is suitable for the size of video data in actual experiment operation processes. Moreover, the target area can be quickly focused on by detecting a region of interest, and the detection accuracy is effectively improved. Through the above method and system, a large amount of human cost is saved, and the detection efficiency of physical and chemical experiment scoring is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for determining whether a filter paper experiment has received a score. Background Technology

[0002] As Shanghai continues to expand its junior high school enrollment, the number of teachers needed for physics and chemistry experiments is also increasing every year. To alleviate this problem, the determination of whether the lower end of the glass rod is in close contact with one side of the three layers of filter paper in the "filtration" experiment can be made using artificial intelligence, reducing manual operation.

[0003] In existing technologies, binary classification is performed directly using convolutional neural networks. This method is theoretically feasible, but it still has the following problems: First, the model has difficulty focusing on the regions that should be focused on in the image, resulting in low model accuracy; Second, a large number of samples are required for training, but in real-world scenarios, the amount of data obtained is not large enough to train a traditional classification model. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for determining whether a filter paper experiment has received a score.

[0005] According to one aspect of the present invention, a method for determining whether a filter paper experiment has been scored is provided, comprising:

[0006] Experimental operations in the video stream are detected to obtain a set of images for the time periods of interest;

[0007] Based on the image set for the time period of interest, a fully convolutional neural network (FCN) model is trained to obtain image labels;

[0008] Based on the trained fully convolutional neural network (FCN) model, strongly supervised target classification is performed frame by frame on the image set of the time period of interest to obtain the classification result of whether the lower end of the glass rod is close to the three layers of filter paper.

[0009] Based on the classification results of each frame in the image set of the time of interest, multi-frame voting is performed to determine whether the experimental operation in the video stream receives a score.

[0010] Preferably, the experimental operation on the video stream is detected to obtain a set of images for the time period of interest, including:

[0011] The image frames from the video stream are input into the detection model to determine whether they contain a region of interest.

[0012] Each frame of the video stream is detected to obtain multiple images containing the region of interest, forming a set of images for the time period of interest.

[0013] Preferably, the conditions for determining the region of interest include: the detection of filter paper, the detection of a glass rod, and the intersection of the circumscribed moments of the filter paper and the glass rod.

[0014] Preferably, training a fully convolutional neural network (FCN) model based on the image set of the time period of interest includes:

[0015] The N images of the obtained image set for the time period of interest are divided into a training set and a test set. The multiple images are manually labeled. Each image is divided into three parts: a glass rod, a filter paper, and a background. These three parts constitute the label maskg.t of the image.

[0016] The FCN model was trained using manually labeled training and test sets.

[0017] Preferably, the step of performing strongly supervised target classification frame by frame on the image set of the time period of interest based on the trained FCN model to determine whether the lower end of the glass rod is in close contact with the three layers of filter paper includes:

[0018] Using a trained fully convolutional neural network (FCN) model, obtain the label mask gt of the input image;

[0019] Based on the obtained label mask gt, four parts are extracted from the original image: the original image, the filter paper, the glass rod, and the target composed of the filter paper and the glass rod. The four parts of the image are then adjusted to be the same size as the original image.

[0020] The four images are input into a CNN network to obtain four feature maps;

[0021] Adjust the label mask gt corresponding to the four feature maps to be the same size as the feature map; multiply the feature map with the corresponding elements of the label mask gt to obtain a new feature map;

[0022] The four new feature maps are then subjected to mean pooling and max pooling to obtain four feature vectors, which are then concatenated together.

[0023] The concatenated feature vectors are input into a fully connected layer and a softmax function to obtain the classification result.

[0024] Preferably, the step of determining whether the experimental operation in the video stream scores a point by performing multi-frame voting based on the judgment results of each frame includes:

[0025] The classification results of each frame in the image set for the time period of interest are statistically analyzed.

[0026] The result with the highest frequency will be used as the final judgment result.

[0027] Preferably, the classification result includes two categories: the side where the lower end of the glass rod is close to the three layers of filter paper and the side where the lower end of the glass rod is not close to the three layers of filter paper.

[0028] According to a second aspect of the present invention, a score point evaluation system based on strongly supervised target classification is provided, comprising:

[0029] The module for detecting the time-of-interest image set is used to detect experimental operations in the video stream of the filter paper experiment and obtain the time-of-interest image set.

[0030] The FCN module trains a fully convolutional neural network (FCN) model to obtain image labels based on the image set of the time period of interest.

[0031] The single-frame judgment module, based on the trained fully convolutional neural network (FCN) model, performs strongly supervised target classification on the image set of the time period of interest frame by frame to obtain the classification result of whether the lower end of the glass rod is close to the three layers of filter paper;

[0032] The timing judgment module performs multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point.

[0033] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method for determining whether a filter paper experiment has been scored, or to run the above-described system for determining whether a filter paper experiment has been scored.

[0034] According to a fourth aspect of the present invention, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method for determining whether a filter paper experiment has been scored, or to run the above-described system for determining whether a filter paper experiment has been scored.

[0035] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0036] The method and system for determining whether a filter paper experiment has been scored in this embodiment of the invention require significantly less data, making it suitable for video data sizes in actual experimental operations; furthermore, by detecting the region of interest, it can quickly focus on the target area, effectively improving detection accuracy.

[0037] The method and system for determining whether a beaker experiment has been scored in this invention embodiment can automatically evaluate whether the experimental operation has been scored, saving a lot of manpower costs and improving the detection efficiency of physicochemical experiment scores. Attached Figure Description

[0038] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0039] Figure 1 This is a flowchart of a method for determining whether a filter paper experiment scores a point in one embodiment of the present invention;

[0040] Figure 2 This is a structural diagram of the FCN model in a preferred embodiment of the present invention;

[0041] Figure 3 This is a diagram of the four-stream Mask-CNN structure in a preferred embodiment of the present invention;

[0042] Figure 4 This is a comparison image of the original image and the mask gt in a preferred embodiment of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0044] See Figure 1 This invention provides an embodiment of a method for determining whether a filter paper experiment has received a score, comprising:

[0045] S100: Detects the video stream and obtains a set of images for the time period of interest;

[0046] S200, based on the image set of the time period of interest obtained in S100, trains a fully convolutional neural network (FCN) model;

[0047] S300, based on the fully convolutional neural network (FCN) model trained by S200, performs strongly supervised target classification on the image set of the time period of interest frame by frame to obtain the classification result of whether the lower end of the glass rod is close to the three layers of filter paper;

[0048] S400 uses multi-frame voting based on the judgment results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point.

[0049] The method for judging scores in the filter paper experiment described in this embodiment requires significantly less data compared to existing technologies, making it suitable for video data sizes used in actual experimental operations. Furthermore, by detecting the region of interest, it can quickly focus on the target area, effectively improving detection accuracy. This method saves substantial labor costs and improves the efficiency of physicochemical experiment score detection.

[0050] In a preferred embodiment of the present invention, step S100 involves inputting the video stream into a detection model to detect whether each frame is a region of interest (ROI). If not, the detection continues for the later parts of the video stream; if so, the image of the time period of interest and the location information of the experimental equipment are input into a classification model. Specifically, this includes two steps:

[0051] S101, The image is input into the detection model to determine whether it contains a region of interest;

[0052] S102, perform the operation S101 on each frame of the video stream to obtain N images of interest.

[0053] In a preferred embodiment, the region of interest is determined by simultaneously satisfying three conditions: first, filter paper is detected; second, a glass rod is detected; and third, the circumscribed moments of the filter paper and the glass rod intersect. These three conditions effectively detect the region of interest in the filter paper experiment. Existing models can be used for detection.

[0054] If it is a region of interest, first calculate the union of the filter paper's outer moments and the glass rod's outer moments, then calculate the outer moments of this union. The image framed by the union's outer moments is then used as a candidate image and input into the FCN model.

[0055] See Figure 2 In a preferred embodiment of the present invention, S200 is implemented, and the specific process includes:

[0056] S201, divide the N images obtained in S102 into training and testing sets, and manually label these N images, dividing them into three parts: glass rod, filter paper, and background. The label mask gt for this image is composed of these three parts, as shown below. Figure 4 As shown, the long strips are glass rods, the serrated edges are filter paper, and the rest is the background.

[0057] S202 uses the labeled data from S201 to train the FCN model.

[0058] A trained FCN model can obtain the label mask ground truth (gt) of the input image. The training set is used to train the FCN model, and the test set is used to verify the model's training effect. For the specific structure of the FCN model, please refer to [link / reference needed]. Figure 2 .

[0059] See Figure 3 In a preferred embodiment of the present invention, S300 is implemented, and the specific process includes:

[0060] S301, using the FCN model trained by S200, predicts the label mask gt of an image;

[0061] S302, based on the label mask gt predicted by S301, extracts four parts from the original image: the original image, the filter paper, the glass rod, and the target (including both the filter paper and the glass rod). These four parts are then resized to match the size of the original image. Figure 3 (a) Inputs;

[0062] S303, input the four parts of the image from S302 into the CNN network respectively, that is Figure 3 (b) CNN, resulting in 4 feature maps, namely: Figure 3 (c)Convolutional activation tensor;

[0063] S304, resize the label mask gt corresponding to the four feature maps to the size of the feature maps, and multiply the feature maps by the corresponding elements of the label mask gt to obtain new feature maps, i.e. Figure 3 (d) Descriptor selection;

[0064] S305, the four new feature maps obtained in S304 are subjected to mean pooling and max pooling respectively to obtain 1024-dimensional feature vectors. Then, these four 1024-dimensional feature vectors are concatenated together to obtain a 4096-dimensional feature vector.

[0065] S306 Finally, the 4096-dimensional feature vector is input into the fully connected layer (fc) and the softmax function to obtain the final classification result.

[0066] In a preferred embodiment, the implementation process of S302 is as follows:

[0067] First, the image input to the FCN model is the original image. The FCN model then produces a glass rod mask and a filter paper mask. The mask size is the same as the original image. In the glass rod mask, the glass rod portion is set to 1, and the rest is set to 0. In the filter paper mask, the filter paper portion is set to 1, and the rest is set to 0.

[0068] Next, calculate the circumcircle moment of the region formed by 1 in the glass rod mask and filter paper mask respectively; this circumcircle moment corresponds to a region in both the original image and the mask, and extract this region from the original image;

[0069] Finally, resize the extracted portion to match the original size. Figure 1 Size, such as Figure 3The inputs on the left, from top to bottom, are the original image, the region corresponding to the circumscribed moment of the filter paper mask, the region corresponding to the circumscribed moment of the glass rod mask, and the region corresponding to the circumscribed moment of the union of the glass rod mask and the filter paper mask.

[0070] In a preferred embodiment, in S304, the feature map and the label mask gt, two matrices of the same size, are multiplied element-wise at corresponding positions to obtain a new matrix. Here, mask refers to a data mask, and the mask is the same size as the feature map. In the mask, the region of interest is represented by 1, and the remaining positions are represented by 0. It can be observed that... Figure 3 In the image in (d), red represents the part of interest and is represented by 1, while black represents the other part and is represented by 0.

[0071] In a preferred embodiment of the present invention, step S400 is implemented, whereby for each test video, the N candidate images selected in step S100 are processed through steps S301 to S306, ultimately obtaining the classification results for these N images. The classification results of all input images in the time series are statistically analyzed, and the most frequent result is taken as the final judgment result. Specifically, the classification results can be divided into two types: the side where the lower end of the glass rod is in close contact with the three layers of filter paper, and the side where the lower end of the glass rod is not in close contact with the three layers of filter paper. If the final judgment result is the side where the lower end of the glass rod is in close contact with the three layers of filter paper, a score is awarded; if the final judgment result is the side where the lower end of the glass rod is not in close contact with the three layers of filter paper, no score is awarded. This automatically completes the evaluation of the test score, saving significant manpower costs and improving the detection efficiency of physicochemical experiment scores.

[0072] Based on the same inventive concept, other embodiments of the present invention provide a scoring point evaluation system based on strongly supervised target classification, including a time-of-interest (TOO) image set detection module, an FCN module, a single-frame judgment module, and a temporal judgment module. The TOO image set detection module detects experimental operations in the video stream to obtain TOO image sets. The FCN module trains a fully convolutional neural network (FCN) model based on the TOO image sets. The single-frame judgment module performs strongly supervised target classification on the TOO image sets frame by frame based on the trained FCN model to determine whether the lower end of the glass rod is close to the three layers of filter paper. The temporal judgment module performs multi-frame voting based on the judgment results of each frame to determine whether the experimental operation in the video stream receives a score.

[0073] This embodiment is still a classification model, but it differs from traditional classification models in several ways. First, it uses strongly supervised learning, incorporating more labeled information. Second, the purpose of adding labeled information is to help the network quickly focus on the labeled regions, whereas traditional classification models struggle to do so. Third, from a data perspective, traditional classification models require a large amount of training data to focus on the relevant regions, which is difficult to achieve in real-world tasks. Strongly supervised classification models, on the other hand, require far less data than traditional models, highlighting their advantages. Fourth, in terms of model accuracy, the strongly supervised classification model, trained with the same amount of data, achieves significantly higher accuracy than the traditional classification model.

[0074] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.

[0075] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0076] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0077] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0078] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0079] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0080] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for determining whether a filter paper experiment has received a score, characterized in that, include: The experimental operations in the video stream of the filter paper experiment were detected to obtain the image set of the time period of interest. Based on the image set for the time period of interest, a fully convolutional neural network (FCN) model is trained to obtain image labels; Based on the trained fully convolutional neural network (FCN) model, strongly supervised target classification is performed frame by frame on the image set of the time period of interest to obtain the classification result of whether the lower end of the glass rod is close to the three layers of filter paper. Based on the classification results of each frame in the image set of the time of interest, multi-frame voting is performed to determine whether the experimental operation in the video stream scores a point. The pre-trained fully convolutional neural network (FCN) model performs strongly supervised target classification frame-by-frame on the image set for the time period of interest, including: Using a trained fully convolutional neural network (FCN) model, obtain the label mask gt of the input image; Based on the obtained label mask gt, four parts are cropped from the original image: the original image, the filter paper, the glass rod, and the target composed of the filter paper and the glass rod. The images of the four parts are then adjusted to match the size of the original image. The images of the four parts are input into a CNN network to obtain four feature maps; Adjust the label mask gt corresponding to the four feature maps to be the same size as the feature map; multiply the feature map with the corresponding elements of the label mask gt to obtain a new feature map; The four new feature maps are then subjected to mean pooling and max pooling to obtain four feature vectors, which are then concatenated together. The concatenated feature vectors are input into a fully connected layer and a softmax function to obtain the classification result.

2. The method for determining whether a filter paper experiment has received a score according to claim 1, characterized in that, The process of detecting experimental operations in the video stream of the filter paper experiment to obtain a set of images for the time period of interest includes: The image frames from the video stream are input into the detection model to determine whether they contain a region of interest. Each frame of the video stream is detected to obtain multiple images containing the region of interest, forming a set of images for the time period of interest.

3. The method for determining whether a filter paper experiment has received a score according to claim 2, characterized in that, The criteria for determining the region of interest include: the presence of filter paper, the presence of a glass rod, and the intersection of the circumscribed moments of the filter paper and the glass rod.

4. The method for determining whether a filter paper experiment has received a score according to claim 1, characterized in that, The step of training a fully convolutional neural network (FCN) model for obtaining image labels based on the image set for the time period of interest includes: The obtained image set for the time period of interest is divided into a training set and a test set. The images are manually labeled. Each image is divided into three parts: a glass rod, a filter paper, and a background. These three parts constitute the label mask gt of the image. The FCN network was trained and tested using manually labeled training and test sets.

5. The method for determining whether a filter paper experiment has received a score according to claim 1, characterized in that, The process of determining whether an experimental operation in the video stream scores a point by performing multi-frame voting based on the judgment results of each frame includes: The classification results of each frame in the image set for the time period of interest are statistically analyzed. The result with the highest frequency will be used as the final judgment result.

6. The method for determining whether a filter paper experiment has been scored according to claim 1, characterized in that, The classification results include two categories: the side where the lower end of the glass rod is close to the three layers of filter paper and the side where the lower end of the glass rod is not close to the three layers of filter paper.

7. A score-based evaluation system for strongly supervised target classification, used to implement the method for determining whether a filter paper experiment has received a score as described in any one of claims 1-6, characterized in that, include: The module for detecting the time-of-interest image set is used to detect experimental operations in the video stream of the filter paper experiment and obtain the time-of-interest image set. The FCN module trains a fully convolutional neural network (FCN) model to obtain image labels based on the image set of the time period of interest. The single-frame judgment module, based on the trained fully convolutional neural network (FCN) model, performs strongly supervised target classification on the image set of the time period of interest frame by frame to obtain the classification result of whether the lower end of the glass rod is close to the three layers of filter paper; The timing judgment module performs multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point.

8. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-6, or to run the system of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-6, or to run the system of claim 7.

Images