A method and system for determining whether a beaker experiment is scored

Abstract

The application provides a method and system for judging whether a beaker experiment is successful, comprising: obtaining a video stream of the beaker experiment and performing region of interest detection; a plurality of images containing the region of interest obtained by the region of interest detection constitute a picture set of a region of interest period of the beaker experiment; a full convolutional neural network (FCN) model is trained based on the picture set of the region of interest period of the beaker experiment; the picture set of the region of interest period is subjected to strong supervision target classification frame by frame based on the trained FCN model, and a classification result of each frame in the picture set of the region of interest period is obtained; and whether the experiment operation in the video stream is successful is determined based on the classification result of each frame in the picture set of the region of interest period. The data required by the application is greatly reduced, and the application is suitable for video data size in actual experiment operation process; and the target region can be quickly focused on by detecting the region of interest, and the detection precision is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for determining whether a beaker experiment has received a score. Background Technology

[0002] As Shanghai continues to expand its junior high school enrollment, the demand for teachers in physics and chemistry experiments is also increasing every year. To alleviate this problem, the judgment of whether the beaker spout is close to the glass rod in the "filtration" experiment of chemistry can be made by modeling and using artificial intelligence to reduce manual operation.

[0003] In existing technologies, binary classification is performed directly using convolutional neural networks. This method is theoretically feasible, but it still has the following problems: First, the model has difficulty focusing on the regions that should be focused on in the image, resulting in low model accuracy; Second, a large number of samples are required for training, but in real-world scenarios, the amount of data obtained is not large enough to train a traditional classification model. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for determining whether a beaker experiment has received a score.

[0005] According to one aspect of the present invention, a method for determining whether a beaker experiment has been scored is provided, comprising:

[0006] Acquire a video stream of the beaker experiment, and perform region of interest detection on the experimental operation of the video stream. The region of interest detection includes: detecting whether there is a beaker, detecting whether there is a glass rod, and detecting whether the circumscribed moments of the beaker and the glass rod intersect.

[0007] Multiple images containing the region of interest obtained from the region of interest detection constitute the image set of the time period of interest in the beaker experiment; wherein the images of the time period of interest contain at least a beaker and a glass rod;

[0008] A fully convolutional neural network (FCN) model was trained based on the image set of the time period of interest from the beaker experiment.

[0009] Based on the trained fully convolutional neural network (FCN) model, strongly supervised target classification is performed frame by frame on the image set of the time period of interest to obtain the classification result of each frame in the image set of the time period of interest. The classification result refers to the judgment result of whether the beaker spout is close to the glass rod in the image of the time period of interest.

[0010] Based on the classification results of each frame in the image set of the time of interest, multi-frame voting is performed to determine whether the experimental operation in the video stream scores a point. If the spout of the beaker is close to the glass rod, a point is awarded.

[0011] Preferably, the experimental operation on the video stream includes region of interest detection, comprising:

[0012] The image frames of the video stream from the beaker experiment are input into the detection model to determine whether they contain the region of interest.

[0013] Each frame of the video stream is detected to obtain N images containing the region of interest.

[0014] Preferably, the criteria for determining the region of interest include: detecting a beaker, detecting a glass rod, and the circumscribed moments of the beaker and the glass rod intersecting; if all three conditions are met, it is considered a region of interest.

[0015] Preferably, the training of a fully convolutional neural network (FCN) model based on the image set of the time period of interest from the beaker experiment includes:

[0016] The obtained image set of the time period of interest is divided into a training set and a test set. Each image is manually labeled into three parts: glass rod, beaker and background. The label mask gt of the image is composed of these three parts.

[0017] The FCN model was trained and tested using manually labeled training and test sets.

[0018] Preferably, the step of performing strongly supervised target classification frame by frame on the image set of the time period of interest based on the trained fully convolutional neural network (FCN) model to obtain the classification result of each frame in the image set of the time period of interest includes:

[0019] Using a trained fully convolutional neural network (FCN) model, obtain the label mask gt of the input image;

[0020] Based on the obtained label mask gt, four parts are extracted from the original image: the original image, the beaker, the glass rod, and the target composed of the beaker and the glass rod. The four parts of the image are then adjusted to be the same size as the original image.

[0021] The four images are input into a CNN network to obtain four feature maps;

[0022] Adjust the label mask gt corresponding to the four feature maps to be the same size as the feature map; multiply the feature map with the corresponding elements of the label mask gt to obtain a new feature map;

[0023] The four new feature maps are then subjected to mean pooling and max pooling to obtain four feature vectors, which are then concatenated together.

[0024] The concatenated feature vectors are input into a fully connected layer and a softmax function for classification, resulting in the classification results for each frame in the image set for the time period of interest.

[0025] Preferably, the step of performing multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point includes:

[0026] The classification results of each frame of the image set for the time period of interest are statistically analyzed;

[0027] The classification result with the highest frequency is taken as the final judgment result.

[0028] Preferably, the classification result includes two categories: the beaker spout is close to the glass rod and the beaker spout is not close to the glass rod.

[0029] According to a second aspect of the present invention, a score point evaluation system based on strongly supervised target classification is provided, comprising:

[0030] The region of interest detection module acquires the video stream of the beaker experiment and performs region of interest detection on the experimental operation of the video stream. The region of interest detection includes: detecting whether there is a beaker, detecting whether there is a glass rod, and detecting whether the circumscribed moments of the beaker and the glass rod intersect.

[0031] The image set integration module combines multiple images containing the region of interest detected by the region of interest into an image set for the time period of interest in the beaker experiment; wherein the images for the time period of interest contain at least a beaker and a glass rod.

[0032] The FCN training module trains a fully convolutional neural network (FCN) model based on the image set of the time period of interest from the beaker experiment.

[0033] The single-frame judgment module, based on the trained fully convolutional neural network (FCN) model, performs strongly supervised target classification on the image set of the time period of interest frame by frame to obtain the classification result of each frame in the image set of the time period of interest. The classification result refers to the judgment result of whether the beaker spout is close to the glass rod in the image of the time period of interest.

[0034] The timing judgment module performs multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point. If the spout of the beaker is close to the glass rod, a point is awarded.

[0035] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described method for determining whether a beaker experiment has been scored, or to run the above-described system for determining whether a beaker experiment has been scored.

[0036] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method for determining whether a beaker experiment has been scored, or to run the above-described system for determining whether a beaker experiment has been scored.

[0037] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0038] The method and system for determining whether a beaker experiment has been scored in this embodiment of the invention require significantly less data, making it suitable for video data sizes in actual experimental operations; furthermore, by detecting the region of interest, it can quickly focus on the target region, effectively improving detection accuracy.

[0039] The method and system for determining whether a beaker experiment has been scored in this invention embodiment can automatically evaluate whether the experimental operation has been scored, saving a lot of manpower costs and improving the detection efficiency of physicochemical experiment scores. Attached Figure Description

[0040] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 This is a flowchart of a method for determining whether a beaker experiment scores a point in one embodiment of the present invention;

[0042] Figure 2 This is a framework diagram of a method for determining whether a beaker experiment scores a point in one embodiment of the present invention;

[0043] Figure 3 This is a structural diagram of the FCN model in a preferred embodiment of the present invention;

[0044] Figure 4 This is a diagram of the four-stream Mask-CNN structure in a preferred embodiment of the present invention;

[0045] Figure 5 This is a comparison image of the original image and the label mask gt in a preferred embodiment of the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0047] See Figure 1 and Figure 2 This invention provides an embodiment of a method for determining whether a beaker experiment has received a score, comprising:

[0048] S100: Acquire the video stream of the beaker experiment and perform region of interest detection on the experimental operation of the video stream;

[0049] The region of interest detection includes: detecting whether a beaker is present, detecting whether a glass rod is present, and detecting whether the circumscribed moments of the beaker and the glass rod intersect.

[0050] S200, multiple images containing regions of interest obtained from region of interest detection constitute the image set of the time period of interest in the beaker experiment;

[0051] Among them, the images of the time period of interest must contain at least a beaker and a glass rod;

[0052] S300, based on the image set of the time of interest from the beaker experiment, trains a fully convolutional neural network (FCN) model;

[0053] S400, based on the fully convolutional neural network (FCN) model trained in S300, performs strongly supervised target classification on the image set of the time period of interest frame by frame, and obtains the classification result of each frame in the image set of the time period of interest. The classification result refers to the judgment result of whether the beaker spout is close to the glass rod in the image of the time period of interest.

[0054] S500 uses multi-frame voting based on the classification results of each frame in the image set of the time period of interest in S400 to determine whether the experimental operation in the video stream scores a point. If the spout of the beaker is close to the glass rod, it scores a point.

[0055] This embodiment, through the coordination of the above steps, significantly reduces the data required for scoring beaker experiments, making it suitable for video data sizes in actual experimental operations. Furthermore, by detecting the region of interest, it can quickly focus on the target area, effectively improving detection accuracy. The above method and system save substantial labor costs and improve the efficiency of physicochemical experiment score detection.

[0056] In a preferred embodiment of the present invention, step S100 involves inputting the video stream of the beaker experiment into a detection model to detect whether each frame is a region of interest (ROI). If not, the detection continues for the later parts of the video stream; if so, the image of the time period of interest and the location information of the experimental equipment are input into a classification model. Specifically, this includes two steps:

[0057] S101, The image is input into the detection model to determine whether it contains a region of interest;

[0058] S102, perform the operation S101 on each frame of the video stream to obtain N images of interest.

[0059] In a preferred embodiment, the region of interest (ROI) is determined by simultaneously satisfying three conditions: ① a beaker is detected; ② a glass rod is detected; ③ the circumscribed moments of the beaker and the glass rod intersect. These three conditions effectively detect the ROI in a beaker experiment. Existing models can be used for detection.

[0060] In a preferred embodiment of the present invention, step S200 is performed, which involves generating a set of N images of interest for the time period of the beaker experiment obtained in step S100 of the above embodiment. Each image of interest is obtained through the following process:

[0061] First, calculate the union of the circumscribed moments of the beaker and the glass rod. Then, calculate the circumscribed moment of the entire union. The image framed by the circumscribed moments of the union is the candidate image, which is used to input the FCN model.

[0062] See Figure 3 In a preferred embodiment of the present invention, S300 is implemented, and the specific process includes:

[0063] S301, the image set of the N images obtained in S200, representing the time period of interest, is divided into a training set and a test set. These N images are manually labeled, divided into three parts: glass rod, beaker, and background. The label mask gt for the corresponding image is composed of these three parts, as shown below. Figure 5 As shown, the long strip is a glass rod, the serrated edge is a beaker, and the rest is the background; the training set is used to train the model, and the test set is used to verify the training effect of the model.

[0064] S302, use the labeled data from S301 to train the FCN model. The trained FCN model can obtain the label mask gt of the input image.

[0065] See Figure 4 In a preferred embodiment of the present invention, S400 is implemented, and the specific process includes:

[0066] S401 uses the FCN model trained by S300 to predict the label mask gt of an input image; the specific structure of the FCN model can be found in [reference needed]. Figure 4 As shown;

[0067] S402, based on the label mask ground value predicted by S401, extracts four parts from the original image: the original image, the beaker, the glass rod, and the target (including both the beaker and the glass rod). These four parts are then resized to match the size of the original image. Figure 4 (a) Inputs;

[0068] S403, input the four parts of the image from S402 into the CNN network respectively, that is Figure 4 (b) CNN, resulting in 4 feature maps, namely: Figure 4 (c)Convolutional activation tensor;

[0069] S404: Resize the label mask gt corresponding to the four feature maps to the size of the feature maps, and multiply the feature maps by the corresponding elements of the label mask gt to obtain new feature maps. Figure 4 (d) Descriptor selection;

[0070] S405, the four new feature maps obtained in S404 are subjected to mean pooling and max pooling respectively to obtain 1024-dimensional feature vectors. Then, these four 1024-dimensional feature vectors are concatenated together to obtain a 4096-dimensional feature vector.

[0071] S406, finally, the 4096-dimensional feature vector is input into the fully connected layer (fc) and the softmax function to obtain the final classification result.

[0072] In a preferred embodiment, S402 is implemented as follows:

[0073] First, the image input to the FCN model is the original image. The FCN model then produces a glass rod mask and a beaker mask. The mask size is the same as the original image. In the glass rod mask, the glass rod portion is set to 1, and the rest is set to 0. In the beaker mask, the filter paper portion is set to 1, and the rest is set to 0.

[0074] Next, calculate the circumcircle moment of the region formed by 1 in the glass rod mask and beaker mask respectively; this circumcircle moment corresponds to a region in both the original image and the mask, and extract this region from the original image;

[0075] Finally, resize the extracted portion to match the original size. Figure 1 Size, such as Figure 3 The inputs on the left, from top to bottom, are the original image, the region corresponding to the circumscribed moment of the beaker mask, the region corresponding to the circumscribed moment of the glass rod mask, and the region corresponding to the circumscribed moment of the union of the beaker mask and the filter paper mask.

[0076] In a preferred embodiment, in S404, the feature map and the label mask gt, two matrices of the same size, are multiplied element-wise at corresponding positions to obtain a new matrix. Here, mask refers to a data mask, and the mask is the same size as the feature map. In the mask, the region of interest is represented by 1, and the remaining positions are 0. It can be observed that... Figure 4 In the image in (d), red represents the part of interest and is represented by 1, while black represents the other part and is represented by 0.

[0077] In a preferred embodiment of the present invention, step S500 is implemented, whereby steps S401 to S406 are executed for each test video and the N candidate images selected in step S100, ultimately obtaining the classification results for these N images. The classification results of all input images in the time series are statistically analyzed, and the most frequent result is taken as the final judgment result. Specifically, the classification results can be divided into two types: the beaker spout is close to the glass rod, and the beaker spout is not close to the glass rod. If the final judgment result is that the beaker spout is close to the glass rod, a score is awarded; if the final judgment result is that the beaker spout is not close to the glass rod, no score is awarded. This automatically completes the evaluation of the test score, saving significant manpower costs and improving the detection efficiency of physicochemical experiment scores.

[0078] The classification model used in the above embodiments of the present invention differs from traditional classification models in the following ways: First, the above embodiments use strongly supervised learning, i.e., incorporating more labeled information; second, the purpose of adding labeled information is to help the network quickly focus on the labeled regions, while traditional classification models struggle to quickly focus on the regions that should be focused on; third, from a data perspective, traditional classification models require a large amount of training data to focus on the regions of interest, which is difficult to achieve in real-world tasks, while strongly supervised classification models require far less data than traditional models, thus demonstrating the advantages of strongly supervised classification models; fourth, in terms of model accuracy, the strongly supervised classification model, trained with the same amount of data, achieves significantly higher accuracy than traditional classification models. Therefore, the above embodiments of the present invention require significantly less data, making them suitable for video data sizes in actual experimental operations, and by detecting regions of interest, the target region can be quickly identified, effectively improving detection accuracy.

[0079] Based on the same inventive concept, other embodiments of the present invention provide a system for determining whether a beaker experiment has been scored, including a region of interest (ROI) image set detection module, an FCN module, a single-frame judgment module, and a temporal judgment module; wherein: the ROI detection module acquires the video stream of the beaker experiment and performs ROI detection on the experimental operations of the video stream, wherein ROI detection includes: detecting whether a beaker is present, detecting whether a glass rod is present, and detecting whether the circumcircle moments of the beaker and the glass rod intersect. The image set integration module constructs a ROI image set for the beaker experiment from multiple images containing ROI obtained by ROI detection; wherein the ROI images contain at least a beaker and a glass rod. The FCN training module trains a fully convolutional neural network (FCN) model based on the ROI image set of the beaker experiment. The single-frame judgment module performs strongly supervised target classification on the ROI image set frame by frame based on the trained FCN model, obtaining the classification result for each frame in the ROI image set, where the classification result refers to the judgment result of whether the beaker spout is close to the glass rod in the ROI image. The timing judgment module performs multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point. If the spout of the beaker is close to the glass rod, a point is awarded.

[0080] The specific implementation techniques of each module of the system for determining whether a beaker experiment has been scored in the above embodiments of the present invention can be referred to the implementation techniques of the corresponding steps in the previous embodiments of the method for determining whether a beaker experiment has been scored, and will not be repeated here.

[0081] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.

[0082] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDRSDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0083] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0084] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0085] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0086] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0087] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for determining whether a beaker experiment has received a score, characterized in that, include: Acquire a video stream of the beaker experiment, and perform region of interest detection on the experimental operation of the video stream. The region of interest detection includes: detecting whether there is a beaker, detecting whether there is a glass rod, and detecting whether the circumscribed moments of the beaker and the glass rod intersect. Multiple images containing the region of interest obtained by the region of interest detection constitute the image set of the time period of interest in the beaker experiment, wherein the images of the time period of interest contain at least a beaker and a glass rod. A fully convolutional neural network (FCN) model was trained based on the image set of the time period of interest from the beaker experiment. Based on the trained fully convolutional neural network (FCN) model, strongly supervised target classification is performed frame by frame on the image set of the time period of interest to obtain the classification result of each frame in the image set of the time period of interest. The classification result refers to the judgment result of whether the beaker spout is close to the glass rod in the image of the time period of interest. Based on the classification results of each frame in the image set of the time of interest, multi-frame voting is performed to determine whether the experimental operation in the video stream scores a point. If the spout of the beaker is close to the glass rod, a point is awarded. The trained fully convolutional neural network (FCN) model performs strongly supervised target classification frame by frame on the image set for the time period of interest, obtaining the classification result for each frame in the image set for the time period of interest, including: Using a trained fully convolutional neural network (FCN) model, obtain the label mask gt of the input image; Based on the obtained label mask gt, four parts are extracted from the original image: the original image, the beaker, the glass rod, and the target composed of the beaker and the glass rod. The four parts are then adjusted to the same size as the original image. The four images are input into the CNN network to obtain four feature maps; Adjust the label mask gt corresponding to the four feature maps to be the same size as the feature maps; multiply the feature maps with the corresponding elements of the label mask gt to obtain new feature maps; The four new feature maps are then subjected to mean pooling and max pooling to obtain four feature vectors, which are then concatenated together. The concatenated feature vectors are input into a fully connected layer and a softmax function for classification, resulting in the classification results for each frame in the image set for the time period of interest.

2. The method for determining whether a beaker experiment has received a score according to claim 1, characterized in that, The experimental operation on the video stream, which involves region of interest detection, includes: The image frames of the video stream from the beaker experiment are input into the detection model to determine whether they contain the region of interest. Each frame of the video stream is detected to obtain N images containing the region of interest.

3. The method for determining whether a beaker experiment has received a score according to claim 2, characterized in that, The criteria for determining the region of interest include: the detection of a beaker, the detection of a glass rod, and the intersection of the circumscribed moments of the beaker and the glass rod; if all three conditions are met, it is considered a region of interest.

4. The method for determining whether a beaker experiment has received a score according to claim 1, characterized in that, The image set based on the time period of interest from the beaker experiment is used to train a fully convolutional neural network (FCN) model, including: The obtained image set of the time period of interest is divided into a training set and a test set. Each image is manually labeled into three parts: glass rod, beaker and background. The label mask gt of the image is composed of these three parts. The FCN model was trained and tested using manually labeled training and test sets.

5. The method for determining whether a beaker experiment has received a score according to claim 1, characterized in that, The step of performing multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point includes: The classification results of each frame of the image set for the time period of interest are statistically analyzed; The classification result with the highest frequency is taken as the final judgment result.

6. The method for determining whether a beaker experiment has received a score according to claim 1, characterized in that, The classification results include two categories: the beaker spout is close to the glass rod and the beaker spout is not close to the glass rod.

7. A score point evaluation system based on strongly supervised target classification, characterized in that, include: The region of interest detection module acquires the video stream of the beaker experiment and performs region of interest detection on the experimental operation of the video stream. The region of interest detection includes: detecting whether there is a beaker, detecting whether there is a glass rod, and detecting whether the circumscribed moments of the beaker and the glass rod intersect. The image set integration module combines multiple images containing the region of interest detected by the region of interest into an image set for the time period of interest in the beaker experiment; wherein the images for the time period of interest contain at least a beaker and a glass rod. The FCN training module trains a fully convolutional neural network (FCN) model based on the image set of the time period of interest from the beaker experiment. The single-frame judgment module, based on the trained fully convolutional neural network (FCN) model, performs strongly supervised target classification on the image set of the time period of interest frame by frame to obtain the classification result of each frame in the image set of the time period of interest. The classification result refers to the judgment result of whether the beaker spout is close to the glass rod in the image of the time period of interest. The timing judgment module performs multi-frame voting based on the classification results of each frame in the image set of the time period of interest to determine whether the experimental operation in the video stream scores a point. If the spout of the beaker is close to the glass rod, a point is awarded. The trained fully convolutional neural network (FCN) model performs strongly supervised target classification frame by frame on the image set for the time period of interest, obtaining the classification result for each frame in the image set for the time period of interest, including: Using a trained fully convolutional neural network (FCN) model, obtain the label mask gt of the input image; Based on the obtained label mask gt, four parts are extracted from the original image: the original image, the beaker, the glass rod, and the target composed of the beaker and the glass rod. The four parts are then adjusted to the same size as the original image. The four images are input into the CNN network to obtain four feature maps; Adjust the label mask gt corresponding to the four feature maps to be the same size as the feature maps; multiply the feature maps with the corresponding elements of the label mask gt to obtain new feature maps; The four new feature maps are then subjected to mean pooling and max pooling to obtain four feature vectors, which are then concatenated together. The concatenated feature vectors are input into a fully connected layer and a softmax function for classification, resulting in the classification results for each frame in the image set for the time period of interest.

8. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-6, or to run the system of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-6, or to run the system of claim 7.

Images