An experimental scoring method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides an experimental scoring method, device, electronic device, and storage medium, which are applied to the field of image processing technology. The scoring method includes: obtaining an operation video of a person to be scored during the operation of a target experiment; performing object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; according to the obtained position information of each object, identifying whether there is an interaction relationship between each device, and identifying whether there is an interaction relationship between a specified part and each device, to obtain an identification result; for each scoring point of the target experiment, based on the identification result, determining whether the experimental event represented by the scoring point occurs, to obtain a judgment result corresponding to the scoring point; based on the judgment results corresponding to each scoring point, calculating the scoring result of the person to be scored regarding the target experiment. Through this solution, the workload of teachers in scoring students' experimental operations can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an experimental scoring method, device, electronic device and storage medium. Background Art

[0002] Experimental operation is an important means to help students master knowledge content. For example, density measurement is a key point for students in physics learning. Usually, during the teaching process of density measurement, teachers will demonstrate the operation of the object density measurement experiment.

[0003] In addition, in order to help students better understand and remember the learned knowledge content, teachers will require students to complete experimental operations independently and evaluate the standardization of students' experimental operations by scoring the students' experimental operation situations.

[0004] Currently, the method for teachers to score students' experimental operation situations is as follows: When students are conducting experiments, teachers observe the experimental operation process of students beside and score based on what they observe. Obviously, this scoring method requires teachers to participate in the experimental operation process of each student, resulting in a large workload for teachers.

[0005] Then, for experiments such as object density measurement where scoring is affected by the standardization of operation actions, how to reduce the workload of teachers in scoring students' experimental operations is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of the embodiments of the present invention is to provide an experimental scoring method, device, electronic device and storage medium to reduce the workload of teachers in scoring students' experimental operations. The specific technical solutions are as follows:

[0007] In a first aspect, the embodiments of the present invention provide an experimental scoring method, and the method includes:

[0008] Obtain an operation video of a person to be scored when operating a target experiment;

[0009] Perform object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; wherein, each object includes: each device required for the target experiment and a designated part of the person to be scored;

[0010] According to the obtained position information of each object, identify whether there is an interaction relationship between each device, and identify whether there is an interaction relationship between the designated part and each device, to obtain an identification result;

[0011] For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event represented by the scoring point occurs, and obtain the judgment result corresponding to the scoring point;

[0012] Based on the judgment results corresponding to each scoring point, calculate the scoring result of the person to be scored for the target experiment.

[0013] In a second aspect, an embodiment of the present invention provides an experimental scoring device, which includes:

[0014] An acquisition module, configured to acquire the operation video of the person to be scored during the operation of the target experiment;

[0015] A detection module, configured to perform object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; wherein, each object includes: each device required for the target experiment and the designated part of the person to be scored;

[0016] An identification module, configured to identify whether there is an interaction relationship between each device according to the obtained position information of each object, and identify whether there is an interaction relationship between the designated part and each device, and obtain an identification result;

[0017] A judgment module, configured to, for each scoring point of the target experiment, based on the recognition result, judge whether the experimental event represented by the scoring point occurs, and obtain the judgment result corresponding to the scoring point;

[0018] A calculation module, configured to calculate the scoring result of the person to be scored for the target experiment based on the judgment results corresponding to each scoring point.

[0019] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0020] The memory is used to store a computer program;

[0021] The processor is configured to, when executing the program stored on the memory, implement any experimental scoring method.

[0022] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, any experimental scoring method is implemented.

[0023] The embodiment of the present invention also provides a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the above-mentioned experimental scoring methods.

[0024] Advantages of the embodiments of the present invention:

[0025] An experimental scoring method provided by an embodiment of the present invention, when scoring the experimental operation situation of students, does not require a teacher to observe the experimental operation process of the students beside and score based on the observed content. Instead, by obtaining the operation video of the students operating the target experiment, it is determined whether there is an interaction relationship between devices and whether there is an interaction relationship between a specified part and a device based on the operation video. Furthermore, the scoring result of the student for the target experiment is automatically obtained. In this way, the workload of the teacher for scoring the experimental operation of the students can be greatly reduced.

[0026] Of course, when implementing any product or method of the present invention, it is not necessarily required to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following-described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0028] Figure 1 It is a schematic flowchart of an experimental scoring method provided by an embodiment of the present invention;

[0029] Figure 2 It is another schematic flowchart of an experimental scoring method provided by an embodiment of the present invention;

[0030] Figure 3 It is another schematic flowchart of an experimental scoring method provided by an embodiment of the present invention;

[0031] Figure 4 It is another schematic flowchart of an experimental scoring method provided by an embodiment of the present invention;

[0032] Figure 5 It is another schematic flowchart of an experimental scoring method provided by an embodiment of the present invention;

[0033] Figure 6 It is another schematic flowchart of an experimental scoring method provided by an embodiment of the present invention;

[0034] Figure 7 It is a schematic principle diagram of an experimental scoring method provided by an embodiment of the present invention;

[0035] Figure 8 It is another schematic principle diagram of an experimental scoring method provided by an embodiment of the present invention;

[0036] Figure 9 Another schematic diagram of the principle of an experimental scoring method provided by an embodiment of the present invention;

[0037] Figure 10 Schematic diagram of an experimental scoring device provided by an embodiment of the present invention;

[0038] Figure 11 Schematic diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on this application belong to the scope of protection of the present invention.

[0040] The importance of experiments in education has been increasing year by year. When scoring experiments, generally, teachers watch the experimental operations of students or review the videos of students' experimental operations, and manually score. In this way, teachers need to watch the operations of each student one by one and score, which has problems such as low scoring efficiency and scoring fairness. Then, for experiments such as density measurement where the scoring is affected by the standardization of operation actions, how to reduce the workload of teachers in scoring students' experimental operations is an urgent problem to be solved.

[0041] Based on this, the embodiments of the present invention provide an experimental scoring method, device, electronic device and storage medium to reduce the workload of teachers in scoring students' experimental operations.

[0042] Next, an experimental scoring method provided by the present invention will be introduced first.

[0043] Among them, an experimental scoring method provided by an embodiment of the present invention can be applied to an electronic device, and the electronic device can be a terminal device or a server. Exemplarily, the terminal device can be: a smart phone, a tablet computer, a desktop computer, a notebook computer, etc. The present invention does not limit the specific form of the electronic device. An experimental scoring method provided by an embodiment of the present invention can be applied to a scenario of scoring any experiment affected by the standardization of operation actions, for example: a scenario of scoring students' object density measurement experiments, a scenario of scoring students' object mass measurement experiments, or a scenario of scoring students' object volume measurement experiments, etc.

[0044] Specifically, the execution entity of this experimental scoring method can be an experimental scoring device. Exemplarily, when this experimental scoring method is applied to a terminal device, the experimental scoring device can be a functional software running on the terminal device, such as: functional software for experimental scoring; the experimental scoring device can also be a plugin in an existing client, such as: a plugin in a client for student information management. Exemplarily, when this experimental scoring method is applied to a server, the experimental scoring device can be a functional module in the server-side program corresponding to the client that plays videos.

[0045] An experimental scoring method provided by an embodiment of the present invention may include the following steps:

[0046] Obtain the operation video of the person to be scored during the operation of the target experiment;

[0047] Perform object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; wherein, each of the objects includes: each device required for the target experiment and the designated part of the person to be scored;

[0048] According to the obtained position information of each object, identify whether there is an interaction relationship between each device, and identify whether there is an interaction relationship between the designated part and each device, to obtain an identification result;

[0049] For each scoring point of the target experiment, based on the identification result, determine whether the experimental event characterized by this scoring point occurs, to obtain the judgment result corresponding to this scoring point;

[0050] Based on the judgment results corresponding to each scoring point, calculate the scoring result of the person to be scored regarding the target experiment.

[0051] An experimental scoring method provided by an embodiment of the present invention, when scoring the experimental operation situation of students, does not require a teacher to observe the experimental operation process of students beside and score based on the observed content, but obtains the operation video of the students operating the target experiment, and thus determines whether there is an interaction relationship between devices and whether there is an interaction relationship between the designated part and the devices based on this operation video, and further automatically obtains the scoring result of this student regarding this target experiment. In this way, the workload of teachers scoring the experimental operations of students can be greatly reduced.

[0052] Next, with reference to the accompanying drawings, an experimental scoring method provided by an embodiment of the present invention will be introduced exemplarily.

[0053] As Figure 1 shown, an experimental scoring method provided by an embodiment of the present invention may include the following steps:

[0054] S101: Obtain the operation video of the person to be scored during the operation of the target experiment.

[0055] It can be understood that an experimental scoring method provided by an embodiment of the present invention is completed based on the operation video of the person to be scored during the operation of the target experiment. Therefore, the electronic device needs to first obtain the operation video of the person to be scored during the operation of the target experiment. That is, when the person to be scored operates the target experiment, the operation process of the target experiment needs to be monitored.

[0056] It should be noted that the process of the person to be scored operating the target experiment can be monitored in various ways to obtain the operation video. Here, the method of obtaining the operation video is not limited.

[0057] Exemplarily, one or more cameras aligned with the experimental bench can be installed on each experimental bench. When the person to be scored operates the target experiment on the experimental bench, the camera can monitor the process of the person to be scored operating the target experiment to obtain the operation video. Furthermore, the camera can send the operation video to the above-mentioned electronic device in communication with it, or save it to the local storage space. This is all reasonable. It should be noted that when multiple cameras are used, the directions of the multiple cameras can be different. For example: the side, the front, directly above, or diagonally above at a certain angle. This is all reasonable. And the multiple cameras only monitor the person to be scored from different angles, and the experiment, operation time, operation actions, etc. performed by the person to be scored are all the same.

[0058] In addition, when the camera saves the operation video of the person to be scored operating the target experiment in the local storage space of the camera, the electronic device can obtain the operation video in various ways.

[0059] Exemplarily, the method of obtaining the operation video of the person to be scored operating the target experiment in the camera can be: receiving the operation video sent by the camera communicatively connected to the electronic device, sending a video acquisition request to the camera communicatively connected to the electronic device, or obtaining the memory card containing the operation video in the camera, etc., so as to obtain the operation video of the person to be scored operating the target experiment.

[0060] It should be noted that when the electronic device is communicatively connected to the camera, during the monitoring process of the person to be scored, the operation video can be obtained in real time. When the electronic device obtains the operation video, it can execute the experimental scoring method provided by the embodiment of the present invention in real time, so as to realize the real-time scoring of the process of the person to be scored operating the target experiment. In this way, when the person to be scored completes the operation of the target experiment, the scoring result of the person to be scored operating the target experiment can be obtained immediately.

[0061] S102: Perform object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object;

[0062] Among them, each of the objects includes: each device required for the target experiment and the designated part of the person to be scored.

[0063] It can be understood that when performing an experimental operation, it is usually necessary to use experimental devices to complete the experimental operation. Moreover, when scoring the experiment, it is necessary to check whether the operator correctly uses the experimental devices to obtain the score corresponding to the operator for the target experiment. Therefore, it is necessary to perform object detection on each video frame in the operation video of the target experiment to obtain each object existing in the video frame and its position information. It should be noted that each object includes each device required for the person to be scored to operate the target experiment and the designated part of the person to be scored; for example, if the target experiment is an object density measurement experiment, each device required for the target experiment can be: experimental devices such as a balance, weights, tweezers, an object, a dropper, and a graduated cylinder; the designated part of the person to be scored can be: the hand of the person to be scored; of course, for target experiments with some special requirements, the designated part can be: parts such as the shoulder or arm of the person to be scored, which is not limited here.

[0064] Among them, object detection can also be called target detection, which refers to using theories and methods in the fields of image processing and pattern recognition to detect each object existing in the video frame, that is, to determine the semantic category of each object and calibrate the position of the target object in the image. The semantic category can be used to classify each object. For example, an object belonging to what kind of device or an object belonging to the designated part. It can be understood that there can be multiple ways to implement object detection. For example, the video frame can be subjected to object detection through a pre-trained object detection model to obtain the objects existing in the video frame and the object positions. The structure and training process of the object detection model are not limited in the embodiments of the present invention; for example, other image analysis algorithms can also be used to detect the objects in the video frame.

[0065] After obtaining the position information of each object, the subsequent steps can be executed using the position information of each object to implement scoring of the operation of the target experiment.

[0066] S103: According to the obtained position information of each object, identify whether there is an interaction relationship between each device and whether there is an interaction relationship between the designated part and each device to obtain an identification result;

[0067] It should be noted that in some scenarios, even if all the devices required for the target experiment and the designated parts of the person to be scored exist in each video frame of the operation video, it cannot be directly determined that the person to be scored has performed the correct experimental operation on the target experiment. For example, before the experiment starts, the person to be scored is reading the experiment instructions or the experimental code. At this time, although all the devices of the target experiment and the designated parts of the person to be scored exist, the person to be scored has not performed any operations, so it cannot be directly determined that the person to be scored has performed the correct experimental operation, and the operation video of the person to be scored for the target experiment cannot be directly scored.

[0068] Therefore, after obtaining the position information of each object, it is possible to further identify whether there is an interaction relationship between each device, and between each device and the designated part, to obtain an identification result. Specifically, the position information of each object can be used to identify the interaction relationship of each object, and then, by performing subsequent experimental scoring steps, the scoring of the target experiment can be achieved.

[0069] Exemplarily, the method for identifying whether there is an interaction relationship between each device may include: for at least two devices among each device, if based on the position information of the at least two devices, it is identified that the at least two devices have overlapping positions in multiple consecutive frames, it is determined that within the time range of the multiple consecutive frames, the at least two devices have an interaction relationship; otherwise, it is determined that the at least two devices do not have an interaction relationship. It should be noted that under normal circumstances, all the devices of the target experiment will be stored in non-overlapping designated positions. Therefore, for the position information of at least two devices, if the positions of at least two devices overlap in multiple consecutive frames, it can be directly determined that within the time range of the multiple consecutive frames, these at least two devices have an interaction relationship, that is, an interaction has occurred; otherwise, it is determined that there is no interaction relationship. Taking the object density measurement experiment as an example, if in a certain consecutive video frame, the position box of the object is completely contained within the position box of the left tray of the balance, it can be directly determined that the object is located on the left tray of the balance, that is, within the range of the consecutive frames, the object and the balance have an interaction relationship.

[0070] Exemplarily, the method for identifying whether there is an interaction relationship between the specified part and each device includes: If, based on the position information of the specified part and the position information of at least one device, it is identified that there is a position overlap between the specified part and the at least one device in multiple consecutive frames, then the device candidate set corresponding to the first specified frame is used to determine whether there is an interaction relationship between the specified part and the at least one device; wherein, the first specified frame is a video frame in the multiple consecutive frames, and the device candidate set corresponding to the first specified frame includes at least one candidate device, and each candidate device is a device used by the specified part or a device to be used by the specified part within the time range of the first specified frame. The device used by the specified part or the device to be used by the specified part is: a device used by the person to be scored or a device to be used by the person to be scored in order to complete the target experiment within the time range of the first specified frame.

[0071] It should be noted that since the video frame is two-dimensional data, even if it is identified through image analysis that there is a position overlap between the specified part and any device, it is possible that the specified part does not actually contact any device, but the position overlap is caused by the perspective relationship. Therefore, within the time range of multiple consecutive frames, if there is a position overlap between the specified part and at least one device, it is obviously inappropriate to directly determine that there is an interaction relationship between the at least one device and the specified part. Moreover, during the experiment, the specified part of the person to be scored will be in a specific operating posture. At this time, there is no interaction between the specified part of the person to be scored and some devices, while there may be an interaction with some other devices. Therefore, if there is a position overlap between the specified part and at least one device in multiple consecutive frames, it is not directly determined that there is an interaction relationship between the specified part and the at least one device, but the device candidate set corresponding to the first specified frame can be further used to determine whether the specified part interacts with the at least one device. For example: In multiple consecutive frames, the hand overlaps with the weight, the balance, and the forceps. However, only the forceps are in the device candidate set, and it can be determined that there is an interaction relationship between the forceps and the hand, while there is no interaction relationship with the weight and the balance.

[0072] In addition, when the operation video is captured by a single camera, the interaction relationship between the devices or the interaction relationship between the specified part and the devices can be directly determined by whether the devices overlap, or whether the specified part overlaps with the devices and whether the devices are in the device candidate set. When the operation video is captured by multiple cameras, if there is an overlap between the devices or an overlap between the specified part and the devices in one direction, at this time, it is inappropriate to directly determine the interaction relationship between the devices or the interaction relationship between the specified part and the devices. Therefore, it is possible to check whether there is an overlap between the devices or an overlap between the specified part and the devices in the operation video in the same time in other directions, so as to accurately determine the interaction relationship between the devices or the interaction relationship between the specified part and the devices.

[0073] It should be noted that the above descriptions of the interaction relationships between devices and the interaction relationships between specified parts and devices are only examples and should not constitute a limitation to the present invention. The detailed determination method for the interaction relationships will be introduced later and will not be elaborated here.

[0074] S104: For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event represented by this scoring point occurs, and obtain the judgment result corresponding to this scoring point;

[0075] For the target experiment, there may be one or more scoring points in the target experiment. After obtaining the result of the recognized interaction relationship, it is possible to determine whether the experimental event represented by this scoring point occurs according to the recognition result, obtain the judgment result corresponding to this scoring point, and thus execute the subsequent experimental scoring steps according to the judgment result of each scoring point to obtain the scoring result of the target experiment.

[0076] Exemplarily, in one implementation manner, for each scoring point of the target experiment, the method of determining whether the experimental event represented by this scoring point occurs based on the recognition result and obtaining the judgment result corresponding to this scoring point may include: for each scoring point of the target experiment, compare the recognition result with the correct experimental operation corresponding to each scoring point, and determine whether the correct experimental operation corresponding to this scoring point occurs in the recognition result, so as to obtain the corresponding judgment result. For example: in the experiment of measuring the density of an object, there is a scoring point for leveling the balance. If the recognition result includes the correct interaction relationship between devices or the interaction relationship between the specified part and the device corresponding to leveling the balance, the obtained judgment result is: the result of the correct operation corresponding to this target experiment.

[0077] It should be noted that the above description of the judgment result for each scoring point is only an example. The detailed judgment process for each scoring point will be introduced in detail in combination with a specific embodiment later and will not be elaborated here.

[0078] S105: Calculate the scoring result of the person to be scored regarding the target experiment based on the judgment results corresponding to each scoring point;

[0079] After obtaining the judgment result corresponding to each scoring point, the scoring result of the person to be scored regarding the target experiment can be calculated according to the judgment result.

[0080] Exemplarily, calculating the scoring result of the person to be scored regarding the target experiment based on the judgment results corresponding to each scoring point includes: for each scoring point, determining the scoring type corresponding to this scoring point, and based on the scoring type corresponding to this scoring point and the judgment result corresponding to this scoring point, determining the score value corresponding to this scoring point; wherein, the scoring type includes a scoring type or a deduction type; adding up the score values corresponding to each scoring point to obtain the scoring result of the person to be scored regarding the target experiment. For example: for the experiment of measuring the density of an object, for the scoring point of leveling the balance, if the scoring type corresponding to this scoring point is the score corresponding to the correct operation, that is, the scoring type, and the judgment result is the operation corresponding to the correct leveling of the balance, determining the score value corresponding to this scoring point as: the score corresponding to the correct operation corresponding to this scoring point; or, if the scoring type corresponding to this scoring point is the score corresponding to the wrong operation, that is, the deduction type, and the judgment result is the operation corresponding to the wrong leveling of the balance, determining the score value corresponding to this scoring point as: the score corresponding to the wrong operation corresponding to this scoring point; in addition, if the judgment result is the operation corresponding to the partially wrong leveling of the balance, the score value corresponding to this scoring point is: the score corresponding to the partially wrong operation corresponding to this scoring point.

[0081] It should be noted that, after obtaining the judgment result of each scoring point, the scoring result of the entire target experiment can be calculated through the calculation result of each scoring point, or, after obtaining the judgment result of any scoring point, first calculate the scoring result corresponding to this scoring point, and after obtaining the scoring results corresponding to all scoring points, then calculate the scoring result of the entire target experiment. In addition, after obtaining the score values corresponding to each scoring point, the scoring result of the person to be scored regarding the target experiment can also be calculated according to a predetermined weight ratio; this weight can be set on average, or can be set according to the different experimental difficulties of the scoring points. The embodiments of the present invention do not limit the specific calculation order and calculation method.

[0082] By calculating the scoring result of any scoring point, the detailed scores of each scoring point can be displayed in detail. For the person to be scored, it is possible to understand the specific links where points are lost during the experimental operation, which is convenient for the person to be scored to consolidate and improve the experiment or the corresponding knowledge points of the experiment.

[0083] An experimental scoring method provided by an embodiment of the present invention, when scoring the experimental operation situation of students, does not require a teacher to observe the experimental operation process of students beside and score based on the observed content, but by obtaining the operation video of the students operating the target experiment, thereby determining whether there is an interaction relationship between devices and whether there is an interaction relationship between a specified part and a device based on this operation video, and further automatically obtaining the scoring result of this student regarding this target experiment. In this way, the workload of teachers scoring the experimental operations of students can be greatly reduced.

[0084] Optionally, in another embodiment of the present invention, determining whether there is an interaction relationship between the specified part and the at least one device by using the device candidate set corresponding to the first specified frame includes steps A1-A4:

[0085] Step A1: Detect whether the at least one device is in the device candidate set corresponding to the first specified frame to obtain a detection result;

[0086] It can be understood that after detecting that there is a position overlap between the specified part and at least one device, it is necessary to continue to detect whether the at least one device exists in the device candidate set corresponding to the first specified frame. If not, even if there is a position overlap between the specified part and the at least one device, there is no interaction relationship between the two; if it exists, it is necessary to analyze according to the specific situation to determine whether the device has an interaction relationship with the specified part.

[0087] Step A2: If the detection result indicates that none of the at least one device is in the device candidate set, it is determined that there is no interaction relationship between the specified part and the at least one device within the time range of the consecutive multiple frames;

[0088] If the detection result is that none of the at least one device exists in the device candidate set, even if there is a position overlap between the device and the specified part, the determination result is still that there is no interaction relationship between the device and the specified part. For example: within a consecutive frame time range, the hand of the person to be scored overlaps with the measuring cylinder, but in the object density measurement experiment, the hand does not operate the measuring cylinder. At this time, it is necessary to further use the device candidate set to determine whether there is an interaction relationship between the measuring cylinder and the hand. If the measuring cylinder does not exist in the device candidate set, it can be determined that there is no interaction relationship between the hand and the measuring cylinder.

[0089] Step A3: If the detection result indicates that one device is in the device candidate set, it is determined that within the time range of the consecutive multiple frames, there is an interaction relationship between the specified part and one device existing in the device candidate set, and there is no interaction relationship between the specified part and the other devices among the at least one device;

[0090] If the detection result shows that only one device is in the device candidate set, it can be directly determined at this time that there is an interaction relationship between this device and the specified part, and there is no interaction relationship between the specified part and other devices except this device. For example: in an object volume measurement experiment, at least one device includes: weights, a balance, and a graduated cylinder. However, in the first specified frame of the object volume measurement experiment, the corresponding device candidate set is: a dropper, a graduated cylinder, and an object. At this time, only the graduated cylinder is in the device candidate set. It can be determined that within the range of the first specified frame of the object volume measurement experiment, there is an interaction relationship between the hand of the person to be scored and the graduated cylinder, and there is no interaction relationship between the weights and the balance and the hand of the person to be scored.

[0091] Step A4: If the detection result indicates that multiple devices are in the device candidate set, among the multiple devices, identify the target device that is closest to the specified part, and determine that within the time range of this continuous multiple frames, there is an interaction relationship between the specified part and the target device, and there is no interaction relationship between the specified part and other devices in the at least one device.

[0092] If the detection result is that multiple devices are in the device candidate set, it is obviously inappropriate to directly determine at this time that all the multiple devices in the existing device candidate set have an interaction relationship with the specified part. Moreover, if a device has an interaction relationship with the specified part, the distance between the two is the smallest compared to the distance between the specified part and other devices. Therefore, the target device interacting with the specified part can be further determined through the distance relationship. For example: during an object density measurement experiment, the specified part is the hand, and the multiple devices detected in the device candidate set are: a dropper and a graduated cylinder. It is obviously inappropriate to directly determine that there is an interaction relationship between the hand and both the graduated cylinder and the dropper within the range of the first specified frame. Moreover, if a device is the closest to the hand, usually there is an interaction relationship between this device and the hand. Therefore, within the continuous frame time range of the first specified frame, the device closest to the hand can be determined. When the device closest to the hand is the dropper, it can be determined at this time that there is an interaction relationship between the hand and the dropper, and there is no interaction relationship between the hand and other devices in the at least one device.

[0093] Optionally, the identifying the target device that is closest to the specified part among the multiple devices includes:

[0094] Based on the depth information of the multiple devices in the second specified frame, identify the target device that is closest to the specified part among the multiple devices;

[0095] The second specified frame is a video frame in this continuous multiple frames.

[0096] The distance between the specified part and multiple devices can be determined by the depth information of the multiple devices, and this depth information is the distance information of the multiple devices from the camera. After obtaining the depth information of the multiple devices, based on the subsequent steps, the target device closest to the specified part can be determined.

[0097] Exemplarily, the identifying, among the multiple devices, the target device closest to the specified part based on the depth information of the multiple devices in the second specified frame includes:

[0098] Based on the depth information of the multiple devices in the second specified frame, select the depth information representing the minimum depth as the depth information to be utilized;

[0099] Determine the device with the depth information to be utilized as the target device closest to the specified part.

[0100] Generally speaking, the specified part of the person to be scored is in the position closest to the camera relative to the multiple devices. Therefore, after obtaining the depth information of the multiple devices, the depth information representing the minimum depth can be selected as the depth information to be utilized, and the device corresponding to this depth information to be utilized can be the target device closest to the specified part. That is, the device with the minimum distance from the camera is the target device with the minimum distance from the specified part of the person to be scored.

[0101] It should be noted that any other method that can determine the target device closest to the specified part is applicable to the present invention and is not limited herein.

[0102] Among them, the method for determining the depth information of any object in the second specified frame includes:

[0103] Determine the depth information of each pixel point in the second specified frame;

[0104] Based on the depth information of the pixel points in the target area in the second specified frame, determine the depth information of the any object in the second specified frame;

[0105] Among them, the target area is the area represented by the position information of the any object in the second specified frame.

[0106] Exemplarily, in one implementation, a monocular depth estimation model can be used to determine the depth information of each pixel point in the second specified frame. After obtaining the depth information of each pixel point, the object corresponding to each pixel point can be determined according to the regional positions of the objects detected by object detection, that is, the target regions. For example, the objects detected by object detection are: the hand of the person to be scored, the dropper, and the graduated cylinder. For the graduated cylinder, the position information of the graduated cylinder obtained by object detection can be used to determine which pixel points in the second specified frame belong to the graduated cylinder, so as to determine the depth information of the graduated cylinder according to the depth information of the pixel points belonging to the graduated cylinder. Among them, the monocular depth estimation model can be a model trained using sample images and the true depth information of each pixel point in the sample images. The specific model structure and training process of the monocular depth estimation model in the embodiments of the present invention are not limited. In addition, any implementation that can determine the depth information of each pixel point in the second specified frame can be applied to the embodiments of the present invention.

[0107] Moreover, the depth information can be obtained by using the monocular depth estimation model to perform depth estimation on each object in each video frame while performing object detection on each video frame in the operation video, or can be obtained subsequently, which is not limited herein.

[0108] By the method of determining the interaction relationship through the device candidate set, it is further determined whether each device is in the set composed of the devices used at the specified part or the devices to be used at the specified part, and for various possible detection results, it can be determined that there is an interaction relationship between the specified part and at least one of the devices, so as to more accurately determine the interaction relationship between the specified part and the device.

[0109] Optionally, in another embodiment of the present invention, the construction method of the device candidate set corresponding to the first specified frame includes:

[0110] Determine the operation state of the specified part in the first specified frame as the target operation state;

[0111] Search for the devices corresponding to the target operation state from the pre-established mapping relationships between the various operation states of the specified part and the devices to obtain candidate devices; the devices corresponding to each operation state are the devices used or to be used by the specified part when in this operation state;

[0112] Use the obtained candidate devices to construct the device candidate set corresponding to the first specified frame.

[0113] It should be noted that the device candidate set corresponding to the first specified frame can be pre-constructed. When it is necessary to use this device candidate set to determine the interaction relationship between the specified part and each of the at least one device, it can be directly queried. Since this device candidate set is a set for the specified part, therefore, the operation state of the specified part in the first specified frame can be first determined as the target operation state, and then the candidate devices corresponding to the target operation state can be searched from the mapping relationship between the target operation state and the devices. After that, the obtained candidate devices can be used to construct the device candidate set corresponding to the first specified frame.

[0114] Among them, there can be multiple operation states of the specified part, and different mapping relationships between the operation states and the devices can be preset according to experience. The operation state of the specified part can be recognized by an image classification model. Specifically, it can be: input the video frame marked with the position information of the specified part into the pre-trained image classification model for recognition to obtain the operation state of the specified part in this video frame. Taking the hand as an example, if the target experiment is an object density measurement experiment, at this time, the operation states of the hand can include: no operation state, the state of holding tweezers with the hand, the state of holding a dropper with the hand, or the state of holding a graduated cylinder with the hand, etc.

[0115] Exemplarily, in one implementation, the image classification model can recognize the probability that the specified part belongs to each operation state, and the maximum value among the probabilities can be used to determine the operation state of the specified part. Taking the hand as an example, the result recognized by the image classification model is: the probability of belonging to the state of holding tweezers with the hand is 90%, and the probability of belonging to the state of holding a dropper with the hand is 10%; the operation state with the largest probability can be determined as the operation state of the hand, that is, the hand is in the operation state of holding tweezers; of course, it is not limited to this.

[0116] By constructing the device candidate set according to the operation state of the specified part, when constructing the candidate set, various operation states of the hand can be comprehensively and accurately recognized, and the device candidate set can be determined by using the devices corresponding to the possible operation states of the hand, which can avoid incorrect judgment results in the judgment of the interaction relationship, thereby improving the accuracy of the subsequent experiment scoring.

[0117] Optionally, in another embodiment of the present invention, the target experiment is: an object density measurement experiment, and the specified part is the hand; the scoring points of the target experiment include: the first scoring point, the second scoring point, the third scoring point, the fourth scoring point, and the fifth scoring point;

[0118] Among them, the experimental event represented by the first scoring point includes: adjusting the balance nut of the balance to make the balance balanced;

[0119] The experimental events represented by the second scoring point include: the object is located on the left side of the balance, and weights and the rider are adjusted to make the balance horizontally balanced;

[0120] The experimental events represented by the third scoring point include: during the experimental events of the second scoring point, the balance nut is not adjusted;

[0121] The experimental events represented by the fourth scoring point include: using a dropper, a graduated cylinder, a beaker and water to measure the volume of the object;

[0122] The experimental events represented by the fifth scoring point include: using a rag to wipe the object and putting the used devices back in place.

[0123] It should be noted that for the object density measurement experiment, an operation video of the person to be scored during the object density measurement experiment can be obtained, and object detection is performed on each video frame in the video to obtain the position information of each object and each object in the video frame. Each object is: each device required for the object density measurement experiment and the designated part of the person to be scored. Each device may include: tweezers, weights, a balance, an object, a graduated cylinder, a dropper and water, etc. The designated part of the person to be scored may be the hand of the person to be scored. Then, using the position information of each object, the interaction relationships between each device and between each device and the hand of the person to be scored can be identified to obtain an identification result. For example: there is an interaction relationship between the hand and the tweezers, an interaction relationship between the object and the balance, or an interaction relationship between the hand and the dropper, etc.

[0124] Next, when the target experiment is the object density measurement experiment, step S104 in an experimental scoring method provided by an embodiment of the present invention will be specifically introduced.

[0125] For the first scoring point, as Figure 2 shown, the above step S104 may include: S201 - S202;

[0126] S201: For the first scoring point of the target experiment, based on the identification result, detect whether there is an interaction relationship between the hand and the balance nut of the balance within a continuous multi-frame time range, and based on the position information of the scale of the balance in these continuous multi-frames, identify whether the pointer in the scale is located at the central scale line position;

[0127] For the first scoring point, the various objects present in the obtained operation video can be: the balance and the hands of the person to be scored. When the recognition result is obtained using the positional relationship between the hands and the balance, since the first scoring point is the balance point for leveling the balance, if the person to be scored is to complete the operation corresponding to the first scoring point at this time, it is necessary to use the hands to adjust the balance nuts on both sides of the balance to make the balance balanced. Therefore, based on the recognized result, it can be detected whether there is an interaction relationship between the hands and the balance nuts on both sides of the balance within a continuous multi-frame time range, that is, the hands adjust the balance nuts on both sides of the balance, and based on the position information of the balance scale in these continuous multi-frames, it is recognized whether the pointer on the scale is located at the central scale line position, that is, it is recognized whether the balance is in a balanced state. Thus, according to the detection result, the judgment result corresponding to the first scoring point can be obtained.

[0128] S202: If all the detection results are yes, it is determined that the experimental event represented by this first scoring point has occurred, and the judgment result corresponding to this first scoring point is obtained;

[0129] If all the detection results are yes, that is, the person to be scored uses the hands to adjust the balance nuts of the balance to make the balance balanced, it can be determined that the operation of the person to be scored is the experimental event represented by the first scoring point, and the judgment result corresponding to the first scoring point is obtained. The judgment result can be: the correct experimental operation for the first scoring point. The judgment result corresponding to this first scoring point can be used to determine the score corresponding to the first scoring point.

[0130] For the second scoring point, as Figure 3 shown, the above step S104 may include: S301 - S302;

[0131] S301: For the second scoring point of the target experiment, based on the recognition result, it is detected whether there is an interaction relationship between the object and the left tray of the balance within a first continuous multi-frame time range, and it is detected whether there is an interaction relationship between the hand and the forceps, between the forceps and the weights, between the weights and the right tray of the balance within a second continuous multi-frame time range, and it is detected whether there is an interaction relationship between the hand and the forceps, between the forceps and the balance rider within a third continuous multi-frame time range, and based on the position information of the balance scale in these third continuous multi-frames, it is detected whether the pointer on the scale is located at the central scale line position;

[0132] Among them, the first continuous multi-frame is any continuous multi-frame, the start time of the second continuous multi-frame is not earlier than the start time of the first continuous multi-frame, and the start time of the third continuous multi-frame is not earlier than the start time of the second continuous multi-frame.

[0133] For the second scoring point, the various objects existing in the obtained operation video can be: a balance, tweezers, weights, an object, and the hand of the person to be scored. After the person to be scored adjusts the balance nut of the balance to make the balance balanced, the mass of the object can be measured using the balance first. When using the balance to measure the mass of the object, following the operation sequence of the experiment, first place the object on the left tray of the balance, and then the person to be scored can use the tweezers to place the weights on the right tray of the balance to make the balance approximately balanced. Finally, the person to be scored can use the tweezers to adjust the rider in the balance to make the balance balanced. According to the readings of the weights and the rider, the mass of the object can be measured. Therefore, it is possible to first detect whether there is an interaction relationship between the object and the left tray of the balance within the time range of the first consecutive multiple frames, and then detect whether there is an interaction relationship between the hand and the tweezers, between the tweezers and the weights, and between the weights and the right tray of the balance within the time range of the second consecutive multiple frames, that is, the process in which the person to be scored uses the tweezers to put the weights into the right tray through the hand. Finally, within the time range of the third consecutive multiple frames, detect whether there is an interaction relationship between the hand and the tweezers, between the tweezers and the rider, and whether the pointer of the balance scale is located at the central scale line position, that is, the person to be scored adjusts the rider to make the balance balanced. Finally, the mass of the object can be obtained as m.

[0134] It should be noted that when using the balance, the object should be placed on the left tray of the balance, and the weights should be placed on the right tray of the balance, otherwise it will cause measurement errors. Also, during the process of measuring the mass of the object, the hand of the person to be scored should not directly contact the weights or the rider, but can use the tweezers for operation, otherwise it will cause inaccurate weight mass or unclear rider scale lines, resulting in errors in measuring the mass of the object.

[0135] S302: If all the detection results are yes, determine that the experimental event characterized by the second scoring point has occurred, and obtain the judgment result corresponding to the second scoring point;

[0136] If all the detection results are yes, that is, the person to be scored interacts with each device through the hand, making the object located on the left side of the balance, the weights located on the right side of the balance, and the balance balanced, thereby measuring the mass of the object. Then it can be determined that the operation of the person to be scored is the experimental event characterized by the second scoring point, and obtain the judgment result corresponding to the second scoring point. The judgment result can be: the correct experimental operation for the second scoring point. The judgment result corresponding to the second scoring point can be used to determine the score corresponding to the second scoring point.

[0137] For the third scoring point, as Figure 4 shown, the above step S104 can include: S401 - S402;

[0138] S401: For the third scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the balance nut within the time range of a specified number of consecutive frames.

[0139] Among them, the starting time point of the specified number of consecutive frames is later than the starting time point of the first consecutive frames and earlier than the ending time point of the third consecutive frames. Within the time range of the first consecutive frames, there is an interaction relationship between the object and the left tray of the balance. Within the time range of the third consecutive frames, there is an interaction relationship between the hand and the rider of the balance.

[0140] For the third scoring point, since the balance has been leveled at the first scoring point, during the process of measuring the mass of the object at the second scoring point, that is, from the starting time of the first consecutive frames to the ending time of the third consecutive frames, the balance nut cannot be adjusted, otherwise it will cause an error in the measured mass of the object. Therefore, within the time range corresponding to the second scoring point, the hands of the person to be scored cannot touch the balance nuts on both sides of the balance. It can be detected whether there is an interaction relationship between the hand and the balance nut within the time range of a specified number of consecutive frames, that is, whether the hand adjusts the balance nut. Thus, according to the detection result, the judgment result corresponding to the third scoring point can be obtained.

[0141] S402: If the detection result is negative, determine that the experimental event of the third scoring point has occurred, and obtain the judgment result corresponding to the third scoring point.

[0142] If the detection result is negative, that is, the hand does not adjust the balance nuts on both sides of the balance, it can be determined that the operation of the person to be scored is the experimental event represented by the third scoring point, and the judgment result corresponding to the second scoring point can be obtained. The judgment result can be: the correct experimental operation for the third scoring point. The score corresponding to the third scoring point can be determined by using the judgment result corresponding to the third scoring point.

[0143] For the fourth scoring point, as Figure 5 shown, the above step S104 may include: S501 - S502;

[0144] S501: For the fourth scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the dropper within the time range of a consecutive number of frames, and detect whether there is an interaction relationship between the dropper and the measuring cylinder within the time range of a consecutive number of frames.

[0145] For the fourth scoring point, the various objects existing in the obtained operation video can be: a dropper, a graduated cylinder, an object, a beaker, and the hand of the person to be scored. After the mass of the object is measured, the volume of the object can be measured, and then based on the density calculation formula, the ratio of the mass to the volume is used as the density of the object. When measuring the volume of the object, usually the volume of water overflowed by the object is used as the volume of the object. First, an appropriate amount of water needs to be added to the graduated cylinder using a beaker and a dropper, that is, a large amount of water is first poured into the graduated cylinder using the beaker, and then a small amount of water is added using the dropper. At this time, there is an appropriate amount of water in the graduated cylinder; after the liquid level stabilizes, read the liquid level reading V1 of the water before the object is put in. Then, put the object into the graduated cylinder. After the liquid level stabilizes, read the total volume V2 of the water and the object. The volume of the object can be obtained by using the difference between the two: V2 - V1. Then the density of the object can be m / (V2 - V1). The so-called appropriate amount of water means that after the object is put into the graduated cylinder, the water can submerge the object, and the total volume of the water and the object after submerging does not exceed the scale line of the graduated cylinder.

[0146] It should be noted that when adding water to the graduated cylinder using a dropper, the person to be scored needs to use the hand to add an appropriate amount of water to the graduated cylinder using the dropper. At this time, if the interaction relationship between the hand and the dropper and the interaction relationship between the dropper and the graduated cylinder are detected, it can be detected whether the person to be scored has performed the corresponding operation for the experimental event of the fourth scoring point. In addition, after detecting the time range of multiple consecutive frames corresponding to the fourth scoring point, in the order of video playback, it can continue to be detected whether there is an interaction relationship between the object and the graduated cylinder within another time range of multiple consecutive frames, that is, it is detected whether the object is put into the graduated cylinder.

[0147] S502: If the detection result is yes, it is determined that the experimental event represented by the fourth scoring point occurs, and the judgment result corresponding to the fourth scoring point is obtained;

[0148] If the detection result is yes, that is, the person to be scored uses the hand to use the dropper to add an appropriate amount of water to the graduated cylinder to measure the volume of the object, it can be determined that the operation of the person to be scored is the experimental event represented by the fourth scoring point, and the judgment result corresponding to the fourth scoring point is obtained. The judgment result can be: the correct experimental operation for the fourth scoring point. The judgment result corresponding to this fourth scoring point can be used to determine the score corresponding to the fourth scoring point.

[0149] For the fifth scoring point, as Figure 6 shown, the above step S104 may include: S601 - S602;

[0150] S601: For the fifth scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the rag within a time range of a continuous plurality of frames, and detect whether there is an interaction relationship between the rag and the object within a time range of a continuous plurality of frames; and, detect whether the positions of each device are at predetermined positions in the video frames after the time range in which the rag and the object have an interaction relationship;

[0151] For the fifth scoring point, when the density of the object is calculated, the experiment is completed. At this time, the experimental devices need to be sorted out so that the experimental devices are in the positions at the start of the experiment. For the measured object, the water on the surface of the object needs to be wiped with a rag. For other experimental devices used, they can be directly returned to their positions. Therefore, it is possible to detect whether there is an interaction relationship between the hand and the rag, whether there is an interaction relationship between the rag and the object, and whether each device is at a predetermined position within a time range of a continuous plurality of frames, so as to detect whether there is an experimental event for the fifth scoring point in the operation of the person to be scored.

[0152] S602: If all the detection results are yes, determine that the experimental event represented by the fifth scoring point has occurred, and obtain the judgment result corresponding to the fifth scoring point;

[0153] If all the detection results are yes, that is, the person to be scored uses the hand to wipe the object with a rag and returns the used experimental devices to their positions, it can be determined that the operation of the person to be scored is the experimental event represented by the fifth scoring point, and the judgment result corresponding to the fifth scoring point is obtained. The judgment result can be: correct experimental operation for the fifth scoring point. The score corresponding to the fifth scoring point can be determined using the judgment result corresponding to the fifth scoring point.

[0154] By determining the judgment result corresponding to each scoring point, the operation of each scoring point of the person to be scored can be identified in detail, and for the subsequent scoring steps, the judgment result can be directly used to quickly calculate the scoring result of the target experiment.

[0155] Next, a specific embodiment is used to introduce in detail an experimental scoring method provided by an embodiment of the present invention.

[0156] As Figure 7 shown, the experimental scoring method may include the following steps:

[0157] Two-way video inflow: The scenario of this embodiment is: Two cameras are installed above each experimental table to obtain the video in the top view direction and the side view direction of the student's experimental operation. The two-way video is the video of the operation target experiment of the person to be scored based on the top view and the side view directions, corresponding to the operation video of obtaining the operation target experiment of the person to be scored described above.

[0158] Target perception: The input of the target perception module is two synchronized videos from the top view and the side view, that is, the operation videos in two directions at the same time. The target perception module contains multiple models such as image classification, object detection, semantic segmentation, and monocular depth estimation. After inputting the two video frames, information such as the positions, sizes, angles, and depths of the operator and various experimental devices in the image can be obtained. Corresponding to the above, object detection is performed on each video frame in the operation video to obtain each object and its position information.

[0159] Taking the experiment of measuring the density of an object as an example, the object detection model can analyze the two videos to obtain the position information of the main experimental instruments such as the balance (including components such as trays, riders, and dials), weights, stones, tweezers, measuring cylinders, beakers, droppers, and rags from two perspectives, as well as the position information of the operator's hands, eyes, and head. Based on the detected position of the hand, the image classification model can further distinguish the state of the hand, such as: the hand holding tweezers, the hand holding a rag, the hand holding a dropper, etc., to obtain candidate instruments and construct a set of instrument candidates, corresponding to the above-mentioned set of device candidates. Based on the position of the dial detected from the top view, the semantic segmentation model can further obtain the position of the pointer and the position of the central scale line of the dial. Based on the detection result from the top view, the monocular depth estimation model can further obtain the distance of each detected target relative to the camera.

[0160] Interaction relationship construction: Based on the continuous frame results output by the target perception module, the interaction relationship construction module establishes and determines the interaction relationships between the hand and the instrument, and between the instruments, and can further perceive the experimental link where the current picture is located in combination with the specific characteristics of different experiments. Corresponding to the above step of identifying whether there is an interaction relationship between each device based on the obtained position information of each object, and identifying whether there is an interaction relationship between the specified part and each device to obtain the recognition result.

[0161] The interaction relationship between instruments is mainly judged according to whether there is an overlap in the spatial positions of the two instruments, corresponding to the above-mentioned recognition method of whether there is an interaction relationship between each device. For example: within the range of continuous frames, the position box of the stone is completely contained in the position box of the left tray of the balance, then it can be regarded as the stone being placed in the balance.

[0162] Experimental operation recognition: Based on the results of the interaction relationship construction module and combined with the scoring rules of specific experiments, the experimental operation recognition module can realize the recognition of the whole operation state. It can be flexibly applied and designed according to the characteristics of each experiment and the changes in the exam questions. Corresponding to the above step of, for each scoring point of the target experiment, based on the recognition result, judging whether the experimental time characterized by this scoring point has occurred to obtain the judgment result corresponding to this scoring point.

[0163] Comprehensive evaluation: The comprehensive evaluation module can score each test point according to the scoring rules based on the results of the experimental operation recognition module, corresponding to the judgment results corresponding to each scoring point above, and calculate the scoring results of the person to be scored for the target experiment. Moreover, the comprehensive evaluation module supports outputting all-round evaluation indicators such as the final scores of students, points deducted and reasons, overall experiment completion, proficiency, etc. In addition, when used in daily experimental teaching, this module can give real-time experimental operation guidance and reminders for incorrect experimental operations. At the same time, it can record and trace the experimental process to improve the teaching quality and efficiency.

[0164] Output scoring results: For the results of the comprehensive evaluation module, the scoring results of the overall experiment of the student and the scoring results corresponding to each scoring point can be output.

[0165] The construction and determination method of the interaction relationship between the hand and the instrument, and the experimental operation recognition module will be specifically introduced in the following embodiments and will not be elaborated here.

[0166] An experimental scoring method provided by an embodiment of the present invention supports various types of experimental instruments by comprehensively applying various technologies such as artificial intelligence. It can identify key experimental operations through modular design to achieve intelligent scoring of multiple test points. At the same time, the test points and scoring requirements can be freely adjusted according to needs, with strong versatility and being easy to promote. Moreover, the entire process from video input to experimental intelligent scoring does not require manual intervention, greatly reducing the scoring burden on teachers. And the machine scoring uses the same standard, reducing the differences between the scores given by different people and ensuring the fairness of scoring. It can meet the regionalized needs under different regional conditions and give play to the energy efficiency of artificial intelligence technology in promoting learning through examinations in the education industry.

[0167] Next, a specific embodiment will be combined to specifically introduce the determination of the interaction relationship between the hand and the instrument in an experimental scoring method provided by an embodiment of the present invention.

[0168] As Figure 8 shown, since the construction and judgment of the interaction relationship between the hand and the instrument are relatively complex, it is first necessary to determine the spatial position of the hand and the instrument, that is, obtain the position information of the specified part and at least one device, and judge whether there is an overlap in the spatial position of the hand and the instrument, that is, identify whether there is a position overlap between the specified part and at least one device; if there is an overlap, further judge whether the instrument is in the instrument candidate set classified by the hand. Corresponding to the above device candidate set, if there is only one instrument in the candidate set, it can be directly determined that there is an interaction between the instrument and the hand; if there is no overlap or the instrument is not in the candidate set of the instrument classified by the hand, it is determined that the instrument has nothing to do with the hand.

[0169] If multiple instruments overlap with the hand simultaneously and are in the instrument candidate set, the distance of each instrument relative to the camera can be further obtained according to the monocular depth estimation model, and combined with the characteristics of different experiments, comprehensively determine which instrument the hand interacts with, that is, perform the judgment step of the interaction relationship between the specified part and at least one device as described above; for example: judge whether the depth distance is the closest, and take the instrument with the closest depth distance as the instrument interacting with the hand. Generally speaking, determine the instrument closer to the camera as the interaction object. For example: when a student uses tweezers to add weights to a tray, the hand will overlap with instruments such as tweezers, weight balance trays, etc. at the same time, but the tweezers are the closest to the camera, so it can be determined that the hand interacts with the tweezers. It can be understood that during normal operation of the experiment, generally there is one instrument with the closest depth distance to the hand, and of course, multiple instruments may also be used simultaneously. For other instruments with a depth distance that is not the closest, it can be determined that the instrument has nothing to do with the hand.

[0170] The interaction relationship construction module establishes the interaction relationships between the hand and instruments, and between instruments, which is applicable to various middle school experiments such as physics and chemistry, especially applicable to experiments that emphasize the standardization of students' operation actions, and on this basis, quickly identify experiment operations and intelligently score.

[0171] Next, in combination with a specific embodiment, taking the measurement of the density of a stone as an example, introduce the realization of the full-process operation state recognition based on the interaction relationship and the position information of each object, combined with the specific experiment scoring rules.

[0172] As Figure 9 shown, the experiment operation recognition module used to realize the full-process operation state recognition may include: a reading and recording sub-module, an instrument position monitoring sub-module, and a key operation recognition sub-module.

[0173] The reading and recording sub-module can take readings and record for the distance of the balance pointer, the recording of the stone mass, and the measurement of the volume of the graduated cylinder. The distance of the balance pointer is to calculate the distance from the pointer on the balance scale to the central scale line of the scale. When this distance is less than the threshold for consecutive frames, it is considered that the balance is in a balanced state. The recording of the stone mass is when the stone is placed on the left tray of the balance, record the value of the weights placed on the right tray of the balance, and calculate the value of the rider on the scale. The sum of the weight value and the rider value is the mass of the stone. When the balance is in a balanced state, the recorded stone mass is the final stone mass. The measurement of the volume of the graduated cylinder is in the side-shot perspective, and according to the position of the liquid level in the graduated cylinder and the main scale position of the graduated cylinder, calculate the volume value. Among them, it is necessary to record the initial volume value V1 before the stone is put in and the total volume value V2 when the stone is stably placed in the graduated cylinder. The final volume of the stone is V2 - V1. The readings recorded by this sub-module can be used to further determine the accuracy of the results obtained from the experiment of measuring the density of an object.

[0174] The instrument position monitoring sub-module can monitor the taking and placing back of instruments and the stable placement of stones. The taking and placing back of instruments means judging whether the main measuring instruments such as the balance, weight box, and graduated cylinder are in the specified operating positions, and whether they are restored after the experiment ends. The stable placement of stones means judging whether the experiment is in the weighing state or the volume measuring state according to the spatial interaction relationship between the stone and the left tray of the balance and the graduated cylinder in consecutive frames. The functions implemented by the instrument position monitoring sub-module correspond to the above: during the experiment, when there is an interaction relationship between the specified part and the device or between devices, the posture of the specified part and the position of the device. The instrument position detected by this sub-module can be used to judge the standard degree of the operation of the target experiment by the person to be scored.

[0175] The key operation recognition sub-module can recognize key operations such as adjusting the balance nut, using a dropper, and wiping the stone with a rag. Among them, adjusting the balance nut means judging whether the student adjusts the balance nuts on both sides of the balance with the hand before and during weighing, and comprehensively judging by detecting the interaction between the hand and the trays on both sides of the balance in consecutive frames and combining the change range of the pointer distance on the scale. Among them, the use of the dropper means that when the hand interacts with the dropper and the graduated cylinder interacts with the dropper, it is regarded as the student adding water to the graduated cylinder with the dropper; at the same time, according to the size and angle of the detection frame of the dropper, it is judged whether the student has illegal operations such as tilting and dripping or inserting into the mouth of the graduated cylinder. Among them, wiping the stone with a rag means that when the hand interacts with the rag and the stone interacts with the rag in consecutive frames, it is regarded as the student using the rag to wipe the stone. The functions implemented by the key operation recognition sub-module correspond to the above: for each scoring point of the target experiment, based on the recognition result, it is judged whether the experimental event corresponding to this scoring point occurs. The experimental operations recorded by this sub-module can be used to judge whether the person to be scored has carried out the experimental events corresponding to each scoring point and to accurately score.

[0176] The experimental operation recognition module can record, detect, and recognize the readings, instrument positions, and key operations of the operator during the experiment, and can accurately record the operation data and operation actions of the operator for each scoring point during the experimental operation process, facilitating subsequent experimental scoring for information such as key operations, instrument positions, and readings. Moreover, the person to be scored can use this recording result to improve the target experiment and consolidate the knowledge points corresponding to the target experiment.

[0177] Based on the above experimental scoring method, an embodiment of the present invention also provides an experimental scoring device, as Figure 10 shown. This device includes:

[0178] An acquisition module 1010, configured to acquire the operation video of the person to be scored when operating the target experiment;

[0179] The detection module 1020 is configured to perform object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; wherein, each object includes: each device required for the target experiment and the designated part of the person to be scored.

[0180] The recognition module 1030 is configured to identify whether there is an interaction relationship between each device according to the position information of each obtained object, and to identify whether there is an interaction relationship between the designated part and each device, so as to obtain a recognition result.

[0181] The judgment module 1040 is configured to, for each scoring point of the target experiment, based on the recognition result, judge whether the experimental event characterized by the scoring point occurs, so as to obtain the judgment result corresponding to the scoring point.

[0182] The calculation module 1050 is configured to calculate the scoring result of the person to be scored regarding the target experiment based on the judgment results corresponding to each scoring point.

[0183] An experimental scoring method provided by an embodiment of the present invention, when scoring the experimental operation situation of students, does not require a teacher to observe the experimental operation process of students beside and score based on the observed content, but obtains the operation video of the students operating the target experiment, and then determines whether there is an interaction relationship between devices and whether there is an interaction relationship between the designated part and the devices based on this operation video. Furthermore, the scoring result of the student regarding the target experiment is automatically obtained. In this way, the workload of teachers scoring the experimental operations of students can be greatly reduced.

[0184] Optionally, the method for identifying whether there is an interaction relationship between each device includes:

[0185] For at least two devices among each device, if it is identified based on the position information of the at least two devices that the at least two devices have overlapping positions in multiple consecutive frames, it is determined that there is an interaction relationship between the at least two devices within the time range of the multiple consecutive frames; otherwise, it is determined that there is no interaction relationship between the at least two devices.

[0186] Optionally, the method for identifying whether there is an interaction relationship between the designated part and each device includes:

[0187] If it is identified based on the position information of the designated part and the position information of at least one device that the designated part and the at least one device have overlapping positions in multiple consecutive frames, the device candidate set corresponding to the first designated frame is used to judge whether there is an interaction relationship between the designated part and the at least one device.

[0188] Among them, the first specified frame is a video frame in the continuous multiple frames, and the device candidate set corresponding to the first specified frame includes at least one candidate device. Each candidate device is a device used by the specified part or a device to be used by the specified part within the time range of the first specified frame.

[0189] Optionally, the determining whether there is an interaction relationship between the specified part and the at least one device by using the device candidate set corresponding to the first specified frame includes:

[0190] Detect whether the at least one device is in the device candidate set corresponding to the first specified frame to obtain a detection result;

[0191] If the detection result indicates that none of the at least one device is in the device candidate set, it is determined that there is no interaction relationship between the specified part and the at least one device within the time range of the continuous multiple frames;

[0192] If the detection result indicates that one device is in the device candidate set, it is determined that within the time range of the continuous multiple frames, there is an interaction relationship between the specified part and one device existing in the device candidate set, and there is no interaction relationship between the specified part and the other devices among the at least one device;

[0193] If the detection result indicates that multiple devices are in the device candidate set, among the multiple devices, identify the target device closest to the specified part, and determine that within the time range of the continuous multiple frames, there is an interaction relationship between the specified part and the target device, and there is no interaction relationship between the specified part and the other devices among the at least one device.

[0194] Optionally, the identifying the target device closest to the specified part among the multiple devices includes:

[0195] Based on the depth information of the multiple devices in the second specified frame, identify the target device closest to the specified part among the multiple devices;

[0196] The second specified frame is a video frame in the continuous multiple frames.

[0197] Optionally, the identifying the target device closest to the specified part among the multiple devices based on the depth information of the multiple devices in the second specified frame includes:

[0198] Based on the depth information of the multiple devices in the second specified frame, select the depth information representing the minimum depth as the depth information to be used;

[0199] Determine the device with the depth information to be utilized as the target device closest to the specified part.

[0200] Optionally, the method for determining the depth information of any object in the second specified frame includes:

[0201] Determine the depth information of each pixel point in the second specified frame;

[0202] Based on the depth information of the pixel points in the target area of the second specified frame, determine the depth information of the any object in the second specified frame;

[0203] Wherein, the target area is the area characterized by the position information of the any object in the second specified frame.

[0204] Optionally, the method for constructing the device candidate set corresponding to the first specified frame includes:

[0205] Determine the operation state of the specified part in the first specified frame as the target operation state;

[0206] Search for the device corresponding to the target operation state from the pre-established mapping relationship between each operation state of the specified part and the device to obtain candidate devices; the device corresponding to each operation state is the device used or to be used when the specified part is in this operation state;

[0207] Use the obtained candidate devices to construct the device candidate set corresponding to the first specified frame.

[0208] Optionally, the target experiment is: an object density measurement experiment, the specified part is the hand; the scoring points of the target experiment include: the first scoring point, the second scoring point, the third scoring point, the fourth scoring point and the fifth scoring point;

[0209] Wherein, the experimental event characterized by the first scoring point includes: adjusting the balance nut of the balance to make the balance balanced;

[0210] The experimental event characterized by the second scoring point includes: the object is on the left side of the balance, adjusting the weights and the rider to make the balance horizontally balanced;

[0211] The experimental event characterized by the third scoring point includes: during the experimental event of the second scoring point, the balance nut is not adjusted;

[0212] The experimental event characterized by the fourth scoring point includes: using a dropper, a measuring cylinder, a beaker and water to measure the volume of the object;

[0213] The experimental events characterized by the fifth scoring point include: wiping the object with a rag and returning the used device to its original position.

[0214] Optionally, the judgment module includes:

[0215] The first judgment sub-module is used to, for the first scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the balance nut of the balance within a continuous multi-frame time range, and based on the position information of the scale dial of the balance in this continuous multi-frame, identify whether the pointer in the scale dial is located at the central scale line position; if the detection results are all yes, it is determined that the experimental event characterized by this first scoring point occurs, and the judgment result corresponding to this first scoring point is obtained.

[0216] Optionally, the judgment module includes:

[0217] The second judgment sub-module is used to, for the second scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the object and the left tray of the balance within the first continuous multi-frame time range, and detect whether there is an interaction relationship between the hand and the tweezers, between the tweezers and the weights, and between the weights and the right tray of the balance within the second continuous multi-frame time range, and detect whether there is an interaction relationship between the hand and the tweezers, and between the tweezers and the rider of the balance within the third continuous multi-frame time range, and based on the position information of the scale dial of the balance in this third continuous multi-frame, detect whether the pointer in the scale dial is located at the central scale line position; where the first continuous multi-frame is any continuous multi-frame, the start time of the second continuous multi-frame is not earlier than the start time of the first continuous multi-frame, and the start time of the third continuous multi-frame is not earlier than the start time of the second continuous multi-frame; if the detection results are all yes, it is determined that the experimental event characterized by the second scoring point occurs, and the judgment result corresponding to the second scoring point is obtained.

[0218] Optionally, the judgment module includes:

[0219] The third judgment sub-module is used to, for the third scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the balance nut within the specified continuous multi-frame time range; where the start time point of the specified continuous multi-frame is later than the start time point of the first continuous multi-frame and earlier than the end time point of the third continuous multi-frame, there is an interaction relationship between the object and the left tray of the balance within the time range of the first continuous multi-frame, and there is an interaction relationship between the hand and the rider of the balance within the time range of the third continuous multi-frame; if the detection result is no, it is determined that the experimental event of the third scoring point occurs, and the judgment result corresponding to the third scoring point is obtained.

[0220] Optionally, the determination module includes:

[0221] A fourth determination sub-module, configured to, for a fourth scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the dropper within a time range of a continuous plurality of frames, and detect whether there is an interaction relationship between the dropper and the graduated cylinder within a time range of a continuous plurality of frames; if the detection result is yes, determine that the experimental event characterized by the fourth scoring point occurs, and obtain a determination result corresponding to the fourth scoring point.

[0222] Optionally, the determination module includes:

[0223] A fifth determination sub-module, configured to, for a fifth scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the rag within a time range of a continuous plurality of frames, and detect whether there is an interaction relationship between the rag and the object within a time range of a continuous plurality of frames; and detect whether the positions of each device are in predetermined positions in video frames after the time range in which the rag and the object have an interaction relationship; if the detection results are all yes, determine that the experimental event characterized by the fifth scoring point occurs, and obtain a determination result corresponding to the fifth scoring point.

[0224] Optionally, the calculation module is specifically configured to:

[0225] For each scoring point, determine the scoring type corresponding to this scoring point, and based on the scoring type corresponding to this scoring point and the determination result corresponding to this scoring point, determine the score value corresponding to this scoring point; where the scoring type includes a scoring type or a deduction type;

[0226] Accumulate the score values corresponding to each scoring point to obtain a scoring result of the person to be scored for the target experiment.

[0227] An embodiment of the present invention further provides an electronic device, as Figure 11 shown, including a processor 1101, a communication interface 1102, a memory 1103, and a communication bus 1104, where the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104,

[0228] The memory 1103 is used to store a computer program;

[0229] The processor 1101, when executing the program stored on the memory 1103, implements the steps of any experimental scoring method.

[0230] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0231] The communication interface is used for communication between the above electronic device and other devices.

[0232] The memory can include a Random Access Memory (RAM), and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located far from the aforementioned processor.

[0233] The above-mentioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0234] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above experimental scoring methods are implemented.

[0235] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the experimental scoring methods in the above embodiments.

[0236] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0237] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including an..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0238] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0239] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included within the protection scope of the present invention.

Claims

1. An experimental scoring method, characterized in that, The method includes: Obtaining an operation video of the person to be scored during the operation of the target experiment; Performing object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; wherein, each of the objects includes: each device required for the target experiment and the designated part of the person to be scored; According to the position information of each obtained object, identifying whether there is an interaction relationship between each device, and identifying whether there is an interaction relationship between the designated part and each device, to obtain an identification result; wherein, the method for identifying whether there is an interaction relationship between the designated part and each device includes: if, based on the position information of the designated part and the position information of at least one device, it is identified that the designated part and the at least one device have position overlaps in a continuous plurality of frames, then using the device candidate set corresponding to the first designated frame to determine whether there is an interaction relationship between the designated part and the at least one device; wherein, the first designated frame is a video frame in the continuous plurality of frames, and the device candidate set corresponding to the first designated frame includes at least one candidate device, and each candidate device is determined based on the operation state of the designated part in the first designated frame within the time range of the first designated frame, the device used by the designated part or the device to be used by the designated part; the device used by the designated part or the device to be used by the designated part is: within the time range of the first designated frame, the device used or to be used by the person to be scored to complete the target experiment; For each scoring point of the target experiment, based on the identification result, determining whether the experimental event characterized by the scoring point occurs, to obtain the determination result corresponding to the scoring point; Based on the determination results corresponding to each scoring point, calculating the scoring result of the person to be scored regarding the target experiment.

2. The method according to claim 1, wherein The method for identifying whether there is an interaction relationship between each device includes: For at least two devices among each device, if, based on the position information of the at least two devices, it is identified that the at least two devices have position overlaps in a continuous plurality of frames, then it is determined that within the time range of the continuous plurality of frames, the at least two devices have an interaction relationship, otherwise, it is determined that the at least two devices do not have an interaction relationship.

3. The method according to claim 1, wherein The using the device candidate set corresponding to the first designated frame to determine whether there is an interaction relationship between the designated part and the at least one device includes: Detecting whether the at least one device is in the device candidate set corresponding to the first designated frame, to obtain a detection result; If the detection result indicates that none of the at least one device is in the device candidate set, it is determined that within the time range of the continuous plurality of frames, the designated part and the at least one device do not have an interaction relationship; If the detection result indicates that one device is in the device candidate set, it is determined that within the time range of the continuous plurality of frames, the designated part and one device existing in the device candidate set have an interaction relationship, and the designated part and the other devices among the at least one device do not have an interaction relationship; If the detection result indicates that multiple devices are in the device candidate set, among the multiple devices, identify the target device that is closest to the specified part, determine that there is an interaction relationship between the specified part and the target device within the time range of the consecutive multiple frames, and there is no interaction relationship between the specified part and other devices among the at least one device.

4. The method according to claim 3, wherein Among the multiple devices, identifying the target device that is closest to the specified part includes: Based on the depth information of the multiple devices in the second specified frame, among the multiple devices, identify the target device that is closest to the specified part; The second specified frame is a video frame in the consecutive multiple frames.

5. The method according to claim 4, wherein Based on the depth information of the multiple devices in the second specified frame, among the multiple devices, identifying the target device that is closest to the specified part includes: Based on the depth information of the multiple devices in the second specified frame, select the depth information representing the minimum depth as the depth information to be utilized; Determine the device with the depth information to be utilized as the target device that is closest to the specified part.

6. The method according to claim 4, wherein The determination method of the depth information of any object in the second specified frame includes: Determine the depth information of each pixel point in the second specified frame; Based on the depth information of the pixel points in the target area in the second specified frame, determine the depth information of the any object in the second specified frame; Wherein, the target area is the area represented by the position information of the any object in the second specified frame.

7. The method according to claim 1, characterized in that The construction method of the device candidate set corresponding to the first specified frame includes: Determine the operation state of the specified part in the first specified frame as the target operation state; Among the pre-established mapping relationships between each operation state of the specified part and devices, search for the devices corresponding to the target operation state to obtain candidate devices; the devices corresponding to each operation state are the devices used or to be used by the specified part when in this operation state; Use the obtained candidate devices to construct the device candidate set corresponding to the first specified frame.

8. The method according to any one of claims 1-7, characterized in that, The target experiment is: an object density measurement experiment, and the specified part is the hand; the scoring points of the target experiment include: the first scoring point, the second scoring point, the third scoring point, the fourth scoring point, and the fifth scoring point; Wherein, the experimental event represented by the first scoring point includes: adjusting the balance nut of the balance to make the balance balanced; The experimental event represented by the second scoring point includes: the object is on the left side of the balance, adjusting the weights and the rider to make the balance horizontally balanced; The experimental event represented by the third scoring point includes: during the experimental event of the second scoring point, not adjusting the balance nut; The experimental event represented by the fourth scoring point includes: using a dropper, a graduated cylinder, a beaker, and water to measure the volume of the object; The experimental event represented by the fifth scoring point includes: using a rag to wipe the object and putting the used devices back in place.

9. The method according to claim 8, characterized in that, For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event represented by the scoring point occurs, and obtain the judgment result corresponding to the scoring point, including: For the first scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the balance nut of the balance within a time range of a continuous multi-frame, and based on the position information of the scale dial of the balance in the continuous multi-frame, identify whether the pointer in the scale dial is located at the central scale line position; If the detection results are all yes, determine that the experimental event represented by the first scoring point occurs, and obtain the judgment result corresponding to the first scoring point.

10. The method according to claim 8, wherein For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event represented by the scoring point occurs, and obtain the judgment result corresponding to the scoring point, including: For the second scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the object and the left tray of the balance within a first continuous multi-frame time range, and detect whether there is an interaction relationship between the hand and the forceps, an interaction relationship between the forceps and the weights, and an interaction relationship between the weights and the right tray of the balance within a second continuous multi-frame time range, and detect whether there is an interaction relationship between the hand and the forceps, and an interaction relationship between the forceps and the rider of the balance within a third continuous multi-frame time range, and based on the position information of the scale dial of the balance in the third continuous multi-frame, detect whether the pointer in the scale dial is located at the central scale line position; where the first continuous multi-frame is any continuous multi-frame, the start time of the second continuous multi-frame is not earlier than the start time of the first continuous multi-frame, and the start time of the third continuous multi-frame is not earlier than the start time of the second continuous multi-frame; If the detection results are all yes, determine that the experimental event represented by the second scoring point occurs, and obtain the judgment result corresponding to the second scoring point.

11. The method according to claim 8, wherein For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event represented by the scoring point occurs, and obtain the judgment result corresponding to the scoring point, including: For the third scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the balance nut within a specified continuous multi-frame time range; where the start time point of the specified continuous multi-frame is later than the start time point of the first continuous multi-frame and earlier than the end time point of the third continuous multi-frame, there is an interaction relationship between the object and the left tray of the balance within the time range of the first continuous multi-frame, and there is an interaction relationship between the hand and the rider of the balance within the time range of the third continuous multi-frame; If the detection result is no, determine that the experimental event of the third scoring point occurs, and obtain the judgment result corresponding to the third scoring point.

12. The method according to claim 8, wherein For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event represented by the scoring point occurs, and obtain the judgment result corresponding to the scoring point, including: For the fourth scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the dropper within a time range of a continuous plurality of frames, and detect whether there is an interaction relationship between the dropper and the graduated cylinder within a time range of a continuous plurality of frames; If the detection result is yes, determine that the experimental event characterized by the fourth scoring point has occurred, and obtain the judgment result corresponding to the fourth scoring point.

13. The method according to claim 8, characterized in that For each scoring point of the target experiment, based on the recognition result, determine whether the experimental event characterized by this scoring point has occurred, and obtain the judgment result corresponding to this scoring point, including: For the fifth scoring point of the target experiment, based on the recognition result, detect whether there is an interaction relationship between the hand and the rag within a time range of a continuous plurality of frames, and detect whether there is an interaction relationship between the rag and the object within a time range of a continuous plurality of frames; and, detect whether the positions of each device are in predetermined positions in the video frames after the time range in which the rag and the object have an interaction relationship; If the detection results are all yes, determine that the experimental event characterized by the fifth scoring point has occurred, and obtain the judgment result corresponding to the fifth scoring point.

14. The method according to any one of claims 1-7, characterized in that Calculating the scoring result of the person to be scored for the target experiment based on the judgment results corresponding to each scoring point, including: For each scoring point, determine the scoring type corresponding to this scoring point, and based on the scoring type corresponding to this scoring point and the judgment result corresponding to this scoring point, determine the score value corresponding to this scoring point; wherein, the scoring type includes a scoring type or a deduction type; Accumulate the score values corresponding to each scoring point to obtain the scoring result of the person to be scored for the target experiment.

15. An experimental scoring device, characterized in that, The device includes: An acquisition module, configured to acquire the operation video of the person to be scored during the operation of the target experiment; A detection module, configured to perform object detection on each video frame in the operation video to obtain each object existing in the video frame and the position information of each object; wherein, each object includes: each device required for the target experiment and the designated part of the person to be scored; An identification module, configured to identify whether there is an interaction relationship between each device and whether there is an interaction relationship between the specified part and each device according to the obtained position information of each object, and obtain an identification result; wherein, the method for identifying whether there is an interaction relationship between the specified part and each device includes: if, based on the position information of the specified part and the position information of at least one device, it is identified that there is a position overlap between the specified part and the at least one device in multiple consecutive frames, then use the device candidate set corresponding to the first specified frame to determine whether there is an interaction relationship between the specified part and the at least one device; wherein, the first specified frame is a video frame in the multiple consecutive frames, and the device candidate set corresponding to the first specified frame includes at least one candidate device, and each candidate device is determined based on the operation state of the specified part in the first specified frame within the time range of the first specified frame, and is a device used by the specified part or a device to be used by the specified part; the device used by the specified part or the device to be used by the specified part is: within the time range of the first specified frame, a device used or to be used by the person to be scored to complete the target experiment; A judgment module, configured to, for each scoring point of the target experiment, based on the identification result, judge whether the experimental event characterized by the scoring point occurs, and obtain the judgment result corresponding to the scoring point; A calculation module, configured to calculate the scoring result of the person to be scored for the target experiment based on the judgment results corresponding to each scoring point.

16. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is configured to, when executing the program stored on the memory, implement the method steps described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-14.

Images