Standard operation scoring methods, systems, equipment and storage media for experimental instruments

Abstract

The present invention discloses a method, system, device and storage medium for scoring standard operations of experimental instruments, including: marking the data of the experimental test video to be scored and detecting the scoring points to determine the scoring points; detecting key objects and hands through a pre-established YOLOX detector and detecting hand postures through a pre-established MediaPipe detector to obtain detection results of the YOLOX detector and the MediaPipe detector; constructing a finite state machine model for the action and state sets within the scoring points, and sequentially detecting the states or actions that should appear; using different scoring schemes according to the scoring point category, judging the correctness of the scoring points to obtain the completion status of each operation at the scoring point; judging the completion status of each operation at the scoring point in combination with different perspectives according to the scoring scheme, and scoring the test video. The present invention establishes a finite state machine model, determines the start and end of the scoring point by detecting the scoring point mark, and can accurately divide the scoring points.

Description

Technical Field

[0001] The present invention belongs to the technical field of quality assessment in computer vision, and specifically relates to a standard operating method, system, equipment and storage medium for an experimental instrument. Background Art

[0002] Experimental courses are an important part of teaching. Traditional experimental exams are often manually graded by teachers, which is not only labor-intensive but also leads to inconsistent grading standards and low efficiency. Therefore, there is a need for a method that can automatically grade to save labor costs. In the prior art, the invention patent with application publication number CN114419736A provides a method for intelligent grading of high school entrance examination experiments. This patent trains a similarity calculation model based on scored videos and standard operation sample videos, and then uses this model to calculate the matching degree between the video to be scored and the reference video at the scoring point. The matching degree threshold is calculated through clustering, and each scoring point is scored according to the threshold. The shortcomings of this prior art are as follows:

[0003] 1. This technology obtains scoring point video clips by locating the keyframe at the scoring point in the video to be scored and capturing a fixed time window before and after the keyframe. This method is prone to under- or over-capturing the video, as the duration of movements varies from person to person, and thus the exact range of movement cannot be accurately covered. This can affect the final calculated match score and lead to inaccurate scoring.

[0004] 2. This technology does not take into account that certain states or actions in the experiment are dependent on previous states and actions. It only cuts out the video clips where the scoring point appears and scores them. The scoring criteria are not reasonable.

[0005] 3. This technology determines the score through a video matching calculation model. It can only determine whether each scoring point is correct or not, but cannot provide specific operation error information for each scoring point, and lacks explainability.

[0006] 4. This technology trains a similarity calculation model based on previously rated videos and standard operation example videos in a specific scenario. During testing, if the input video to be rated is in a different scenario than the previously rated video, the similarity calculation model may be difficult to transfer.

[0007] Therefore, the present invention proposes a scoring scheme for standard operation of experimental instruments. By constructing a finite state machine based on the positioning and state of objects and hands, the scoring strategy can be applied to different scenarios. Summary of the Invention

[0008] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a standard operation scoring method for experimental instruments. By establishing a finite state machine model and detecting the scoring point marks to determine the start and end of the scoring points, the scoring points can be accurately divided, which is conducive to improving the accuracy of the scoring and facilitating manual review.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a method for scoring standard operation of an experimental instrument, comprising the following steps:

[0011] Mark the data and detection scoring point marks of the experimental test video to be scored, and determine the scoring points;

[0012] Determining the category of the scoring point; the categories of the scoring point include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's hand and the object, and the examinee performing a sequence of actions;

[0013] Detect key objects and hands using a pre-established YOLOX detector and detect hand postures using a pre-established MediaPipe detector, and obtain detection results of the YOLOX detector and the MediaPipe detector;

[0014] Based on the detection results of the YOLOX detector and the MediaPipe detector and the category of the scoring point, a finite state machine model for object behavior modeling is constructed for the action and state set within each scoring point. The state or action that should appear is detected in sequence, and a correct or incorrect judgment is performed to obtain the completion status of each operation at the scoring point.

[0015] The completion status of each operation at the scoring point is judged from different perspectives to obtain the scoring results.

[0016] As a preferred technical solution, the data of the labeled experimental test video to be scored includes the labeling of objects related to the scoring points and the labeling of objects at key points;

[0017] The labeling of objects related to the scoring points includes labeling the position, length, width and object category label of each object related to the scoring point, and labeling the necessary actions of the experiment frame by frame; the remaining waiting actions are labeled frame by frame or not labeled, and for objects with multiple states, different labels are labeled for each state; the object labeling of key points is based on the specific positional relationship between objects, or the specific relationship between the examinee's hand and the object, and the key points of the object are labeled.

[0018] As a preferred technical solution, the scoring point of the state presented by the object is to detect the object using a YOLOX detector and judge the state of the object. If the object continues to be in the correct state in N frames, the scoring point is considered to be correctly completed, where N is a set threshold.

[0019] As a preferred technical solution, the scoring points with specific positional relationships between objects are detected by the YOLOX detector to detect multiple objects involved in the scoring points, and the intersection relationship of the objects or the positional relationship between the key points of the objects is judged; for the positional relationship between objects that do not require marking of key points, the intersection-and-union ratio between the objects is calculated. If the intersection-and-union ratio is greater than a manually set threshold α and lasts for N frames, the positional relationship between the objects is correct, that is, the scoring point is correctly completed; for the positional relationship between objects that require marking of key points, it is judged whether the object and the detected key points have a correct positional relationship. If the positional relationship between the object and the detected key points is correct, the scoring point is correctly completed.

[0020] As a preferred technical solution, the scoring points where the examinee's hand has a characteristic positional relationship with the object are scored by using the YOLOX detector to detect multiple objects involved in the scoring points, and using the MediaPipe detector to detect the coordinates of the hand joints; first, the hand posture is judged whether it is correct based on the joints, and then the relationship between the hand joints and the object is judged. If the posture and positional relationship are both correct and last for N frames, the scoring point is considered to be completed correctly.

[0021] As a preferred technical solution, the scoring point of the examinee's execution of the sequence action requires the correctness judgment of the sequence action and state;

[0022] First, a finite state machine model is established based on the order in which the sequence states occur. Each state in the finite state machine model is one of the states of an object, a specific positional relationship between objects, and a specific positional relationship between the examinee's hand and the object.

[0023] Secondly, the state transfer is carried out. According to the operation order of the scoring point, the state or action that should appear is detected. If the state or action that should appear is successfully transferred to the final state, the scoring point is considered to be completed correctly.

[0024] As a preferred technical solution, the method of combining different perspectives to judge the completion status of each operation at the scoring point and obtain the scoring result is a multi-perspective fusion scoring solution for special occlusion situations. Since objects or hand movements related to the scoring point may be occluded by a single perspective during the exam, cameras with two perspectives are used to shoot the exam video. When scoring each scoring point, if one of the perspectives completes the operation correctly, the scoring point is considered to have scored.

[0025] In a second aspect, the present invention further provides a standard operating system for an experimental instrument, which is applied to the standard operating scoring method for an experimental instrument, and includes a scoring point determination module, a scoring point category judgment module, a scoring point category detection module, a finite state machine model construction module, and a scoring module;

[0026] The scoring point determination module is used to mark the data of the experimental test video to be scored, detect the scoring points, and determine the scoring points;

[0027] The scoring point category determination module is used to determine the category of the scoring point; the categories of the scoring point include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's hand and the object, and the examinee performing a sequence of actions;

[0028] The scoring point category detection module is used to detect key objects and hands through a pre-established YOLOX detector and to detect hand postures through a pre-established MediaPipe detector, thereby obtaining detection results of the YOLOX detector and the MediaPipe detector;

[0029] The finite state machine model building module is used to build a finite state machine model for object behavior modeling for the action and state set within each scoring point based on the detection results of the YOLOX detector and the MediaPipe detector and the category of the scoring point, sequentially detect the states or actions that should appear, and perform correct or incorrect judgment to obtain the completion status of each operation of the scoring point;

[0030] The scoring module is used to judge the completion status of each operation of the scoring point in combination with different perspectives to obtain a scoring result.

[0031] In a third aspect, the present invention provides an electronic device, comprising:

[0032] at least one processor; and,

[0033] a memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the experimental instrument standard operation scoring method.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the standard operation scoring method for experimental instruments.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] 1. Existing technologies calculate the matching degree between two videos, which takes a long time to calculate and has low scoring efficiency. This invention combines object location and object state judgment through joint learning, requiring only a single model to infer each frame of the test video, resulting in high recognition efficiency.

[0038] 2. Existing technologies use the same evaluation method for each scoring point, which can only assess the overall correctness of each scoring point and cannot provide a detailed evaluation of each sub-action. This invention divides all scoring points into four categories and designs evaluation rules based on finite state machines for each category. This fully considers the interaction between equipment and equipment, as well as between people and equipment. It can provide a fine-grained evaluation of experimental operations and obtain the specific behavioral quality of each key operation point. It is highly interpretable and convenient for manual review.

[0039] 3. The existing technology uses video matching calculation to score, which is easily affected by scene differences, has poor transferability, and high training costs. The scoring strategy based on the finite state machine of the present invention can be better applied to different scenes, is easy to transfer, and has low training costs.

[0040] 4. The existing technology uses a fixed, manually set time window to capture operation clips, which can easily result in under- or over-capturing of the video, affecting scoring accuracy. The present invention establishes a finite state machine model and detects scoring point markers to determine the start and end of scoring points. This allows for accurate demarcation of scoring points, improving scoring accuracy and facilitating manual review. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 This is a flow chart of a standard operation scoring method for an experimental instrument according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of marking objects related to scoring points according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of objects marking key points of an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a finite state machine model is constructed for an embodiment of the present invention;

[0046] Figure 5 4 is a block diagram of a standard operation scoring system for experimental instruments according to an embodiment of the present invention;

[0047] Figure 6 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0050] Example

[0051] In order to better illustrate the technical solution of the present invention, this embodiment uses some key scoring points in the process of "Chemical Properties of Acids" in a certain city's middle school entrance examination as an example. The experimental process in this experimental test is generally divided into three steps: acid-base reaction, rust removal of iron nails with acid, and cleaning and sorting. Among them, the acid-base reaction part has two scoring points, namely, taking a small amount of calcium hydroxide powder, tilting the test tube, carefully sending the medicine spoon to the bottom of the test tube, and then making the test tube upright; placing the dilute hydrochloric acid bottle stopper upside down on the table; adding 2-3mL of dilute hydrochloric acid to the test tube by pouring method, covering the bottle stopper after taking the liquid, shaking the test tube, and observing the phenomenon. The acid-based nail rust removal section has seven scoring points: Place the test tube horizontally, use tweezers to pick up a rusty nail, place it in the test tube, and slowly turn the test tube upright, allowing the nail to slide to the bottom of the tube; Add dilute hydrochloric acid by pouring until the nail is just submerged; Use a test tube clamp to clamp the test tube from the bottom, putting it on and off, clamping it about 1 / 3 of the way from the tube's opening; When heating, do not point the test tube's opening toward yourself or others; Extinguish the alcohol burner with a lamp cap and place the test tube on a test tube rack to cool; Carefully pour the reaction solution and the nail into a small beaker, remove the nail with tweezers, rinse it under the tap, and place it in a large beaker labeled "Recycle Nails." The cleaning and tidying section has one scoring point: Pour the waste liquid from the test tube and beaker into a waste collection bucket, rinse the test tube and beaker, and wipe the tweezers and medicine spoon. Tidy up the table and return the lab supplies to their original locations. This experiment is worth 10 points, with a total of 10 scoring points.

[0052] See also Figure 1In one embodiment of the present application, a standard operation scoring method for an experimental instrument is provided, comprising the following steps:

[0053] S1. Labeling data of the experimental test video to be scored, detecting scoring points, and determining scoring points; further, the labeling data of the experimental test video to be scored includes labeling objects related to scoring points and object labeling of key points.

[0054] Specifically, the labeling scheme for objects related to scoring points: In this embodiment, for each object related to the scoring point, the object position, length and width, and object category label need to be labeled. The actions necessary for the experiment are labeled frame by frame, and the remaining waiting actions can be labeled frame by frame or not. This embodiment lists the objects related to the experimental scoring points. For objects with multiple states, different labels are given to each state. Taking the experiment "Chemical Properties of Acids" as an example, this embodiment lists 27 labeling labels such as medicine spoons, hydrochloric acid bottles, and nail bottles. Among them, the color of the test tube liquid has two states of "white" and "green", which are labeled as two different categories; the labeling samples are as follows: Figure 2 shown.

[0055] Furthermore, this embodiment requires combining two perspectives, so both the front and side videos are annotated. Because each school's experimental environment varies significantly, each school needs to annotate a certain amount of video. For experimental phenomena that occur, additional annotated samples are needed. For example: bottle caps, liquid surfaces, flames, and paper towels. For other conventional experimental instruments, since they appear in every frame, the number of annotated images can be appropriately reduced.

[0056] Object marking scheme for key points: Some scoring points require that objects have a specific positional relationship, or that the examinee's hand has a specific positional relationship with the object. This embodiment marks key points for such objects. Taking the scoring point "When clamping the test tube with a test tube clamp, the test tube clamp should be clamped at about 1 / 3 of the distance from the test tube mouth" as an example, the present invention marks the three key points of the test tube head, test tube tail, and test tube clamp clamping position, such as Figure 3 shown.

[0057] S2. Determine the category of the scoring point; the categories of the scoring point include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's head and the object, and the examinee performing a sequence of actions.

[0058] S3. Detect key objects and hands using a pre-established YOLOX detector and detect hand postures using a pre-established MediaPipe detector, and obtain detection results of the YOLOX detector and the MediaPipe detector;

[0059] For scoring points based on the state of an object, this embodiment first uses the YOLOX detector to detect the object and determine its state. If the object remains in the correct state for N frames, the scoring point is considered successfully completed; N is a manually set threshold. For example, the scoring point "Placing the cork of a dilute hydrochloric acid bottle upside down on the table" has specific requirements for the state of the cork. The YOLOX detector is used to detect the corks of the dilute hydrochloric acid bottle and the reagent bottle, and the corks are classified into upside-down and other states. If the detector detects the cork is upside down and this state persists for N frames, the scoring point is considered successfully completed.

[0060] For scoring points with specific positional relationships between objects, this embodiment first uses the YOLOX detector to detect multiple objects involved in the scoring points, and judges the intersection relationship of the objects or the positional relationship between the key points of the objects. Specifically, for the positional relationship between objects that do not require marking of key points, the intersection-and-union ratio between the objects is calculated. If the intersection-and-union ratio is greater than α and persists for N frames, the positional relationship between the objects is considered correct. Taking the scoring point "use tweezers to pick up a rusty nail" as an example, the nail bottle and tweezers are detected. If it is detected that the two experimental instruments are very close (the intersection-and-union ratio is greater than α), it is considered that the nail is correctly obtained. For the positional relationship between objects that require marking of key points, it is necessary to judge whether the object and the detected key points have a correct positional relationship. Taking the scoring point "When clamping the test tube with a test tube clamp, put it on and take it off from the bottom of the test tube, and clamp it about 1 / 3 of the way from the test tube mouth" as an example, there is a requirement for a specific positional relationship between the test tube clamp and the test tube mouth; by inspecting the test tube clamp and the test tube, and inspecting the head and tail of the test tube and the key points clamped by the test tube clamp, if the test tube clamp overlaps with the test tube from the bottom of the test tube and leaves from the bottom of the test tube, the position of the key point clamped by the test tube clamp should be in the upper half of the test tube, then the scoring point is considered to be completed correctly.

[0061] For scoring points where the examinee's hand has a specific positional relationship with an object, this embodiment first uses the YOLOX detector to detect multiple objects involved in the scoring point, and then uses the MediaPipe detector to detect the coordinates of the hand joints. The hand posture is determined based on the joints, and then the relationship between the hand joints and the object is determined. If both the posture and relationship are correct and persist for N frames, the scoring point is considered correctly completed. For the scoring point "When heating, hold the long handle of the test tube clamp" as an example, the examinee's hand should hold the long handle, not the short handle. After using the YOLOX detector to detect the test tube and the test tube clamp, and MediaPipe to detect the hand joints, it is first determined whether the hand is holding the test tube clamp, and then the distance from the thumb tip to the short handle of the test tube clamp is used to make a judgment. If the test tube clamp is held correctly and the distance between the thumb tip and the short handle of the test tube clamp is greater than the manually set threshold s, and persists for N frames, the scoring point is considered correctly completed.

[0062] For the scoring points where the examinee performs sequence actions, the state of the objects, the specific positional relationship between objects, and the specific positional relationship between the examinee's hands and objects may appear at the same time, and the correctness of the sequence actions and states needs to be judged.

[0063] S4. Based on the detection results of the YOLOX detector and the MediaPipe detector, a finite state machine model is constructed for each set of actions and states in the category of the scoring point. The states or actions that should appear are detected in sequence, and a correct or incorrect judgment is performed to obtain the completion status of each operation of the scoring point.

[0064] Furthermore, the finite state machine model is constructed according to the order in which the sequence states occur. Each state in the model will be one of the states in which the object is presented, the objects have a specific positional relationship, and the examinee's hand has a specific positional relationship with the object. The state is then transferred according to the finite state machine model. If the state is successfully transferred to the final state of "correct completion", it is considered that the scoring point is correctly completed. Taking the scoring point "Put the cork of the dilute hydrochloric acid bottle upside down on the table; add 2 to 3 mL of dilute hydrochloric acid to the test tube by pouring (with the label facing the palm of your hand); cover the cork after taking the liquid. Shake the test tube and observe the phenomenon." as an example, a finite state machine model is established as follows: Figure 4 .according to Figure 4 Perform state transition, check whether the hydrochloric acid bottle cap is in an inverted state, check the specific position relationship between the examinee's hand and the hydrochloric acid bottle and the bottle label, check the specific position relationship between the hydrochloric acid bottle and the test tube, and check whether the hydrochloric acid bottle cap is in a closed state. If the hydrochloric acid bottle cap is inverted, the hand is holding the bottle body, and the label is facing the palm of the hand, the intersection ratio of the test tube and the hydrochloric acid bottle exceeds α, and the hydrochloric acid bottle cap is closed, and all of these conditions last for N frames, then the scoring point is considered to be completed correctly.

[0065] S5. Determine the completion status of each operation at the scoring point from different perspectives to obtain a scoring result.

[0066] Furthermore, since some scoring points may be obscured by objects or hand movements related to a single perspective during the test, the present invention uses cameras with two perspectives to shoot the test video; when scoring each scoring point, if one of the perspectives completes the operation correctly, the scoring point is considered to have been scored.

[0067] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0068] Based on the same concept as the laboratory instrument standard operation scoring method in the above embodiment, the present invention also provides a laboratory instrument standard operation scoring system, which can be used to implement the above laboratory instrument standard operation scoring method. For ease of explanation, the structural diagram of the embodiment of the laboratory instrument standard operation scoring system only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it can include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0069] See also Figure 5 In another embodiment of the present application, a standard operation scoring system 100 for an experimental instrument is provided, the system comprising a scoring point determination module 101, a scoring point category determination module 102, a scoring point category detection module 103, a finite state machine model construction module 104, and a scoring module 105;

[0070] The scoring point determination module 101 is used to mark the data of the experimental test video to be scored and detect the scoring points to determine the scoring points;

[0071] The scoring point category determination module 102 is used to determine the category of the scoring point; the scoring point categories include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's hand and the object, and the examinee performing a sequence of actions;

[0072] The scoring point category detection module 103 is configured to detect key objects and hands using a pre-established YOLOX detector and detect hand gestures using a pre-established MediaPipe detector, thereby obtaining detection results of the YOLOX detector and the MediaPipe detector.

[0073] The finite state machine model building module 104 is used to build a finite state machine model for each set of actions and states in the category of the scoring point based on the detection results of the YOLOX detector and the MediaPipe detector, sequentially detect the states or actions that should appear, and perform correct or incorrect judgment to obtain the completion status of each operation of the scoring point;

[0074] The scoring module 105 is used to determine the completion status of each operation at a scoring point by combining different perspectives to obtain a scoring result.

[0075] It should be noted that the experimental instrument standard operation scoring system of the present invention corresponds one-to-one to the experimental instrument standard operation scoring method of the present invention. The technical features and beneficial effects described in the embodiment of the above-mentioned experimental instrument standard operation scoring method are all applicable to the embodiment of the experimental instrument standard operation scoring system. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0076] In addition, in the implementation of the laboratory instrument standard operation scoring system in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the laboratory instrument standard operation scoring system is divided into different program modules to complete all or part of the functions described above.

[0077] See also Figure 6 In one embodiment, an electronic device for implementing a standard operation scoring method for an experimental instrument is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a standard operation scoring program 203 for an experimental instrument.

[0078] Wherein, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the standard operating scoring program 203 of the laboratory instrument, but can also be used to temporarily store data that has been output or is to be output.

[0079] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0080] Figure 6 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 6 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0081] The experimental instrument standard operation scoring program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When executed in the first processor 201, it can achieve the following:

[0082] Mark the data and test scoring points of the experimental test video to be scored, and determine the scoring points;

[0083] Determining the category of the scoring point; the categories of the scoring point include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's hand and the object, and the examinee performing a sequence of actions;

[0084] The scoring point categories detect key objects and hands using a pre-established YOLOX detector and detect hand postures using a pre-established MediaPipe detector to obtain detection results of the YOLOX detector and the MediaPipe detector;

[0085] Based on the detection results of the YOLOX detector and the MediaPipe detector, a finite state machine model is constructed for each set of actions and states in the category of the scoring point, and the states or actions that should appear are detected in sequence, and correct or incorrect judgments are made to obtain the completion status of each operation of the scoring point;

[0086] The completion status of each operation at the scoring point is judged from different perspectives to obtain the scoring results.

[0087] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0088] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. The standard operation scoring method of the experimental instrument is characterized by: The steps include: Mark the data and detection scoring point marks of the experimental test video to be scored, and determine the scoring points; Determining the category of the scoring point; the categories of the scoring point include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's hand and the object, and the examinee performing a sequence of actions; Scoring points where the candidate's hand has a characteristic positional relationship with an object are scored by using the YOLOX detector to detect multiple objects involved in the scoring point and the MediaPipe detector to detect the coordinates of the hand joints. The hand posture is first determined based on the joints, and then the relationship between the hand joints and the object is determined. If both the posture and positional relationship are correct and persist for N frames, the scoring point is considered completed correctly. The scoring point for the candidate's execution of the sequence action requires judgment on the correctness of the sequence action and state; First, a finite state machine model is established based on the order in which the sequence states occur. Each state in the finite state machine model is one of the states of an object, a specific positional relationship between objects, and a specific positional relationship between the examinee's hand and the object. Next, the state transfer is performed. According to the operation order of the scoring point, the state or action that should appear is detected. If the state or action that should appear is successfully transferred to the final state, the scoring point is considered to be completed correctly. Detect key objects and hands using a pre-established YOLOX detector and detect hand postures using a pre-established MediaPipe detector, and obtain detection results of the YOLOX detector and the MediaPipe detector; Based on the detection results of the YOLOX detector and the MediaPipe detector and the category of the scoring point, a finite state machine model for object behavior modeling is constructed for the action and state set within each scoring point. The state or action that should appear is detected in sequence, and a correct or incorrect judgment is performed to obtain the completion status of each operation at the scoring point. The scoring result is obtained by combining different perspectives to judge the completion of each operation at the scoring point. This is a multi-perspective fusion scoring scheme for special occlusion situations. Since objects or hand movements related to the scoring point may be occluded by a single perspective during the exam, cameras with two perspectives are used to shoot the exam video. When scoring each scoring point, if the operation is completed correctly from one of the perspectives, the scoring point is considered to have scored.

2. The standard operation scoring method for experimental instruments according to claim 1, characterized in that: The data of the labeled experimental test video to be scored includes the labeling of objects related to the scoring points and the labeling of objects at key points; The labeling of objects related to the scoring points includes labeling the position, length, width and object category label of each object related to the scoring point, and labeling the necessary actions of the experiment frame by frame; the remaining waiting actions are labeled frame by frame or not labeled, and for objects with multiple states, different labels are labeled for each state; the object labeling of key points is based on the specific positional relationship between objects, or the specific relationship between the examinee's hand and the object, and the key points of the object are labeled.

3. The standard operation scoring method for experimental instruments according to claim 1, characterized in that: The scoring point of the state presented by the object is to detect the object using the YOLOX detector and judge the state of the object. If the object continues to be in the correct state for N frames, the scoring point is considered to be completed correctly, where N is the set threshold.

4. The standard operation scoring method for experimental instruments according to claim 1, characterized in that: The scoring points with specific positional relationships between objects are detected by the YOLOX detector using multiple objects involved in the scoring points, and the intersection relationship of the objects or the positional relationship between the key points of the objects are judged; For the positional relationship between objects that do not require key point annotation, the intersection-over-union ratio between the objects is calculated. If the intersection-over-union ratio is greater than the manually set threshold α and persists for N frames, the positional relationship between the objects is correct, that is, the scoring point is completed correctly; For the positional relationship between objects that need to be marked with key points, it is determined whether the positional relationship between the object and the detection key points is correct. If the positional relationship between the object and the detection key points is correct, the scoring point is completed correctly.

5. Standard operating system for experimental instruments, characterized in that: The standard operation scoring method for an experimental instrument applied to any one of claims 1 to 4 comprises a scoring point determination module, a scoring point category judgment module, a scoring point category detection module, a finite state machine model construction module, and a scoring module; The scoring point determination module is used to mark the data of the experimental test video to be scored, detect the scoring points, and determine the scoring points; The scoring point category determination module is used to determine the category of the scoring point; the categories of the scoring point include the state of the object, the specific positional relationship between objects, the specific positional relationship between the examinee's hand and the object, and the examinee performing a sequence of actions; The scoring point category detection module is used to detect key objects and hands through a pre-established YOLOX detector and to detect hand postures through a pre-established MediaPipe detector, thereby obtaining detection results of the YOLOX detector and the MediaPipe detector; The finite state machine model building module is used to build a finite state machine model for object behavior modeling for the action and state set within each scoring point based on the detection results of the YOLOX detector and the MediaPipe detector and the category of the scoring point, sequentially detect the states or actions that should appear, and perform correct or incorrect judgment to obtain the completion status of each operation of the scoring point; The scoring module is used to judge the completion status of each operation of the scoring point in combination with different perspectives to obtain a scoring result.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the standard operating scoring method for an experimental instrument according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the standard operation scoring method of the experimental instrument according to any one of claims 1 to 4 is implemented.

Images