Behavior recognition algorithm test method, electronic equipment and storage medium

Abstract

The invention relates to the field of video data processing, and discloses a test method of a behavior recognition algorithm, electronic equipment and a storage medium, the method comprises the following steps: determining an annotation event list and an actual event list through video data, the annotation event list comprising a plurality of annotation events, the actual event list comprising a plurality of actual events, calculating the intersection-union ratio of the annotation event and the actual event, determining the comparison result of the annotation event list and the actual event list according to the intersection-union ratio, and determining a confusion matrix according to the comparison result; and calculating an evaluation result of the behavior recognition algorithm according to the confusion matrix. According to the method, the confusion matrix is determined by calculating the intersection-union ratio of the annotation events and the actual events and determining the comparison result of the annotation event list and the actual event list according to the intersection-union ratio, and the evaluation result of the behavior recognition algorithm is further calculated, so that the test quality of the behavior recognition algorithm can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video data processing, and in particular to a testing method, electronic device and storage medium for a behavior recognition algorithm. Background Art

[0002] Behavior recognition algorithm is a technology used to automatically identify and analyze human behavior in video or image sequences. It has a wide range of applications. Therefore, it is of great significance to study the accuracy of behavior recognition algorithms.

[0003] Currently, behavior recognition in videos is achieved by converting videos into image sequences to extract features from the image sequences, and then further performing feature fusion to identify the behavior in the video. Alternatively, related algorithms are used to annotate behaviors that appear in video data to further determine the correctness of the behavior. However, feature recognition errors often occur, or video data where no behavior appears is still annotated, which leads to behavior recognition errors, inaccurate behavior recognition algorithms, and unsatisfactory results. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application provide a testing method, an electronic device and a storage medium for a behavior recognition algorithm to solve the problem of low recognition accuracy of existing behavior recognition algorithms.

[0005] The embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for testing a behavior recognition algorithm, comprising:

[0007] Acquire video data, wherein the video data includes picture frames;

[0008] Obtain the annotation results of each picture frame of the video data;

[0009] Obtaining a labeled event according to the labeling result to determine a labeled event list, wherein the labeled event list includes a plurality of labeled events, and each labeled event includes a plurality of picture frames;

[0010] Based on the behavior recognition algorithm, the video data is recognized to obtain the recognition result corresponding to each picture frame of the video data;

[0011] According to the recognition result, an actual event is obtained to determine an actual event list, wherein the actual event list includes a plurality of actual events, and each actual event includes a plurality of picture frames;

[0012] Calculate the intersection and union ratio of the labeled events and the actual events;

[0013] According to the intersection-union ratio, the comparison result between the marked event list and the actual event list is determined;

[0014] According to the comparison results, the confusion matrix is ​​determined;

[0015] Based on the confusion matrix, the evaluation results of the behavior recognition algorithm are calculated.

[0016] In some embodiments, the recognition result includes the first action type, and the video data is recognized to obtain the recognition result corresponding to each picture frame of the video data, including:

[0017] Identify each picture frame of the video data to obtain a first person frame corresponding to each picture frame;

[0018] Identify the first person frame corresponding to each picture frame to obtain multiple key points corresponding to each first person frame;

[0019] Connect multiple key points to obtain a skeleton model;

[0020] A first action type is obtained according to the skeleton model, wherein the first action type corresponds to a picture frame one by one.

[0021] In some embodiments, the marked event includes a first start frame and a first end frame, the actual event includes a second start frame and a second end frame, and the actual event is obtained according to the recognition result to determine the actual event list, including:

[0022] Determining whether the first action type in the recognition result is the same as the preset action type;

[0023] If they are the same, pre-marking the picture frame corresponding to the first action type to obtain the pre-marked picture frame;

[0024] Based on the pre-marked picture frames, the actual events are obtained;

[0025] Subtract a preset value from the sequence number corresponding to the first start frame to obtain a first sequence number, and add the preset value to the sequence number corresponding to the first end frame to obtain a second sequence number;

[0026] If the sequence number corresponding to the second start frame is greater than the first sequence number, and the sequence number corresponding to the second end frame is less than the second sequence number, the same actual event corresponding to the second start frame and the second end frame is filtered to obtain the filtered actual event, and the filtered actual event is added to the actual event list.

[0027] In some embodiments, determining whether the first action type in the recognition result is the same as the preset action type includes:

[0028] Based on the behavior recognition algorithm, the picture frame is recognized to obtain the credibility of the first action type corresponding to the picture frame;

[0029] If the credibility of the first action type is greater than or equal to the credibility threshold, it means that the first action type is the same as the preset action type;

[0030] If the credibility of the first action type is less than the credibility threshold, it means that the first action type is different from the preset action type.

[0031] In some embodiments, the annotation result of each picture frame includes a second person frame, and calculating the intersection-over-union ratio of the annotated event and the actual event includes:

[0032] Intersecting and merging the first character frame and the second character frame to obtain an intersection frame;

[0033] According to the intersection-and-union frame, the intersection-and-union ratio of the first person frame and the second person frame is calculated.

[0034] In some embodiments, the first character frame includes first diagonal coordinates, and the second character frame includes second diagonal coordinates;

[0035] According to the intersection-and-union frame, the intersection-and-union ratio of the first person frame and the second person frame is calculated, including:

[0036] Calculate the diagonal coordinates of the intersection frame according to the first diagonal coordinates and the second diagonal coordinates to obtain the third diagonal coordinates;

[0037] According to the third diagonal coordinate, calculating the intersection of the first character frame and the second character frame;

[0038] Calculate the union of the first character frame and the second character frame according to the first diagonal coordinates, the second diagonal coordinates and the intersection;

[0039] Divide the union by the intersection to get the intersection ratio.

[0040] In some embodiments, determining the comparison result between the marked event list and the actual event list according to the intersection-over-union ratio includes:

[0041] Select an actual event with the largest intersection-and-union ratio in the actual event list to obtain a first successful matching event, and select a labeled event with the largest intersection-and-union ratio in the labeled event list to obtain a second successful matching event;

[0042] Compare whether the action types of the first successful matching event and the second successful matching event are the same;

[0043] If they are the same, marking the first successful matching event and the second successful matching event as a first mark;

[0044] If they are not the same, marking the first successful matching event and the second successful matching event as a second mark;

[0045] A comparison result between the marked event list and the actual event list is determined according to the first mark and the second mark.

[0046] In some embodiments, the evaluation result of the behavior recognition algorithm is calculated according to the confusion matrix, including:

[0047] According to the confusion matrix, an evaluation result corresponding to each action type is calculated, wherein the evaluation result includes at least one of recall rate, accuracy rate, precision rate, sensitivity, error rate, and specificity.

[0048] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0049] At least one processor and at least one memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the testing method of the behavior recognition algorithm of the first aspect.

[0050] In a third aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the testing method of the behavior recognition algorithm of the first aspect.

[0051] The beneficial effect of the implementation mode of the present application is: different from the prior art, the implementation mode of the present application provides a testing method for a behavior recognition algorithm, the method comprising: obtaining video data, wherein the video data includes picture frames; obtaining the labeling results of each picture frame of the video data; according to the labeling results, obtaining labeled events to determine a labeled event list, wherein the labeled event list includes multiple labeled events, and each labeled event includes multiple picture frames; based on the behavior recognition algorithm, the video data is identified to obtain the recognition results corresponding to each picture frame of the video data; according to the recognition results, actual events are obtained to determine an actual event list, wherein the actual event list includes multiple actual events, and each actual event includes multiple picture frames; calculating the intersection-and-union ratio of the labeled events and the actual events; according to the intersection-and-union ratio, determining the comparison results of the labeled event list and the actual event list; according to the comparison results, determining a confusion matrix; according to the confusion matrix, calculating the evaluation results of the behavior recognition algorithm.

[0052] By calculating the intersection-and-union ratio of labeled events and actual events, and determining the comparison results of the labeled event list and the actual event list based on the intersection-and-union ratio, to determine the confusion matrix, and further calculate the evaluation results of the behavior recognition algorithm, the present application can improve the quality of the behavior recognition algorithm test. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] One or more embodiments are exemplarily described by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and the figures in the drawings do not constitute proportional limitations unless otherwise stated.

[0054] Figure 1 It is a flowchart of a testing method of a behavior recognition algorithm provided in an embodiment of the present application;

[0055] Figure 2 yes Figure 1 A detailed flow chart of step S104 in FIG.

[0056] Figure 3 yes Figure 1 A detailed flow chart of step S105 in FIG.

[0057] Figure 4 yes Figure 3 A detailed flowchart of step S151 in FIG.

[0058] Figure 5 yes Figure 1 A detailed flow chart of step S106 in FIG.

[0059] Figure 6 yes Figure 5 A detailed flow chart of step S162 in FIG.

[0060] Figure 7 is an example schematic diagram of calculating the intersection-over-union ratio provided in an embodiment of the present application;

[0061] Figure 8 yes Figure 1 A detailed flow chart of step S107 in FIG.

[0062] Fig. 9 yes Figure 1 A detailed flow chart of step S109 in FIG.

[0063] Fig.10 It is a schematic diagram of the overall structure of a testing method for a behavior recognition algorithm provided in an embodiment of the present application;

[0064] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0065] Description of Figure Numbers:

[0066] Label name Label name 110 Electronic devices 102 Memory 101 processor DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, and all are within the scope of protection of the present application. In addition, the words "first", "second", "third", etc. used in the present application do not limit the data, but only divide the same or similar items with basically the same functions and effects.

[0069] The technical solution of the present application is described in detail below in conjunction with the accompanying drawings:

[0070] See also Figure 1 , Figure 1 It is a flowchart of a testing method of a behavior recognition algorithm provided in an embodiment of the present application;

[0071] The testing method of the behavior recognition algorithm is applied to an electronic device. Specifically, the executing subject of the testing method of the behavior recognition algorithm is one or at least two processors of the electronic device.

[0072] The electronic device is used to execute all steps of the test method of the behavior recognition algorithm. It is understandable that the electronic device includes a tablet computer, a computer terminal, and other network-connected devices.

[0073] like Figure 1 As shown, the method includes:

[0074] Step S101: acquiring video data, wherein the video data includes picture frames;

[0075] Specifically, a video download request is initiated to a video website, and the required video is downloaded locally, such as some course videos, and the downloaded video is preliminarily screened using a behavior recognition algorithm to identify the number of actions in the video, convert the video into picture frames one by one, and identify whether there is an action in each picture frame and the action type corresponding to the action. Finally, the action type and number of actions in each video are summarized to further screen videos with a large number of actions and action types per unit time, and obtain the screened video data, which is used as test data for the behavior recognition algorithm. It can be understood that the action is the action performed by the character, for example, the course video includes a teacher and a student, and the student's raising of hands, standing up, moving, and turning are actions, and the video is preliminarily screened to remove some irrelevant video data, such as some videos in which the teacher and the student do not appear in the course video.

[0076] In the embodiment of the present application, the video data is preliminarily screened so as to better evaluate the performance and robustness of the behavior recognition algorithm later.

[0077] Step S102: Obtaining the annotation result of each picture frame of the video data;

[0078] In the embodiment of the present application, there are two ways to annotate the picture frames of the video data, namely manual annotation and automatic annotation. Manual annotation requires manual annotation of each picture frame, and automatic annotation is to automatically annotate the video data through the behavior recognition algorithm. It can be understood that the use of two methods to annotate the video data is to improve the accuracy and reliability of the annotation, so as to test the behavior recognition algorithm later. The specific manual annotation method is as follows:

[0079] Specifically, the video data is converted into picture frames, and the converted picture frames are displayed on the interface of the electronic device. Based on a manual method, it is judged whether the actions of the characters in the continuous picture frames are correct, and whether the characters in the picture frames exist completely in the pictures. For example, whether the students in the course video raise their hands, and the students' heads, hands, legs, and feet appear in the picture. If the actions are correct and the characters exist completely in the picture, each picture frame in the continuous picture frames is annotated to obtain annotated picture frames, that is, to obtain annotated results, wherein the annotated results include the annotated picture frames.

[0080] Further click on the rectangular box on the interface, and use the rectangular box to frame the person in the marked picture frame to obtain multiple continuous person frames, wherein each person frame includes the head, hands, legs, and feet of the person, and each person frame has corresponding attributes to record the action type in the person frame, such as the person coordinates, actions and other attributes in the person frame, and each person frame corresponds to a picture frame one by one.

[0081] In the embodiment of the present application, the action type can be classified by the character's hand movement, the stretch of the body, the moving position of the character, and the rotation direction of the character's face. For example, the character's hand movement from flat to raised can be regarded as an action type, and the character's movement from sitting still to standing, the character moving from one position to another, and the character's face turning from front to back can each be regarded as an action type.

[0082] In an embodiment of the present application, manual annotation of picture frames is performed by determining whether the action composed of continuous picture frames is complete, for example, whether the process of a student's hand movement from raising to lowering in a course video can be displayed in continuous picture frames. If the action is complete, the picture frames corresponding to the complete action are annotated.

[0083] Step S103: obtaining a marked event according to the marking result to determine a marked event list, wherein the marked event list includes a plurality of marked events, and each marked event includes a plurality of picture frames;

[0084] In the embodiment of the present application, the actions in the video are labeled in order to obtain different types of actions for comparison with the results of subsequent behavior recognition algorithms.

[0085] Specifically, the annotation result includes annotated picture frames. Each time consecutive picture frames are annotated, the consecutive picture frames are connected to obtain video data containing an action type. The video data containing an action type is used as an annotation event until all picture frames corresponding to videos with complete actions in all unannotated videos are annotated, and multiple video data containing only one action type are obtained, that is, multiple annotation events are obtained, wherein each annotation event is video data containing an action type, and the video data includes multiple picture frames, that is, each annotation event includes multiple picture frames, and each picture frame corresponds to a character frame one by one.

[0086] Furthermore, multiple annotated events are sorted, and the continuous character frames corresponding to the continuous multiple picture frames in each annotated event are grouped into a field, such as a JSON field. That is, each annotated event corresponds to a field one by one, and multiple annotated events have multiple fields. The multiple fields are grouped into a list in chronological order to obtain a annotated event list, such as a JSON list, wherein the annotated event list includes multiple annotated events.

[0087] Step S104: Based on the behavior recognition algorithm, the video data is recognized to obtain a recognition result corresponding to each picture frame of the video data;

[0088] In the embodiment of the present application, there are many human actions in the video data.

[0089] Specifically, the video data is identified based on the behavior recognition algorithm, that is, the video data is converted using the behavior recognition algorithm to obtain multiple picture frames, so as to further identify the multiple picture frames and obtain the recognition results corresponding to each picture frame of the video data, wherein the recognition results include the action type of the character in the picture frame and the credibility of the action type.

[0090] Please refer to Figure 2 , Figure 2 yes Figure 1 A detailed flow chart of step S104 in FIG.

[0091] like Figure 2 As shown, step S104 includes:

[0092] Step S141: identifying each picture frame of the video data to obtain a first character frame corresponding to each picture frame;

[0093] Specifically, a behavior recognition algorithm is used to identify each picture frame of the video data. If there is a person in the picture frame, a coordinate axis is constructed in the picture. For example, an xy coordinate is constructed with the upper left corner of the picture as the origin. According to the constructed coordinate axis, the width and height of the person in the picture are obtained, and further based on the width and height of the person, a character frame is drawn. The character frame is the first character frame, wherein the first character frame corresponds one to one to a picture frame, and the first character frame includes the complete person.

[0094] Step S142: identifying the first person frame corresponding to each picture frame to obtain a plurality of key points corresponding to each first person frame;

[0095] In an embodiment of the present application, the key points are the key points of a character, wherein the key points of a character are representative feature points in an image or video, such as the joints of a character's hands. By identifying the key points of a character, further analysis of the character's movements can be achieved.

[0096] Specifically, the first character frame corresponding to each picture frame is identified, that is, the character in the first character frame corresponding to each picture frame is identified, and all key points of the character in the first character frame are extracted to obtain multiple key points corresponding to the character in the first character frame.

[0097] Step S143: Connect multiple key points to obtain a skeleton model;

[0098] In the embodiment of the present application, the skeleton model is used to represent a simplified model of the human form.

[0099] Specifically, multiple key points corresponding to each picture frame are connected to obtain key point trajectories, that is, a skeleton model is obtained, wherein the skeleton model includes the key point trajectories, and each skeleton model corresponds to a picture frame one by one.

[0100] In the embodiment of the present application, the skeleton model is used to determine the actions performed by the character, so as to further identify the action type of the character based on the actions.

[0101] Step S144: obtaining a first action type according to the skeleton model;

[0102] Specifically, each picture frame corresponds to a skeleton model, and the skeleton model includes a key point trajectory. According to the skeleton model, a first action type is obtained, that is, it is identified whether the key point trajectory of the skeleton model corresponding to each consecutive picture frame has undergone obvious changes. If the key point trajectory has undergone obvious changes, the changed part in the key point trajectory is found to correspond to a certain part of the character, so as to judge the type of action made by the character according to the part, and obtain the first action type. For example, if the part in the key point trajectory that has undergone obvious changes corresponds to the hand position of the character, it means that the character is raising his hand, and the first action type is raised.

[0103] Step S105: obtaining actual events according to the recognition results to determine an actual event list, wherein the actual event list includes multiple actual events, and each actual event includes multiple picture frames;

[0104] Specifically, the recognition result includes a first action type. When it is recognized that a person appears in each of the consecutive picture frames, and the actions performed by the person in the consecutive picture frames are of the same action type, the consecutive picture frames corresponding to the person are marked, and the consecutive picture frames are connected to obtain video data containing an action type. The video data containing an action type is used as an actual event. The above steps are repeated to obtain multiple actual events, and the multiple actual events constitute an actual event set. The actual events in the actual event set are filtered based on the marked event list to obtain multiple filtered actual events. The multiple filtered actual events are arranged in chronological order to form an actual event list, wherein the actual event list includes multiple actual events, and each actual event includes multiple picture frames.

[0105] Please refer to Figure 3 , Figure 3 yes Figure 1 A detailed flow chart of step S105 in FIG.

[0106] like Figure 3 As shown, step S105 includes:

[0107] Step S151: determining whether the first action type in the recognition result is the same as the preset action type;

[0108] Specifically, based on the behavior recognition algorithm, determine whether the first action type in the recognition result is the same as the preset action type. When the first action type in the recognition result is the same as the preset action type, jump to step S152; otherwise, jump to step S159.

[0109] Please refer to Figure 4 , Figure 4 yes Figure 3 A detailed flowchart of step S151 in FIG.

[0110] like Figure 4 As shown, step S151 includes:

[0111] Step S1511: Based on the behavior recognition algorithm, the picture frame is recognized to obtain the credibility of the first action type corresponding to the picture frame;

[0112] In the embodiment of the present application, credibility is a measure for evaluating the degree of similarity of an action type. The closer the credibility is to 1, the more accurate the behavior recognition algorithm is in identifying the action.

[0113] Specifically, a behavior recognition algorithm is used to recognize the picture frame of the video data. If there is a person in the picture frame, the action type corresponding to the person in the picture frame and the credibility of the action type are output.

[0114] Step S1512: determining whether the credibility of the first action type is greater than or equal to a credibility threshold;

[0115] In an embodiment of the present application, before using a behavior recognition algorithm to identify picture frames of video data, it is necessary to preset a credibility threshold. The credibility threshold is used to compare with the credibility of the action type to determine whether the action type of the person in the picture frame recognized by the behavior recognition algorithm is correct, and the preset credibility threshold needs to be within a reasonable range. A credibility threshold that is too high or too low will cause some real behaviors to be missed. The specific reasonable range needs to be adjusted according to the specific scenario. For example, to identify the actions of students in a course video, the credibility threshold can be set to 0.8, which can accurately identify the actions of students in the course video.

[0116] Specifically, the first action type is an action type obtained by identifying the picture frame based on the behavior recognition algorithm. When the credibility of the first action type is greater than or equal to the credibility threshold, jump to step S1513; when the credibility of the first action type is less than the credibility threshold, jump to step S1514.

[0117] Step S1513: Determine that the first action type is the same as the preset action type;

[0118] In an embodiment of the present application, the preset action type is an action type predefined when training the behavior recognition algorithm so that the behavior recognition algorithm can classify and identify the action type. For example, when identifying the actions of students in a course video, it is necessary to preset the action types that the students will make, such as raising their hands, moving, etc.

[0119] In an embodiment of the present application, before using the behavior recognition algorithm to pre-mark the picture frames to obtain the actual events, a lower credibility threshold is set, such as 0.4 or 0.5. According to the lower credibility threshold, more action types can be obtained.

[0120] Specifically, the first action type is an action type obtained by identifying the picture frame based on the behavior recognition algorithm. When the credibility of the first action type is greater than or equal to the credibility threshold, it indicates that the first action type is the same as the preset action type. For example, the preset action types include raising hands and moving, and the credibility threshold is 0.4. The first action type recognized and output by the behavior recognition algorithm is raising hands, indicating that the first action type is within the preset action type range, and the credibility of the output first action type is 0.4. It is further obtained that the credibility of the first action type is the same as the credibility threshold, then the action recognition is considered correct, that is, the first action type is the same as the preset action type.

[0121] Step S1514: determining that the first action type is different from a preset action type;

[0122] Specifically, when the credibility of the first action type output by the behavior recognition algorithm for the picture frame is less than the credibility threshold, it is considered that the first action type is different from the preset action type.

[0123] In the embodiment of the present application, the preset credibility threshold is compared with the feasibility of the action type, which can improve the accuracy of the behavior recognition algorithm to further annotate more video data with the same action type as the actual video.

[0124] Step S152: pre-marking the picture frame corresponding to the current first action type to obtain a pre-marked picture frame;

[0125] Specifically, when the first action type corresponding to the picture frame in the recognition result is the same as the preset action type, the picture frame corresponding to the first action type is marked to obtain a pre-marked picture frame, wherein the pre-marked picture frame includes the first action type, and based on the constructed coordinate axis, a character frame in the picture frame is obtained, wherein the character frame includes multiple attributes to mark the action type corresponding to the character frame.

[0126] Step S153: obtaining the actual event based on the pre-marked picture frame;

[0127] Specifically, the pre-marked picture frames are picture frames marked by the behavior recognition algorithm. According to the time sequence, the pre-marked continuous picture frames are connected to obtain video data containing an action type. The video data containing an action type is used as an actual event until all the picture frames corresponding to the video data with complete actions in the unmarked video data are marked, and multiple video data containing only one action type are obtained, that is, multiple actual events are obtained, and the multiple actual events constitute an actual event data set.

[0128] Step S154: subtracting a preset value from the sequence number corresponding to the first start frame to obtain a first sequence number, and adding the preset value to the sequence number corresponding to the first end frame to obtain a second sequence number;

[0129] In an embodiment of the present application, the labeled event is video data obtained based on manual labeling, and the actual event is video data obtained based on labeling based on a behavior recognition algorithm, wherein each labeled event and each actual event include multiple picture frames, each labeled event includes a first start frame and a first end frame, and each actual event includes a second start frame and a second end frame, wherein the start frame is the picture frame corresponding to the first action in the event, and the end frame is the picture frame corresponding to the end action in the event, for example: for the action of raising hands, in the start frame, the character's elbow is vertically upward, and in the end frame, the character's elbow is placed horizontally.

[0130] In the embodiment of the present application, when the video data is pre-labeled based on the behavior recognition algorithm, only the actual video data is used as the basis. There will be certain errors in the behavior recognition algorithm identifying the actions in the video data. Moreover, due to the low credibility threshold set, the recognized actions may not match the actual ones, but they are considered to be correct, and wrong data is obtained, and then the video data corresponding to the actual events need to be screened.

[0131] Specifically, a marked event is selected from the marked event list, wherein the marked event includes a plurality of picture frames, and the plurality of picture frames are numbered in chronological order, that is, each picture frame corresponds to a sequence number one by one.

[0132] It is understandable that, for video data, each picture frame in the video data is numbered in chronological order, starting from 1 to the last picture frame. For example, if the video data includes 1000 picture frames, the video data is numbered from 1 to 1000.

[0133] Further, a preset value is subtracted from the sequence number corresponding to the first start frame in the selected marking event to obtain a first sequence number, and a preset value is added to the sequence number corresponding to the first end frame in the selected marking event to obtain a second sequence number.

[0134] In an embodiment of the present application, subtracting a preset value from the serial number corresponding to the first start frame and adding a preset value to the serial number corresponding to the first end frame actually increases the number of picture frames for the first start frame and the first end frame in the annotation event at the same time to obtain the annotation event after the number of picture frames is increased. The behavior recognition algorithm uses the annotation event after the number of picture frames is increased as a reference to improve the accuracy of identifying actions in the video. For example, to identify whether a student raises his hand in a course video, it is necessary to judge from several picture frames before the hand-raising action occurs. For example, assuming that the video data in which the hand-raising action actually occurs starts from the 1st frame, and there are 20 picture frames corresponding to the video data in which the hand-raising action actually occurs, the hand-raising action begins to appear from the 10th frame, and the hand is placed horizontally in the 20th frame. Under normal circumstances, there will be certain errors in the behavior recognition algorithm, such as the hand-raising action will be recognized from the 5th or 6th frame. In order to reduce the error, 5 frames are subtracted from the start frame and 5 frames are added to the end frame to increase the number of frames of the annotation event, thereby improving the accuracy of the behavior recognition algorithm.

[0135] Step S155: comparing the sequence number corresponding to the second start frame with the first sequence number, and comparing the sequence number corresponding to the second end frame with the second sequence number;

[0136] Specifically, based on the marked events after increasing the number of picture frames, wherein the marked events after increasing the number of picture frames include a first serial number and a second serial number, an actual event is selected from the actual event set, the actual event includes a second start frame and a second end frame, the serial number corresponding to the second start frame corresponding to the selected actual event is compared with the first serial number, and the serial number corresponding to the second end frame corresponding to the selected actual event is compared with the second serial number.

[0137] In an embodiment of the present application, the serial numbers corresponding to the picture frames in the actual event are compared with the serial numbers corresponding to the picture frames in the marked event after the number of picture frames is increased, so as to better identify the entire process of the action, so as to further screen out the actual events corresponding to the actions that conform to the actual video data.

[0138] Step S156: if the sequence number corresponding to the second start frame is greater than the first sequence number, and the sequence number corresponding to the second end frame is less than the second sequence number, the same actual event corresponding to the second start frame and the second end frame is filtered to obtain the filtered actual event, and the filtered actual event is added to the actual event list;

[0139] Specifically, when the sequence number corresponding to the second start frame of the actual event is greater than the first sequence number, and the sequence number corresponding to the second end frame of the actual event is less than the second sequence number, the same actual event corresponding to the second start frame and the second end frame is filtered to obtain the filtered actual event, and the filtered actual event is added to the actual event list.

[0140] In an embodiment of the present application, actual events are screened to remove erroneous actual events. For example, if it is necessary to identify a student's hand-raising action in a course video, and hand-raising and movement appear simultaneously in the same frame, the behavior recognition algorithm chooses to identify the movement action, which will result in an erroneous actual event.

[0141] Step S157: if the sequence number corresponding to the second start frame is smaller than the first sequence number, or the sequence number corresponding to the second end frame is larger than the second sequence number, then the current actual event is not screened;

[0142] Specifically, when the sequence number corresponding to the second start frame of the actual event is smaller than the first sequence number, or the sequence number corresponding to the second end frame of the actual event is larger than the second sequence number, it means that the current actual event does not include a complete action, and the current actual event will not be added to the actual event list.

[0143] It can be understood that a complete action is an action from start to end. For example, the action of students raising their hands in the course video is from raising their hands to putting them down. The complete action of students standing up is from sitting down to standing up, and then from standing to sitting down or from standing to leaving (leaving can be understood as the students not appearing in the video). Another example is the complete action of a group discussion is to only recognize 20% of the students' faces (indicating that relative to the direction of the podium, the direction of most faces is opposite to the direction of the podium), and then to increase from recognizing only 20% of the students' faces to recognizing more than 80% of the students' faces, thus ending the group discussion.

[0144] Step S158: Move to the next actual event;

[0145] Specifically, when the current actual event is not filtered, move to the next actual event adjacent to the current actual event, and further continue to execute step S155 until the actual events that meet the filtering conditions in the actual event set are filtered, such as adding the actual event including the complete action to the actual event list.

[0146] Step S159: not marking the picture frame corresponding to the current first action type;

[0147] Specifically, when the first action type corresponding to the picture frame in the recognition result is different from the preset action type, the picture frame corresponding to the current first action type is not marked.

[0148] Step S1510: moving to the next picture frame;

[0149] Specifically, when the first action type in the recognition result is different from the preset action type, the picture frame corresponding to the current first action type is not marked, and the process moves to the next picture frame of the picture frame corresponding to the current first action type, and continues to execute step S151.

[0150] In an embodiment of the present application, based on the behavior recognition algorithm and manual labeling of the picture frames, the obtained picture frame data is used to evaluate the behavior recognition algorithm to verify the accuracy of the behavior recognition algorithm.

[0151] Step S106: Calculate the intersection-over-union ratio of the labeled events and the actual events;

[0152] In an embodiment of the present application, the intersection-and-union ratio is used to calculate the error between the character frame corresponding to the picture frame of the manually labeled event and the character frame corresponding to the picture frame of the actual event labeled based on the behavior recognition algorithm, where the intersection-and-union ratio is the ratio of the intersection and the union.

[0153] Specifically, the specific steps for calculating the intersection-over-union ratio of the labeled events and the actual events are as follows.

[0154] Please refer to Figure 5 , Figure 5 yes Figure 1 A detailed flow chart of step S106 in FIG.

[0155] like Figure 5 As shown, step S106 includes:

[0156] Step S161: intersecting and merging the first character frame and the second character frame to obtain an intersection frame;

[0157] In an embodiment of the present application, the first character frame corresponds to a picture frame in the actual event, that is, each picture frame in the actual event corresponds to a first character frame, and the second character frame corresponds to a picture frame in the annotated event, that is, each picture frame in the annotated event corresponds to a second character frame, wherein the character frame is a rectangular frame, and each character frame has corresponding attributes, such as: the coordinates of the character frame, the action type of the character in the character frame, and other attributes.

[0158] Specifically, the second character frame corresponding to the picture frame in the marked event and the first character frame corresponding to the picture frame in the actual event are intersected to obtain an intersection frame, wherein the intersection frame is a partial area where the second character frame and the first character frame intersect.

[0159] Step S162: calculating the intersection-and-union ratio of the first person frame and the second person frame according to the intersection-and-union frame;

[0160] Specifically, the diagonal coordinates of the intersection frame are calculated, so as to further calculate the intersection ratio between the first person frame and the second person frame according to the diagonal coordinates of the intersection frame.

[0161] Please refer to Figure 6 , Figure 6 yes Figure 5 A detailed flow chart of step S162 in FIG.

[0162] like Figure 6 As shown, step S162 includes:

[0163] Step S1621: Calculate the diagonal coordinates of the intersection frame according to the first diagonal coordinates and the second diagonal coordinates to obtain the third diagonal coordinates;

[0164] In an embodiment of the present application, the first diagonal coordinates are the diagonal coordinates of the first character frame corresponding to the picture frame in the actual event, and the second diagonal coordinates are the diagonal coordinates of the second character frame corresponding to the picture frame in the annotated event, wherein the first diagonal coordinates and the second diagonal coordinates are known during annotation.

[0165] Specifically, the diagonal coordinates of the intersection frame are calculated according to the known first diagonal coordinates and the second diagonal coordinates to obtain the third diagonal coordinates, wherein the third diagonal coordinates are the diagonal coordinates of the intersection frame.

[0166] Step S1622: Calculate the intersection of the first person frame and the second person frame according to the third diagonal coordinates;

[0167] Specifically, the third diagonal coordinate is known, and the intersection of the first character frame and the second character frame is calculated according to the known third diagonal coordinate, wherein the intersection is a partial area where the first character frame and the second character frame overlap.

[0168] Step S1623: Calculate the union of the first character frame and the second character frame according to the first diagonal coordinates, the second diagonal coordinates and the intersection;

[0169] Specifically, according to the known first diagonal coordinates, the second diagonal coordinates and the intersection, the union of the first character frame and the second character frame is calculated, wherein the union is the area where the first character frame and the second character frame do not overlap.

[0170] Step S1624: Divide the union by the intersection to obtain the intersection-union ratio;

[0171] Specifically, the intersection-over-union ratio is equal to the union of the first character frame and the second character frame divided by the intersection of the first character frame and the second character frame.

[0172] Please refer to Figure 7 , Figure 7 is an example schematic diagram of calculating the intersection-over-union ratio provided in an embodiment of the present application;

[0173] like Figure 7As shown, the figure includes a first character frame A corresponding to the picture frame in the actual event, a second character frame B corresponding to the picture frame in the annotated event, and area C is the intersection frame obtained by intersecting the first character frame A and the second character frame B. a and e are the diagonal coordinate points of the first character frame A, b and d are the diagonal coordinate points of the second character frame B. The coordinates of a, b, e, and d are known. Let the coordinate of a be (xmin A , ymin A ), e coordinate is (xmax A , ymax A ), b coordinate is (xmin B , ymin B ), d coordinate is (xmax B , ymax B ), define the diagonal coordinates of the intersection box as (xmin AB , ymin AB )、(xmax AB , ymax AB ), the diagonal coordinates of the intersection box are calculated as follows:

[0174] xmin AB =max(xmin A , xmin B )

[0175] ymin AB =max(ymin A , ymin B )

[0176] xmax AB =min(xmax A , xmax B )

[0177] ymax AB =min(ymax A , ymax B )

[0178] Among them, max(xmin A , xmin B ) means taking xmin A and xmin B The maximum value between A , ymin B ) means taking ymin A and ymin B The maximum value between A , xmax B ) means taking xmax A and xmaxB The minimum value between A , ymax B ) means taking ymax A and ymax B The minimum value between .

[0179] Furthermore, given the diagonal coordinates of the intersection frame, the intersection of the first person frame A and the second person frame B is calculated. The specific method of calculating the intersection is as follows:

[0180] A∩B=(xmax AB -xmin AB )*(ymax AB -ymin AB )

[0181] W A = xmax A -xmin A

[0182] H A =ymax A -ymin A

[0183] W B = xmax B -xmin B

[0184] H B =ymax B -ymin B

[0185] Among them, A∩B is to calculate the intersection of the first character frame A and the second character frame B, W A is the width of the first character frame A, H A The height of the first character frame A, W B is the width of the second character frame B, H B The height of the second character frame B.

[0186] Further, the specific method of calculating the union of the first person frame A and the second person frame B is as follows:

[0187] A∪B=W A *H A +W B *H B -A∩B

[0188] Among them, W A *H A is the area of ​​the first character frame A, W B *H B is the area of ​​the second character frame B.

[0189] Furthermore, the intersection-over-union ratio of the first person frame A and the second person frame B is calculated as follows:

[0190]

[0191] Among them, IOU is the intersection-over-union ratio of the first character frame A and the second character frame B.

[0192] In the embodiment of the present application, the intersection-over-union ratio of the labeled events and the actual events is calculated so that the comparison result obtained by comparing the labeled events and the actual events is more accurate.

[0193] In an embodiment of the present application, it is necessary to perform preliminary screening on the actual events in the actual event list to screen out actual events that meet the conditions, for example: actual events that meet the conditions are actual events that include complete character actions, such as raising a hand.

[0194] In an embodiment of the present application, an intersection-and-union ratio threshold is preset, and the intersection-and-union ratio threshold is used to screen actual events that meet the conditions. The specific intersection-and-union ratio threshold can be set manually, for example: calling the interface for obtaining video data in the behavior recognition algorithm to obtain video data, setting different intersection-and-union ratio thresholds respectively, and manually comparing the recognition effects of the behavior recognition algorithm under different intersection-and-union ratio thresholds to select the intersection-and-union ratio threshold with the best recognition effect.

[0195] Perform a preliminary screening of the actual events in the actual event list. Specifically, select an actual event in the actual event list and select a labeled event in the labeled event list, calculate the intersection-and-union ratio between the character frame corresponding to the second start frame in the selected actual event and the character frame corresponding to the first start frame in the selected labeled event, and obtain a first intersection-and-union ratio, and calculate the intersection-and-union ratio between the character frame corresponding to the second end frame in the selected actual event and the character frame corresponding to the first end frame in the selected labeled event, and obtain a second intersection-and-union ratio.

[0196] Further, compare the first intersection-in-union ratio with the intersection-in-union ratio threshold, and compare the second intersection-in-union ratio with the intersection-in-union ratio threshold. If the first intersection-in-union ratio is greater than the intersection-in-union ratio threshold, or the second intersection-in-union ratio is greater than the intersection-in-union ratio threshold, it means that the selected actual event includes a complete action, and the selected actual event is filtered to obtain the filtered actual event. Further, compare the person in the filtered actual event with the person in the selected labeled event. If the first intersection-in-union ratio and the second intersection-in-union ratio are both less than the intersection-in-union ratio threshold, mark the selected actual event and the selected labeled event to indicate that the selected actual event does not meet the conditions, for example, the selected actual event does not include a complete character action, such as a hand-raising action. Repeat the above steps until all actual events that meet the conditions are filtered out to obtain actual events that meet the conditions.

[0197] Further, determine the most suitable actual event among all actual events that meet the conditions, for example: compare whether the person in the actual event that meets the conditions is the same person as that in the marked event, and whether the time when the people in the two events appear is the same. Specifically, select the actual event that meets the conditions and the marked event with the same number of frames for intersection and merger, or select the actual event that meets the conditions and the marked event with the closest number of frames for intersection and merger, calculate the intersection-and-union ratio of the selected actual event that meets the conditions and the marked event, repeat the above steps until all actual events that meet the conditions in the list are intersection-and-union with the marked event, and obtain multiple intersection-and-union ratios, wherein each actual event that meets the conditions corresponds to an intersection-and-union ratio one by one, and each intersection-and-union ratio corresponding to the actual event that meets the conditions also corresponds to a marked event, and the actual event that meets the conditions and the marked event corresponding to the same intersection-and-union ratio have the same number of frames or the closest number of frames.

[0198] Step S107: Determine the comparison result between the marked event list and the actual event list according to the intersection-union ratio;

[0199] Specifically, the actual event list includes multiple actual events that meet the conditions, each actual event that meets the conditions corresponds to an intersection-and-union ratio, and each intersection-and-union ratio corresponding to the actual event that meets the conditions also corresponds to a labeled event. The labeled events in the labeled event list with the same intersection-and-union ratio are compared with the action types corresponding to the actual events that meet the conditions in the actual event list to see if they are the same, and a comparison result is obtained. The comparison result between the labeled event list and the actual event list is determined based on the comparison result.

[0200] Please refer to Figure 8 , Figure 8 yes Figure 1 A detailed flow chart of step S107 in FIG.

[0201] like Figure 8 As shown, step S107 includes:

[0202] Step S171: Select an actual event with the largest intersection-and-union ratio in the actual event list to obtain a first successful matching event, and select a labeled event with the largest intersection-and-union ratio in the labeled event list to obtain a second successful matching event;

[0203] Specifically, compare the intersection-and-union ratios of all actual events in the actual event list to obtain the largest intersection-and-union ratio, and further obtain the actual event and the labeled event corresponding to the largest intersection-and-union ratio, wherein a larger intersection-and-union ratio indicates that the actual event and the labeled event corresponding to the intersection-and-union ratio are successfully matched, that is, the characters in the actual event and the labeled event and the time when the characters appear are the same, and the actual event with the largest intersection-and-union ratio is taken as the first successfully matched event, and the labeled event with the largest intersection-and-union ratio is taken as the second successfully matched event.

[0204] Step S172: Compare the action types of the first successful matching event and the second successful matching event to see if they are the same;

[0205] In an embodiment of the present application, the first successful match event is an actual event, which includes a first character frame and the attributes of the first character frame. The second successful match event is a marked event, which includes a second character frame and the attributes of the second character frame, wherein the attributes are used to record the action type in the character frame.

[0206] Specifically, compare whether the attributes of the first character frame corresponding to the actual event are the same as the attributes of the second character frame corresponding to the annotated event, that is, compare whether the action types of the first successful matching event and the second successful matching event are the same.

[0207] Step S173: marking the first successful matching event and the second successful matching event as a first mark;

[0208] Specifically, if the attributes of the first character box corresponding to the actual event are the same as the attributes of the second character box corresponding to the annotated event, that is, the action type corresponding to the actual event is the same as the action type corresponding to the annotated event, then the actual event and the annotated event are marked as the first mark, that is, the first successful match event and the second successful match event are marked as the first mark, and the actual event corresponding to the first successful match event is deleted from the actual event list, and the annotated event corresponding to the second successful match event is deleted from the annotated event list, where the first mark is represented by T.

[0209] Step S174: marking the first successful matching event and the second successful matching event as a second mark;

[0210] Specifically, if the attributes of the first character box corresponding to the actual event are different from the attributes of the second character box corresponding to the annotated event, that is, the action type corresponding to the actual event is different from the action type corresponding to the annotated event, then the actual event and the annotated event are marked as the second mark, that is, the first successful matching event and the second successful matching event are marked as the second mark, where the second mark is represented by F.

[0211] Repeat the above steps of comparing the first successful matching event with the second successful matching event, that is, repeat steps S171 to S174 until one of the lists is empty, and mark all events in the non-empty list as the second mark.

[0212] Step S175: determining a comparison result between the marked event list and the actual event list according to the first mark and the second mark;

[0213] Specifically, the first mark indicates that the action type of the marked event is the same as that of the actual event, which is denoted as T. The second mark indicates that the action type of the marked event is different from that of the actual event, which is denoted as F. The events in the marked event list and the actual event list are marked accordingly based on the comparison results, that is, the comparison results of the marked event list and the actual event list are determined, wherein the comparison results are the results obtained by comparing the action types of the marked event and the actual event, for example: if the action types are the same, the marked marked event and the actual event are T.

[0214] In an embodiment of the present application, the actual events are preliminarily screened, and then the intersection and union ratio of the screened actual events and the labeled events is calculated to determine whether the characters in the screened actual events are the same as those in the labeled events, and whether the time of appearance of the characters is the same, so as to obtain successfully matched actual events, and further compare the action types of the successfully matched actual events and the labeled events to obtain comparison results, which can compare expected actual events, and based on the comparison between events, the effect of the testing method of the behavior recognition algorithm can be improved.

[0215] Step S108: Determine a confusion matrix according to the comparison result;

[0216] In the embodiment of the present application, the confusion matrix is ​​used to evaluate the performance of the behavior recognition algorithm, such as the accuracy and recall rate of the algorithm.

[0217] Specifically, the confusion matrix includes TP, FP, FN, and TN, and the comparison result includes a first mark and a second mark. According to the first mark and the second mark, the TP, FP, FN, and TN corresponding to each action type are calculated to determine the confusion matrix corresponding to each action type. For example, the action type of the student raising his hand in the course video is identified, and all other actions identified are marked as F, and the comparison result corresponding to the action type of raising the hand is obtained to further confirm the confusion matrix corresponding to the action type of raising the hand. Among them, TP (Truepositives) is the number of samples correctly identified as positive samples, FP (Falsepositives) is the number of samples incorrectly identified as positive samples, FN (Falsenegatives) is the number of samples incorrectly identified as negative samples, TN (Truenegatives) is the number of samples correctly identified as negative samples, P (positive) is a positive sample, N (negative) is a negative sample, a positive sample is an event corresponding to the action type that meets the current definition, and a negative sample is an event corresponding to the action type that does not meet the current definition. For example, if the action type currently defined is raising the hand, then the event corresponding to raising the hand is a positive sample, and the events corresponding to other action types are all negative samples.

[0218] In an embodiment of the present application, when a behavior recognition algorithm is used to identify an action type, only the action type that needs to be identified currently will be marked as the first mark, and other action types will be marked as the second mark, that is, the comparison result corresponds to an action type one-to-one, and each action type corresponds to a confusion matrix one-to-one. For example: assuming that there are 4 action types in the course video, namely raising hands, standing up, group discussion, and going on stage, it is currently necessary to identify the action type of students raising their hands in the course video. When the action type of students raising their hands is identified, the event corresponding to the students raising their hands is marked as T. When other action types are identified, the corresponding events are marked as F, until one of the event lists is empty, and the comparison result is further obtained, and the confusion matrix corresponding to the currently identified action type is determined according to the comparison result.

[0219] Step S109: Calculate the evaluation result of the behavior recognition algorithm according to the confusion matrix;

[0220] Specifically, the data in the confusion matrix is ​​calculated to obtain the evaluation result of the behavior recognition algorithm.

[0221] Please refer to Fig. 9 , Fig. 9 yes Figure 1 A detailed flow chart of step S109 in FIG.

[0222] like Fig. 9 As shown, step S109 includes:

[0223] Step S191: Calculate the evaluation result corresponding to each action type according to the confusion matrix, wherein the evaluation result includes at least one of recall rate, accuracy rate, precision rate, sensitivity, error rate, and specificity;

[0224] In the embodiment of the present application, recall rate, accuracy rate, precision rate, sensitivity, error rate, and specificity are evaluation indicators of the testing method of the behavior recognition algorithm.

[0225] Specifically, according to TP, FP, FN, and TN in the confusion matrix, the evaluation result corresponding to each action type is calculated, that is, at least one of the recall rate, accuracy rate, precision rate, sensitivity, error rate, and specificity of each action type is calculated.

[0226] The specific methods for calculating recall, accuracy, precision, sensitivity, error rate, and specificity are as follows:

[0227] Recall:

[0228]

[0229] in, The recall rate is the ratio of the number of samples correctly identified as positive to the sum of the number of samples correctly identified as positive and the number of samples incorrectly identified as negative. It is used to evaluate the ability of behavior recognition algorithms to identify positive samples.

[0230] Accuracy:

[0231]

[0232] in, It is the ratio of the number of all correctly identified samples (including positive samples and negative samples) to the sum of all positive samples and all negative samples. The higher the accuracy, the stronger the recognition ability of the behavior recognition algorithm.

[0233] Precision:

[0234]

[0235] in, It is the ratio of the number of samples correctly identified as positive to the sum of the number of samples correctly identified as positive and the number of samples incorrectly identified as positive. The precision rate is used to measure the accuracy of the recognition effect of the behavior recognition algorithm.

[0236] Sensitivity:

[0237]

[0238] in, The sensitivity is the ratio of the number of samples correctly identified as positive samples to the number of all positive samples. It is used to evaluate the ability of the behavior recognition algorithm to identify positive samples.

[0239] Error rate:

[0240]

[0241] in, It is the ratio of the sum of the number of samples incorrectly identified as positive and the number of samples incorrectly identified as negative to the sum of all positive samples and all negative samples. The error rate and accuracy rate are opposite.

[0242] Specificity:

[0243]

[0244] in, It is the ratio of the number of samples correctly identified as negative samples to the number of all negative samples. Specificity is used to evaluate the ability of behavior recognition algorithms to identify positive samples.

[0245] In an embodiment of the present application, by calculating the intersection-and-union ratio of the labeled events and the actual events, the action types of the events with the largest intersection-and-union ratio are compared to further calculate the evaluation results. The present application can improve the quality of the behavior recognition algorithm test.

[0246] Please refer to Fig.10 , Fig.10 It is a schematic diagram of the overall structure of a testing method for a behavior recognition algorithm provided in an embodiment of the present application;

[0247] like Fig.10 As shown, the overall structure of the test method of the behavior recognition algorithm includes:

[0248] (1) Obtaining a labeled event list based on manual labeling, and obtaining an actual event set based on a behavior recognition algorithm. Further, obtaining a labeled event in the labeled event set, the labeled event including a first start frame and a first end frame, subtracting a preset value from a sequence number corresponding to the first start frame, and adding a preset value to a sequence number corresponding to the first end frame to obtain a labeled event with an increased number of frames. Based on the labeled event with an increased number of frames, filtering actual events in the actual event set to obtain an actual event list, wherein the actual time list includes actual events of multiple action types.

[0249] (2) Further, the marked events in the marked event list are compared with the actual events in the actual event list to preliminarily screen the actual events in the actual event list that include complete actions and complete characters, and obtain the screened actual events.

[0250] (3) Further, the actual events and the marked events after screening with the same number of frames or adjacent frames are selected for intersection and union comparison to obtain the actual events after screening with the largest intersection and union ratio, and obtain the successfully matched events, wherein the successfully matched events are the actual events after screening with the largest intersection and union ratio.

[0251] (4) Compare the action types of the successfully matched events and the labeled events to obtain the comparison results. Based on the comparison results, determine the confusion matrix corresponding to each action type and calculate it to obtain the evaluation results.

[0252] In an embodiment of the present application, by calculating the intersection-and-union ratio of labeled events and actual events, and comparing whether the action types of labeled events and actual events are the same based on the intersection-and-union ratio, a comparison result of the labeled event list and the actual event list is obtained to determine the confusion matrix, and further calculate the evaluation result of the behavior recognition algorithm. The present application can improve the quality of the behavior recognition algorithm test.

[0253] Please refer to Fig.11 , Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0254] like Fig.11 As shown, the electronic device 110 includes one or more processors 101 and a memory 102. Fig.11 A processor 101 is taken as an example.

[0255] The processor 101 and the memory 102 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0256] A processor for executing a test method for a behavior recognition algorithm in any embodiment of the present application, comprising: obtaining video data, wherein the video data includes picture frames; obtaining a labeling result for each picture frame of the video data; obtaining a labeling event based on the labeling result to determine a labeling event list, wherein the labeling event list includes multiple labeling events, and each labeling event includes multiple picture frames; based on a behavior recognition algorithm, identifying the video data to obtain a recognition result corresponding to each picture frame of the video data; obtaining an actual event based on the recognition result to determine an actual event list, wherein the actual event list includes multiple actual events, and each actual event includes multiple picture frames; calculating an intersection-and-union ratio of the labeling event to the actual event; determining a comparison result between the labeling event list and the actual event list based on the intersection-and-union ratio; determining a confusion matrix based on the comparison result; and calculating an evaluation result of the behavior recognition algorithm based on the confusion matrix.

[0257] By calculating the intersection-and-union ratio of labeled events and actual events, and determining the comparison results of the labeled event list and the actual event list based on the intersection-and-union ratio, to determine the confusion matrix, and further calculate the evaluation results of the behavior recognition algorithm, the present application can improve the quality of the behavior recognition algorithm test.

[0258] The memory 102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the test method of the behavior recognition algorithm in the embodiment of the present invention. The processor 101 executes various functional applications and data processing of the electronic device by running the non-volatile software programs, instructions and modules stored in the memory 102, that is, the test method of the behavior recognition algorithm in the above method embodiment is implemented.

[0259] The memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 102 may optionally include a memory remotely arranged relative to the processor 101. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0260] One or more modules are stored in the memory 102, and when executed by one or more processors 101, the test method of the behavior recognition algorithm in any of the above method embodiments is executed, for example, the test method described above is executed. Figure 1 The steps shown.

[0261] The present application also provides a non-volatile computer-readable storage medium, such as a memory including a program code, and the program code can be executed by a processor to perform the test method of the behavior recognition algorithm in any of the above method embodiments. For example, the non-volatile computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0262] An embodiment of the present application also provides a computer program product, which includes one or more program codes, which are stored in a non-volatile computer-readable storage medium. A processor of an electronic device reads the program code from the non-volatile computer-readable storage medium, and the processor executes the program code to complete the steps of the testing method of the behavior recognition algorithm provided in the above embodiment.

[0263] Through the description of the above implementation methods, ordinary technicians in this field can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or by hardware related to program code, and the program can be stored in a non-volatile computer-readable storage medium. The above-mentioned non-volatile computer-readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0264] Through the description of the above implementation methods, a person of ordinary skill in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the non-volatile computer-readable storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0265] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as mentioned above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for testing a behavior recognition algorithm, characterized in that: The method comprises: Acquire video data, wherein the video data includes picture frames; Obtaining a labeling result for each picture frame of the video data; Obtaining a labeled event according to the labeling result to determine a labeled event list, wherein the labeled event list includes a plurality of labeled events, and each labeled event includes a plurality of picture frames; Based on the behavior recognition algorithm, the video data is recognized to obtain a recognition result corresponding to each picture frame of the video data; According to the recognition result, an actual event is obtained to determine an actual event list, wherein the actual event list includes a plurality of actual events, and each actual event includes a plurality of picture frames; Calculating the intersection-over-union ratio of the marked event and the actual event; Determining a comparison result between the marked event list and the actual event list according to the intersection-over-union ratio; Determine a confusion matrix according to the comparison result; According to the confusion matrix, an evaluation result of the behavior recognition algorithm is calculated.

2. The method according to claim 1, characterized in that The recognition result includes a first action type, and the video data is recognized to obtain a recognition result corresponding to each picture frame of the video data, including: Identify each picture frame of the video data to obtain a first character frame corresponding to each picture frame; Identify the first character frame corresponding to each of the picture frames to obtain a plurality of key points corresponding to each of the first character frames; Connecting the multiple key points to obtain a skeleton model; A first action type is obtained according to the skeleton model.

3. The method according to claim 2, characterized in that The marked event includes a first start frame and a first end frame, the actual event includes a second start frame and a second end frame, and the actual event is obtained according to the recognition result to determine the actual event list, including: Determining whether the first action type in the recognition result is the same as a preset action type; If they are the same, pre-marking the picture frame corresponding to the first action type to obtain a pre-marked picture frame; Based on the pre-marked picture frame, obtaining the actual event; Subtract a preset value from the sequence number corresponding to the first start frame to obtain a first sequence number, and add a preset value to the sequence number corresponding to the first end frame to obtain a second sequence number; If the sequence number corresponding to the second start frame is greater than the first sequence number, and the sequence number corresponding to the second end frame is less than the second sequence number, the same actual event corresponding to the second start frame and the second end frame is filtered to obtain the filtered actual event, and the filtered actual event is added to the actual event list.

4. The method according to claim 3, characterized in that Determining whether the first action type in the recognition result is the same as the preset action type includes: Based on the behavior recognition algorithm, the picture frame is recognized to obtain the credibility of the first action type corresponding to the picture frame; If the credibility of the first action type is greater than or equal to the credibility threshold, it means that the first action type is the same as the preset action type; If the credibility of the first action type is less than the credibility threshold, it means that the first action type is different from the preset action type.

5. The method according to any one of claims 2 to 4, characterized in that: The annotation result of each picture frame includes a second person frame, and the calculating of the intersection-and-union ratio of the annotated event and the actual event includes: Intersecting and merging the first character frame and the second character frame to obtain an intersection frame; According to the intersection-and-union frame, an intersection-and-union ratio of the first person frame and the second person frame is calculated.

6. The method according to claim 5, characterized in that The first character frame includes first diagonal coordinates, and the second character frame includes second diagonal coordinates; Calculating the intersection-and-union ratio of the first person frame and the second person frame according to the intersection-and-union frame includes: Calculate the diagonal coordinates of the intersection frame according to the first diagonal coordinates and the second diagonal coordinates to obtain third diagonal coordinates; Calculate the intersection of the first character frame and the second character frame according to the third diagonal coordinates; Calculate the union of the first character frame and the second character frame according to the first diagonal coordinates, the second diagonal coordinates and the intersection; The union is divided by the intersection to obtain the intersection-union ratio.

7. The method according to any one of claims 1 to 4, characterized in that: Determining a comparison result between the marked event list and the actual event list according to the intersection-over-union ratio includes: Select an actual event with the largest intersection-and-union ratio in the actual event list to obtain a first successful matching event, and select a labeled event with the largest intersection-and-union ratio in the labeled event list to obtain a second successful matching event; Comparing whether the action types of the first successful matching event and the second successful matching event are the same; If they are the same, marking the first successful matching event and the second successful matching event as a first mark; If they are not the same, marking the first successful matching event and the second successful matching event as a second mark; A comparison result between the marked event list and the actual event list is determined according to the first mark and the second mark.

8. The method according to any one of claims 1 to 4, characterized in that: Calculating the evaluation result of the behavior recognition algorithm according to the confusion matrix includes: According to the confusion matrix, an evaluation result corresponding to each action type is calculated, wherein the evaluation result includes at least one of recall rate, accuracy rate, precision rate, sensitivity, error rate, and specificity.

9. An electronic device, characterized in that: include: At least one processor and at least one memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the testing method of the behavior recognition algorithm described in any one of claims 1-8.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the testing method of the behavior recognition algorithm described in any one of claims 1-8.

Images