Training method for abnormal behavior classification model, abnormal behavior classification method

Abstract

The present disclosure provides a training method for an abnormal behavior classification model and an abnormal behavior classification method, which can be applied to the financial field or other fields. The training method includes: obtaining training set data, the training set data including video feature information; inputting the video feature information into an RGB feature extraction module to obtain RGB feature information; inputting the video feature information into an optical flow feature extraction module to obtain optical flow feature information; inputting the RGB feature information and the optical flow feature information into a temporal feature extraction module to obtain temporal feature information, wherein the loss function of the temporal feature extraction module is a temporal similarity loss function; inputting the temporal feature information into a classification module to output the classification result of the action in the video at the current moment; and adjusting the training parameters of the abnormal behavior classification model according to the classification result and the classification label to obtain a trained abnormal behavior classification model. The present disclosure also provides a training device, equipment, storage medium and program product for an abnormal behavior classification model.

Description

Technical Field

[0001] The present disclosure relates to the financial field, specifically to the field of computer vision, and in particular to a training method for an abnormal behavior classification model, an abnormal behavior classification method and device thereof, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Action detection involves analyzing an uncut video and identifying the action category and corresponding start and end points at the end of the video. Online action detection uses a model to detect the action category in real time within the input video stream, without having to wait until the entire video is fed into the model before outputting information such as the action category.

[0003] Videos contain two types of information: appearance and motion, captured by the RGB stream and optical flow, respectively. Existing models based on online action detection algorithms cannot characterize the differences in the temporal information trends between the RGB and optical flows, nor can they reflect the correlation of temporal information within an action. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a training method, abnormal behavior classification method, device, equipment, medium and program product for an abnormal behavior classification model, which is used to characterize the changing trend of the temporal information between RGB stream and optical flow, reflect the correlation of the temporal information within the action, and improve the detection accuracy of the abnormal behavior classification model.

[0005] According to a first aspect of the present disclosure, a training method for an abnormal behavior classification model is provided, where the abnormal behavior classification model includes an RGB feature extraction module, an optical flow feature extraction module, a temporal feature extraction module, and a classification module; the method includes: obtaining training set data, where the training set data includes video feature information; inputting the video feature information into the RGB feature extraction module to obtain RGB feature information; inputting the video feature information into the optical flow feature extraction module to obtain optical flow feature information, where the loss function of the optical flow feature extraction module is an optical flow stability loss function; inputting the RGB feature information and the optical flow feature information into the temporal feature extraction module to obtain temporal feature information, where the loss function of the temporal feature extraction module is a temporal similarity loss function; inputting the temporal feature information into the classification module to output a classification result of the action in the video at the current moment; and adjusting the training parameters of the abnormal behavior classification model according to the classification result and the classification label to obtain a trained abnormal behavior classification model.

[0006] According to an embodiment of the present disclosure, the optical flow stability loss function is:

[0007]

[0008] Among them, L fs represents the optical flow stability loss function, represents the original optical flow features input into the optical flow feature extraction module at time t, represents the optical flow feature at the tth moment extracted by the optical flow feature extraction module, σ 2 Represents the variance of the original optical flow feature information.

[0009] According to an embodiment of the present disclosure, the temporal similarity loss function is:

[0010]

[0011] Among them, L ts represents the temporal similarity loss function, represents the RGB feature information at the tth moment output by the RGB feature extraction module, f t f Represents the optical flow feature information at the tth moment output by the optical flow feature extraction module.

[0012] According to an embodiment of the present disclosure, the RGB feature information and the optical flow feature information are input into the timing feature extraction module to obtain the timing feature information, including: converting the RGB feature information and the optical flow feature information into an m-dimensional RGB column vector matrix and an n-dimensional optical flow column vector matrix, where m is an integer greater than 2, and n is an integer greater than 2; based on the m-dimensional RGB column vector matrix and the n-dimensional optical flow column vector matrix, determining an L-dimensional timing information matrix, where L is equal to m+n; and inputting the L-dimensional timing information matrix into the timing feature extraction module to obtain the timing feature information.

[0013] According to an embodiment of the present disclosure, obtaining training set data includes: inputting training video data of a preset length into a video feature extraction module to obtain video feature information, wherein the video feature information includes image information of all frames of the training video data within the preset length; and obtaining training set data based on the video feature information, wherein the training set includes video feature information obtained based on video data of multiple different preset lengths.

[0014] According to an embodiment of the present disclosure, the optical flow feature extraction module includes multiple long short-term memory units; the RGB feature extraction module includes multiple long short-term memory units.

[0015] According to a second aspect of the present disclosure, a method for classifying abnormal behavior is provided, comprising: obtaining sample data to be classified, the sample data including video feature information; inputting the video feature information into an abnormal behavior classification model to obtain a classification result, wherein the abnormal behavior classification model is trained using the above-mentioned abnormal behavior classification model training method.

[0016] According to a third aspect of the present disclosure, a training device for an abnormal behavior classification model is provided, comprising: a first acquisition module for acquiring training set data, the training set data including video feature information; a first determination module for inputting the video feature information into an RGB feature extraction module to obtain RGB feature information; a second determination module for inputting the video feature information into an optical flow feature extraction module to obtain optical flow feature information, the loss function of the optical flow feature extraction module being an optical flow stability loss function; a third determination module for inputting the RGB feature information and the optical flow feature information into a temporal feature extraction module to obtain temporal feature information, the loss function of the temporal feature extraction module being a temporal similarity loss function; a first classification module for inputting the temporal feature information into the classification module to output a classification result of the action in the video at the current moment; and a training module for adjusting the training parameters of the abnormal behavior classification model according to the classification result and the classification label to obtain a trained abnormal behavior classification model.

[0017] According to a fourth aspect of the present disclosure, an abnormal behavior classification device is provided, comprising: a second acquisition module for acquiring sample data to be classified, the sample data including video feature information; a second classification module for inputting the video feature information into an abnormal behavior classification model to obtain a classification result, wherein the abnormal behavior classification model is trained according to the above method.

[0018] According to the fifth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the above-mentioned training method of the abnormal behavior classification model.

[0019] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor implements the above-mentioned training method of the abnormal behavior classification model.

[0020] According to a seventh aspect of the present disclosure, a computer program product is provided, which includes computer-executable instructions, and when the instructions are executed, they are used to implement the above-mentioned training method for the abnormal behavior classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 The following schematically illustrates a system architecture of a training method for an abnormal behavior classification model and an abnormal behavior classification method according to an embodiment of the present disclosure;

[0023] Figure 2The flowchart of the training method of the abnormal behavior classification model according to the embodiment of the present disclosure is schematically shown;

[0024] Figure 3 The flowchart of the method for obtaining time series feature information according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 4 The following schematically shows a flow chart of a method for obtaining training set data according to an embodiment of the present disclosure;

[0026] Figure 5 The following schematically shows a flow chart of a method for classifying abnormal behaviors according to an embodiment of the present disclosure;

[0027] Figure 6 The structure block diagram of the training device of the abnormal behavior classification model according to the embodiment of the present disclosure is schematically shown;

[0028] Figure 7 Schematically shows a structural block diagram of an abnormal behavior classification device according to an embodiment of the present disclosure; and

[0029] Figure 8 The block diagram schematically shows an electronic device suitable for the training method of abnormal behavior classification model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] An embodiment of the present disclosure provides a training method for an abnormal behavior classification model, which includes an RGB feature extraction module, an optical flow feature extraction module, a temporal feature extraction module, and a classification module; the method includes: obtaining training set data, the training set data including video feature information; inputting the video feature information into the RGB feature extraction module to obtain RGB feature information; inputting the video feature information into the optical flow feature extraction module to obtain optical flow feature information, and the loss function of the optical flow feature extraction module is an optical flow stability loss function; inputting the RGB feature information and the optical flow feature information into the temporal feature extraction module to obtain temporal feature information, and the loss function of the temporal feature extraction module is a temporal similarity loss function; inputting the temporal feature information into the classification module to output the classification result of the action in the video at the current moment; and adjusting the training parameters of the abnormal behavior classification model according to the classification result and the classification label to obtain the trained abnormal behavior classification model.

[0035] Figure 1 The diagram schematically illustrates a system architecture of an abnormal behavior classification model training method and an abnormal behavior classification method according to an embodiment of the present disclosure.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0037] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0038] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0039] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the training method for the abnormal behavior classification model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device for the abnormal behavior classification model provided in the embodiment of the present disclosure can generally be set in the server 105. The training method for the abnormal behavior classification model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the training device for the abnormal behavior classification model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] The following will be based on Figure 1 The system architecture described by Figures 2 to 5 The application review method of the disclosed embodiment is described in detail.

[0043] Figure 2 The flowchart of the training method of the abnormal behavior classification model according to the embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the method includes operations S201 to S206.

[0045] In operation S201 , training set data is acquired, where the training set data includes video feature information.

[0046] According to an embodiment of the present disclosure, an abnormal behavior classification model is used to analyze a video stream, determine the starting point and end point of an action in the video, and then determine the multiple actions present in the video; by analyzing the identified actions, the actions in the video are classified into abnormal behaviors and non-abnormal behaviors.

[0047] According to embodiments of the present disclosure, when video data is obtained for training an abnormal behavior classification model, the video data is converted into image data for each frame of the video to obtain training data. Videos contain not only the appearance of the displayed image but also information about the actions occurring within the video. Therefore, the video feature information contained in the training data can represent the appearance and action information represented by the video.

[0048] In operation S202, the video feature information is input into an RGB feature extraction module to obtain RGB feature information.

[0049] According to the embodiments of the present disclosure, the RGB feature extraction module can process the input video feature information and extract RGB feature information from the video feature information frame by frame. Specifically, the RGB feature extraction module is a single-stream network and can only extract RGB feature information from the video feature information. Although the video feature information contains optical flow information, the RGB feature extraction module cannot extract this optical flow information.

[0050] In operation S203, the video feature information is input into an optical flow feature extraction module to obtain optical flow feature information.

[0051] According to an embodiment of the present disclosure, video feature information includes optical flow information, and an optical flow feature extraction module is capable of extracting optical flow feature information from the video feature information. The optical flow feature extraction module is also a single-stream network and has a loss function, which is an optical flow stability loss function. Based on the optical flow stability loss function, the optical flow feature extraction module is capable of removing interference information present in the optical flow feature information, thereby ensuring the stability of the extracted optical flow feature information.

[0052] In operation S204 , the RGB feature information and the optical flow feature information are input into a temporal feature extraction module to obtain temporal feature information.

[0053] According to the embodiments of the present disclosure, both RGB feature information and optical flow feature information are extracted from video feature information. However, due to the different single-stream networks used for extraction, the extracted RGB feature information and optical flow feature information may differ. Specifically, the RGB feature information extracted by the RGB feature extraction module may contain more temporal features than the optical flow feature information in some parts of the same video, while containing fewer temporal features than the optical flow feature information in other parts.

[0054] According to an embodiment of the present disclosure, for the video feature information of the same video, RGB feature information and optical flow feature information can complement each other. RGB feature information is information extracted by the RGB single-stream network, that is, the information extracted by the RGB feature extraction module, and optical flow feature information is information extracted by the optical flow single-stream network. The abnormal behavior classification model is actually a dual-stream network model. The RGB feature information and optical flow feature information extracted by the dual-stream network model are combined and input into the timing feature extraction module, and the timing feature extraction module can extract timing feature information containing more information. The timing feature module has a loss function, which is a timing similarity loss function. The timing feature module can improve its own timing feature extraction ability based on the timing similarity loss function, and can extract stable timing feature information.

[0055] In operation S205 , the temporal feature information is input into a classification module, and a classification result of the action in the video at the current moment is output.

[0056] According to an embodiment of the present disclosure, the temporal feature information includes the temporal feature information of each frame of the input video stream, and the classification module determines the classification result of the video at the current moment based on the temporal feature information at the current moment and the temporal feature information before the current moment. For example, the input video stream includes action A and action B, and the current moment is the moment when action B is being executed. According to the temporal feature information of the image frame at the current moment, the relevant data of action B before the current moment is determined, including the starting point time of action B; if action B is an abnormal behavior, action B is determined to be an abnormal behavior based on historical information, and if the image frame data at the current moment determines that action B has not ended, the classification result of the action in the video at the current moment is output as abnormal behavior; from the starting point of action B to the current moment, the actions in the video are all non-abnormal behaviors, and according to the temporal feature information of the input current moment, the current action is determined to be an abnormal behavior, and the classification result of the action in the video at the current moment is output as abnormal behavior; if action B is associated with action A, action A is determined to be an abnormal behavior, and the action in the video at the current moment is output as abnormal behavior.

[0057] According to an embodiment of the present disclosure, the classification model adopts a cross entropy loss function as the loss function.

[0058] In operation S206 , the training parameters of the abnormal behavior classification model are adjusted according to the classification result and the classification label to obtain a trained abnormal behavior classification model.

[0059] According to an embodiment of the present disclosure, the training parameters of the abnormal behavior classification model are adjusted based on the classification results and the classification labels manually marked on the training set data, and the trained abnormal behavior classification model is obtained when the loss function meets the preset conditions.

[0060] According to an embodiment of the present disclosure, the loss function of the abnormal behavior classification model includes an optical flow stability loss function, a temporal similarity loss function, and a cross-entropy loss function. The loss function of the abnormal behavior classification model is obtained by weighted summation of the optical flow stability loss function, the temporal similarity loss function, and the cross-entropy loss function. The weight values of the above three loss functions are determined according to the actual situation. For example, for the classification of action A, the weight values of the above three loss functions are one case, and for the classification of action B, the weight values of the above three loss functions will change accordingly.

[0061] Specifically, the training parameters of the abnormal behavior classification model are adjusted according to the calculated loss function of the abnormal behavior classification model, and when the loss function meets the preset conditions, the trained abnormal behavior classification model is obtained.

[0062] The training method for the abnormal behavior classification model provided in this disclosure uses a dual-stream network to extract RGB feature information and optical flow information. This information is then combined to extract temporal feature information, which is then used to determine the classification result of the action in the video at the current moment. This improves the model's ability to detect abnormal behavior in real time. By setting loss functions in the optical flow feature extraction module and the temporal feature extraction module, the model's accuracy in abnormal behavior detection is improved.

[0063] According to an embodiment of the present disclosure, the optical flow stability loss function is:

[0064]

[0065] Among them, L fs represents the optical flow stability loss function, represents the original optical flow features input into the optical flow feature extraction module at time t, represents the optical flow feature at the tth moment extracted by the optical flow feature extraction module, σ 2 Represents the variance of the original optical flow feature information.

[0066] According to the embodiment of the present disclosure, both optical flow feature information and RGB feature information are determined from the same video feature information. RGB feature information can only represent the appearance information, while optical flow feature information represents the motion information between two frames. For the two frames of video feature information, the RGB feature information does not change, but the optical flow feature information does change. The optical flow stability loss function is based on the Laplace eigenmap. The original optical flow feature at time t is and the original optical flow features at time t+1 When the feature changes between the two are small, the optical flow feature extraction module obtains the optical flow feature at time t and the optical flow features at time t+1 The change of is also very small, ensuring that the extracted optical flow features can maintain the same change trend as the original optical flow features, and ensuring the stability of the optical flow feature information extracted by the optical flow feature extraction module.

[0067] According to an embodiment of the present disclosure, Represents the optical flow features at the tth moment extracted by the optical flow feature extraction module, and the optical flow features at the next moment It is necessary to use the optical flow features at time t Optical flow features extracted by the optical flow feature extraction module The optical flow feature information output by the optical flow feature extraction module Different, optical flow features After processing by the fully connected layer, the optical flow feature information is obtained

[0068] According to an embodiment of the present disclosure, the temporal similarity loss function is:

[0069]

[0070] Among them, L ts represents the temporal similarity loss function, represents the RGB feature information at the tth moment output by the RGB feature extraction module, f t f Represents the optical flow feature information at the tth moment output by the optical flow feature extraction module.

[0071] According to the embodiments of the present disclosure, the RGB feature information and the optical flow feature information have different forms of expression, but are both extracted based on the feature information of the same video segment. Although there are differences between the RGB feature information and the optical flow feature information, the information based on the timing part is similar. Combining the RGB feature information and the optical flow feature information and inputting them into the timing feature extraction module can supplement more timing information, but it also increases interference information. Since interference information exists in both the optical flow feature information and the RGB feature information, based on the similarity between the optical flow feature information and the RGB feature information, the interference information that does not provide data for either the optical flow feature information or the RGB feature information can be deleted, thereby improving the data extraction capability of the timing feature extraction module.

[0072] Figure 3 The flowchart of the method for obtaining time series feature information according to an embodiment of the present disclosure is schematically shown.

[0073] like Figure 3 As shown, the method includes operations S301 to S303.

[0074] In operation S301 , RGB feature information and optical flow feature information are converted into an m-dimensional RGB column vector matrix and an n-dimensional optical flow column vector matrix.

[0075] According to an embodiment of the present disclosure, before combining the RGB feature information and the optical flow feature information and inputting them into a temporal feature extraction module, the RGB feature information and the optical flow feature information are concatenated. The RGB feature information is converted into an m-dimensional RGB column vector matrix, and the optical flow feature information is converted into an n-dimensional optical flow column vector matrix. Where m is an integer greater than 2, and n is an integer greater than 2.

[0076] In operation S302 , an L-dimensional temporal information matrix is determined based on the m-dimensional RGB column vector matrix and the n-dimensional optical flow column vector matrix.

[0077] According to an embodiment of the present disclosure, after determining the RGB column vector matrix and the optical flow column vector matrix, the RGB data and the optical flow data are concatenated as column vectors to obtain an L-dimensional temporal information matrix, where L is equal to m+n.

[0078] In operation S303 , the L-dimensional time series information matrix is input into a time series feature extraction module to obtain time series feature information.

[0079] According to an embodiment of the present disclosure, after combining the feature dimensions, an L-dimensional time series information matrix is obtained, and the L-dimensional time series information matrix is input into a time series feature extraction module to obtain time series feature information.

[0080] For example, the RGB feature matrix is a 1024-dimensional column vector matrix, the optical flow feature matrix is a 2048-dimensional column vector matrix, and the resulting time series feature matrix is a 3072-dimensional column vector matrix. The time series feature extraction module extracts data from the above time series feature matrix to obtain time series feature information.

[0081] Figure 4 The flowchart of the method for obtaining training set data according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 4 As shown, the method includes operations S401 to S402.

[0083] In operation S401, training video data of a preset duration is input into a video feature extraction module to obtain video feature information.

[0084] According to an embodiment of the present disclosure, the abnormal behavior classification model processes training video data of different preset durations. For training video data of different preset durations, the video feature extraction module decomposes the video data to obtain image information of all frames of the training video data within the preset duration.

[0085] In operation S402, training set data is obtained based on the video feature information, wherein the training set includes video feature information obtained based on video data of multiple different preset durations.

[0086] According to an embodiment of the present disclosure, the training video information includes a 15-minute video A and a 10-minute video B. Videos A and B are decomposed frame by frame to obtain image data for each frame of video A and each frame of video B. Because the abnormal behavior classification model processes image data for each frame, and determines the classification result of the action in the current video based on the image data of the current frame and historical data, the preset duration can be different.

[0087] According to an embodiment of the present disclosure, the optical flow feature extraction module includes multiple long short-term memory (LSTM) units and an optical flow stability loss function. This function is used to reversely adjust the optical flow feature extraction module so that it can extract stable optical flow feature information. The RGB feature extraction module also includes multiple long short-term memory units for extracting RGB feature information.

[0088] This paper uses a dual-stream network to extract RGB feature information and optical flow information, combining this information to extract temporal feature information, and then determines the classification result of the action in the video at the current moment. This can improve the model's ability to detect abnormal behavior in real time. By setting loss functions in the optical flow feature extraction module and the temporal feature extraction module, the model's accuracy in detecting abnormal behavior is improved.

[0089] Figure 5 The flowchart of the abnormal behavior classification method according to the embodiment of the present disclosure is schematically shown.

[0090] like Figure 5 As shown, the method includes operations S501 to S502.

[0091] In operation S501 , sample data to be classified is obtained, where the sample data includes video feature information.

[0092] According to an embodiment of the present disclosure, video data to be classified is acquired, and sample data containing video feature information is obtained based on the video data to be classified. The sample data to be classified is image data for all frames of the video data to be classified.

[0093] In operation S502 , the video feature information is input into an abnormal behavior classification model to obtain a classification result.

[0094] According to an embodiment of the present disclosure, feature information of a video to be classified is input into a trained abnormal behavior analysis model, and the abnormal behavior analysis model outputs a classification result of the action in the video at the current moment.

[0095] For example, if the video to be classified is 5 minutes long, the video is processed to obtain the video feature information of all image frames in the 5-minute period. This video feature information is input into the abnormal behavior analysis model, which analyzes the video feature data in real time and outputs the classification results in real time.

[0096] Figure 6 The structural block diagram of the training device for the abnormal behavior classification model according to an embodiment of the present disclosure is schematically shown.

[0097] like Figure 6 As shown, the abnormal behavior classification model training device 600 of this embodiment includes a first acquisition module 601 , a first determination module 602 , a second determination module 603 , a third determination module 604 , a first classification module 605 and a training module 606 .

[0098] The first acquisition module 601 is used to acquire training set data, where the training set data includes video feature information. In one embodiment, the first acquisition module 601 can be used to perform the operation S201 described above, which will not be described in detail here.

[0099] The first determination module 602 is configured to input the video feature information into the RGB feature extraction module to obtain RGB feature information. In one embodiment, the first determination module 602 may be configured to perform the operation S202 described above, which will not be described in detail herein.

[0100] The second determination module 603 is configured to input the video feature information into the optical flow feature extraction module to obtain optical flow feature information. The loss function of the optical flow feature extraction module is an optical flow stability loss function. In one embodiment, the second determination module 603 can be configured to perform the operation S203 described above, which will not be described in detail here.

[0101] The third determination module 604 is configured to input the RGB feature information and the optical flow feature information into the temporal feature extraction module to obtain temporal feature information. The temporal feature extraction module uses a temporal similarity loss function. In one embodiment, the third determination module 604 can be configured to perform operation S204 described above and will not be further described here.

[0102] The first classification module 605 is used to input the temporal feature information into the classification module and output the classification result of the action in the video at the current moment. In one embodiment, the first classification module 605 can be used to perform the operation S205 described above, which will not be repeated here.

[0103] The training module 606 is used to adjust the training parameters of the abnormal behavior classification model according to the classification results and the classification labels to obtain a trained abnormal behavior classification model. In one embodiment, the training module 606 can be used to perform the operation S206 described above, which will not be repeated here.

[0104] According to an embodiment of the present disclosure, the second determining module 603 includes a first determining subunit, a second determining subunit, and a third determining subunit.

[0105] The first determining subunit is configured to convert the RGB feature information and the optical flow feature information into an m-dimensional RGB column vector matrix and an n-dimensional optical flow column vector matrix, where m is an integer greater than 2 and n is an integer greater than 2. In one embodiment, the first determining subunit may be configured to perform operation S301 described above, which will not be described in detail herein.

[0106] The second determining subunit is configured to determine an L-dimensional temporal information matrix based on the m-dimensional RGB column vector matrix and the n-dimensional optical flow column vector matrix, where L equals m + n. In one embodiment, the second determining subunit may be configured to perform operation S302 described above, which will not be described in detail here.

[0107] The third determining subunit is used to input the L-dimensional time series information matrix into the time series feature extraction module to obtain time series feature information. In one embodiment, the third determining subunit can be used to perform the operation S303 described above, which will not be repeated here.

[0108] According to an embodiment of the present disclosure, the first acquisition module 601 includes a first acquisition subunit and a second acquisition subunit.

[0109] The first acquisition subunit is configured to input the training video data of a preset duration into the video feature extraction module to obtain video feature information. The video feature information includes image information of all frames of the training video data within the preset duration. In one embodiment, the first acquisition subunit can be configured to perform operation S401 described above, which will not be further described here.

[0110] The second acquisition subunit is used to obtain training set data based on the video feature information, wherein the training set includes video feature information obtained based on video data of multiple different preset durations. In one embodiment, the second acquisition subunit can be used to perform the operation S402 described above, which will not be repeated here.

[0111] Figure 7 The following schematically shows a structural block diagram of an abnormal behavior classification device according to an embodiment of the present disclosure.

[0112] like Figure 7 As shown, the abnormal behavior classification device 700 of this embodiment includes a second acquisition module 701 and a second classification module 702 .

[0113] The second acquisition module 701 is configured to acquire sample data to be classified, wherein the sample data includes video feature information. In one embodiment, the second acquisition module 701 may be configured to execute the operation S501 described above, which will not be described in detail herein.

[0114] The second classification module 702 is configured to input the video feature information into an abnormal behavior classification model to obtain a classification result, wherein the abnormal behavior classification model is trained using the above method. In one embodiment, the second classification module 702 can be configured to perform the operation S502 described above, which will not be described in detail here.

[0115] Figure 8 The block diagram schematically shows an electronic device suitable for the training method of abnormal behavior classification model according to an embodiment of the present disclosure.

[0116] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0117] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0118] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 808 including a hard disk; and a communication portion 809 including a network interface card such as a LAN card or a modem. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 810 as needed, so that a computer program read therefrom can be installed into the storage portion 808 as needed.

[0119] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0120] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0121] Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the training method for the abnormal behavior analysis model provided in the embodiments of the present disclosure.

[0122] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0123] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0124] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the processor 801, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0125] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0128] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A training method for an abnormal behavior classification model, wherein the abnormal behavior classification model includes an RGB feature extraction module, an optical flow feature extraction module, a temporal feature extraction module, and a classification module; the method comprises: Acquire training set data, where the training set data includes video feature information; Inputting the video feature information into the RGB feature extraction module to obtain RGB feature information; Inputting the video feature information into the optical flow feature extraction module to obtain optical flow feature information, wherein the loss function of the optical flow feature extraction module is an optical flow stability loss function; Inputting the RGB feature information and the optical flow feature information into the temporal feature extraction module to obtain temporal feature information, wherein the loss function of the temporal feature extraction module is a temporal similarity loss function; Input the temporal feature information into the classification module, and output the classification result of the action in the video at the current moment; as well as Adjusting the training parameters of the abnormal behavior classification model according to the classification results and the classification labels to obtain a trained abnormal behavior classification model; The optical flow stability loss function is: in, represents the optical flow stability loss function, represents the original optical flow features input into the optical flow feature extraction module at time t, represents the optical flow feature at time t extracted by the optical flow feature extraction module, Represents the variance of the original optical flow feature information.

2. The method according to claim 1, wherein The temporal similarity loss function is: in, represents the temporal similarity loss function, Represents the RGB feature information at the tth moment output by the RGB feature extraction module, Represents the optical flow feature information at the tth moment output by the optical flow feature extraction module.

3. The method according to claim 1, wherein Inputting the RGB feature information and the optical flow feature information into a temporal feature extraction module to obtain temporal feature information includes: Converting the RGB feature information and the optical flow feature information into an m-dimensional RGB column vector matrix and an n-dimensional optical flow column vector matrix, wherein m is an integer greater than 2 and n is an integer greater than 2; Determine an L-dimensional temporal information matrix based on the m-dimensional RGB column vector matrix and the n-dimensional optical flow column vector matrix, where L is equal to m+n; and The L-dimensional time series information matrix is input into the time series feature extraction module to obtain time series feature information.

4. The method according to claim 1, wherein The obtaining of training set data comprises: Inputting training video data of a preset duration into a video feature extraction module to obtain video feature information, wherein the video feature information includes image information of all frames of the training video data within the preset duration; and The training set data is obtained based on the video feature information, wherein the training set includes video feature information obtained based on video data of multiple different preset durations.

5. The method according to claim 1, wherein The optical flow feature extraction module includes multiple long short-term memory units; the RGB feature extraction module includes multiple long short-term memory units.

6. A method for classifying abnormal behavior, comprising: Acquiring sample data to be classified, wherein the sample data includes video feature information; The video feature information is input into an abnormal behavior classification model to obtain a classification result, wherein the abnormal behavior classification model is trained using the method according to any one of claims 1-5.

7. A training device for an abnormal behavior classification model, comprising: A first acquisition module is used to acquire training set data, where the training set data includes video feature information; A first determination module is used to input the video feature information into an RGB feature extraction module to obtain RGB feature information; A second determination module is configured to input the video feature information into an optical flow feature extraction module to obtain optical flow feature information, wherein the loss function of the optical flow feature extraction module is an optical flow stability loss function; A third determination module is configured to input the RGB feature information and the optical flow feature information into a temporal feature extraction module to obtain temporal feature information, wherein the loss function of the temporal feature extraction module is a temporal similarity loss function; A first classification module, configured to input the temporal feature information into a classification module and output a classification result of the action in the video at the current moment; as well as A training module, configured to adjust the training parameters of the abnormal behavior classification model according to the classification results and the classification labels to obtain a trained abnormal behavior classification model; The optical flow stability loss function is: in, represents the optical flow stability loss function, represents the original optical flow features input into the optical flow feature extraction module at time t, represents the optical flow feature at time t extracted by the optical flow feature extraction module, Represents the variance of the original optical flow feature information.

8. An abnormal behavior classification device, comprising: A second acquisition module is used to acquire sample data to be classified, wherein the sample data includes video feature information; The second classification module is configured to input the video feature information into an abnormal behavior classification model to obtain a classification result, wherein the abnormal behavior classification model is trained according to the method according to any one of claims 1-6.

9. An electronic device comprising: one or more processors; a memory for storing one or more instructions, When the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 6.

11. A computer program product comprising computer executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images