Video action recognition method and device based on artificial intelligence, equipment and medium

Abstract

The invention relates to the technical field of medical rehabilitation training, in particular to a video action recognition method and device based on artificial intelligence, equipment and a medium. The method comprises the following steps: acquiring a target video of a patient in a rehabilitation training process, and obtaining M sub-fragments by using a sliding window formed by a preset duration window; classifying each sub-fragment by using a trained feature extractor and trained N binary classifiers constructed based on N action categories to obtain N target results, determining an optimal target result from the N target results, and determining the optimal target result according to the optimal target result. And obtaining the action category to which the binary classifier corresponding to the optimal target result belongs as the action category of the target video, and carrying out classification identification on each sub-fragment in the target video by utilizing the binary classifier corresponding to each action category so as to take the action category to which the binary classifier corresponding to the optimal classification result belongs as the action category of the target video. The interference of invalid information is reduced, and the video action recognition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical rehabilitation training technology, and in particular to an artificial intelligence-based video motion recognition method, device, equipment and medium. Background Art

[0002] With the development of machine learning and deep learning technologies, artificial intelligence (AI) technology has not only appeared in many sub-fields such as clinical nursing, registration, disease-assisted diagnosis, and remote consultation, but has also gained an important position in the field of rehabilitation training. In rehabilitation training, AI human motion recognition technology can build a three-dimensional model of human motion and obtain information on each key point of the human body, which can help patients evaluate the effect of rehabilitation training. Patients do not need to wear any equipment, but only need to obtain a video of the patient's training movements through a camera, and then use the trained motion recognition model to realize whether the patient's motion recognition meets the standard, so as to assist doctors in evaluating the rehabilitation effect.

[0003] For the trained action recognition model, some video actions are usually used as training sets. However, in the process of training with the training sets, the action recognition model takes the feature information of all frames of the video action as the main basis for prediction. Due to the slowness of the patient's movements during rehabilitation training, the feature information of most frames of the video action is invalid information, and the noise caused by this invalid information reduces the efficiency of video action recognition, thereby reducing the patient's experience during rehabilitation training. Therefore, how to improve the efficiency of video action recognition in rehabilitation training has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a video action recognition method, device, equipment and medium based on artificial intelligence to solve the problem of low efficiency of video action recognition in rehabilitation training.

[0005] In a first aspect, an embodiment of the present invention provides a video action recognition method based on artificial intelligence, the video action recognition method comprising: Obtain a target video to be identified, and use a sliding window composed of a preset time window to slide on the target video with a preset time step to obtain M sub-segments, wherein the video in the sliding window is a sub-segment when the sliding stops, and M is an integer greater than zero; Obtain a trained feature extractor and N trained binary classifiers constructed based on N action categories, and use the trained feature extractor to perform feature extraction on the M sub-segments respectively to obtain feature extraction results of the corresponding sub-segments, where N is an integer greater than zero; For any binary classifier, use the binary classifier to classify each feature extraction result respectively, obtain a classification result corresponding to the feature extraction result, and determine the classification result that meets the preset conditions as the target result corresponding to the binary classifier; All binary classifiers are traversed to obtain N target results, an optimal target result is determined from the N target results, and the action category to which the binary classifier corresponding to the optimal target result belongs is the action category of the target video.

[0006] In a second aspect, an embodiment of the present invention provides a video action recognition device based on artificial intelligence, the video action recognition device comprising: A video segmentation module is used to obtain a target video to be identified, and use a sliding window composed of a preset time window to slide on the target video with a preset time step to obtain M sub-segments, wherein the video in the sliding window is a sub-segment when the sliding stops, and M is an integer greater than zero; A feature extraction module, used to obtain a trained feature extractor and N trained binary classifiers constructed based on N action categories, and use the trained feature extractor to perform feature extraction on the M sub-segments respectively to obtain feature extraction results of the corresponding sub-segments, where N is an integer greater than zero; A target acquisition module is used to classify each feature extraction result using any binary classifier to obtain a classification result corresponding to the feature extraction result, and determine the classification result that meets the preset conditions as the target result corresponding to the binary classifier; The category confirmation module is used to traverse all binary classifiers to obtain N target results, determine the optimal target result from the N target results, and obtain the action category to which the binary classifier corresponding to the optimal target result belongs as the action category of the target video.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video action recognition method as described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video action recognition method as described in the first aspect is implemented.

[0009] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: Obtain a target video to be identified, use a sliding window composed of a preset time window, slide on the target video with a preset time step, and obtain M sub-segments, wherein the video in the sliding window when the sliding stops is a sub-segment, and M is an integer greater than zero; obtain a trained feature extractor and N trained binary classifiers constructed based on N action categories, use the trained feature extractor to extract features from the M sub-segments respectively, and obtain feature extraction results of the corresponding sub-segments, where N is an integer greater than zero; for any binary classifier, use the binary classifier to classify each feature extraction result respectively, and obtain a classification result of the corresponding feature extraction result. The result is determined, and the classification result that meets the preset conditions is determined as the target result of the corresponding binary classifier; all binary classifiers are traversed to obtain N target results, and the optimal target result is determined from the N target results, and the action category of the binary classifier corresponding to the optimal target result is obtained as the action category of the target video, and the binary classifier corresponding to each action category is used to classify and identify each sub-segment in the target video, so that the action category of the binary classifier corresponding to the optimal classification result is used as the action category of the target video, which reduces the interference of invalid information, improves the efficiency of video action recognition in rehabilitation training, and makes patients have a better experience in the rehabilitation training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment of a video action recognition method based on artificial intelligence provided by Embodiment 1 of the present invention; Figure 2 1 is a flow chart of a method for video action recognition based on artificial intelligence provided by Embodiment 1 of the present invention; Figure 3 It is a flow chart of training a feature extractor and a binary classifier in a video action recognition method provided by Embodiment 2 of the present invention; Figure 4 is a structural diagram of a video action recognition device based on artificial intelligence provided by Embodiment 3 of the present invention; Figure 5 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0019] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0020] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0021] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0022] The video action recognition method based on artificial intelligence provided by the first embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] The video motion recognition method is applied in medical rehabilitation training scenarios. The client may be a mobile phone. The mobile phone connects to the server via a wireless network to obtain corresponding background data. The doctor or patient obtains the rehabilitation training video during the rehabilitation training process via the mobile phone and performs motion recognition on the rehabilitation training video to detect whether the patient's motion conforms to the standard motion.

[0024] See also Figure 2, is a flow chart of a video action recognition method based on artificial intelligence provided by the first embodiment of the present invention. The video action recognition method can be applied to Figure 1 The client in the client, the computer device corresponding to the client connects to the server to obtain the corresponding background data, such as the target video to be identified and the action category, which is used to identify the action category of the target video. In addition, the server can also provide the client with the data collected by it, so that the client can call it to train the feature extractor and the binary classifier. Figure 2 As shown, the video action recognition method may include the following steps: Step S201, obtaining a target video to be identified, using a sliding window consisting of a preset time window, sliding on the target video with a preset time step, to obtain M sub-segments.

[0025] In the present invention, the target video to be identified refers to a video of an uncertain video action, the size of the sliding window refers to the time range corresponding to the image frame data covered by the sliding window, and the duration step refers to the time range that the sliding window slides over each time. In general, the duration step can be equal to the size of the sliding window, so that the image frame data can be obtained in sequence and continuously, but the duration step can also be smaller than or larger than the size of the sliding window. When the duration step is smaller than the size of the sliding window, the image frame data covered before and after the sliding window overlap, and when the duration step is larger than the size of the sliding window, the sliding window cannot achieve the sequential and continuous acquisition of the image frame data. Therefore, the target video to be identified is obtained, and a sliding window composed of a preset duration window is used to slide on the target video with a preset duration step to obtain M sub-segments, wherein the video in the sliding window when the sliding stops is a sub-segment, and M is an integer greater than zero. The target video is divided into multiple sub-segments using the sliding window composed of a preset duration window and a preset duration step, which can adaptively ensure the division accuracy requirements of the target video.

[0026] Among them, in one embodiment, the time step can be set equal to the size of the sliding window. After obtaining the target video to be identified, the sliding window is used to slide on the target video with a preset time step to obtain M sub-segments. It should be noted that if the target video cannot be completely divided by the sliding window, that is, when the target video is divided by the sliding window, there will be some remaining image frame data with a time range smaller than the size of the sliding window, then these remaining image frame data will be treated as a sub-segment separately to ensure the integrity of the target data.

[0027] In some possible implementations, the target video to be identified is a rehabilitation training video, that is, the movements of a patient during rehabilitation training in the field of medical rehabilitation training. According to the acquisition time of the rehabilitation training video, a sliding window consisting of a preset time window is used to slide on the rehabilitation training video with a preset time step. Each time the sliding window stops, the rehabilitation training video in the sliding window is used as a sub-segment, so that the rehabilitation training video can be divided into multiple sub-segments.

[0028] Step S202, obtaining a trained feature extractor and N trained binary classifiers constructed based on N action categories, and using the trained feature extractor to perform feature extraction on the M sub-segments respectively to obtain feature extraction results of the corresponding sub-segments.

[0029] In the present invention, a trained feature extractor refers to a feature extractor that has completed training, and the trained feature extractor is obtained by training based on existing video data. Action categories refer to categories distinguished according to different actions, for example, action categories include leg raising, arm raising, and squatting. The number of trained binary classifiers corresponds to the number of action categories, that is, one action category corresponds to one binary classifier. For example, the trained binary classifier corresponding to the leg raising action is obtained by training the video of the leg raising action.

[0030] In some possible implementations, the action categories are rehabilitation actions in the field of medical rehabilitation training, including elbow extension, raising both upper limbs, raising the affected upper limb, sit-ups, squats, etc. Then, a trained binary classifier is corresponding to each rehabilitation action. For example, a binary classifier corresponding to the elbow extension class is trained using an action video of the elbow extension class, and the trained binary classifier can only recognize the elbow extension action.

[0031] The trained binary classifier is trained based on the feature extraction results obtained by the trained feature extractor based on the video big data, and has the ability to accurately classify. Therefore, a trained feature extractor and N trained binary classifiers constructed based on N action categories are obtained, and the trained feature extractor is used to extract features of the M sub-segments respectively to obtain the feature extraction results of the corresponding sub-segments, where N is an integer greater than zero. The trained feature extractor obtained by training based on the video big data has a strong feature extraction capability, so that the feature extraction results of the sub-segments obtained by using the trained feature extractor can accurately characterize the features of the sub-segments.

[0032] Optionally, before obtaining the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, the following is further included: Obtain a first training set and a feature extractor, where the first training set includes unlabeled video samples; The feature extractor is trained unsupervisedly using the first training set to obtain a trained feature extractor.

[0033] The unlabeled video sample refers to a video of an uncertain action category, and at least one unlabeled video sample constitutes a first training set. The feature extractor can use a convolutional neural network, and then use the first training set to perform unsupervised training on the feature extractor to obtain a trained feature extractor, so that the trained feature extractor has a strong feature extraction capability.

[0034] Optionally, before obtaining the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, the following is further included: Obtain a second training set, where the second training set includes video samples containing labels of action categories; Use the trained feature extractor to extract features from all video samples in the second training set to obtain feature extraction results of the corresponding video samples. For any video sample, use the action category corresponding to the video sample and the corresponding feature extraction result as a training sample; For any action category, a binary classifier corresponding to the action category is constructed, and the binary classifier corresponding to the action category is trained using the training samples corresponding to the action category to obtain a trained binary classifier corresponding to the action category.

[0035] Among them, at least one video sample containing a label of an action category constitutes a second training set. Since the trained feature extractor has a strong feature extraction capability, the trained feature extractor is used to extract features from all video samples in the second training set to obtain feature extraction results of the corresponding video samples. Since the feature extraction results obtained by the trained feature extractor can accurately characterize the features of the video samples, for any video sample, the action category corresponding to the video sample and the corresponding feature extraction result are used as a training sample.

[0036] The binary classifier can use a support vector machine (SVM). Therefore, for any action category, a binary classifier corresponding to the action category is constructed, and the binary classifier corresponding to the action category is trained using the training samples corresponding to the action category to obtain a trained binary classifier corresponding to the action category, so that the trained binary classifier has the ability to accurately classify.

[0037] Step S203: for any binary classifier, use the binary classifier to classify each feature extraction result respectively, obtain the classification result corresponding to the feature extraction result, and determine the classification result that meets the preset conditions as the target result of the corresponding binary classifier.

[0038] In the present invention, a binary classifier refers to a trained binary classifier. The classification result may refer to the classification prediction value output by the binary classifier, which is used to characterize the probability that the sub-segment belongs to the action category corresponding to the binary classifier. The target result refers to the subsequent impact of the development of things or the expected state at the end of the stage, that is, the classification prediction value that meets the expectations. Therefore, for any binary classifier, each feature extraction result is classified separately using the binary classifier to obtain the classification result of the corresponding feature extraction result, and the classification result that meets the preset conditions is determined as the target result of the corresponding binary classifier. The classification results are screened by the preset conditions to obtain effective classification results and reduce the impact of invalid information.

[0039] Optionally, before determining that the classification result that meets the preset condition is the target result of the corresponding binary classifier, the following steps are included: For any binary classifier, sort the classification results corresponding to each sub-segment according to the segmentation order of the sliding window to obtain the sorting result of the corresponding binary classifier; Align all sorting results to obtain aligned classification results, and construct a classification result matrix based on the aligned classification results; Determining the classification result that meets the preset conditions as the target result of the corresponding binary classifier includes: For any binary classifier, the classification result that meets the preset conditions in the row or column corresponding to the binary classifier is determined from the classification result matrix as the target result of the corresponding binary classifier.

[0040] Among them, for any binary classifier, obtain the classification results corresponding to each sub-segment classified by the binary classifier, sort the classification results of all sub-segments according to the segmentation order of the sliding window on the target video, and obtain the sorting results of the binary classifier. Similarly, obtain the sorting results of each binary classifier, obtain N sorting results, align all the sorting results by row, obtain the classification results after row alignment, and confirm that the classification results after row alignment are the classification result matrix. Alternatively, align all the sorting results by column, obtain the classification results after column alignment, and confirm that the classification results after column alignment are the classification result matrix.

[0041] Since the classification result matrix can be the classification result after row alignment or column alignment, for any binary classifier, the classification result that meets the preset conditions in the row or column corresponding to the binary classifier is determined from the classification result matrix as the target result of the corresponding binary classifier.

[0042] Optionally, the classification result is a data value dimension, and for any binary classifier, the classification result that meets the preset conditions in the row or column corresponding to the binary classifier is determined from the classification result matrix as the target result of the corresponding binary classifier, including: For any binary classifier, determine the K elements with the largest data values ​​in the row or column corresponding to the binary classifier from the classification result matrix as the target result of the corresponding binary classifier, where K is an integer greater than zero.

[0043] Among them, for any binary classifier, all elements in the row or column corresponding to the binary classifier are obtained from the classification result matrix, the data values ​​of all elements are compared, and the K elements with the largest data values ​​are taken as the target results of the classifier, where K is an integer greater than zero, for example, K is 3.

[0044] In the field of medical rehabilitation training, the classification result refers to the probability value of the rehabilitation action. For example, the binary classifier corresponding to the elbow extension class is used to classify the feature extraction results of each sub-segment of the rehabilitation training video, and the probability value of the corresponding sub-segment belonging to the rehabilitation action of the elbow extension class is obtained. The probability value that meets the preset conditions is then confirmed as the target result of the binary classifier corresponding to the elbow extension class.

[0045] Step S204, traverse all binary classifiers to obtain N target results, determine the optimal target result from the N target results, and obtain the action category to which the binary classifier corresponding to the optimal target result belongs as the action category of the target video.

[0046] In the present invention, the optimal target result refers to a target result that meets the preset requirements. The preset requirements can be set according to actual needs. For example, if the number of requirements is 1, then one target result is finally determined from N target results. From step S203, it can be seen that a binary classifier will eventually obtain one target result, that is, the final target results are N.

[0047] The optimal target result is determined from N target results, and the action category to which the binary classifier corresponding to the optimal target result belongs is the action category of the target video. On the premise of accurately identifying the action category of the target video, the efficiency of video action recognition is also improved.

[0048] Optionally, determining the optimal target result from the N target results includes: For any target result, perform weighted summation on all elements in the target result to obtain the weighted summation result of the corresponding target result, and obtain the weighted summation result of all target results; Determine the target result with the largest weighted sum as the optimal target result.

[0049] Each target result includes K elements. For any target result, the data values ​​of all elements in the target result are normalized to obtain the normalized data value of each element. The sum of the normalized data values ​​of all elements is 1. The normalization method is: add the data values ​​of all elements to obtain the sum of the data values, and take the ratio of the data value of each element to the sum of the data values ​​as the normalized data value of the corresponding element. The normalized data value is used as the weight of the element, and the data values ​​of all elements in the target result are weighted and summed to obtain the weighted sum result of the target result.

[0050] Similarly, the weighted sum of all target results is obtained, and the weighted sum of all target results is compared, and the target result with the largest weighted sum is taken as the optimal target result.

[0051] In the field of medical rehabilitation training, it is assumed that there are three binary classifiers for rehabilitation movements, and three target results are obtained. The three target results are the target results of the binary classifier corresponding to the elbow extension class, the target results of the binary classifier corresponding to the sit-up class, and the target results of the binary classifier corresponding to the squat class. The optimal target result is determined from the three target results. Since each target result corresponds to a binary classifier for a rehabilitation movement, the rehabilitation movement to which the binary classifier corresponding to the optimal target result belongs is confirmed as the rehabilitation movement of the rehabilitation training video.

[0052] The embodiment of the present invention obtains a target video to be identified, uses a sliding window composed of a preset time window, slides on the target video with a preset time step, and obtains M sub-segments, wherein the video in the sliding window when the sliding stops is a sub-segment, and M is an integer greater than zero; obtains a trained feature extractor and N trained binary classifiers constructed based on N action categories, uses the trained feature extractors to extract features from the M sub-segments respectively, and obtains feature extraction results of the corresponding sub-segments, and N is an integer greater than zero; for any binary classifier, uses the binary classifier to classify each feature extraction result respectively , obtain the classification result of the corresponding feature extraction result, determine the classification result that meets the preset conditions as the target result of the corresponding binary classifier; traverse all binary classifiers to obtain N target results, determine the optimal target result from the N target results, and obtain the action category of the binary classifier corresponding to the optimal target result as the action category of the target video, and use the binary classifier corresponding to each action category to classify and identify each sub-segment in the target video, so as to use the action category of the binary classifier corresponding to the optimal classification result as the action category of the target video, thereby reducing the interference of invalid information and improving the efficiency of video action recognition.

[0053] See also Figure 3 , is a flow chart of the feature extractor and binary classifier training in a video action recognition method provided by the second embodiment of the present invention. Figure 3 As shown, the training of the feature extractor and the binary classifier in the video action recognition method may include the following steps: Step S301, obtaining N preset action categories and historical videos corresponding to each action category, and taking a historical video and the corresponding action category as a group of training samples.

[0054] The description of the action categories will not be repeated here, and please refer to the description in the above embodiment 1.

[0055] In the present invention, N action categories are preset according to the video action to be identified, and the preset N action categories and the historical videos corresponding to each action category are obtained. Each historical video corresponds to an action category, which is equivalent to labeling each historical video, and then a historical video and the corresponding action category are used as a group of training samples.

[0056] Step S302, construct a feature extractor and a binary classifier corresponding to each action category, copy N feature extractors, and use each binary classifier and a feature extractor as a classification model for the corresponding action category.

[0057] Among them, the classification model is composed of a binary classifier and a feature extractor. Therefore, a feature extractor and a binary classifier corresponding to each action category are constructed, N feature extractors are copied, and each binary classifier is respectively associated with a feature extractor as the classification model of the corresponding action category, that is, one action category corresponds to one classification model, wherein the parameters of the feature extractor in the classification model corresponding to each action category are the same.

[0058] It should be noted that training a convolutional neural network is equivalent to training a binary classifier and a feature extractor at the same time, so the classification model can use a convolutional neural network.

[0059] Step S303: for any action category, use the training samples corresponding to the action category to train the classification model corresponding to the action category to obtain the training result, and adjust the parameters of the binary classifier corresponding to the action category based on the training result.

[0060] In the present invention, the training result refers to the classification result prediction value output by the classification model during the training process. Therefore, after obtaining the training samples and classification model of each action category, for any action category, the classification model corresponding to the action category is trained using the training samples corresponding to the action category to obtain the training result. Based on the training result, the parameters of the binary classifier corresponding to the action category are adjusted, so that the parameters of the binary classifier are set more rigorously, and the classification result is more reliable and accurate.

[0061] Among them, the classification model is supervised trained. For any action category, the classification model corresponding to the action category is trained using the training samples corresponding to the action category to obtain the classification result prediction value, and the loss between the classification result prediction value and the action category of the training sample is calculated. According to the loss, the parameters of the binary classifier in the classification model are adjusted.

[0062] It should be noted that the action category of the training sample refers to the label that is manually annotated on the training sample in advance, and the label represents the actual action category of the training sample.

[0063] Similarly, the classification model corresponding to each action category is trained, and the parameters of the binary classifier in the classification model are adjusted according to the training results. Step S304: combine the training results of all action categories, and adjust the parameters of all feature extractors in a unified manner based on the combined results to obtain a trained feature extractor and N trained binary classifiers.

[0064] In the present invention, by training the classification model of each action category, the training results of all action categories are obtained. Therefore, the training results of all action categories are combined, and the parameters of all feature extractors are uniformly adjusted based on the combined results to ensure the consistency of feature extraction capabilities of the feature extractors and reduce feature extraction errors.

[0065] Among them, according to the training results of all action categories, the parameters of the feature extractor in the classification model of each action category are obtained, and the parameter mean is calculated according to the parameters of all feature extractors, and then the parameters of all feature extractors are uniformly adjusted to the parameter mean, thereby obtaining the trained feature extractor and N trained binary classifiers.

[0066] The embodiment of the present invention obtains N preset action categories and historical videos corresponding to each action category, and uses a historical video and a corresponding action category as a group of training samples; constructs a feature extractor and a binary classifier corresponding to each action category, copies N feature extractors, and uses each binary classifier and a feature extractor as a classification model of the corresponding action category; for any action category, uses the training sample corresponding to the action category to train the classification model corresponding to the action category to obtain a training result, and adjusts the parameters of the binary classifier corresponding to the action category based on the training result; combines the training results of all action categories, and uniformly adjusts the parameters of all feature extractors based on the combined result to obtain a trained feature extractor and N trained binary classifiers, and uses the training results of the classification model of each action category to adjust the parameters of the binary classifier in the classification model, so that the parameters of the trained binary classifier are set more rigorously and the classification results are more reliable and accurate, and then the parameters of all feature extractors are uniformly adjusted according to the training results of all action categories to ensure that the feature extraction capabilities of the trained feature extractors are consistent and reduce feature extraction errors. Corresponding to the video action recognition method of the above embodiment, Figure 4 A structural block diagram of a video action recognition device based on artificial intelligence provided by Embodiment 3 of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0067] See also Figure 4 , the video action recognition device comprises: The video segmentation module 41 is used to obtain a target video to be identified, and use a sliding window composed of a preset time window to slide on the target video with a preset time step to obtain M sub-segments, wherein the video in the sliding window when the sliding stops is a sub-segment, and M is an integer greater than zero; A feature extraction module 42 is used to obtain a trained feature extractor and N trained binary classifiers constructed based on N action categories, and use the trained feature extractor to perform feature extraction on the M sub-segments respectively to obtain feature extraction results of the corresponding sub-segments, where N is an integer greater than zero; The target acquisition module 43 is used to classify each feature extraction result using the binary classifier for any binary classifier, obtain the classification result corresponding to the feature extraction result, and determine the classification result that meets the preset conditions as the target result of the corresponding binary classifier; The category confirmation module 44 is used to traverse all binary classifiers to obtain N target results, determine the optimal target result from the N target results, and obtain the action category of the binary classifier corresponding to the optimal target result as the action category of the target video.

[0068] Optionally, the feature extraction module 42 includes: A sample acquisition unit is used to acquire N preset action categories and historical videos corresponding to each action category before acquiring the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, and use a historical video and the corresponding action category as a set of training samples; A model acquisition unit is used to construct a feature extractor and a binary classifier corresponding to each action category, replicate N feature extractors, and use each binary classifier and a feature extractor as a classification model for the corresponding action category; A model training unit is used to train a classification model corresponding to any action category using training samples corresponding to the action category to obtain a training result, and adjust parameters of a binary classifier corresponding to the action category based on the training result; The parameter adjustment unit is used to combine the training results of all action categories, and based on the combined results, uniformly adjust the parameters of all feature extractors to obtain a trained feature extractor and N trained binary classifiers.

[0069] Optionally, the feature extraction module 42 further includes: A first training set acquisition unit, used for acquiring a first training set and a feature extractor before acquiring a trained feature extractor and trained N binary classifiers constructed based on N action categories, wherein the first training set includes unlabeled video samples; The first training unit is used to perform unsupervised training on the feature extractor using the first training set to obtain a trained feature extractor.

[0070] Optionally, the feature extraction module 42 further includes: The second training set acquisition unit acquires a second training set before acquiring the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, wherein the second training set includes video samples containing labels of the action categories; A second sample acquisition unit is used to use the trained feature extractor to extract features from all video samples in the second training set to obtain feature extraction results corresponding to the video samples, and for any video sample, the action category corresponding to the video sample and the corresponding feature extraction result are used as a training sample; The second training unit constructs a binary classifier corresponding to any action category, and uses the training samples corresponding to the action category to train the binary classifier corresponding to the action category to obtain the trained binary classifier corresponding to the action category.

[0071] Optionally, the target acquisition module 43 includes: A result sorting unit, used for sorting the classification results corresponding to each sub-segment according to the segmentation order of the sliding window for any binary classifier before determining the classification result that meets the preset condition as the target result corresponding to the binary classifier, to obtain the sorting result corresponding to the binary classifier; A matrix construction unit is used to align all sorting results to obtain aligned classification results, and to construct a classification result matrix according to the aligned classification results; The target result acquisition unit is used to determine, for any binary classifier, from the classification result matrix, the classification result that meets the preset conditions in the row or column corresponding to the binary classifier as the target result of the corresponding binary classifier.

[0072] Optionally, if the classification result is a data value dimension, the target result acquisition unit includes The target screening subunit is used to determine, for any binary classifier, from the classification result matrix, the K elements with the largest data values ​​in the row or column corresponding to the binary classifier as the target result of the corresponding binary classifier, where K is an integer greater than zero.

[0073] Optionally, the category confirmation module 44 includes: A result summing unit, for any target result, performs weighted summation on all elements in the target result to obtain a weighted summation result corresponding to the target result, and obtains a weighted summation result of all target results; The optimal result confirmation unit is used to determine that the target result with the largest weighted sum result is the optimal target result.

[0074] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0075] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned video action recognition method embodiments are implemented.

[0076] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 5This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0077] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0079] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0080] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0083] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A video action recognition method based on artificial intelligence, characterized in that: The video action recognition method comprises: Obtain a target video to be identified, and use a sliding window composed of a preset time window to slide on the target video with a preset time step to obtain M sub-segments, wherein the video in the sliding window is a sub-segment when the sliding stops, and M is an integer greater than zero; Obtain a trained feature extractor and N trained binary classifiers constructed based on N action categories, and use the trained feature extractor to perform feature extraction on the M sub-segments respectively to obtain feature extraction results of the corresponding sub-segments, where N is an integer greater than zero; For any binary classifier, use the binary classifier to classify each feature extraction result respectively, obtain a classification result corresponding to the feature extraction result, and determine the classification result that meets the preset conditions as the target result corresponding to the binary classifier; All binary classifiers are traversed to obtain N target results, an optimal target result is determined from the N target results, and the action category to which the binary classifier corresponding to the optimal target result belongs is the action category of the target video.

2. The video action recognition method according to claim 1, characterized in that: Before obtaining the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, it also includes: Obtaining N preset action categories and historical videos corresponding to each action category, and using a historical video and the corresponding action category as a set of training samples; Construct a feature extractor and a binary classifier corresponding to each action category, copy N feature extractors, and use each binary classifier with a feature extractor as a classification model for the corresponding action category; For any action category, use the training samples corresponding to the action category to train the classification model corresponding to the action category to obtain a training result, and adjust the parameters of the binary classifier corresponding to the action category based on the training result; The training results of all action categories are combined, and based on the combined results, all feature extractors are uniformly parameter adjusted to obtain a trained feature extractor and N trained binary classifiers.

3. The video action recognition method according to claim 1, characterized in that: Before obtaining the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, it also includes: Obtaining a first training set and a feature extractor, wherein the first training set includes unlabeled video samples; The feature extractor is unsupervisedly trained using the first training set to obtain a trained feature extractor.

4. The video action recognition method according to claim 3, characterized in that: Before obtaining the trained feature extractor and the trained N binary classifiers constructed based on the N action categories, it also includes: Obtain a second training set, wherein the second training set includes video samples containing labels of action categories; Using the trained feature extractor to perform feature extraction on all video samples in the second training set to obtain feature extraction results of corresponding video samples, and for any video sample, using the action category corresponding to the video sample and the corresponding feature extraction result as a training sample; For any action category, a binary classifier corresponding to the action category is constructed, and the binary classifier corresponding to the action category is trained using the training samples corresponding to the action category to obtain a trained binary classifier corresponding to the action category.

5. The video action recognition method according to any one of claims 1 to 4, characterized in that: Before determining that the classification result that meets the preset condition is the target result corresponding to the binary classifier, it includes: For any binary classifier, sort the classification results corresponding to each sub-segment according to the segmentation order of the sliding window to obtain the sorting result of the corresponding binary classifier; Aligning all sorting results to obtain aligned classification results, and constructing a classification result matrix according to the aligned classification results; Determining the classification result that meets the preset conditions as the target result corresponding to the binary classifier includes: For any binary classifier, the classification result that meets the preset conditions in the row or column corresponding to the binary classifier is determined from the classification result matrix as the target result corresponding to the binary classifier.

6. The video action recognition method according to claim 5, characterized in that: The classification result is a data value dimension, and for any binary classifier, determining from the classification result matrix the classification result that satisfies a preset condition in a row or column corresponding to the binary classifier as a target result corresponding to the binary classifier includes: For any binary classifier, determine from the classification result matrix the K elements with the largest data values ​​in the row or column corresponding to the binary classifier as the target result corresponding to the binary classifier, where K is an integer greater than zero.

7. The video action recognition method according to claim 6, characterized in that: The determining the optimal target result from the N target results comprises: For any target result, perform weighted summation on all elements in the target result to obtain a weighted summation result corresponding to the target result, and obtain a weighted summation result of all target results; Determine the target result with the largest weighted sum result as the optimal target result.

8. A video action recognition device based on artificial intelligence, characterized in that: The video action recognition device comprises: A video segmentation module is used to obtain a target video to be identified, and use a sliding window composed of a preset time window to slide on the target video with a preset time step to obtain M sub-segments, wherein the video in the sliding window is a sub-segment when the sliding stops, and M is an integer greater than zero; A feature extraction module, used to obtain a trained feature extractor and N trained binary classifiers constructed based on N action categories, and use the trained feature extractor to perform feature extraction on the M sub-segments respectively to obtain feature extraction results of the corresponding sub-segments, where N is an integer greater than zero; A target acquisition module is used to classify each feature extraction result using any binary classifier to obtain a classification result corresponding to the feature extraction result, and determine the classification result that meets the preset conditions as the target result corresponding to the binary classifier; The category confirmation module is used to traverse all binary classifiers to obtain N target results, determine the optimal target result from the N target results, and obtain the action category to which the binary classifier corresponding to the optimal target result belongs as the action category of the target video.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the video action recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video action recognition method according to any one of claims 1 to 7 is implemented.

Images