Action detection method and system, computer equipment and storage medium

Abstract

The invention relates to an action detection method and system, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-detected video frame; extracting multi-time scale information in the to-be-detected video frame, and respectively obtaining a fast video feature and a slow video feature; determining a current action according to the slow video feature and a preset action; constructing a classification network by fusing the slow video features and the fast video features; and detecting the current action based on the classification network to obtain a detection result that the current action is a correct action or a wrong action. Through the method and the device, the current action in the to-be-detected video frame can be identified, the classification network is constructed by fusing the features of the multi-time scale information, the current action can be detected more accurately through the classification network, and a detection result indicating whether the current action is correct or not is obtained.

Description

Technical Field

[0001] The present application relates to the field of medical imaging technology, and in particular to an action detection method, system, computer device and storage medium. Background Art

[0002] In recent years, with the further deepening of research in the field of artificial intelligence endoscopic diagnosis, especially in minimally invasive surgeries such as laparoscopy, artificial intelligence methods are combined to further analyze the surgical process. By detecting whether the doctor's movements during the operation are correct, it is more conducive to the doctor's understanding and analysis of the surgical process. Moreover, in surgical training and education, motion detection can help trainees immediately identify and correct incorrect movements, thereby improving training effectiveness.

[0003] Most of the current motion detection methods only recognize surgical movements and cannot analyze the unqualified or incorrect parts of surgical movements, which cannot meet the actual needs of surgery or training. Therefore, there is an urgent need for a method that can accurately detect whether the movement is correct. Summary of the invention

[0004] Based on this, it is necessary to provide an action detection method, system, computer device and storage medium that can accurately detect whether an action is correct in order to solve the above technical problems.

[0005] In a first aspect, an action detection method is provided in this embodiment, including:

[0006] Get the video frame to be tested;

[0007] Extracting multi-time scale information in the video frame to be tested to obtain fast video features and slow video features respectively;

[0008] Determining a current action according to the slow video feature and the preset action;

[0009] By fusing the fast video feature and the fast video feature, constructing a classification network;

[0010] The current action is detected based on the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

[0011] In some embodiments, extracting the multi-time scale information in the video frame to be tested to obtain fast video features and slow video features respectively includes:

[0012] Sampling the video frames to be tested at different sampling rates to obtain video frame sequences at different time scales;

[0013] Based on a pre-trained feature extraction network, time scale information of video frame sequences at different time scales is extracted respectively to obtain the fast video feature and the slow video feature.

[0014] In some embodiments, the feature extraction network based on pre-training extracts time scale information of video frame sequences at different time scales to obtain the fast video features and the slow video features, including:

[0015] Inputting a short-time-scale video frame sequence into a first feature extraction network, and outputting the fast video feature; the first feature extraction network is constructed by a temporal convolutional network and a fully connected layer;

[0016] A long-time-scale video frame sequence is input into a second feature extraction network, and the slow video features are output; the second feature extraction network is constructed by a temporal convolutional network and a feature pyramid network.

[0017] In some embodiments, determining the current action according to the quick video feature and the preset action includes:

[0018] Preliminarily extracting text features from the preset action to obtain text features;

[0019] Based on a pre-trained feature matching network, feature matching is performed on the quick video feature and the text feature to determine the current action.

[0020] In some embodiments, the training process of the classification network is:

[0021] Using the text features of the preset action and a short-time-scale video frame sequence, a feature matching network is trained;

[0022] The classification network is trained based on the feature matching network and a long-time scale video frame sequence.

[0023] In some of the embodiments, it also includes:

[0024] Storing all actions and corresponding detection results in the video frame to be tested;

[0025] The action is retrieved based on the detection result; and the retrieval result is visualized.

[0026] In some of the embodiments, it also includes:

[0027] The video frame to be tested, the current action and the detection result are visually displayed.

[0028] In a second aspect, in this embodiment, a motion detection system is provided, including:

[0029] A video acquisition module is used to acquire the video frame to be tested;

[0030] An action determination module is used to extract multi-time scale information in the video frame to be tested, and obtain fast video features and slow video features respectively; determine the current action according to the fast video features and the preset action;

[0031] The action detection module is used to construct a classification network by fusing the slow video features and the fast video features; detect the current action based on the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

[0032] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the action detection method described in the first aspect when executing the computer program.

[0033] In a fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the action detection method described in the first aspect is implemented.

[0034] Compared with the related art, the action detection method, system, computer device and storage medium provided in the present embodiment obtain the video frame to be tested; extract the multi-time scale information in the video frame to be tested, and obtain the fast video feature and the slow video feature respectively; determine the current action according to the fast video feature and the preset action; construct a classification network by fusing the slow video feature and the fast video feature; detect the current action based on the classification network to obtain the detection result of whether the current action is a correct action or an incorrect action. Through this embodiment, the current action in the video frame to be tested can be identified, and the features of multi-time scale information can be fused to construct a classification network. The classification network can detect the current action more accurately to obtain the detection result of whether the current action is correct, and then analyze the unqualified or incorrect parts in the surgical action to meet the actual needs of surgery or training.

[0035] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1is a hardware structure block diagram of a terminal of an action detection method in an embodiment;

[0038] Figure 2 is a flow chart of an action detection method in an embodiment;

[0039] Figure 3 is a schematic diagram of the structure of a second feature extraction network in an embodiment;

[0040] Figure 4 is a flow chart of a motion detection method in another embodiment;

[0041] Figure 5 is a schematic diagram of an action detection process in an embodiment;

[0042] Figure 6 is a structural block diagram of a motion detection system in an embodiment;

[0043] Figure 7 is a schematic diagram of a motion detection system in an embodiment.

[0044] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, video acquisition module; 20, action determination module; 30, action detection module. DETAILED DESCRIPTION

[0045] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0046] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the motion detection method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the motion detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0049] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0050] In recent years, with the further deepening of research in the field of artificial intelligence endoscopic diagnosis, especially in minimally invasive surgeries such as laparoscopy, artificial intelligence methods are combined to further analyze the surgical process. By detecting whether the doctor's movements during the operation are correct, it is more conducive to the doctor's understanding and analysis of the surgical process. Moreover, in surgical training and education, motion detection can help trainees immediately identify and correct incorrect movements, thereby improving training effectiveness.

[0051] Most of the current motion detection methods only recognize surgical movements and cannot analyze the unqualified or incorrect parts of surgical movements, which cannot meet the actual needs of surgery or training. Therefore, there is an urgent need for a method that can accurately detect whether the movement is correct.

[0052] In this embodiment, a motion detection method is provided. Figure 2 is a flow chart of the action detection method in this embodiment. Figure 2 As shown, the method comprises the following steps:

[0053] Step S210, obtaining a video frame to be tested.

[0054] Specifically, the video frames to be tested are obtained during the surgical process or the training simulation surgical process, and the video frames to be tested contain surgical actions, surgical instruments, and operation objects. The video frames to be tested can be directly captured from the collected surgical video, or the video stream of the surgical video stream can be parsed into multiple frames of video frames arranged in sequence. Furthermore, the obtained video frames to be tested are preprocessed, and the preprocessing includes but is not limited to denoising, color conversion, size conversion, etc.

[0055] The method for obtaining the video frames to be tested in the embodiment of the present application includes but is not limited to obtaining the video frames to be tested that meet the requirements from the surgical videos pre-stored in the database; it can also be downloading the video frames to be tested that meet the requirements from the network platform; it can also be generating the corresponding video frames to be tested according to the needs, etc. The embodiment of the present application does not limit the method for obtaining the video frames to be tested.

[0056] In addition, the operation object can be different parts of the simulated human body model, or local simulated tissue and organ models, and the video frames to be tested can be captured in real time through endoscopes, laparoscopes, or image acquisition devices such as cameras and video recorders.

[0057] Step S220, extracting multi-time scale information in the video frame to be tested, and obtaining fast video features and slow video features respectively.

[0058] Specifically, the acquired video frames to be tested are subjected to intelligent analysis, for example, multimodal feature extraction. Multi-scale time information in the video frames to be tested can be extracted through multimodal feature extraction. The multi-scale time information represents multiple time scale information, specifically including long time scale and short time scale information. The long time scale information focuses on capturing global changes and long-term trends in the video frames to be tested, that is, the characteristics of slow movements during surgery, and the short time scale information focuses on capturing local changes and immediate events in the video frames to be tested, that is, the characteristics of fast movements during surgery. Correspondingly, the multi-time scale information in the video frames to be tested is extracted to obtain fast video features corresponding to fast movements during surgery, and slow video features corresponding to slow movements during surgery. It can be understood that fast video features and slow video features can be extracted through deep learning neural networks.

[0059] Step S230, determining the current action according to the quick video feature and the preset action.

[0060] Specifically, feature matching is performed based on the fast video features of the video frame to be tested and the features of the preset action to identify and determine the current action. The preset actions include but are not limited to correct actions such as holding the needle with the right hand, placing the needle, puncturing the tissue, transferring the needle from the left to the right, moving the needle to the center, and releasing more sutures, as well as incorrect actions such as accidental slipping of the instrument, inverted needle tip, failed thread release, and multiple attempts at contact.

[0061] In some embodiments, the preset actions may be input in advance in text form and categorized into the above correct actions and incorrect actions.

[0062] Step S240, constructing a classification network by fusing the slow video features and the fast video features.

[0063] Specifically, the fast video features and slow video features at multiple time scales are fused, and the fused features of the two dimensions are used as input to construct a classification network. The classification network can be a fully connected neural network or a more complex deep learning model, such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). The classification network is trained to learn the mapping relationship from fused features to preset action classifications (correct actions and incorrect actions). This can improve the sensitivity of the classification network to actions of different speeds, describe actions more comprehensively, and help the classification network detect the type of action more accurately.

[0064] Step S250, detecting the current action based on the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

[0065] Specifically, the identified current action is input into the classification network constructed above, and the classification network outputs the detection result of whether the current action is a correct action or an incorrect action.

[0066] It is understandable that the test results can be applied to surgical training or training, or to the analysis of surgical cases, to assist doctors in practicing and learning better. In addition, the results can be further processed or analyzed, and applied in different scenarios according to the processing results or analysis results.

[0067] Through the above steps, the current action in the video frame to be tested can be identified through feature matching, and the fast video features and slow video features under multi-time scale information can be integrated to construct a classification network. The classification network can more accurately detect the current action and obtain a detection result of whether the current action is correct, thereby analyzing the unqualified or incorrect surgical actions to meet the actual needs of surgery or training.

[0068] In some embodiments, the step S220 extracts multi-time scale information in the video frame to be tested, and obtains fast video features and slow video features respectively, including the following steps:

[0069] Step S221 , sampling the video frames to be tested at different sampling rates to obtain video frame sequences at different time scales.

[0070] Specifically, different sampling rates are used to sample the same sequence of video frames to be tested, and video frame sequences at different time scales are obtained. The sampling rate determines the frequency of extracting frames from the original sequence of video frames to be tested, thereby affecting the final time scale, and obtaining video frame sequences at different time scales. Exemplarily, for the input video frame to be tested, sampling is performed at intervals of t frames at a sampling rate to obtain a video frame sequence of a short time scale, and then sampling is performed at intervals of 2t frames or 3t frames at another sampling rate to obtain a video frame sequence of a long time scale, wherein t can be taken as 16, corresponding to a video frame sequence of 16 frames of a short time scale.

[0071] Step S222: Based on the pre-trained feature extraction network, the time scale information of the video frame sequence at different time scales is extracted to obtain fast video features and slow video features.

[0072] Specifically, for the video frame sequences at different time scales obtained above, corresponding feature extraction networks can be trained respectively to extract features of different dimensions through the feature extraction network, including fast video features in video frame sequences at short time scales, and slow video features in video frame sequences at long time scales.

[0073] Among them, the short-time-scale video frame sequence is input into the first feature extraction network to output fast video features; the first feature extraction network is constructed by a temporal convolutional network and a fully connected layer; the long-time-scale video frame sequence is input into the second feature extraction network to output slow video features; the second feature extraction network is constructed by a temporal convolutional network and a feature pyramid network.

[0074] The first feature extraction network is used to extract fast video features in a short-time-scale video frame sequence. Based on the network structure of the temporal convolutional network and the fully connected layer in the first feature extraction network, the temporal convolutional network is used to extract local detail features of the action in the short-time-scale video frame sequence. The local detail features are passed through the fully connected layer, and the features at different moments extracted by the temporal convolutional network are fused in the fully connected layer and converted to obtain fast video features. Among them, the temporal convolutional network includes but is not limited to LSTM (Long Short-Term Memory).

[0075] The slow video features in the long-time video frame sequence are extracted by the second feature extraction network. Figure 3 is a schematic diagram of the structure of the second feature extraction network in this embodiment, such as Figure 3As shown, based on the network structure of the temporal convolutional network and the feature pyramid network in the second feature extraction network, downsampling is performed through the temporal convolutional network to continuously obtain feature layers F1, F2, ..., Fn of multiple spatial dimensions, and the feature layers are fused in the feature pyramid network (FPN) to obtain slow video features. Among them, the temporal convolutional network includes but is not limited to TCN (Temporal Convolutional Network). In order to be able to perform real-time processing, TCN can perform efficient parallel convolution operations.

[0076] In this embodiment, multimodal feature extraction is performed on the video frames to be tested, and fast video features and slow video features that can characterize multi-time scale information are obtained. At the same time, the features of fast and slow movements during the operation are obtained, and more comprehensive motion feature information is provided when a classification network is subsequently constructed.

[0077] In some embodiments, determining the current action according to the fast video feature and the preset action in step S230 includes the following steps:

[0078] The text features of the preset actions are extracted in advance to obtain the text features; based on the pre-trained feature matching network, the fast video features and the text features are matched to determine the current action.

[0079] Specifically, the preset actions are input in text form in advance and classified into correct actions and incorrect actions. Among them, the correct actions include but are not limited to holding the needle with the right hand, placing the needle, puncturing the tissue, transferring the needle from the left to the right, moving the needle to the center, releasing more sutures, etc., and the incorrect actions include but are not limited to accidental slipping of the instrument, inverted needle tip, failed thread release, multiple attempts at contact, etc.

[0080] The text features of the preset action in text form are extracted by a text encoder (Text-Encoder) to obtain the text features of the preset action. The text encoder includes but is not limited to BERT (Bidirectional Encoder Representations from Transformers), ULMFiT (Universal Language Model Fine-Tuning), etc.

[0081] In the pre-trained feature matching network, the fast video features and text features are converted into high-dimensional embedding representations (feature matrices), the feature matrices are dot-multiplied, and the covariance between the features is calculated for feature matching. The higher the similarity of the feature matching, the greater the covariance, and the final match determines the current action.

[0082] By extracting multimodal features in this embodiment and fusing text features and fast video features for action recognition, the accuracy of action recognition can be improved.

[0083] In some of the embodiments, the training process of the classification network is:

[0084] A feature matching network is trained using text features of preset actions and a short-time-scale video frame sequence; a classification network is trained based on the feature matching network and a long-time-scale video frame sequence.

[0085] Specifically, a feature matching network is obtained by training a deep learning network based on text features and fast video features of a short-time-scale video frame sequence to identify the current action. On the basis of the feature matching network, features of two different dimensions, fast video features and slow video features, are fused. Among them, the feature fusion method is to concatenate different features to generate a larger tensor feature, and then perform convolution. After feature fusion, the network parameters of the classification network are jointly fine-tuned as a whole through the convolutional network and the fully connected layer, and the classification network is trained to learn the mapping relationship from the fused features to the preset action classification (correct action and wrong action) to obtain the classification network.

[0086] Furthermore, the short-time-scale video frame sequence and the long-time-scale video frame sequence can be taken as two branches. First, the network training of the text features and the short-time-scale video frame sequence branch is performed, and then the network parameters of the short-time-scale video frame sequence branch are maintained, and the network training of the long-time-scale video frame sequence branch is performed. Finally, the overall fine-tuning is combined to obtain the classification network.

[0087] By adopting a multi-stage temporal convolutional network in this embodiment, by capturing features at different time scales on different video branches and fusing them, a classification network is trained to improve the accuracy of the classification network in detecting actions.

[0088] In some of the embodiments, it also includes:

[0089] Visualize the video frame to be tested, the current action and the detection results.

[0090] Specifically, during the operation or in the analysis of the surgical video, the current action in the video frame to be tested and the detection result corresponding to the current action are identified in real time, and the current video frame to be tested, as well as the current action and detection result obtained by identification are displayed through the visual interaction module. Exemplarily, an intuitive and easy-to-use display interface can be designed in the visual interaction module according to the application scenario, and the interface may include elements such as a video window, a result display area, and control buttons. The identified current action (such as "holding the needle with the right hand", "needle tip inverted", etc.) and the detection result ("correct action", "wrong action") are displayed in real time in a designated area on the interface. With the continuous input of the video frame to be tested, the display interface of the visual interaction module should also be updated in real time to reflect the latest current action and detection results. Necessary interactive functions are also added to the display interface, such as pause / continue, playback, selection of different viewing angles, etc., so that users can view and operate as needed.

[0091] Furthermore, information related to the surgical action can be pre-integrated, such as the operation precautions, potential risks, and degree of danger of the action. When the current action is identified, the related information can be retrieved and displayed through the visual interaction module.

[0092] In some of the embodiments, it also includes:

[0093] Store all actions and corresponding detection results in the video frame to be tested; retrieve actions based on the detection results; and visualize the retrieval results.

[0094] Specifically, when storing all actions in the video frame to be tested and the corresponding detection results, the start time and end time of a certain action, as well as additional information such as the surgical instrument used for the action, can also be stored. Through query search, all incorrect actions can be retrieved according to the detection results, or a specific action can be retrieved according to the operation time, so as to review and repeatedly learn to improve the learning effect. Among them, the time when the action begins to be recognized is used as the start time, and the time when it is recognized as other actions is used as the end time.

[0095] The present embodiment is described and illustrated below through preferred embodiments.

[0096] Figure 4 is a flow chart of the action detection method in this embodiment. Figure 4 As shown, the method comprises the following steps:

[0097] Step S410, obtaining a video frame to be tested and a preset action.

[0098] Step S420, sampling the video frames to be tested at different sampling rates to obtain video frame sequences at different time scales; based on the pre-trained feature extraction network, extracting fast video features of the video frame sequence at a short time scale and slow video features of the video frame sequence at a long time scale.

[0099] Step S430: extracting text features of the preset action through a text encoder.

[0100] Step S440, performing feature matching based on the quick video feature and the text feature to determine the current action.

[0101] Step S450, constructing a classification network by fusing the slow video features and the fast video features; and detecting the current action through the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

[0102] Step S460, storing all actions and corresponding detection results in the video frame to be tested; and providing a retrieval mechanism to display the current action, detection results, retrieval results and related information in real time through the display interface of the visual interaction module.

[0103] Figure 5 is a schematic diagram of the action detection process in this embodiment. Figure 5 As shown in the figure, in multimodal feature extraction, the text features of the input preset action, as well as the fast video features and slow video features of the video frame to be tested at different time scales are extracted through the text encoder, wherein the fast video features in the short time scale video frame sequence are extracted through the long short-term memory network and the fully connected layer, and the slow video features in the long time scale video frame sequence are extracted through the temporal convolution network and the feature pyramid network. In action recognition and detection, feature matching is performed based on the slow video features and text features, and the current action is output. Then, the classification network constructed by fusing the slow video features and the fast video features is used to detect the current action to obtain the detection result of the correct action or the wrong action.

[0104] refer to Figure 5 In the training process of the classification network, the short-time-scale video frame sequence and the long-time-scale video frame sequence are taken as two branches. The network training of the text features and the short-time-scale video frame sequence branch is first performed, and then the network parameters of the short-time-scale video frame sequence branch are maintained, and the network training of the long-time-scale video frame sequence branch is performed. Finally, the overall fine-tuning is combined to obtain the classification network.

[0105] This embodiment uses a multimodal feature extraction method to fuse text features and video features at different time scales. On the one hand, it enriches the input information of the action and can accurately identify the current action. On the other hand, by fusing features of different dimensions, it constructs a classification network that can perform action detection more comprehensively, and obtains more accurate detection results to meet the actual needs of surgery and training, which is helpful for doctors to review and train and learn.

[0106] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here. For example, the order of extracting multimodal features in steps S420 and S430 does not limit the order of execution.

[0107] In this embodiment, a motion detection system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0108] Figure 6 : is a structural block diagram of the motion detection system of this embodiment. Figure 6 As shown, the system includes:

[0109] The video acquisition module 10 is used to acquire the video frame to be tested.

[0110] The action determination module 20 is used to extract multi-time scale information in the video frame to be tested, and obtain fast video features and slow video features respectively; and determine the current action according to the fast video features and the preset action.

[0111] The action detection module 30 is used to construct a classification network by fusing slow video features and fast video features; detect the current action based on the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

[0112] Through the system provided in this embodiment, the current action in the video frame to be tested can be identified through feature matching, and the fast video features and slow video features under multi-time scale information can be integrated to construct a classification network. The classification network can more accurately detect the current action and obtain a detection result of whether the current action is correct, thereby analyzing the unqualified or incorrect surgical actions to meet the actual needs of surgery or training.

[0113] In some of the embodiments, the system further includes a visualization interaction module, which is used to visualize the video frame to be tested, the current action and the detection result, and to provide a retrieval mechanism and visualize the retrieval result.

[0114] Figure 7 is a schematic diagram of the motion detection system in this embodiment. Figure 7 As shown, the video acquisition module includes image acquisition equipment, surgical instruments, etc. The image acquisition equipment acquires the video frames to be tested in real time, and transmits the video frames to be tested to the visualization interaction module for display, and inputs them into the server for intelligent analysis. The surgical instrument also has relevant data transmission configurations, such as an endoscope, etc., which collects and outputs information through the endoscope lens. At the same time, the doctor can capture and save images at any time through the surgical instrument. Finally, the collected video frames to be tested and other image information are cached, stored and converted, and pre-processed such as denoising, color conversion, and size conversion.

[0115] In intelligent analysis, the acquired video frames to be tested and preset actions are mainly intelligently analyzed, including multimodal feature extraction, action recognition and detection, storage and control, and information integration. In multimodal feature extraction, the text features of the preset action, as well as the fast video features and slow video features at different time scales of the video frames to be tested are extracted, the current action is identified based on the text features and slow video features, and the detection result of the current action is obtained through classification network detection. In information integration, information associated with surgical actions can be pre-integrated, such as the operation precautions, potential risks, and degree of danger of the action. When the current action is identified and determined, the associated information can be retrieved and displayed through a visual interaction module. In storage and control, all actions and corresponding detection results during the operation can be stored, as well as the output of the control analysis results, for example, the start time and end time of a certain action, as well as additional information such as the surgical instruments used for the action, can be stored. Through query search, all wrong actions can be retrieved according to the detection results, or a specific action can be retrieved according to the operation time, and displayed through a visual interaction module.

[0116] The original video frame to be tested, the current action, the detection results and the retrieval mechanism are displayed in the visual interaction module.

[0117] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0118] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0119] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0120] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0121] In addition, in combination with the action detection method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any action detection method in the above embodiments is implemented.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0123] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0124] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0125] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0126] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A motion detection method, characterized in that: include: Get the video frame to be tested; Extracting multi-time scale information in the video frame to be tested to obtain fast video features and slow video features respectively; Determining a current action according to the quick video feature and the preset action; By fusing the slow video features and the fast video features, a classification network is constructed; The current action is detected based on the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

2. The motion detection method according to claim 1, characterized in that: The step of extracting multi-time scale information from the video frame to be tested to obtain fast video features and slow video features respectively includes: Sampling the video frames to be tested at different sampling rates to obtain video frame sequences at different time scales; Based on a pre-trained feature extraction network, time scale information of video frame sequences at different time scales is extracted respectively to obtain the fast video feature and the slow video feature.

3. The motion detection method according to claim 2, characterized in that: The feature extraction network based on the pre-training respectively extracts the time scale information of the video frame sequence at different time scales to obtain the fast video feature and the slow video feature, including: Inputting a short-time-scale video frame sequence into a first feature extraction network, and outputting the fast video feature; the first feature extraction network is constructed by a temporal convolutional network and a fully connected layer; A long-time-scale video frame sequence is input into a second feature extraction network, and the slow video features are output; the second feature extraction network is constructed by a temporal convolutional network and a feature pyramid network.

4. The motion detection method according to claim 1, characterized in that: The determining the current action according to the quick video feature and the preset action includes: Preliminarily extracting text features from the preset action to obtain text features; Based on a pre-trained feature matching network, feature matching is performed on the fast video feature and the text feature to determine the current action.

5. The motion detection method according to claim 1, characterized in that: The training process of the classification network is: Using the text features of the preset action and a short-time-scale video frame sequence, a feature matching network is trained; The classification network is trained based on the feature matching network and a long-time scale video frame sequence.

6. The motion detection method according to claim 1, characterized in that: Also includes: Storing all actions and corresponding detection results in the video frame to be tested; Retrieving the action based on the detection result; The search results are displayed visually.

7. The motion detection method according to claim 1, characterized in that: Also includes: The video frame to be tested, the current action and the detection result are visually displayed.

8. A motion detection system, characterized in that: include: A video acquisition module is used to acquire the video frame to be tested; An action determination module is used to extract multi-time scale information in the video frame to be tested, and obtain fast video features and slow video features respectively; Determining a current action according to the quick video feature and the preset action; An action detection module, used to construct a classification network by fusing the slow video features and the fast video features; The current action is detected based on the classification network to obtain a detection result of whether the current action is a correct action or an incorrect action.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the action detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the action detection method according to any one of claims 1 to 7 are implemented.

Images