Motion recognition method and motion recognition system combining segmentation and depth estimation

Abstract

The invention relates to an action recognition method and system combining segmentation and depth estimation, and the method comprises the steps: obtaining a to-be-recognized video frame sequence, and enabling a picture of the to-be-recognized video frame sequence to comprise a target instrument for carrying out an action and a target object for bearing the action; based on the to-be-recognized video frame sequence, extracting a target instrument and a target object to obtain a mask graph sequence; extracting depth information to obtain a depth map sequence; inputting the mask image sequence, the depth image sequence and the video frame sequence into a completely trained action recognition model to obtain action recognition information between the target instrument and the target object; based on the action recognition information, the rationality of the currently used target instrument and the implemented action is verified, and the verification result and the corresponding action associated information are output, so that the problem of low action recognition precision is solved, segmentation and depth estimation are combined and applied to action recognition, the action recognition precision is improved, and the accuracy of action recognition is improved. Therefore, a more accurate action analysis effect is obtained.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an action recognition method and an action recognition system combining segmentation and depth estimation. Background Art

[0002] Laparoscopic surgery, also known as minimally invasive surgery, is a surgical method that uses a laparoscope and related instruments through small incisions. Compared with traditional open surgery, laparoscopic surgery has many advantages, such as only a few small incisions instead of a large opening, which means less tissue damage, bleeding and pain. These small incisions are usually only a few centimeters long and can significantly reduce the recovery time and scar formation after surgery. By using high-definition cameras and other imaging technologies, surgeons can clearly observe the intraoperative situation. However, doctors can only capture the structure and actual situation of the human abdominal cavity through the images taken by the lens. Unlike human eyes, a single lens is difficult to show the depth difference of details, which increases the difficulty of doctors' judgment. At present, the execution of surgical actions depends entirely on the doctor's personal experience and lacks high-precision motion analysis assistance. Among them, the most critical thing is to achieve accurate surgical action recognition.

[0003] In recent years, with the further deepening of domestic research in the field of artificial intelligence, action recognition based on artificial intelligence has gradually matured. For example, in the existing action recognition scheme, the action in the video data is recognized through the convolutional network, in which the action recognition network can extract multi-scale features and fuse the multi-scale feature information through the pyramid to recognize the action.

[0004] However, when working in macro scenarios such as laparoscopy, higher accuracy is required for motion recognition. The accuracy of existing motion recognition networks is still insufficient and cannot provide accurate data support for further motion analysis. Summary of the invention

[0005] In this embodiment, an action recognition method and an action recognition system combining segmentation and depth estimation are provided to solve the problem of low action recognition accuracy in related technologies.

[0006] In a first aspect, in this embodiment, a method for action recognition combining segmentation and depth estimation is provided, the method comprising:

[0007] Acquire a video frame sequence to be identified, wherein the video frame sequence to be identified includes a target device for performing an action and a target object for receiving the action;

[0008] Based on the video frame sequence to be identified, the target device and the target object are extracted to obtain a mask map sequence; depth information is extracted from the video frame sequence to be identified to obtain a depth map sequence;

[0009] Inputting the mask image sequence, the depth image sequence and the video frame sequence into a fully trained action recognition model to obtain action recognition information between the target device and the target object; wherein the fully trained action recognition model is trained based on the mask image and the depth image used for training;

[0010] Based on the action recognition information, the target device currently in use and the action being performed are verified to obtain a verification result and corresponding action association information.

[0011] In some embodiments, based on the video frame sequence to be identified, extracting the target device and the target object to obtain a mask image sequence includes:

[0012] Inputting the video frame sequence to be identified into a well-trained detection and segmentation model, detecting and segmenting the target device and the target object, and obtaining a mask image sequence;

[0013] The fully trained detection and segmentation model includes: a feature extraction unit, a detection unit and a segmentation unit;

[0014] The feature extraction unit is used to extract features of each video frame image in the video frame sequence;

[0015] The detection unit is used to predict the features of the video frame image to obtain a detection result; the detection result includes the position and category of the target device and the target object;

[0016] The segmentation unit is used to segment the video frame image based on the detection result to obtain a mask image.

[0017] In some of the embodiments, the feature extraction unit is obtained based on contrastive learning training;

[0018] Wherein, the method for obtaining the feature extraction unit based on the contrastive learning training includes:

[0019] Extracting sample images from the first video data used for training, and inputting the sample images into two initial neural networks with the same structure respectively to obtain two sample features; the sample images include the target device and the target object;

[0020] Calculating a loss function based on the similarity of the two sample features;

[0021] The initial neural network is trained based on the loss function to obtain a feature extraction unit.

[0022] In some of the embodiments, the detection unit includes a multi-scale fusion network and a detection head;

[0023] The multi-scale fusion network is used to fuse the features of the video frame output by the feature extraction unit to obtain a fused feature;

[0024] The detection head is used to output the detection frame coordinates and the category of the detection target based on the fusion features; the detection target includes a target device and a target object.

[0025] In some embodiments, segmenting the video frame image based on the detection result to obtain a mask image includes:

[0026] Based on the detection frame in the detection result, the video frame image is cropped to obtain small frame images of the target device and the target object;

[0027] Inputting the small frame image into a segmentation network to obtain small frame mask images of the target device and the target object;

[0028] Based on the small frame mask images in the same video frame image, a mask image corresponding to the video frame image is obtained by splicing.

[0029] In some embodiments, extracting depth information from the video frame sequence to be identified to obtain a depth map sequence includes:

[0030] Inputting the video frame sequence to be identified into a well-trained depth estimation model to obtain a depth map sequence;

[0031] The fully trained depth estimation model is obtained based on an initial depth estimation network and self-supervised label training; the self-supervised label is obtained based on prior depth information of the target object.

[0032] In some embodiments, the method of obtaining the fully trained depth estimation model based on the initial depth estimation network and the self-supervised labels includes:

[0033] Performing depth estimation on the second video data for training based on the initial depth estimation network to obtain initial depth information of each position in the picture; the picture of the second video data includes the target object;

[0034] Predicting the second video data based on the fully trained detection and segmentation model to obtain category information of the target object in the picture; obtaining prior depth information of the target object in the picture based on the category information;

[0035] Optimizing parameters of the initial depth estimation network based on the initial depth information and the priori depth information to obtain a fully trained depth estimation model.

[0036] In some embodiments, the process of obtaining a fully trained action recognition network includes:

[0037] Segmenting the first video data for training based on the action categories to obtain video segments with action labels;

[0038] Extracting action frame images from the video segment with action labels;

[0039] Inputting the action frame image into the well-trained detection segmentation model and the well-trained depth estimation model respectively to obtain a set of prediction results for training, wherein the prediction results include the mask map for training and the depth map for training;

[0040] Inputting a plurality of continuous groups of the prediction results and the corresponding action frame images into an initial action recognition network, the action recognition network is trained to obtain the action recognition network.

[0041] In a second aspect, an action recognition system is provided in this embodiment, the system comprising: an intelligent analysis module; the intelligent analysis module is used to implement the steps of any one of the methods in the first aspect.

[0042] In some of the embodiments, the system further comprises: an image acquisition module and a data display module;

[0043] The image acquisition module is used to shoot the action video to be identified, and pre-process the action video to be identified and then send it to the intelligent analysis module and the data display module;

[0044] The data display module is used to display the video of the action to be identified, receive the verification result and action association information output by the intelligent analysis module, and visually display the verification result and the action association information;

[0045] The intelligent analysis module is also used to extract the video frame sequence to be identified from the pre-processed action video to be identified.

[0046] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for action recognition combining segmentation and depth estimation described in the first aspect is implemented.

[0047] Compared with the related art, the action recognition method and action recognition system combined with segmentation and depth estimation provided in the present embodiment obtain a video frame sequence to be recognized, wherein the screen of the video frame sequence to be recognized includes a target device for performing an action and a target object for undergoing the action; based on the video frame sequence to be recognized, the target device and the target object are extracted to obtain a mask image sequence; depth information is extracted from the video frame sequence to be recognized to obtain a depth image sequence; the mask image sequence, the depth image sequence and the video frame sequence are input into a fully trained action recognition model to obtain action recognition information between the target device and the target object; based on the action recognition information, the rationality of the target device currently in use and the action performed is verified, and the verification result and the corresponding action association information are output, thereby solving the problem of low action recognition accuracy, realizing the application of image depth information to action recognition, and improving the accuracy of action recognition through the combination of target segmentation and depth estimation, thereby obtaining a more accurate action analysis effect.

[0048] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1 A hardware structure block diagram of a terminal for an action recognition method combining segmentation and depth estimation in one embodiment;

[0051] Figure 2 is a flow chart of an action recognition method combining segmentation and depth estimation in one embodiment;

[0052] Figure 3 A schematic diagram of the structure of a detection segmentation model in one embodiment;

[0053] Figure 4 is a schematic diagram of cropping a video frame image based on a detection result in one embodiment;

[0054] Figure 5 A schematic diagram of a process for generating a mask image in one embodiment;

[0055] Figure 6 A schematic diagram of the structure of a motion recognition system in a preferred embodiment;

[0056] Figure 7Schematic diagram of the processing flow of surgical video data by the detection and segmentation network in a preferred embodiment. DETAILED DESCRIPTION

[0057] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0058] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0059] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the action recognition method combining segmentation and depth estimation of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations shown.

[0060] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the action recognition method combined with segmentation and depth estimation in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0062] In this embodiment, a method for action recognition combining segmentation and depth estimation is provided. Figure 2 is a flow chart of the action recognition method combining segmentation and depth estimation of this embodiment. Figure 2 As shown, the process includes the following steps:

[0063] Step S210, obtaining a video frame sequence to be identified, wherein the video frame sequence to be identified includes a target device for performing an action and a target object for receiving the action.

[0064] Specifically, this embodiment can be applied to multiple fields such as medicine and industry. For example, in the medical field, the video frame sequence is extracted from an endoscopic surgery video, the target instrument is a surgical instrument, and the target object is an organ in a living body. For example, in the industrial manufacturing field, the video frame sequence is extracted from a parts processing video, the target instrument is a processing instrument, and the target object is a part to be processed. By accurately identifying the actions of the target instrument, the monitoring of surgical operations, processing procedures and other processes can be further achieved. The video frame sequence can be obtained by extracting a frame sequence within a period of time from the action video to be identified, or the pre-processed video frame sequence to be identified can be directly obtained from the storage unit. The method of obtaining the video frame sequence to be identified is not limited in this embodiment.

[0065] Step S220: based on the video frame sequence to be identified, extract the target device and the target object to obtain a mask map sequence; extract depth information from the video frame sequence to be identified to obtain a depth map sequence.

[0066] Specifically, after the positions of the target device and the target object are detected, the target device and the target object are segmented, and a mask image including the target device and the target object is output. Since some video frames may only capture the target object or only capture the target device, the obtained mask image may include only the target object, only the target device, or both the target object and the target device.

[0067] Specifically, the depth information can be extracted based on an existing depth estimation network, or the existing depth estimation network can be fine-tuned so that the distance scale that it can recognize is consistent with the current action scene, thereby improving the calculation accuracy of the depth information.

[0068] Step S230, inputting the mask image sequence, the depth image sequence and the video frame sequence into a fully trained action recognition model to obtain action recognition information between the target device and the target object; wherein the fully trained action recognition model is trained based on the mask image and the depth image used for training.

[0069] Specifically, our input data contains a variety of information (i.e., mask maps and depth maps), which is more conducive to improving the accuracy of action recognition. For example, there are multiple target objects in the current action scene. Through the depth difference between the target device and the nearby target objects, it can be estimated whether the current target device is operating on a certain target object. At the same time, through the depth change of the target object between the previous and next frames, it can be estimated that the current action is pulling, grasping, dissecting, etc.

[0070] Step S240, based on the action recognition information, verify the currently used target device and the implemented action to obtain the verification result and the corresponding action association information.

[0071] Specifically, after obtaining the current action, it will reversely check whether the currently used equipment and the operation part (the area where the target object is located) are reasonable, and integrate and output the relevant information such as the precautions for the current action. Some information related to the action can be pre-set, such as the operation precautions for the action, the general operation time of the action, the equipment used for the action, etc. When the action is recognized, the relevant information can be retrieved for control output.

[0072] In this embodiment, by obtaining a video frame sequence to be identified, the screen of the video frame sequence to be identified includes a target device for performing an action and a target object for receiving the action; based on the video frame sequence to be identified, the target device and the target object are extracted to obtain a mask map sequence; the depth information is extracted from the video frame sequence to be identified to obtain a depth map sequence; the mask map sequence, the depth map sequence and the video frame sequence are input into a well-trained action recognition model to obtain action recognition information between the target device and the target object; based on the action recognition information, the rationality of the currently used target device and the action to be performed is verified, and the verification result and the corresponding action association information are output, thereby realizing action recognition based on artificial intelligence. Compared with the traditional method of analyzing actions by users observing video images autonomously, the accuracy and efficiency of action recognition are improved, and more abundant information assistance is provided to users based on verification and association information; compared with the existing action recognition neural network, the present application applies the depth information of the image to action recognition, improves the accuracy of action recognition through the combination of target segmentation and depth estimation, obtains more accurate action verification and the output effect of association information, and further improves the auxiliary performance for users.

[0073] In some of the embodiments, based on the video frame sequence to be identified, the target device and the target object are extracted to obtain a mask image sequence, including:

[0074] Step S221, input the video frame sequence to be identified into a well-trained detection and segmentation model, detect and segment the target device and the target object, and obtain a mask image sequence.

[0075] In some of the embodiments, see Figure 3 ,The fully trained detection and segmentation model includes: feature extraction unit, detection unit and segmentation unit.

[0076] The feature extraction unit is used to extract features of each video frame image in the video frame sequence.

[0077] The detection unit is used to predict the features of the video frame image to obtain a detection result; the detection result includes the position and category of the target device and the target object.

[0078] The segmentation unit is used to segment the video frame image based on the detection result to obtain a mask image.

[0079] In some embodiments, the feature extraction unit is obtained based on contrastive learning training; wherein the method of obtaining the feature extraction unit based on contrastive learning training includes:

[0080] Step S310, extracting sample images from the first video data used for training, and inputting the sample images into two initial neural networks with the same structure respectively to obtain two sample features; the sample images include a target device and a target object.

[0081] Step S320, calculating the loss function based on the similarity between the two sample features.

[0082] Step S330, training the initial neural network based on the loss function to obtain a feature extraction unit.

[0083] Specifically, the principle of the contrastive learning method is: feature extraction is performed through two neural networks with the same structure, such as two (resnet, vit, etc.). The two neural networks each input an image. When the input images are the same, the output features should be the same. When the two images are different, the output features should be different. The video is subjected to certain frame extraction and sample screening (the difference between two adjacent frames in the video is very small, so moderate frame extraction and filtering are required), and then input into the neural network. The loss function of the neural network is designed as follows:

[0084]

[0085] Where i, j are the sample numbers that enter the two neural networks at the same time, z i 、z j is the feature obtained after the sample passes through the neural network, sim is the similarity calculation function, τ is the number of data obtained by enhancement, and it is 1 without enhancement. Contrastive learning can help us learn richer image features.

[0086] In some of the embodiments, the detection unit includes a multi-scale fusion network and a detection head.

[0087] The multi-scale fusion network is used to fuse the features of the video frames output by the feature extraction unit to obtain fused features.

[0088] The detection head is used to output the detection frame coordinates and the category of the detection target based on the fusion features; the detection target includes the target device and the target object.

[0089] Specifically, when training the detection unit, the network parameters of the feature extraction unit are frozen so that it is not iteratively updated, and only the parameters of the multi-scale fusion network and the detection head are updated, thereby improving the training efficiency.

[0090] In some of the embodiments, see Figure 4 and Figure 5 , based on the detection results, the video frame image is segmented to obtain a mask map, including:

[0091] Step S410: cropping the video frame image based on the detection frame in the detection result to obtain a small frame image of the target device and the target object.

[0092] Step S420, input the small frame image into the segmentation network to obtain the small frame mask image of the target device and the target object.

[0093] Step S430 , based on the small frame mask images in the same video frame image, a mask image corresponding to the video frame image is obtained by splicing.

[0094] Specifically, the small image corresponding to the detected target (target object or target device) is sent to a segmentation network (the segmentation network is not limited, such as UNet), so as to segment the target in each detection frame and obtain a small frame mask image of each target; then the different small frame mask images obtained by segmentation are spliced ​​according to the position of their detection frames in the original video frame image, among which the position that does not enter the segmentation network is the background, and finally the mask of the entire large image and the category of each target in the large image are obtained.

[0095] In this embodiment, on the basis of positioning and identifying the target device and the target object, the segmentation mask information of the details is obtained at the same time, thereby enriching the input information of the action recognition and improving the accuracy of the action recognition.

[0096] In some embodiments, extracting depth information from a video frame sequence to be identified to obtain a depth map sequence includes:

[0097] Step S222: input the video frame sequence to be identified into a fully trained depth estimation model to obtain a depth map sequence.

[0098] In some of the embodiments, the fully trained depth estimation model is obtained based on an initial depth estimation network and self-supervised labels; the self-supervised labels are obtained based on prior depth information of the target object.

[0099] Specifically, since the video frame images to be detected are usually monocular images, lacking depth information, and the existing depth estimation networks are mostly trained through images in daily life scenes, their depths are mostly between tens of centimeters and tens of meters, and the errors are large when applied in macro scenes such as laparoscopic surgery. Therefore, this embodiment improves the accuracy of the depth estimation network through self-supervised labels.

[0100] In some embodiments, a method for obtaining a fully trained depth estimation model based on an initial depth estimation network and self-supervised labels includes:

[0101] Step S510, performing depth estimation on the second video data used for training based on the initial depth estimation network to obtain initial depth information of each position in the picture; the picture of the second video data includes the target object.

[0102] Step S520, predicting the second video data based on the fully trained detection and segmentation model to obtain category information of the target object in the picture; and obtaining prior depth information of the target object in the picture based on the category information.

[0103] Step S530, obtaining a self-supervised label based on the initial depth information and the prior depth information; optimizing the parameters of the initial depth estimation network based on the self-supervised label to obtain a fully trained depth estimation model.

[0104] Specifically, it is difficult to obtain annotated information for monocular images, and annotation is usually performed manually, which is inefficient. This embodiment also proposes a method for producing self-supervised labels. The method first trains a detection and segmentation module based on the video data in the current scene, and uses the detection and segmentation model to accurately predict the position of the target object in the current scene. Since the deepest target object has been recorded in advance, the depth of the known target object can be used as prior depth information. In addition, the initial depth estimation network can output the predicted depth information of each position in a picture. Using the prior depth information and the predicted depth information (i.e., the initial depth information) at the same position, the actual depth of each position in the picture can be calculated, that is, the self-supervised label is obtained.

[0105] For example, in a laparoscopic surgery scenario, an initial depth estimation network (such as the publicly available dinov2 depth estimation network) is used to perform depth estimation on each image in the laparoscopic video to obtain the predicted depth difference in each image, that is, the maximum depth d extracted by the initial depth estimation network. pmax Min. depth d pmin At the same time, the detection and segmentation network is used to predict the category and location of the organ (i.e., the target object) in the laparoscopic video. Based on the predicted data of the detection and segmentation network and the prior information of the actual physiological structure, the prior depth difference is obtained, that is, the prior maximum depth d rmax (i.e., the depth of the deepest organ in the laparoscopic video) minus the prior minimum depth d rmin (Generally, the depth of the laparoscopic instrument entering the abdominal cavity can be preset to 0.) The predicted depth difference is normalized by the actual depth difference to obtain the self-supervised label of the actual depth of each position. The calculation formula is as follows:

[0106]

[0107] Among them, d(i,j) represents the actual depth of each position, and pre(i,j) is the predicted depth output by the initial depth estimation network.

[0108] In some embodiments, the process of obtaining a fully trained action recognition network includes:

[0109] Step S610: segment the first video data for training based on action categories to obtain video segments with action labels.

[0110] Step S620: extracting action frame images from the video segment with action labels.

[0111] Step S630, inputting the action frame image into a well-trained detection segmentation model and a well-trained depth estimation model respectively, to obtain a set of prediction results for training, wherein the prediction results include a mask map for training and a depth map for training.

[0112] Step S640: input multiple groups of continuous prediction results and corresponding action frame images into an initial action recognition network to train the action recognition network.

[0113] Specifically, each time, eight consecutive sets of prediction result data are sent to the action recognition network for training. For example, groups 1 to 8 are taken for the first time, groups 2 to 9 are taken for the second time, and so on.

[0114] In this embodiment, a motion recognition system is also provided, and the system includes: an intelligent analysis module; the intelligent analysis module is used to implement the method in any of the above embodiments.

[0115] The system also includes: an image acquisition module and a data display module.

[0116] The image acquisition module is used to shoot the video of the action to be identified, and pre-process the video of the action to be identified and then send it to the intelligent analysis module and the data display module.

[0117] The data display module is used to display the video of the action to be identified, receive the verification result and action association information output by the intelligent analysis module, and visually display the verification result and the action association information.

[0118] The intelligent analysis module is also used to extract the video frame sequence to be identified from the pre-processed action video to be identified.

[0119] In this embodiment, the system is able to identify the information of the target device and the target object, and input this information into the motion recognition network to achieve accurate motion recognition, while also providing some related information to assist the user's operation.

[0120] The present embodiment is described and illustrated below through a preferred embodiment. The present preferred embodiment takes a laparoscopic surgery scenario as an example.

[0121] In order to improve the accuracy of surgical action recognition during laparoscopic surgery, this preferred embodiment proposes a method and system for action recognition that combines segmentation and depth estimation. The main components and connections of the system are as follows: Figure 6 The system includes an image acquisition module, an intelligent analysis module and a data display module.

[0122] The image acquisition module includes lenses, laparoscopic surgical instruments, and data acquisition. The lens is mainly used in laparoscopic surgery. It captures real-time internal human image information during the operation, sends it to the original image display through the instrument, and inputs it into the server for intelligent processing. Laparoscopic surgical instruments: used for various medical devices used in laparoscopic surgery, as well as related transmission configurations, conversion and control of image output through information collected by the endoscopic lens. At the same time, doctors can use the instrument to capture and save the currently required images at any time. Data acquisition: The internal human body data collected by the lens is parsed, cached, converted, and related preprocessing is performed, such as image denoising, color conversion, size transformation, etc.

[0123] The intelligent analysis module includes a key information extraction submodule, an action recognition submodule, an associated information processing submodule, and a server control submodule. The key information extraction submodule is used to extract the video frame sequence to be identified from the preprocessed action video to be identified, extract the surgical instrument and the tissue organ based on the video frame sequence to be identified, and obtain a mask map sequence; extract depth information from the video frame sequence to be identified to obtain a depth map sequence. The action recognition submodule is used to input the mask map sequence, the depth map sequence, and the video frame sequence into a fully trained action recognition model to obtain the action recognition information between the surgical instrument and the tissue organ. The associated information processing submodule is used to verify the currently used surgical instrument and the action performed based on the action recognition information, and obtain the verification result and the corresponding action association information. The server control submodule is used to control the information flow between the key information extraction submodule, the action recognition submodule, and the associated information processing submodule, and to interact with the data display module.

[0124] Among them, the key information extraction submodule includes a detection and segmentation network and a depth estimation network. The processing flow of the detection and segmentation network on surgical video data is as follows: Figure 7As shown. The detection and segmentation network includes a detection network and a segmentation network, wherein the detection network includes a backbone network based on contrastive learning, followed by a feature pyramid network, and followed by a detection head. When the surgical video acquired by the laparoscope lens is input into the detection network, the detection network outputs the detection results of tissues and organs and surgical instruments respectively, and then the detection results of tissues and organs and surgical instruments are input into the segmentation network to segment the tissues and organs or surgical instruments in each detection frame; finally, we splice the masks of different small frames obtained by segmentation according to the position of their detection frames based on the original image, and the position that does not enter the segmentation network is the background. Finally, we will obtain the segmentation of the entire large image and the specific categories of tissues and organs or surgical instruments in the large image.

[0125] Since the laparoscope lens acquires monocular images, it is difficult to obtain labeled information. The currently available depth estimation networks are mainly trained with images from daily life scenes, and their depths are mostly between tens of centimeters and tens of meters. When applied to laparoscopic scenes, the error is large. Therefore, we designed a self-supervised labeling method to fine-tune the currently available dinov2 depth estimation network. Fine-tuning dinov2 is mainly divided into three steps: the first step is to use the original dinov2 to estimate the depth of each image in a complete video. At this time, we count the maximum depth difference in a picture, that is, use d pmax Min. depth d pmin ; The second step is to obtain the minimum depth d by detecting the segmentation network rmin The actual position (usually the position where the laparoscopic instrument enters the abdominal cavity, the depth can be preset to 0), and the maximum depth d rmax In the actual human body structure, there is usually an approximate depth difference of the actual physiological structure available for use. The third step is to normalize the predicted depth difference by the actual depth difference to obtain the label depth d(i, j) of the position (i, j). The depth estimation network is fine-tuned using the surgical video data and labels. During the fine-tuning process, the new label depth is continuously updated and calculated, and finally the depth estimation network in the laparoscopic scenario is fine-tuned.

[0126] The data display module includes original image display and visual display. The original image display is mainly used to display the image collected by the current lens in real time, which is convenient for doctors to analyze. The visual display is mainly used to display information such as surgical instruments, tissue and organ locations, and precautions, that is, the content input by the intelligent analysis module.

[0127] In this preferred embodiment, information such as surgical instruments, tissues and organs can be identified, and this information can be input into the action recognition network to identify the actions during laparoscopic surgery, and at the same time, some precautions and other information can be reminded to assist the doctor in the operation. Surgical instruments, tissues and organs can be located and identified, and their detailed segmentation mask information can be obtained, which enriches the input information of the action and improves the accuracy of action recognition. This preferred embodiment also proposes a depth estimation and fine-tuning method for laparoscopic images. This method can calculate the depth of each part in the laparoscopic image when there is no annotation information for the monocular laparoscope, and improve the accuracy of action recognition through changes in depth and the distance between the instrument and the tissue.

[0128] In addition, in combination with the action recognition method combined with segmentation and depth estimation provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the action recognition methods combined with segmentation and depth estimation in the above embodiments is implemented.

[0129] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0130] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0131] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0132] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An action recognition method combining segmentation and depth estimation, characterized in that: The method comprises: Acquire a video frame sequence to be identified, wherein the video frame sequence to be identified includes a target device for performing an action and a target object for receiving the action; Based on the video frame sequence to be identified, the target device and the target object are extracted to obtain a mask map sequence; depth information is extracted from the video frame sequence to be identified to obtain a depth map sequence; Inputting the mask image sequence, the depth image sequence and the video frame sequence into a fully trained action recognition model to obtain action recognition information between the target device and the target object; wherein the fully trained action recognition model is trained based on the mask image and the depth image used for training; Based on the action recognition information, the target device currently in use and the action being performed are verified to obtain a verification result and corresponding action association information.

2. The method for action recognition combining segmentation and depth estimation according to claim 1, characterized in that: Based on the video frame sequence to be identified, the target device and the target object are extracted to obtain a mask image sequence, including: Inputting the video frame sequence to be identified into a well-trained detection and segmentation model, detecting and segmenting the target device and the target object, and obtaining a mask image sequence; The fully trained detection and segmentation model includes: a feature extraction unit, a detection unit and a segmentation unit; The feature extraction unit is used to extract features of each video frame image in the video frame sequence; The detection unit is used to predict the features of the video frame image to obtain a detection result; the detection result includes the position and category of the target device and the target object; The segmentation unit is used to segment the video frame image based on the detection result to obtain a mask image.

3. The method for action recognition combining segmentation and depth estimation according to claim 2, characterized in that: The feature extraction unit is obtained based on contrastive learning training; Wherein, the method for obtaining the feature extraction unit based on the contrastive learning training includes: Extracting sample images from the first video data used for training, and inputting the sample images into two initial neural networks with the same structure respectively to obtain two sample features; the sample images include the target device and the target object; Calculating a loss function based on the similarity of the two sample features; The initial neural network is trained based on the loss function to obtain a feature extraction unit.

4. The method for action recognition combining segmentation and depth estimation according to claim 2, characterized in that: The detection unit includes a multi-scale fusion network and a detection head; The multi-scale fusion network is used to fuse the features of the video frame output by the feature extraction unit to obtain a fused feature; The detection head is used to output the detection frame coordinates and the category of the detection target based on the fusion features; the detection target includes a target device and a target object.

5. The method for action recognition combining segmentation and depth estimation according to claim 2, characterized in that: The video frame image is segmented based on the detection result to obtain a mask image, including: Based on the detection frame in the detection result, the video frame image is cropped to obtain small frame images of the target device and the target object; Inputting the small frame image into a segmentation network to obtain small frame mask images of the target device and the target object; Based on the small frame mask images in the same video frame image, a mask image corresponding to the video frame image is obtained by splicing.

6. The method for action recognition combining segmentation and depth estimation according to claim 2, characterized in that: Extracting depth information from the video frame sequence to be identified to obtain a depth map sequence includes: Inputting the video frame sequence to be identified into a well-trained depth estimation model to obtain a depth map sequence; The fully trained depth estimation model is obtained based on an initial depth estimation network and self-supervised label training; the self-supervised label is obtained based on prior depth information of the target object.

7. The method for action recognition combining segmentation and depth estimation according to claim 6, characterized in that: The method for obtaining the fully trained depth estimation model based on the initial depth estimation network and the self-supervised labels comprises: Performing depth estimation on the second video data for training based on the initial depth estimation network to obtain initial depth information of each position in the picture; the picture of the second video data includes the target object; Predicting the second video data based on the fully trained detection and segmentation model to obtain category information of the target object in the picture; obtaining prior depth information of the target object in the picture based on the category information; Optimizing parameters of the initial depth estimation network based on the initial depth information and the priori depth information to obtain a fully trained depth estimation model.

8. The method for action recognition combining segmentation and depth estimation according to claim 6, characterized in that: The process of obtaining a fully trained action recognition network includes: Segmenting the first video data for training based on the action categories to obtain video segments with action labels; Extracting action frame images from the video segment with action labels; Inputting the action frame image into the well-trained detection segmentation model and the well-trained depth estimation model respectively to obtain a set of prediction results for training, wherein the prediction results include the mask map for training and the depth map for training; Inputting a plurality of continuous groups of the prediction results and the corresponding action frame images into an initial action recognition network, and training the action recognition network.

9. A motion recognition system, characterized in that: The system comprises: an intelligent analysis module; the intelligent analysis module is used to implement the steps of any one of the methods of claim 1 to claim 8.

10. The motion recognition system according to claim 9, characterized in that: The system also includes: an image acquisition module and a data display module; The image acquisition module is used to shoot the action video to be identified, and pre-process the action video to be identified and then send it to the intelligent analysis module and the data display module; The data display module is used to display the video of the action to be identified, receive the verification result and action association information output by the intelligent analysis module, and visually display the verification result and the action association information; The intelligent analysis module is also used to extract the video frame sequence to be identified from the pre-processed action video to be identified.

Images