Operation stage identification method and device based on video image intelligent understanding

Abstract

The invention relates to the technical field of computer vision, in particular to an operation stage identification method and device based on video image intelligent understanding. The method comprises the following steps: training an AVT network by using picture information and label information of each video frame to obtain a trained AVT network; based on the trained AVT network, performing feature extraction on the video frame information to obtain spatial features of each video frame; on the basis of a local time feature aggregator, according to the spatial features of each video frame, operation stage prediction is carried out through a head module, and a frame-by-frame operation stage prediction result is obtained; constructing a plurality of time proposals of the operation stage according to the spatial features of each video frame based on the classification network; and according to the plurality of time proposals, correcting the frame-by-frame operation stage prediction result to obtain a final operation stage prediction result. The operation stage identification method is high in intra-stage prediction consistency and high in accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method and device for identifying surgical stages based on intelligent understanding of video images. Background Art

[0002] In recent years, automatic computer-assisted surgery systems have proven to have made outstanding contributions to the support of operating room technology solutions. By monitoring the surgical process through video, the system can warn of potential surgical deviations and errors. Surgical stage prediction has a wide range of applications in surgical diagnosis, surgical report generation, and surgical quality assessment.

[0003] Using computer vision and deep learning technology, surgical video data is analyzed to identify and predict surgical stages. The system used in this method includes a visual model module and a language model module to extract and convert visual information in surgical videos. The multimodal surgical large model CARES Copilot 1.0 combines the extensive knowledge of the large model with the professional precision of the small model. It is capable of advanced image and video semantic extraction and uses language models for deep understanding, including surgical risk assessment and detailed analysis of surgical stages. However, the current recognition results of surgical video stages usually have problems such as phase jitter within the stage and poor prediction consistency.

[0004] In the prior art, there is a lack of a surgical stage identification method with strong intra-stage prediction consistency and high accuracy. Summary of the invention

[0005] In order to solve the technical problems of poor prediction consistency and phase jitter in the surgical stage in the prior art, the embodiment of the present invention provides a surgical stage recognition method and device based on intelligent understanding of video images. The technical solution is as follows:

[0006] On the one hand, a method for identifying a surgical stage based on intelligent understanding of a video image is provided, the method being implemented by a surgical stage identification device, the method comprising:

[0007] Acquire video frame information of the surgical video; perform data processing on the video frame information to obtain image information and label information of each video frame;

[0008] Using the image information and label information of each video frame, training the AVT network to obtain a trained AVT network;

[0009] Based on the trained AVT network, feature extraction is performed on the video frame information to obtain spatial features of each video frame;

[0010] Based on the local time feature aggregator, according to the spatial features of each video frame, the head module predicts the surgical stage to obtain a frame-by-frame surgical stage prediction result;

[0011] Based on the classification network, multiple time proposals of the surgical stage are constructed according to the spatial features of each video frame;

[0012] The frame-by-frame surgical stage prediction results are corrected according to the multiple time proposals to obtain a final surgical stage prediction result.

[0013] On the other hand, a device for identifying a surgical stage based on intelligent understanding of video images is provided, and the device is applied to a method for identifying a surgical stage based on intelligent understanding of video images, and the device comprises:

[0014] An information acquisition module is used to acquire video frame information of the surgical video; perform data processing on the video frame information to obtain image information and label information of each video frame;

[0015] An AVT network training module, used to train the AVT network using the image information and label information of each video frame to obtain a trained AVT network;

[0016] A spatial feature extraction module, used to extract features from the video frame information based on the trained AVT network to obtain spatial features of each video frame;

[0017] A surgery stage prediction module, which is used to predict the surgery stage through a head module based on a local time feature aggregator and the spatial features of each video frame to obtain a frame-by-frame surgery stage prediction result;

[0018] A time proposal construction module, used for constructing multiple time proposals of the surgical stage according to the spatial features of each video frame based on a classification network;

[0019] The prediction result correction module is used to correct the frame-by-frame surgical stage prediction result according to the multiple time proposals to obtain the final surgical stage prediction result.

[0020] On the other hand, a surgical stage identification device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned surgical stage identification methods based on intelligent understanding of video images is implemented.

[0021] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned surgical stage identification methods based on intelligent understanding of video images.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] The present invention proposes a surgical stage recognition method based on intelligent understanding of video images, extracting the spatial features of each video frame through an AVT model; extracting and aggregating local features and global features based on the spatial features of each video frame, fusing the local features and global features, and obtaining frame-by-frame surgical stage prediction results; calculating the stage start time point and end time point based on the spatial features of each video frame; using the surgical stage start time point and end time point, correcting the results of frame-by-frame surgical stage prediction, jointly predicting the final surgical stage, and improving the accuracy of intra-stage prediction. The present invention is a surgical stage recognition method with strong intra-stage prediction consistency and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 It is a flow chart of a surgical stage recognition method based on intelligent understanding of video images provided by an embodiment of the present invention;

[0026] Figure 2 It is a block diagram of a surgical stage identification device based on intelligent understanding of video images provided by an embodiment of the present invention;

[0027] Figure 3 It is a structural schematic diagram of a surgical stage identification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0029] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0030] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0031] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0032] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0033] The embodiment of the present invention provides a method for identifying surgical stages based on intelligent understanding of video images. The method can be implemented by a surgical stage identification device, which can be a terminal or a server. Figure 1 The flowchart of the surgical stage recognition method based on intelligent understanding of video images is shown in the figure. The processing flow of the method may include the following steps:

[0034] S1. Obtain video frame information of the surgical video; perform data processing on the video frame information to obtain image information and label information of each video frame.

[0035] Optionally, data processing is performed on the video frame information to obtain image information and label information of each video frame, including:

[0036] Perform data enhancement processing on the video frame information to obtain a video frame data set;

[0037] Constructing a surgery stage index dictionary based on the video frame dataset;

[0038] According to the surgical stage index dictionary, the image information and label information of each video frame are obtained.

[0039] In a feasible implementation manner, the present invention performs data enhancement methods such as cropping and rotating video frames in surgical video data to produce a surgical video frame data set; constructs a surgical stage index dictionary, which contains a preset number of surgical stages and corresponding labels.

[0040] The surgical video frame information is dimensional video frame information, where is the resolution of the image, 3 is the number of RGB channels of the image, and T is the number of video frames, that is, the time dimension.

[0041] S2. Use the image information and label information of each video frame to train the AVT network to obtain the trained AVT network.

[0042] In a feasible implementation, the present invention adopts a video model architecture (Anticipatory Video Transformer, AVT) for predicting future actions, the core of which is to use the Transformer architecture to process and predict future actions in videos. Unlike traditional convolutional neural networks or recurrent neural networks, AVT uses a causal attention mechanism to predict future actions and features based only on observed frames.

[0043] S3. Based on the trained AVT network, feature extraction is performed on the video frame information to obtain the spatial features of each video frame.

[0044] Optionally, feature extraction is performed on the video frame information through the trained AVT network to obtain spatial features of each video frame, including:

[0045] Input the video frame information into the ViT model for feature vector encoding processing to obtain a first encoding feature;

[0046] The first coding feature is input into the Casual Transformer layer for future information hiding processing to obtain the second coding feature;

[0047] According to the second encoding feature, a softmax function is used to perform linear mapping to obtain the spatial features of each video frame.

[0048] In one feasible implementation, in this step, input dimensional video frame information, the output is dimensional spatial characteristics.

[0049] Based on the ViT model, each resolution dimension The video frames are divided into Non-overlapping small blocks, 196 small blocks are obtained, each small block is flattened into a 256-dimensional vector; then linearly projected to 768 dimensions, each video frame is encoded into Before encoding each frame of the image, add a randomly initialized class token (the dimension of the class token is 768 dimensions, used for classification) to get The dimensional features are used for subsequent processing.

[0050] Bundle The features are fed into a 12-head, 12-cascaded 768-dimensional Transformer encoder. The output feature dimension is still .

[0051] Take the class token features of each video frame, and the dimension becomes , adjust the dimension , and then use linear mapping to project the dimension into The Casual Transformer layer consists of 4 heads and 6 cascaded 2048-dimensional Transformer encoders with causal masks. The causal mask is used to hide future information and not consider future information when calculating attention. The dimension of the output of the Casual Transformer layer is .

[0052] Use linear mapping to adjust the 2048 dimensions to the number of stages, and then use the softmax function to get the probability of each stage. The stage with the largest probability is the prediction result.

[0053] When extracting spatial features, the Casual Transformer layer and classification head are removed, and the class token of ViT-B / 16 is used as the output spatial feature, that is, dimensional spatial characteristics.

[0054] In the next training phase, the spatial features of all videos in the training and test sets are extracted. Corresponding to a spatial feature .

[0055] S4. Based on the local temporal feature aggregator, the head module is used to predict the surgical stage according to the spatial features of each video frame to obtain frame-by-frame surgical stage prediction results.

[0056] Optionally, based on the local temporal feature aggregator, according to the spatial features of each video frame, the head module is used to predict the surgical stage, and the frame-by-frame surgical stage prediction results are obtained, including:

[0057] According to the spatial features of each video frame, feature extraction is performed through a local time feature aggregator to obtain the local features of each video frame; the local features of each video frame are subjected to dimensionality reduction processing to obtain the dimensionality-reduced local features of each video frame;

[0058] In the time dimension, the element-by-element maximum value is calculated based on the dimension-reduced local features of each video frame to obtain the global features of each video frame;

[0059] According to the local features of each video frame and the global features of each video frame, the head module is used to fuse them to obtain the frame-by-frame surgical stage prediction results.

[0060] In a feasible implementation manner, the present invention uses local feature extraction in this step (local features are denoted as , l is the abbreviation of local) combined with global feature extraction (global features are recorded as , g is the abbreviation of global) to predict the surgical stage.

[0061] The local features are reduced from 512 to 64 dimensions using linear mapping to obtain the intermediate quantity . Function represents a linear mapping operation; is the weight matrix of the linear mapping, with dimension ; is the bias vector with dimension .

[0062] Initialize the global features at time i=1: ; At other times when i>1, use the element-by-element maximum calculation method to use and the global features of the previous moment The global feature at the i-th moment is calculated and the mathematical expression of the global feature is as follows (1).

[0063] (1)

[0064] in, represents the jth bit of the global feature at the i-th moment, represents the intermediate value of the local feature at the i-th moment, Represents the jth bit of the intermediate value of the local feature at the i-th moment.

[0065] The global feature uses a linear mapping method to make it consistent with the local feature dimension. The output result is as follows:

[0066] (2)

[0067] in, Represents global features, Represents local features, It is an intermediate quantity. The fused features obtained through a residual layer are as follows (3):

[0068] (3)

[0069] in, Represents the final fused features.

[0070] Perform linear changes on the fused features, adjust the dimension to the number of stages, and use the softmax function to get the probability of each stage , the stage with the highest probability is the predicted result. The mathematical expression of this process is as follows (4):

[0071] (4)

[0072] In order to attach importance to the previous key information in the frame-by-frame prediction network, especially during the phase change, a one-dimensional asymmetric Gaussian kernel is used to project the transition frame onto the phase change map h. The mapping value of the tth frame is expressed as follows (5):

[0073] (5)

[0074] in, Current phase The index of the frame where the start is, and Deviation The left and right nucleus lengths.

[0075] The loss function of the frame-by-frame prediction network is calculated as follows (6):

[0076] (6)

[0077] in, Represents the prediction heat map Its grounded true value Between loss, and Represents the predicted phase Its grounded true value The cross entropy loss between .

[0078] Among them, the local temporal feature aggregator consists of a 2-layer cascaded transformer self-attention encoder and a 2-layer cascaded transformer cross-attention decoder.

[0079] In one feasible implementation, a local temporal feature aggregator based on transfomer is used with a window size of 100 to extract the temporal context information of the video frame as the local feature The local temporal feature aggregator consists of two branches, including a 2-layer cascaded transformer self-attention encoder and a 2-layer cascaded transformer cross-attention decoder. The spatial features extracted by the AVT model are copied into two groups, one as the input of the encoder and the other as the input of the decoder. The output of the encoder will be used as another input of the decoder to calculate the cross attention.

[0080] S5. Based on the classification network, multiple temporal proposals of surgical stages are constructed according to the spatial features of each video frame.

[0081] Optionally, based on the classification network, multiple temporal proposals of the surgical stage are constructed according to the spatial features of each video frame, including:

[0082] In the time dimension, according to the spatial features of each video frame, the video is downsampled to obtain the spatial features of each video at multiple time scales;

[0083] Based on the preset time center point, according to the spatial features of each video at multiple time scales, probability prediction is performed through the classification network to obtain the prediction probability of each video frame at each time scale;

[0084] Construct multiple preliminary temporal proposals based on the predicted probability of each video frame at each time scale;

[0085] For multiple preliminary time proposals, the soft-nms method is used to deduplicate and obtain time proposals.

[0086] In one feasible implementation, the spatial features are downsampled six times with a sampling rate of 2, and the obtained , ,……, Spatial features of different time dimensions. These six spatial features of different time dimensions, together with the original spatial features, form seven layers of features, and the number of layers is recorded as (1≤ ≤7), the first layer is the original spatial features.

[0087] According to the extracted video features of multiple time scales, probability prediction is performed through the classification network to obtain the prediction probability of each video frame, where the prediction probability includes the start time probability, the end time probability and the stage prediction probability, as shown in the following equations (7), (8) and (9):

[0088] (7)

[0089] (8)

[0090] (9)

[0091] The preset time center point is randomly generated based on the time center point regression network: it contains two input channels with 512 dimensions, the output channel with 512 dimensions, and the convolution kernel size is One-dimensional convolution layer; the number of input channels is 512, and the number of output channels is , the convolution kernel size is One-dimensional convolution. (bin is a hyperparameter, Indicates the length of the time period. The relative boundary probability is calculated within this time period, not using the entire video. From all the downsampled video features, select a time length of The time center point regression network outputs a conditional distribution probability. , It indicates the probability that the tth moment is the starting point or end point of the stage relative to the ith feature when the ith moment is the time center point.

[0092] The calculation method of the starting point and the end point is as follows (10) and (11):

[0093] (10)

[0094] (11)

[0095] in, The function is a function that identifies the time position in the sequence that produces the maximum value. Therefore, given the central feature, the two probability sequences are added and the position with the highest probability is selected as the starting point and the end point. The time indicated The sampling rate of the layer. The starting point of the stage in the time proposal, The end point of the stage in the time proposal.

[0096] The format of the generated time proposal is stage label, stage prediction probability, start time point, and end time point.

[0097] Each time center point generates a time proposal, and different time points may generate the same or very close time start and end points. Use the soft-nms method to remove duplicates. For all time proposals, first classify them according to the predicted stage category; for each stage category, first find the time proposal with the highest prediction probability; for other time proposals in the stage category, calculate its time overlap with the time proposal with the highest prediction probability. If the time overlap is greater than the given threshold, reduce the stage prediction probability. If the time overlap is less than the threshold, do not change it.

[0098] For the segment regression loss calculated from the start to the end features and offsets, the intersection loss (IoU) is adjusted to a one-dimensional time context. The loss function for surgical stage localization is calculated as follows (12):

[0099] (12)

[0100] in, , , , is the actual value and the predicted distance between the center point and the start or end point. is calculated by dividing the union of the ground truth segment and the predicted segment by the intersection of the two segments in the time dimension.

[0101] Optionally, based on a preset time center point and according to video features at multiple time scales, probability prediction is performed through a classification network to obtain a prediction probability of each video frame at each time scale, including:

[0102] Based on the preset time center point, according to the video features of multiple time scales, the video features of each time scale are input into the stage starting point network to obtain the starting time probability of each video frame of each time scale;

[0103] Based on the preset time center point, according to the video features of multiple time scales, the video features of each time scale are input into the stage end point network to obtain the end time probability of each video frame of each time scale;

[0104] Based on the preset time center point and according to the video features of multiple time scales, the video features of each time scale are input into the stage recognition network to obtain the stage prediction probability of each video frame of each time scale.

[0105] In a feasible implementation manner, the classification network used in the present invention includes a stage identification network, a stage starting point network, and a stage ending point network.

[0106] The stage recognition network contains two input channels with 512 dimensions, the output channels with 512 dimensions, and the convolution kernel size is One-dimensional convolution layer; the number of input channels used as a classification head is 512 dimensions, the number of output channels is the number of stages, and the convolution kernel size is One-dimensional convolution. It represents the predicted probability of the stage at the i-th moment. The stage with the largest probability is the stage label in the time proposal, and its probability is the stage prediction probability in the time proposal.

[0107] The network at the starting point of the stage contains two input channels with 512 dimensions, the output channel with 512 dimensions, and the convolution kernel size is One-dimensional convolution layer; the number of input channels used as a classification head is 512 dimensions, the number of output channels is the number of stages, and the convolution kernel size is The output result is recorded as , i represents the i-th moment.

[0108] The structure of the network at the end of the stage is the same as that at the start of the stage. The output result is recorded as The outputs of the stage start point network and the stage end point network represent the probability of each moment being the starting point or end point of the action, respectively.

[0109] S6. According to the multiple time proposals, the frame-by-frame surgical stage prediction results are corrected to obtain the final surgical stage prediction results.

[0110] In a feasible implementation, for each stage of surgical prediction, several time proposals belonging to this stage are obtained, such as (stage 1, 0.9, 10, 20), (stage 1, 0.8, 15, 25), (stage 1, 0.7, 19, 29). The time proposal with the largest probability is selected as the corrected data, that is, the data (stage 1, 0.9, 10, 20).

[0111] When the number of stages is 7, each stage corresponds to a time proposal. For the 7 time proposals finally selected, there may be time overlaps, for example: (stage 1, 0.9, 10, 20) and (stage 2, 0.8, 15, 25). When correcting, make the modification according to the result with the larger stage prediction probability.

[0112] For the range not covered by the time proposal, for example: (stage 1, 0.9, 10, 20) and (stage 2, 0.8, 30, 40), 20, 30), no processing is done for this result.

[0113] The present invention proposes a surgical stage recognition method based on intelligent understanding of video images, extracting the spatial features of each video frame through an AVT model; extracting and aggregating local features and global features based on the spatial features of each video frame, fusing the local features and global features, and obtaining frame-by-frame surgical stage prediction results; calculating the stage start time point and end time point based on the spatial features of each video frame; using the surgical stage start time point and end time point, correcting the results of frame-by-frame surgical stage prediction, jointly predicting the final surgical stage, and improving the accuracy of intra-stage prediction. The present invention is a surgical stage recognition method with strong intra-stage prediction consistency and high accuracy.

[0114] Figure 2 The block diagram of a device for identifying surgical stages based on intelligent understanding of video images according to an exemplary embodiment is shown. The device is used in a method for identifying surgical stages based on intelligent understanding of video images. Figure 2 The device includes a first acquisition module 310, a second acquisition module 320, a checking module 330 and a training module 340. Among them:

[0115] The information acquisition module 210 is used to acquire video frame information of the surgical video; perform data processing on the video frame information to obtain image information and label information of each video frame;

[0116] An AVT network training module 220 is used to train the AVT network using the image information and label information of each video frame to obtain a trained AVT network;

[0117] A spatial feature extraction module 230 is used to extract features from video frame information based on the trained AVT network to obtain spatial features of each video frame;

[0118] A surgery stage prediction module 240 is used to predict the surgery stage through a head module based on a local time feature aggregator and according to the spatial features of each video frame, and obtain a frame-by-frame surgery stage prediction result;

[0119] A temporal proposal construction module 250, for constructing a plurality of temporal proposals of the surgical stage according to the spatial features of each video frame based on a classification network;

[0120] The prediction result correction module 260 is used to correct the frame-by-frame surgical stage prediction results according to multiple time proposals to obtain the final surgical stage prediction results.

[0121] Optionally, the information acquisition module 210 is further configured to:

[0122] Perform data enhancement processing on the video frame information to obtain a video frame data set;

[0123] Constructing a surgery stage index dictionary based on the video frame dataset;

[0124] According to the surgical stage index dictionary, the image information and label information of each video frame are obtained.

[0125] Optionally, the spatial feature extraction module 230 is further used to:

[0126] Input the video frame information into the ViT model for feature vector encoding processing to obtain a first encoding feature;

[0127] The first coding feature is input into the Casual Transformer layer for future information hiding processing to obtain the second coding feature;

[0128] According to the second encoding feature, a softmax function is used to perform linear mapping to obtain the spatial features of each video frame.

[0129] Optionally, the surgery stage prediction module 240 is further configured to:

[0130] According to the spatial features of each video frame, feature extraction is performed through a local time feature aggregator to obtain the local features of each video frame; the local features of each video frame are subjected to dimensionality reduction processing to obtain the dimensionality-reduced local features of each video frame;

[0131] In the time dimension, the element-by-element maximum value is calculated based on the dimension-reduced local features of each video frame to obtain the global features of each video frame;

[0132] According to the local features of each video frame and the global features of each video frame, the head module is used to fuse them to obtain the frame-by-frame surgical stage prediction results.

[0133] Among them, the local temporal feature aggregator consists of a 2-layer cascaded transformer self-attention encoder and a 2-layer cascaded transformer cross-attention decoder.

[0134] Optionally, the time proposal construction module 250 is further configured to:

[0135] In the time dimension, according to the spatial features of each video frame, the video is downsampled to obtain the spatial features of each video at multiple time scales;

[0136] Based on the preset time center point, according to the spatial features of each video at multiple time scales, probability prediction is performed through the classification network to obtain the prediction probability of each video frame at each time scale;

[0137] Construct multiple preliminary temporal proposals based on the predicted probability of each video frame at each time scale;

[0138] For multiple preliminary time proposals, the soft-nms method is used to deduplicate and obtain time proposals.

[0139] Optionally, the time proposal construction module 250 is further configured to:

[0140] Based on the preset time center point, according to the video features of multiple time scales, the video features of each time scale are input into the stage starting point network to obtain the starting time probability of each video frame of each time scale;

[0141] Based on the preset time center point, according to the video features of multiple time scales, the video features of each time scale are input into the stage end point network to obtain the end time probability of each video frame of each time scale;

[0142] Based on the preset time center point and according to the video features of multiple time scales, the video features of each time scale are input into the stage recognition network to obtain the stage prediction probability of each video frame of each time scale.

[0143] The present invention proposes a surgical stage recognition method based on intelligent understanding of video images, extracting the spatial features of each video frame through an AVT model; extracting and aggregating local features and global features based on the spatial features of each video frame, fusing the local features and global features, and obtaining frame-by-frame surgical stage prediction results; calculating the stage start time point and end time point based on the spatial features of each video frame; using the surgical stage start time point and end time point, correcting the results of frame-by-frame surgical stage prediction, jointly predicting the final surgical stage, and improving the accuracy of intra-stage prediction. The present invention is a surgical stage recognition method with strong intra-stage prediction consistency and high accuracy.

[0144] Figure 3 is a schematic diagram of the structure of a surgical stage identification device provided by an embodiment of the present invention. Figure 3 As shown, the surgical stage identification device may include the above Figure 2 The surgical stage recognition device based on intelligent understanding of video images is shown. Optionally, the surgical stage recognition device 310 may include a first processor 2001 .

[0145] Optionally, the surgical stage identification device 310 may further include a memory 2002 and a transceiver 2003 .

[0146] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0147] Combine the following Figure 3 The components of the surgical stage identification device 310 are specifically introduced as follows:

[0148] The first processor 2001 is the control center of the surgical stage identification device 310, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0149] Optionally, the first processor 2001 may perform various functions of the surgical stage identification device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0150] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0151] In a specific implementation, as an embodiment, the surgical stage identification device 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0152] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0153] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0154] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0155] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0156] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and be connected to the first processor 2001 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0157] It should be noted that Figure 3 The structure of the surgical stage identification device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0158] In addition, the technical effects of the surgical stage identification device 310 can refer to the technical effects of the surgical stage identification method based on intelligent understanding of video images described in the above method embodiment, and will not be repeated here.

[0159] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0160] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0161] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0162] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0163] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0164] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0165] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0170] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0171] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A surgical stage recognition method based on intelligent understanding of video images, characterized in that: The method comprises: Acquire video frame information of the surgical video; perform data processing on the video frame information to obtain image information and label information of each video frame; Using the image information and label information of each video frame, training the AVT network to obtain a trained AVT network; Based on the trained AVT network, feature extraction is performed on the video frame information to obtain spatial features of each video frame; Based on the local time feature aggregator, according to the spatial features of each video frame, the head module predicts the surgical stage to obtain a frame-by-frame surgical stage prediction result; Based on the classification network, multiple time proposals of the surgical stage are constructed according to the spatial features of each video frame; The frame-by-frame surgical stage prediction results are corrected according to the multiple time proposals to obtain a final surgical stage prediction result.

2. The surgical stage recognition method based on video image intelligent understanding according to claim 1 is characterized in that: The data processing of the video frame information to obtain the picture information and label information of each video frame includes: Performing data enhancement processing on the video frame information to obtain a video frame data set; constructing a surgery stage index dictionary according to the video frame dataset; According to the surgical stage index dictionary, image information and label information of each video frame are obtained.

3. The surgical stage recognition method based on intelligent understanding of video images according to claim 1 is characterized in that: The feature extraction of the video frame information through the trained AVT network to obtain the spatial features of each video frame includes: Inputting the video frame information into the ViT model for feature vector encoding processing to obtain a first encoding feature; Input the first coding feature into the Casual Transformer layer for future information hiding processing to obtain a second coding feature; According to the second encoding feature, a softmax function is used to perform linear mapping to obtain the spatial feature of each video frame.

4. The surgical stage recognition method based on intelligent understanding of video images according to claim 1, characterized in that: The local time feature aggregator performs surgery stage prediction through the head module according to the spatial features of each video frame to obtain frame-by-frame surgery stage prediction results, including: According to the spatial features of each video frame, feature extraction is performed through a local time feature aggregator to obtain local features of each video frame; dimensionality reduction processing is performed on the local features of each video frame to obtain reduced-dimensional local features of each video frame; In the time dimension, performing element-by-element maximum value calculation according to the dimension-reduced local features of each video frame to obtain the global features of each video frame; According to the local features of each video frame and the global features of each video frame, a head module is used to fuse them to obtain frame-by-frame surgical stage prediction results.

5. The surgical stage recognition method based on intelligent understanding of video images according to claim 4 is characterized in that: The local temporal feature aggregator includes a 2-layer cascaded transformer self-attention encoder and a 2-layer cascaded transformer cross-attention decoder.

6. The surgical stage recognition method based on intelligent understanding of video images according to claim 1, characterized in that: The classification network is based on the spatial features of each video frame to construct multiple time proposals of the surgical stage, including: In the time dimension, according to the spatial features of each video frame, down-sampling the video is performed to obtain the spatial features of each video at multiple time scales; Based on a preset time center point, according to the spatial features of each video at the multiple time scales, a probability prediction is performed through a classification network to obtain a prediction probability of each video frame at each time scale; constructing a plurality of preliminary temporal proposals based on the predicted probability of each video frame at each of the temporal scales; The multiple preliminary time proposals are deduplicated using the soft-nms method to obtain time proposals.

7. The surgical stage recognition method based on intelligent understanding of video images according to claim 6, characterized in that: The method of performing probability prediction based on a preset time center point and according to the video features of the multiple time scales through a classification network to obtain the prediction probability of each video frame at each time scale includes: Based on a preset time center point, according to the video features of the multiple time scales, the video features of each time scale are input into a stage starting point network to obtain a starting time probability of each video frame of each time scale; Based on a preset time center point, according to the video features of the multiple time scales, the video features of each time scale are input into the stage end point network to obtain the end time probability of each video frame of each time scale; Based on a preset time center point and according to the video features of the multiple time scales, the video features of each time scale are input into a stage recognition network to obtain the stage prediction probability of each video frame of each time scale.

8. A surgical stage recognition device based on intelligent understanding of video images, the surgical stage recognition device based on intelligent understanding of video images is used to implement the surgical stage recognition method based on intelligent understanding of video images as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An information acquisition module is used to acquire video frame information of the surgical video; perform data processing on the video frame information to obtain image information and label information of each video frame; An AVT network training module, used to train the AVT network using the image information and label information of each video frame to obtain a trained AVT network; A spatial feature extraction module, used to extract features from the video frame information based on the trained AVT network to obtain spatial features of each video frame; A surgery stage prediction module, which is used to predict the surgery stage through a head module based on a local time feature aggregator and the spatial features of each video frame to obtain a frame-by-frame surgery stage prediction result; A time proposal construction module, used for constructing multiple time proposals of the surgical stage according to the spatial features of each video frame based on a classification network; The prediction result correction module is used to correct the frame-by-frame surgical stage prediction result according to the multiple time proposals to obtain the final surgical stage prediction result.

9. A surgical stage identification device, characterized in that: The surgical stage identification device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images