Video data analysis method and device and computer equipment

Abstract

The invention relates to a video data analysis method and device and computer equipment. The method is applied to video data collected by an endoscope, and comprises the following steps: acquiring original video data; segmenting the original video data into video segments with preset lengths; determining feature consistency loss, reconstruction loss and dynamic matching loss of each video segment; based on the feature consistency loss, the reconstruction loss and the dynamic matching loss of each video segment, obtaining the overall loss of each video segment; on the basis of the overall loss, pre-training is carried out on the initial Timeform model, and a pre-trained Timeform model is obtained; obtaining endoscope video data to be analyzed; on the basis of the endoscope video data to be analyzed, the pre-trained Timeform model is adjusted, and a complete Timeform model is obtained; and analyzing the endoscope video data to be analyzed by using the complete Timesformer model to obtain an analysis result. By adopting the method, the problem that the cost is high due to the fact that a large amount of marked data is needed in an existing video data analysis method can be solved.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and in particular to a method, device and computer equipment for analyzing video data. Background Art

[0002] With the advancement of science and technology, endoscopy technology has also developed to a certain extent. The process of using an endoscope is to insert an endoscope into the body to observe the lesion site and perform operations. Therefore, in some cases, such as monitoring the operation process of the endoscope, learning and training medical students on endoscopy operations, conducting postoperative analysis, or providing technical support and services to doctors during telemedicine, it is necessary to automatically analyze the video data generated during the endoscopy operation to identify, understand and predict specific behaviors and events in the endoscopy operation assignment video.

[0003] Existing analysis methods for endoscopic video data are mainly based on deep learning technology, designing reasonable neural networks and loss functions, and using supervised learning methods to achieve intelligent analysis of endoscopic video data. Supervised learning methods require a large amount of labeled data. However, endoscopic cameras often collect 30fps 4K resolution video data, and the collected endoscopic video data contains rich information, such as the movement of surgical instruments, morphological changes of tissues, blood flow, etc. Therefore, it takes a lot of time and complex algorithms to make detailed annotations of endoscopic video data, and the annotation cost is high.

[0004] Existing video data analysis methods require a large amount of labeled data and are costly, and no effective solution has been proposed yet. Summary of the invention

[0005] Based on this, it is necessary to provide a video data analysis method, device and computer equipment to address the above technical issues.

[0006] In a first aspect, the present application provides a method for analyzing video data. Applied to video data collected by an endoscope, the method comprises:

[0007] Acquiring raw video data collected by the endoscope;

[0008] The obtained original video data is divided into video segments of preset length;

[0009] Determining a feature consistency loss, a reconstruction loss, and a dynamic matching loss for each of the video segments;

[0010] Based on the feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments, an overall loss of each of the video segments is obtained;

[0011] Pre-training an initial Timesformer model based on the overall loss of each of the video segments to obtain a pre-trained Timesformer model;

[0012] Acquiring endoscope video data to be analyzed;

[0013] Adjusting the pre-trained Timesformer model based on the endoscope video data to be analyzed to obtain a complete Timesformer model;

[0014] The complete Timesformer model is used to analyze the endoscopic video data to be analyzed to obtain analysis results of the endoscopic video data to be analyzed.

[0015] In one embodiment, determining the feature consistency loss, reconstruction loss and dynamic matching loss of each of the video segments includes:

[0016] For each of the video segments, image frames are captured according to a preset first capture standard, a second capture standard, and a third capture standard, respectively, to obtain a first frame set, a second frame set, and a third frame set for each of the video segments; the first capture standard is to capture image frames according to a preset first capture frequency, the second capture standard is to capture image frames according to a preset second capture frequency, and the third capture standard is to capture one frame of image according to the preset first capture frequency, and then capture one frame of image according to the preset second capture frequency, as a capture cycle, to periodically capture image frames, and the preset first capture frequency is greater than the preset second capture frequency;

[0017] The feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments are determined based on the first frame set, the second frame set and the third frame set of each of the video segments.

[0018] In one of the embodiments, after performing image frame acquisition for each of the video segments according to a preset first acquisition standard, a second acquisition standard, and a third acquisition standard to obtain a first frame set, a second frame set, and a third frame set for each of the video segments, the method further includes:

[0019] A mask operation is performed on the first frame set of each of the video segments to obtain a masked first frame set of each of the video segments.

[0020] In one embodiment, determining the feature consistency loss, the reconstruction loss, and the dynamic matching loss of each of the video segments based on the first frame set, the second frame set, and the third frame set of each of the video segments includes:

[0021] Based on the first frame set, the second frame set and the third frame set of each video segment, respectively, characteristics of the first frame set, characteristics of the second frame set and characteristics of the third frame set of each video segment are obtained;

[0022] The feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments are determined based on the features of the first frame set, the features of the second frame set and the features of the third frame set of each of the video segments.

[0023] In one embodiment, obtaining the features of the first frame set, the features of the second frame set, and the features of the third frame set of each video segment based on the first frame set, the second frame set, and the third frame set of each video segment respectively includes:

[0024] The Encoder module of the initial Timesformer model is used to extract features of the first frame set, the second frame set and the third frame set of each video segment, respectively, to obtain features of the first frame set, features of the second frame set and features of the third frame set of each video segment.

[0025] In one embodiment, determining the feature consistency loss, the reconstruction loss, and the dynamic matching loss of each of the video segments based on the features of the first frame set, the features of the second frame set, and the features of the third frame set of each of the video segments includes:

[0026] Based on the features of the first frame set and the features of the second frame set of each of the video segments, obtaining the feature consistency loss of each of the video segments;

[0027] Based on the features of the first frame set and the features of the second frame set of each video segment, respectively, a first image and a second image of each video segment are constructed; the first image is a next frame image of the current video segment constructed according to the features of the first frame set of the video segment; the second image is a next frame image of the current video segment constructed according to the features of the second frame set of the video segment;

[0028] Determining the next frame image of each of the video segments;

[0029] Determine the reconstruction loss of each of the video segments based on the first image and the second image of each of the video segments and a next frame image of each of the video segments;

[0030] The dynamic matching loss of each video segment is obtained based on the features of the first frame set and the features of the third frame set of each video segment.

[0031] In one embodiment, constructing the first image and the second image of each of the video segments based on the features of the first frame set and the features of the second frame set of each of the video segments respectively includes:

[0032] Based on the features of the first frame set and the features of the second frame set of each video segment, respectively, the first image and the second image of each video segment are constructed using the Decoder module of the initial Timesformer model.

[0033] In one embodiment, the pre-trained Timesformer model is adjusted based on the endoscopic video data to be analyzed to obtain a complete Timesformer model, including:

[0034] Determining a downstream task head of the pre-trained Timesformer model based on the task type of the endoscopic video data to be analyzed;

[0035] Based on the downstream task head of the pre-trained Timesformer model, the pre-trained Timesformer model is adjusted to obtain a complete Timesformer model.

[0036] In a second aspect, the present application also provides a video data analysis device. Applicable to video data collected by an endoscope, the device comprises:

[0037] A raw data acquisition module, used to acquire raw video data collected by the endoscope;

[0038] A segmentation module, used to segment the acquired original video data into video segments of preset length;

[0039] A loss determination module, used to determine the feature consistency loss, reconstruction loss and dynamic matching loss of each of the video segments;

[0040] An overall loss calculation module, used for obtaining an overall loss of each of the video segments based on the feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments;

[0041] A model training module, used for pre-training an initial Timesformer model based on the overall loss of each of the video segments to obtain a pre-trained Timesformer model;

[0042] A module for acquiring data to be analyzed, used for acquiring endoscope video data to be analyzed;

[0043] A fine-tuning module, used for adjusting the pre-trained Timesformer model based on the endoscope video data to be analyzed to obtain a complete Timesformer model;

[0044] and an analysis module, which is used to analyze the endoscopic video data to be analyzed by using the complete Timesformer model to obtain analysis results of the endoscopic video data to be analyzed.

[0045] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video data analysis method described in the first aspect is implemented.

[0046] The above-mentioned video data analysis method, device and computer equipment divide the acquired original video data into video segments of preset lengths, and determine the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment. Then, based on the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, the overall loss of each video segment is obtained. Based on the overall loss of each video segment, the initial Timesformer model is pre-trained to obtain a pre-trained Timesformer model. During the pre-training process, a self-supervised learning mechanism is adopted, and labeled endoscopic data is not required. Based on this, for the endoscopic video data to be analyzed, the required downstream modules are added to the pre-trained Timesformer model for adjustment, and a small amount of data is added for supervised learning to obtain a complete Timesformer model related to the task (such as common tasks such as classification, detection and segmentation). Then, the complete Timesformer model is used to analyze the endoscopic video data to be analyzed to obtain the analysis results of the endoscopic video data, which can solve the problem that the existing video data analysis method requires a large amount of labeled data and has high costs.

[0047] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 A hardware structure block diagram of a terminal for a method for analyzing video data provided in one embodiment of the present application;

[0050] Figure 2A flowchart of a method for analyzing video data provided by an embodiment of the present application;

[0051] Figure 3 A flowchart of a method for analyzing video data provided in a preferred embodiment of the present application;

[0052] Figure 4 This is a structural block diagram of a video data analysis device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0054] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 4 is a hardware structure diagram of a terminal of the video data analysis method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0056] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the video data analysis method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0057] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0058] In this embodiment, a video data analysis method is provided, which is applied to video data collected by an endoscope. Figure 2 is a flow chart of the video data analysis method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0059] Step S210, obtaining original video data collected by the endoscope.

[0060] The above-mentioned original video data can be obtained by setting the video recording parameters of the endoscope to meet the preset requirements. For example, the video recording parameters of the endoscope can be pre-set to obtain video data with an FPS (Frames Per Second) of 30, a resolution of 4k, an RGB type, and an MP4 format. The video recording parameters of the endoscope can also be pre-set to obtain video data with an FPS of 20, a resolution of 4k, an RGB type, and an MP4 format. Specifically, original video data with different parameters can be obtained according to requirements, and the specific parameters of the original video data obtained in this embodiment are not specifically limited. It should be noted that because the original video data is used to pre-train the initial Timesformer model, in order to ensure the reliability of the model training and avoid the failure of the initial Timesformer model training due to the single feature of the endoscope scene acquisition, the above-mentioned original video data needs to be guaranteed to be a mixture of natural scene data and specific medical scene data.

[0061] Step S220: dividing the acquired original video data into video segments of preset length.

[0062] In order to facilitate the subsequent image frame acquisition of the original video data, simplify the calculation process, and improve the data processing efficiency, the original video data needs to be divided into video segments of the same length. The above preset length can be specifically set according to the needs. For example, the preset length can be 2s, that is, the long video of the endoscope collected is divided into video segments of 2s in length.

[0063] Step S230 , determining the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment.

[0064] For the same video segment, the behavior information between the video frame sets collected according to different acquisition frequencies should be close. Therefore, for the same video frame, the features extracted from different video frame sets collected based on different acquisition frequencies should be consistent. The feature consistency loss of each video segment mentioned above can be used to quantify the metric value of the consistency between the features extracted from different video frame sets collected based on different acquisition frequencies for the same video frame. In the process of using the model to predict the next frame image of the same video segment, the same model is used to reconstruct different video frame sets of the same video segment collected according to different acquisition frequencies, and the next frame image of the obtained video segment should be consistent with the actual next frame image of the video segment. The above reconstruction loss is used to quantify the metric value of the consistency between the next frame image of the video segment obtained by reconstructing different video frame sets of the same video segment collected according to different acquisition frequencies using the same model and the actual next frame image of the video segment. Taking into account the application scenarios of endoscopes, there may be inconsistent movement speeds in the video data collected by the endoscope. Based on this, this embodiment uses dynamic matching loss to quantify the consistency between the features extracted from a set of video frames collected according to the same acquisition frequency and a set of video frames collected according to different acquisition frequencies for the same video frame.

[0065] Step S240 , obtaining the overall loss of each video segment based on the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment.

[0066] In this step, the feature consistency loss, reconstruction loss and dynamic matching loss based on each video segment are used to obtain the overall loss of each video segment. The feature consistency loss, reconstruction loss and dynamic matching loss of each video segment are weightedly calculated based on preset weights to obtain the overall loss of each video segment.

[0067] The specific calculation process is as follows:

[0068] L all =λ1L stm +λ2L recon +λ3L dm

[0069] Among them, L all is the overall loss of the current video segment, λ1 is the regularization coefficient of feature consistency loss, λ2 is the regularization coefficient of reconstruction loss, λ3 is the regularization coefficient of dynamic matching loss, and L sim is the feature consistency loss of the current video segment, L recon is the reconstruction loss of the current video segment, L dm is the dynamic matching loss of the current video segment.

[0070] Step S250, pre-training the initial Timesformer model based on the overall loss of each video segment to obtain a pre-trained Timesformer model.

[0071] The above-mentioned initial Timesformer model at least includes an Encoder module and a Decoder module. Among them, the above-mentioned Encoder module can adopt the Timesformer structure to better model the spatiotemporal features for feature extraction of different frame sets of each video segment. The above-mentioned Decoder module can adopt a single-layer Timesformer Block for constructing the next frame image of the current video segment based on the features extracted from the different frame sets of each video segment. It should be noted that because the original video data used in the process of pre-training the initial Timesformer model includes both endoscopic scene data and natural scene data, it can ensure that the trained Timesformer model has good generalization ability, which is convenient for the subsequent migration of downstream tasks according to the endoscopic video data to be analyzed.

[0072] Step S260, obtaining the endoscope video data to be analyzed.

[0073] The endoscopic video data to be analyzed may be endoscopic video data that can determine the task type and needs to be analyzed in actual applications. The task type may be one or more of classification, detection, segmentation, and the like.

[0074] Step S270, adjusting the pre-trained Timesformer model based on the endoscope video data to be analyzed to obtain a complete Timesformer model.

[0075] Because the pre-trained Timesformer model has domain prior knowledge, fewer samples can be collected for downstream tasks. The pre-trained Timesformer model can be fine-tuned using the small sample paradigm to obtain a complete Timesformer model that is suitable for the current endoscopic video data to be analyzed.

[0076] Step S280, using the complete Timesformer model to analyze the endoscope video data to be analyzed, to obtain the analysis result of the endoscope video data to be analyzed.

[0077] In the above steps S210 to S280, the acquired original video data is divided into video segments of preset lengths, and the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment are determined. Then, based on the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, the overall loss of each video segment is obtained. Based on the overall loss of each video segment, the initial Timesformer model is pre-trained to obtain a pre-trained Timesformer model. During the pre-training process, a self-supervised learning mechanism is adopted, and labeled endoscopic data is not required. Based on this, the pre-trained Timesformer model is adjusted for the endoscopic video data to be analyzed to obtain a complete Timesformer model. Then, the complete Timesformer model is used to analyze the endoscopic video data to be analyzed to obtain the analysis results of the endoscopic video data, which can solve the problem that the existing video data analysis method requires a large amount of labeled data and has high costs.

[0078] In one embodiment, the above step S230, determining the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, includes the following steps:

[0079] Step S232, performing image frame acquisition for each video segment according to a preset first acquisition standard, a second acquisition standard and a third acquisition standard, respectively, to obtain a first frame set, a second frame set and a third frame set for each video segment; the first acquisition standard is to acquire image frames according to a preset first acquisition frequency, the second acquisition standard is to acquire image frames according to a preset second acquisition frequency, the third acquisition standard is to acquire one frame of image according to the preset first acquisition frequency, and then to acquire one frame of image according to the preset second acquisition frequency, as a acquisition cycle for periodically acquiring image frames, and the preset first acquisition frequency is greater than the preset second acquisition frequency.

[0080] The above-mentioned acquisition of image frames according to the preset first acquisition frequency may be acquired in a manner of acquiring one image frame every N image frames. The above-mentioned acquisition of image frames according to the preset second acquisition frequency may be acquired in a manner of acquiring one image frame every M image frames. The above-mentioned periodic acquisition of image frames in a manner of acquiring one image frame according to the preset first acquisition frequency and then acquiring one image frame according to the preset second acquisition frequency as an acquisition cycle may be acquired in a manner of periodically acquiring image frames in a manner of acquiring one image frame every N image frames and then acquiring one image frame every M image frames as an acquisition cycle. Wherein, N is less than M. For example, N is 2 frames and M is 4 frames. Therefore, the first frame set acquired according to the preset first acquisition standard may be referred to as a dense frame, and the first frame set acquired according to the preset second acquisition standard may be referred to as a sparse frame. Preferably, the above-mentioned third acquisition standard can also be to periodically acquire image frames in a manner of acquiring one frame of image at a preset first acquisition frequency and then acquiring one frame of image at a preset second acquisition frequency as an acquisition cycle, and then randomly extract some frames in a sequential interpolation manner to ensure that the number of frames in the third frame set is consistent with the number of frames in the first frame set.

[0081] In order to facilitate the calculation of losses and ensure that the model can learn local details and capture global information during the training process. The above-mentioned sparse frames and dense frames can be amplified by different resolution amplification methods. For example, the scale of each image frame in the sparse frame and the dense frame can be adjusted to a size of 256×256. Then, the adjusted image frames are randomly cropped at the center. For example, the center random cropping method can be used to crop each image frame in the sparse frame to a size of 128×128, so that the cropped sparse frame carries local field of view information. The center random cropping method can be used to crop each image frame in the dense frame to a size of 224×224, so that the cropped dense frame carries global field of view information. Preferably, the center random cropping method can also be used to crop each image frame in the third frame set to a size of 128×128 to ensure that the image frames in the third frame set are the same size as the image frames in the sparse frame.

[0082] Step S234: determining the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment based on the first frame set, the second frame set and the third frame set of each video segment.

[0083] The above-mentioned determination of the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment based on the first frame set, the second frame set and the third frame set of each video segment can be based on the first frame set, the second frame set and the third frame set of each video segment, obtaining the features of the first frame set, the features of the second frame set and the features of the third frame set of each video segment, and then determining the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment based on the features of the first frame set, the features of the second frame set and the features of the third frame set of each video segment.

[0084] In the above steps S232 to S234, image frames are captured for each video segment according to the preset first capture standard, second capture standard and third capture standard, to obtain the first frame set, the second frame set and the third frame set of each video segment, and then the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment are determined according to the first frame set, the second frame set and the third frame set of each video segment, so as to obtain the overall loss of each video segment according to the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, and train the model according to the overall loss of each video segment.

[0085] Specifically, in one embodiment, after step S232, the method further includes:

[0086] Step S236: performing a mask operation on the first frame set of each video segment to obtain the masked first frame set of each video segment.

[0087] Because the first frame set is a dense frame, there may be a certain amount of redundancy in the first frame set. Therefore, a mask operation can be performed on the first frame set of each video segment. By adding a mask, a part of the image is deliberately blocked in the self-supervised learning task, so that the model can learn to predict or restore the blocked part, that is, to learn more features through existing features. The above-mentioned mask operation on the first frame set of each video segment can generate a mask matrix according to a preset mask strategy, and then obtain the masked first frame set of each video segment according to the mask matrix. The above-mentioned preset mask strategy can include one or more of a random mask, a center area mask, an edge area mask, a geometric shape mask, and a feature-based mask.

[0088] The above process of performing mask operation on the first frame set of each video segment to obtain the masked first frame set of each video segment is as follows:

[0089] X D1 =Mask(Crop 244 (X D ))

[0090] X D1 , X D ∈R [T，3，H，W]

[0091] Among them, X D1 is the first frame set after the mask of the video segment, X D is the first frame set of the video segment, Crop 244 (X D ) is the first frame set cropped to a size of 224×224 using the center random cropping method. T is the sequence length of the first frame set, H is the height of the image in the first frame set of the video segment, and W is the width of the image in the first frame set of the video segment.

[0092] Further, in one embodiment, based on step S234, based on the first frame set, the second frame set and the third frame set of each video segment, determining the feature consistency loss, the reconstruction loss and the dynamic matching loss of each video segment includes:

[0093] Step S2342 , obtaining features of the first frame set, features of the second frame set, and features of the third frame set of each video segment based on the first frame set, the second frame set, and the third frame set of each video segment respectively.

[0094] The features of the first frame set, the second frame set and the third frame set of each video segment are obtained based on the first frame set, the second frame set and the third frame set of each video segment. The features of the first frame set, the second frame set and the third frame set of each video segment can be obtained based on the first frame set, the second frame set and the third frame set of each video segment using a preset method. The preset method can be one of a preset deep learning model (such as a convolutional neural network), an optical flow method and the like.

[0095] Step S2344, based on the features of the first frame set, the features of the second frame set and the features of the third frame set of each video segment, determine the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment.

[0096] In the above steps S2342 to S2344, features of the first frame set, features of the second frame set and features of the third frame set of each video segment are obtained based on the first frame set, the second frame set and the third frame set of each video segment. Furthermore, the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment are determined according to the features of the first frame set, the second frame set and the third frame set of each video segment. By determining the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, it is convenient to obtain the overall loss of each video segment according to the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, and train the model according to the overall loss of each video segment.

[0097] In addition, in one embodiment, based on step S2342, based on the first frame set, the second frame set and the third frame set of each video segment, respectively, the features of the first frame set, the features of the second frame set and the features of the third frame set of each video segment are obtained, including:

[0098] Step S23422, using the Encoder module of the initial Timesformer model, feature extraction is performed on the first frame set, the second frame set and the third frame set of each video segment to obtain features of the first frame set, the second frame set and the third frame set of each video segment.

[0099] The features of the first frame set of each of the above-mentioned video segments may be features obtained by extracting features from each image of the first frame set of each of the above-mentioned video segments using the Encoder module of the initial Timesformer model. The features of the second frame set of each of the above-mentioned video segments may be features obtained by extracting features from each image of the second frame set of each of the above-mentioned video segments using the Encoder module of the initial Timesformer model. The features of the third frame set of each of the above-mentioned video segments may be features obtained by extracting features from each image of the third frame set of each of the above-mentioned video segments using the Encoder module of the initial Timesformer model.

[0100] In one embodiment, based on step S2344, based on the features of the first frame set, the features of the second frame set, and the features of the third frame set of each video segment, determining the feature consistency loss, the reconstruction loss, and the dynamic matching loss of each video segment includes:

[0101] Step S23441, based on the features of the first frame set and the features of the second frame set of each video segment, obtain the feature consistency loss of each video segment.

[0102] It should be noted that in order to prevent the initial Timesformer model from falling into "collapse", that is, to prevent the initial Timesformer model from converging to a suboptimal solution or losing diversity prematurely, a gradient stopping strategy can be adopted. Specifically, a gradient stopping strategy can be adopted for the features of the first frame set to prevent the initial Timesformer model from overfitting on this branch or falling into a local optimal solution.

[0103] The above process of obtaining the feature consistency loss of each video segment by using the gradient stopping strategy based on the features of the first frame set and the features of the second frame set of each video segment is as follows:

[0104] L sim =∑∑-f D ·log(sg(f s ))

[0105] Among them, fD is the feature of the second frame set of the video segment, f s is the feature of the first frame set of the video segment, and sg() represents the gradient stopping operation.

[0106] Step S23442, constructing the first image and the second image of each video segment based on the characteristics of the first frame set and the characteristics of the second frame set of each video segment respectively; the first image is the next frame image of the current video segment constructed according to the characteristics of the first frame set of the video segment; the second image is the next frame image of the current video segment constructed according to the characteristics of the second frame set of the video segment.

[0107] The first image and the second image of each video segment are constructed based on the features of the first frame set and the features of the second frame set of each video segment, and the first image and the second image of each video segment are constructed based on the features of the first frame set and the features of the second frame set of each video segment, using a preset image prediction method. The preset image prediction method can be one or more of a generative adversarial network, a variational autoencoder, or a recursive neural network.

[0108] Step S23443, determining the next frame image of each video segment.

[0109] In this step, the next frame image of each video segment is determined by reading the next frame image of each video segment from the video stream of the original video data collected by the endoscope.

[0110] Step S23444: Determine the reconstruction loss of each video segment based on the first image and the second image of each video segment and the next frame image of each video segment.

[0111] Based on the first image and the second image of each video segment and the next frame image of each video segment, the reconstruction loss L of each video segment is determined. recon , the specific calculation process is as follows:

[0112]

[0113] Among them, P D is the first image of the video segment, P N is the next frame image of the video segment, P S is the second image of the video segment.

[0114] Step S23445: obtaining the dynamic matching loss of each video segment based on the features of the first frame set and the features of the third frame set of each video segment.

[0115] Based on the features of the first frame set and the third frame set of each video segment, the gradient stopping strategy is adopted to obtain the dynamic matching loss L of each video segment.dm The process is as follows:

[0116] L dm =∑∑-f DS ·log(sg(f D ))

[0117] Among them, f DS is the feature of the third frame set of the video segment.

[0118] The above steps S23441 to S23445 determine the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment based on the features of the first frame set, the features of the second frame set and the features of the third frame set of each video segment. By determining the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, it is convenient to obtain the overall loss of each video segment according to the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, and train the model according to the overall loss of each video segment.

[0119] In addition, in one embodiment, the above step S23442 constructs the first image and the second image of each video segment based on the features of the first frame set and the features of the second frame set of each video segment, respectively, including:

[0120] Based on the features of the first frame set and the features of the second frame set of each video segment, the first image and the second image of each video segment are constructed using the Decoder module of the initial Timesformer model.

[0121] The above-mentioned first image and second image of each video segment are constructed based on the features of the first frame set and the features of the second frame set of each video segment, respectively, using the Decoder module of the initial Timesformer model. The features of the first frame set of each video segment can be used as the input of the Decoder module of the initial Timesformer model to output the first image of each video segment, and the features of the second frame set of each video segment can be used as the input of the Decoder module of the initial Timesformer model to output the second image of each video segment.

[0122] In one embodiment, based on step S270, the pre-trained Timesformer model is adjusted based on the endoscope video data to be analyzed to obtain a complete Timesformer model, including:

[0123] Step S272, based on the task type of the endoscopic video data to be analyzed, determine the downstream task head of the pre-trained Timesformer model.

[0124] The above-mentioned determination of the downstream task head of the pre-trained Timesformer model based on the task type of the endoscopic video data to be analyzed may be based on the task type of the endoscopic video data to be analyzed, determining the downstream task head of the pre-trained Timesformer model corresponding to the task type of the endoscopic video data to be analyzed.

[0125] Step S274, based on the downstream task head of the pre-trained Timesformer model, adjust the pre-trained Timesformer model to obtain a complete Timesformer model.

[0126] The above adjustments may include but are not limited to adding specific task heads, freezing some layers, adjusting the model architecture to adapt to the characteristics of the input data, etc.

[0127] In the above steps S272 to S274, the downstream task head of the pre-trained Timesformer model is determined based on the task type of the endoscopic video data to be analyzed, and the pre-trained Timesformer model is adjusted based on the downstream task head of the pre-trained Timesformer model to obtain a complete Timesformer model suitable for the task type of the endoscopic video data to be analyzed. This facilitates the subsequent use of the complete Timesformer model to analyze the endoscopic video data to be analyzed and obtain the analysis results of the endoscopic video data to be analyzed.

[0128] The present embodiment is described and illustrated below through preferred embodiments.

[0129] Figure 3 FIG. 1 is a flow chart of a method for analyzing video data provided by a preferred embodiment of the present application. The method is applied to video data collected by an endoscope, such as Figure 3 As shown, the video data analysis method includes the following steps:

[0130] Step S301, obtaining original video data collected by the endoscope;

[0131] Step S302, dividing the acquired original video data into video segments of a preset length;

[0132] Step S303, performing image frame acquisition for each video segment according to a preset first acquisition standard, a second acquisition standard, and a third acquisition standard, respectively, to obtain a first frame set, a second frame set, and a third frame set for each video segment;

[0133] Step S304, determining a feature consistency loss, a reconstruction loss, and a dynamic matching loss of each video segment based on the first frame set, the second frame set, and the third frame set of each video segment;

[0134] Step S305, obtaining the overall loss of each video segment based on the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment;

[0135] Step S306, pre-training the initial Timesformer model based on the overall loss of each video segment to obtain a pre-trained Timesformer model;

[0136] Step S307, obtaining endoscope video data to be analyzed;

[0137] Step S308, determining the downstream task head of the pre-trained Timesformer model based on the task type of the endoscope video data to be analyzed;

[0138] Step S309, based on the downstream task head of the pre-trained Timesformer model, adjusting the pre-trained Timesformer model to obtain a complete Timesformer model;

[0139] Step S310, using a complete Timesformer model to analyze the endoscope video data to be analyzed, and obtaining analysis results of the endoscope video data to be analyzed.

[0140] In the above steps S301 to S310, the acquired original video data is divided into video segments of preset lengths, and the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment are determined. Then, based on the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, the overall loss of each video segment is obtained. Based on the overall loss of each video segment, the initial Timesformer model is pre-trained to obtain a pre-trained Timesformer model. During the pre-training process, a self-supervised learning mechanism is adopted, and labeled endoscopic data is not required. Based on this, the pre-trained Timesformer model is adjusted for the endoscopic video data to be analyzed to obtain a complete Timesformer model. Then, the complete Timesformer model is used to analyze the endoscopic video data to be analyzed to obtain the analysis results of the endoscopic video data, which can solve the problem that the existing video data analysis method requires a large amount of labeled data and has high costs.

[0141] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0142] Based on the same inventive concept, a video data analysis device is also provided in this embodiment, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0143] In one embodiment, Figure 4 is a structural block diagram of a video data analysis device provided in an embodiment of the present application, which is applied to video data collected by an endoscope, such as Figure 4 As shown, the video data analysis device comprises:

[0144] The raw data acquisition module 41 is used to acquire the raw video data collected by the endoscope;

[0145] A segmentation module, used to segment the acquired original video data into video segments of preset length;

[0146] A loss determination module 42, for determining a feature consistency loss, a reconstruction loss and a dynamic matching loss for each video segment;

[0147] An overall loss calculation module 43, used to obtain an overall loss of each video segment based on a feature consistency loss, a reconstruction loss, and a dynamic matching loss of each video segment;

[0148] A model training module 44 is used to pre-train the initial Timesformer model based on the overall loss of each video segment to obtain a pre-trained Timesformer model;

[0149] The to-be-analyzed data acquisition module 45 is used to acquire the endoscope video data to be analyzed;

[0150] A fine-tuning module 46, for adjusting the pre-trained Timesformer model based on the endoscope video data to be analyzed to obtain a complete Timesformer model;

[0151] and an analysis module 47, which is used to analyze the endoscope video data to be analyzed by using a complete Timesformer model to obtain analysis results of the endoscope video data to be analyzed.

[0152] The above-mentioned video data analysis device divides the acquired original video data into video segments of preset lengths, and determines the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment. Then, based on the feature consistency loss, reconstruction loss and dynamic matching loss of each video segment, the overall loss of each video segment is obtained. Based on the overall loss of each video segment, the initial Timesformer model is pre-trained to obtain a pre-trained Timesformer model. During the pre-training process, a self-supervised learning mechanism is adopted, and labeled endoscopic data is not required. Based on this, the pre-trained Timesformer model is adjusted for the endoscopic video data to be analyzed to obtain a complete Timesformer model. Then, the complete Timesformer model is used to analyze the endoscopic video data to be analyzed to obtain the analysis results of the endoscopic video data. This can solve the problem that the existing video data analysis method requires a large amount of labeled data and has high costs.

[0153] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0154] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the video data analysis methods in the above embodiments is implemented.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0156] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video data analysis method, applied to video data collected by an endoscope, characterized in that: The method comprises: Acquiring raw video data collected by the endoscope; The obtained original video data is divided into video segments of preset length; Determining a feature consistency loss, a reconstruction loss, and a dynamic matching loss for each of the video segments; Based on the feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments, an overall loss of each of the video segments is obtained; Pre-training an initial Timesformer model based on the overall loss of each of the video segments to obtain a pre-trained Timesformer model; Acquiring endoscope video data to be analyzed; Adjusting the pre-trained Timesformer model based on the endoscope video data to be analyzed to obtain a complete Timesformer model; The complete Timesformer model is used to analyze the endoscopic video data to be analyzed to obtain analysis results of the endoscopic video data to be analyzed.

2. The video data analysis method according to claim 1, characterized in that: The determining of the feature consistency loss, reconstruction loss and dynamic matching loss of each of the video segments comprises: For each of the video segments, image frames are captured according to a preset first capture standard, a second capture standard, and a third capture standard, respectively, to obtain a first frame set, a second frame set, and a third frame set for each of the video segments; the first capture standard is to capture image frames according to a preset first capture frequency, the second capture standard is to capture image frames according to a preset second capture frequency, and the third capture standard is to capture one frame of image according to the preset first capture frequency, and then capture one frame of image according to the preset second capture frequency, as a capture cycle, to periodically capture image frames, and the preset first capture frequency is greater than the preset second capture frequency; The feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments are determined based on the first frame set, the second frame set and the third frame set of each of the video segments.

3. The video data analysis method according to claim 2, characterized in that: After performing image frame acquisition for each of the video segments according to the preset first acquisition standard, the second acquisition standard and the third acquisition standard to obtain the first frame set, the second frame set and the third frame set of each of the video segments, the method further includes: A mask operation is performed on the first frame set of each of the video segments to obtain a masked first frame set of each of the video segments.

4. The video data analysis method according to claim 2, characterized in that: The determining the feature consistency loss, the reconstruction loss, and the dynamic matching loss of each of the video segments based on the first frame set, the second frame set, and the third frame set of each of the video segments comprises: Based on the first frame set, the second frame set and the third frame set of each video segment, respectively, characteristics of the first frame set, characteristics of the second frame set and characteristics of the third frame set of each video segment are obtained; The feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments are determined based on the features of the first frame set, the features of the second frame set and the features of the third frame set of each of the video segments.

5. The video data analysis method according to claim 4, characterized in that: The obtaining, based on the first frame set, the second frame set and the third frame set of each video segment respectively, features of the first frame set, features of the second frame set and features of the third frame set of each video segment comprises: The Encoder module of the initial Timesformer model is used to extract features of the first frame set, the second frame set and the third frame set of each video segment, respectively, to obtain features of the first frame set, features of the second frame set and features of the third frame set of each video segment.

6. The video data analysis method according to claim 4, characterized in that: The determining the feature consistency loss, the reconstruction loss, and the dynamic matching loss of each of the video segments based on the features of the first frame set, the features of the second frame set, and the features of the third frame set of each of the video segments comprises: Based on the features of the first frame set and the features of the second frame set of each of the video segments, obtaining the feature consistency loss of each of the video segments; Based on the features of the first frame set and the features of the second frame set of each video segment, respectively, a first image and a second image of each video segment are constructed; the first image is a next frame image of the current video segment constructed according to the features of the first frame set of the video segment; the second image is a next frame image of the current video segment constructed according to the features of the second frame set of the video segment; Determining the next frame image of each of the video segments; Determine the reconstruction loss of each of the video segments based on the first image and the second image of each of the video segments and a next frame image of each of the video segments; The dynamic matching loss of each video segment is obtained based on the features of the first frame set and the features of the third frame set of each video segment.

7. The video data analysis method according to claim 6, characterized in that: The constructing the first image and the second image of each of the video segments based on the features of the first frame set and the features of the second frame set of each of the video segments respectively comprises: Based on the features of the first frame set and the features of the second frame set of each video segment, respectively, the first image and the second image of each video segment are constructed using the Decoder module of the initial Timesformer model.

8. The video data analysis method according to claim 1, characterized in that: The pre-trained Timesformer model is adjusted based on the endoscope video data to be analyzed to obtain a complete Timesformer model, including: Determining a downstream task head of the pre-trained Timesformer model based on the task type of the endoscopic video data to be analyzed; Based on the downstream task head of the pre-trained Timesformer model, the pre-trained Timesformer model is adjusted to obtain a complete Timesformer model.

9. A video data analysis device, applied to video data collected by an endoscope, characterized in that: The device comprises: A raw data acquisition module, used to acquire raw video data collected by the endoscope; A segmentation module, used to segment the acquired original video data into video segments of preset length; A loss determination module, used to determine the feature consistency loss, reconstruction loss and dynamic matching loss of each of the video segments; An overall loss calculation module, used for obtaining an overall loss of each of the video segments based on the feature consistency loss, the reconstruction loss and the dynamic matching loss of each of the video segments; A model training module, used for pre-training an initial Timesformer model based on the overall loss of each of the video segments to obtain a pre-trained Timesformer model; A module for acquiring data to be analyzed, used for acquiring endoscope video data to be analyzed; A fine-tuning module, used for adjusting the pre-trained Timesformer model based on the endoscope video data to be analyzed to obtain a complete Timesformer model; and an analysis module, which is used to analyze the endoscopic video data to be analyzed by using the complete Timesformer model to obtain analysis results of the endoscopic video data to be analyzed.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the video data analysis method according to any one of claims 1 to 8 are implemented.

Images