Video generation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of digital medical treatment, and relates to a video generation method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining a medical video sequence; performing low-dimensional conversion on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; extracting a space relation and a time-space relation from the first sequence feature vector and the second sequence feature vector to obtain a video feature space relation and a video feature time relation; editing the medical video sequence to obtain a standard medical video sequence; performing potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; and fusing the first potential state sequence and the second potential state sequence to obtain an effective medical video sequence. According to the method and the device, the real and effective medical video sequence with high correlation can be effectively generated according to the provided medical video sequence.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, specifically to the field of digital medicine, and in particular to a video generation method, apparatus, computer equipment and storage medium. Background Art

[0002] In the modern medical service system, with the rapid development of information technology, medical videos have become an indispensable tool for improving the quality of medical services, promoting the popularization of medical education, and optimizing the patient's rehabilitation experience. These videos cover everything from high-tech surgical demonstrations to in-depth and easy-to-understand medical theory explanations, to intuitive and easy-to-understand medical equipment operation guides and personalized rehabilitation training guidance, fully supporting medical practice, teaching and scientific research activities.

[0003] However, traditional video generation technology often faces many challenges, especially when dealing with complex and ever-changing scenarios in the medical field. Due to the high degree of professionalism, sophistication and strict safety requirements of the medical process, traditional methods rely on accurate modeling of scenes, specific medical objects and complex action sequences, which not only requires huge computing resources, but also makes it difficult to ensure the authenticity and relevance of the generated videos. Especially when faced with rare cases, new surgical techniques or personalized rehabilitation programs, traditional methods often find it difficult to quickly adapt and generate high-quality video content, thus limiting the potential of medical videos in promoting knowledge dissemination, improving diagnosis and treatment efficiency and patient self-management. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video generation method, apparatus, computer device and storage medium to solve the problem that a true and highly relevant extended video sequence cannot be generated based on a provided medical video sequence.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a video generation method, which adopts the following technical solution:

[0006] Acquiring medical video sequences;

[0007] Performing low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector;

[0008] Extracting spatial and spatiotemporal relationships of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a spatial relationship of video features and a temporal relationship of video features;

[0009] Editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence;

[0010] Performing latent state extraction on the medical video sequence and the standard medical video sequence to obtain a first latent state sequence and a second latent state sequence;

[0011] The first latent state sequence and the second latent state sequence are fused to obtain a valid medical video sequence.

[0012] Furthermore, the step of acquiring the medical video sequence specifically includes:

[0013] Get the video extraction request identifier;

[0014] Extracting the original medical video from the medical video database according to the video extraction request identifier;

[0015] The original medical video is converted into the medical video sequence.

[0016] Furthermore, the step of performing low-dimensional transformation on the medical video sequence to obtain the first sequence feature vector and the second sequence feature vector specifically includes:

[0017] Acquiring video frame time information of the medical video sequence;

[0018] Extracting the first frame of the medical video sequence as a video static image frame according to the video frame time information;

[0019] Extracting non-first frames of the medical video sequence as a video image frame set according to the video frame time information;

[0020] The video static image frame and the video image frame set are input into a preset convolution encoder for compression encoding to obtain the first sequence feature vector and the second sequence feature vector.

[0021] Furthermore, the step of extracting the spatial relationship and the spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on the spatial window and the spatiotemporal window to obtain the spatial relationship of video features and the temporal relationship of video features specifically includes:

[0022] Obtain preset spatial window definition rules and spatiotemporal window definition rules;

[0023] Creating the spatial window and the spatiotemporal window according to the spatial window definition rule and the spatiotemporal window definition rule;

[0024] Performing spatial relationship analysis on the first sequence feature vectors and the second sequence feature vectors according to the spatial window to obtain the video feature spatial relationship;

[0025] The spatiotemporal relationship between the first sequence feature vectors and the second sequence feature vectors is analyzed according to the spatiotemporal window to obtain the video feature time relationship.

[0026] Furthermore, the step of editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence specifically includes:

[0027] Selecting a medical video key frame in the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features;

[0028] Logically segmenting the medical video sequence according to the medical video key frames to obtain medical video sequence segments;

[0029] The medical video sequence segments are adjusted, enhanced and combined to obtain the standard medical video sequence.

[0030] Furthermore, the step of extracting latent states from the medical video sequence and the standard medical video sequence to obtain a first latent state sequence and a second latent state sequence specifically includes:

[0031] Obtaining a first model extraction identifier and a second model extraction identifier, and extracting a feature extraction model and a potential state extraction model from a model database according to the first model extraction identifier and the second model extraction identifier;

[0032] Inputting the medical video sequence and the standard medical video sequence into the feature extraction model to obtain a first video sequence feature and a second video sequence feature;

[0033] The first video sequence feature and the second video sequence feature are input into the latent state extraction model to obtain the first latent state sequence and the second latent state sequence.

[0034] Furthermore, the step of fusing the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence specifically includes:

[0035] Time-aligning the first latent state sequence and the second latent state sequence based on dynamic time warping;

[0036] Acquire a preset fusion strategy, and perform sequence fusion on the time-aligned first latent state sequence and the second latent state sequence according to the preset fusion strategy to obtain a fused state sequence;

[0037] The fusion state sequence should be set to a video content space to generate the valid medical video sequence.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a video generation device, which adopts the following technical solution:

[0039] A sequence acquisition module, used to acquire medical video sequences;

[0040] A sequence conversion module, used for performing low-dimensional conversion on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector;

[0041] A relationship extraction module, used for extracting spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window, so as to obtain a spatial relationship of video features and a temporal relationship of video features;

[0042] A sequence editing module, used for editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence;

[0043] A state extraction module, used for performing potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence;

[0044] The state fusion module is used to fuse the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0046] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above video generation methods when executing the computer-readable instructions.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0048] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method described in any one of the above items are implemented.

[0049] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment obtains a medical video sequence; performs a low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; extracts the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a video feature spatial relationship and a video feature temporal relationship; edits the medical video sequence according to the video feature spatial relationship and the video feature temporal relationship to obtain a standard medical video sequence; performs potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; fuses the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence. Thus, it is effectively realized to generate a true and highly relevant valid medical video sequence according to the provided medical video sequence, so as to provide effective data support for medical research and operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0052] Figure 2 A flowchart of an embodiment of a video generation method according to the present application;

[0053] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S10;

[0054] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S20;

[0055] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S30;

[0056] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S40;

[0057] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S50;

[0058] Figure 8 yes Figure 2 A flowchart of a specific implementation of step S60;

[0059] Fig. 9 is a structural schematic diagram of an embodiment of a video generating device according to the present application;

[0060] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0065] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0067] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0068] It should be noted that the video generating method provided in the embodiment of the present application is generally executed by a server, and accordingly, the video generating device is generally arranged in the server.

[0069] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of the method for image recognition according to the present application. The video generation method comprises the following steps:

[0071] Step S10, acquiring a medical video sequence;

[0072] In this embodiment, the medical video sequence is a video sequence obtained by converting the original medical video. The video sequence refers to a set of static images indexed by time sequence, and contains all information content including audio, text, and images. The medical video sequence can come from hospitals and medical institutions, medical teaching and research institutions, medical equipment manufacturers, etc.

[0073] Step S20, performing low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector;

[0074] In this embodiment, low-dimensional conversion refers to the process of converting high-dimensional data into low-dimensional data through dimensionality reduction techniques such as mapping or transformation. In this embodiment, the first sequence feature vector is a feature vector obtained by low-dimensional conversion of the first frame in the medical video sequence, and the second sequence feature vector is a feature vector obtained by low-dimensional conversion of the remaining frames in the medical video sequence.

[0075] Step S30, extracting the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on the spatial window and the spatiotemporal window to obtain the spatial relationship of video features and the temporal relationship of video features;

[0076] In this embodiment, the spatial window refers to one or more window areas defined on a static video image frame. This window area can be the entire image or a part of the image. In this embodiment, the spatial window corresponds to several window areas in the video frame of the first sequence feature vector. In the specific implementation, the size and shape of the spatial window can be adjusted as needed to more finely capture and analyze the information in the space. The spatiotemporal window adds the consideration of the time dimension on the basis of the spatial window. It not only focuses on the spatial information at a certain moment, but also focuses on the changes of these spatial information over time. In this embodiment, the spatiotemporal window refers to the same spatial area in the second sequence feature vector including multiple consecutive frames, which is used to analyze the changes of objects or features in the area over time. The spatial relationship of video features refers to the spatial relationship between each window area in the static video image frame corresponding to the first sequence feature vector, including similarity, degree of difference, etc. The temporal relationship of video features refers to the temporal change relationship of the same spatial area of ​​the consecutive video frames corresponding to the second sequence feature vector, including the rate of change, trajectory, motion mode, etc.

[0077] Step S40, editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence;

[0078] In this embodiment, editing the medical video sequence refers to logically segmenting the medical video sequence and adjusting and enhancing the medical video sequence to obtain a standard medical video sequence that meets the requirements and is convenient for subsequent processing.

[0079] Step S50, performing latent state extraction on the medical video sequence and the standard medical video sequence to obtain a first latent state sequence and a second latent state sequence;

[0080] In this embodiment, the extraction of latent states is achieved through a pre-trained latent state extraction model, and the first latent state sequence refers to extracting hidden, high-level features or patterns from a medical video sequence, which can reflect key information or dynamic changes in the medical video sequence. The second latent state sequence refers to extracting hidden, high-level features or patterns from a standard medical video sequence, which can reflect key information or dynamic changes in the standard medical video sequence.

[0081] Step S60: fusing the first latent state sequence and the second latent state sequence to obtain a valid medical video sequence.

[0082] In this embodiment, a sequence fusion operation is performed on the first latent state sequence and the second latent state sequence based on a preset fusion strategy to generate a valid medical video sequence that has both the representativeness and information content of the first latent state sequence and the second latent state sequence.

[0083] This embodiment obtains a medical video sequence; performs a low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; extracts the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a video feature spatial relationship and a video feature temporal relationship; edits the medical video sequence according to the video feature spatial relationship and the video feature temporal relationship to obtain a standard medical video sequence; performs potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; and fuses the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence. This effectively realizes the generation of a true and highly relevant valid medical video sequence based on the provided medical video sequence, so as to provide effective data support for medical research and operation.

[0084] Continue to refer Figure 3 In some optional implementations of this embodiment, step S10 includes the following steps:

[0085] Step S101, obtaining a video extraction request identifier;

[0086] In this embodiment, the video extraction request identifier is a unique request identifier corresponding to the original medical video, and the request identifier can be an ID or a file name. The video extraction request identifier can be generated by a user sending a video extraction request in the system, and the operation of the user sending a video extraction request in the system can be achieved by the user clicking a video extraction button on the system.

[0087] Step S102, extracting the original medical video from the medical video database according to the video extraction request identifier;

[0088] In this embodiment, the video extraction request identifier is used as a query condition, and a query match is performed in the medical video database to find the corresponding medical original video. In this embodiment, the medical original video refers to a medical-related video file, including a surgical process record: recording the entire process of the doctor performing the operation. Case analysis and discussion: showing the imaging data of a specific case, the patient's clinical manifestations, and the doctor's detailed analysis and discussion of the case. Medical teaching materials: including anatomical demonstrations, pathological specimen observations, physiological function demonstrations, etc. Medical equipment operation demonstrations: showing the use methods, operation steps, and equipment principles of medical equipment (such as CT, MRI, ultrasound, etc.). Rehabilitation training guidance: providing video guidance for rehabilitation training for patients, including physical therapy, rehabilitation training action demonstrations, etc. Patient education and communication: videos that explain disease knowledge, treatment plans, rehabilitation processes, etc. to patients. Medical research and experiments: videos that record medical experiments or research processes, including animal experiments, cell culture, drug trials, etc. Emergency rescue and first aid demonstrations: videos that show emergency rescue and first aid skills, such as cardiopulmonary resuscitation (CPR), hemostasis, bandaging, etc.

[0089] Step S103: converting the original medical video into the medical video sequence.

[0090] In this embodiment, the step of converting the original medical video into a medical video sequence includes video decoding: the process of restoring the compressed data in the original medical video to a playable frame sequence; frame extraction: after video decoding, reading the pictures in the original medical video frame by frame to obtain medical video frames; frame saving: saving the extracted medical video frames as image files and arranging them in chronological order to obtain a medical video sequence.

[0091] This embodiment obtains a video extraction request identifier; extracts a medical original video from a medical video database according to the video extraction request identifier; and converts the medical original video into the medical video sequence, thereby effectively obtaining a medical video sequence arranged in chronological order to facilitate subsequent low-dimensional conversion processing.

[0092] Continue to refer Figure 4 In some optional implementations of this embodiment, step S20 includes the following steps:

[0093] Step S201, obtaining video frame time information of the medical video sequence;

[0094] In this embodiment, by parsing the metadata of the medical video sequence, the timestamp corresponding to each frame of the medical video sequence is obtained to obtain the video frame time information.

[0095] Step S202, extracting the first frame of the medical video sequence as a video static image frame according to the video frame time information;

[0096] In this embodiment, the first frame of the medical video sequence is located by using the video frame time information. The first frame refers to the video frame that is the earliest in time in the medical video sequence according to the video frame time information. The video frame is used as a video static image frame representing the starting state of the medical video sequence.

[0097] Step S203, extracting non-first frames of the medical video sequence as a video image frame set according to the video frame time information;

[0098] In this embodiment, after the video static image frame is extracted, the remaining frames (ie, not the first frame) in the medical video sequence are selected and extracted, thereby obtaining a video image frame set representing the dynamic changes of the medical video sequence.

[0099] Step S204: input the video static image frame and the video image frame set into a preset convolution encoder for compression encoding to obtain the first sequence feature vector and the second sequence feature vector.

[0100] In this embodiment, the preset convolution encoder is a convolutional neural network encoder including a causal three-dimensional convolutional layer, and the step of inputting the video still image frame and the video image frame set into the preset convolutional encoder for compression encoding includes: setting the number of channels of the still image frame to the same number of channels as the video frame (for example, for RGB images, the number of channels is 3); expanding the still image frame into three-dimensional data through an additional dimension (for example, the time dimension size is 1) to obtain a three-dimensional still image frame; converting the three-dimensional still image frame into a one-dimensional first sequence feature vector through the causal three-dimensional convolutional layer, activation layer, and pooling layer of the convolutional neural network; inputting the video image frame set into the convolutional neural network encoder, extracting a two-dimensional feature map (number of time steps x number of features) through the causal three-dimensional convolutional layer, activation layer, and pooling layer, and further processing the two-dimensional feature map into a one-dimensional second sequence feature vector.

[0101] This embodiment obtains the video frame time information of the medical video sequence; extracts the first frame of the medical video sequence as a video static image frame according to the video frame time information; extracts the non-first frame of the medical video sequence as a video image frame set according to the video frame time information; inputs the video static image frame and the video image frame set into a preset convolution encoder for compression encoding, thereby effectively obtaining a first sequence feature vector and a second sequence feature vector for encoding the first frame of the medical video sequence and the non-first frame of the medical video sequence, respectively, to facilitate the subsequent extraction of spatial relationships and spatiotemporal relationships.

[0102] Continue to refer Figure 5 In some optional implementations of this embodiment, step S30 includes the following steps:

[0103] Step S301, obtaining preset spatial window definition rules and spatiotemporal window definition rules;

[0104] In this embodiment, the spatial window definition rule defines how to select one or more regions (i.e., windows) in the two-dimensional space of the video frame to analyze spatial features. For example, it can be a fixed rectangular area, an area dynamically adjusted according to the target position, or an area obtained based on an image segmentation algorithm. The spatiotemporal window definition rule defines how to select one or more spatiotemporal blocks (i.e., spatiotemporal windows) in a continuous sequence of video frames to analyze spatiotemporal features. This can include a continuous frame sequence of fixed length, a window based on event triggering, or a spatiotemporal block combined with a motion trajectory.

[0105] Step S302, creating the spatial window and the spatiotemporal window according to the spatial window definition rule and the spatiotemporal window definition rule;

[0106] In this embodiment, the spatial window and the spatiotemporal window are created according to the above-mentioned spatial window definition rules and the spatiotemporal window definition rules. The spatial window and the spatiotemporal window can be static (remain unchanged during the entire analysis process) or dynamic (adjusted according to changes in video content).

[0107] Step S303, performing spatial relationship analysis on the first sequence feature vectors and the second sequence feature vectors according to the spatial window to obtain the video feature spatial relationship;

[0108] In this embodiment, the first sequence feature vectors and the second sequence feature vectors are input into a spatial window for processing, and their spatial position relationship (such as adjacent, overlapping, and distant), similarity (such as Euclidean distance or cosine similarity of feature vectors), direction relationship, etc. are analyzed to obtain the spatial relationship of video features.

[0109] Step S304: performing a spatiotemporal relationship analysis on the first sequence feature vectors and the second sequence feature vectors according to the spatiotemporal window to obtain the video feature temporal relationship.

[0110] In this embodiment, the first sequence feature vectors and the second sequence feature vectors are input into a spatiotemporal window for processing, and their changes between consecutive frames are analyzed, including their motion trajectories, speeds, accelerations, and their interactions and dependencies, so as to obtain the video feature time relationship.

[0111] This embodiment obtains preset spatial window definition rules and spatiotemporal window definition rules; creates the spatial window and the spatiotemporal window according to the spatial window definition rules and the spatiotemporal window definition rules; performs spatial relationship analysis on the first sequence feature vector and the second sequence feature vector according to the spatial window to obtain the video feature spatial relationship; performs spatiotemporal relationship analysis on the first sequence feature vector and the second sequence feature vector according to the spatiotemporal window to obtain the video feature temporal relationship. Thus, the first sequence feature vector and the second sequence feature vector are analyzed from the spatial and spatiotemporal dimensions respectively to obtain the corresponding video feature spatial relationship and video feature temporal relationship, which facilitates the subsequent editing and processing of the medical video sequence.

[0112] Continue to refer Figure 6 In some optional implementations of this embodiment, step S40 includes the following steps:

[0113] Step S401, selecting a medical video key frame in the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features;

[0114] In this embodiment, based on the spatial relationship of video features (such as the similarity and positional relationship of feature vectors) and the temporal relationship of video features (such as the change trajectory, speed, acceleration, etc. of feature vectors) obtained by the previous analysis, frames with significant changes or important information in the video can be identified in the medical video sequence as medical video key frames. The medical video key frames usually represent the transition of surgical steps, significant changes in lesions, important operations of doctors, etc.

[0115] Step S402, logically segmenting the medical video sequence according to the medical video key frames to obtain medical video sequence segments;

[0116] In this embodiment, the medical video sequence is segmented using medical video key frames as segmentation nodes, thereby dividing the medical video sequence into multiple logically coherent segments to obtain medical video sequence segments. Each medical video sequence segment generally contains one or more related medical operations, observations or discussions, and has a key frame as the starting point or end point.

[0117] Step S403: adjusting, enhancing and combining the medical video sequence segments to obtain the standard medical video sequence.

[0118] In this embodiment, the adjustment process includes cutting unnecessary parts (such as repeated shots, meaningless blanks), adjusting the frame rate and resolution to meet the standard format, and correcting possible color or brightness problems; the enhancement process includes improving the video quality through image enhancement technology (such as sharpening, denoising, and contrast adjustment) to make key details more visible, and at the same time, necessary annotations, tags, or timestamps can be added to enhance the comprehensibility and traceability of information; the combination includes recombining the processed video sequence segments into a complete standard medical video sequence in a logical order. Through the above adjustment process, enhancement process, and combination operation, the medical video sequence segments are effectively organized into a standard medical video sequence.

[0119] This embodiment selects medical video key frames in the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features; logically segments the medical video sequence according to the medical video key frames to obtain medical video sequence segments; and adjusts, enhances and combines the medical video sequence segments, thereby effectively obtaining a standard medical video sequence that is edited according to the spatial relationship of the video features and the temporal relationship of the video features, so as to facilitate subsequent potential relationship extraction processing.

[0120] Continue to refer Figure 7 In some optional implementations of this embodiment, step 50 includes the following steps:

[0121] Step S501, obtaining a first model extraction identifier and a second model extraction identifier, and extracting a feature extraction model and a potential state extraction model from a model database according to the first model extraction identifier and the second model extraction identifier;

[0122] In this embodiment, the first model extraction identifier is the unique identification information corresponding to the feature extraction model, and the second model extraction identifier is the unique identification information corresponding to the latent state extraction model. The first model extraction identifier and the second model extraction identifier are used to traverse and query the model database to find the corresponding feature extraction model and latent state extraction model and extract them.

[0123] Step S502, inputting the medical video sequence and the standard medical video sequence into the feature extraction model to obtain a first video sequence feature and a second video sequence feature;

[0124] In this embodiment, the feature extraction model can adopt a recurrent neural network (RNN) model, and the steps of extracting the first video sequence feature and the second video sequence feature through the recurrent neural network (RNN) model specifically include: taking the first video frame sequence and the second video frame sequence as the input of the RNN model respectively, and inputting the frames in each sequence into the RNN in order; learning the dependency between the frames through the RNN model, and gradually constructing the feature representation of the video sequence; selecting the state of the last hidden layer of the RNN model as the feature representation of the entire video sequence to obtain the first video sequence feature and the second video sequence feature. If the dimension of the extracted feature vector is too high, the dimension can be reduced by dimensionality reduction technology (such as PCA, t-SNE, etc.) for subsequent processing and analysis, and the extracted feature vectors are normalized to ensure that they have the same scale.

[0125] Step S503: input the first video sequence feature and the second video sequence feature into the latent state extraction model to obtain the first latent state sequence and the second latent state sequence.

[0126] In this embodiment, the latent state extraction model can adopt a dynamic Bayesian network (DBN) model, and the steps of extracting the first latent state sequence and the second latent state sequence according to the latent state extraction model include: selecting an initial state distribution for the latent state nodes in the DBN, which can be uniform or based on prior knowledge; using the transition probability and observation probability of the DBN, starting from the initial state, gradually calculating the posterior probability distribution of the latent state at each time step. The probability distribution calculation can be performed using the Bayesian formula and the chain rule; according to the posterior probability distribution of the latent state at each time step, selecting the most likely latent state as the latent state of the time step, and arranging these latent states in order to form the first latent state sequence and the second latent state sequence.

[0127] This embodiment obtains a first model extraction identifier and a second model extraction identifier, and extracts a feature extraction model and a latent state extraction model from a model database according to the first model extraction identifier and the second model extraction identifier; inputs the medical video sequence and the standard medical video sequence into the feature extraction model to obtain a first video sequence feature and a second video sequence feature; inputs the first video sequence feature and the second video sequence feature into the latent state extraction model, thereby effectively obtaining a first latent state sequence and a second latent state sequence based on the medical video sequence and the standard medical video sequence, so as to facilitate subsequent fusion processing.

[0128] Continue to refer Figure 8In some optional implementations of this embodiment, step 60 includes the following steps:

[0129] Step 601, time-aligning the first potential state sequence and the second potential state sequence based on dynamic time warping;

[0130] In this embodiment, time alignment is achieved through the following steps: calculating the distance matrix: calculating the distances between all possible state pairs in two potential state sequences (for example, using Euclidean distance) to obtain a state distance matrix; finding the optimal path: using the dynamic time warping (DTW) algorithm to find a path from the upper left corner to the lower right corner in the distance matrix, with the minimum cumulative distance on the path, which represents the best alignment between the two sequences; time alignment: based on the optimal path found, resample or interpolate the two potential state sequences in time so that they are aligned in time.

[0131] Step 602, obtaining a preset fusion strategy, and performing sequence fusion on the time-aligned first latent state sequence and the second latent state sequence according to the preset fusion strategy to obtain a fused state sequence;

[0132] In this embodiment, the preset fusion strategy is in, is the pixel value of the fusion state sequence, ⊙ is the element multiplication, γ∈(0,1), γ represents the weight coefficient, m i,j To fuse the weights, we can collect the cross-attention maps between text features and visual features during the DDIM inversion process, binarize the time-related cross-attention maps by thresholding, and pool them into a global binary map to estimate m. i,j , is the pixel value of the first potential state sequence, is the second potential state sequence pixel value. After the fused state sequence pixel value is calculated by a preset fusion strategy, a corresponding fused state sequence is generated according to the fused state sequence pixel value.

[0133] Step 603: The fusion state sequence should be set to a video content space to generate the valid medical video sequence.

[0134] In this embodiment, each state in the fusion state sequence is mapped to a corresponding element or attribute in the video content space to be converted into features, colors, textures, motion patterns, etc. of the video frame, thereby generating a valid medical video sequence.

[0135] This embodiment performs time alignment on the first latent state sequence and the second latent state sequence based on dynamic time warping; obtains a preset fusion strategy, and performs sequence fusion on the time-aligned first latent state sequence and the second latent state sequence according to the preset fusion strategy to obtain a fused state sequence; and sets the fused state sequence to a video content space, thereby effectively generating a valid medical video sequence based on the fusion of the first latent state sequence and the second latent state sequence.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0137] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0138] Further references Fig. 9 , as a response to the above Figure 1 The present application provides an embodiment of a video generation device, and the device embodiment is Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0139] like Fig. 9 As shown, the video generation device 700 described in this embodiment includes: a sequence acquisition module 701, a sequence conversion module 702, a relationship extraction module 703, a sequence editing module 704, a state extraction module 705, and a state fusion module 706. Among them:

[0140] A sequence acquisition module 701 is used to acquire a medical video sequence;

[0141] A sequence conversion module 702 is used to perform low-dimensional conversion on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector;

[0142] A relationship extraction module 703 is used to extract the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a video feature spatial relationship and a video feature temporal relationship;

[0143] A sequence editing module 704, configured to edit the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence;

[0144] A state extraction module 705 is used to extract potential states from the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence;

[0145] The state fusion module 706 is used to fuse the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence.

[0146] By adopting the above-mentioned device, this embodiment can obtain a medical video sequence; perform low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; extract the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a video feature spatial relationship and a video feature temporal relationship; edit the medical video sequence according to the video feature spatial relationship and the video feature temporal relationship to obtain a standard medical video sequence; perform potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; fuse the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence. Thus, it is effectively realized that a true and highly relevant valid medical video sequence is generated according to the provided medical video sequence to provide effective data support for medical research and operation.

[0147] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0148] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0149] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0150] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions of the video generation method, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0151] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer-readable instructions stored in the memory 81 or process data, such as computer-readable instructions for running the video generation method.

[0152] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0153] By adopting the above-mentioned computer device, this embodiment can obtain a medical video sequence; perform low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; extract the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a video feature spatial relationship and a video feature temporal relationship; edit the medical video sequence according to the video feature spatial relationship and the video feature temporal relationship to obtain a standard medical video sequence; perform potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; fuse the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence. This effectively realizes the generation of a true and highly relevant valid medical video sequence based on the provided medical video sequence, so as to provide effective data support for medical research and operation.

[0154] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the video generation method as described above.

[0155] This embodiment, by using the above-mentioned computer-readable storage medium, can obtain a medical video sequence; perform a low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; extract the spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a video feature spatial relationship and a video feature temporal relationship; edit the medical video sequence according to the video feature spatial relationship and the video feature temporal relationship to obtain a standard medical video sequence; perform potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; fuse the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence. This effectively realizes the generation of a true and highly relevant valid medical video sequence based on the provided medical video sequence, so as to provide effective data support for medical research and operation.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0157] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A video generation method, characterized in that: The steps include: Acquiring medical video sequences; Performing low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; Extracting spatial and spatiotemporal relationships of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window to obtain a spatial relationship of video features and a temporal relationship of video features; Editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence; Performing latent state extraction on the medical video sequence and the standard medical video sequence to obtain a first latent state sequence and a second latent state sequence; The first latent state sequence and the second latent state sequence are fused to obtain a valid medical video sequence.

2. The video generation method according to claim 1, characterized in that: The step of acquiring the medical video sequence specifically includes: Get the video extraction request identifier; Extracting the original medical video from the medical video database according to the video extraction request identifier; The original medical video is converted into the medical video sequence.

3. The video generation method according to claim 1, characterized in that: The step of performing low-dimensional transformation on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector specifically includes: Acquiring video frame time information of the medical video sequence; Extracting the first frame of the medical video sequence as a video static image frame according to the video frame time information; Extracting non-first frames of the medical video sequence as a video image frame set according to the video frame time information; The video static image frame and the video image frame set are input into a preset convolution encoder for compression encoding to obtain the first sequence feature vector and the second sequence feature vector.

4. The video generation method according to claim 1, characterized in that: The step of extracting the spatial relationship and the spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on the spatial window and the spatiotemporal window to obtain the spatial relationship of video features and the temporal relationship of video features specifically includes: Obtain preset spatial window definition rules and spatiotemporal window definition rules; Creating the spatial window and the spatiotemporal window according to the spatial window definition rule and the spatiotemporal window definition rule; Performing spatial relationship analysis on the first sequence feature vectors and the second sequence feature vectors according to the spatial window to obtain the video feature spatial relationship; The spatiotemporal relationship between the first sequence feature vectors and the second sequence feature vectors is analyzed according to the spatiotemporal window to obtain the video feature time relationship.

5. The video generation method according to claim 1, characterized in that: The step of editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence specifically includes: Selecting a medical video key frame in the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features; Logically segmenting the medical video sequence according to the medical video key frames to obtain medical video sequence segments; The medical video sequence segments are adjusted, enhanced and combined to obtain the standard medical video sequence.

6. The video generation method according to claim 1, characterized in that: The step of extracting latent states from the medical video sequence and the standard medical video sequence to obtain a first latent state sequence and a second latent state sequence specifically includes: Obtaining a first model extraction identifier and a second model extraction identifier, and extracting a feature extraction model and a potential state extraction model from a model database according to the first model extraction identifier and the second model extraction identifier; Inputting the medical video sequence and the standard medical video sequence into the feature extraction model to obtain a first video sequence feature and a second video sequence feature; The first video sequence feature and the second video sequence feature are input into the latent state extraction model to obtain the first latent state sequence and the second latent state sequence.

7. The video generation method according to claim 1, characterized in that: The step of fusing the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence specifically includes: Time-aligning the first latent state sequence and the second latent state sequence based on dynamic time warping; Acquire a preset fusion strategy, and perform sequence fusion on the time-aligned first latent state sequence and the second latent state sequence according to the preset fusion strategy to obtain a fused state sequence; The fusion state sequence should be set to a video content space to generate the valid medical video sequence.

8. A video generating device, characterized in that: include: A sequence acquisition module, used to acquire medical video sequences; A sequence conversion module, used for performing low-dimensional conversion on the medical video sequence to obtain a first sequence feature vector and a second sequence feature vector; A relationship extraction module, used for extracting spatial relationship and spatiotemporal relationship of the first sequence feature vector and the second sequence feature vector based on a spatial window and a spatiotemporal window, so as to obtain a spatial relationship of video features and a temporal relationship of video features; A sequence editing module, used for editing the medical video sequence according to the spatial relationship of the video features and the temporal relationship of the video features to obtain a standard medical video sequence; A state extraction module, used for performing potential state extraction on the medical video sequence and the standard medical video sequence to obtain a first potential state sequence and a second potential state sequence; The state fusion module is used to fuse the first potential state sequence and the second potential state sequence to obtain a valid medical video sequence.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are implemented.

Images