Video retrieval method, device and computer equipment

Abstract

The present application relates to a video retrieval method, apparatus and computer equipment. The method performs element detection on each video frame of a video to be retrieved to obtain a target structure set and a target instrument set; based on the target instrument set, operation information of the target instrument is extracted from the video frame; based on the operation information, the video to be retrieved is segmented to obtain an action segmentation set; based on the target structure set and the action segmentation set, a first structured feature library of the video to be retrieved is established; during retrieval, retrieval results are obtained based on retrieval data provided by a user and the first structured feature library; the problem of poor description ability of medical target videos is solved, and the accuracy of video retrieval is improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics and image video processing, and in particular to a video retrieval method, device and computer equipment. Background Art

[0002] Video retrieval is a technology and method that involves locating, retrieving, and obtaining video content of interest to users in large-scale video databases, aiming to help users quickly find the content they need in massive video resources. Video retrieval systems have broad application prospects in many fields, including but not limited to: digital libraries, film and television entertainment, e-commerce, and medical assistance. For example, a surgical video retrieval system can help doctors retrieve surgical videos of similar cases, and learn surgical techniques and operating procedures by analyzing and processing surgical videos. With the advancement of technology, compared with the traditional method of manually labeling videos, there are solutions for retrieval based on video image features, audio features, etc. as video descriptions, which improves the accuracy and efficiency of retrieval.

[0003] In the prior art, some solutions extract multimodal features from videos. In order to compare the similarity between multimodal features and vector features of retrieved text, it is necessary to perform operations such as pooling, alignment, and multimodal fusion on the multimodal features, resulting in the lack of semantic descriptiveness of the extracted multimodal features of surgical videos. Other solutions extract features from each frame of the video stream, extract the text description of each frame through a text sequence model, and construct a mapping relationship based on chapter attributes for each video. According to the text retrieval information sent by the user, the target chapter and target section are retrieved in turn, and finally the target frame image is obtained. Its main purpose is to retrieve relevant frames in the video, and it uses traditional video retrieval methods to extract video text tags and match them with the retrieved text, which loses the image representation information of the surgical video itself.

[0004] In summary, existing technical video retrieval solutions have the problem of poor video description capabilities, which affects the accuracy of video retrieval. Summary of the invention

[0005] Based on this, it is necessary to provide a video retrieval method, device and computer equipment that can enhance video description capabilities and improve retrieval quality in response to the above technical problems.

[0006] In a first aspect, the present application proposes a video retrieval method, the method comprising:

[0007] Performing element detection on each video frame of the video to be retrieved to obtain a target structure set and a target device set; the target structure set is a set of the video frames containing target structure elements, and the target device set is a set of the video frames containing target device elements;

[0008] Extracting operation information of the target device from the video frames of the target device set; segmenting the video to be retrieved based on the operation information to obtain an action segment set;

[0009] Based on the target structure set and the action segment set, establishing a first structured feature library of the video to be retrieved;

[0010] During the search, the search results are obtained based on the search data provided by the user and the first structured feature library.

[0011] In some embodiments, element detection is performed on each video frame of the video to be retrieved to obtain a set of target structures and a set of target devices, including:

[0012] Based on the organ detection model, lesion detection model and device detection model respectively, element detection is performed on each video frame of the retrieved video to obtain an organ set, a lesion set and a target device set.

[0013] In some embodiments, extracting the operation information of the target device from the video frame of the target device set includes:

[0014] Extracting key area videos from video frames of the target device set;

[0015] Extracting spatial movement information, operation category, and operation features of the target device from the key area video;

[0016] Based on the movement information, the operation category and the operation feature, the operation information of each target device in each frame is obtained.

[0017] In some embodiments, the spatial movement information, operation category and operation feature of the target device are extracted from the key area video, including:

[0018] Input the key area video into a deep optical flow model, predict and obtain optical flow vector graphs of the previous and next two frames in the X direction and the Y direction, and obtain the movement amount of the target device in the X direction and the Y direction based on the optical flow vector graph;

[0019] Inputting the key area video into a depth estimation model, predicting a depth value map of each pixel, and obtaining a movement amount of the target device in the Z direction based on the depth value map;

[0020] Based on the movement amounts in the X direction, the Y direction, and the Z direction, obtaining movement information of the target device in space;

[0021] The key area video is divided into video segments based on a fixed frame number and a sliding step size; the video segments are input into a three-dimensional convolution classification network model, and the operation category and operation feature corresponding to each of the video segments are output.

[0022] In some embodiments, segmenting the video to be retrieved based on the operation information to obtain an action segment set includes:

[0023] Based on the operation category in the operation information, construct an operation encoding vector corresponding to each video frame;

[0024] The operation coding vectors are traversed, and when the operation coding vectors of two adjacent frames are not equal, the previous frame is used as a dividing point to segment the video to be retrieved to obtain an action segment set.

[0025] In some embodiments, establishing a first structured feature library of the video to be retrieved based on the target structure set and the action segment set includes:

[0026] Obtaining organ information and lesion information based on the target structure set;

[0027] Obtaining action encoding information, temporal normalization features and action scores based on the action segment set;

[0028] Based on the organ information, lesion information, action coding information, time-normalized features and action scores, a first structured feature library of the video to be retrieved is established.

[0029] In some embodiments, obtaining action encoding information, temporal normalization features and action segmentation scores based on the action segment set includes:

[0030] Based on the operation information, acquiring segment operation information of each of the action segments;

[0031] Obtaining action encoding information based on the operation encoding vector in the segmented operation information;

[0032] Based on the movement information in the segmented operation information, calculating a time-series normalized displacement feature;

[0033] Based on the operation features in the segmented operation information, calculating temporally normalized visual features;

[0034] The time-normalized displacement feature and the time-normalized visual feature are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is input into a fully trained action evaluation model to obtain an action score.

[0035] In some of the embodiments, during the search, based on the search data provided by the user and the structured feature library, the search results are obtained, including:

[0036] When the search data is a search item, a target video is obtained based on a matching result between the search item and the first structured feature library;

[0037] When the retrieval data is a reference video, a second structured feature library of the reference video is established, the second structured feature library is compared with the first structured feature library to obtain a target video matching the reference video, and a structured comparison result is output.

[0038] In a second aspect, the present application further provides a video retrieval device, the device comprising:

[0039] An element detection module is used to perform element detection on each video frame of the video to be retrieved to obtain a target structure set and a target device set; the target structure set is a set of the video frames containing target structure elements, and the target device set is a set of the video frames containing target device elements;

[0040] An action segmentation module is used to extract operation information of the target device from the video frame of the target device set; segment the video to be retrieved based on the operation information to obtain an action segmentation set;

[0041] A feature library construction module is used to establish a first structured feature library of the video to be retrieved based on the target structure set and the action segment set;

[0042] The retrieval module is used to obtain retrieval results based on the retrieval data provided by the user and the first structured feature library during retrieval.

[0043] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video retrieval method described in the first aspect is implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video retrieval method described in the first aspect is implemented.

[0045] The above-mentioned video retrieval method, device and computer equipment obtain a target structure set and a target instrument set by performing element detection on each video frame of the video to be retrieved; the target structure set is a set of the video frames containing target structure elements, and the target instrument set is a set of the video frames containing target instrument elements; the operation information of the target instrument is extracted from the video frames of the target instrument set; the video to be retrieved is segmented based on the operation information to obtain an action segmentation set; based on the target structure set and the action segmentation set, a first structured feature library of the video to be retrieved is established; when retrieving, a retrieval result is obtained based on the retrieval data provided by the user and the first structured feature library; the problem of poor description ability of medical target videos is solved, and the accuracy of video retrieval is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 is a hardware structure block diagram of a video retrieval method in one embodiment;

[0048] Figure 2 A schematic diagram of a video retrieval method in one embodiment;

[0049] Figure 3 A schematic diagram of a process for extracting operation information in one embodiment;

[0050] Figure 4 A schematic diagram of a flow chart of a video retrieval method in a preferred embodiment;

[0051] Figure 5 FIG. 4 is a structural block diagram of a video retrieval device in one embodiment.

[0052] Figure numerals: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 51, element detection module; 52, action segmentation module; 53, feature library construction module; 54, retrieval module. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0054] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as the content disclosed in this application is insufficient.

[0055] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0056] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0057] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1FIG. 4 is a hardware structure diagram of the video retrieval method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0058] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the video retrieval method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0059] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0060] In this embodiment, a video retrieval method is provided. Figure 2 is a flow chart of the video retrieval method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0061] Step S210, perform element detection on each video frame of the video to be retrieved to obtain a target structure set and a target device set. The target structure set is a set of video frames containing target structure elements, and the target device set is a set of video frames containing target device elements.

[0062] Specifically, this embodiment can be applied in the fields of education, security monitoring, and e-commerce. Preferably, this embodiment is applied in the field of education, and the video to be retrieved is surgical teaching video data in a medical scenario, and the efficiency of teaching resource utilization is improved by optimizing the video retrieval method. Among them, the video to be retrieved includes but is not limited to surgical video data that meets the above requirements obtained from a database pre-stored; it can also be surgical video data that meets the requirements downloaded from a network platform; it can also be corresponding surgical video data generated according to demand, etc. The embodiment of this application does not limit the method of obtaining surgical video data.

[0063] Among them, feature detection mainly focuses on identifying, analyzing, judging and other processing of the elements involved in the video. The feature categories can be pre-set, such as the target structural elements of organs and lesions involved in the video, and medical device elements.

[0064] Step S220, extracting operation information of the target device from the video frames of the target device set; segmenting the video to be retrieved based on the operation information to obtain an action segment set.

[0065] Specifically, the target device is a dynamic element in the video, and the video can be motion analyzed based on the target device to obtain richer content information.

[0066] Step S230: establishing a first structured feature library of the video to be retrieved based on the target structure set and the action segment set.

[0067] Specifically, a corresponding first structured feature library is constructed for each video to be retrieved, and all first structured feature libraries constitute a retrieval library, so that retrieval is performed in the retrieval library.

[0068] Step S240, during retrieval, obtaining retrieval results based on the retrieval data provided by the user and the first structured feature library.

[0069] Specifically, the search data provided by the user can be text data or video data. That is, this embodiment supports searching for videos by text, and also supports searching for videos by video. Among them, the reference video provided by the user includes but is not limited to video data obtained from pre-stored in a database; it can also be video data that meets the requirements downloaded from a network platform; it can also be corresponding video data generated according to the requirements. The embodiment of this application does not limit the method of obtaining the reference video.

[0070] Specifically, the search results include the searched target video and structured prompt information, wherein the structured prompt information includes but is not limited to the element types in the video, the overlap ratio of the types of each element, the overlap ratio of the segmented actions, the scores of the same segmented actions, the similarity of the segmented actions, etc. After obtaining the search results, the information can be further processed or analyzed according to the search results, and applied in different scenarios according to the processing results or analysis results.

[0071] In this embodiment, by performing element detection on each video frame of the video to be retrieved, a target structure set and a target instrument set are obtained; based on the target instrument set, operation information of the target instrument is extracted from the video frame; based on the operation information, the video to be retrieved is segmented to obtain an action segmentation set; based on the target structure set and the action segmentation set, a first structured feature library of the video to be retrieved is established; when searching, a search result is obtained based on the search data provided by the user and the first structured feature library; this embodiment extracts structured video element information from the video to be retrieved by action segmentation, which is more semantically descriptive, solves the problem of poor description ability of medical target videos, and improves the accuracy of video retrieval.

[0072] In some embodiments, element detection is performed on each video frame of the video to be retrieved to obtain a set of target structures and a set of target devices, including:

[0073] Step S211 , based on the organ detection model, the lesion detection model and the instrument detection model, each video frame of the video to be retrieved is subjected to element detection to obtain an organ set, a lesion set and a target instrument set.

[0074] Specifically, the organ detection model is used to identify organs in the video to be retrieved, and the organ set includes video frames containing organs. The lesion detection model is used to identify lesions in the video to be retrieved, and the lesion set includes video frames containing lesions. The instrument detection model is used to identify operating instruments in the video to be retrieved, and the target instrument set includes a set of video frames containing operating instruments.

[0075] In some of the embodiments, Figure 3 As shown, extracting the operation information of the target device from the video frame of the target device set includes the following steps:

[0076] Step S310: extract key area videos from video frames of the target device set.

[0077] Specifically, after detecting the target instrument, the operation vertex of the target instrument (such as the center point of the forceps head and the center point of the surgical knife head) is detected using a pre-trained key point detection network to obtain the instrument key point. For each frame of the image, the instrument key point is taken as the center, the length and width are preset, and the instrument operation key area map is obtained by cropping to form the operation key area video of each instrument.

[0078] Step S320: extracting the spatial movement information, operation category and operation features of the target device from the key area video.

[0079] Step S330 , obtaining operation information of each target device in each frame based on the movement information, the operation category and the operation feature.

[0080] In this embodiment, the key area video helps to focus on the content of the target device, reduce the interference of background or other irrelevant elements on the processing results, and avoid comprehensive processing of the entire video screen, significantly reducing the amount of processed data and improving processing efficiency. In addition, this embodiment extracts movement information, operation categories and operation features, and strengthens the description performance of operation information.

[0081] In some embodiments, the spatial movement information, operation category and operation features of the target device are extracted from the key area video, including:

[0082] Step S321, input the key area video into the deep optical flow model, predict the optical flow vector graphs of the previous and next two frames in the X direction and the Y direction, and obtain the movement amount of the target device in the X direction and the Y direction based on the optical flow vector graph.

[0083] Step S322: Input the key area video into the depth estimation model, predict the depth value map of each pixel, and obtain the movement amount of the target device in the Z direction based on the depth value map.

[0084] Step S323, based on the movement amounts in the X direction, the Y direction, and the Z direction, obtain the movement information of the target device in space.

[0085] Step S324: divide the key area video into video segments based on a fixed frame number and a sliding step size; input the video segments into a three-dimensional convolution classification network model, and output the operation category and operation feature corresponding to each video segment.

[0086] Specifically, the 3D convolution classification network model is a pre-trained model. After inputting the above video segment, the 3D convolution classification network model can identify the instrument operation o (moving, cutting, shearing, clamping, lifting, suturing, etc.) performed in each segment, and output the last fully connected layer of the network as the operation feature f of the image frame in the current segment. o .

[0087] In this embodiment, the state of the target device is analyzed from different dimensions through a deep optical flow model, a depth estimation model, and a three-dimensional convolution classification network model.

[0088] In some embodiments, the video to be retrieved is segmented based on the operation information to obtain an action segment set, including:

[0089] Step S340: construct an operation coding vector corresponding to each video frame based on the operation category in the operation information.

[0090] Step S350, traverse the operation coding vectors, when the operation coding vectors of two adjacent frames are not equal, use the previous frame as a dividing point to segment the video to be retrieved to obtain an action segment set.

[0091] Specifically, obtain the operation category o of each device in each frame m,i , the number of surgical instruments is preset to M, then each frame image i can obtain a 1×M instrument operation encoding vector O i =[o 1,i , o 2,i , o 2,i ,…,o M,i ], where the operation of the instrument that is not detected is marked as -1. Traverse the instrument operation vector O of each frame i , if O i With O i+1 If they are not equal, the i frame is used as the dividing point to segment the entire video to be retrieved and obtain the action segmentation set S.

[0092] In this embodiment, by operating the coding vector, the long video is segmented quickly and accurately, so that each action segment focuses on one operation action.

[0093] In some embodiments, a first structured feature library of a video to be retrieved is established based on a target structure set and an action segment set, including:

[0094] Step S231, obtaining organ information and lesion information based on the target structure set.

[0095] Step S232, obtaining action coding information, temporal normalization features and action scores based on the action segment set.

[0096] Specifically, because the operation information of each target device in each frame is obtained in the above step S330, the operation information can be reintegrated based on the target device in the action segment to obtain the segmented operation information (action encoding information, temporal normalization features and action scores) corresponding to each action segment.

[0097] Step S233, based on the organ information, lesion information, action coding information, time-normalized features and action scores, a first structured feature library of the video to be retrieved is established.

[0098] In this embodiment, feature archiving is performed for each video to be retrieved, and the first structured feature library contains organ information, lesion information, and action segment set S. For each action segment in set S, it contains action coding information of each segment, time-normalized features, and action scores of the action segment. The first structured feature library contains structured video element information and time-series feature information, which further improves the semantic descriptiveness of the feature library.

[0099] In some embodiments, obtaining action encoding information, temporal normalization features and action segmentation scores based on the action segmentation set includes:

[0100] Step S410: acquiring segmented operation information of each action segment based on the operation information.

[0101] Step S420, obtaining action coding information based on the operation coding vector in the segmented operation information.

[0102] Step S430, calculating the time-normalized displacement feature based on the movement information in the segmented operation information.

[0103] Assume that in the action segment set S, the segment starts from frame t and ends at frame t+n-1. The operation information of the instrument m in the segment [t, t+n-1] is normalized in a unified time period to obtain the time-series related displacement features and visual features.

[0104] Specifically, the frame length corresponding to the unified time of the segment is set to N. The displacements △x, △y and △z in the X, Y and Z directions are processed and normalized respectively to obtain the displacement quantization information of the action segment in the three directions, that is, the time-series normalized displacement feature, whose dimension is N×3×1.

[0105] Taking the displacement △x in the X direction as an example, its displacement in the segment [t, t+n-1] constitutes an n-dimensional △x vector:

[0106] ;

[0107] Interpolate the vector to the set frame length N to get the original vector; calculate the interval of frames in the original vector: dt=n / N; for the jth frame in the original vector, calculate its corresponding position j×dt in the original vector according to the interval dt, where 0≤j<N; take the two integer position indexes index1 and index2 closest to j×dt, average them, and use the result as the displacement vector of the jth frame in the X direction:

[0108] ;

[0109] Finally, we get the N-dimensional displacement vector in the X direction:

[0110] .

[0111] The same processing is performed on the Y direction and the Z direction. After processing, a time-normalized displacement feature with a dimension of N×3×1 is obtained based on the displacement vectors in the three directions.

[0112] Step S440: Calculate temporally normalized visual features based on the operation features in the segmented operation information.

[0113] Specifically, the frame length corresponding to the segmented uniform time is set to N. For each operation’s visual feature f o , whose feature vector dimension is 4096, and its visual features on the segment [t, t+n-1] constitute a n×4096 vector:

[0114] ;

[0115] Interpolate the vector to the set frame length N to get the original vector; calculate the interval of frames in the original vector: dt=n / N; for the jth frame in the original vector, calculate its corresponding position j×dt in the original vector according to the interval dt, where 0≤j<N; take the two integer position indexes index1 and index2 closest to j×dt, perform maximum pooling on them, and use the result as the visual feature vector of the jth frame:

[0116] ;

[0117] Finally, an N×4096 visual feature vector is obtained.

[0118] Step S450, splicing the temporally normalized displacement feature and the temporally normalized visual feature to obtain a spliced ​​feature, and inputting the spliced ​​feature into a fully trained action evaluation model to obtain an action score.

[0119] In some of the embodiments, during the search, based on the search data provided by the user and the structured feature library, the search results are obtained, including:

[0120] Step S241, when the search data is a search item, a target video is obtained based on a matching result between the search item and the first structured feature library.

[0121] Specifically, relevant videos are retrieved based on diversified fuzzy information such as the element name in the search entry (such as the organ or lesion name), the name of the action performed (action score, operation sequence of the action), the name of the target instrument (specific operation of the instrument), etc.

[0122] Step S242, when the retrieved data is a reference video, a second structured feature library of the reference video is established, the second structured feature library is compared with the first structured feature library to obtain a target video matching the reference video, and a structured comparison result is output.

[0123] Specifically, the reference video uploaded by the user is subjected to the same feature processing process as the video to be retrieved to extract structured feature information. Specifically, each video frame of the reference video is subjected to feature detection to obtain a target structure set and a target instrument set; based on the target instrument set, the operation information of the target instrument is extracted from the video frame; based on the operation information, the video to be retrieved is segmented to obtain an action segmentation set; based on the target structure set and the action segmentation set, a second structured feature library of the reference video is established. When retrieving videos through videos, the features of interest are selected as the retrieval criteria, and finally the videos that are closer to the retrieval criteria are displayed, and the structured comparison results are displayed as structured prompt information.

[0124] Taking surgical videos as an example, the structured comparison results include: 1. Types of organs, lesions, and instruments in the video; 2. Overlap ratio of organ, lesion, and instrument types between the two videos: the number of elements (organs, lesions, instruments) in the reference video that appear in the elements (organs, lesions, instruments) in the retrieval library video, and the ratio of the number of elements (organs, lesions, instruments) in the reference video; 3. Overlap ratio of segmented action sets: the number of actions in the reference video that appear in the action set of the retrieval library video, and the ratio of the number of actions in the reference video; 4. Comparison of scores for the same segmented actions: For the overlapping segmented actions in the reference video and the video library, the action scores are compared respectively, and users can filter and display the clips with higher scores; 5. Similarity of segmented actions: For the overlapping segmented actions in the uploaded reference video and the video library, the cosine similarity is calculated for each of them through N×(3+4096) displacement and visual features.

[0125] Figure 4 FIG. 4 is a flow chart of the video retrieval method of the preferred embodiment. Figure 4 As shown, the process includes the following steps:

[0126] Step 1: Detect surgical elements in all video frames of the retrieved video, including organs, lesions, and surgical instruments. Use the trained organ detection model, lesion detection model, and instrument detection model to detect organs, lesions, and surgical instruments respectively, and obtain the organ set, lesion set, and surgical instrument set contained in the current frame. After detecting the surgical instruments, use the pre-trained key point detection network to detect the key points of the operating vertices of the surgical instruments (such as the center point of the forceps head and the center point of the scalpel head).

[0127] Step 2: Extract the operation information of each instrument in each frame, including position movement information and instrument operation classification. For each frame of the image, the instrument key point is centered, the length and width are preset, and the instrument operation key area map is obtained by cropping to form the operation key area video of each instrument. The deep optical flow model is used to predict the optical flow vector map on x and y between the previous and next frames, where the midpoint movement is the movement of the key point in the x and y directions (△x, △y). The depth estimation model is used to predict the depth value map of each pixel. The depth value difference between the midpoints of the previous and next frames is the movement of the key point in the z direction △z, and the movement information of the key point in space is obtained (△x, △y, △z). Each instrument key area video is segmented with a fixed number of frames and sliding steps, and the pre-trained 3D convolution classification network model is input to identify the operation category o (moving, cutting, shearing, clamping, lifting, suturing, etc.) performed in each segment, and the last fully connected layer output of the network is used as the operation feature f of the image frame in the current segment. o The operation information I of the mth instrument in the i-th frame m,i It can be expressed as:

[0128] .

[0129] Step 3: According to the operation information of the entire instrument set, the video to be retrieved is segmented into actions. Step 2 obtains the operation category of each instrument in each frame. The surgical instruments are preset to be M types, so each frame image i can get a 1×M instrument operation encoding vector. If the instrument is not detected, the operation mark is -1. Traverse the instrument operation vector of each frame. If O i With O i+1 If they are not equal, the i frame is used as the dividing point to segment the entire video to be retrieved and obtain the action segmentation set S.

[0130] Step 4: Integrate the feature information of each action segment to obtain the time-normalized displacement feature and time-normalized visual feature of each action segment. The action segment set S obtained in step 3 starts from frame t and ends at frame t+n-1. The operation information of the instrument m in the segment [t, t+n-1] is normalized in a unified time period to obtain the time-related displacement feature and visual feature.

[0131] Step 5: Concatenate the time-normalized displacement features and time-normalized visual features obtained in step 4 to obtain N×(3+4096) features, input the features into the trained action evaluation model, and score each encoded surgical action.

[0132] Step 6: Through the above operations, a feature file is created for each video to be detected. The feature library contains the detected organs, lesion information and action segment set S. For each action segment in the set, it contains the action coding information of each segment, the time-normalized features and the score of the action segment. The action coding information contains the current operation status of each instrument. The operation status of the instrument that does not exist in the picture is -1. Action coding information O s Expressed as: s =[o 1,s , o 2,s , o 2,s ,…,o M ,s ];

[0133] The time-normalized features include: time-normalized displacement features and time-normalized visual features, wherein the time-normalized displacement features contain the displacement quantification information of each instrument in three directions, and its dimension is N×3×1; the time-normalized visual features contain the visual features of each instrument, and its dimension is N×4096.

[0134] Step 7: In the retrieval stage, it includes operations such as text retrieval video and video retrieval video. In the text entry retrieval operation, the user retrieves the relevant video based on the organ name or lesion name, the name of the action performed (action score, action operation sequence), the instrument name (specific operation category of the instrument), and other diversified fuzzy information. In the video retrieval video operation, the structured feature information of the video uploaded by the user is extracted through steps 1 to 5. And a structured comparison is performed with the feature library of the video to be retrieved. The user can select the feature of interest as the retrieval criteria. Finally, the video with the closest search criteria is displayed, and the similarity information is structured. The structured information is as follows: 1. Types of organs, lesions, and instruments in the video; 2. Overlap ratio of organ, lesion, and instrument types between the two videos: the number of elements (organs, lesions, and instruments) in the reference video that appear in the elements (organs, lesions, and instruments) in the search library video, and the ratio of the number of elements (organs, lesions, and instruments) in the reference video; 3. Overlap ratio of segmented action sets: the number of actions in the reference video that appear in the action set of the search library video, and the ratio of the number of actions in the reference video; 4. Comparison of scores for the same segmented actions: For the overlapping segmented actions in the reference video and the video library, the action scores are compared respectively, and users can filter and display the clips with higher scores; 5. Similarity of segmented actions: For the overlapping segmented actions in the uploaded reference video and the video library, the cosine similarity is calculated by N×(3+4096) displacement and visual features.

[0135] In this preferred embodiment, surgical action segmentation based on instrument status is performed, the action displacement characteristics and visual characteristics of surgical elements in timing are extracted, and structured surgical video retrieval is performed in combination with the timing visual characteristics, which improves the description ability of medical surgical videos, enables structured display of similar information, and supports using video instead of text as retrieval information.

[0136] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, a video retrieval device is also provided in this embodiment, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0138] In one embodiment, if Figure 5 As shown, a video retrieval device is provided, which includes: an element detection module 51, an action segmentation module 52, a feature library construction module 53, and a retrieval module 54.

[0139] The element detection module 51 is used to perform element detection on each video frame of the video to be retrieved to obtain a target structure set and a target device set; the target structure set is a set of video frames containing target structure elements, and the target device set is a set of video frames containing target device elements.

[0140] The action segmentation module 52 is used to extract the operation information of the target instrument from the video frames of the target instrument set; segment the video to be retrieved based on the operation information to obtain an action segmentation set.

[0141] The feature library building module 53 is used to build a first structured feature library of the video to be retrieved based on the target structure set and the action segment set.

[0142] The retrieval module 54 is used to obtain retrieval results based on the retrieval data provided by the user and the first structured feature library during retrieval.

[0143] In some of the embodiments, the element detection module 51 is further used to perform element detection on each video frame of the retrieved video based on the organ detection model, the lesion detection model and the instrument detection model, so as to obtain an organ set, a lesion set and a target instrument set.

[0144] In some of the embodiments, the action segmentation module 52 is also used to extract key area videos from the video frames of the target device set; extract the spatial movement information, operation category and operation characteristics of the target device from the key area videos; and obtain the operation information of each target device in each frame based on the movement information, operation category and operation characteristics.

[0145] In some of the embodiments, the action segmentation module 52 is further used to input the key area video into the deep optical flow model, predict the optical flow vector map of the previous and next frames in the X direction and the Y direction, and obtain the movement amount of the target device in the X direction and the Y direction based on the optical flow vector map; input the key area video into the depth estimation model, predict the depth value map of each pixel, and obtain the movement amount of the target device in the Z direction based on the depth value map; obtain the movement information of the target device in space based on the movement amount in the X direction, the Y direction, and the Z direction; divide the key area video into video segments based on a fixed frame number and a sliding step size; input the video segments into the three-dimensional convolution classification network model, and output the operation category and operation feature corresponding to each video segment.

[0146] In some of the embodiments, the action segmentation module 52 is further used to construct an operation coding vector corresponding to each video frame based on the operation category in the operation information; traverse the operation coding vectors, and when the operation coding vectors of two adjacent frames are not equal, use the previous frame as a dividing point to segment the video to be retrieved, and obtain an action segmentation set.

[0147] In some of the embodiments, the feature library module 53 is constructed and is also used to obtain organ information and lesion information based on a target structure set; obtain action coding information, timing normalized features and action scores based on an action segmentation set; and establish a first structured feature library of the video to be retrieved based on the organ information, lesion information, action coding information, timing normalized features and action scores.

[0148] In some of the embodiments, the feature library module 53 is constructed, and is also used to obtain segmented operation information of each action segment based on the operation information; obtain action coding information based on the operation coding vector in the segmented operation information; calculate the time-normalized displacement feature based on the movement information in the segmented operation information; calculate the time-normalized visual feature based on the operation feature in the segmented operation information; splice the time-normalized displacement feature and the time-normalized visual feature to obtain the spliced ​​feature, and input the spliced ​​feature into a fully trained action evaluation model to obtain an action score.

[0149] In some of the embodiments, the retrieval module 54 is also used to obtain a target retrieval video based on the matching result between the retrieval item and the first structured feature library when the retrieval data is a retrieval item; when the retrieval data is a reference video, establish a second structured feature library of the reference video, compare the second structured feature library with the first structured feature library, obtain a target video that matches the reference video, and output a structured comparison result.

[0150] Each module in the above video retrieval device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the video retrieval method in any of the above embodiments are implemented.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video retrieval method in any of the above embodiments are implemented.

[0153] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the video retrieval method in any of the above embodiments when executed by a processor.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0156] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video retrieval method, characterized in that: The method comprises: Performing element detection on each video frame of the video to be retrieved to obtain a target structure set and a target device set; the target structure set is a set of the video frames containing target structure elements, and the target device set is a set of the video frames containing target device elements; Extracting operation information of the target device from the video frames of the target device set; segmenting the video to be retrieved based on the operation information to obtain an action segment set; Based on the target structure set and the action segment set, establishing a first structured feature library of the video to be retrieved; During the search, based on the search data provided by the user and the first structured feature library, a search result is obtained; Wherein, based on the target structure set and the action segment set, establishing a first structured feature library of the video to be retrieved includes: Obtaining organ information and lesion information based on the target structure set; Obtaining action encoding information, temporal normalization features and action scores based on the action segment set; Based on the organ information, lesion information, action coding information, time-normalized features and action scores, a first structured feature library of the video to be retrieved is established.

2. The video retrieval method according to claim 1, characterized in that: Perform element detection on each video frame of the video to be retrieved to obtain a set of target structures and a set of target devices, including: Based on the organ detection model, lesion detection model and device detection model respectively, element detection is performed on each video frame of the retrieved video to obtain an organ set, a lesion set and a target device set.

3. The video retrieval method according to claim 1, characterized in that: Extracting operation information of the target device from the video frame of the target device set includes: Extracting key area videos from video frames of the target device set; Extracting spatial movement information, operation category, and operation features of the target device from the key area video; Based on the movement information, the operation category and the operation feature, the operation information of each target device in each frame is obtained.

4. The video retrieval method according to claim 3, characterized in that: Extracting the spatial movement information, operation category and operation features of the target device from the key area video respectively, including: Input the key area video into a deep optical flow model, predict and obtain optical flow vector graphs of the previous and next two frames in the X direction and the Y direction, and obtain the movement amount of the target device in the X direction and the Y direction based on the optical flow vector graph; Inputting the key area video into a depth estimation model, predicting a depth value map of each pixel, and obtaining a movement amount of the target device in the Z direction based on the depth value map; Based on the movement amounts in the X direction, the Y direction, and the Z direction, obtaining movement information of the target device in space; The key area video is divided into video segments based on a fixed frame number and a sliding step size; the video segments are input into a three-dimensional convolution classification network model, and the operation category and operation feature corresponding to each of the video segments are output.

5. The video retrieval method according to claim 3, characterized in that: Segmenting the video to be retrieved based on the operation information to obtain an action segment set, including: Based on the operation category in the operation information, construct an operation encoding vector corresponding to each video frame; The operation coding vectors are traversed, and when the operation coding vectors of two adjacent frames are not equal, the previous frame is used as a dividing point to segment the video to be retrieved to obtain an action segment set.

6. The video retrieval method according to claim 1, characterized in that: Based on the action segment set, action coding information, temporal normalization features and action segment scores are obtained, including: Based on the operation information, acquiring segment operation information of each of the action segments; Obtaining action encoding information based on the operation encoding vector in the segmented operation information; Based on the movement information in the segmented operation information, calculating a time-series normalized displacement feature; Based on the operation features in the segmented operation information, calculating temporally normalized visual features; The time-normalized displacement feature and the time-normalized visual feature are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is input into a fully trained action evaluation model to obtain an action score.

7. The video retrieval method according to claim 1, characterized in that: During the search, based on the search data provided by the user and the structured feature library, the search results are obtained, including: When the search data is a search item, a target video is obtained based on a matching result between the search item and the first structured feature library; When the retrieval data is a reference video, a second structured feature library of the reference video is established, the second structured feature library is compared with the first structured feature library to obtain the target video matching the reference video, and a structured comparison result is output.

8. A video retrieval device, characterized in that: The device comprises: An element detection module is used to perform element detection on each video frame of the video to be retrieved to obtain a target structure set and a target device set; the target structure set is a set of the video frames containing target structure elements, and the target device set is a set of the video frames containing target device elements; An action segmentation module is used to extract operation information of the target device from the video frame of the target device set; segment the video to be retrieved based on the operation information to obtain an action segmentation set; A feature library module is constructed, which is used to establish a first structured feature library of the video to be retrieved based on the target structure set and the action segment set; wherein, the first structured feature library of the video to be retrieved is established based on the target structure set and the action segment set, including: obtaining organ information and lesion information based on the target structure set; obtaining action coding information, time-normalized features and action scores based on the action segment set; establishing the first structured feature library of the video to be retrieved based on the organ information, lesion information, action coding information, time-normalized features and action scores; The retrieval module is used to obtain retrieval results based on the retrieval data provided by the user and the first structured feature library during retrieval.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images