Medical video retrieval method and device, equipment, storage medium and program product

Abstract

The invention relates to a medical video retrieval method and device, equipment, a storage medium and a program product, and the method comprises the steps: detecting a to-be-retrieved object, and determining target retrieval information; the target retrieval information comprises a name of a target object in the to-be-retrieved object or a feature of the target object; determining target video information corresponding to the target retrieval information according to the target retrieval information and a tag library; a corresponding relation between the retrieval information and the video information is stored in the label library. The medical video retrieval method provided by the invention has relatively high retrieval efficiency.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a medical video retrieval method, apparatus, device, storage medium and program product. Background Art

[0002] With the development of medical technology, the number of surgical videos in hospitals has increased exponentially. Hospitals often lack effective management of a large number of surgical videos, which requires medical staff to spend a lot of energy to find surgical videos after surgery. In addition, a surgical video is usually more than an hour long and contains a lot of redundant information. Medical staff need a lot of time to locate the required information.

[0003] In traditional technology, the required surgical videos are usually retrieved manually. However, when faced with massive video retrieval tasks, this retrieval method is extremely inefficient. Summary of the invention

[0004] Based on this, it is necessary to provide a medical video retrieval method, device, equipment, storage medium and program product that can improve the efficiency of video retrieval in response to the above technical problems.

[0005] In a first aspect, the present application provides a medical video retrieval method, the method comprising:

[0006] Detect the object to be searched and determine target search information; the target search information includes the name of the target object in the object to be searched or the characteristics of the target object;

[0007] According to the target retrieval information and the tag library, the target video information corresponding to the target retrieval information is determined; wherein the tag library stores the corresponding relationship between the retrieval information and the video information.

[0008] In one embodiment, detecting the object to be searched and determining target search information includes:

[0009] Detecting the target object in the object to be retrieved and determining the detection result;

[0010] Determine the target retrieval information based on the detection results.

[0011] In one embodiment, the detection result includes a detection name and a confidence level of the target object, and determining the target retrieval information according to the detection result includes:

[0012] If the confidence is greater than a preset confidence threshold, the detected name of the target object is used as the target retrieval information;

[0013] If the confidence is not greater than a preset confidence threshold, the features of the target object are extracted and used as target retrieval information.

[0014] In one embodiment, determining target video information corresponding to the target search information according to the target search information and the tag library includes:

[0015] Calculate the similarity between the target retrieval information and the retrieval information in the tag library;

[0016] The video information corresponding to the search information whose similarity is greater than a preset similarity threshold is determined as the target video information.

[0017] In one embodiment, detecting a target object in the object to be searched and determining a detection result includes:

[0018] The object to be retrieved is input into a pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

[0019] In one of the embodiments, the target video information includes a target video corresponding to the target object and a position of the target object in the target video.

[0020] In one embodiment, the method for obtaining the corresponding relationship between the search information and the video information includes:

[0021] Traverse the videos to be processed in the video library, determine the retrieval information and video information corresponding to the videos to be processed, and build a corresponding relationship between the retrieval information and the video information; the corresponding relationship between the retrieval information and the video information includes the corresponding relationship between the name and the video information, and the corresponding relationship between the feature and the video information.

[0022] In one embodiment, determining the search information corresponding to the video to be processed includes:

[0023] Extracting each video frame in the video to be processed, and determining the search information corresponding to the object to be processed in each video frame;

[0024] The search information corresponding to the object to be processed in each video frame is determined as the search information corresponding to the video to be processed.

[0025] In one embodiment, if the search information is a feature of the target object, determining the search information corresponding to the video to be processed includes:

[0026] Divide the video to be processed into multiple video segments, and determine the features corresponding to each video segment;

[0027] The features corresponding to each video segment are determined as the features corresponding to the video to be processed.

[0028] In one embodiment, determining the features corresponding to each video segment includes:

[0029] For each video segment, extract each key frame of the video segment and determine the initial features of the object to be processed in each key frame;

[0030] The average value of the initial features of the object to be processed in all key frames in the video segment is calculated to obtain the features corresponding to the video segment.

[0031] In a second aspect, the present application also provides a medical video retrieval device, the device comprising:

[0032] A detection module is used to detect the object to be searched and determine target search information; the target search information includes the name information of the target object in the object to be searched or the characteristics of the target object;

[0033] The determination module is used to determine the target video information corresponding to the target search information according to the target search information and the tag library; wherein the tag library stores the corresponding relationship between the search information and the video information.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method provided in the first aspect are implemented.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect above.

[0036] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method provided in the first aspect when the computer program is executed by a processor.

[0037] The above-mentioned medical video retrieval method, device, equipment, storage medium and program product determine the target retrieval information by detecting the object to be retrieved; the target retrieval information includes the name of the target object in the object to be retrieved or the characteristics of the target object; according to the target retrieval information and the tag library, the target video information corresponding to the target retrieval information is determined. In this embodiment, the target retrieval information is determined by detecting the object to be retrieved; the corresponding relationship between the retrieval information and the video information is stored in the tag library, and by searching the tag library for the same retrieval information as the target retrieval information, the video information corresponding to the found retrieval information is determined as the target video information corresponding to the target retrieval information, that is, the target video information corresponding to the target object. In this way, by performing a series of analysis and processing on the object to be retrieved, the target video information corresponding to the target object in the object to be retrieved can be obtained without manual retrieval, which can improve the retrieval efficiency and reduce the consumption of manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a process flow of a medical video retrieval method in one embodiment;

[0039] Figure 2 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0040] Figure 3 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0041] Figure 4 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0042] Figure 5 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0043] Figure 6 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0044] Figure 7 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0045] Figure 8 A schematic diagram of a process flow of a medical video retrieval method in another embodiment;

[0046] Fig. 9 A schematic diagram of the structure of a medical video retrieval device in one embodiment;

[0047] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Before specifically introducing the technical solution of the embodiment of the present application, the technical background and technical evolution context based on the embodiment of the present application are introduced. With the development of medical technology, the number of surgical videos in hospitals has increased exponentially. Hospitals often lack effective management of a large number of surgical videos, which requires medical staff to spend a lot of energy when searching for surgical videos after surgery. In addition, a surgical video is usually more than one hour long, which contains a lot of redundant information, and medical staff need a lot of time to locate the required information. For example, a video including the use process of a surgical instrument and the specific location of the surgical instrument in the found video is retrieved from a large number of surgical videos. At present, the more mature video retrieval technology on natural scenes has corresponding retrieval technologies as support for different input situations, such as text-video or image-video retrieval. However, in the medical field, for the retrieval of surgical videos, the time of a single surgical video is long, and there are many types of surgical instruments, and the scenes in the video are similar, and the retrieval difficulty is greatly improved compared to natural scenes. In traditional technology, for surgical videos, the required surgical videos are usually retrieved manually. However, when facing a large number of videos for retrieval, the efficiency of this retrieval method is low. In this regard, the present application provides a medical video retrieval method.

[0050] The following is an introduction to the technical solutions involved in the embodiments of the present application in combination with the scenarios in which the embodiments of the present application are applied.

[0051] The medical video retrieval method provided in the present application can be applied to computer devices, and the computer devices may include but are not limited to industrial computers, laptops, and tablet computers.

[0052] In one embodiment, Figure 1 As shown, a medical video retrieval method is provided. This embodiment is described by applying the method to a computer device. In this embodiment, the method may include the following steps:

[0053] Step 100: Detect the object to be searched and determine target search information; the target search information includes the name of the target object in the object to be searched or the characteristics of the target object;

[0054] The object to be retrieved may be a picture including the target object, or a video including the target object. The target object may be a surgical instrument. This embodiment does not limit the method for obtaining the object to be retrieved, as long as its function can be achieved. The target retrieval information refers to the information used when retrieving the target video. The target retrieval information may include the name of the target object; the target retrieval information may also include the characteristics of the target object, such as the appearance and shape of the target object.

[0055] If the object to be retrieved is a picture, the object to be retrieved can be a picture cropped by the user from the video currently being watched, that is, a screenshot of the current video screen; it can also be a region of interest customized (outlined) by the user from the picture displayed on the current interface, and the region of interest includes the target object; it can also be a picture of the target object. After obtaining the object to be retrieved, the computer device detects the object to be retrieved, that is, detects the pictures including the target object, and determines the name of the target object in the object to be retrieved or the characteristics of the target object. If the object to be retrieved is a video, the object to be retrieved can be a video stored in a video database. After obtaining the object to be retrieved, the computer device detects the object to be retrieved, that is, detects the video including the target object, and determines the name of the target object in the object to be retrieved or the characteristics of the target object.

[0056] In an optional embodiment, when the object to be retrieved is a video, the computer device may first extract a video frame from the video, detect the extracted video frame, and determine the name of the target object or the characteristics of the target object in the video frame when detecting the object to be retrieved. When the computer device extracts the video frame from the video, it may randomly extract one or more video frames from the video, or periodically extract video frames at a preset interval, which is not limited in this embodiment.

[0057] Step 110: Determine the target video information corresponding to the target search information according to the target search information and the tag library; wherein the tag library stores the corresponding relationship between the search information and the video information.

[0058] The tag library may be pre-stored in a computer device. The tag library stores a large amount of search information, a large amount of video information, and the corresponding relationship between the search information and the video information. The search information and the video information may correspond one to one, that is, one search information corresponds to one video information; it may also correspond one to many, that is, one search information corresponds to multiple video information; it may also correspond many to one, that is, multiple search information corresponds to one video information. The video information may include the storage location of the video, the name of the video, the duration of the video, etc. This embodiment does not limit the specific content of the video information, as long as its function can be realized.

[0059] After determining the target search information corresponding to the target object in the object to be searched, the computer device searches for search information similar to the target search information in multiple search information in the tag library, and determines the video information corresponding to the search information as the target video information corresponding to the target search information.

[0060] The medical video retrieval method provided in the embodiment of the present application determines the target retrieval information by detecting the object to be retrieved; the target retrieval information includes the name of the target object in the object to be retrieved or the characteristics of the target object; and the target video information corresponding to the target retrieval information is determined according to the target retrieval information and the tag library. In this embodiment, the target retrieval information is determined by detecting the object to be retrieved; the corresponding relationship between the retrieval information and the video information is stored in the tag library, and by searching the tag library for the retrieval information that is the same as the target retrieval information, the video information corresponding to the found retrieval information is determined as the target video information corresponding to the target retrieval information, that is, the target video information corresponding to the target object. In this way, by performing a series of analysis and processing on the object to be retrieved, the target video information corresponding to the target object in the object to be retrieved can be obtained without manual retrieval, which can improve the retrieval efficiency and reduce the consumption of manpower.

[0061] In one embodiment, Figure 2 As shown, it is related to an implementation method of detecting the object to be searched and determining the target search information, and the steps of the implementation method include:

[0062] Step 200: Detect the target object in the search object and determine the detection result.

[0063] After obtaining the object to be retrieved, the computer device detects the target object included in the object to be retrieved to obtain a detection result. The detection result is used to characterize whether the target object in the object to be retrieved is an object in a known category, and the known category includes objects whose names are known to the user. This embodiment does not limit the specific method by which the computer device detects the target object in the object to be retrieved, as long as its function can be achieved.

[0064] Step 210: Determine target retrieval information according to the detection result.

[0065] After the computer device detects the target object in the search object and obtains the detection result, it determines the target search information according to the detection result. Different detection results lead to different target search information, that is, the computer device selects the target search information as the name of the target object or the feature of the target object according to the detection result.

[0066] In this embodiment, the target object in the object to be retrieved is detected to obtain the detection result first, and then the target retrieval information is determined according to the detection result. Different detection results lead to different target retrieval information. In this way, the target retrieval information can be determined quickly and accurately, thereby improving the efficiency and reliability of the medical video retrieval method.

[0067] In one embodiment, a method for detecting a target object in an object to be searched and determining a detection result includes:

[0068] The object to be retrieved is input into a pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

[0069] The detection model can be obtained by pre-training the neural network using training samples and gold standards. The training sample is a picture or video including an object, and the gold standard is the specific name of the object in the picture or video including the object. The output of the detection model includes the name of the object in the input picture or video, and the confidence, which is used to characterize the reliability of the name of the object output by the detection model. Among them, the neural network can be a network of the YOLO (you only look once) series, or a convolutional neural network.

[0070] The computer device inputs the object to be retrieved into the detection model, and the detection model detects the target object in the object to be retrieved and outputs the detection result. The detection result includes the detection name and confidence of the target object in the object to be retrieved, and the confidence is used to characterize the reliability of the detection name of the target object. For example, if the target object in the object to be retrieved is a puncture, the output result of the detection model includes the name and confidence of the target object, the name of the target object is a puncture, and the confidence indicates the probability that the name of the target object is a puncture.

[0071] In this embodiment, the detection model is directly used to detect the target object in the search object to obtain the detection result. This method of determining the detection result is fast and can improve the efficiency of determining the target search information, thereby improving the efficiency of the medical video retrieval method.

[0072] In another optional embodiment, the computer device may input the object to be retrieved into a pre-trained object detection model. The object detection model is obtained by training a neural network using a picture or video including the object as a training sample and the name of the object in the picture or video as a gold standard. The object detection model detects the target object in the object to be retrieved and obtains a detection result; if the detection result is the name of the object, it means that the target object in the object to be retrieved is an object in a known category; if the detection result is empty, it means that the target object in the object to be retrieved is not an object in a known category.

[0073] In another optional embodiment, when the object to be retrieved is a picture, the computer device can directly input the object to be retrieved into a pre-trained detection model to obtain a detection result. When the object to be retrieved is a video, the computer device can first extract the object to be retrieved, extract the video frame, and input the video frame into the pre-trained detection model to obtain a detection result.

[0074] In one embodiment, Figure 3 As shown, in the case where the detection result includes the detection name and confidence of the target object, an implementation method of determining the target retrieval information according to the detection result is involved, and the steps of the implementation method include:

[0075] Step 300: If the confidence is greater than a preset confidence threshold, the detected name of the target object is used as target retrieval information.

[0076] The preset confidence threshold is preset by the user and stored in the computer device. After obtaining the detection result, the computer device compares the confidence in the detection result with the preset confidence threshold to determine whether the confidence is greater than the preset confidence threshold. If the computer device determines that the confidence in the detection result is greater than the preset confidence threshold, it means that the detection name of the target object in the detection result has a high reliability, and the detection name of the target object is used as the target retrieval information.

[0077] Step 310: If the confidence is not greater than the preset confidence threshold, the features of the target object are extracted and the features of the target object are used as target retrieval information.

[0078] If the computer device determines that the confidence in the detection result is not greater than the preset confidence threshold by comparing the confidence in the detection result with the preset confidence threshold, indicating that the reliability of the detection name of the target object in the detection result is low, then the characteristics of the target object are extracted and the characteristics of the target object are used as target retrieval information. This embodiment does not limit the specific method of extracting the characteristics of the target object, as long as its function can be achieved.

[0079] In an optional embodiment, the computer device may use feature descriptors, such as scale-invariant feature transform (SIFT) and histogram of oriented gradient (HOG) to extract features of the target object. The extracted features of the target object may be global features or local features.

[0080] In another optional embodiment, the computer device may use a pre-trained feature extraction model to extract the features of the target object. Specifically, the method for extracting the features of the target object includes: performing segmentation processing on the object to be retrieved to obtain a picture of the target object; inputting the picture of the target object into the feature extraction model to obtain the features of the target object. Among them, the segmentation processing of the object to be retrieved can adopt a deep learning method, using a pre-trained segmentation network. The segmentation network can be a large-scale convolutional neural network (Mask-RCNN). In this way, the target object is segmented from the object to be retrieved for feature extraction, which can avoid feature interference from other areas in the object to be retrieved and improve the accuracy of the extracted features of the target object.

[0081] In this embodiment, if the detection result after detecting the target object in the object to be retrieved includes the detection name and confidence of the target object, when the confidence is greater than the preset confidence threshold, the detection name of the target object is directly determined as the target retrieval information; when the confidence is not greater than the preset confidence threshold, the characteristics of the target object are determined as the target retrieval information. In this way, by comparing the confidence with the preset confidence threshold, according to the comparison result, the method of using the detection name of the target object or the characteristics of the target object as the target retrieval information is fast and easy to implement, which can improve the efficiency of determining the target retrieval information, thereby improving the efficiency of the medical video retrieval method, that is, it can quickly determine the target video information.

[0082] In one embodiment, Figure 4 As shown, an implementation method of determining target video information corresponding to the target retrieval information according to the target retrieval information and the tag library includes the following steps:

[0083] Step 400: Calculate the similarity between the target search information and the search information in the tag library.

[0084] After obtaining the target search information corresponding to the target object, the computer device calculates the similarity between the target search information and each search information stored in the tag library. This embodiment does not limit the specific method of calculating the similarity, as long as its function can be achieved.

[0085] In an optional embodiment, the computer device can use a pre-trained similarity calculation model to calculate the similarity, that is, the target retrieval information and each retrieval information in the tag library are input into the similarity calculation model to obtain the similarity between the target retrieval information and each retrieval information.

[0086] In another optional embodiment, the computer device may use a BERT (Bidirectional Encoder Representation from Transformers) model to calculate the similarity.

[0087] Step 410: Determine the video information corresponding to the search information whose similarity is greater than a preset similarity threshold as the target video information.

[0088] The preset similarity threshold may be pre-stored in the computer device by the user. After calculating the similarity between each retrieval information in the tag library and the target retrieval information, the computer device compares each similarity with the preset similarity threshold, obtains the retrieval information with a similarity greater than the preset similarity threshold, and determines the video information corresponding to the retrieval information as the target video information corresponding to the target object. There may be multiple retrieval information with a similarity greater than the preset similarity threshold, and there are also multiple target video information to be determined.

[0089] In an optional embodiment, when there are multiple target video information, the multiple target video information are sorted from large to small according to the similarity of the search information corresponding to the target video information, and displayed on the display interface of the computer device.

[0090] In this embodiment, by calculating the similarity between the target retrieval information and each retrieval information stored in the tag library, the video information corresponding to the retrieval information whose similarity is greater than a preset similarity threshold is determined as the target video information. This method of determining the target video information is quick, and the medical video retrieval method has higher efficiency.

[0091] In one embodiment, the target video information includes a target video corresponding to the target object and a position of the target object in the target video.

[0092] The target video corresponding to the target object refers to a video including the target object stored in the database. Specifically, the target video information may include the storage location and name of the target video corresponding to the target object. The position of the target object in the target video refers to the time period when the target object appears in the target video.

[0093] In an optional embodiment, when the target video information is displayed on the display interface, the position of the target object in the target video is also displayed on the progress bar of the target video.

[0094] In this embodiment, the target video information includes not only the target video corresponding to the target object, but also the position of the target object in the target video. In this way, the medical video retrieval method provided by the embodiment of the present application can not only efficiently and accurately retrieve the target video corresponding to the target object, but also automatically locate the time period of interest in the target video, that is, the time period when the target object appears, making the medical video retrieval method more practical.

[0095] When searching using the medical video retrieval method provided in the embodiment of the present application, the tag library used stores the correspondence between the search information and the video information. In this embodiment, the method for obtaining the correspondence between the search information and the video information includes:

[0096] Traverse the videos to be processed in the existing video library, determine the retrieval information and video information corresponding to the videos to be processed, and build a correspondence between the retrieval information and the video information; the correspondence between the retrieval information and the video information includes the correspondence between the name and the video information, and the correspondence between the features and the video information.

[0097] The video library can be a database storing videos in a computer device, and the video library includes a large number of videos to be retrieved. There can be multiple video libraries in the computer device.

[0098] The computer device traverses all videos in the video library, takes each video as a video to be processed, obtains the retrieval information and video information of the video to be processed, and constructs a corresponding relationship between the retrieval information and the video information. The retrieval information of the video to be processed includes the name of the target object and / or the characteristics of the target object in the video to be processed. The video information includes the storage location, name and time period of the target object in the video. The corresponding relationship between the retrieval information and the video information includes the corresponding relationship between the name and the video information, and the corresponding relationship between the characteristics and the video information.

[0099] In this embodiment, by traversing all video libraries in the video library, the retrieval information and video information corresponding to all videos are determined, and a correspondence between the retrieval information and the video information is constructed. The correspondence includes not only the correspondence between the name and the video information, but also the correspondence between the feature and the video information. In this way, when using the tag library provided by this embodiment, the target retrieval information of the target object obtained, whether it is the name of the target object or the feature of the target object, can retrieve the corresponding target video information in the tag library, so that the medical video retrieval method provided by this embodiment has high practicality.

[0100] In an optional embodiment, the method for obtaining the correspondence between the search information and the video information, that is, the method for obtaining the tag library, can be performed by the computer device when the server is idle, which will not affect the normal use of the server and can improve the utilization rate of the server.

[0101] In an optional embodiment, the tag library used includes a name tag library and a feature tag library; the name tag library stores the correspondence between the name and the video information, and the feature tag library stores the correspondence between the feature and the video information. When the medical video retrieval method is used for retrieval, if the target retrieval information determined after the object to be retrieved is the name of the target object, then the video information corresponding to the name of the target object is determined in the name tag library according to the name of the target object; if the target retrieval information determined after the object to be retrieved is the feature of the target object, then the video information corresponding to the feature of the target object is determined in the feature tag library according to the feature of the target object.

[0102] In one embodiment, Figure 5 As shown, an implementation method for determining the search information corresponding to the video to be processed is involved, and the steps of the implementation method include:

[0103] Step 500: extract each video frame in the video to be processed, and determine the search information corresponding to the object to be processed in each video frame.

[0104] After obtaining the video to be processed from the video library, the computer device extracts each video frame in the video to be processed (i.e., the image of each frame in the video to be processed), analyzes and processes each video frame respectively, and determines the search information corresponding to the object to be processed in each video frame. A video frame may include multiple objects to be processed, so it is necessary to determine the search information corresponding to each object to be processed in a video frame. The search information corresponding to the object to be processed may include the name of the object to be processed, and may also include the characteristics of the object to be processed.

[0105] When the search information corresponding to the object to be processed includes the name of the object to be processed, the computer device can use a deep learning method to determine the name of the object to be processed in each video frame. That is, the computer device inputs each video frame into a pre-trained detection model to obtain a detection result; the detection result includes the detection name of the object to be processed in each video frame, and the confidence; if the confidence is greater than a preset confidence threshold, the detection name of the object to be processed is determined as the name of the object to be processed; if the confidence is not greater than the preset confidence threshold, it is necessary to obtain the features of the object to be processed.

[0106] When the search information corresponding to the object to be processed includes the features of the object to be processed, for each video frame, the computer device performs segmentation processing on the video frame to obtain a picture of the object to be processed in the video frame; and extracts features of the object to be processed to obtain the features of the object to be processed. The description of extracting the features of the object to be processed can refer to the specific description of extracting the features of the target object in the above embodiment, which will not be repeated here.

[0107] Step 510: Determine the search information corresponding to the object to be processed in each video frame as the search information corresponding to the video to be processed.

[0108] After obtaining the search information corresponding to the object to be processed in each video frame, the computer device determines the search information corresponding to each video frame as the search information corresponding to the video to be processed. In other words, there may be multiple search information for one video to be processed.

[0109] In this embodiment, by extracting each video frame in the video to be processed, the retrieval information corresponding to each video frame is determined as the retrieval information corresponding to the video to be processed. This method of determining the retrieval information corresponding to the video to be processed is quick and easy to implement, and can improve the accuracy and efficiency of obtaining the tag library.

[0110] In one embodiment, when the search information corresponding to the object to be processed is a feature of the target object, such as Figure 6 As shown, another implementation method for determining the search information corresponding to the video to be processed is provided, and the steps of the implementation method include:

[0111] Step 600: Divide the video to be processed into multiple video segments, and determine the features corresponding to each video segment.

[0112] The computer device divides each video to be processed into multiple video segments and extracts the features of each video segment. The feature of each video segment refers to the global feature of the video segment. The computer device can divide the video to be processed according to the length of the processed video. This embodiment does not limit the specific method of dividing the video to be processed, as long as its function can be achieved.

[0113] In an optional embodiment, the computer device can use a region convolutional 3D network (R-C3D) model to divide the video to be processed, that is, the video to be processed is input into the R-C3D model, and multiple video segments can be obtained. In this embodiment, the R-C3D model is used to divide the video to be processed, and fewer redundant video segments are obtained, which can ensure that each video segment includes the complete operation process of the target object, thereby improving the accuracy of locating the target object in the target video using the medical video retrieval method provided by this embodiment.

[0114] In one embodiment, Figure 7 As shown, an implementation method for determining features corresponding to each video segment includes the following steps:

[0115] Step 700: For each video segment, extract each key frame of the video segment, and determine the initial features of the object to be processed in each key frame.

[0116] After obtaining each video segment of the video to be processed, the computer device extracts multiple key frames in each video segment. A key frame refers to a video frame in a video segment.

[0117] In an optional embodiment, the computer device may sample the video segment at a preset interval to obtain multiple key frames; or may use a deep learning method to extract each key frame in the video segment.

[0118] After obtaining each key frame in the video frame, the computer device extracts the initial features of the object to be processed in the key frame for each key frame. The initial features are global features of the object to be processed. The description of extracting the initial features of the object to be processed in the key frame can refer to the above-mentioned specific description of extracting the features of the target object, which will not be repeated here.

[0119] Step 710: Calculate the average value of the initial features of the object to be processed in all key frames in the video segment to obtain the features corresponding to the video segment.

[0120] After obtaining the initial features of the object to be processed in each key frame in the video segment in the computer device, the average value of the initial features of the object to be processed in all key frames is calculated, and the average value is used as the feature corresponding to the video segment.

[0121] Step 610: Determine the features corresponding to each video segment as the features corresponding to the video to be processed.

[0122] After determining the features corresponding to each video frame in the video to be processed, the computer device determines the features corresponding to each video frame as the features corresponding to the video to be processed. In other words, each video to be processed may correspond to multiple features.

[0123] In this embodiment, the video to be processed in the video library is divided into multiple video segments, and the features corresponding to each video segment obtained by calculation are determined as the features corresponding to the video to be processed. In this way, a corresponding relationship between the features and the video segments in the video to be processed can be established, which facilitates the rapid and accurate positioning of the target object in the target video during subsequent video retrieval.

[0124] See also Figure 8 An embodiment of the present application provides a medical video retrieval method, the method comprising the steps of:

[0125] Step 800: Input the object to be retrieved into the pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object;

[0126] Step 810: If the confidence is greater than a preset confidence threshold, the detection name of the target object in the detection result is used as the target retrieval information;

[0127] Step 820: If the confidence is not greater than the preset confidence threshold, extract the features of the target object and use the features of the target object as target retrieval information;

[0128] Step 830: calculating the similarity between the target search information and the target search information in the tag library; the tag library stores the corresponding relationship between the search information and the video information;

[0129] Step 840: Determine the video information corresponding to the search information with a similarity greater than a preset similarity threshold as the target video information; the target video information includes the target video corresponding to the target object and the position of the target object in the target video.

[0130] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a medical video retrieval device for implementing the medical video retrieval method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more medical video retrieval device embodiments provided below can refer to the limitations of the medical video retrieval method above, and will not be repeated here.

[0132] In one embodiment, Fig. 9 As shown, a medical video retrieval device 10 is provided, comprising: a detection module 11 and a determination module 12, wherein:

[0133] The detection module 11 is used to detect the object to be searched and determine target search information; the target search information includes the name information of the target object in the object to be searched or the characteristics of the target object.

[0134] The determination module 12 is used to determine the target video information corresponding to the target search information according to the target search information and the tag library; wherein the tag library stores the corresponding relationship between the search information and the video information.

[0135] In one embodiment, the detection module 11 includes a detection unit and a determination unit; the detection unit is used to detect the target object in the search object and determine the detection result; the determination unit is used to determine the target search information according to the detection result.

[0136] In one embodiment, the determination module 12 includes a judgment unit. The judgment unit is used to use the detected name of the target object as the target retrieval information if the confidence is greater than a preset confidence threshold; if the confidence is not greater than the preset confidence threshold, extract the features of the target object and use the features of the target object as the target retrieval information.

[0137] In one embodiment, the determination module 12 further includes a calculation unit and a determination unit. The calculation unit is used to calculate the similarity between the target search information and the search information in the tag library; the determination unit is used to determine the video information corresponding to the search information whose similarity is greater than a preset similarity threshold as the target video information.

[0138] In one embodiment, the retrieval module 11 is used to input the object to be retrieved into a pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

[0139] In one embodiment, the target video information includes a target video corresponding to the target object and a position of the target object in the target video.

[0140] In one embodiment, the medical video retrieval device 10 further includes an acquisition module, which is used to acquire the correspondence between the retrieval information and the video information. The acquisition module is specifically used to traverse the videos to be processed in the video library, determine the retrieval information and video information corresponding to the videos to be processed, and construct the correspondence between the retrieval information and the video information; the correspondence between the retrieval information and the video information includes the correspondence between the name and the video information, and the correspondence between the feature and the video information.

[0141] In one embodiment, the determination module 12 also includes an extraction unit, which is used to extract each video frame in the video to be processed, determine the retrieval information corresponding to the object to be processed in each video frame; and determine the retrieval information corresponding to the object to be processed in each video frame as the retrieval information corresponding to the video to be processed.

[0142] In one embodiment, the determination module 12 further includes a division unit, which is used to divide the video to be processed into multiple video segments, determine the features corresponding to each video segment, and determine the features corresponding to each video segment as the features corresponding to the video to be processed.

[0143] In one embodiment, the division unit is specifically used to extract each key frame of the video segment for each video segment, determine the initial features of the object to be processed in each key frame; calculate the average value of the initial features of the object to be processed in all key frames in the video segment, and obtain the features corresponding to the video segment.

[0144] Each module in the above video retrieval device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a medical video retrieval method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0146] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0148] Detecting the object to be searched and determining target search information; the target search information includes the name of the target object or the characteristics of the target object;

[0149] According to the target retrieval information and the tag library, the target video information corresponding to the target retrieval information is determined; wherein the tag library stores the corresponding relationship between the retrieval information and the video information.

[0150] In one embodiment, when the processor executes the computer program, the following steps are further implemented: detecting the target object in the objects to be retrieved and determining the detection result; and determining the target retrieval information according to the detection result.

[0151] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if the confidence is greater than a preset confidence threshold, the detected name of the target object is used as the target retrieval information; if the confidence is not greater than the preset confidence threshold, the features of the target object are extracted, and the features of the target object are used as the target retrieval information.

[0152] In one embodiment, when the processor executes the computer program, the following steps are also implemented: calculating the similarity between the target retrieval information and the retrieval information in the tag library; and determining the video information corresponding to the retrieval information whose similarity is greater than a preset similarity threshold as the target video information.

[0153] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the object to be retrieved is input into a pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

[0154] In one embodiment, the target video information includes a target video corresponding to the target object and a position of the target object in the target video.

[0155] In one embodiment, when the processor executes the computer program, the following steps are also implemented: traversing the videos to be processed in the video library, determining the retrieval information and video information corresponding to the videos to be processed, and constructing a correspondence between the retrieval information and the video information; the correspondence between the retrieval information and the video information includes the correspondence between the name and the video information, and the correspondence between the features and the video information.

[0156] In one embodiment, when the processor executes the computer program, the following steps are also implemented: extracting each video frame in the video to be processed, and determining the retrieval information corresponding to the object to be processed in each video frame; determining the retrieval information corresponding to the object to be processed in each video frame as the retrieval information corresponding to the video to be processed.

[0157] In one embodiment, when the processor executes the computer program, the following steps are also implemented: dividing the video to be processed into multiple video segments, determining the features corresponding to each video segment; and determining the features corresponding to each video segment as the features corresponding to the video to be processed.

[0158] In one embodiment, when the processor executes the computer program, the following steps are also implemented: for each video segment, each key frame of the video segment is extracted, and the initial features of the object to be processed in each key frame are determined; the average value of the initial features of the object to be processed in all key frames in the video segment is calculated to obtain the features corresponding to the video segment.

[0159] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0160] Detect the object to be searched and determine target search information; the target search information includes the name of the target object to be searched or the characteristics of the target object;

[0161] According to the target retrieval information and the tag library, the target video information corresponding to the target retrieval information is determined; wherein the tag library stores the corresponding relationship between the retrieval information and the video information.

[0162] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: detecting the target object in the objects to be retrieved and determining the detection result; and determining the target retrieval information according to the detection result.

[0163] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: if the confidence is greater than a preset confidence threshold, the detected name of the target object is used as the target retrieval information; if the confidence is not greater than the preset confidence threshold, the features of the target object are extracted, and the features of the target object are used as the target retrieval information.

[0164] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: calculating the similarity between the target retrieval information and the retrieval information in the tag library; and determining the video information corresponding to the retrieval information whose similarity is greater than a preset similarity threshold as the target video information.

[0165] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the object to be retrieved is input into a pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

[0166] In one embodiment, the target video information includes a target video corresponding to the target object and a position of the target object in the target video.

[0167] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: traversing the videos to be processed in the video library, determining the retrieval information and video information corresponding to the videos to be processed, and constructing a correspondence between the retrieval information and the video information; the correspondence between the retrieval information and the video information includes the correspondence between the name and the video information, and the correspondence between the features and the video information.

[0168] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: extracting each video frame in the video to be processed, and determining the retrieval information corresponding to the object to be processed in each video frame; determining the retrieval information corresponding to the object to be processed in each video frame as the retrieval information corresponding to the video to be processed.

[0169] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: dividing the video to be processed into multiple video segments, determining the features corresponding to each video segment; and determining the features corresponding to each video segment as the features corresponding to the video to be processed.

[0170] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: for each video segment, each key frame of the video segment is extracted, and the initial features of the object to be processed in each key frame are determined; the average value of the initial features of the object to be processed in all key frames in the video segment is calculated to obtain the features corresponding to the video segment.

[0171] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0172] Detect the object to be searched and determine target search information; the target search information includes the name of the target object in the object to be searched or the characteristics of the target object;

[0173] According to the target retrieval information and the tag library, the target video information corresponding to the target retrieval information is determined; wherein the tag library stores the corresponding relationship between the retrieval information and the video information.

[0174] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: detecting the target object in the objects to be retrieved and determining the detection result; and determining the target retrieval information according to the detection result.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: if the confidence is greater than a preset confidence threshold, the detected name of the target object is used as the target retrieval information; if the confidence is not greater than the preset confidence threshold, the features of the target object are extracted, and the features of the target object are used as the target retrieval information.

[0176] The similarity between the target retrieval information and the retrieval information in the tag library is calculated; and the video information corresponding to the retrieval information whose similarity is greater than a preset similarity threshold is determined as the target video information.

[0177] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the object to be retrieved is input into a pre-trained detection model to obtain a detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

[0178] In one embodiment, the target video information includes a target video corresponding to the target object and a position of the target object in the target video.

[0179] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: traversing the videos to be processed in the video library, determining the retrieval information and video information corresponding to the videos to be processed, and constructing a correspondence between the retrieval information and the video information; the correspondence between the retrieval information and the video information includes the correspondence between the name and the video information, and the correspondence between the features and the video information.

[0180] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: extracting each video frame in the video to be processed, and determining the retrieval information corresponding to the object to be processed in each video frame; determining the retrieval information corresponding to the object to be processed in each video frame as the retrieval information corresponding to the video to be processed.

[0181] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: dividing the video to be processed into multiple video segments, determining the features corresponding to each video segment; and determining the features corresponding to each video segment as the features corresponding to the video to be processed.

[0182] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: for each video segment, each key frame of the video segment is extracted, and the initial features of the object to be processed in each key frame are determined; the average value of the initial features of the object to be processed in all key frames in the video segment is calculated to obtain the features corresponding to the video segment.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A medical video retrieval method, characterized in that: The method comprises: Detect the object to be searched and determine target search information; the target search information includes the name of the target object in the object to be searched or the characteristics of the target object; According to the target search information and the tag library, target video information corresponding to the target search information is determined; wherein the tag library stores the corresponding relationship between the search information and the video information.

2. The method according to claim 1, characterized in that The detecting of the object to be searched and determining the target search information includes: Detecting the target object in the to-be-searched object and determining a detection result; The target retrieval information is determined according to the detection result.

3. The method according to claim 2, characterized in that The detection result includes the detection name and confidence level of the target object, and determining the target retrieval information according to the detection result includes: If the confidence level is greater than a preset confidence threshold, the detected name of the target object is used as the target retrieval information; If the confidence is not greater than the preset confidence threshold, the features of the target object are extracted and the features of the target object are used as the target retrieval information.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining target video information corresponding to the target search information according to the target search information and the tag library includes: Calculating the similarity between the target search information and the search information in the tag library; The video information corresponding to the search information whose similarity is greater than a preset similarity threshold is determined as the target video information.

5. The method according to any one of claims 1 to 3, characterized in that: The step of detecting the target object in the object to be searched and determining the detection result includes: The object to be retrieved is input into a pre-trained detection model to obtain the detection result; the detection result includes the detection name and confidence of the target object in the object to be retrieved; the confidence is used to characterize the reliability of the detection name of the target object.

6. The method according to any one of claims 1 to 3, characterized in that: The target video information includes a target video corresponding to the target object and a position of the target object in the target video.

7. The method according to any one of claims 1 to 3, characterized in that: The method for obtaining the corresponding relationship between the search information and the video information includes: Traverse the videos to be processed in the video library, determine the retrieval information and video information corresponding to the videos to be processed, and construct a correspondence between the retrieval information and the video information; the correspondence between the retrieval information and the video information includes a correspondence between a name and video information, and a correspondence between a feature and video information.

8. The method according to claim 7, characterized in that The determining of the search information corresponding to the video to be processed includes: Extracting each video frame in the video to be processed, and determining the search information corresponding to the object to be processed in each video frame; The search information corresponding to the object to be processed in each of the video frames is determined as the search information corresponding to the video to be processed.

9. The method according to claim 8, characterized in that If the search information is a feature of the target object, determining the search information corresponding to the video to be processed includes: Dividing the video to be processed into multiple video segments, and determining features corresponding to each of the video segments; The features corresponding to each of the video segments are determined as the features corresponding to the video to be processed.

10. The method according to claim 9, characterized in that The determining the features corresponding to each of the video segments includes: For each of the video segments, extract each key frame of the video segment, and determine the initial features of the object to be processed in each of the key frames; The average value of the initial features of the object to be processed in all the key frames in the video segment is calculated to obtain the features corresponding to the video segment.

11. A medical video retrieval device, characterized in that: The device comprises: A detection module, used to detect the object to be searched and determine target search information; the target search information includes the name of the target object in the object to be searched or the characteristics of the target object; The determination module is used to determine the target video information corresponding to the target search information according to the target search information and the tag library; wherein the tag library stores the corresponding relationship between the search information and the video information.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images