Endoscope video processing method and device and electronic device

Abstract

The invention relates to an endoscope video processing method and device and an electronic device.The endoscope video processing method comprises the steps that non-target frames are determined in an initial endoscope video, the non-target frames are removed, and intermediate endoscope video data are obtained; the non-target frame comprises a non-information frame; determining two adjacent video frames of which the frame spacing is greater than a preset threshold value in the intermediate endoscope video data as a first target frame and a second target frame; wherein the preset threshold value is greater than the unit frame spacing; and generating an intermediate frame according to the first target frame and the second target frame, and inserting the intermediate frame between the first target frame and the second target frame to obtain a target endoscope video. According to the method and the device, after the non-information frames of the endoscope video are removed, frame supplement processing is carried out between the two video frames with a relatively large frame distance, so that the video content of the processed endoscope video can be more coherent and smooth. According to the invention, the problem that the video content of the endoscope video after non-information frame removal processing is not coherent in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of endoscopic video processing, and in particular to an endoscopic video processing method, device and electronic device. Background Art

[0002] There are a lot of non-information clips in endoscopic videos. The existence of non-information clips will not only interfere with the doctor's judgment, but also take up more disk space, consume more time for doctors to review, and cause a waste of resources. Therefore, it is necessary to remove non-information clips in endoscopic videos.

[0003] In the related art, non-information frames in endoscopic videos can be selected and removed to achieve the removal of non-information segments. However, after the non-information frames are removed, the endoscopic video has the problem of incoherent video content and affected fluency, and there is currently no effective solution. Summary of the invention

[0004] The present invention provides an endoscopic video processing method, device and storage medium to solve the problem in the related art that the endoscopic video after non-information frame removal processing has incoherent video content and affected fluency.

[0005] In a first aspect, the present invention provides an endoscopic video processing method, the processing method comprising:

[0006] Determine non-target frames in the initial endoscopic video, remove the non-target frames, and obtain intermediate endoscopic video data; the non-target frames include non-information frames;

[0007] Determine two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than a unit frame spacing;

[0008] An intermediate frame is generated according to the first target frame and the second target frame, and the intermediate frame is inserted between the first target frame and the second target frame to obtain a target endoscopic video.

[0009] In some embodiments, generating an intermediate frame according to the first target frame and the second target frame includes:

[0010] determining a reference frame among the removed non-target frames between the first target frame and the second target frame;

[0011] The intermediate frame is generated according to the first target frame, the second target frame and the reference frame.

[0012] In some of the embodiments, each pixel in the intermediate frame corresponds to each pixel in the reference frame one by one;

[0013] The step of generating the intermediate frame according to the first target frame, the second target frame and the reference frame comprises:

[0014] Respectively determining matching points of each pixel in the reference frame in the first target frame or the second target frame;

[0015] The pixel value of the corresponding pixel point in the intermediate frame is determined according to the pixel value of the matching point of each pixel point in the reference frame.

[0016] In some embodiments, determining matching points of each pixel in the reference frame in the first target frame and the second target frame includes:

[0017] identifying feature points of the first target frame, the second target frame, and the reference frame;

[0018] Matching feature points between the first target frame and the reference frame to obtain a first matching point pair;

[0019] Matching feature points between the second target frame and the reference frame to obtain a second matching point pair;

[0020] According to the first matching point pair and the second matching point pair, matching points of each pixel point in the reference frame in the first target frame or the second target frame are determined respectively.

[0021] In some of the embodiments, the first matching point pair includes a first feature point in the reference frame and its matching point in the first target frame, and the second matching point pair includes a second feature point in the reference frame and its matching point in the second target frame;

[0022] The step of determining, according to the first matching point pair and the second matching point pair, matching points of each pixel point in the reference frame in the first target frame or the second target frame, respectively, includes:

[0023] Determining a first pose relationship between the first target frame and the reference frame, and determining a second pose relationship between the second target frame and the reference frame;

[0024] For a pixel point that has not been matched in the reference frame, determining a first distance between the pixel point and the first feature point, and determining a second distance between the pixel point and the second feature point;

[0025] For the pixel points whose first spacing is smaller than the second spacing, determining matching points of the pixel points in the first target frame according to the first posture relationship;

[0026] For the pixel points whose second spacing is smaller than the first spacing, the matching points of the pixel points in the second target frame are determined according to the second posture relationship.

[0027] In some embodiments, determining a first pose relationship between the first target frame and the reference frame, and determining a second pose relationship between the second target frame and the reference frame, comprises:

[0028] Matching feature points between the first target frame and the second target frame to obtain a target matching point pair;

[0029] Determining a target pose relationship between the first target frame and the second target frame according to the target matching point pair;

[0030] The first posture relationship and the second posture relationship are determined according to the target posture relationship.

[0031] In some of the embodiments, determining non-target frames in the initial endoscopic video includes:

[0032] Performing frame classification on the initial endoscopic video to determine the non-target frames and the confidence of each of the non-target frames;

[0033] The step of determining a reference frame from the removed non-target frames between the first target frame and the second target frame comprises:

[0034] Among the non-target frames that have been removed between the first target frame and the second target frame, a reference frame is determined according to the confidence of each of the non-target frames.

[0035] In some of the embodiments, determining non-target frames in the initial endoscopic video includes:

[0036] Performing frame classification on the initial endoscopic video, and determining information frames and the non-information frames in the initial endoscopic video;

[0037] The continuous non-information frames are determined as the non-target frames; or, the continuous information frames are determined as the target frames, and the remaining frames are determined as the non-target frames.

[0038] In a second aspect, the present invention provides an endoscopic video processing device, the processing device comprising:

[0039] A frame removal processing module, used to determine non-target frames in the initial endoscopic video, remove the non-target frames, and obtain intermediate endoscopic video data; the non-target frames include non-information frames;

[0040] An intermediate processing module, used for determining two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than a unit frame spacing;

[0041] The frame interpolation processing module is used to generate an intermediate frame according to the first target frame and the second target frame, and insert the intermediate frame between the first target frame and the second target frame to obtain a target endoscopic video.

[0042] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the endoscopic video processing method described in the first aspect when executing the computer program.

[0043] In a fourth aspect, the present invention provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the endoscopic video processing method described in the first aspect is implemented.

[0044] The endoscopic video processing method, device and electronic device provided in the present invention first remove non-target frames in the endoscopic video; then generate an intermediate frame between two adjacent target frames whose frame spacing is greater than a preset threshold to achieve frame interpolation; finally, a target endoscopic video with non-target frames removed and containing intermediate frames is obtained. Among them, the non-target frames include non-information frames. Therefore, compared with the prior art, the endoscopic video processing method provided by the present invention, after removing the non-information frames of the endoscopic video, also performs frame interpolation processing between two video frames with a larger frame spacing, so that the video content of the processed endoscopic video can be more coherent and smooth. Thereby, the problem in the related art that the endoscopic video after the non-information frame removal processing has incoherent video content and affected fluency is solved.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 It is a hardware structure block diagram of a terminal for executing the endoscopic video processing method provided by the present invention;

[0048] Figure 2 is a flow chart of the endoscopic video processing method provided by the present invention;

[0049] Figure 3 is a flow chart of an endoscopic video processing method in a specific embodiment of the present invention;

[0050] Figure 4 It is a structural block diagram of the endoscopic video processing device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal for executing the endoscopic video processing method provided by the present invention. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0054] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the endoscopic video processing method provided in the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0056] The present invention provides an endoscopic video processing method. Figure 2 FIG. 1 is a flow chart of the endoscopic video processing method provided by the present invention. Figure 2 As shown, the process includes the following steps:

[0057] Step S210, determining non-target frames in the initial endoscopic video, removing the non-target frames, and obtaining intermediate endoscopic video data; the non-target frames include non-information frames.

[0058] Step S220, determining two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than a unit frame spacing.

[0059] Step S230, generating an intermediate frame according to the first target frame and the second target frame, and inserting the intermediate frame between the first target frame and the second target frame to obtain a target endoscopic video.

[0060] Specifically, the initial endoscopic video may be an endoscopic video directly captured by an endoscopic lens. However, not all contents of the initial endoscopic video can assist in diagnosis, so it is necessary to remove invalid contents in the initial endoscopic video, that is, to remove non-target frames in the initial endoscopic video. Non-target frames are video frames in the initial endoscopic video that cannot be used for assisting diagnosis and need to be removed.

[0061] Furthermore, non-target frames at least include non-information frames, and removing non-target frames will also remove non-information frames accordingly. Non-information frames refer to video frames that do not contain valid information. Among them, valid information refers to relevant information that can be used as a reference for medical staff during diagnosis. Due to its lack of valid information, non-information frames are difficult to use for auxiliary diagnosis. Exemplarily, non-information frames can be video frames where effective information is partially or completely missing due to problems such as the lens being outside the patient's body, endoscope examination, highlight / liquid reflection, contrast problems, motion / out-of-focus blur, bubbles, effusion, and excessive feces. It should be noted that in different usage scenarios or requirements, the effective information that can be used for auxiliary diagnosis is different, and the criteria for determining non-information frames are also different. The above is only an exemplary description of non-information frames.

[0062] After removing the non-target frames, the video frames retained in the initial endoscopic video constitute the intermediate endoscopic video data. At this time, the frame spacing between any two adjacent video frames in the intermediate endoscopic video data can be analyzed. Among them, the frame spacing can be the acquisition time interval of each frame, which can be determined based on the timestamp of the video frame. In the initial endoscopic video, each video frame has a timestamp. In some cases, the timestamp indicates the absolute acquisition time of the corresponding video frame; while in other cases, the timestamp indicates the time difference between the corresponding video frame and the first video frame. Therefore, in both cases, the frame spacing between the two video frames can be determined by the timestamp difference between the two video frames. In the complete initial endoscopic video, the frame spacing between adjacent video frames is related to the acquisition frequency of the endoscopic lens. The higher the acquisition frequency, the smaller the frame spacing. At this time, the frame spacing between adjacent video frames can be defined as the unit frame spacing. In the intermediate endoscopic video data, the frame spacing between the two video frames originally before and after the non-target frame must be greater than the unit frame spacing, and an excessively large frame spacing will make the video content incoherent, and the higher the frame spacing between adjacent video frames, the more serious the problem of incoherent video content.

[0063] In order to solve the above problems, the present invention performs frame interpolation processing on the intermediate endoscopic video data. First, two adjacent video frames whose frame spacing is greater than a preset threshold value in the intermediate endoscopic video data are determined. The preset threshold value needs to be greater than the unit frame spacing, so as to determine the adjacent video frames whose frame spacing is too large, and determine the two as the first target frame and the second target frame; then an intermediate frame is generated according to the first target frame and the second target frame, and the intermediate frame is a newly generated video frame; finally, the intermediate frame is inserted between the first target frame and the second target frame, so as to complete the frame interpolation operation and generate the target endoscopic video. It should be noted that the preset threshold value of the frame spacing can be determined according to the actual situation, such as two unit frame spacings, three unit frame spacings, etc. Exemplarily, when the frame spacing of adjacent video frames is greater than three unit frame spacings, which will affect the video fluency, the preset threshold value can be determined as three unit video frames. At the same time, the number of intermediate frames between the first target frame and the second target frame can also be determined according to the actual situation, mainly based on the frame spacing between the first target frame and the second target frame, the consistency requirements of the endoscopic video, and other factors. Generally, the frame spacing between the first target frame and the second target frame is negatively correlated with the number of intermediate frames generated, and the continuity requirement of the endoscopic video is positively correlated with the number of intermediate frames generated.

[0064] In summary, the endoscopic video processing method provided by the present invention first removes non-target frames in the endoscopic video; then generates an intermediate frame between two adjacent target frames whose frame spacing is greater than a preset threshold to achieve frame interpolation; and finally obtains a target endoscopic video with non-target frames removed and containing intermediate frames. Among them, the non-target frames include non-information frames. Therefore, compared with the prior art, the endoscopic video processing method provided by the present invention, after removing the non-information frames of the endoscopic video, also performs frame interpolation processing between two video frames with a larger frame spacing, so that the video content of the processed endoscopic video can be more coherent and smooth. Thereby, the problem in the related art that the endoscopic video after the non-information frame removal processing has incoherent video content and affected fluency is solved.

[0065] As an embodiment of the present application, some specific frame supplementation methods are provided as follows.

[0066] In some embodiments, an intermediate frame can be generated by a frame interpolation algorithm based on an optical flow method. First, a dense optical flow between a first target frame and a second target frame is calculated, that is, the displacement of pixels between the first target frame and the second target frame. The first target frame I s With the second target frame I e The dense optical flows between are: V x (u,v) and V y (u,v), where u and v represent the pixel coordinate values ​​on the video frame, V x (u,v) and V y(u, v) represent the displacement of the pixel point (u, v) in the x direction and the y direction respectively. s With the second target frame I e n intermediate frames are inserted between the i-th intermediate frame I i It can be expressed as:

[0067]

[0068] In addition, in some other embodiments, an intermediate frame can be generated by a deep learning-based interpolation algorithm. A depth-aware video frame interpolation method (DAIN) can be used to interpolate frames between the first target frame and the second target frame. The clear and smooth endoscopic video is split into video frames, and sampling is performed at intervals of n frames (for example, n=5). The sth frame I s With the s+nth frame I s+n As the input of model training, I s with I s+n All video frames between the first target frame and the second target frame are used as the standard to train the model. The trained model is used to generate the intermediate frame between the first target frame and the second target frame.

[0069] In addition to the above two frame interpolation methods, the present invention also provides a new frame interpolation method.

[0070] In some of the embodiments, in step S230, generating an intermediate frame according to the first target frame and the second target frame includes: steps S231 to S232.

[0071] Step S231 , determining a reference frame from the removed non-target frames between the first target frame and the second target frame.

[0072] Step S232: generating an intermediate frame according to the first target frame, the second target frame and the reference frame.

[0073] Different from the above-mentioned frame interpolation method, the frame interpolation method provided in this embodiment also utilizes the non-target frames that have been removed between the first target frame and the second target frame. Specifically, according to the number of intermediate frames to be generated, the same number of non-target frames can be determined from the non-target frames that have been removed as reference frames, and finally the corresponding intermediate frames can be generated based on the reference frames.

[0074] It should be noted that, although the reference frame does not have effective information that can be used for auxiliary diagnosis, some image features of the reference frame can be used to assist in the construction of the image of the intermediate frame. The main function of the intermediate frame is to replace the corresponding non-target frame, and the replaced non-target frame is the reference frame. When the spatial pose of the intermediate frame is the same as the spatial pose of the reference frame, a better frame supplementation effect can be achieved. In this embodiment, some image features of the reference frame are used to assist in the construction of the intermediate frame, so that the spatial pose of the intermediate frame can be consistent with the spatial pose of the reference frame. In other frame supplementation methods, the intermediate frame is usually generated directly based on the first target frame and the second target frame. At this time, it is difficult to ensure that the spatial pose of the intermediate frame is the same as the spatial pose of the removed non-target frame.

[0075] Therefore, in some of the embodiments, each pixel in the intermediate frame corresponds to each pixel in the reference frame one by one; step S232, generating an intermediate frame according to the first target frame, the second target frame and the reference frame, includes: step S310 and step S320.

[0076] Step S310 , respectively determining matching points of each pixel point in the reference frame in the first target frame or the second target frame.

[0077] Step S320 , determining the pixel value of the corresponding pixel point in the intermediate frame according to the pixel value of the matching point of each pixel point in the reference frame.

[0078] Specifically, the main function of the intermediate frame is to replace the corresponding non-target frame, so the spatial pose of the intermediate frame can be the same as the corresponding non-target frame. In this embodiment, an intermediate frame with the same spatial pose as the reference frame is generated, and each pixel of the intermediate frame corresponds to each pixel of the reference frame and has the same spatial coordinates. The matching points of each pixel of the reference frame in the first target frame or the second target frame can be determined respectively, that is, for any pixel of the reference frame, the pixel with the highest similarity is found in the first target frame or the second target frame. Since each pixel of the intermediate frame corresponds to each pixel of the reference frame one by one, the matching points of each pixel of the intermediate frame in the first target frame or the second target frame are determined through the above steps. Therefore, each pixel of the intermediate frame can determine its own pixel value according to the pixel value of its own matching point. For example, the pixel value of the matching point in the target frame is directly determined as the pixel value of the corresponding pixel in the intermediate frame. Alternatively, the pixel value of the matching point in the target frame is determined as the pixel value of the corresponding pixel in the intermediate frame after being adjusted by a preset ratio.

[0079] In this embodiment, on the basis of keeping the spatial pose of the intermediate frame the same as that of the reference frame, the pixel value of each pixel in the intermediate frame is further determined through the first target frame and the second target frame, so that the intermediate frame can better connect the two target frames in terms of both spatial pose and pixel value, making the final endoscopic video content more smooth and coherent.

[0080] Furthermore, the feature points in the video frame may be used to determine the matching points of each pixel of the reference frame in the first target frame or the second target frame.

[0081] Therefore, step S320, determining the matching point of each pixel point in the reference frame in the first target frame or the second target frame, includes: steps S321 to S324.

[0082] Step S321, identifying feature points of the first target frame, the second target frame and the reference frame.

[0083] Step S322, matching feature points between the first target frame and the reference frame to obtain a first matching point pair.

[0084] Step S323, matching feature points between the second target frame and the reference frame to obtain a second matching point pair.

[0085] Step S324: determining matching points of each pixel point in the reference frame in the first target frame or the second target frame according to the first matching point pair and the second matching point pair.

[0086] The first step is to identify the feature points in each frame. The feature points can be SIFT (Scale-invariant feature transform) feature points. Then, feature point matching is performed between the reference frame and the target frame. For any feature point in the reference frame, the feature point with the highest similarity is determined among all feature points in the first target frame and the second target frame. Among them, the similarity can be determined based on the feature descriptor of the feature point. The closer the feature descriptors of the two feature points are, the more similar they are. After the feature point matching is completed, the first matching point pair and the second matching point pair will be obtained. At this point, the matching points of each feature point in the reference frame in the first target frame or the second target frame have been determined, and then the matching relationship of other pixel points in the reference frame can be determined through the matching relationship of each feature point in the reference frame. Exemplarily, the pixel points around a feature point can be matched with the pixel points around the matching point of the feature point.

[0087] In this embodiment, the feature point is a pixel point with a high degree of recognition in the image, and the matching relationship of the feature point is usually relatively accurate. By using feature point matching, the matching points of each pixel point in the intermediate frame in the two target frames can be determined more accurately, thereby more accurately determining the pixel value of each pixel point, so that the intermediate frame has a better connection effect with the two target frames.

[0088] As follows, a specific means of determining the matching relationship of other pixel points through the matching relationship of feature points is provided.

[0089] Specifically, through the above steps, it can be known that the first matching point pair includes the first feature point in the reference frame and its matching point in the first target frame, and the second matching point pair includes the second feature point in the reference frame and its matching point in the second target frame. Among them, the first feature point in the reference frame refers to the feature point of its matching point in the first target frame, and the second feature point in the reference frame refers to the feature point of its matching point in the second target frame.

[0090] Step S324, determining the matching points of each pixel point in the reference frame in the first target frame or the second target frame according to the first matching point pair and the second matching point pair, includes: steps S410 to S440.

[0091] Step S410, determining a first pose relationship between the first target frame and the reference frame, and determining a second pose relationship between the second target frame and the reference frame.

[0092] Step S420: for the pixel points that have not been matched in the reference frame, determine a first distance between the pixel point and the first feature point, and determine a second distance between the pixel point and the second feature point.

[0093] Step S430: for pixel points whose first spacing is smaller than the second spacing, a matching point of the pixel point in the first target frame is determined according to the first pose relationship.

[0094] Step S440: For pixel points whose second spacing is smaller than the first spacing, a matching point of the pixel point in the second target frame is determined according to the second posture relationship.

[0095] After determining the matching points of each feature point in the reference frame, it is also necessary to determine the first pose relationship between the first target frame and the reference frame, and determine the second pose relationship between the second target frame and the reference frame. The pose relationship represents the coordinate transformation relationship between the corresponding points of the two frames, which can be expressed by the pose matrix.

[0096] For any pixel point in the reference frame that has not been matched, first determine whether the matched feature point closest to the pixel point is the first feature point or the second feature point. If the pixel point is closer to the first feature point, then determine that the matching point of the pixel point is in the first target frame, and determine the matching point of the pixel point in the first target frame through the first pose relationship; if the pixel point is closer to the second feature point, then determine that the matching point of the pixel point is in the second target frame, and determine the matching point of the pixel point in the second target frame through the second pose relationship.

[0097] Through the above matching method, the matching points of each pixel in the reference frame in the two target frames can be accurately determined, so as to more accurately determine the pixel value of each pixel, so that the intermediate frame has a better connection effect with the two target frames.

[0098] In one embodiment, step S410, determining a first pose relationship between a first target frame and a reference frame, and determining a second pose relationship between a second target frame and a reference frame, includes: steps S411 to S413.

[0099] Step S411, matching feature points between the first target frame and the second target frame to obtain a target matching point pair.

[0100] Step S412, determining the target posture relationship between the first target frame and the second target frame according to the target matching point pair.

[0101] Step S413, determining a first posture relationship and a second posture relationship according to the target posture relationship.

[0102] In this embodiment, a specific means for determining the first pose relationship and the second pose relationship is provided. First, the feature points in the first target frame and the second target frame are used to obtain several groups of target matching point pairs. Furthermore, the target pose relationship between the first target frame and the second target frame can be determined based on the pose conversion relationship between the two feature points in the target matching point pair, and the target pose relationship is represented by a relative pose matrix. Finally, the relative pose matrix can be converted into a quaternion and a position vector, and the first pose relationship and the second pose relationship can be determined by an interpolation algorithm. Specifically, the interpolation position is mainly determined based on the original position of the reference frame between the first target frame and the second target frame.

[0103] Through the technical means in this embodiment, the first pose relationship and the second pose relationship can be determined more accurately, so that the matching points of each pixel point in the reference frame in the two target frames can be determined more accurately.

[0104] In addition, in another embodiment, the first pose relationship may be directly determined by the first matching point pair, and the second pose relationship may be determined by the second matching point pair.

[0105] Furthermore, the clarity of the reference frame will directly affect the recognition and matching of its feature points. The clearer the reference frame, the more accurate its feature point recognition and the more accurate the feature point matching between it and the target frame.

[0106] Therefore, in some of the embodiments, in step S210, non-target frames are determined in the initial endoscopic video, including: step S211, performing frame classification on the initial endoscopic video to determine the non-target frames and the confidence of each non-target frame.

[0107] Step S231, determining a reference frame from the removed non-target frames between the first target frame and the second target frame, includes: determining a reference frame from the removed non-target frames between the first target frame and the second target frame according to the confidence of each non-target frame.

[0108] Specifically, firstly, each video frame of the initial endoscopic video is classified to determine the non-target frames that need to be removed and the confidence of the non-target frames. At this time, other video frames can be determined as target frames. The confidence of the non-target frame can also be called the classification probability, which refers to the probability that the corresponding video frame is a non-target frame. Exemplarily, a binary classification network model can be used to classify the video frames of the initial endoscopic video. While obtaining the classification results of each video frame through the network model, the classification confidence of each video frame can also be obtained.

[0109] Since the clarity of the video frame is one of the factors for classification, the probability that a clear video frame is a target frame is higher, and the probability that a blurred video frame is a non-target frame is higher. Therefore, for a non-target frame, its confidence is higher, indicating that the clarity of the frame is relatively low, and the lower its confidence, the higher the clarity of the frame. Therefore, among the non-target frames that have been removed, the non-target frames with lower confidence can be selected as reference frames, thereby obtaining reference frames with relatively higher clarity.

[0110] Therefore, in this embodiment, a non-target frame with higher definition is selected as a reference frame. Compared with selecting a non-target frame with lower definition as a reference frame, the feature point recognition and matching of the reference frame with higher definition is more accurate. Therefore, the feature point matching relationship between the reference frame and the two target frames can be established more accurately.

[0111] In some of the embodiments, in step S210, non-target frames are determined in the initial endoscopic video, including: step S212 and step S213.

[0112] Step S212, classifying the frames of the initial endoscopic video, and determining information frames and non-information frames in the initial endoscopic video.

[0113] Step S213, determining the continuous non-information frames as non-target frames; or determining the continuous information frames as target frames and determining the remaining frames as non-target frames.

[0114] In this embodiment, a binary classification network model or other image classification algorithm is first used to classify each video frame in the initial endoscopic video. The classification function is mainly to screen out information frames and non-information frames. As described above, non-information frames refer to video frames that do not contain valid information and are video frames that need to be removed during the endoscopic video processing. However, considering that the classification results are not completely accurate, some video frames containing valid information may be misjudged as non-information frames, and some video frames that do not contain valid information may be misjudged as information frames.

[0115] In this embodiment, in order to more accurately remove video frames that do not contain valid information, the non-information frames identified by classification are not directly removed. By analyzing the endoscopic video, it is found that video frames that do not contain valid information usually appear continuously, and video frames that contain valid information usually appear continuously. Using the above rules, non-target frames that need to be removed can be determined based on the classification results.

[0116] In one approach, consecutive non-information frames are determined as non-target frames, thereby obtaining non-target segments. In endoscopic video processing, non-target segments are removed while the remaining video parts are retained. This approach can be used when it is necessary to retain as much video data as possible while requiring lower video data quality.

[0117] In another method, continuous information frames are determined as target frames to obtain target segments, and the remaining video frames in the endoscopic video are determined as non-target frames. In endoscopic video processing, non-target frames are removed and target segments are retained. This method can be used when the video data quality requirements are high.

[0118] Furthermore, a continuous frame number threshold may be set, and non-target frames may be determined based on the continuous frame number threshold. Exemplarily, non-information frames that appear continuously and have a continuous frame number greater than 5 frames are determined as non-target frames.

[0119] It should be noted that the above two processing methods can improve the accuracy of removing video frames that do not contain valid information. At the same time, for endoscopic videos in which non-information frames and information frames frequently appear alternately in the classification results, the above video processing method does not remove individual non-information frames, thereby ensuring the continuity of such endoscopic videos.

[0120] The technical solution of the present invention is described in detail below through a specific embodiment.

[0121] Figure 3 FIG. 3 is a flow chart of an endoscopic video processing method in a specific embodiment of the present invention. Referring to FIG. 3, the process of the method is as follows:

[0122] Step S510, training a binary classification network model.

[0123] First, data division is performed. The historical endoscopic video data of each patient is split to obtain a series of sequence frames, that is, video frames arranged in time sequence. And the series of sequence frame data is divided into a test set and a training set according to the patient. In order to ensure rigor, the sequence frame data of the endoscopic video of the same patient will not appear in both the training set and the test set.

[0124] Then, the binary classification model is trained. The sequence frames of the training set are classified into information frames and non-information frames. Among them, non-information frames include video frames with loss of effective information due to problems such as the patient's body outside the body, endoscope examination, highlight / liquid reflection, contrast problems, motion / out-of-focus blur, bubbles, effusion and excessive feces. It should be noted that in different scenarios, non-information frames can be determined according to different standards. Then the labeled sequence frame data is input into the neural network model for training until the model converges.

[0125] Finally, the binary classification network model is tested. The video frames of the test set are input into the binary classification network model in sequence, and the binary classification network model returns 0 or 1. Among them, 0 represents a non-information frame and 1 represents an information frame. A series of 0 or 1 is stored, and each endoscopic video corresponds to a set of 0 / 1 sequences.

[0126] Step S520, performing frame classification on the initial endoscopic video by training a complete binary classification network model.

[0127] Specifically, the initial endoscopic video to be frame classified is input into a well-trained binary classification network model. The binary classification network model outputs the frame classification results of the initial endoscopic video, and the classification results are a set of 0 / 1 sequences.

[0128] Step S530, determining the information segment and the non-information segment of the endoscopic video.

[0129] In one case, the 0 / 1 sequence obtained by the binary classification network model has very frequent transitions from 0 to 1 or from 1 to 0. If all non-information frames are simply removed, the video will become very incoherent. At the same time, the binary classification network model may predict false negatives, such as mistaking information frames for non-information frames, which will mistakenly remove some video frames containing valid information. Therefore, an endoscopic video post-processing method is used.

[0130] First, the continuity of non-information frames is analyzed. When non-information frames appear continuously and the number of frames that appear continuously is greater than a given threshold (e.g., 5 frames, 10 frames, etc.), the continuous non-information frames are determined as non-information segments (non-target segments). All non-information segments are extracted, and the rest are information segments (target segments).

[0131] Secondly, the analysis of the continuity of information segments is basically the same as the analysis of the continuity of non-information segments. The difference is that the information frames are directly selected based on the continuity. That is, when the continuous appearance of information frames reaches a certain threshold, the continuous information frames are determined as information segments (target segments), and the other parts are determined as non-information segments (non-target segments). Finally, the non-information segments are removed. This method can avoid false positives and ensure the quality of data.

[0132] When it is necessary to retain as much data as possible and the data quality requirement is not high, the processing method of the first aspect can be adopted. On the contrary, when the data quality requirement is high but the quantity requirement is not high, the processing method of the second aspect can be adopted.

[0133] Step S540 , splicing the information segments of the initial endoscopic video and performing frame insertion processing between adjacent information segments to obtain the target endoscopic video.

[0134] After obtaining all the information fragments, if the information fragments are simply connected, the video will be incoherent at the connection. In order to solve this problem, a transition video is generated between the last frame of the i-th information fragment and the first frame of the i+1-th information fragment through a frame filling algorithm, so that the target endoscopy video after the connection is smoother. The last frame of the i-th information fragment is denoted as I s , the first frame of the i+1th information fragment is I f ,Generating transition videos can be achieved in the following three ways.

[0135] 1. Frame interpolation algorithm based on optical flow method.

[0136] First, the dense optical flow between the two frames is calculated, that is, the displacement of the pixels between the two frames. s With image I e The optical flow between is: V x (u,v),V y (u,v). Among them, u and v represent the pixel coordinate values ​​on the image, V x (u,v) and V y (u,v) represent the displacement of the pixel point (u,v) in the x direction and y direction respectively. s with I e If n frames are inserted between, the i-th frame image can be expressed as:

[0137]

[0138] 2. Frame interpolation algorithm based on deep learning.

[0139] The depth-aware video interpolation method (DAIN) can be used to interpolate frames between video frames. The sample endoscopic video is split into video frames, and sampling is performed at intervals of n frames (for example, n=5). s With the s+nth frame I s+n As the input of model training, I s with I s+n All video frames between are used as criteria to train the model. The trained model is used to generate transition segments between informative video segments.

[0140] 3. Interpolation algorithm based on camera pose estimation.

[0141] First, the last frame of the Nth segment is recorded as I s and the first frame of the N+1th segment is denoted as I e , need I s with I e Generate intermediate frames I` m A reference frame is selected from the original non-information frame between the Nth segment and the second (N+1) segment, and the reference frame is denoted as I m In one case, reference frame I m It is for I s To I e The clearest video frame. m The classification probability (confidence) output by the above two-classification network model can be determined. The smaller the probability of the output non-information frame is, the clearer the video frame is likely to be.

[0142] Then, SIFT feature points in three video frames are identified and s with I e , I s with I m , I m with I e Match feature points.

[0143] Secondly, using I s with I e The matching point pairs are used to estimate the camera position at I s Moment and I e The relative pose matrix H at the moment e s . The relative pose matrix H e s Convert to quaternion and position vector, and then get I by interpolation m At this moment relative to I s Moment and I e The relative pose matrix H at the moment m s With H m e .

[0144] Finally, the intermediate frame I` m The pixel value of the previous pixel is determined by the following two methods.

[0145] First, I s with I m , I m with I e A matching point pair is formed. At the pixel coordinates of the matching point, I` m The pixel points on the original image I are directly taken s or I eThe pixel values ​​on I are called anchor points. There are two types of anchor points: one is the pixel value taken from I s Denoted as p s , a pixel value is taken from I e Denoted as p e .

[0146] Second, other s or I e The pixel that forms the matching point is compared with the distance of the anchor point. If the pixel is at a distance p s Closer, then through the relative pose matrix H m s With video frame I s Get the pixel value of the point, otherwise through the relative pose matrix H m e With video frame I e Get its pixel value.

[0147] The present invention also provides an endoscopic video processing device, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be repeated hereafter. The terms "module", "unit", "subunit" and the like used below may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0148] Figure 4 is a structural block diagram of the endoscopic video processing device provided by the present invention, such as Figure 4 As shown, the device comprises:

[0149] The frame removal processing module 610 is used to determine non-target frames in the initial endoscopic video, remove the non-target frames, and obtain intermediate endoscopic video data; the non-target frames include non-information frames;

[0150] An intermediate processing module 620 is used to determine two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than a unit frame spacing;

[0151] The frame interpolation processing module 630 is used to generate an intermediate frame according to the first target frame and the second target frame, and insert the intermediate frame between the first target frame and the second target frame to obtain a target endoscopic video.

[0152] Through the above-mentioned endoscopic video processing device, non-target frames in the endoscopic video are first removed; then an intermediate frame is generated between two adjacent target frames whose frame spacing is greater than a preset threshold to achieve frame interpolation; finally, a target endoscopic video with non-target frames removed and containing intermediate frames is obtained. Among them, non-target frames include non-information frames. Therefore, compared with the prior art, the endoscopic video processing device provided by the present invention, after removing the non-information frames of the endoscopic video, also performs frame interpolation processing between two video frames with a larger frame spacing, so that the video content of the processed endoscopic video can be more coherent and smooth. Thereby, the problem in the related art that the endoscopic video after the non-information frame removal processing has incoherent video content and affected fluency is solved.

[0153] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0154] The present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0155] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0156] Optionally, in one embodiment, the processor may be configured to perform the following steps through a computer program:

[0157] S1, determining non-target frames in the initial endoscopic video, removing the non-target frames, and obtaining intermediate endoscopic video data; the non-target frames include non-information frames.

[0158] S2, determining two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than the unit frame spacing.

[0159] S3, generating an intermediate frame according to the first target frame and the second target frame, and inserting the intermediate frame between the first target frame and the second target frame to obtain a target endoscopic video.

[0160] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0161] In addition, in combination with the endoscopic video processing method provided by the present invention, a storage medium can also be provided in the present invention to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the endoscopic video processing methods in the above embodiments is implemented.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0163] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0164] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0165] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0166] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An endoscopic video processing method, characterized in that: The processing method comprises: Determine non-target frames in the initial endoscopic video, remove the non-target frames, and obtain intermediate endoscopic video data; the non-target frames include non-information frames; Determine two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than a unit frame spacing; An intermediate frame is generated according to the first target frame and the second target frame, and the intermediate frame is inserted between the first target frame and the second target frame to obtain a target endoscopic video.

2. The endoscopic video processing method according to claim 1, characterized in that: Generating an intermediate frame according to the first target frame and the second target frame comprises: determining a reference frame among the removed non-target frames between the first target frame and the second target frame; The intermediate frame is generated according to the first target frame, the second target frame and the reference frame.

3. The endoscopic video processing method according to claim 2, characterized in that: Each pixel in the intermediate frame corresponds to each pixel in the reference frame one by one; The step of generating the intermediate frame according to the first target frame, the second target frame and the reference frame comprises: Respectively determining matching points of each pixel in the reference frame in the first target frame or the second target frame; The pixel value of the corresponding pixel point in the intermediate frame is determined according to the pixel value of the matching point of each pixel point in the reference frame.

4. The endoscopic video processing method according to claim 3, characterized in that: The determining of matching points of each pixel in the reference frame in the first target frame or the second target frame includes: identifying feature points of the first target frame, the second target frame, and the reference frame; Matching feature points between the first target frame and the reference frame to obtain a first matching point pair; Matching feature points between the second target frame and the reference frame to obtain a second matching point pair; According to the first matching point pair and the second matching point pair, matching points of each pixel point in the reference frame in the first target frame or the second target frame are determined respectively.

5. The endoscopic video processing method according to claim 4, characterized in that: The first matching point pair includes a first feature point in the reference frame and a matching point thereof in the first target frame, and the second matching point pair includes a second feature point in the reference frame and a matching point thereof in the second target frame; The step of determining, according to the first matching point pair and the second matching point pair, matching points of each pixel point in the reference frame in the first target frame or the second target frame, respectively, includes: Determining a first pose relationship between the first target frame and the reference frame, and determining a second pose relationship between the second target frame and the reference frame; For a pixel point that has not been matched in the reference frame, determining a first distance between the pixel point and the first feature point, and determining a second distance between the pixel point and the second feature point; For the pixel points whose first spacing is smaller than the second spacing, determining matching points of the pixel points in the first target frame according to the first posture relationship; For the pixel points whose second spacing is smaller than the first spacing, the matching points of the pixel points in the second target frame are determined according to the second posture relationship.

6. The endoscopic video processing method according to claim 5, characterized in that: The determining of a first pose relationship between the first target frame and the reference frame, and the determining of a second pose relationship between the second target frame and the reference frame, comprises: Matching feature points between the first target frame and the second target frame to obtain a target matching point pair; Determining a target pose relationship between the first target frame and the second target frame according to the target matching point pair; The first posture relationship and the second posture relationship are determined according to the target posture relationship.

7. The endoscopic video processing method according to claim 2, characterized in that: The step of determining a non-target frame in the initial endoscopic video comprises: Performing frame classification on the initial endoscopic video to determine the non-target frames and the confidence of each of the non-target frames; The step of determining a reference frame from the removed non-target frames between the first target frame and the second target frame comprises: Among the non-target frames that have been removed between the first target frame and the second target frame, a reference frame is determined according to the confidence of each of the non-target frames.

8. The endoscopic video processing method according to claim 1, characterized in that: The step of determining a non-target frame in the initial endoscopic video comprises: Performing frame classification on the initial endoscopic video, and determining information frames and the non-information frames in the initial endoscopic video; The consecutive non-information frames are determined as the non-target frames; or, the consecutive information frames are determined as the target frames, and the remaining frames are determined as the non-target frames.

9. An endoscopic video processing device, characterized in that: The processing device comprises: A frame removal processing module, used to determine non-target frames in the initial endoscopic video, remove the non-target frames, and obtain intermediate endoscopic video data; the non-target frames include non-information frames; An intermediate processing module, used for determining two adjacent video frames in the intermediate endoscopic video data whose frame spacing is greater than a preset threshold as a first target frame and a second target frame; wherein the preset threshold is greater than a unit frame spacing; The frame interpolation processing module is used to generate an intermediate frame according to the first target frame and the second target frame, and insert the intermediate frame between the first target frame and the second target frame to obtain a target endoscopic video.

10. An electronic device, characterized in that: The device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the endoscopic video processing method according to any one of claims 1 to 8 when executing the computer program.

Images