Hepatolithiasis operation video processing method

Abstract

The invention relates to the technical field of image processing, and discloses a hepatobiliary calculus operation video processing method, which comprises the following steps: acquiring an operation video of a hepatobiliary calculus operation; extracting images in the operation video by adopting a first time window to obtain a first operation image sequence; determining a plurality of image segmentation rules associated with the operation image sequence by using the operation images in the first operation image sequence, wherein the pre-trained calculus model and the plurality of image segmentation rules are used for identifying consistent calculus images in adjacent operation images in the first operation image sequence; the time range is obtained based on video time corresponding to the operation images in the first operation image sequence. According to the technical scheme, the operation efficiency and accuracy are improved, and the auxiliary operation capability of doctors is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for processing a video of a hepatobiliary stone operation. Background Art

[0002] In hepatobiliary stone surgery, doctors usually use endoscopes or other imaging devices to observe the patient's hepatobiliary area in real time and determine the location of the stones through video images. However, the morphology of stones is complex and they are often hidden in different tissues of the liver and gallbladder, especially small or partially embedded stones, which are difficult to be identified by the naked eye during the operation. Existing surgical video analysis methods mainly rely on the doctor's personal experience and manual marking, which are easily affected by subjective factors, resulting in missed judgments and misjudgments. In addition, the limited field of view during the operation, the occlusion of imaging equipment and individual differences among patients may further increase the difficulty of the operation. These problems increase the workload of doctors, prolong the operation time, and increase the risk of surgery. Therefore, how to accurately and real-time identify the location of stones during the operation and assist doctors in the operation has become a key technical requirement for improving the success rate of hepatobiliary stone surgery.

[0003] At present, there are some imaging technology systems for assisting surgery on the market, such as surgical navigation systems based on image recognition, which can provide reference for doctors through the analysis of single-frame images. However, the existing technology still has great limitations in the processing of continuous surgical videos, especially when dealing with complex dynamic scenes, it is difficult to accurately detect and mark stones in real time. Most of the existing auxiliary systems can only identify specific types of tissue structures in specific scenes, and cannot combine multiple frames of images for overall analysis. In addition, the existing system has limited ability to recognize stones at different resolutions and image sizes, and cannot effectively deal with the image size differences caused by the perspective and equipment changes during surgery. Therefore, the deficiencies of the existing technology in real-time, multi-scale recognition and automated processing directly affect its application effect in hepatobiliary stone surgery. These technical bottlenecks urgently need to be solved through smarter and more sophisticated automated processing methods. Summary of the invention

[0004] At least one aspect or advantage of the invention will be set forth in part in the following description, or may be obvious from the description, or may be learned by practice of the disclosed subject matter.

[0005] According to a first aspect of the present invention, a method for processing a video of a hepatobiliary stone surgery comprises:

[0006] Get surgical videos of hepatobiliary stone surgery;

[0007] Extracting images from the surgical video using a first time window to obtain a first surgical image sequence;

[0008] Using the surgical images in the first surgical image sequence, a plurality of image segmentation rules associated with the surgical image sequence are determined, and the pre-trained stone model and the plurality of image segmentation rules are used to identify consistent stone images in adjacent surgical images in the first surgical image sequence;

[0009] The time range is obtained based on the video time corresponding to the surgical images in the first surgical image sequence.

[0010] According to one embodiment of the present invention, the process of determining the image segmentation rule includes:

[0011] Performing image segmentation and stone image recognition on the first surgical image sequence to obtain image segmentation rules corresponding to images in the first surgical image sequence;

[0012] The image segmentation rules corresponding to the images in the first surgical image sequence are merged to obtain a plurality of image segmentation rules associated with the first surgical image sequence.

[0013] According to one embodiment of the present invention, the process of obtaining the image segmentation rule corresponding to the surgical image includes:

[0014] Segmenting the surgical image according to a plurality of preset sizes to obtain segmented images corresponding to each size;

[0015] Resizing the segmented image to obtain an adjusted image, wherein the adjusted image has a consistent resolution;

[0016] The adjusted image is recognized using a pre-trained stone model to obtain stone area images and non-stone area images corresponding to each size;

[0017] All stone region images are superimposed based on the image positions to obtain the total area of ​​the stone image region;

[0018] Superimposing the stone region images corresponding to each size based on the image position to obtain the stone region image area corresponding to each size;

[0019] Determine the stone recognition rate corresponding to each size according to the image area of ​​the stone region corresponding to each size and the total area of ​​the stone image region;

[0020] Determine an image segmentation rule corresponding to the surgical image, wherein the image segmentation rule includes a time range and two or more image segmentation sizes, wherein the stone recognition rate corresponding to the image segmentation size is not lower than the mean of the stone recognition rate, and the time range is determined according to the video time corresponding to the surgical image and the first time window;

[0021] The image position is the position of the segmented image in the surgical image.

[0022] According to one embodiment of the present invention, merging image segmentation rules corresponding to images in the first surgical image sequence includes:

[0023] Acquire a first surgical image and a second surgical image that are adjacent to each other in a first surgical image sequence;

[0024] Obtaining a consistent image segmentation size in the image segmentation rules corresponding to the first surgical image and the second surgical image, setting the end point of the time range of the image segmentation rule corresponding to the first surgical image as the end point of the time range of the image segmentation rule corresponding to the second surgical image, and using the image segmentation rule corresponding to the first surgical image as the image segmentation rule corresponding to the second surgical image;

[0025] The image segmentation dimensions in the first image that are not included in the corresponding image segmentation rule in the second surgical image are deleted.

[0026] According to one embodiment of the present invention, a second time window is used to extract images in the surgical video to obtain a second surgical image sequence;

[0027] Identify images in the second surgical image sequence based on image segmentation rules to obtain location information of stone images in the surgical images in the second surgical image sequence;

[0028] Image markers are drawn in the surgical video based on the location information of the stone image.

[0029] According to one embodiment of the present invention, the stone image regions are merged based on the location information of the stone image, and markers are drawn based on the merged stone image regions, and the drawn markers are associated with the image segmentation size within the image segmentation rule.

[0030] According to one embodiment of the present invention, the image mark drawn in the surgical video corresponds to the first image segmentation size, the first image segmentation size is an image segmentation rule that matches the surgical video time, and the image segmentation size has a maximum unexhausted time window, and the unexhausted time window is the duration of the surgical video time and the end point of the image segmentation size time range.

[0031] According to one embodiment of the present invention, identifying images in the second surgical image sequence based on image segmentation rules includes:

[0032] Determine a first image segmentation rule including surgical video time corresponding to an image in a second surgical image sequence;

[0033] Segmenting the images in the second surgical image sequence based on the image segmentation size included in the first image segmentation rule to obtain a second segmented image;

[0034] resizing the segmented second image to obtain an adjusted second segmented image;

[0035] The adjusted second segmented image is recognized using the pre-trained stone model to obtain the location information of the stone image.

[0036] According to a second aspect of the present invention, an electronic device includes a processor and a memory; the memory is used to store a program; the processor executes the program to implement a method for processing a video of a hepatobiliary stone surgery.

[0037] According to a third aspect of the present invention, a computer-readable storage medium stores a program, and the program is executed by a processor to implement a method for processing a video of a hepatobiliary stone surgery.

[0038] Compared with the prior art, the technical solution of the present invention has the following technical effects:

[0039] Through the pre-trained stone model and image segmentation rules, stones can be automatically and accurately identified at different resolutions and image sizes, and the recognition rate can be calculated, thereby realizing multi-level and multi-angle stone detection.

[0040] The present invention helps doctors understand the specific location of stones in real time by drawing marks on the locations of stones in surgical videos, assists surgical operations, and reduces the risks of missed diagnosis and misdiagnosis.

[0041] Through the automatic merging of image segmentation, resizing, and recognition rules, manual intervention in the surgical video analysis process is reduced, and surgical efficiency and consistency of stone detection are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a method for processing video of hepatobiliary stone surgery provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, rather than implying any limitation on the scope of the present disclosure.

[0044] As used herein, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "based at least in part on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment". The term "another embodiment" is to be interpreted as "at least one other embodiment". The orientation or position relationship indicated by the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "vertical", "horizontal", "lateral", "longitudinal" and the like is based on the orientation or position relationship shown in the accompanying drawings. These terms are mainly for better describing the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation. In addition, in addition to being used to indicate an orientation or position relationship, some of the above terms may also be used to indicate other meanings, such as the term "upper" may also be used to indicate a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to the specific circumstances. In addition, the terms "install", "set", "provided with", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be an internal connection between two devices, elements or components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. In addition, the terms "first", "second", etc. are mainly used to distinguish different devices, elements or components (the specific types and structures may be the same or different), and are not used to indicate or imply the relative importance and quantity of the indicated devices, elements or components. Unless otherwise specified, "plurality" means two or more.

[0045] During hepatobiliary stone surgery, doctors observe the hepatobiliary area through an endoscope or other imaging devices and manually identify the location of stones. However, stones are complex in shape and may be obscured by surrounding tissues. Relying on the doctor's experience for identification can easily lead to missed or misjudgment, increasing surgical risks. Although existing surgical assistance systems can provide single-frame image analysis, they are limited in processing continuous video and marking stones in real time, and are difficult to cope with image size differences caused by changes in viewing angles and equipment interference during surgery. Therefore, there is an urgent need for an automated, high-precision surgical video processing method to improve the accuracy of stone identification and surgical efficiency.

[0046] According to one embodiment of the present invention, referring to Figure 1 As shown, a method for processing a video of a hepatobiliary stone surgery is provided, comprising:

[0047] Get surgical videos of hepatobiliary stone surgery.

[0048] Specifically, the system first obtains the hepatobiliary stone surgery video generated during the operation. This video is usually shot by imaging equipment such as an endoscope and contains real-time image information of the liver and gallbladder area. By obtaining the surgical video, it can provide basic data for subsequent image processing, and this real-time video contains rich surgical scene information, which can provide more stone image clues for automated processing and improve subsequent recognition accuracy.

[0049] The images in the surgical video are extracted using the first time window to obtain a first surgical image sequence.

[0050] Specifically, the system extracts continuous image frames from the surgical video according to the set first time window to form a first surgical image sequence. The length of this time window and the frequency of extraction can be adjusted according to the complexity of the surgical scene to ensure that sufficiently continuous images are extracted for subsequent processing. Using a time window to extract image sequences can reduce unnecessary video frames and reduce the burden of data processing while maintaining the continuity of key images. This design can effectively balance the efficiency of image processing and recognition accuracy.

[0051] Several image segmentation rules associated with the surgical image sequence are determined using surgical images in the first surgical image sequence, and the pre-trained stone model and the several image segmentation rules are used to identify consistent stone images in adjacent surgical images in the first surgical image sequence.

[0052] Specifically, after acquiring the first surgical image sequence, the system will determine several image segmentation rules associated with the sequence according to the image features of the sequence, including the segmentation size and segmentation method of the surgical image and the stone distribution characteristics in different areas of the image.

[0053] Image segmentation rules are the core of surgical video image analysis. By setting corresponding segmentation sizes for different surgical images, the present invention can ensure accurate identification of stones of various sizes and shapes. The segmentation rules can effectively improve the segmentation effect of stones in complex backgrounds, especially when stones are similar to other tissue structures, they can still be effectively distinguished by segmentation rules.

[0054] The system will call the pre-trained stone model to analyze the surgical images in the first surgical image sequence and use image segmentation rules to find consistent stone images in adjacent surgical images. The pre-trained stone model can identify the stone area based on specific texture, color and other features in the image.

[0055] The pre-trained stone model can automatically identify the characteristics of stones through machine learning technology, getting rid of the errors of manual judgment, and by identifying consistent stone images in adjacent images, it ensures the continuity and consistency of stone identification. This design greatly improves the accuracy of stone identification and avoids misjudgment or missed judgment caused by single-frame image processing.

[0056] The time range is obtained based on the video time corresponding to the surgical images in the first surgical image sequence.

[0057] Specifically, the system determines the video time corresponding to each surgical image by calculating the time range of the surgical image in the video. This time range is used for subsequent image marking and real-time display to ensure that the marking results are synchronized with the surgical process.

[0058] The order of image processing and corresponding markings are determined based on the time range, which can ensure that the system gives accurate stone location prompts in real time during the operation, thereby enhancing the doctor's operating efficiency. The design of the present invention can achieve real-time monitoring and feedback, making the surgical assistance system more intelligent and practical.

[0059] According to one embodiment of the present invention, the process of determining the image segmentation rule includes:

[0060] Image segmentation and stone image recognition are performed on the first surgical image sequence to obtain image segmentation rules corresponding to the images in the first surgical image sequence.

[0061] Specifically, the system first performs image segmentation on each surgical image in the first surgical image sequence. The image segmentation method can divide each image according to a preset segmentation size to obtain a more detailed image area. Then, the pre-trained stone model is used to identify each segmented image area to determine the specific location and distribution of stones in these images. Based on the stone recognition results, the system can generate corresponding image segmentation rules for each surgical image, including segmentation size, stone location, recognition rate and other information.

[0062] Through image segmentation, complex surgical images can be decomposed into smaller areas that are easier to process, thereby improving the accuracy of stone recognition. The stone recognition model can quickly locate the stone area based on the trained stone features and reduce human errors. The generated image segmentation rules provide an important basis for subsequent processing, ensuring that the system has a high recognition ability for images of different resolutions and sizes during the processing process.

[0063] The image segmentation rules corresponding to the images in the first surgical image sequence are merged to obtain a plurality of image segmentation rules associated with the first surgical image sequence.

[0064] Specifically, after obtaining the segmentation rules for each image, the system merges these rules. The merging process includes comparing the segmentation results of adjacent surgical images, retaining the image segmentation rules with higher recognition rates at the same size, and eliminating noise or irrelevant image area information. Through the merged rules, the system can determine several general segmentation rules applicable to the entire surgical image sequence. These segmentation rules will play a core role in subsequent surgical image processing and stone recognition.

[0065] The benefit of merging segmentation rules is that it reduces redundant and unnecessary information, unifies the segmentation rules of multiple images into a few universally applicable rules, and thus improves processing efficiency. Comparison and merging of adjacent images can also ensure the continuity and stability of stone recognition, avoiding recognition inconsistency caused by misjudgment of individual frames. This design can optimize the real-time and efficiency of the entire surgical process while ensuring recognition accuracy.

[0066] According to one embodiment of the present invention, the process of obtaining the image segmentation rule corresponding to the surgical image includes:

[0067] The surgical image is segmented according to a plurality of preset sizes to obtain segmented images corresponding to each size.

[0068] Specifically, the system first segments the surgical image according to multiple preset sizes. The segmentation of different sizes is to ensure that the stone area information at different scales is captured, which can deal with both small stones and larger stone areas.

[0069] Multi-size segmentation can adapt to different sizes and shapes of stones, ensuring that even smaller stones can be fully captured and analyzed. This design can improve the system's detection accuracy for stones of different sizes and avoid missed detection or inaccurate identification.

[0070] The segmented image is resized to obtain an adjusted image, wherein the adjusted image has a consistent resolution.

[0071] Specifically, after the segmentation is completed, the system resizes the segmented images of different sizes to have a uniform resolution. The uniform resolution ensures that the subsequent recognition and analysis steps will not be biased due to resolution differences.

[0072] The resizing provides a unified input format for subsequent recognition using the pre-trained stone model, avoiding the loss of recognition accuracy due to different image resolutions. By resizing all segmented images to the same resolution, the system can ensure consistency and stability during stone recognition.

[0073] The pre-trained stone model is used to identify the adjusted image to obtain stone area images and non-stone area images corresponding to each size.

[0074] Specifically, the adjusted images are input into the pre-trained stone model for analysis. The model will identify each segmented image and separate the stone area image and the non-stone area image.

[0075] The pre-trained stone model can automatically analyze the stone area and non-stone area in the surgical image, reducing manual intervention and improving recognition efficiency. The model's automatic recognition capability makes the entire surgical image processing faster and more accurate.

[0076] All stone region images are superimposed based on the image position to obtain the total area of ​​the stone image region.

[0077] Specifically, the system then superimposes all stone area images, merges all identified stone areas based on the image positions, and calculates the total area of ​​the stone image area.

[0078] By superimposing all the stone area images, the system can combine the recognition results of different sizes to obtain more complete stone area information. This design avoids the possibility of missed detection under a single size and ensures comprehensive stone detection.

[0079] The stone region images corresponding to each size are superimposed based on the image position to obtain the stone region image area corresponding to each size.

[0080] Specifically, the stone region images at each size are superimposed to calculate the corresponding stone region area at each size. The superposition process at each size helps to determine the effectiveness of the size for stone identification.

[0081] By superimposing images of each size, the system can analyze the coverage and effectiveness of stone recognition at different sizes, ensuring that the system can accurately identify stones at multiple resolutions. This multi-level analysis design can significantly improve the stone recognition rate.

[0082] The stone recognition rate corresponding to each size is determined according to the image area of ​​the stone region corresponding to each size and the total area of ​​the stone image region.

[0083] Specifically, by comparing the area of ​​the stone region at each size with the total stone area, the system can calculate the stone recognition rate of each size. The recognition rate is used to measure the effectiveness of each segmented size in stone recognition.

[0084] The area-based recognition rate calculation method of the present invention provides a quantitative index to evaluate the contribution of different segmentation sizes to stone recognition. Through this design, the system can give priority to the size with a higher recognition rate, thereby optimizing the overall recognition effect.

[0085] Determine an image segmentation rule corresponding to the surgical image, wherein the image segmentation rule includes a time range and more than two image segmentation sizes, wherein the stone recognition rate corresponding to the image segmentation size is not lower than the mean stone recognition rate, and the time range is determined according to the video time corresponding to the surgical image and the first time window.

[0086] Specifically, the system finally selects two or more segmentation sizes with recognition rates higher than the mean based on the stone recognition rate of each size, and determines the final image segmentation rules in combination with the time range corresponding to the surgical image.

[0087] This method of the present invention can ensure that the image segmentation rules used in the end have a high recognition accuracy rate, while taking into account the temporal continuity of the surgical video. This optimized design enables the system to maintain a high recognition efficiency in different time windows, ensuring real-time recognition and feedback of stones during surgery.

[0088] The image position is the position of the segmented image in the surgical image.

[0089] Specifically, the position of each segmented image corresponds to the actual position in the surgical image, and this information is used for subsequent image labeling and visualization.

[0090] By retaining the position information after image segmentation, the system can accurately mark the actual location of the stone in the surgical video, providing doctors with real-time surgical assistance. This design improves the practicality and visualization of the system.

[0091] According to one embodiment of the present invention, merging image segmentation rules corresponding to images in the first surgical image sequence includes:

[0092] A first surgical image and a second surgical image adjacent to each other in a first surgical image sequence are acquired.

[0093] Specifically, the system first acquires adjacent images in the first surgical image sequence, namely, the first surgical image and the second surgical image. The two adjacent images are usually frames captured continuously in the surgical video, ensuring temporal and spatial consistency between the images.

[0094] By selecting adjacent images for analysis, the similarity of image content can be guaranteed to the greatest extent, especially when the instrument or tissue does not change much during surgery, the segmentation results of adjacent images are often similar, reducing unnecessary computational burden. This step ensures processing efficiency and lays the foundation for the subsequent rule merging.

[0095] Obtain the consistent image segmentation size in the image segmentation rules corresponding to the first surgical image and the second surgical image.

[0096] Specifically, the system extracts the image segmentation rules from the two surgical images and compares these rules to find out their consistency in segmentation size. Only the rules with the same segmentation size will be retained, and other segmentation sizes will be ignored.

[0097] By comparing and extracting consistent segmentation sizes in adjacent images, the consistency of image segmentation can be ensured, thereby improving the stability and accuracy of stone recognition. This design avoids interruptions or errors in stone recognition caused by inconsistent segmentation rules on different images, and improves the consistency and overall efficiency of image processing.

[0098] The end point of the time range of the first surgical image corresponding to the image segmentation rule is set as the end point of the time range of the second surgical image corresponding to the image segmentation rule.

[0099] Specifically, the system adjusts the time range of the first surgical image segmentation rule to the same time range as the second surgical image rule, that is, unifies the time end point, so that the segmentation rule of the first surgical image can be extended to the time period of the second surgical image.

[0100] The unified processing of the time range ensures that the segmentation rules of adjacent images are valid in the same time period. Especially in video sequence processing, time consistency is crucial for continuous processing, which enables the system to maintain the consistency of image segmentation rules when processing consecutive frames, thereby reducing image processing interruptions caused by time inconsistency.

[0101] The image segmentation rule corresponding to the first surgical image is used as the image segmentation rule corresponding to the second surgical image.

[0102] Specifically, after identifying the consistent segmentation size and time range in adjacent images, the system will directly use the segmentation rule of the first surgical image as the segmentation rule of the second surgical image, which can reduce repeated calculations and analyses.

[0103] By directly inheriting the segmentation rules of adjacent images, the system avoids repeated rule calculations for each image, which greatly improves processing speed and efficiency. In addition, using the same segmentation rules can also ensure consistency in processing between adjacent images, which helps to improve the continuity and reliability of stone identification results.

[0104] The image segmentation dimensions in the first image that are not included in the corresponding image segmentation rule in the second surgical image are deleted.

[0105] Specifically, in the last step, the system will delete those segmentation dimensions that exist in the first surgical image but do not appear in the segmentation rules of the second surgical image. This is to further optimize and streamline the rules so that only valid segmentation rules are retained during the entire processing process.

[0106] By removing useless or inconsistent segmentation sizes, the system can reduce unnecessary computational burden and improve the accuracy of recognition rules. Retaining segmentation rules that are only valid in adjacent images can reduce recognition errors and redundant processing, making image processing more efficient and accurate.

[0107] According to one embodiment of the present invention, a second time window is used to extract images in the surgical video to obtain a second surgical image sequence.

[0108] Specifically, the system extracts image frames from the surgical video according to the set second time window to form a second surgical image sequence. This time window can be dynamically adjusted according to the surgical process to ensure that the acquired image sequence has sufficient continuity and representativeness to facilitate subsequent stone identification.

[0109] Extracting images through time windows can reduce the processing of unnecessary frames, reduce the complexity and time overhead of data processing. At the same time, this method can also ensure that key frames are continuously captured during important surgical procedures, which helps to capture the complete information of stones, thereby improving the accuracy and efficiency of recognition.

[0110] The images in the second surgical image sequence are identified based on the image segmentation rule to obtain the position information of the stone images in the surgical images in the second surgical image sequence.

[0111] The specific extracted second surgical image sequence is processed by the defined image segmentation rules. The system uses the pre-trained stone model to identify the stone area in each frame of the image, and combines the size, resolution and other information in the image segmentation rules to determine the specific location of the stone image.

[0112] The combination of image segmentation rules and pre-trained stone models can automatically identify the stone area in each frame, reduce manual intervention, and improve the accuracy of image recognition. Through this design, the system can efficiently extract the location information of stones, ensure consistent and stable stone location recognition in consecutive frames, and facilitate subsequent real-time marking and display.

[0113] Image markers are drawn in the surgical video based on the location information of the stone image.

[0114] Specifically, after obtaining the stone location information, the system matches the location information with the image frames in the surgical video and draws image marks of the stone location in the video. The mark can be a visual box, dot or other highlight form to clearly indicate the location of the stone.

[0115] By drawing image markers in real time in the surgical video, the system provides doctors with an intuitive reference to help them accurately locate stones during surgery. This real-time marking design not only reduces the doctor's visual fatigue, but also improves surgical efficiency and avoids missing stones or improper operations. It makes surgical assistance more intelligent and can indicate the specific location of stones at any time in dynamic videos, further optimizing surgical results.

[0116] According to one embodiment of the present invention, the stone image regions are merged based on the location information of the stone image, and markers are drawn based on the merged stone image regions, and the drawn markers are associated with the image segmentation size within the image segmentation rule.

[0117] Specifically, in the surgical video, the system has obtained the location information of the stones in each frame of the surgical image. In order to improve the accuracy of recognition, the system merges the stone areas identified in different frames of images in space and time. The merging process is based on the stone area information in the same or similar positions in consecutive frames. The system superimposes and integrates these areas to form a complete and more accurate stone image area. Merging stone image areas can eliminate the noise and errors that may exist in the single-frame image recognition process, ensuring the accuracy and completeness of the stone position. In addition, through the superposition processing in time and space, the system can capture the full picture of the stone and avoid missing stones due to instrument movement or changes in viewing angle. This design improves the consistency and reliability of stone recognition results.

[0118] After merging the stone areas, the system marks the stones based on the merged areas. Marking can highlight the location of the stone on the surgical video, such as using a box, dot or other visual cue symbols, so that the specific location of the stone is clearly visible in the video. The position and size of the mark are determined based on the merged stone area to ensure that the doctor can accurately determine the location of the stone. The merged mark can provide a more intuitive and accurate stone location prompt. This design eliminates the instability that may be caused by single-frame marking and provides a continuous, dynamic marking display, which facilitates doctors to accurately and in real time locate and handle stones during surgery. This step improves the accuracy of the operation and reduces the workload of doctors.

[0119] In the present invention, the marking is not only based on the stone area, but also associated with the image segmentation size within the image segmentation rules. That is, the system ensures that the drawn marking meets the previously defined image segmentation size, so that the size and position of the marking are consistent with the segmentation rules. This association ensures the coordination and consistency of the marking and the image processing process. Associating the marking with the image segmentation rules can ensure that the size and accuracy of the marking meet the preset image processing standards. Through this design, the system can ensure that the marking information is consistent with the stone area under different segmentation dimensions, avoiding the situation where the marking does not match the actual stone area. This design further improves the accuracy of stone identification and display, making the marking in surgery more reliable.

[0120] According to one embodiment of the present invention, the image mark drawn in the surgical video corresponds to the first image segmentation size, the first image segmentation size is an image segmentation rule that matches the surgical video time, and the image segmentation size has a maximum unexhausted time window, and the unexhausted time window is the duration of the surgical video time and the end point of the image segmentation size time range.

[0121] Specifically, the system first obtains the current playback time from the surgical video, and finds the corresponding image segmentation size in the previously defined image segmentation rules based on this time. The image segmentation rules match the video time to ensure that the appropriate segmentation size is used for image processing in each time period. Matching the image segmentation size according to the video time ensures that the system uses the most appropriate segmentation rules when processing surgical videos, avoiding recognition errors caused by mismatched rules. This design accurately adjusts the image segmentation rules according to time, ensuring the accuracy of the recognition process.

[0122] Among multiple image segmentation rules that match the current video time, the system will select the segmentation size with the largest unexhausted time window. The unexhausted time window refers to the time difference between the current surgical video time and the end time of the image segmentation rule. The system gives priority to the segmentation size with the largest unexhausted time window to ensure that the segmentation size can be continuously used in the subsequent image processing. Selecting the segmentation size with the largest unexhausted time window can reduce the number of times the system frequently switches segmentation rules and improve the stability and consistency of system processing. This design allows the system to maintain the same segmentation size for a longer period of time, improving processing efficiency and consistency in stone identification.

[0123] The system processes the images in the surgical video and draws stone markers based on the selected image segmentation size. The size and position of the markers are determined based on the selected image segmentation size, ensuring that the markers are consistent with the segmented stone area and displayed in real time in the video. By drawing markers based on the image segmentation size, the system can ensure the accuracy and consistency of the stone position markings. Real-time marking not only provides doctors with an intuitive reference, but also reduces the possibility of misoperation. This design improves visualization and operation accuracy during surgery.

[0124] As the surgical video plays, the system automatically adjusts the segmentation rules based on the current unexhausted time window. When the time window is about to run out, the system matches the appropriate image segmentation rules based on the new time point and updates the size and position of the markers. This dynamic adjustment mechanism ensures that the system continues to provide accurate image processing and labeling throughout the operation, avoiding recognition interruptions or labeling errors due to time window exhaustion. By automatically updating segmentation rules and labels, the system can respond to changes during the operation in real time and maintain efficient and accurate stone recognition and labeling.

[0125] According to one embodiment of the present invention, identifying images in the second surgical image sequence based on image segmentation rules includes:

[0126] A first image segmentation rule including surgical video time corresponding to an image in the second surgical image sequence is determined.

[0127] Specifically, the system first extracts the current time period from the time information of the surgical video, and finds a suitable first image segmentation rule based on this time period. The segmentation rule includes segmentation sizes and rules related to the current surgical video time, which are used for subsequent image segmentation and processing.

[0128] By matching the image segmentation rules according to the surgical video time, the system can ensure that the segmentation rules used at each time point are the most suitable for the current surgical video progress. This design can improve the accuracy of image processing and ensure the consistency of image segmentation, avoiding errors caused by time mismatch.

[0129] The images in the second surgical image sequence are segmented based on the image segmentation size included in the first image segmentation rule to obtain second segmented images.

[0130] Specifically, after matching the appropriate image segmentation rule, the system segments the images in the second surgical image sequence using the segmentation size specified in the segmentation rule. In this way, the second surgical image segmented according to different sizes can be obtained, ensuring that the stone area of ​​each frame of the image can be accurately extracted.

[0131] Image segmentation is the basis of the entire image processing. By using the preset segmentation size, the system can segment complex images into easy-to-process parts. This segmentation method helps to improve the recognition accuracy of the stone area and reduce the interference of background noise on the recognition. At the same time, this step also ensures that the system can handle stones of different sizes and shapes.

[0132] The segmented second image is resized to obtain an adjusted second segmented image.

[0133] Specifically, in order to ensure the consistency of subsequent processing, the system resizes the second image after segmentation and adjusts all segmented images to a uniform resolution. This adjustment ensures that images can be consistently input into the model for processing at different segmentation sizes.

[0134] Resizing can unify the image input format and avoid recognition errors caused by different resolutions. This design helps improve the stability and consistency of image recognition, allowing the system to maintain efficiency and accuracy when processing images of different segmentation sizes.

[0135] The adjusted second segmented image is recognized using the pre-trained stone model to obtain the location information of the stone image.

[0136] Specifically, the adjusted image is input into the pre-trained stone model for processing. The model automatically identifies the stone area and outputs the specific location information of the stone image by learning and analyzing the image features.

[0137] The pre-trained stone model can accurately identify stones based on the characteristics of a large number of stone images, reducing the possibility of manual intervention. Through the automated recognition process, the system can quickly and accurately determine the location of the stone in each frame. This design step significantly improves the efficiency and accuracy of stone recognition, especially in complex surgical scenarios, and can greatly reduce the possibility of recognition errors.

[0138] According to one embodiment of the present invention, an electronic device includes a processor and a memory; the memory is used to store a program; the processor executes the program to implement a method for processing a hepatobiliary stone surgery video.

[0139] According to one embodiment of the present invention, a computer-readable storage medium stores a program, and the program is executed by a processor to implement a method for processing a video of a hepatobiliary stone surgery.

[0140] The beneficial effects of the present invention are:

[0141] The pre-trained stone model is combined with multi-scale image segmentation to ensure accurate identification of stones of different shapes and maintain identification consistency, reducing missed and misjudgments.

[0142] The system automatically detects stones, reduces manual intervention, and improves surgical efficiency and the doctor's operating accuracy.

[0143] The system draws real-time markers in the surgical video to help doctors accurately locate stones and enhance the safety and accuracy of the operation.

[0144] Time window image extraction and dynamic adjustment of segmentation rules optimize processing efficiency and ensure real-time and stability of recognition during surgery.

[0145] Those of ordinary skill in the art will appreciate that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0148] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0149] In addition, each functional module in the embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0150] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the energy-saving signal sending / receiving method of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0151] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

[0152] It should be understood that the size of the sequence number of each step in the content of the invention and the embodiments of the present invention does not absolutely mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. For the purpose of example and description, the foregoing description of the implementation of the present disclosure has been given. The foregoing description is not exhaustive and is not intended to limit the present disclosure to the exact form disclosed. Various deformations and modifications may exist according to the above teachings, or various deformations and modifications may be obtained from the practice of the present disclosure. These embodiments are selected and described to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can use the present disclosure in various embodiments and various modifications suitable for the specific purpose conceived.

Claims

1. A method for processing a video of a hepatobiliary stone surgery, characterized in that: include: Get surgical videos of hepatobiliary stone surgery; Extracting images from the surgical video using a first time window to obtain a first surgical image sequence; Using the surgical images in the first surgical image sequence, a plurality of image segmentation rules associated with the surgical image sequence are determined, and the pre-trained stone model and the plurality of image segmentation rules are used to identify consistent stone images in adjacent surgical images in the first surgical image sequence; The time range is obtained based on the video time corresponding to the surgical images in the first surgical image sequence.

2. A method for processing a video of a hepatobiliary stone surgery as claimed in claim 1, characterized in that: The process of determining the image segmentation rule includes: Performing image segmentation and stone image recognition on the surgical images in the first surgical image sequence, and acquiring image segmentation rules corresponding to the surgical images in the first surgical image sequence; The image segmentation rules corresponding to the surgical images in the first surgical image sequence are merged to obtain a plurality of image segmentation rules associated with the first surgical image sequence.

3. A method for processing a video of a hepatobiliary stone surgery as claimed in claim 2, characterized in that: The process of obtaining the image segmentation rules corresponding to the surgical images includes: Segmenting the surgical image according to a plurality of preset sizes to obtain segmented images corresponding to each size; Resizing the segmented image to obtain an adjusted image, wherein the adjusted image has a consistent resolution; The adjusted image is identified using a pre-trained stone model to obtain stone area images and non-stone area images corresponding to each size; All stone region images are superimposed based on the image positions to obtain the total area of ​​the stone image region; Superimposing the stone region images corresponding to each size based on the image position to obtain the stone region image area corresponding to each size; Determine the stone recognition rate corresponding to each size according to the image area of ​​the stone region corresponding to each size and the total area of ​​the stone image region; Determine an image segmentation rule corresponding to the surgical image, wherein the image segmentation rule includes a time range and two or more image segmentation sizes, wherein the stone recognition rate corresponding to the image segmentation size is not lower than the mean of the stone recognition rate, and the time range is determined according to the video time corresponding to the surgical image and the first time window; The image position is the position of the segmented image in the surgical image.

4. A method for processing a video of a hepatobiliary stone surgery as claimed in claim 3, characterized in that: The image segmentation rules corresponding to the images in the first surgical image sequence are merged as follows: Acquire a first surgical image and a second surgical image that are adjacent to each other in a first surgical image sequence; Obtaining a consistent image segmentation size in the image segmentation rules corresponding to the first surgical image and the second surgical image, setting the end point of the time range of the image segmentation rule corresponding to the first surgical image as the end point of the time range of the image segmentation rule corresponding to the second surgical image, and using the image segmentation rule corresponding to the first surgical image as the image segmentation rule corresponding to the second surgical image; The image segmentation dimensions in the first image that are not included in the corresponding image segmentation rule in the second surgical image are deleted.

5. A method for processing a video of a hepatobiliary stone surgery as claimed in claim 3, characterized in that: Using the second time window to extract images in the surgical video to obtain a second surgical image sequence; Identify images in the second surgical image sequence based on image segmentation rules to obtain location information of stone images in the surgical images in the second surgical image sequence; Image markers are drawn in the surgical video based on the location information of the stone image.

6. A method for processing a video of a hepatobiliary stone surgery as claimed in claim 5, characterized in that: The stone image regions are merged based on the position information of the stone image, and markers are drawn based on the merged stone image regions, and the drawn markers are associated with the image segmentation size within the image segmentation rule.

7. A method for processing a video of a hepatobiliary stone surgery according to claim 6, characterized in that: The image mark drawn in the surgical video corresponds to the first image segmentation size, which is an image segmentation rule that matches the surgical video time, and the image segmentation size has a maximum unexhausted time window, which is the duration of the surgical video time and the end point of the image segmentation size time range.

8. The method for processing a video of a hepatobiliary stone surgery according to claim 5, characterized in that: Recognizing the images in the second surgical image sequence based on the image segmentation rule includes: Determine a first image segmentation rule including surgical video time corresponding to an image in a second surgical image sequence; Segmenting the images in the second surgical image sequence based on the image segmentation size included in the first image segmentation rule to obtain a second segmented image; resizing the segmented second image to obtain an adjusted second segmented image; The adjusted second segmented image is recognized using the pre-trained stone model to obtain the location information of the stone image.

9. An electronic device, characterized in that: It comprises a processor and a memory; the memory is used to store a program; the processor executes the program to implement the hepatobiliary stone surgery video processing method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the hepatobiliary stone surgery video processing method according to any one of claims 1-8.

Images