Image processing method and device, equipment, medium and product

Abstract

The invention discloses an image processing method and device, equipment, a medium and a product. According to the image processing method provided by the embodiment of the invention, in the operation process, the local area image containing the key part of the operation object is determined in the first video frame image, and compared with the complete first video frame image, the local area image is smaller in size and less in data needing to be processed; and the obtained result is more accurate. And finally displaying the target area in the first video frame image differently. As the local area image is determined according to the current position of the surgical instrument, it can be guaranteed that the area is the area concerned by an operator of the surgical instrument, and as the range of the local area image is smaller, higher precision can be obtained under the same condition; the segmentation speed is ensured while the segmentation precision of the operator attention area of the surgical instrument is improved, the important organs obtained through segmentation are displayed to the operator, and the surgical safety is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image processing method, device, equipment, medium and product. Background Art

[0002] Surgery is a medical technology that aims to treat diseases and is performed locally to remove diseased tissues, repair injuries, reconstruct morphology or function, transplant cell tissues or organs, and implant medical devices. During the operation, surgical instruments are required to perform various operations on the parts of the surgical object. However, during the operation, especially surgical operations, there may be important organs of the surgical object in the operation area. When operating the instruments, care must be taken to avoid important organs to avoid serious medical accidents.

[0003] In order to avoid damaging the vital organs of the surgical subject during the operation, a computer system is generally used to assist the surgery. The image of the surgical area is first acquired, and then the image is processed. After the important organs are segmented and marked, they are displayed to the operator of the instrument to avoid touching the important organs.

[0004] However, in some surgeries, the boundaries between important organs and surrounding tissues are relatively blurred, and the exposed area changes dynamically in real time, affected by laparoscope lighting and smoke, etc. The above situations will lead to the limited accuracy of traditional image segmentation methods and cannot guarantee the safety of the operation. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, device, equipment, medium and product, which can improve the accuracy of video image segmentation during surgery, thereby ensuring the safety of the surgery.

[0006] On the one hand, an embodiment of the present application provides an image processing method, including:

[0007] Acquire a first video frame image, where the first video frame image is an image acquired during a surgery, and the first video frame image includes a surgical instrument;

[0008] Identifying a surgical instrument in the first video frame image;

[0009] Based on the identified surgical instrument, determining a local area image containing a key part of the surgical object in the first video frame image;

[0010] Inputting the first video frame image into a segmentation model to obtain a first segmentation result of a key part of the surgical object; and inputting the local area image into the segmentation model to obtain a second segmentation result of the key part of the surgical object;

[0011] fusing the first segmentation result and the second segmentation result to obtain a target area that matches a key part of the surgical object;

[0012] The target area is displayed distinctively in the first video frame image.

[0013] On the other hand, the determining of a local area image containing a key part of the surgical object in the first video frame image based on the identified surgical instrument includes:

[0014] Extracting instrument features of surgical instruments;

[0015] Based on the instrument features, determining key points about key parts of the surgical object in the first video frame image;

[0016] Based on the key points, the local area image is generated.

[0017] On the other hand, determining the key points in the first video frame image based on the device features includes:

[0018] Based on the instrument feature, determining the position of at least one target feature point of the surgical instrument;

[0019] Predicting the position of a key part of the surgical object in the first video frame image according to the part of the surgical object to which the first video frame image belongs, to obtain a prediction result;

[0020] Determining a reference point based on the position of a key part of the surgical object in the prediction result;

[0021] The midpoint of the line connecting each of the reference points and each of the target feature points is determined as the key point.

[0022] On the other hand, generating the local area image based on the key points includes:

[0023] Generate a target graphic using the key points as graphic feature points of the target graphic;

[0024] The local area image is determined according to the target graphic.

[0025] On the other hand, the target figure is an ellipse, and the step of generating the target figure using the key points as the graphic feature points of the target figure includes:

[0026] Determine the distance between the reference point corresponding to the key point and the target feature point;

[0027] The size of the ellipse is determined based on the distance, and the ellipse is generated with the key point as the center; wherein the distance is inversely correlated with the size of the ellipse.

[0028] On the other hand, before fusing the first segmentation result and the second segmentation result to obtain a target area matching a key part of the surgical object, the method further includes:

[0029] Acquire a second video frame image, where the second video frame image is a previous frame image of the first video frame image;

[0030] comparing the similarity between the second video frame image and the first video frame image;

[0031] The step of fusing the first segmentation result and the second segmentation result to obtain a target area that matches a key part of the surgical object includes:

[0032] When the similarity is higher than a threshold, the first segmentation result of the first video frame image and the second segmentation result of the second video frame image are fused to obtain the target area that matches the key part of the surgical object.

[0033] On the other hand, an embodiment of the present application provides an image processing device, including:

[0034] An acquisition module, used for acquiring a first video frame image, where the first video frame image is an image acquired during a surgery, and the first video frame image includes a surgical instrument;

[0035] An identification module, used for identifying surgical instruments in the first video frame image;

[0036] A determination module, configured to determine, based on the identified surgical instrument, a local area image containing a key part of the surgical object in the first video frame image;

[0037] An input module, used for inputting the first video frame image into a segmentation model to obtain a first segmentation result of a key part of the surgical object; and inputting the local area image into the segmentation model to obtain a second segmentation result of the key part of the surgical object;

[0038] A fusion module, used for fusing the first segmentation result and the second segmentation result to obtain a target area matching a key part of a surgical object;

[0039] A display module is used to distinguish and display the target area in the first video frame image.

[0040] In another aspect, an embodiment of the present application provides an image processing device, the device comprising: a processor and a memory storing computer program instructions;

[0041] The processor implements the above-mentioned image processing method when executing the computer program instructions.

[0042] On the other hand, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned image processing method is implemented.

[0043] On the other hand, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device performs the above-mentioned image processing method.

[0044] The image processing method of the embodiment of the present application collects a first video frame image including a surgical instrument during the operation, and identifies the surgical instrument in the first video frame image; based on the identified surgical instrument, a local area image containing a key part of the surgical object is determined in the first video frame image. Then the first video frame image is input into the segmentation model to obtain a first segmentation result of the key part of the surgical object; and the local area image is input into the segmentation model to obtain a second segmentation result of the key part of the surgical object. Because the local area image is smaller in size than the complete first video frame image, less data needs to be processed, and the result obtained is more accurate. Finally, the first segmentation result and the second segmentation result are fused to obtain a target area matching the key part of the surgical object, and the target area is displayed in the first video frame image. Because the local area image is determined according to the current position of the surgical instrument, it can be ensured that the area is the area of ​​concern to the operator of the surgical instrument, and because the range of the local area image is smaller, higher accuracy can be obtained under the same conditions; that is, the segmentation accuracy of the area of ​​concern to the operator of the surgical instrument is improved, and the segmented important organs are displayed to the operator, which can ensure the safety of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 A schematic diagram showing a flow chart of an image processing method provided by an embodiment of the present application is shown;

[0047] Figure 2 A schematic diagram showing the complete data flow of the framework proposed in the embodiment of the present application and the connection relationship between different modules;

[0048] Figure 3 A schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application is shown;

[0049] Figure 4 A schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0050] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0051] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0052] As mentioned above, during surgery, the subject's important organs may be present in the operating area. When operating instruments, care must be taken to avoid important organs to avoid causing serious medical accidents.

[0053] Taking rectal cancer as an example, total mesorectal excision (TME) has become the standard surgical treatment for rectal cancer because of its significant effect in controlling local recurrence.

[0054] However, laparoscopic TME surgery faces many challenges due to the narrow and deep pelvic space, especially in the case of mid- and low-position rectal cancer. Using a bottom-up approach, transanal total mesorectal excision (TaTME) solves some technical difficulties in pelvic anatomy and improves the assessment of the distal resection margin, thereby ensuring the quality of TME. However, the implementation of taTME may significantly increase the risk of perioperative complications.

[0055] Multiple studies have reported that the incidence of urethral injury (UI) in TaTME is 0.4-1%. The most common factor leading to UI is the failure to correctly identify the anterior TME cutting plane. Factors leading to incorrect dissection include post-radiotherapy fibrosis and anterior tumors causing fusion or disappearance of tissue planes. In addition, intraoperative smoke can also obscure the field of vision, increasing the difficulty of surgery. The correct dissection plane is between the anterior rectal wall and the posterior prostate. Deviation from this plane upward may cause prostate injury, thereby causing UI, while deviation downward may cause rectal perforation or positive circumferential margins.

[0056] Computer-assisted intervention systems can monitor and analyze the surgical process, provide automated assistance to clinical staff, and warn clinicians when potential complications occur. There is an urgent need to develop an accurate and real-time computer-assisted intervention system to identify prostate areas during TaTME and help surgeons avoid this area, but it also faces major challenges.

[0057] Most studies on automatic prostate segmentation focus on preoperative prostate segmentation, including prostate segmentation studies based on ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI) images.

[0058] It is very challenging to directly apply these segmentation results from the patient's preoperative examination data to TaTME. This process involves converting the data of the two modalities into a shared coordinate system, and then aligning and fusing them. At the same time, as the operation progresses, the patient's anatomical structure changes, which greatly increases the difficulty. Therefore, a better solution is to use intraoperative video for prostate segmentation.

[0059] Traditional solutions usually process the entire image collected during the operation, segment and mark important areas (such as the prostate area) so that surgical instrument operators can avoid this area. However, there are various influencing factors during the operation, such as large deformation of the prostate, the influence of smoke and bleeding, etc. These factors will lead to low image segmentation accuracy, making it impossible for instrument operators to accurately determine the location of important areas, and thus unable to ensure the safety of the operation.

[0060] In order to solve the problems of the prior art, the embodiments of the present application provide an image processing method, device, equipment, medium and product. The image processing method provided by the embodiments of the present application is first introduced below. Figure 1 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application; Figure 1 As shown, the method comprises the following steps:

[0061] S101: Acquire a first video frame image.

[0062] The first video frame image is an image acquired during the operation, and the first video frame image to be processed includes surgical instruments. The embodiment of the present application does not limit which frame image the first video frame image is, and generally any video frame image during the operation is required to perform each step of the present solution. The image is usually displayed to the operator of the surgical instrument so that the operation can be completed according to the image content.

[0063] S102: Identify surgical instruments in the first video frame image.

[0064] As mentioned above, the first video frame image includes surgical instruments, and the surgical instruments accidentally touching the important parts of the surgical object is an important factor causing surgical accidents. Therefore, the area that the operator of the instrument pays attention to is related to the position of the surgical instrument, and the surgical instruments in the first video frame image need to be identified. Here, there is no limitation on how to identify the surgical instruments in the first video frame image, and a commonly used image recognition model can be used, or a feature extraction module can be used to extract the instrument features of the surgical instruments to complete the identification of the surgical instruments.

[0065] S103: Based on the identified surgical instrument, determine a local area image containing a key part of the surgical object in the first video frame image.

[0066] After identifying the surgical instrument in the first video frame image, it is necessary to determine the local area image containing the key parts of the surgical object in the first video frame image. The local area image is an image with a smaller size than the first video frame image, because the subsequent segmentation is performed on the key parts of the surgical object, so the local area image contains the key parts of the surgical object. As mentioned above, accidental contact of the surgical instrument with the important parts of the surgical object is an important factor causing surgical accidents. Therefore, in addition to including the key parts of the surgical object, the area that the operator of the instrument pays attention to is also related to the position of the surgical instrument. Therefore, the local area image is specifically determined based on the identified surgical instrument.

[0067] S104: Inputting the first video frame image into the segmentation model to obtain a first segmentation result of the key part of the surgical object; and inputting the local area image into the segmentation model to obtain a second segmentation result of the key part of the surgical object.

[0068] This embodiment performs two segmentations in total. The first segmentation is based on the first video frame image as a whole. The key parts of the surgical object are segmented from the complete image to obtain the first segmentation result. The second segmentation is based on the local area image calculated above. As mentioned above, the local area image includes the key parts of the surgical object. The second segmentation is to segment the key parts of the surgical object from the local area image to obtain the second segmentation result. Because the local area image is smaller in size than the complete first video frame image, less data needs to be processed, and the result obtained is more accurate.

[0069] S105: Fusing the first segmentation result and the second segmentation result to obtain a target area that matches the key part of the surgical object.

[0070] After obtaining the segmentation results of the above two segmentations, the two segmentation results need to be fused so that the final image includes both global content and more accurate segmentation results for the local content where the key parts of the surgical object are located. Finally, the target area matching the key parts of the surgical object is obtained.

[0071] S106: Distinctively displaying the target area in the first video frame image.

[0072] Because the first video frame image is a complete image, usually during the operation, the surgical instrument operator needs to view the overall situation, so the image displayed to the instrument operator is the complete first video frame image. In order to prevent the instrument operator from accidentally touching the key parts of the surgical object, the target area will be displayed distinguishably in the first video frame image.

[0073] This embodiment can achieve image segmentation through a video semantic segmentation (VSS) framework. The VSS framework of this solution includes a convolutional neural network (CNN) for semantic image segmentation and two modules (Instrument Guidance (IG) and Key Frame Selection (KFS)). CNN can be any network for semantic image segmentation. As for the other two modules, they focus on improving the segmentation accuracy of the first frame (I frame) and accelerating the processing speed of the remaining frames (P frames). The IG module is designed to extract instrument features and calculate surgical areas of interest. Afterwards, the KFS module is designed to select P frames that are not suitable for segmentation. All I frames are directly sent to instrument-guided segmentation. As for P frames, they are judged by the KFS module.

[0074] Figure 2The complete data flow of the framework proposed in the embodiment of the present application and the schematic diagram of the connection relationship between different modules are shown. Figure 2 As shown, it includes an acquired first video frame image 201, an instrument guidance module 202, an image segmentation network module 203, a first segmentation result 204, a second segmentation result 205, and a final segmentation image 206. The first video frame image 201 is input into the segmentation model to obtain a first segmentation result 204 of a key part of the surgical object; and the local area image is input into the segmentation model to obtain a second segmentation result 205 of a key part of the surgical object; the first segmentation result 204 and the second segmentation result 205 are fused to obtain a final segmentation image 206.

[0075] The instrument guidance module 202 includes three main parts: instrument feature extraction, key point detection and region of interest generation. The instrument feature extraction uses the image segmentation network module 203 to obtain the instrument features. Key point detection is to find multiple discrete key points on the instrument feature map. These key points represent the focus of the surgeon. The region of interest generation is to generate a region on these discrete key points, that is, a local area image. The above three components complete the generation process and map the image to the surgeon's region of interest. The details of these components will be elaborated in the subsequent sections.

[0076] In the process of instrument feature extraction, we used an image segmentation model. For the input image at time t, its size is H*W*G, H is the height of the input image, W is the width of the input image, and G represents the number of channels of the input image. Since the input image is in color, G=3, we can obtain the corresponding instrument feature map, whose size is H*W*1. This method effectively identifies and extracts features related to surgical instruments by analyzing the content in the input image, which will be used in the subsequent key point detection and focus area generation steps.

[0077] When detecting key points, for a given instrument feature map IF (t) , we can obtain the set of instrument profiles C = {C1, C2, ..., C i , ..., C n}, where C i Composed of a group of pixels, C i ={(X i1 ,y i1 ), (X i1 ,y i1 ), (X i2 ,y i2 ),...,(X in ,y in )}. Find C i The vertex P to form KP1, and then calculate with IF (t)The midpoint between the midpoints of the upper edges (X mid , y0), denoted as KP2. Finally, KP is generated as the output to form a point set, KP={KP1,KP2}={(x1,y1), ······, (x n ,y n )}.

[0078] When generating a region of interest (local region image), taking prostate dissection as an example, the process involves generating a single sub-region AR based on the location of each key point (i) Subsequently, all these sub-regions are combined together to form the final region of interest AR (t) Due to the characteristics of TaTME, the prostate mainly appears in the upper middle part of the input image during the process. The generated AR can be adjusted by considering the distance of each point from the upper edge. (i) When a point is located at a greater distance from the upper edge, it corresponds to a smaller ellipse AR (i) Conversely, if a point is closer, then AR (i) Bigger.

[0079] The key frame selection module KFS is described below. (t) and Focus Area AR (t) Afterwards, an adaptive KFS module is constructed to select video frames that do not need to be segmented for feature fusion based on the similarity between two adjacent video frames.

[0080] To demonstrate the effectiveness of KFS, experiments were conducted using GOP values ​​of 3, 6, 9, 12, 15, and 18 for the number of frames between two key frames. Experimental results show that choosing an appropriate GOP value is crucial to optimizing segmentation performance. The KFS module significantly improves the overall segmentation effect by optimizing key frame selection and segmentation strategies under specific GOP settings. Higher GOP values ​​help achieve higher IoUs, showing the potential to reduce computation while maintaining image quality. This approach not only improves processing speed but also ensures segmentation accuracy, which is the key to efficient video processing.

[0081] This application introduces a new video semantic segmentation framework called IGNet, which integrates the features of surgical instruments into the real-time and accurate prostate segmentation task. Specifically, an instrument guidance module is designed to extract a priori instrument features and calculate the surgeon's area of ​​interest. Then a segmentation network is proposed to fuse the two segmentation results to improve segmentation accuracy. Finally, an adaptive keyframe selection module based on instrument features is used to optimize segmentation speed while maintaining accuracy.

[0082] The image processing method of the embodiment of the present application collects a first video frame image including a surgical instrument during the operation, and identifies the surgical instrument in the first video frame image; based on the identified surgical instrument, calculates a local area image containing a key part of the surgical object in the first video frame image. Then the first video frame image is input into the segmentation model to obtain a first segmentation result of the key part of the surgical object; and the local area image is input into the segmentation model to obtain a second segmentation result of the key part of the surgical object. Because the local area image is smaller in size than the complete first video frame image, less data needs to be processed, and the result obtained is more accurate. Finally, the first segmentation result and the second segmentation result are fused to obtain a target area matching the key part of the surgical object, and the target area is displayed in the first video frame image. Because the local area image is determined according to the current position of the surgical instrument, it can be ensured that the area is the area of ​​concern to the operator of the surgical instrument, and because the range of the local area image is smaller, higher accuracy can be obtained under the same conditions; that is, the segmentation accuracy of the area of ​​concern to the operator of the surgical instrument is improved, and the segmented important organs are displayed to the operator, which can ensure the safety of the operation.

[0083] In the above-mentioned embodiment, it is mentioned that after the surgical instrument in the first video frame image is identified, it is necessary to calculate a local area image including a key part of the surgical object in the first video frame image based on the identified surgical instrument.

[0084] For the local area image, different determination methods will result in different sizes and positions of the obtained local area images. This embodiment provides a specific implementation scheme, which first extracts the instrument features of the surgical instrument from the first video frame image, then determines the key points of the key parts of the surgical object in the first video frame image based on the instrument features, and finally generates the local area image based on the key points.

[0085] By using the method of determining key points through instrument features provided in this embodiment, the instrument features are quickly extracted through existing feature extraction means, thereby quickly determining the key points, and being able to quickly and accurately determine the local area image containing the key parts of the surgical object in the first video frame image, thereby accelerating the image segmentation efficiency.

[0086] In the above embodiment, it is not specifically limited how to determine the key points in the first video frame image based on the device features. Because the key points are used to calculate the local focus area, the determination of the key points is very important.

[0087] This embodiment provides a solution, which first determines the position of at least one target feature point of the surgical instrument based on the instrument features. The target feature point is a specific representative point on the surgical instrument, and generally the point on the top of the instrument is used as the target feature point.

[0088] As mentioned above, since the subsequent segmentation is performed on the key parts of the surgical object, the key parts of the surgical object need to be included in the local focus area. Therefore, further, the reference point needs to be determined according to the part of the surgical object in the first video frame image.

[0089] In this embodiment, the midpoint of the line connecting the reference point and each target feature point is determined as a key point, and then the local focus area is calculated based on the key point. By combining the reference point, it can be ensured that the local focus area contains the key parts of the surgical object; by combining the target feature point, it can be ensured that the generated local area image is the area that the instrument operator pays more attention to.

[0090] This embodiment provides a specific solution for calculating the operator's attention area, that is, the local attention area, based on key points. In practical applications, there are no specific requirements for the size, shape and position of the local attention area, which can be set according to actual conditions. In general, the local attention area can be set as a rectangle, and its position and size can be determined based on the reference points and target feature points in the above embodiment.

[0091] In practical applications, the midpoint of the line connecting each reference point and each target feature point can be determined as a key point, and then a target figure is generated with the key point as the graphic feature point of the target figure. There is no requirement for the shape of the target figure. The shape of the target figure can be set according to the approximate shape of the key part of the surgical object, and then the local area image is determined according to the target figure.

[0092] Because in the specific implementation, there may be more than one reference point and target feature point, there are multiple key points and multiple target graphics; when the local area image is finally determined based on the target graphics, the local area image needs to include all the target graphics.

[0093] This embodiment provides a solution for generating a target graph based on key points and then generating a local area image based on the target graph. Through this solution, the key area of ​​the surgical object can be more accurately included in the local area image, while avoiding including too many unnecessary areas. While ensuring the accuracy of target area segmentation, the segmentation speed is improved.

[0094] The above does not limit the specific shape of the target graphic. This embodiment provides a specific solution. The target graphic is specifically set to be an ellipse. The process of generating the target graphic with the key points as the graphic feature points of the target graphic specifically includes:

[0095] First determine the distance between the reference point and the target feature point corresponding to the key point, then determine the size of the ellipse based on the distance, and generate the ellipse with the key point as the center. Because the larger the distance between the reference point and the target feature point, the farther the surgical instrument is from the key part of the surgical object, the smaller the probability of mis-touch, and the less attention the instrument operator pays to the target shape, so a smaller target shape (ellipse) can be generated. Therefore, the embodiment of the present application stipulates that the distance between the reference point and the target feature point is inversely correlated with the size of the ellipse.

[0096] The solution provided in the embodiment of the present application uses an ellipse as the target graphic and a key point as the center of the ellipse. The size of the ellipse is inversely correlated with the distance between the surgical instrument and the key part of the surgical object, that is, the size of the ellipse is related to the attention of the instrument operator. When it is not necessary, a smaller local area image is obtained to speed up the segmentation speed.

[0097] Because the present application performs secondary segmentation on the key parts of the surgical object, it inevitably causes an increase in the calculation data. Under the condition of the same computing power, the operation speed will slow down. Therefore, in order to ensure the segmentation speed, this embodiment proposes a solution, that is, to compare the similarity between the current video frame (the first video frame) and the previous video frame (the second video frame). When the similarity is high enough, the segmentation result of the previous video frame can be reused. When performing the similarity comparison, the focus is on the comparison of the device features in the two video frame images.

[0098] If the similarity between the second video frame image and the first video frame image is high, it indicates that the position and size of the instrument and the key parts of the surgical object in the two video frames have little change. At this time, in order to speed up the segmentation speed, the segmentation result of the previous video frame can be reused to determine the target area of ​​the current video frame. Specifically, when the similarity is higher than the threshold, the first segmentation result of the first video frame image and the second segmentation result of the second video frame image are fused to obtain an accurate segmentation result.

[0099] Through the solution proposed in the embodiment of the present application, when conditions permit, the historical segmentation results are used as a reference to speed up the segmentation speed without affecting the accuracy as much as possible.

[0100] In order to solve the above technical problems, the present application provides an image processing device. Figure 3 FIG. 4 shows a schematic diagram of the hardware structure of the image processing device provided in the embodiment of the present application; Figure 3 As shown, the device includes the following modules:

[0101] An acquisition module 301 is used to acquire a first video frame image, where the first video frame image is an image acquired during a surgery, and the first video frame image includes a surgical instrument;

[0102] An identification module 302, configured to identify a surgical instrument in a first video frame image;

[0103] A determination module 303 is used to determine a local area image containing a key part of the surgical object in the first video frame image based on the identified surgical instrument;

[0104] An input module 304 is used to input the first video frame image into the segmentation model to obtain a first segmentation result of the key part of the surgical object; and input the local area image into the segmentation model to obtain a second segmentation result of the key part of the surgical object;

[0105] A fusion module 305 is used to fuse the first segmentation result and the second segmentation result to obtain a target area that matches the key part of the surgical object;

[0106] The display module 306 is used to distinguish and display the target area in the first video frame image.

[0107] The image processing device provided in this embodiment corresponds to the image processing method mentioned in the above embodiment, and therefore has the same effect as the image processing method, which will not be described in detail here.

[0108] Figure 4 FIG. 1 shows a schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application; Figure 4 As shown, the image processing device may include a processor 401 and a memory 402 storing computer program instructions.

[0109] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0110] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0111] The memory 402 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0112] The processor 401 implements any one of the image processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0113] In one example, the image processing device may further include a communication interface 403 and a bus 410. The processor 401, the memory 402, and the communication interface 403 are connected via the bus 410 and communicate with each other.

[0114] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0115] Bus 410 includes hardware, software or both, and the parts of image processing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0116] In addition, in combination with the image processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image processing methods in the above embodiment is implemented.

[0117] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the image processing methods in the above embodiments when the computer program is processed and executed.

[0118] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0119] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0120] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0121] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0122] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image processing method, characterized in that: include: Acquire a first video frame image, where the first video frame image is an image acquired during a surgery, and the first video frame image includes a surgical instrument; Identifying a surgical instrument in the first video frame image; Based on the identified surgical instrument, determining a local area image containing a key part of the surgical object in the first video frame image; Inputting the first video frame image into a segmentation model to obtain a first segmentation result of a key part of the surgical object; and inputting the local area image into the segmentation model to obtain a second segmentation result of the key part of the surgical object; fusing the first segmentation result and the second segmentation result to obtain a target area that matches a key part of the surgical object; The target area is displayed distinctively in the first video frame image.

2. The image processing method according to claim 1, characterized in that: The method of determining a local area image containing a key part of the surgical object in the first video frame image based on the identified surgical instrument includes: Extracting instrument features of surgical instruments; Based on the instrument features, determining key points about key parts of the surgical object in the first video frame image; Based on the key points, the local area image is generated.

3. The image processing method according to claim 2, characterized in that: The determining of key points in the first video frame image based on the device features includes: Based on the instrument feature, determining the position of at least one target feature point of the surgical instrument; Predicting the position of a key part of the surgical object in the first video frame image according to the part of the surgical object to which the first video frame image belongs, to obtain a prediction result; Determining a reference point based on the position of a key part of the surgical object in the prediction result; The midpoint of the line connecting each of the reference points and each of the target feature points is determined as the key point.

4. The image processing method according to claim 3, characterized in that: The step of generating the local area image based on the key points includes: Generate a target graphic using the key points as graphic feature points of the target graphic; The local area image is determined according to the target graphic.

5. The image processing method according to claim 4, characterized in that: The target graphic is an ellipse, and the step of generating the target graphic using the key points as graphic feature points of the target graphic includes: Determine the distance between the reference point corresponding to the key point and the target feature point; The size of the ellipse is determined based on the distance, and the ellipse is generated with the key point as the center; wherein the distance is inversely correlated with the size of the ellipse.

6. The image processing method according to claim 1, characterized in that: Before fusing the first segmentation result and the second segmentation result to obtain a target area matching a key part of the surgical object, the method further includes: Acquire a second video frame image, where the second video frame image is a previous frame image of the first video frame image; comparing the similarity between the second video frame image and the first video frame image; The step of fusing the first segmentation result and the second segmentation result to obtain a target area that matches a key part of the surgical object includes: When the similarity is higher than a threshold, the first segmentation result of the first video frame image and the second segmentation result of the second video frame image are fused to obtain the target area that matches the key part of the surgical object.

7. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first video frame image, where the first video frame image is an image acquired during a surgery, and the first video frame image includes a surgical instrument; An identification module, used for identifying surgical instruments in the first video frame image; A determination module, configured to determine, based on the identified surgical instrument, a local area image containing a key part of the surgical object in the first video frame image; An input module, used for inputting the first video frame image into a segmentation model to obtain a first segmentation result of a key part of a surgical object; and inputting the local area image into the segmentation model to obtain a second segmentation result of the key part of the surgical object; A fusion module, used for fusing the first segmentation result and the second segmentation result to obtain a target area matching a key part of a surgical object; A display module is used to distinguish and display the target area in the first video frame image.

8. An image processing device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image processing method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image processing method according to any one of claims 1 to 6.

Images