Laparoscope and target object edge extraction method

Abstract

The embodiment of the invention provides a laparoscope and a target object edge extraction method, whether an initial edge curve comprises a plurality of discontinuous line segments or not can be determined according to the initial edge curve of an object, and if the initial edge curve comprises the plurality of discontinuous line segments, adjacent points are marked for a plurality of end points on the plurality of line segments; and taking the end points marked as the adjacent points as target end points, and performing curve fitting according to the target end points and the adjacent points of the target end points to obtain a continuous edge curve of the target object. According to the method, the problem of discontinuous target object edge extraction under the laparoscope can be solved, and the accuracy and coherence of target object edge extraction are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a laparoscope and a method for extracting the edge of a target object. Background Art

[0002] Compared with traditional liver surgery, laparoscopic surgery has the advantages of small incision, less pain and scarring for patients, and faster recovery. Therefore, laparoscopic surgery has become the current development trend of liver surgery. The key to laparoscopic surgery is to extract the liver edge curve based on the medical image of the liver.

[0003] Currently, the method of extracting liver edge curves mainly focuses on how to extract edge curves with a single pixel width. The extracted edge curves are prone to disconnection, resulting in inaccurate surgical planning and increased surgical difficulty. Summary of the invention

[0004] In order to solve the above problems in the prior art, the present application example provides a laparoscope and a target object edge extraction method, which can determine a continuous liver edge curve through image processing technology.

[0005] In a first aspect, an embodiment of the present application provides a laparoscope, including an image acquisition component and a processor;

[0006] The image acquisition component is configured to acquire an image containing a target object;

[0007] The processor is configured to:

[0008] Performing image segmentation on the image to obtain an initial edge curve of the target object;

[0009] If the initial edge curve includes a plurality of discontinuous line segments, marking an adjacent point for each of the plurality of endpoints according to positions of the plurality of endpoints of the plurality of line segments;

[0010] The endpoints marked as adjacent points are used as target endpoints, and curve fitting is performed according to the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

[0011] In a possible implementation, the processor is specifically configured to:

[0012] It is determined whether the initial edge curve includes a plurality of discontinuous line segments according to the number of endpoints included in the initial edge curve.

[0013] In a possible implementation, the processor is specifically configured to:

[0014] If the number of endpoints included in the initial edge curve is greater than a set number threshold, it is determined that the initial edge curve includes a plurality of discontinuous line segments.

[0015] In a possible implementation, the processor is specifically configured to:

[0016] For any pixel point on the initial edge curve, determine whether the pixel point is an endpoint according to the pixel values ​​of the adjacent pixel points of the pixel point;

[0017] The number of pixels determined as endpoints is used as the number of endpoints included in the initial edge curve.

[0018] In a possible implementation, the processor is specifically configured to:

[0019] The image is input into an image processing network model, and the image is segmented by the image processing network model to obtain an initial edge curve of the target object.

[0020] In a possible implementation, the processor is specifically configured to:

[0021] For each of the multiple endpoints, perform the following operations respectively:

[0022] taking the endpoint as the current endpoint;

[0023] Determine, according to the positions of the multiple endpoints, the distances between the remaining endpoints except the current endpoint and the current endpoint respectively;

[0024] The remaining endpoints whose distances from the current endpoint are less than a set distance threshold are used as candidate endpoints;

[0025] According to the number of the candidate endpoints, adjacent points of the current endpoint are marked in the remaining endpoints.

[0026] In a possible implementation, the processor is specifically configured to:

[0027] If there are multiple candidate endpoints, select a target candidate endpoint from the multiple candidate endpoints, and mark the target candidate endpoint as an adjacent point of the current endpoint;

[0028] If the number of the candidate endpoints is one, marking the candidate endpoint as an adjacent point of the current endpoint;

[0029] If there is no candidate endpoint, the remaining endpoint with the smallest distance from the current endpoint is marked as an adjacent point of the current endpoint.

[0030] In a possible implementation, the processor is specifically configured to:

[0031] The slope of the connecting line between each candidate endpoint and the current endpoint is determined respectively, and the candidate endpoint with the smallest slope difference between the connecting line slope and the slope of the curve at the current endpoint of the initial edge curve is used as the target candidate endpoint.

[0032] In a second aspect, an embodiment of the present application provides a method for extracting an edge of a target object, the method comprising:

[0033] Performing image segmentation on an image containing a target object to obtain an initial edge curve of the target object;

[0034] If the initial edge curve includes a plurality of discontinuous line segments, marking an adjacent point for each of the plurality of endpoints according to positions of the plurality of endpoints of the plurality of line segments;

[0035] The endpoints marked as adjacent points are used as target endpoints, and curve fitting is performed according to the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

[0036] In a third aspect, an embodiment of the present application provides a target object edge extraction device, the target object edge extraction device comprising:

[0037] An edge curve extraction unit, used for performing image segmentation on an image containing a target object to obtain an initial edge curve of the target object;

[0038] an adjacent point determination unit, configured to mark an adjacent point for each of the multiple endpoints according to the positions of the multiple endpoints of the multiple line segments if the initial edge curve includes multiple discontinuous line segments;

[0039] The curve fitting unit is used to take the endpoints marked as adjacent points as target endpoints, perform curve fitting according to the target endpoints and the adjacent points of the target endpoints, and obtain the continuous edge curve of the target object.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target object edge extraction method is implemented.

[0041] The embodiment of the present application provides a laparoscope and a method for extracting the edge of a target object. It can determine whether the initial edge curve includes multiple discontinuous line segments according to the initial edge curve of the object. If the initial edge curve includes multiple discontinuous line segments, the adjacent points are marked for multiple endpoints on the multiple line segments, and the endpoints marked as adjacent points are used as target endpoints. According to the target endpoint and the adjacent points of the target endpoint, curve fitting is performed to obtain a continuous edge curve of the target object. This method can solve the problem of discontinuous edge extraction of target objects under laparoscopy and improve the accuracy and consistency of edge extraction of target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A schematic diagram of the structure inside a laparoscope provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of the external structure of a laparoscope provided in an embodiment of the present application;

[0045] Figure 3 A flow chart of a target object edge extraction method provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the structure of an image processing network model provided in an embodiment of the present application;

[0047] Figure 5 An image including a liver provided in an embodiment of the present application;

[0048] Figure 6 A liver segmentation image provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of a continuous edge curve without disconnection provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of an initial edge curve including multiple discontinuous line segments provided in an embodiment of the present application;

[0051] Fig. 9 A flowchart for determining whether an initial edge curve includes multiple discontinuous line segments provided in an embodiment of the present application;

[0052] Fig.10A flow chart for determining the number of endpoints of an initial edge curve provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of a method for determining a curve endpoint provided in an embodiment of the present application;

[0054] Fig.12 A schematic diagram of an initial edge curve of a target object provided in an embodiment of the present application;

[0055] Fig.13 A flowchart for marking adjacent points for a current endpoint provided in an embodiment of the present application;

[0056] Fig.14 A flowchart of marking adjacent points of a current endpoint among remaining endpoints according to the number of candidate endpoints provided in an embodiment of the present application;

[0057] Fig.15 A flowchart of target object edge extraction performed by a laparoscope processor provided in an embodiment of the present application;

[0058] Fig.16 A structural block diagram of a target object edge extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0060] It should be noted that the application scenarios described in the following embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. A person of ordinary skill in the art can know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0061] At present, the development trend of liver surgery is to use laparoscopic technology for surgery. The key to laparoscopic surgery is to extract the liver edge curve based on the medical image of the liver. The edge curve extracted by traditional methods is prone to disconnection, resulting in inaccurate surgical planning.

[0062] Based on this, an embodiment of the present application provides a laparoscope that can determine whether the initial edge curve includes multiple discontinuous line segments based on the initial edge curve of the target object. If the initial edge curve includes multiple discontinuous line segments, adjacent points are marked for multiple endpoints on the multiple line segments, and the endpoints marked as adjacent points are used as target endpoints. Curve fitting is performed based on the target endpoint and the adjacent points of the target endpoint to obtain a continuous edge curve of the target object. This method can obtain a continuous edge curve of the target object, solve the problem of discontinuous liver edge extraction under laparoscopy, improve the accuracy and consistency of liver edge extraction, and thus improve the efficiency of doctors in locating the liver edge under laparoscopy. The continuous liver edge curve provides accurate anatomical information for surgery, and doctors can perform more accurate surgical planning, reducing the difficulty of surgery.

[0063] Figure 1 FIG. 2 shows a schematic diagram of the internal structure of a laparoscope provided in an embodiment of the present application. The laparoscope may be a standard laparoscope, a double-hole laparoscope, or a multi-hole laparoscope. Figure 1 As shown, the laparoscope 100 may include an image acquisition component 110, a processor 120 and a memory 130. The image acquisition component 110 and the memory 130 may be connected to the processor 120 via a bus.

[0064] The image acquisition component 110 may include a light source 111, a lens 112, and an image collector 113. The light source 111 is connected to the lens 112, and the light source 111 may generate high-brightness light, which is transmitted to the front end of the laparoscope through an optical fiber to provide illumination during surgery; the lens 112 is composed of a lens, which is used to magnify and focus the image in the abdominal cavity; the image collector 113 is located behind the lens 112, and is used to collect the magnified image, convert the magnified image containing the target object into an electronic signal, and transmit it to a display connected to the laparoscope for the doctor to observe and operate.

[0065] The memory 130 may be a volatile memory, such as a random access memory; the memory may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 130 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 130 may be a combination of the above memories. The memory 130 may be used to store software programs and modules. The memory 130 may also store images containing target objects transmitted by the image collector 113, as well as processing results of the images.

[0066] The processor 120 may include one or more processors. The processor 120 executes the method for determining the edge curve of the target object provided in the embodiment of the present application by running the software programs and modules stored in the memory 130 .

[0067] Exemplarily, the processor 120 can be used to process an image containing a target object stored in the memory 130. First, the image containing the target object can be segmented to obtain an initial edge curve of the target object. If the initial edge curve includes multiple discontinuous line segments, adjacent points can be marked for each endpoint in the multiple endpoints according to the positions of multiple endpoints on the multiple line segments. Subsequently, the endpoints marked as adjacent points can be used as target endpoints, and curve fitting can be performed based on the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

[0068] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the node device. In other embodiments of the present application, the node device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0069] Figure 2 FIG. 2 shows the appearance of a laparoscope provided in an embodiment of the present application. Figure 2 As shown, the laparoscope 100 may include an operating rod 140, an outer sleeve 150 and a gas channel 160. The operating rod 140 connects the image acquisition component and the handle 170, and can control the movement, rotation and focus of the image collector and the lens in the image acquisition component; the outer sleeve 150 is the outer shell of the laparoscope, protecting the internal optical fiber, lens and operating rod and other parts; the gas channel 160 introduces carbon dioxide into the abdominal cavity to maintain the space in the abdominal cavity during surgery, providing better visibility and operating space. The memory 130 and the processor 120 can be set in the handle 170, or in other locations inside the laparoscope 100.

[0070] Figure 3 A flow chart of a target object edge extraction method provided in an embodiment of the present application is shown. Figure 3 As shown, the method may include the following steps:

[0071] Step S301, performing image segmentation on an image containing a target object to obtain an initial edge curve of the target object.

[0072] In some embodiments, during operation, the laparoscope can collect images containing the target object through the image acquisition component and transmit the images containing the target object to the processor. The processor inputs the images containing the target object into the image processing network model to obtain the initial edge curve of the target object output by the image processing network model.

[0073] For example, Figure 4 As shown, the image processing network model may include an image segmentation network and an edge extraction network. The processor may input an image containing a target object into the image segmentation network, and use the image segmentation network to segment the target object from the image containing the target object, thereby obtaining a target object segmentation image output by the image segmentation network. The processor may input the target object segmentation image into the edge extraction network, and use the edge extraction network to extract the edge information of the target object from the target object segmentation image, thereby obtaining an initial edge curve of the target object output by the edge extraction network.

[0074] For example, in liver surgery, the target object may be the liver. The image acquisition component of the laparoscope may acquire an image containing the liver and transmit the image containing the liver to the processor. The image containing the liver may be as follows: Figure 5 As shown. The processor inputs the image containing the liver into the image segmentation network. Exemplarily, the image segmentation network can use a deep feature fusion (DFF) network to segment the image containing the liver, and the output of the DFF network can be a liver segmentation image. Exemplarily, the DFF network is usually composed of two or more sub-networks of different levels, each of which can extract different levels of feature representation of the input data. In the DFF network, features from different levels can be fused by using appropriate fusion strategies to produce richer and more expressive feature representations.

[0075] After the image segmentation network performs image segmentation on the image containing the liver, the following can be obtained: Figure 6 The liver segmentation image is shown. The largest area is the liver, and the edge of the liver is blocked by other organs. In addition to liver surgery, different surgical operations have different target objects. For example, in cardiac surgery, the target object can be the heart; in thoracic surgery, the target object can be the lungs.

[0076] After obtaining the target object segmentation image using the image segmentation network, the target object segmentation image can be input into the edge extraction network. For example, the edge extraction network can use a deep centerline DeepCenterline network for edge extraction. The DeepCenterline network is a deep learning method for image edge extraction, which can accurately extract edge information from the target object segmentation image output by the DFF network and obtain the output initial edge curve of the target object.

[0077] In some embodiments, the initial edge curve in the image containing the initial edge curve of the target object may be as follows: Figure 7 As shown, the initial edge curve is a continuous edge curve without disconnection.

[0078] In some other embodiments, the initial edge curve in the image containing the initial edge curve of the target object may be as follows: Figure 8 As shown, the initial edge curve includes multiple discontinuous line segments, which are disconnected.

[0079] Step S302: if the initial edge curve includes multiple discontinuous line segments, mark adjacent points for each of the multiple endpoints according to the positions of the multiple endpoints of the multiple line segments.

[0080] In some embodiments, Figure 7 As shown, the initial edge curve may be a continuous edge curve without disconnection, and no subsequent operation is required, and the curve may be directly output.

[0081] In other embodiments, Figure 8 As shown, the initial edge curve may include multiple discontinuous line segments, resulting in disconnection, and thus subsequent operations are required.

[0082] For example, after obtaining an image containing an initial edge curve of a target object, the following method can be used to obtain the image: Fig. 9 The method shown in the figure determines whether the initial edge curve includes multiple discontinuous line segments. The process may include the following steps:

[0083] Step S901, determining the number of endpoints of the initial edge curve.

[0084] For example, the process can be performed by Fig.10 The process can include the following steps:

[0085] Step S9011, performing a binarization operation on the image containing the initial edge curve of the target object.

[0086] In some embodiments, the binarization operation can simplify the image and simplify the complex information in the image into two colors, black and white, where the black pixel value is 1 and the white pixel value is 0, thereby reducing the complexity and computational complexity of subsequent processing.

[0087] Step S9012: for any pixel point on the initial edge curve, determine the sum of the pixel values ​​of the eight pixel points surrounding the pixel point.

[0088] For example, if any pixel point on the input initial edge curve is recorded as P1, the positions of the eight neighboring points P2, P3, P4, P5, P6, P7, P8, and P9 of P1 are as follows: Fig.11 As shown, after binarization, the image only contains black and white pixels, where the pixel value of the black pixel is 1 and the pixel value of the white pixel is 0. sum can be defined as the sum of the pixel values ​​of the eight pixels around P1, that is, sum = P2+P3+P4+P5+P6+P7+P8+P9.

[0089] Step S9013, determine whether the sum of the pixel values ​​is equal to 1, if not, execute step S9014; if yes, execute step S9015.

[0090] For example, if the sum of the pixel values ​​of the eight points around P1 is sum=0, it can be considered that there is no pixel point with a pixel value equal to 1 among the eight neighborhood points around P1, and P1 is an isolated point; if sum=1, it can be considered that there is one and only one pixel point with a pixel value equal to 1 among the eight neighborhood points around P1, and P1 is an endpoint; if sum>1, it can be considered that there is at least one pixel point with a pixel value equal to 1 among the eight neighborhood points around P1, and P1 is an internal point.

[0091] Step S9014: determine that the pixel point is not an endpoint on the initial edge curve.

[0092] Specifically, neither the isolated point nor the internal point is an endpoint on the initial edge curve, so the point can be marked as not being an endpoint on the initial edge curve.

[0093] Step S9015: mark the pixel point as an endpoint on the initial edge curve.

[0094] Specifically, the endpoint is marked as an endpoint on the initial edge curve.

[0095] Step S9016, determine whether there are any unmarked pixels, if so, execute step S9017; if not, execute step S9018.

[0096] Specifically, the position of each endpoint can be found by traversing each pixel point on the initial edge curve.

[0097] Step S9017: for the next unmarked pixel point, determine the sum of the pixel values ​​of the eight pixels around the pixel point.

[0098] Step S9018, recording the number of endpoints of the initial edge curve.

[0099] After determining the number of endpoints of the initial edge curve in step S901, the following can be obtained: Fig.12 The schematic diagram of the initial edge curve of the target object shown in the figure numbers each endpoint, the first endpoint 1201, the second endpoint 1202, the third endpoint 1203, the fourth endpoint 1204, the fifth endpoint 1205, the sixth endpoint 1206, the seventh endpoint 1207, the eighth endpoint 1208, and continues to execute step S902.

[0100] Step S902, determining whether the number of endpoints included in the initial edge curve is greater than a set number threshold, if not, executing step S903; if yes, executing step S904.

[0101] For example, the set number threshold may be 2. Therefore, if the number of endpoints included in the initial edge curve is greater than 2, it can be considered that the initial edge curve includes multiple discontinuous line segments; if the number of endpoints included in the initial edge curve is less than 2, it can be considered that the initial edge curve is a continuous curve without any disconnection.

[0102] Step S903: determine that the initial edge curve is a continuous line segment.

[0103] In some embodiments, if the initial edge curve is determined to be a continuous line segment, the initial edge curve of the target object in the next frame of image can be obtained through the image processing network model.

[0104] Step S904: determine whether the initial edge curve includes multiple discontinuous line segments.

[0105] After determining that the initial edge curve includes a plurality of discontinuous line segments and positions of a plurality of endpoints of the plurality of line segments, an adjacent point may be marked for each of the plurality of endpoints.

[0106] For example, taking any endpoint as the current endpoint, the process of marking adjacent points for the current endpoint can be performed by Fig.13 The process can include the following steps:

[0107] Step S3021, select any endpoint as the current endpoint.

[0108] Step S3022: Determine the distances between the current endpoint and the remaining endpoints except the current endpoint according to the positions of the multiple endpoints.

[0109] For example, Fig.12 As shown, there are eight endpoints in total. The second endpoint 1202 is selected as the current endpoint. The distances between the remaining endpoints except the current endpoint and another endpoint on the same line segment as the current endpoint and the current endpoint, i.e., the distance d between the second endpoint and the third endpoint, can be determined respectively. 23 , the distance d between the second endpoint and the fourth endpoint 24 , the distance d between the second endpoint and the fifth endpoint 25 , the distance d between the second endpoint and the sixth endpoint 26 , the distance d between the second endpoint and the seventh endpoint 27 , the distance d between the second endpoint and the eighth endpoint 28 , where the distance calculation formula is as follows: Among them, x i is the horizontal coordinate of the i-th endpoint, x j is the horizontal coordinate of the jth endpoint, y i is the ordinate of the i-th endpoint, y j is the ordinate of the jth endpoint, i∈(0, num], j∈(0, num], where num is the number of endpoints.

[0110] Step S3023: The remaining endpoints whose distances to the current endpoint are less than the set distance threshold are taken as candidate endpoints.

[0111] In some embodiments, the set distance threshold may be 80 pixels. If the current endpoint is the second endpoint, the distance d between the second endpoint and the third endpoint is 23 is 57 pixels, which is less than the set distance threshold. The distance d between the second endpoint and the third endpoint is 27 is 65 pixels, which is also less than the set distance threshold. The third endpoint and the seventh endpoint are both candidate endpoints for the second endpoint. If the current endpoint is the fourth endpoint, only the distance d between the fourth endpoint and the fifth endpoint is 45 It is 72 pixels, which is less than the set distance threshold. Therefore, there is only the fifth endpoint that is a candidate endpoint of the fourth endpoint.

[0112] In other embodiments, the set distance threshold may also be 95 pixels or other values. The set distance threshold varies according to the accuracy of image segmentation of the image containing the target object. For example, the higher the accuracy of image segmentation of the image containing the target object, the smaller the set distance threshold.

[0113] Step S3024: Mark the adjacent points of the current endpoint among the remaining endpoints according to the number of candidate endpoints.

[0114] For example, the process can be performed by Fig.14The process can include the following steps:

[0115] Step S1401, determine whether the number of candidate endpoints is 0, if so, execute step S1402; if not, execute step S1403.

[0116] Step S1402: Mark the remaining endpoints with the smallest distance from the current endpoint as adjacent points of the current endpoint.

[0117] For example, the set distance threshold may be 80 pixels. If there are eight endpoints in total and the current endpoint is the second endpoint, the distance d between the second endpoint and the third endpoint is 23 is the distance d between 94 pixels, the second endpoint and the fourth endpoint 24 is the distance d between 137 pixels, the second endpoint and the fifth endpoint 25 is 165 pixels, the distance d between the second endpoint and the sixth endpoint 26 is the distance d between 203 pixels, the second endpoint and the seventh endpoint 27 is the distance d between 97 pixels, the second endpoint and the eighth endpoint 28 If the number of pixels is 105, it can be concluded that the remaining endpoint with the smallest distance from the current endpoint is the third endpoint, and the third endpoint is marked as an adjacent point of the current endpoint.

[0118] Step S1403, determine whether the number of candidate endpoints is 1, if so, execute step S1404; if not, execute step S1405.

[0119] Step S1404: mark the candidate endpoint as an adjacent point of the current endpoint.

[0120] For example, the set distance threshold may be 80 pixels. If there are eight endpoints in total and the current endpoint is the second endpoint, the distance d between the second endpoint and the third endpoint is 23 is the distance d between 74 pixels, the second endpoint and the fourth endpoint 24 is the distance d between 137 pixels, the second endpoint and the fifth endpoint 25 is 165 pixels, the distance d between the second endpoint and the sixth endpoint 26 is the distance d between 203 pixels, the second endpoint and the seventh endpoint 27 is the distance d between 87 pixels, the second endpoint and the eighth endpoint 28 If the number of pixels is 105, the third endpoint can be marked as an adjacent point of the current endpoint.

[0121] Step S1405, respectively determine the slope of the connection line between each candidate endpoint and the current endpoint, and mark the candidate endpoint with the smallest slope difference between the connection line slope and the slope of the curve at the current endpoint of the initial edge curve as an adjacent point of the current endpoint.

[0122] For example, the set distance threshold may be 80 pixels. If there are eight endpoints in total and the current endpoint is the second endpoint, the distance d between the second endpoint and the third endpoint is 23 is the distance d between 74 pixels, the second endpoint and the fourth endpoint 24 is the distance d between 137 pixels, the second endpoint and the fifth endpoint 25 is 165 pixels, the distance d between the second endpoint and the sixth endpoint 26 is the distance d between 203 pixels, the second endpoint and the seventh endpoint 27 is the distance d between the 67 pixel points, the second endpoint and the eighth endpoint 28 105 pixels, where the distance between the second endpoint and the third endpoint and the distance between the second endpoint and the seventh endpoint are both less than the set distance threshold, the second endpoint and a point as close as possible to the second endpoint and located on the same line segment as the second endpoint can be used to determine the slope k0 of the curve at the current endpoint, and then the slopes k of the second endpoint and the third endpoint can be calculated respectively. 23 and the slope k between the second endpoint and the seventh endpoint 27 , if k0 is 1.2, k 23 is 1.6, k 27 is -0.2, the third endpoint can be marked as the adjacent point of the second endpoint.

[0123] Among them, each endpoint in the initial edge curve can be traversed as the current endpoint, and the adjacent point of each endpoint can be marked by the above method.

[0124] Step S303: taking the endpoints marked as adjacent points as target endpoints, performing curve fitting based on the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

[0125] In some embodiments, the moving least squares (MLS) method can be used to perform curve fitting, and a smooth curve can be generated by performing least squares fitting in a local neighborhood around the data point. The data point set of the curve to be fitted, i.e., the target endpoint and the adjacent points of the target endpoint, can be determined first. For each data point, the local neighborhood around it is determined according to the set neighborhood radius. For each data point and its neighborhood, the weight is calculated according to the distance or other geometric features for the subsequent least squares fitting. For each data point and its neighborhood, the least squares fitting is performed using the weight matrix. According to the result of the least squares fitting, the curve interpolation or smoothing operation can be performed.

[0126] Moving least squares is a variant of least squares that can be fitted in a weighted manner within a local neighborhood. Instead of minimizing all data points as a whole, it fits based on the local neighborhood of each data point. Specifically, for each data point, moving least squares selects a local neighborhood and uses weighted least squares to fit the data points within the neighborhood. This can accommodate the variability of the data in a local range and provide better fitting results.

[0127] In other embodiments, other curve fitting methods such as polynomial fitting or parametric curve fitting may also be used.

[0128] In a specific embodiment, a target object edge extraction process performed by a laparoscope processor may be as follows: Fig.15 As shown, the following steps are included:

[0129] Step S1501: Capture an image containing a target object through an image acquisition component.

[0130] Step S1502: performing image segmentation on the image containing the target object to obtain an initial edge curve of the target object.

[0131] Step S1503, determining whether the number of endpoints included in the initial edge curve is greater than a set number threshold, if not, executing step S1504; if yes, executing step S1505.

[0132] Step S1504, obtaining the initial edge curve of the target object in the next frame of image.

[0133] Step S1505, select any endpoint as the current endpoint.

[0134] Step S1506, based on the positions of the multiple endpoints, respectively determine the distances between the remaining endpoints except the current endpoint and the current endpoint, and use the remaining endpoints whose distances to the current endpoint are less than a set distance threshold as candidate endpoints.

[0135] Step S1507: Mark the adjacent points of the current endpoint according to the number of candidate endpoints.

[0136] If the number of candidate endpoints is 0, the remaining endpoint with the smallest distance to the current endpoint is marked as the adjacent point of the current endpoint; if the number of candidate endpoints is 1, the candidate endpoint is marked as the adjacent point of the current endpoint; if the number of candidate endpoints is greater than 1, the slope of the connecting line between each candidate endpoint and the current endpoint is determined respectively, and the candidate endpoint with the smallest slope difference between the connecting line slope and the slope of the curve at the current endpoint of the initial edge curve is marked as the adjacent point of the current endpoint.

[0137] Step S1508, determine whether there is an endpoint that has not been selected as the current endpoint, if so, execute step S1509, if not, execute step S1510.

[0138] Step S1509: updating the current endpoint according to any endpoint that is not selected as the current endpoint.

[0139] Step S1510: The endpoints marked as adjacent points are used as target endpoints, and curve fitting is performed based on the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

[0140] Based on the same inventive concept, the present application also provides a target object edge extraction device, such as Fig.16 As shown, the target object edge extraction device may include:

[0141] The edge curve extraction unit 1601 is used to perform image segmentation on an image containing a target object to obtain an initial edge curve of the target object;

[0142] An adjacent point determining unit 1602 is configured to mark an adjacent point for each of the multiple endpoints according to the positions of the multiple endpoints of the multiple line segments if the initial edge curve includes multiple discontinuous line segments;

[0143] The curve fitting unit 1603 is used to take the endpoints marked as adjacent points as target endpoints, and perform curve fitting based on the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

[0144] In an optional implementation manner, the adjacent point determining unit 1602 is specifically configured to:

[0145] It is determined whether the initial edge curve includes a plurality of discontinuous line segments according to the number of endpoints included in the initial edge curve.

[0146] In an optional implementation manner, the adjacent point determining unit 1602 is specifically configured to:

[0147] If the number of endpoints included in the initial edge curve is greater than a set number threshold, it is determined that the initial edge curve includes a plurality of discontinuous line segments.

[0148] In an optional implementation manner, the adjacent point determining unit 1602 is specifically configured to:

[0149] For any pixel point on the initial edge curve, determine whether the pixel point is an endpoint according to the pixel values ​​of the adjacent pixel points of the pixel point;

[0150] The number of pixels determined as endpoints is used as the number of endpoints included in the initial edge curve.

[0151] In an optional implementation, the edge curve extraction unit 1601 is specifically configured to:

[0152] The image is input into an image processing network model, and the image is segmented by the image processing network model to obtain an initial edge curve of the target object.

[0153] In an optional implementation manner, the adjacent point determining unit 1602 is specifically configured to:

[0154] For each of the multiple endpoints, perform the following operations respectively:

[0155] taking the endpoint as the current endpoint;

[0156] Determine, according to the positions of the multiple endpoints, the distances between the remaining endpoints except the current endpoint and the current endpoint respectively;

[0157] The remaining endpoints whose distances from the current endpoint are less than a set distance threshold are used as candidate endpoints;

[0158] According to the number of the candidate endpoints, adjacent points of the current endpoint are marked in the remaining endpoints.

[0159] In an optional implementation manner, the adjacent point determining unit 1602 is specifically configured to:

[0160] If there are multiple candidate endpoints, select a target candidate endpoint from the multiple candidate endpoints, and mark the target candidate endpoint as an adjacent point of the current endpoint;

[0161] If the number of the candidate endpoints is one, marking the candidate endpoint as an adjacent point of the current endpoint;

[0162] If there is no candidate endpoint, the remaining endpoint with the smallest distance from the current endpoint is marked as an adjacent point of the current endpoint.

[0163] In an optional implementation manner, the adjacent point determining unit 1602 is specifically configured to:

[0164] The slope of the connecting line between each candidate endpoint and the current endpoint is determined respectively, and the candidate endpoint with the smallest slope difference between the connecting line slope and the slope of the curve at the current endpoint of the initial edge curve is used as the target candidate endpoint.

[0165] In an optional implementation, the curve fitting unit 1603 is specifically configured to:

[0166] The slope of the connecting line between each candidate endpoint and the current endpoint is determined respectively, and the candidate endpoint with the smallest slope difference between the connecting line slope and the slope of the curve at the current endpoint of the initial edge curve is used as the target candidate endpoint.

[0167] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0168] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0171] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A laparoscope, characterized in that: including an image acquisition component and a processor; The image acquisition component is configured to acquire an image containing a target object; The processor is configured to: Performing image segmentation on the image to obtain an initial edge curve of the target object; If the initial edge curve includes a plurality of discontinuous line segments, marking an adjacent point for each of the plurality of endpoints according to positions of the plurality of endpoints of the plurality of line segments; The endpoints marked as adjacent points are used as target endpoints, and curve fitting is performed according to the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

2. The laparoscope according to claim 1, characterized in that: The processor is specifically configured to: It is determined whether the initial edge curve includes a plurality of discontinuous line segments according to the number of endpoints included in the initial edge curve.

3. The laparoscope according to claim 2, characterized in that: The processor is specifically configured to: If the number of endpoints included in the initial edge curve is greater than a set number threshold, it is determined that the initial edge curve includes a plurality of discontinuous line segments.

4. The laparoscope according to claim 2 or 3, characterized in that: The processor is specifically configured to: For any pixel point on the initial edge curve, determine whether the pixel point is an endpoint according to the pixel values ​​of the adjacent pixel points of the pixel point; The number of pixels determined as endpoints is used as the number of endpoints included in the initial edge curve.

5. The laparoscope according to claim 1, characterized in that: The processor is specifically configured to: The image is input into an image processing network model, and the image is segmented by the image processing network model to obtain an initial edge curve of the target object.

6. The laparoscope according to claim 1, characterized in that: The processor is specifically configured to: For each of the multiple endpoints, perform the following operations respectively: taking the endpoint as the current endpoint; Determine, according to the positions of the multiple endpoints, the distances between the remaining endpoints except the current endpoint and the current endpoint respectively; The remaining endpoints whose distances from the current endpoint are less than a set distance threshold are used as candidate endpoints; According to the number of the candidate endpoints, adjacent points of the current endpoint are marked in the remaining endpoints.

7. The laparoscope according to claim 6, characterized in that: The processor is specifically configured to: If there are multiple candidate endpoints, select a target candidate endpoint from the multiple candidate endpoints, and mark the target candidate endpoint as an adjacent point of the current endpoint; If the number of the candidate endpoints is one, marking the candidate endpoint as an adjacent point of the current endpoint; If there is no candidate endpoint, the remaining endpoint with the smallest distance from the current endpoint is marked as an adjacent point of the current endpoint.

8. The laparoscope according to claim 7, characterized in that: The processor is specifically configured to: The slope of the connecting line between each candidate endpoint and the current endpoint is determined respectively, and the candidate endpoint having the smallest slope difference between the connecting line slope and the slope of the curve at the current endpoint of the initial edge curve is used as the target candidate endpoint.

9. A method for extracting the edge of a target object, characterized in that: The method comprises: Performing image segmentation on an image containing a target object to obtain an initial edge curve of the target object; If the initial edge curve includes a plurality of discontinuous line segments, marking an adjacent point for each of the plurality of endpoints according to positions of the plurality of endpoints of the plurality of line segments; The endpoints marked as adjacent points are used as target endpoints, and curve fitting is performed according to the target endpoints and the adjacent points of the target endpoints to obtain a continuous edge curve of the target object.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed by a processor, the method described in claim 9 is implemented.

Images