An in-vivo tissue detection method, apparatus and system

Abstract

The application provides an in-vivo tissue detection method, device and system. The method comprises: when a detection probe contacts a specified type of tissue inside a target object, an endoscope collects an original image corresponding to the specified type of tissue inside the target object; position information of the detection probe and M candidate targets are obtained by identifying the original image, M being a positive integer greater than 1; based on the position information and regions where the M candidate targets are located, a to-be-processed target belonging to the specified type of tissue is selected from the M candidate targets; and a region where the to-be-processed target is located is prompted. Through the technical solution of the application, the accuracy of to-be-processed target detection can be improved, and the accuracy of target recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of medical processing technology, and in particular to a method, apparatus and system for in vivo tissue detection. Background Technology

[0002] Endoscopes are commonly used medical devices consisting of a beam guide structure and a set of lenses. After the endoscope enters the interior of the target object, it can be used to acquire raw images of the interior of the target object, and the target object can be examined and treated based on the raw images of the interior of the target object.

[0003] When it is necessary to examine and treat a specific type of tissue inside a target object, it is necessary to first acquire the original image of the specified type of tissue inside the target object through an endoscope, analyze the actual location of the specified type of tissue based on the original image, and then examine and treat the specified type of tissue.

[0004] However, analyzing the original image to determine the actual location of a specific type of tissue requires advanced professional knowledge and extensive practical experience. Inexperienced doctors may not be able to locate the actual location of the specified type of tissue. Even with the original image acquired by the endoscope, it may be impossible to determine the actual location of the specified type of tissue inside the target object based on the original image, making it impossible to conduct examinations and treatments. Summary of the Invention

[0005] This application provides an in vivo tissue detection method, the method comprising:

[0006] When the probe comes into contact with a specified type of tissue inside the target object, the endoscope acquires the original image corresponding to the specified type of tissue inside the target object;

[0007] The original image is identified to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1;

[0008] Based on the location information and the regions where the M candidate targets are located, select the target to be processed that belongs to the specified type of organization from the M candidate targets;

[0009] The system indicates the area where the target to be processed is located.

[0010] This application provides an in vivo tissue detection device, the device comprising:

[0011] The acquisition module is used to acquire raw images of the specified type of tissue inside the target object when the detection probe comes into contact with the target object.

[0012] The acquisition module is used to identify the original image to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1;

[0013] The determination module is used to filter out the target to be processed that belongs to the specified type of organization from the M candidate targets based on the location information and the area where the M candidate targets are located;

[0014] The prompting module is used to indicate the area where the target to be processed is located.

[0015] This application provides an in vivo tissue detection system, which includes: an endoscope, a main unit camera system, a detection probe, a specified type of tissue detector, and a display screen; wherein:

[0016] The detection probe is used to contact a specified type of tissue inside the target object;

[0017] The specified type tissue detector is used to send a successful detection response message to the host camera system when the detection probe comes into contact with the specified type of tissue inside the target object;

[0018] The endoscope is used to acquire raw images of the specified type of tissue inside the target object and send the raw images to the host camera system.

[0019] The host camera system is configured to, upon receiving the response information, determine that the detection probe has contacted a specified type of tissue inside the target object; identify the original image to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1; based on the position information and the regions where the M candidate targets are located, filter out the target to be processed that belongs to the specified type of tissue from the M candidate targets; and indicate the region where the target to be processed is located.

[0020] The display screen is used to overlay the region boundary of the target to be processed onto the original image.

[0021] As can be seen from the above technical solutions, in this embodiment, based on the location information of the detection probe and the area where M candidate targets are located, a target belonging to a specified type of tissue can be selected from the M candidate targets. The location of the target to be processed is the actual location of the specified type of tissue, thereby accurately knowing the actual location of the specified type of tissue inside the target object, quickly and accurately locating the actual location of the specified type of tissue, reminding the doctor to find the actual location of the specified type of tissue as soon as possible, and to examine and treat the specified type of tissue. In the above method, when determining the target to be processed belonging to the specified type of tissue by combining the location information of the detection probe, the target to be processed can be found more accurately, improving the accuracy of target detection and increasing the recognition accuracy rate. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0023] Figure 1 This is a flowchart illustrating an in vivo tissue detection method according to one embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the recurrent laryngeal nerve in one embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the structure of an in vivo tissue detection system according to one embodiment of this application;

[0026] Figure 4 This is a flowchart illustrating an in vivo tissue detection method according to one embodiment of this application;

[0027] Figure 5 This is a schematic diagram showing the position of the detection probe in one embodiment of this application;

[0028] Figure 6 This is a graph showing the allocation of pixel-level weights in one embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the attenuation curve in one embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the in vivo tissue detection device according to one embodiment of this application. Detailed Implementation

[0031] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0033] This application proposes an in vivo tissue detection method that can be applied to any type of device. See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of the in vivo tissue detection method, which may include:

[0034] Step 101: When the detection probe comes into contact with the specified type of tissue inside the target object, the endoscope acquires the original image corresponding to the specified type of tissue inside the target object.

[0035] Step 102: Identify the original image to obtain the location information of the detection probe and M candidate targets, where M is a positive integer greater than 1. For example, the original image can be input into the first target network model to obtain M candidate targets and the initial confidence score of each candidate target belonging to a specified type of organization. The original image can be input into the second target network model to obtain the location information of the detection probe.

[0036] Step 103: Based on the location information and the regions where the M candidate targets are located, filter out the targets belonging to the specified type of organization from the M candidate targets. For example, the region weight of each candidate target can be determined based on the location information; the initial confidence of the candidate target can be adjusted based on the region weight of the candidate target to obtain the target confidence of the candidate target; based on the target confidence of each candidate target, the targets belonging to the specified type of organization can be determined from the M candidate targets.

[0037] Step 104: Prompt the location of the target to be processed.

[0038] For example, when indicating the location of a target to be processed, the boundary of that target area can be overlaid on the original image. For instance, the outline of the target area can be overlaid on the original image, representing the boundary of the target area. Alternatively, the target color can be overlaid on the target area of ​​the original image, with the edge of the target color representing the boundary of the target area. Or, text information can be overlaid on the original image to explain the boundary of the target area. Or, a wireframe of the target can be overlaid on the original image, representing the boundary of the target area. Of course, these are just a few examples of indicating the location of a target, and there are no limitations, as long as the location of the target area can be distinguished.

[0039] In one possible implementation, in step 103, based on location information and the regions where the M candidate targets are located, the target to be processed belonging to a specified type of organization is selected from the M candidate targets. This may include, but is not limited to: obtaining the initial confidence scores of the M candidate targets from the original image; determining the regional weight of each candidate target based on the location information; adjusting the initial confidence score of the candidate target based on the regional weight of the candidate target to obtain the target confidence score of the candidate target; and determining the target to be processed belonging to the specified type of organization from the M candidate targets based on the target confidence score of each candidate target.

[0040] For example, the region weight of each candidate target is determined based on the location information, including but not limited to: determining the centroid of the detection probe based on the location information; determining the pixel-level weight of each pixel in the original image based on the centroid; wherein, the farther the distance between the pixel and the centroid, the smaller the pixel-level weight of the pixel, and the closer the distance between the pixel and the centroid, the larger the pixel-level weight of the pixel; and determining the region weight of the candidate target based on the pixel-level weight of each pixel in the candidate target.

[0041] In one possible implementation, determining the centroid of the detection probe based on the location information may include, but is not limited to: if the location information includes the coordinates of the four vertices of the quadrilateral corresponding to the detection probe, the centroid of the quadrilateral can be determined based on these four vertex coordinates; or, if the location information includes the coordinates of the three vertices of the triangle corresponding to the detection probe, the centroid of the triangle can be determined based on these three vertex coordinates; or, if the location information includes the coordinates of multiple circumferential points of the circle corresponding to the detection probe, the center of the circle can be determined based on these multiple circumferential point coordinates.

[0042] In one possible implementation, determining the pixel-level weight of each pixel in the original image based on the centroid may include, but is not limited to: for each pixel in the original image, determining the normalized distance between the pixel and the centroid based on the pixel's position coordinates and the maximum target distance; and determining the pixel-level weight of the pixel based on the normalized distance and the configured weight allocation modulation parameters.

[0043] The target maximum distance represents the maximum distance between a pixel in the original image and the centroid, and the target maximum distance can be determined based on the width and height of the original image and the coordinates of the centroid.

[0044] In one possible implementation, determining the region weight of the candidate target based on the pixel-level weight of each pixel in the candidate target may include, but is not limited to: determining the average weight value corresponding to the pixel-level weight of each pixel in the candidate target; and determining the region weight of the candidate target based on the average weight value. For example, the average weight value can be used as the region weight of the candidate target.

[0045] In one possible implementation, determining the target to be processed from M candidate targets that belongs to a specified type of organization may include, but is not limited to: selecting the candidate target with the highest target confidence from the M candidate targets; if the target confidence of the candidate target is greater than a preset threshold, then the candidate target is determined as a target to be processed belonging to the specified type of organization; if the target confidence of the candidate target is not greater than the preset threshold, then the candidate target is not determined as a target to be processed belonging to the specified type of organization, i.e., there is no target to be processed belonging to the specified type of organization. Alternatively, based on the target confidence corresponding to the M candidate targets, all candidate targets with a target confidence greater than a preset threshold are determined as targets to be processed belonging to the specified type of organization.

[0046] In another possible implementation, the candidate target with the highest target confidence can be selected from M candidate targets. If the target confidence of the candidate target is greater than a first threshold and less than a second threshold, then the candidate target is identified as a target to be processed belonging to a specified type of organization. In this case, the target to be processed is a suspected target, and a prompt message is issued indicating that the target to be processed may belong to the specified type of organization. If the target confidence of the candidate target is greater than or equal to the second threshold, then the candidate target is identified as a target to be processed belonging to the specified type of organization. In this case, the target to be processed is a confirmed target, and the region boundary of the target to be processed is overlaid on the original image. If the target confidence of the candidate target is not greater than the first threshold, then the candidate target may not be identified as a target to be processed belonging to the specified type of organization, i.e., there is no target to be processed belonging to the specified type of organization.

[0047] In one possible implementation, after the initial confidence scores of the M candidate targets are obtained from the original image, before adjusting the initial confidence scores of the candidate targets based on their region weights, if the original image does not include the detection probe, the region weight of each candidate target can be determined based on the historical weight value of each pixel in the original image; wherein, for each pixel in the original image, the historical weight value of the pixel can be determined based on the pixel-level weight of the pixel in a reference image, and the reference image is the original image including the detection probe.

[0048] For example, determining the historical weight value of a pixel based on its pixel-level weight in the reference image may include, but is not limited to: determining the attenuation parameter corresponding to the original image based on the number of interval frames and the configured attenuation modulation parameter; wherein, the number of interval frames is the number of images between the reference image and the original image; and determining the historical weight value of the pixel based on the attenuation parameter and its pixel-level weight.

[0049] As can be seen from the above technical solutions, in this embodiment, after obtaining M candidate targets and the initial confidence level of each candidate target belonging to a specified type of tissue based on the original image, the initial confidence level can be adjusted based on the position information of the detection probe to obtain the target confidence level of the candidate targets. Based on the target confidence level of each candidate target, the target to be processed belonging to the specified type of tissue is determined. The position of the target to be processed is also the actual position of the specified type of tissue, thereby accurately knowing the actual position of the specified type of tissue inside the target object, quickly and accurately locating the actual position of the specified type of tissue, reminding the doctor to find the actual position of the specified type of tissue as soon as possible, and to examine and treat the specified type of tissue. In the above method, by adjusting the initial confidence level in combination with the position information of the detection probe, a more accurate and reliable target confidence level is obtained. When determining the target to be processed belonging to the specified type of tissue based on the target confidence level, the target to be processed can be found more accurately, improving the accuracy of target detection and increasing the recognition accuracy.

[0050] The technical solutions described above in this application will be explained below with reference to specific embodiments.

[0051] Endoscopes are commonly used medical devices. When it is necessary to examine and treat a specific type of tissue inside a target object, the endoscope can be inserted into the target object to acquire raw images of the specified type of tissue inside the target object. Based on the analysis of the raw images, the actual location of the specified type of tissue can be determined, and then the specified type of tissue can be examined and treated.

[0052] For example, the specified type of tissue can be any tissue within the target object, that is, the in vivo tissue of the target object (such as a patient awaiting a specimen). For instance, the specified type of tissue can be neural tissue or organ tissue, etc. Nerve tissue can include, but is not limited to, the recurrent laryngeal nerve, olfactory nerve, optic nerve, trigeminal nerve, and facial nerve, while organ tissue can include, but is not limited to, liver tissue and kidney tissue. Of course, the above are just a few examples of specified type tissues, and the type of specified type tissue is not limited. For ease of description, in the following embodiments, the recurrent laryngeal nerve will be used as an example of the specified type of tissue.

[0053] The recurrent laryngeal nerve is the main motor nerve of the laryngeal muscles. Its transverse diameter is (1.95±0.35) mm, and its surface is translucent and whitish in color. (See also...) Figure 2 The diagram shown illustrates the recurrent laryngeal nerve. During thyroid surgery, recurrent laryngeal nerve injury is a serious complication, primarily manifesting as hoarseness due to unilateral recurrent laryngeal nerve injury and respiratory distress or even suffocation due to bilateral recurrent laryngeal nerve injury. Therefore, during thyroid surgery, it is crucial to be able to determine the actual location of the recurrent laryngeal nerve to avoid damage. Based on this requirement, an endoscope can be inserted into the target area to acquire raw images of the recurrent laryngeal nerve within the target area. These raw images can then be analyzed to determine the actual location of the recurrent laryngeal nerve, thus preventing damage.

[0054] However, when analyzing the actual location of the recurrent laryngeal nerve based on the original images, multiple candidate targets may be identified, requiring the selection of the recurrent laryngeal nerve from among them. However, selecting the recurrent laryngeal nerve from multiple candidate targets requires advanced professional knowledge and extensive practical experience. For less experienced physicians, accurately locating the recurrent laryngeal nerve from multiple candidate targets may be impossible, even with the original images acquired endoscopically, as it may not be possible to determine the actual location of the recurrent laryngeal nerve based on these images alone.

[0055] During thyroid surgery, a recurrent laryngeal nerve detector can be used to locate the actual position of the recurrent laryngeal nerve. This detector uses a probe to locate the nerve. For example, the probe stimulates the recurrent laryngeal nerve to generate nerve impulses, which are then transmitted to the muscles that innervate it, producing electromyographic (EMG) signals. These signals create an EMG waveform and accompanying sound, allowing the detector to assess the continuity of the recurrent laryngeal nerve and thus protect it. However, the recurrent laryngeal nerve detector may not be able to accurately locate the nerve.

[0056] In response to the above findings, this application embodiment can select the recurrent laryngeal nerve from multiple candidate targets based on the detection results of the endoscope and the recurrent laryngeal nerve detector, accurately locating the actual position of the recurrent laryngeal nerve and avoiding damage to it. It can also link the detection results of the endoscope and the recurrent laryngeal nerve detector, fusing the information from multiple candidate targets in the original image based on the position information of the detection probe, thereby improving the recognition accuracy in difficult situations (such as dark areas, subfascial areas, etc.).

[0057] To accurately locate the actual position of the recurrent laryngeal nerve, this application proposes an in vivo tissue detection system, see [link to relevant documentation]. Figure 3 The diagram shown illustrates the structure of an in vivo tissue detection system. This system may include an endoscope with a camera (for accessing the target tissue), a cold light source for illumination, a host camera system for image processing, a detection probe for accessing the target tissue, a display screen for real-time imaging, and a recurrent laryngeal nerve detector (i.e., a specific type of tissue detector) for thyroid surgery. Of course, Figure 3 This is merely an example of an in vivo tissue detection system, and its structure is not limited.

[0058] For example, an endoscope acquires raw images of the interior of a target object (such as a patient or other subject) and inputs these raw images into a host camera system. A cold light source provides the light source for the endoscope; that is, illumination light is emitted from the tip of the endoscope, enabling the endoscope to acquire relatively clear raw images of the interior of the target object.

[0059] The detection probe is used to contact the recurrent laryngeal nerve inside the target object. The detection probe stimulates the recurrent laryngeal nerve to generate nerve impulses, which are then transmitted to the innervated muscles to generate electromyographic signals, forming an electromyographic waveform and prompting sound.

[0060] When the detection probe contacts the recurrent laryngeal nerve inside the target object, the recurrent laryngeal nerve detector can sense this and send a successful detection response to the main camera system, indicating that the detection probe has contacted the recurrent laryngeal nerve inside the target object. Alternatively, if the detection probe does not contact the recurrent laryngeal nerve inside the target object, the recurrent laryngeal nerve detector will not send a successful detection response to the main camera system.

[0061] When the host camera system receives a response from the recurrent laryngeal nerve detector, it determines that the detection probe has contacted the recurrent laryngeal nerve inside the target object. If the host camera system does not receive a response from the recurrent laryngeal nerve detector, it determines that the detection probe has not contacted the recurrent laryngeal nerve inside the target object. The host camera system can acquire raw images from the endoscope and process them; the processing procedure is described in subsequent embodiments.

[0062] The main camera system can display the original image on the screen and overlay the boundary of the recurrent laryngeal nerve region onto the original image to indicate the actual location of the recurrent laryngeal nerve. Users can observe the original image in real time on the screen and view the actual location of the recurrent laryngeal nerve displayed overlaid on the original image.

[0063] In the above application scenarios, this application proposes an in vivo tissue detection method, which can be applied to a host camera system. See [link to relevant documentation]. Figure 4 The diagram shown is a flowchart of the method, which may include:

[0064] Step 401: The host camera system acquires the original image corresponding to the specified type of organization inside the target object.

[0065] For example, an endoscope can acquire raw images of a specified type of tissue (such as the recurrent laryngeal nerve) inside a target object and input the raw images to a host camera system, so that the host camera system can obtain raw images corresponding to the specified type of tissue inside the target object.

[0066] Step 402: The host camera system identifies the original image to obtain M candidate targets and the initial confidence score of each candidate target belonging to a specified type of organization. For example, the original image is input into the first target network model to obtain M candidate targets and the initial confidence score of each candidate target belonging to a specified type of organization. M can be a positive integer greater than 1, and the value of M can be configured empirically without restriction.

[0067] For example, before step 402, a network model can be pre-configured as a first initial network model. The first initial network model can be a deep learning-based convolutional neural network model or other types of network models; there are no restrictions on the type of the first initial network model. For instance, the first initial network model can consist of an encoder network and a decoder network. The encoder network is a classification network composed of convolutional layers and downsampling layers. The decoder network is used to map the semantic recognition features learned by the encoder network to the pixel space to obtain the classification result, and it consists of convolutional layers and upsampling layers.

[0068] The network structure of the first initial network model may include, but is not limited to, FCN model, SegNet model, RefineNet model, U-Ner model, DeepLab model, etc., without any restrictions.

[0069] For example, prior to step 402, a large number of sample images can be acquired, and the user can label the information of each sample image. The sample images are images corresponding to a specified type of tissue within the target object. The endoscope acquires sample images of the specified type of tissue within the target object, and the sample images are input to the host camera system, which then acquires the sample images of the specified type of tissue within the target object.

[0070] For each sample image, the user can assign a label to it. This label indicates whether each pixel in the sample image belongs to a specified type of organization. For example, for each pixel, if it belongs to a specified type of organization, a first value can be assigned to it to indicate that the pixel belongs to the specified type of organization; if it does not belong to the specified type of organization, a second value can be assigned to it to indicate that the pixel does not belong to the specified type of organization.

[0071] For example, before step 402, sample images and their label information can be input into the first initial network model to train the first initial network model. In this embodiment, this training process is not limited, and the trained network model is recorded as the first target network model.

[0072] The function of the first target network model is to identify M candidate targets belonging to a specified type of organization and output the initial confidence level of the M candidate targets belonging to the specified type of organization.

[0073] For example, in step 402, after obtaining the original image, the host camera system can input the original image into the first target network model. The first target network model processes the original image to obtain multiple initial targets and the initial confidence score of each initial target belonging to a specified type of organization. The processing procedure of the first target network model is not limited. Then, all initial targets are sorted in descending order of initial confidence score, and the top M initial targets are selected as candidate targets. The M candidate targets and the initial confidence score of each candidate target belonging to the specified type of organization are output.

[0074] In summary, after inputting the original image into the first target network model, the host camera system can obtain M candidate targets and the initial confidence score of each candidate target belonging to a specified type of organization. For example, let the target set be denoted as {A}, the number of elements in the target set be denoted as M, and after sorting according to the initial confidence score, the first element in the target set is the candidate target with the highest initial confidence score, denoted as A1, the second element is the candidate target with the second highest initial confidence score, denoted as A2, and so on, with the last element being the candidate target with the lowest initial confidence score, denoted as A...M .

[0075] In one possible implementation, the host camera system may include a specified type of tissue segmentation unit (such as a recurrent laryngeal nerve segmentation unit), which may perform step 402. That is, the specified type of tissue segmentation unit may input the original image into the first target network model to obtain M candidate targets and the initial confidence of each candidate target belonging to the specified type of tissue.

[0076] Step 403: The host camera system determines whether it has received response information from a specified type of tissue detector (such as a recurrent laryngeal nerve detector). If yes, proceed to step 404; otherwise, proceed to step 406.

[0077] If the host camera system receives a response from the tissue detector of a specified type, it determines that the detection probe has contacted the specified type of tissue inside the target object. In this case, the original image may include the detection probe, and step 404 can be executed. Clearly, when the detection probe contacts the specified type of tissue inside the target object, if an original image corresponding to the specified type of tissue inside the target object is acquired via an endoscope, the original image may include the detection probe, and step 404 is executed in this case.

[0078] If the host camera system does not receive a response from the designated tissue detector, it is determined that the detection probe has not contacted the designated tissue inside the target object. In this case, the original image does not include the detection probe, and step 406 can be executed. Clearly, if an original image corresponding to the designated tissue inside the target object is acquired via an endoscope when the detection probe has not contacted the target object, the original image does not include the detection probe, and step 406 is executed in this case.

[0079] In one possible implementation, the host camera system may include a communication unit for communicating with a specified type of tissue detector. The communication unit may perform step 403, that is, the communication unit may receive response information sent by the specified type of tissue detector. In this way, the communication unit can determine whether it has received the response information sent by the specified type of tissue detector.

[0080] Step 404: The host camera system recognizes the original image to obtain the position information of the detection probe. For example, the original image is input into the second target network model to obtain the position information of the detection probe.

[0081] For example, before step 404, a network model can be pre-configured as a second initial network model. The second initial network model can be a deep learning-based convolutional neural network model or other types of network models. There is no restriction on the type of the second initial network model.

[0082] The network structure of the second initial network model may include, but is not limited to, Fast R-CNN model, YOLOv1-v4 model, SSD model, etc. There are no restrictions on the network structure of the second initial network model.

[0083] For example, prior to step 404, a large number of sample images can be acquired, and the user can label the information of each sample image. The sample images can be images corresponding to a specific type of tissue inside the target object. For instance, when the detection probe contacts a specific type of tissue inside the target object, the endoscope acquires a sample image of the specified type of tissue inside the target object, and inputs the sample image to the host camera system, which then acquires the sample image of the specified type of tissue inside the target object.

[0084] For each sample image, the user can assign a label to the image, which indicates the location of the detection probe within that image. Unlike the first target network model, which is based on pixel-level target segmentation, the second target network model only needs to perform regional target recognition. The recognition result is the location information of the region where the detection probe is located. Therefore, when assigning labels to sample images, only the location information of the detection probe within that sample image needs to be assigned.

[0085] In calibrating the position information of the detection probe in the sample image, the detection probe can correspond to a quadrilateral. In this case, only the coordinates of the four vertices of the quadrilateral corresponding to the detection probe need to be calibrated. See [link to documentation]. Figure 5 The image shows a schematic diagram of the detection probe's position, illustrating the quadrilateral corresponding to the probe. The coordinates of the four vertices of this quadrilateral need to be calibrated. Alternatively, when calibrating the probe's position in the sample image, the probe can correspond to a triangle; in this case, only the coordinates of the three vertices of the triangle need to be calibrated. Or, the probe can correspond to a circle; in this case, only the coordinates of multiple points around the circumference of the circle (i.e., the coordinates of each pixel on the circumference) need to be calibrated. Or, the probe can correspond to a circle; in this case, only the center point and radius of the circle need to be calibrated. Of course, these are just a few examples; other shapes besides quadrilaterals, triangles, and circles are possible and there are no restrictions.

[0086] For example, before step 404, sample images and their label information can be input into the second initial network model to train it. This embodiment does not limit this training process, and the trained network model is referred to as the second target network model. The function of the second target network model can be to identify the location information of the detection probe.

[0087] For example, in step 404, after the host camera system obtains the original image, it can input the original image to the second target network model, which will process the original image to obtain the position information of the detection probe. The processing procedure of the second target network model is not limited.

[0088] In summary, after inputting the original image into the second target network model, the host camera system can obtain the position information of the detection probe. For example, if the label information of the sample image represents the coordinates of the four vertices of the quadrilateral corresponding to the detection probe, then in step 404, the position information obtained by the host camera system is the coordinates of the four vertices of the quadrilateral corresponding to the detection probe. Alternatively, if the label information of the sample image represents the coordinates of the three vertices of the triangle corresponding to the detection probe, then in step 404, the position information obtained by the host camera system is the coordinates of the three vertices of the triangle corresponding to the detection probe. Or, if the label information of the sample image represents the coordinates of multiple circumferential points of the circle corresponding to the detection probe, then in step 404, the position information obtained by the host camera system is the coordinates of multiple circumferential points of the circle corresponding to the detection probe.

[0089] In one possible implementation, the host camera system may include a detection probe recognition unit, which may perform step 404. That is, the detection probe recognition unit may input the original image into the second target network model to obtain the position information of the detection probe.

[0090] Step 405: The host camera system determines the region weight of each candidate target based on the position information of the detection probe. For example, for M candidate targets, after obtaining the position information of the detection probe, the region weight of each candidate target can be determined based on this position information. For instance, the following steps can be used to determine the region weight of each candidate target; however, these steps are merely examples and are not intended to be limiting.

[0091] Step 4051: The host camera system determines the centroid of the detection probe based on the location information.

[0092] For example, if the location information includes the coordinates of the four vertices of the quadrilateral corresponding to the detection probe, then the centroid of the quadrilateral can be determined based on these four vertex coordinates. In other words, if the location information is the coordinates of the four vertices of the quadrilateral surrounding the detection probe, then the centroid is the centroid of the quadrilateral.

[0093] For example, if the location information includes the coordinates of the three vertices of the triangle corresponding to the detection probe, then the centroid of the triangle can be determined based on these three vertex coordinates. In other words, if the location information is the coordinates of the three vertices of the triangle surrounding the detection probe, then the centroid is the centroid of the triangle.

[0094] For example, if the location information includes the coordinates of multiple circumferential points of the circle corresponding to the detection probe, the center of the circle can be determined as the centroid based on these multiple circumferential point coordinates. In other words, if the location information is the circular mask information surrounding the detection probe, then the centroid is the center of the circle.

[0095] For ease of description, taking the location information including the coordinates of the four vertices of a quadrilateral as an example, the centroid of the detection probe is determined by the following formula. The method for determining the centroid in other cases is similar and will not be repeated here.

[0096]

[0097]

[0098] In the above formula, (x i ,y i (x) represents the coordinates of the four vertices of the quadrilateral. The value of i can be 1, 2, 3, or 4, and can be arranged in a counter-clockwise direction. For example, when i is 1, it corresponds to the coordinates of the top-left vertex; when i is 2, it corresponds to the coordinates of the bottom-left vertex; when i is 3, it corresponds to the coordinates of the bottom-right vertex; and when i is 4, it corresponds to the coordinates of the top-right vertex. g ,y g ) represents the coordinates of the centroid of the detection probe.

[0099] Step 4052: The host camera system determines the pixel-level weight of each pixel in the original image based on the centroid. For example, after obtaining the centroid of the detection probe, weights are assigned around the centroid. The farther the pixel is from the centroid, the smaller its pixel-level weight, until the pixel-level weight is 0. The closer the pixel is to the centroid, the larger its pixel-level weight.

[0100] For example, in order to determine the pixel-level weight of each pixel in the original image, Gaussian weight, piecewise linear weight, non-linear weight, etc. can be used. There is no restriction on the method of determining the weight, as long as the distance between the pixel and the centroid is greater, the pixel-level weight of the pixel is smaller.

[0101] For example, for each pixel in the original image, the pixel-level weight of that pixel can be determined by the following steps. Of course, the following steps are just an example, and there are no restrictions on the determination method.

[0102] Step S11: The host camera system determines the maximum target distance based on the width and height of the original image and the coordinates of the centroid. The maximum target distance represents the maximum distance between a pixel in the original image and the centroid.

[0103] For example, the maximum target distance between a pixel and its centroid can be determined using the following formula:

[0104]

[0105] In the above formula, L max The value is used to represent the maximum distance to the target, (x,y) is used to represent the coordinates of the centroid of the detection probe, W is used to represent the width of the original image, and H is used to represent the height of the original image.

[0106] Step S12: For each pixel in the original image, the host camera system determines the normalized distance between the pixel and the centroid based on the pixel's position coordinates and the target's maximum distance.

[0107] For example, the normalized distance between a pixel and its centroid can be determined using the following formula:

[0108]

[0109] In the above formula, L i,j The coordinates (x, y) represent the normalized distance between pixel (i, j) and the centroid, where i represents the x-coordinate of pixel (i, j), j represents the y-coordinate of pixel (i, j), and (x, y) represent the coordinates of the centroid of the detection probe. max Used to indicate the maximum distance to the target.

[0110] Step S13: For each pixel in the original image, the host camera system determines the pixel-level weight of the pixel based on the normalized distance corresponding to the pixel and the configured weight allocation modulation parameters.

[0111] For example, this embodiment designs an inverse S-curve function as the weight determination function. The inverse S-curve function can be found in the following formula, and the pixel-level weights of pixels can be determined using the following formula:

[0112]

[0113] In the above formula, ω i,j The pixel-level weights of pixel (i,j) are used to represent the pixel-level weights. a, b, c, d, e, and f represent the configured weight allocation modulation parameters, which can be configured empirically. L i,j Used to represent the normalized distance between pixel (i,j) and the centroid.

[0114] For example, regarding the weight allocation modulation parameters a, b, c, d, e, and f, different combinations of weight allocation modulation parameters correspond to different pixel-level weights at different distances. For instance, one possible combination of weight allocation modulation parameters could be: a = 1.5, b = 1, c = 1, d = 1, e = 1, f = 0.75. Of course, the above combination of weight allocation modulation parameters is just an example, and there are no restrictions on this combination.

[0115] Under the above combination of weighted modulation parameters, the inverse S-shaped curve function can be found in [reference needed]. Figure 6 As shown, that is, using Figure 6 Determine the pixel-level weights of each pixel. Figure 6 A graph representing the pixel-level weight distribution of pixels. Figure 6 In the diagram, the horizontal axis represents the normalized distance, and the vertical axis represents the pixel-level weights.

[0116] Step 4053: For each candidate target, the host camera system determines the region weight of the candidate target based on the pixel-level weight of each pixel in the candidate target. For example, it determines the average weight value corresponding to the pixel-level weight of each pixel in the candidate target, and determines the region weight of the candidate target based on the average weight value. The average weight value can be used as the region weight of the candidate target.

[0117] In one possible implementation, the host camera system may include a region weight allocation unit, which performs step 405, that is, for each of the M candidate targets, the region weight allocation unit determines the region weight of the candidate target based on the position information of the detection probe.

[0118] In one possible implementation, step 407 may be performed after step 405.

[0119] Step 406: The host camera system determines the region weight of each candidate target based on the historical weight value of each pixel in the original image. For example, for each candidate target, the region weight of the candidate target is determined based on the historical weight value of each pixel in the candidate target. For example, the following steps can be used to determine the region weight of the candidate target. Of course, the following steps are just examples and are not limited thereto.

[0120] Step 4061: The host camera system determines the reference image corresponding to the original image, and the reference image is the original image including the detection probe, that is, the reference image is the original image corresponding to the response information.

[0121] For example, determine whether the first frame preceding the original image includes the detection probe (e.g., if the first frame receives a response from a tissue detector of a specified type, then the first frame includes the detection probe; otherwise, the first frame does not). If yes, the first frame can be used as the reference image corresponding to the original image. If no, determine whether the second frame preceding the original image includes the detection probe. If yes, the second frame can be used as the reference image corresponding to the original image. If no, determine whether the third frame preceding the original image includes the detection probe, and so on, until a reference image corresponding to the original image is found.

[0122] Step 4062: The host camera system determines the attenuation parameters corresponding to the original image based on the number of interval frames and the configured attenuation modulation parameters. The number of interval frames is the number of images between the reference image and the original image.

[0123] For example, when the weight value of the original image in the current frame needs to be determined based on the pixel-level weight of the reference image, the attenuation parameter corresponding to the original image needs to be determined first. For example, linear attenuation, non-linear attenuation, or other methods can be used to determine the attenuation parameter corresponding to the original image, and there is no restriction on the method of determination.

[0124] For example, one feasible way to calculate the attenuation parameter can be seen in the following formula.

[0125] d = exp(-δ*T)

[0126] In the above formula, d represents the attenuation parameter, δ represents the configured attenuation modulation parameter, which can be configured based on experience without restriction, and T represents the number of interval frames, that is, the number of images between the reference image and the original image (the original image of the current frame), i.e., the interval between the two image frames.

[0127] For example, when δ is 0.02, see [reference needed]. Figure 7The figure shows a schematic diagram of the attenuation curve of the attenuation parameter d. The horizontal axis represents the number of interval frames T, and the vertical axis represents the attenuation parameter d.

[0128] Step 4063: For each pixel in the original image, the host camera system determines the historical weight value of the pixel based on the attenuation parameter and the pixel-level weight of the pixel in the reference image.

[0129] For example, the reference image is the original image including the detection probe. The pixel-level weight of each pixel in the reference image can be determined in the manner of steps 404-405. Therefore, in step 4063, the historical weight value of the pixel can be determined based on the pixel-level weight of the pixel in the reference image.

[0130] For example, for pixel (i,j) in the original image, based on the attenuation parameter d, the historical weight value of pixel (i,j) can be determined as follows: if the pixel-level weight ω of pixel (i,j) in the reference image is... i,j If the value is greater than 1, the historical weight value of pixel (i,j) in the original image can be determined using the following formula. Alternatively, if we consider the pixel-level weight ω of pixel (i,j) in the reference image... i,j If the value is less than 1, the historical weight value of pixel (i,j) in the original image can be determined using the following formula. Obviously, as the number of frames accumulates over time, The weights will tend to 1 until new response information is received and the weights are recalculated.

[0131] Step 4064: For each candidate target, the host camera system determines the region weight of the candidate target based on the historical weight value of each pixel in the candidate target. For example, it determines the average weight value corresponding to the historical weight value of each pixel in the candidate target, and determines the region weight of the candidate target based on the average weight value. For example, the average weight value can be used as the region weight of the candidate target.

[0132] In one possible implementation, the host camera system may include a region weight allocation unit, which performs step 406, that is, for each candidate target, the region weight allocation unit determines the region weight of the candidate target based on the historical weight value of each pixel in the original image.

[0133] In one possible implementation, step 407 may be performed after step 406.

[0134] Step 407: For each candidate target, the host camera system adjusts the initial confidence of the candidate target based on the regional weight of the candidate target to obtain the target confidence of the candidate target.

[0135] For example, after obtaining M candidate targets and the initial confidence level of each candidate target belonging to a specified type of organization, the initial confidence level of each candidate target can be adjusted using the following formula to obtain the target confidence level of that candidate target: When α new_k When >100%, then α new_k =100%. In the above formula, α k Let represent the initial confidence level of the k-th candidate target. Let α represent the region weight of the k-th candidate target. new_k This represents the target confidence level of the k-th candidate target.

[0136] In summary, for each candidate target, the host camera system can obtain the target confidence level corresponding to that candidate target. That is, the host camera system can obtain M candidate targets and the target confidence level of each candidate target belonging to a specified type of organization. For example, let the target set be denoted as {S}, and the number of elements in the target set {S} be denoted as M. After sorting the target confidence levels from highest to lowest, the first element in the target set {S} is the candidate target with the highest target confidence level, denoted as S1; the second element is the candidate target with the second highest target confidence level, denoted as S2; and so on, with the last element being the candidate target with the lowest target confidence level, denoted as S... M .

[0137] In one possible implementation, the host camera system may further include a target confidence correction unit, which may perform step 407. That is, for each candidate target, the target confidence correction unit may adjust the initial confidence of the candidate target based on the regional weight of the candidate target to obtain the target confidence of the candidate target, i.e., obtain the target confidence of each candidate target.

[0138] Step 408: Based on the target confidence score of each candidate target, the host camera system determines the target to be processed from the M candidate targets that belongs to the specified type of organization. The target to be processed may be a confirmed target, meaning it belongs to the specified type of organization, or it may be a suspected target, meaning it may or may not belong to the specified type of organization, requiring the user to analyze whether the target to be processed belongs to the specified type of organization.

[0139] For example, in some application scenarios, there can be at most one target to be processed belonging to a specified type of tissue. For instance, when the specified type of tissue is the recurrent laryngeal nerve, since there are only two recurrent laryngeal nerves located on both sides of the trachea, only one recurrent laryngeal nerve can be seen in the original image. Both recurrent laryngeal nerves exist in the possible rows of the original image that do not exist. Therefore, there can be at most one target to be processed belonging to the specified type of tissue.

[0140] In this scenario, the host camera system selects the candidate target with the highest target confidence score from among the M candidate targets, based on the target confidence score of each candidate target, and determines whether the candidate target with the highest target confidence score belongs to a specified type of organization. For example, if the target confidence score of the candidate target is greater than a preset threshold, it can be determined that the candidate target belongs to a specified type of organization; otherwise, it can be determined that the candidate target does not belong to a specified type of organization.

[0141] For example, in some application scenarios, there are at most P targets belonging to a specified type of organization, where P is a positive integer greater than 1. In this case, based on the target set {S} sorted in descending order of target confidence, the host camera system selects the top P candidate targets from the M candidate targets based on the target confidence of each candidate target, and determines whether the P candidate targets belong to the specified type of organization. For instance, for each of the P candidate targets, if the target confidence of the candidate target is greater than a preset threshold, then the candidate target is determined to be a target belonging to the specified type of organization; otherwise, the candidate target is determined not to be a target belonging to the specified type of organization.

[0142] For example, in some application scenarios, the number of pending targets belonging to a specified type of organization is variable, meaning the number of pending targets is uncertain. In this case, the host camera system determines whether a candidate target belongs to the specified type of organization based on the target confidence score of each candidate target. For instance, for each of the M candidate targets, if the target confidence score of the candidate target is greater than a preset threshold, then the candidate target can be determined to belong to the specified type of organization; otherwise, it can be determined that the candidate target does not belong to the specified type of organization.

[0143] For ease of description, let's take the example that there is at most one pending target belonging to a specified type of organization. That is, we only need to determine whether the candidate target with the highest target confidence belongs to the specified type of organization.

[0144] In one possible implementation, the candidate target with the highest target confidence can be selected from M candidate targets, i.e., the first element in the target set {S}, denoted as candidate target S1. If the target confidence of candidate target S1 is greater than a preset threshold, then candidate target S1 is determined as a target to be processed belonging to a specified type of organization, and the region boundary of candidate target S1 is superimposed and displayed on the original image. If the target confidence of candidate target S1 is not greater than the preset threshold, then candidate target S1 is not determined as a target to be processed belonging to a specified type of organization; that is, there is currently no target to be processed belonging to a specified type of organization.

[0145] In another possible implementation, the candidate target with the highest target confidence can be selected from M candidate targets, i.e., the first element in the target set {S}, denoted as candidate target S1. If the target confidence of candidate target S1 is greater than a first threshold and less than a second threshold, then candidate target S1 can be identified as a target to be processed belonging to a specified type of organization. In this case, the target to be processed is a suspected target, and a prompt message needs to be issued for candidate target S1, indicating that candidate target S1 may belong to the specified type of organization. If the target confidence of candidate target S1 is greater than or equal to the second threshold, then candidate target S1 can be identified as a target to be processed belonging to the specified type of organization. In this case, the target to be processed is a confirmed target, and the region boundary of candidate target S1 is superimposed and displayed on the original image. If the target confidence of candidate target S1 is not greater than the first threshold, then candidate target S1 can not be identified as a target to be processed belonging to the specified type of organization; that is, there is currently no target to be processed belonging to the specified type of organization.

[0146] For example, a first threshold λ1 and a second threshold λ2 can be pre-configured, where the first threshold λ1 is less than the second threshold λ2. If the target confidence S of candidate target S1 is... max If the value is greater than or equal to the second threshold λ2, then the candidate target S1 is determined as a target to be processed belonging to the specified type of organization. The target to be processed is the confirmed target, that is, the candidate target S1 belongs to the specified type of organization. Therefore, the region boundary of the candidate target S1 can be superimposed on the original image. That is, the region boundary of the candidate target S1 is the segmentation result of the specified type of organization. The original image needs to be displayed on the screen, and the region boundary of the candidate target S1 needs to be superimposed on the original image.

[0147] If the target confidence level of candidate target S1 is S maxIf the value is greater than the first threshold λ1 and less than the second threshold λ2, then candidate target S1 is identified as a target to be processed belonging to a specified type of organization. This target is considered a suspected target, meaning it may or may not belong to the specified type of organization, requiring further analysis. Therefore, a prompt message can be issued for candidate target S1, indicating that it may belong to the specified type of organization. The prompt message can be in the form of, but is not limited to, displaying text prompts, voice prompts, and warning sound effects. Multiple methods can also be combined, such as flashing the target's bounding box information while simultaneously displaying text prompts. In this case, the boundary of the candidate target S1 region can be overlaid on the original image, or it can be left unoverlaid. For example, the original image can be displayed on the screen, with the boundary of the candidate target S1 region overlaid on it, or the boundary of the candidate target S1 region can be left unoverlaid on the original image.

[0148] If the target confidence level of candidate target S1 is S max If the value is not greater than the first threshold λ1, then candidate target S1 will not be determined as a target to be processed belonging to the specified type of organization, and there is no target to be processed belonging to the specified type of organization.

[0149] In one possible implementation, the host camera system may include an auxiliary prompting unit, which may perform step 408, that is, the auxiliary prompting unit determines the target to be processed from M candidate targets that belong to a specified type of organization based on the target confidence of each candidate target.

[0150] In summary, the target to be processed belonging to a specified tissue type can be identified from all candidate targets. When the specified tissue type is the recurrent laryngeal nerve, the actual location of the recurrent laryngeal nerve can be determined from all candidate targets. The boundary of the recurrent laryngeal nerve region is then overlaid on the original image to represent its actual location. Users can observe the actual location of the recurrent laryngeal nerve in real time on the display screen.

[0151] As can be seen from the above technical solutions, in this embodiment, after obtaining M candidate targets and the initial confidence level of each candidate target belonging to a specified type of tissue based on the original image, the initial confidence level can be adjusted based on the position information of the detection probe to obtain the target confidence level of the candidate targets. Based on the target confidence level of each candidate target, the target to be processed belonging to the specified type of tissue is determined. The position of the target to be processed is also the actual position of the specified type of tissue, thereby accurately knowing the actual position of the specified type of tissue inside the target object, quickly and accurately locating the actual position of the specified type of tissue, reminding the doctor to find the actual position of the specified type of tissue as soon as possible, and to examine and treat the specified type of tissue. In the above method, by adjusting the initial confidence level in combination with the position information of the detection probe, a more accurate and reliable target confidence level is obtained. When determining the target to be processed belonging to the specified type of tissue based on the target confidence level, the target to be processed can be found more accurately, improving the accuracy of target detection and increasing the recognition accuracy.

[0152] In thyroid surgery, the host camera system is linked with a designated type of tissue detector. By utilizing the effective information input from the designated type of tissue detector and identifying the position information of the detection probe, a weighting and correction scheme for the target confidence is designed and fused with the recurrent laryngeal nerve information from image segmentation to improve the recognition accuracy in difficult situations (such as dark areas or subfascial areas). By fusing information from the designated type of tissue detector and intelligent algorithms, the accuracy of image segmentation and detection is improved, providing auxiliary prompts to doctors.

[0153] Based on the same concept as the above method, this application proposes an in vivo tissue detection device, see [link to relevant documentation]. Figure 8 The diagram shown is a structural schematic of the device, which may include:

[0154] The acquisition module 81 is used to acquire the original image corresponding to the specified type of tissue inside the target object when the detection probe comes into contact with the specified type of tissue inside the target object;

[0155] The acquisition module 82 is used to identify the original image to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1;

[0156] The determination module 83 is used to filter out the target to be processed from the M candidate targets that belong to the specified type of organization based on the location information and the area where the M candidate targets are located.

[0157] The prompting module 84 is used to indicate the area where the target to be processed is located.

[0158] For example, when the prompting module 84 prompts the area where the target to be processed is located, it is specifically used to: overlay and display the area boundary of the target to be processed on the original image.

[0159] For example, when the determining module 83 selects the target to be processed belonging to the specified type of organization from the M candidate targets based on the location information and the regions where the M candidate targets are located, it specifically performs the following steps: obtaining the initial confidence scores of the M candidate targets from the original image; determining the region weight of each candidate target based on the location information; adjusting the initial confidence score of the candidate target based on the region weight of the candidate target to obtain the target confidence score of the candidate target; and determining the target to be processed belonging to the specified type of organization from the M candidate targets based on the target confidence score of each candidate target.

[0160] For example, when determining the region weight of each candidate target based on the location information, the determining module 83 is specifically used to: determine the centroid of the detection probe based on the location information; determine the pixel-level weight of each pixel in the original image based on the centroid; wherein, the farther the distance between the pixel and the centroid, the smaller the pixel-level weight of the pixel; then, the region weight of the candidate target can be determined based on the pixel-level weight of each pixel in the candidate target.

[0161] For example, when determining the centroid of the detection probe based on the position information, the determining module 83 is specifically used as follows: if the position information includes the coordinates of the four vertices of the quadrilateral corresponding to the detection probe, then the centroid of the quadrilateral is determined based on the coordinates of the four vertices as the centroid; or, if the position information includes the coordinates of the three vertices of the triangle corresponding to the detection probe, then the centroid of the triangle is determined based on the coordinates of the three vertices as the centroid; or, if the position information includes the coordinates of multiple circumferential points of the circle corresponding to the detection probe, then the center of the circle is determined based on the coordinates of the multiple circumferential points as the centroid.

[0162] For example, when determining the pixel-level weight of each pixel in the original image based on the centroid, the determining module 83 is specifically used to: for each pixel in the original image, determine the normalized distance between the pixel and the centroid based on the pixel's position coordinates and the target maximum distance; determine the pixel-level weight of the pixel based on the normalized distance and the configured weight allocation modulation parameters; wherein, the target maximum distance represents the maximum distance between the pixel in the original image and the centroid, and the target maximum distance is determined based on the width and height of the original image and the coordinates of the centroid.

[0163] For example, when determining the region weight of a candidate target based on the pixel-level weight of each pixel in the candidate target, the determining module 83 is specifically used to: determine the average weight value corresponding to the pixel-level weight of each pixel in the candidate target; and determine the region weight of the candidate target based on the average weight value.

[0164] For example, when determining the target to be processed belonging to the specified type of organization from the M candidate targets, the determining module 83 is specifically used to: select the candidate target with the highest target confidence from the M candidate targets; if the target confidence of the candidate target is greater than a preset threshold, then determine the candidate target as the target to be processed belonging to the specified type of organization; or, based on the target confidence corresponding to the M candidate targets, determine all candidate targets with target confidence greater than the preset threshold as the target to be processed belonging to the specified type of organization.

[0165] For example, after obtaining the initial confidence scores of the M candidate targets from the original image, before adjusting the initial confidence scores of the candidate targets based on their region weights, the determining module 83 is further configured to: if the original image does not include the detection probe, determine the region weight of each candidate target based on the historical weight value of each pixel in the original image; wherein, for each pixel in the original image, the historical weight value of the pixel is determined based on the pixel-level weight of the pixel in a reference image; the reference image is the original image including the detection probe.

[0166] For example, when the determining module 83 determines the historical weight value of the pixel based on the pixel-level weight of the pixel in the reference image, it is specifically used to: determine the attenuation parameter corresponding to the original image based on the number of interval frames and the configured attenuation modulation parameter; the number of interval frames is the number of images between the reference image and the original image; and determine the historical weight value of the pixel based on the attenuation parameter and the pixel-level weight of the pixel.

[0167] Based on the same concept as the above method, this embodiment proposes an in vivo tissue detection system, which includes: an endoscope, a host camera system, a detection probe, a specified type of tissue detector, and a display screen; wherein:

[0168] The detection probe is used to contact a specified type of tissue inside the target object;

[0169] The specified type tissue detector is used to send a successful detection response message to the host camera system when the detection probe comes into contact with the specified type of tissue inside the target object;

[0170] The endoscope is used to acquire raw images of the specified type of tissue inside the target object and send the raw images to the host camera system.

[0171] The host camera system is configured to, upon receiving the response information, determine that the detection probe has contacted a specified type of tissue inside the target object; identify the original image to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1; based on the position information and the regions where the M candidate targets are located, filter out the target to be processed that belongs to the specified type of tissue from the M candidate targets; and indicate the region where the target to be processed is located.

[0172] The display screen is used to overlay the region boundary of the target to be processed onto the original image.

[0173] Based on the same application concept as the above method, this application proposes a host camera system, which may include: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the in vivo tissue detection method disclosed in the above examples of this application.

[0174] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the in vivo tissue detection method disclosed in the above examples of this application.

[0175] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0176] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer entity or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0177] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0178] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0179] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for in vivo tissue detection, characterized in that, The method includes: When the probe comes into contact with a specified type of tissue inside the target object, the endoscope acquires the original image corresponding to the specified type of tissue inside the target object; The original image is identified to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1; The initial confidence levels of the M candidate targets are obtained from the original image; The centroid of the detection probe is determined based on the location information; the pixel-level weight of each pixel in the original image is determined based on the centroid; wherein, the farther the distance between a pixel and the centroid, the smaller the pixel-level weight of that pixel; the region weight of the candidate target is determined based on the pixel-level weight of each pixel in the candidate target; the initial confidence of the candidate target is adjusted based on the region weight of the candidate target to obtain the target confidence of the candidate target; Based on the target confidence score of each candidate target, determine the target to be processed that belongs to the specified type of organization from the M candidate targets; The system indicates the area where the target to be processed is located.

2. The method according to claim 1, characterized in that, The area where the target to be processed is located is indicated, including: The region boundary of the target to be processed is overlaid on the original image.

3. The method according to claim 1, characterized in that, Determining the centroid of the detection probe based on the location information includes: If the location information includes the coordinates of the four vertices of the quadrilateral corresponding to the detection probe, then the centroid of the quadrilateral is determined based on the coordinates of the four vertices as the centroid; or, If the location information includes the coordinates of the three vertices of the triangle corresponding to the detection probe, then the centroid of the triangle is determined based on the coordinates of the three vertices as the centroid; or, If the location information includes the coordinates of multiple circumferential points of the circle corresponding to the detection probe, then the center of the circle is determined as the centroid based on the coordinates of the multiple circumferential points.

4. The method according to claim 1, characterized in that, The step of determining the pixel-level weight of each pixel in the original image based on the centroid includes: For each pixel in the original image, the normalized distance between the pixel and the centroid is determined based on the pixel's position coordinates and the maximum distance to the target; the pixel-level weight of the pixel is determined based on the normalized distance and the configured weight allocation modulation parameters. Wherein, the target maximum distance represents the maximum distance between a pixel in the original image and the centroid, and the target maximum distance is determined based on the width and height of the original image and the coordinates of the centroid.

5. The method according to claim 1, characterized in that, The step of determining the target to be processed from the M candidate targets that belongs to the specified type of organization includes: From the M candidate targets, select the candidate target with the highest target confidence. If the target confidence of this candidate target is greater than a preset threshold, then this candidate target is determined as a target to be processed belonging to the specified type of organization; or... Based on the target confidence scores corresponding to the M candidate targets, all candidate targets with target confidence scores greater than a preset threshold are identified as targets to be processed belonging to the specified type of organization.

6. The method according to claim 1, characterized in that, After obtaining the initial confidence scores of the M candidate targets from the original image, and before adjusting the initial confidence scores of the candidate targets based on their region weights, the method further includes: If the original image does not include the detection probe, then the region weight of each candidate target is determined based on the historical weight value of each pixel in the original image; wherein, for each pixel in the original image, the historical weight value of the pixel is determined based on the pixel-level weight of the pixel in the reference image; the reference image is the original image including the detection probe.

7. The method according to claim 6, characterized in that, The process of determining the historical weight value of a pixel based on its pixel-level weight in the reference image includes: The attenuation parameter corresponding to the original image is determined based on the number of interval frames and the configured attenuation modulation parameters; wherein, the number of interval frames is the number of images between the reference image and the original image; The historical weight value of the pixel is determined based on the attenuation parameter and the pixel-level weight of the pixel.

8. An in vivo tissue detection device, characterized in that, The device includes: The acquisition module is used to acquire raw images of the specified type of tissue inside the target object when the detection probe comes into contact with the target object. The acquisition module is used to identify the original image to obtain the position information of the detection probe and M candidate targets, where M is a positive integer greater than 1; A determination module is used to: identify the initial confidence scores of the M candidate targets from the original image; determine the centroid of the detection probe based on the location information; determine the pixel-level weight of each pixel in the original image based on the centroid; the greater the distance between a pixel and the centroid, the smaller the pixel-level weight of that pixel; determine the region weight of the candidate target based on the pixel-level weight of each pixel in the candidate target; adjust the initial confidence score of the candidate target based on the region weight of the candidate target to obtain the target confidence score of the candidate target; and determine the target to be processed belonging to the specified type of organization from the M candidate targets based on the target confidence score of each candidate target. The prompting module is used to indicate the area where the target to be processed is located.

9. The apparatus according to claim 8, Its features are, in, When the prompting module prompts the region where the target to be processed is located, it is specifically used to: overlay and display the region boundary of the target to be processed on the original image; Specifically, when the determining module determines the centroid of the detection probe based on the position information, it is used as follows: if the position information includes the coordinates of the four vertices of the quadrilateral corresponding to the detection probe, then the centroid of the quadrilateral is determined based on the coordinates of the four vertices as the centroid; or, if the position information includes the coordinates of the three vertices of the triangle corresponding to the detection probe, then the centroid of the triangle is determined based on the coordinates of the three vertices as the centroid; or, if the position information includes the coordinates of multiple circumferential points of the circle corresponding to the detection probe, then the center of the circle is determined based on the coordinates of the multiple circumferential points as the centroid. Specifically, when the determining module determines the pixel-level weight of each pixel in the original image based on the centroid, it is used to: for each pixel in the original image, determine the normalized distance between the pixel and the centroid based on the pixel's position coordinates and the target maximum distance; and determine the pixel-level weight of the pixel based on the normalized distance and the configured weight allocation modulation parameters; wherein the target maximum distance represents the maximum distance between the pixel in the original image and the centroid, and the target maximum distance is determined based on the width and height of the original image and the coordinates of the centroid; Specifically, when the determining module identifies the target to be processed belonging to the specified type of organization from the M candidate targets, it is used to: select the candidate target with the highest target confidence from the M candidate targets; if the target confidence of the candidate target is greater than a preset threshold, then determine the candidate target as the target to be processed belonging to the specified type of organization; or, based on the target confidence corresponding to the M candidate targets, determine all candidate targets with target confidence greater than the preset threshold as the target to be processed belonging to the specified type of organization. Wherein, after obtaining the initial confidence scores of the M candidate targets from the original image, and before adjusting the initial confidence scores of the candidate targets based on their region weights, the determining module is further configured to: if the original image does not include the detection probe, determine the region weight of each candidate target based on the historical weight value of each pixel in the original image; wherein, for each pixel in the original image, the historical weight value of the pixel is determined based on the pixel-level weight of the pixel in a reference image; the reference image is the original image including the detection probe; Specifically, when the determining module determines the historical weight value of a pixel based on its pixel-level weight in the reference image, it is used to: determine the attenuation parameter corresponding to the original image based on the number of interval frames and the configured attenuation modulation parameter; the number of interval frames is the number of images between the reference image and the original image; and determine the historical weight value of the pixel based on the attenuation parameter and its pixel-level weight.

10. An in vivo tissue detection system, characterized in that, The in vivo tissue detection system includes: an endoscope, a main unit camera system, a detection probe, a specified type of tissue detector, and a display screen; wherein: The detection probe is used to contact a specified type of tissue inside the target object; The specified type tissue detector is used to send a successful detection response message to the host camera system when the detection probe comes into contact with the specified type of tissue inside the target object; The endoscope is used to acquire raw images of the specified type of tissue inside the target object and send the raw images to the host camera system. The host camera system, upon receiving the response information, determines the type of tissue that the detection probe has contacted within the target object; identifies the position information of the detection probe and M candidate targets from the original image, where M is a positive integer greater than 1; identifies the initial confidence levels of the M candidate targets from the original image; determines the centroid of the detection probe based on the position information; determines the pixel-level weight of each pixel in the original image based on the centroid; the greater the distance between a pixel and the centroid, the smaller the pixel-level weight of that pixel; determines the region weight of each candidate target based on the pixel-level weight of each pixel in the candidate targets; adjusts the initial confidence level of the candidate target based on the region weight of the candidate target to obtain the target confidence level of the candidate target; and, based on the target confidence level of each candidate target, identifies the target to be processed belonging to the specified type of tissue from the M candidate targets; and indicates the region where the target to be processed is located. The display screen is used to overlay the region boundary of the target to be processed onto the original image.

Images