Optoelectronic imaging tracing method and device

Abstract

Embodiments of the present application provide a photoelectric imaging tracing method and apparatus, relating to the technical field of medical devices. The method includes: obtaining an image to be processed; identifying a photoelectric detection head image region in the image to be processed; obtaining an effective region where the photoelectric detection head interacts with the image to be processed in the photoelectric detection head image region; obtaining diagnostic information of the effective region; generating a diagnostic visualization image according to the diagnostic information and the effective region, the diagnostic visualization image including a color region or a contour line; fusing the visualization image and the image to be processed; and displaying the fused image. The present application can improve the problem that when using the photoelectric detection technology for detection, what is seen is not what is detected, and the discrimination accuracy of the region to be detected is not high, and achieve the effect that when using the photoelectric detection technology for detection, what is seen is what is detected, and the discrimination accuracy of the region to be detected is improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of medical devices, and in particular, to a photoelectric imaging tracing method and apparatus. Background Art

[0002] Photoelectric detection technology has a very wide range of applications in medicine. For example, photoelectric detection technology is applied to finger blood pressure monitors, human infrared imaging examination instruments, microscopes, refractometers, various optical endoscopy examination instruments (such as gastroscopes), various biochemical analysis instruments (colorimeters, spectrometers), and various diagnostic, examination, treatment, and nursing machines to improve the detection accuracy of the equipment.

[0003] In the process of implementing the present invention, the inventors found that when using photoelectric detection technology for detection, what is seen is not what is detected, and the discrimination accuracy for the area to be detected is not high. Summary of the Invention

[0004] Embodiments of the present application provide a photoelectric imaging tracing method and apparatus, which can improve the problem that what is seen is not what is detected and the discrimination accuracy for the area to be detected is not high when using photoelectric detection technology for detection.

[0005] In the first aspect of the present application, a photoelectric imaging tracing method is provided, including:

[0006] Obtain a to-be-processed image, where the to-be-processed image includes an image of a human tissue to be detected and an image of a photoelectric detection head;

[0007] Identify the area of the photoelectric detection head image in the to-be-processed image;

[0008] Obtain an effective area of the interaction between the photoelectric detection head and the to-be-processed image in the area of the photoelectric detection head image;

[0009] Obtain the diagnostic information of the effective area;

[0010] Generate a diagnostic visualization image according to the diagnostic information and the effective area, where the diagnostic visualization image includes a color area or a contour line;

[0011] Fuse the visualization image and the to-be-processed image;

[0012] Display the fused image.

[0013] By adopting the above technical solutions, a to-be-processed image including a human tissue image to be detected and a photoelectric detection head image is obtained, the photoelectric detection head image area is identified, an effective area where the photoelectric detection head interacts with the to-be-processed image and diagnostic information of the effective area are obtained, a visualization image is generated according to the effective area and the diagnostic information, and the visualization image and the to-be-processed image are combined to obtain a fused image, which can improve the problem that when using the photoelectric detection technology for detection, what is seen is not what is detected, and the discrimination accuracy of the area to be detected is not high, and achieve the effect of what is seen is what is detected when using the photoelectric detection technology for detection and improving the discrimination accuracy of the area to be detected.

[0014] In a possible implementation, multiple groups of data are obtained, and each group of data includes a to-be-processed image, a photoelectric detection head image area, an effective area, diagnostic information, a diagnostic visualization image, and a fused image.

[0015] Display multiple fused images.

[0016] Store multiple groups of data.

[0017] In a possible implementation, the identifying the photoelectric detection head image area in the to-be-processed image includes:

[0018] According to the to-be-processed image, the photoelectric detection head image area in the to-be-processed image is identified by a semantic segmentation algorithm.

[0019] In a possible implementation, according to the to-be-processed image, identifying the photoelectric detection head image area in the to-be-processed image by a semantic segmentation algorithm includes:

[0020] According to the to-be-processed image, an identification area of the photoelectric detection head is obtained by an object recognition algorithm.

[0021] Feature extraction is performed on the identification area to obtain feature data of the identification area.

[0022] Based on the feature data, segmentation box information of the identification area is obtained.

[0023] Based on the feature data and the segmentation box information of the identification area, semantic segmentation information of the photoelectric detection head image area of the identification area is obtained, and the photoelectric detection head image area is obtained.

[0024] In a possible implementation, the obtaining an effective area where the photoelectric detection head in the photoelectric detection head image area interacts with the to-be-processed image includes:

[0025] According to the photoelectric detection head image area, the attitude position and tip position of the photoelectric detection head are fitted to obtain a fitted attitude angle and a fitted end point.

[0026] Obtain the effective area according to the fitted attitude angle, the fitted end point and the physical model of the photoelectric detection head.

[0027] In a possible implementation manner, the obtaining of the photoelectric detection head image area in the to-be-processed image includes:

[0028] According to the to-be-processed image, identify the photoelectric detection head image area in the to-be-processed image through a grayscale segmentation algorithm.

[0029] In a possible implementation manner, the identifying of the photoelectric detection head image area in the to-be-processed image according to the to-be-processed image through a grayscale segmentation algorithm includes:

[0030] According to the to-be-processed image, obtain the recognition area of the photoelectric detection head through an object recognition algorithm;

[0031] Transform the recognition area in the RGB color space to the CIE Lab color space;

[0032] Perform Gaussian histogram filtering on each grayscale channel in the CIE Lab color space;

[0033] Calculate the threshold of each grayscale channel using the Otsu threshold method and adjust the threshold to the local minimum;

[0034] Calculate the grayscale separation degree of each grayscale channel in the CIE Lab color space, select the grayscale channel with the largest grayscale separation degree, and perform binary segmentation using the threshold of this grayscale channel to obtain the photoelectric detection head image area.

[0035] In a possible implementation manner, the obtaining of the effective area of the interaction between the photoelectric detection head and the to-be-processed image in the photoelectric detection head image area includes:

[0036] According to the photoelectric detection head image area, fit the coordinates of all the constituent pixels of the photoelectric detection head to obtain the center line of the photoelectric detection head image area;

[0037] Obtain the effective area according to the center line and the physical model of the photoelectric detection head.

[0038] In a second aspect of the present application, there is provided a photoelectric imaging tracing device, including:

[0039] A first obtaining module, configured to obtain a to-be-processed image, where the to-be-processed image includes an image of a human tissue to be detected and an image of a photoelectric detection head;

[0040] An identification module, configured to identify the photoelectric detection head image area in the to-be-processed image;

[0041] A second acquisition module, configured to acquire an effective area in the image area of the optoelectronic detection head where the optoelectronic detection head interacts with the image to be processed;

[0042] A third acquisition module, configured to acquire diagnostic information of the effective area;

[0043] A generation module, configured to generate a diagnostic visualization image according to the diagnostic information and the effective area, where the diagnostic visualization image includes a color area or a contour line;

[0044] A fusion module, configured to fuse the visualization image and the image to be processed;

[0045] A first display module, configured to display the fused image.

[0046] In a possible implementation manner, it further includes

[0047] A fourth acquisition module, configured to acquire multiple groups of data, and each group of data includes an image to be processed, an image area of an optoelectronic detection head, an effective area, diagnostic information, a diagnostic visualization image, and a fused image;

[0048] A second display module, configured to display multiple fused images;

[0049] A storage module, configured to store multiple groups of data.

[0050] It should be understood that the content described in the summary of the invention section is not intended to limit the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Combined with the drawings and referring to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present application will become more obvious. In the drawings, the same or similar reference numerals represent the same or similar elements, where:

[0052] Figure 1 Shows the structural diagram of an optoelectronic imaging tracing system in an embodiment of the present application.

[0053] Figure 2 Shows the structural diagram of an imaging probe of a monocular endoscope used in an optoelectronic imaging tracing system in an embodiment of the present application.

[0054] Figure 3 Shows the structural diagram of an imaging probe of a binocular endoscope used in an optoelectronic imaging tracing system in an embodiment of the present application.

[0055] Figure 4 Shows the flowchart of an optoelectronic imaging tracing method in an embodiment of the present application.

[0056] Figure 5 Shows a schematic diagram of the image to be processed in the embodiment of the present application.

[0057] Figure 6 Shows a schematic diagram of the fused image in the embodiment of the present application.

[0058] Figure 7 Shows a structural diagram of the optoelectronic imaging tracing device in the embodiment of the present application. Detailed implementation manners

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0060] The optoelectronic imaging tracing method provided by the embodiments of the present application can be applied to the technical field of medical devices, for example, applied to various scenarios such as medical diagnosis, examination, treatment, and nursing.

[0061] Figure 1 Shows a structural diagram of the optoelectronic imaging tracing system in the embodiment of the present application. Refer to Figure 1 , in the embodiment of the present application, the optoelectronic imaging tracing method is based on the optoelectronic imaging tracing system.

[0062] In the embodiment of the present application, the optoelectronic imaging tracing system includes an optoelectronic detection subsystem and an imaging tracing subsystem. Among them, the optoelectronic detection subsystem is used to detect tissue lesions and can provide physiological, biochemical, mesoscopic, and microscopic morphological information of local tissues through methods such as optical scattering spectroscopy, fluorescence spectroscopy, electrical impedance spectroscopy, Raman spectroscopy, or OCT imaging (optical coherence tomography, that is, OCT imaging). The imaging tracing subsystem is used to visually examine the lesion and the area around the lesion, generate real-time images, and depict and display the tissue diagnosis information provided by the optoelectronic detection module in an augmented reality manner.

[0063] In the embodiment of the present application, the optoelectronic detection subsystem includes an optoelectronic detection head and an optoelectronic detection module that can be recognized by the imaging tracing subsystem. The imaging tracing subsystem includes an illumination light source, an imaging probe, an image processing unit, and a display unit.

[0064] In the embodiment of the present application, the imaging probe may include one or more optical lenses, one or more filters, and one or more imaging sensors. The imaging angle of the imaging probe for imaging the lesion and the area around it is one or more depending on the equipment included in the imaging probe.

[0065] For example, Figure 2The figure shows a structural diagram of an imaging probe of a monocular endoscope used in the optoelectronic imaging tracing system according to an embodiment of the present application. Refer to Figure 2 , the imaging probe selects a monocular endoscope, which includes an optical lens, a filter, and an imaging sensor. When the monocular endoscope performs a visual examination on a lesion and the area around the lesion, the viewing angle for imaging the lesion and its surrounding area is one.

[0066] For another example, Figure 3 The figure shows a structural diagram of an imaging probe of a binocular endoscope used in the optoelectronic imaging tracing system according to an embodiment of the present application. Refer to Figure 3 , the imaging probe can also select a binocular endoscope, which includes two optical lenses, two filters, and two imaging sensors. When the binocular endoscope performs a visual examination on a lesion and the area around the lesion, the viewing angles for imaging the lesion and its surrounding area are two.

[0067] In the embodiment of the present application, by using the optoelectronic imaging tracing probe of the optoelectronic imaging tracing system to perform real-time non-invasive optoelectronic detection on the surface tissue of the cervix or uterine body, and tracing and positioning the imaging system, it helps the physician to complete optoelectronic or pathological sampling of the area to be detected with "what is seen is what is examined".

[0068] Figure 4 The figure shows a flowchart of the optoelectronic imaging tracing method according to an embodiment of the present application. Refer to Figure 4 , in the embodiment of the present application, the optoelectronic imaging tracing method includes:

[0069] Step 101: Obtain an image to be processed, which includes an image of the human tissue to be detected and an image of the optoelectronic detection head.

[0070] In the embodiment of the present application, the image to be processed is an image obtained by the imaging probe in the imaging and tracing subsystem based on the requirements of medical staff or researchers, such as an image of the surface tissue of the cervix. Among them, the image of the human tissue to be detected included in the image to be processed is the part of the image of the surface tissue of the cervix showing the lesion area observed through the imaging probe; the image of the optoelectronic detection head is the part of the area of the surface tissue of the cervix showing the optoelectronic detection head observed through the imaging probe.

[0071] Figure 5 The figure shows a schematic diagram of the image to be processed according to an embodiment of the present application. Refer to Figure 5 , in the embodiment of the present application, the imaging probe in the imaging and tracing subsystem performs a visual detection on the surface tissue of the cervix including the lesion area and the area around the lesion, and obtains an image of the surface tissue of the cervix, that is, the image to be processed.

[0072] In the embodiment of the present application, the acquired image to be processed will be obtained by the image processing unit in the imaging tracing subsystem for the next operation. The acquired image to be processed will also be stored in the corresponding module for image data retention for easy retrieval at any time.

[0073] Step 102: Identify the image region of the photoelectric detection head in the image to be processed.

[0074] In the embodiment of the present application, the image region of the photoelectric detection head in the image to be processed is identified through deep learning in machine learning. For example, through a neural network system based on convolutional operations, an autoencoder neural network based on multiple layers of neurons, or a deep belief network that is pre-trained in the manner of a multi-layer autoencoder neural network, and then the neural network weights are further optimized by combining discriminant information.

[0075] For example, object recognition algorithms, semantic segmentation algorithms, or grayscale segmentation algorithms, etc., methods that can perform image recognition.

[0076] In the embodiment of the present application, the image region of the photoelectric detection head in the image to be processed is the image of the photoelectric detection head included in the image to be processed. The identified image region of the photoelectric detection head in the image to be processed will be obtained by the image processing unit in the imaging tracing subsystem for the next operation. The identified image region of the photoelectric detection head in the image to be processed will also be stored in the corresponding module for image data retention for easy retrieval at any time.

[0077] Step 103: Obtain the effective region where the photoelectric detection head in the image region of the photoelectric detection head interacts with the image to be processed.

[0078] In the embodiment of the present application, the effective region includes the coordinates of the detectable region of the photoelectric detection head in the coordinate system used by the photoelectric detection subsystem to display the image, corresponding to the coordinates of the detectable region of the imaging probe in the coordinate system used by the imaging tracing subsystem to display the image, that is, the subset obtained by the intersection between the coordinate set in the image to be processed and the coordinate set of the image region of the photoelectric detection head is the effective region.

[0079] Among them, the detectable region of the photoelectric detection head is the image region that observes the microscopic morphological information of the lesion region according to the photoelectric detection head in the photoelectric detection subsystem, and the photoelectric detection module in the photoelectric detection subsystem obtains the image region of the microscopic morphological information of the lesion region through the optical scattering spectroscopy method. The detectable region of the imaging probe is the image of the cervical surface tissue observed by the imaging probe in the imaging tracing subsystem, and the image of the cervical surface tissue obtained by the image processing unit in the imaging tracing subsystem.

[0080] In the embodiments of the present application, the obtained valid region will be acquired by the image processing unit in the imaging tracing subsystem for the next operation. The obtained valid region will also be stored in the corresponding module for image data retention, facilitating retrieval at any time.

[0081] Step 104: Obtain the diagnostic information of the valid region.

[0082] In the embodiments of the present application, the diagnostic information includes diagnostic classification labels. The diagnostic classification labels can be classification data summarized by medical staff based on historical experience from numerous historical diagnostic images and applied to the image processing unit in the imaging tracing subsystem. The diagnostic classification labels can also be classification data obtained through inductive learning or analogical learning in machine learning and applied to the image processing unit in the imaging tracing subsystem.

[0083] In the embodiments of the present application, the diagnostic information will be acquired by the image processing unit in the imaging tracing subsystem for the next operation. The diagnostic information will also be stored in the corresponding module for image data retention, facilitating retrieval at any time. Optionally, the diagnostic classification label is a lesion classification label, which is divided according to the degree of cervical surface tissue lesions. For different degrees of cervical surface tissue lesions, the corresponding pixels in the image have clear boundaries.

[0084] Step 105: Generate a diagnostic visualization image based on the diagnostic information and the valid region. The diagnostic visualization image includes a colored region or a contour line.

[0085] In the embodiments of the present application, according to the processing result in the image processing unit in the imaging tracing subsystem based on the diagnostic information and the valid region, the image of the lesion region in the cervical surface tissue acquired by the photoelectric detection module in the photoelectric detection subsystem is corresponding to the corresponding region in the image of the cervical surface tissue acquired by the image processing unit in the imaging tracing subsystem, and a diagnostic visualization image is generated.

[0086] In the embodiments of the present application, the diagnostic visualization image is a semi-transparent or opaque uniform pseudo-color region. The range of the region is determined by the valid region, and the color of the region coloring is determined by the pseudo-color coding. The pseudo-color coding is enumerated in a specific order based on the lesion classification label.

[0087] In the embodiments of the present application, the diagnostic visualization image can also be a semi-transparent or opaque contour line. The shape of the contour line is determined by the boundary of the valid region, and the color of the contour line coloring is determined by the pseudo-color coding. The pseudo-color coding is enumerated in a specific order based on the lesion classification label.

[0088] Step 106: Fuse the visualization image and the image to be processed.

[0089] In the embodiments of the present application, through the image fusion technology in the image stitching technology, based on the coordinates of the diagnostic visualization image, the image in the photoelectric detection module in the photoelectric detection subsystem is transformed into the coordinate system of the image in the image processing unit in the imaging tracing subsystem, completing the unified coordinate transformation.

[0090] In the embodiments of the present application, through the above coordinate transformation, the image observed by the photoelectric detection head in the photoelectric detection subsystem is fused with the image observed by the imaging probe in the imaging tracing subsystem, obtaining a fused smooth and seamless image, that is, the lesion area in the image of the cervical surface tissue is fused in the image of the cervical surface tissue.

[0091] Step 107: Display the fused image.

[0092] In the embodiments of the present application, the fused image is displayed in the display unit of the imaging tracing subsystem.

[0093] Figure 6 Shows a schematic diagram of the fused image in the embodiments of the present application. Refer to Figure 6 , in the embodiments of the present application, the fused image is the image of the lesion area of the cervical surface tissue displayed in the image to be processed.

[0094] In the embodiments of the present application, step 102, step 103 or step 104 in steps 101 - 107 can be skipped and executed, and the final effect can also be achieved. The order of steps 101 - 107 is not limited, and all those that can obtain the final result are protected by the present application.

[0095] In the embodiments of the present application, by adopting the above technical solutions, a to-be-processed image including a human tissue image containing the area to be detected and a photoelectric detection head image is obtained, the area of the photoelectric detection head image is identified, the effective area of the interaction between the photoelectric detection head and the to-be-processed image and the diagnostic information of the effective area are obtained, a visualization image is generated according to the effective area and the diagnostic information, and the visualization image and the to-be-processed image are used to obtain a fused image, which can improve the problem that when using the photoelectric detection technology for detection, what is seen is not what is detected, and the discrimination accuracy of the area to be detected is not high, and achieve the effect of what is seen is what is detected when using the photoelectric detection technology for detection and improving the discrimination accuracy of the area to be detected.

[0096] In some embodiments, the method further includes: steps 108 - 110.

[0097] Step 108: Obtain multiple groups of data, and each group of data includes a to-be-processed image, an area of the photoelectric detection head image, an effective area, diagnostic information, a diagnostic visualization image, and a fused image.

[0098] Step 109: Display multiple fused images.

[0099] Step 110: Store multiple sets of data.

[0100] In the embodiments of the present application, each set of data obtained in real time can become the historical data corresponding to the corresponding image and act on the generation process of each set of new data to improve the accuracy of obtaining each set of data.

[0101] For example, the processing results corresponding to Step 101, Step 102 - Step 103, Step 104, Step 105, and Step 106, namely the image to be processed, the valid region, the diagnostic information, the diagnostic visualization image, and the fused image, are respectively stored in the cache devices M1, M2, M3, M4, and M5.

[0102] Among them, Step 102 - Step 103 utilize all or a part of the historical data cached in M1 and M2; Step 105 utilizes all or a part of the historical data cached in M2, M3, and M4; Step 106 utilizes all or a part of the historical data cached in M1, M4, and M5.

[0103] In the embodiments of the present application, the image to be processed, the valid region, and the diagnostic information image can be the same image or can be separately distinguished images.

[0104] In some embodiments, Step A is included in Step 102.

[0105] Step A: According to the image to be processed, identify the photoelectric detector image region in the image to be processed through a semantic segmentation algorithm.

[0106] In the embodiments of the present application, optionally, each pixel in the image to be processed is labeled with its corresponding category using image semantic segmentation, and the pixels belonging to the same category are grouped into one category. Optionally, methods such as texture primitive forest and random forest are used to construct a classifier for implementing image semantic segmentation before deep learning.

[0107] In some embodiments, Steps a1 to a4 are included in Step A.

[0108] Step a1: According to the image to be processed, obtain the recognition region of the photoelectric detector through an object recognition algorithm.

[0109] Step a2: Extract features from the recognition region to obtain the feature data of the recognition region.

[0110] Step a3: Based on the feature data, obtain the recognition region segmentation box information.

[0111] Step a4: Based on the feature data and the recognition area segmentation box information, obtain the semantic segmentation information of the photoelectric detection head image area of the recognition area, and acquire the photoelectric detection head image area.

[0112] In the embodiment of the present application, a semantic segmentation model is adopted to screen the image to be processed. The semantic segmentation model includes an encoding unit, a segmentation box decoding unit, and a semantic decoding unit. Through the encoding unit, feature extraction is performed on the image to be processed to obtain the feature data of the image to be processed. Through the segmentation box decoding unit, based on the feature data, the recognition area segmentation box information is obtained. Through the semantic decoding unit, based on the feature data and the recognition area segmentation box information, the semantic segmentation information of the recognition area of the image to be processed is obtained.

[0113] In the embodiment of the present application, the recognition area is a quadrilateral recognition area calculated by an object recognition algorithm, so that when applying the image semantic segmentation algorithm, the segmentation range is reduced and the recognition accuracy is improved. Taking the quadrilateral recognition area as the input, a pixel classification label image of the photoelectric detection head is obtained through the image semantic segmentation algorithm. The photoelectric detection head image area is the pixel classification label image of the photoelectric detection head calculated through the image semantic segmentation algorithm according to the quadrilateral recognition area.

[0114] In the embodiment of the present application, it should be noted that the above image semantic segmentation algorithm is only one of the specific embodiments in the semantic segmentation algorithm, and other types of semantic segmentation algorithms can also be adopted, such as specific implementation manners of multi-class semantic segmentation algorithms, strongest semantic segmentation algorithms, etc.

[0115] In some embodiments, steps B1 - B2 are included in step 103.

[0116] Step B1: According to the photoelectric detection head image area, fit the attitude position and tip position of the photoelectric detection head to obtain the fitted attitude angle and the fitted end point.

[0117] Step B2: According to the fitted attitude angle, the fitted end point, and the physical model of the photoelectric detection head, obtain the effective area.

[0118] In the embodiment of the present application, fitting the attitude position of the photoelectric detection head is the included angle between the line connecting the end of the fitted attitude position of the photoelectric detection head and the coordinate origin and the abscissa in the coordinate system of the display unit in the imaging tracing subsystem, that is, the attitude angle of the photoelectric detection head. The coordinate point of the fitted tip position of the photoelectric detection head is the coordinate of the end point of the fitted tip position of the photoelectric detection head in the coordinate system of the display unit in the imaging tracing subsystem.

[0119] Optionally, when the physical model of the photoelectric detection head is direct detection, the characteristics of photoelectric detection are mainly utilized. Further explanation is that the characteristics of photoelectric detection in direct detection are that during the direct detection process, the optical signal to be detected is directly incident on the photosensitive surface of the photoelectric detection, and the photoelectric detector directly converts the photoelectric signal into current or voltage. According to different requirements, it is then processed by subsequent circuits, and finally a useful signal is obtained.

[0120] In some embodiments, step C may further be included in step 102.

[0121] Step C: According to the image to be processed, through a gray-scale segmentation algorithm, identify the image area of the photoelectric detection head in the image to be processed.

[0122] In the embodiments of the present application, optionally, a gray-scale threshold segmentation algorithm based on the CIE Lab (Commission International de l'Eclairage Lab, that is, the Lab color model) color space is used. CIE Lab uses a digital method to describe human visual perception and is committed to perceptual uniformity.

[0123] Among them, the L component in the Lab color model is used to represent the brightness of the pixel, and the value range is [0, 100], indicating from pure black to pure white; a represents the range from red to green, and the value range is [127, -128]; b represents the range from yellow to blue, and the value range is [127, -128]. Precise color balance can be achieved by modifying the output color levels of the a and b components, or the L component can be used to adjust the brightness contrast.

[0124] In some embodiments, step C includes steps c1 to c5.

[0125] Step c1: According to the image to be processed, through an object recognition algorithm, obtain the recognition area of the photoelectric detection head.

[0126] Step c2: Transform the recognition area in the RGB color space to the CIE Lab color space.

[0127] Step c3: Perform Gaussian histogram filtering on each gray-scale channel in the CIE Lab color space.

[0128] Step c4: Use the Otsu threshold method to calculate the threshold of each gray-scale channel and adjust the threshold to a local minimum.

[0129] Step c5: Calculate the gray-scale separation degree of each gray-scale channel in the CIE Lab color space, select the gray-scale channel with the largest gray-scale separation degree, and perform binary segmentation using the threshold of this gray-scale channel to obtain the image area of the photoelectric detection head.

[0130] In the embodiments of the present application, an algorithm that first converts from the RGB color space to the CIE XYZ color space and then to the CIE Lab color space is adopted to complete the conversion from the RGB color space to the CIE Lab color space. Through the Gaussian function, the gray histogram is smoothed and filtered to obtain the Gaussian histogram filter. The Otsu threshold method calculates the maximum between-class variance through the gray histogram to obtain the threshold. Binary segmentation means that for the gray channel with the largest gray separation degree, according to the corresponding threshold, the gray values greater than the threshold are classified into one category, and those less than the threshold are classified into another category.

[0131] In the embodiments of the present application, the recognition area is a quadrilateral recognition area calculated by an object recognition algorithm, which reduces the segmentation range and improves the recognition accuracy when using the gray threshold segmentation algorithm. Taking the quadrilateral recognition area as the input, a pixel classification marker image of the photoelectric detection head is obtained through the gray threshold segmentation algorithm. The image area of the photoelectric detection head is the pixel classification marker image of the photoelectric detection head calculated through the gray threshold segmentation algorithm according to the quadrilateral recognition area.

[0132] In the embodiments of the present application, it should be noted that the above gray threshold segmentation algorithm is only one of the specific embodiments in the gray segmentation algorithms, and other types of gray segmentation algorithms can also be adopted, such as specific implementation methods like the edge enhancement gray segmentation algorithm and the pixel histogram gray segmentation algorithm.

[0133] In some embodiments, steps D1 - D2 may also be included in step 103.

[0134] Step D1: According to the image area of the photoelectric detection head, the coordinates of all the constituent pixels of the photoelectric detection head are fitted to obtain the center line of the image area of the photoelectric detection head.

[0135] Step D2: According to the center line and the physical model of the photoelectric detection head, the effective area is obtained.

[0136] In the embodiments of the present application, the following method is adopted for center line fitting:

[0137] Step E1: Obtain the coordinates of all the constituent pixels of the photoelectric detection head.

[0138] Step E2: Calculate the average value of the coordinates of all the constituent pixels of the photoelectric detection head.

[0139] Step E3: Fit the center line according to the average value.

[0140] Optionally, when the physical model of the photoelectric detection head is direct detection, the characteristics of photoelectric detection are mainly utilized. Further explanation is that the characteristics of photoelectric detection during direct detection are that during the direct detection process, the optical signal to be detected is directly incident on the photosensitive surface of the photoelectric detector, and the photoelectric detector directly converts the photoelectric signal into current or voltage. According to different requirements, it is then processed by subsequent circuits, and finally a useful signal is obtained.

[0141] It should be noted that in step 102, the use of object recognition algorithms, semantic segmentation algorithms, or grayscale segmentation algorithms is only a specific implementation of deep machine learning. Other methods capable of image recognition in deep machine learning can also be used in step 102.

[0142] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0143] The above is the introduction of method embodiments. The following further illustrates the solution of this application through device embodiments.

[0144] Figure 7 The structural diagram of an optoelectronic imaging tracing device according to an embodiment of this application is shown. Refer to Figure 7 , the optoelectronic imaging tracing device includes a first acquisition module 201, an identification module 202, a second acquisition module 203, a third acquisition module 204, a generation module 205, a fusion module 206, and a display module 207.

[0145] The first acquisition module 201 is used to acquire a to-be-processed image, and the to-be-processed image includes a to-be-detected human tissue image and a photoelectric detection head image.

[0146] The identification module 202 is used to identify the photoelectric detection head image area in the to-be-processed image.

[0147] The second acquisition module 203 is used to acquire the effective area of the interaction between the photoelectric detection head and the to-be-processed image in the photoelectric detection head image area.

[0148] The third acquisition module 204 is used to acquire the diagnostic information of the effective area.

[0149] The generation module 205 is used to generate a diagnostic visualization image according to the diagnostic information and the effective area, and the diagnostic visualization image includes a color area or a contour line.

[0150] A fusion module 206 is configured to fuse the visualization image and the image to be processed.

[0151] A first display module 207 is configured to display the fused image.

[0152] In some embodiments, the optoelectronic imaging tracing device further includes:

[0153] A fourth acquisition module 208 is configured to acquire multiple sets of data, each set of data including an image to be processed, an optoelectronic detection head image area, a valid area, diagnostic information, a diagnostic visualization image, and a fused image.

[0154] A second display module 209 is configured to display multiple fused images.

[0155] A storage module 210 is configured to store multiple sets of data.

[0156] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the described modules can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0157] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated herein, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0158] The above are only some embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An optoelectronic imaging tracing method, characterized in that, Comprising: Obtain an image to be processed, where the image to be processed includes a human tissue image to be detected and an optoelectronic probe head image; Identify the optoelectronic probe head image area in the image to be processed; Obtain the effective area where the optoelectronic probe head interacts with the image to be processed in the optoelectronic probe head image area; Obtain the diagnostic information of the effective area; Generate a diagnostic visualization image according to the diagnostic information and the effective area, where the diagnostic visualization image includes a color area or a contour line; Fuse the visualization image and the image to be processed; Display the fused image; The obtaining the effective area where the optoelectronic probe head interacts with the image to be processed in the optoelectronic probe head image area includes: according to the optoelectronic probe head image area, fitting the attitude position and the tip position of the optoelectronic probe head to obtain a fitted attitude angle and a fitted end point; According to the fitted attitude angle, the fitted end point and the physical model of the optoelectronic probe head, obtain the effective area; or, according to the optoelectronic probe head image area, fit the coordinates of all constituent pixels of the optoelectronic probe head to obtain the center line of the optoelectronic probe head image area; according to the center line and the physical model of the optoelectronic probe head, obtain the effective area; Fitting the attitude position of the optoelectronic probe head is the angle between the line connecting the end of the fitted attitude position of the optoelectronic probe head and the coordinate origin and the abscissa in the coordinate system of the display unit in the imaging tracing subsystem, that is, the attitude angle of the optoelectronic probe head; the coordinate point of the fitted tip position of the optoelectronic probe head is the coordinate of the end point of the fitted tip position of the optoelectronic probe head in the coordinate system of the display unit in the imaging tracing subsystem; the physical model of the optoelectronic probe head is direct detection by the optoelectronic probe head. During the direct detection process, the optical signal to be detected is directly incident on the photosensitive surface of the optoelectronic detector, and the optoelectronic detector directly converts the optoelectronic signal into current or voltage.

2. The method according to claim 1, characterized in that, Also comprising: Obtain multiple groups of data, each group of data includes an image to be processed, an optoelectronic probe head image area, an effective area, diagnostic information, a diagnostic visualization image, and a fused image; Display multiple fused images; Store multiple groups of data.

3. The method according to claim 2, wherein The identifying the optoelectronic probe head image area in the image to be processed includes: According to the image to be processed, identify the optoelectronic probe head image area in the image to be processed through a semantic segmentation algorithm.

4. The method according to claim 3, wherein According to the image to be processed, identifying the optoelectronic probe head image area in the image to be processed through a semantic segmentation algorithm includes: According to the image to be processed, obtain the recognition area of the optoelectronic probe head through an object recognition algorithm; Extract the features of the recognition area to obtain the feature data of the recognition area; Based on the feature data, obtain the segmentation box information of the recognition area; Based on the feature data and the segmentation box information of the recognition area, obtain the semantic segmentation information of the optoelectronic probe head image area of the recognition area, and obtain the optoelectronic probe head image area.

5. The method according to claim 2, wherein The obtaining the optoelectronic probe head image area in the image to be processed includes: Based on the image to be processed, the image region of the photoelectric detection head in the image to be processed is identified by a grayscale segmentation algorithm.

6. The method according to claim 5, characterized in that, The identifying the image region of the photoelectric detection head in the image to be processed by the grayscale segmentation algorithm based on the image to be processed includes: Based on the image to be processed, the recognition region of the photoelectric detection head is obtained by an object recognition algorithm; The recognition region in the RGB color space is transformed to the CIE Lab color space; Gaussian histogram filtering is performed on each grayscale channel in the CIE Lab color space; The threshold of each grayscale channel is calculated by the Otsu threshold method, and the threshold is adjusted to the local minimum value; The grayscale separation degree of each grayscale channel in the CIE Lab color space is calculated, the grayscale channel with the largest grayscale separation degree is selected, and binary segmentation is performed using the threshold of this grayscale channel to obtain the image region of the photoelectric detection head.

7. An optoelectronic imaging tracing device, characterized in that, including: A first acquisition module for acquiring an image to be processed, where the image to be processed includes an image of a human tissue to be detected and an image of a photoelectric detection head; An identification module for identifying the image region of the photoelectric detection head in the image to be processed; A second acquisition module for acquiring the effective region where the photoelectric detection head interacts with the image to be processed in the image region of the photoelectric detection head; A third acquisition module for acquiring the diagnostic information of the effective region; A generation module for generating a diagnostic visualization image according to the diagnostic information and the effective region, where the diagnostic visualization image includes a color region or a contour line; A fusion module for fusing the visualization image and the image to be processed; A first display module for displaying the fused image; The acquiring the effective region where the photoelectric detection head interacts with the image to be processed in the image region of the photoelectric detection head includes: according to the image region of the photoelectric detection head, fitting the attitude position and the tip position of the photoelectric detection head to obtain a fitting attitude angle and a fitting end point; According to the fitting attitude angle, the fitting end point and the physical model of the photoelectric detection head, the effective region is obtained; or, according to the image region of the photoelectric detection head, fitting the coordinates of all the constituent pixels of the photoelectric detection head to obtain the center line of the image region of the photoelectric detection head; according to the center line and the physical model of the photoelectric detection head, the effective region is obtained; Fitting the attitude position of the photoelectric detection head is the angle between the line connecting the end of the attitude position of the photoelectric detection head and the coordinate origin and the abscissa in the coordinate system of the display unit in the imaging tracing subsystem, that is, the attitude angle of the photoelectric detection head; the coordinate point of the tip position of the photoelectric detection head is the coordinate of the end point of the tip position of the photoelectric detection head in the coordinate system of the display unit in the imaging tracing subsystem; the physical model of the photoelectric detection head is direct detection by the photoelectric detection head. During the direct detection process, the optical signal to be detected is directly incident on the photosensitive surface of the photoelectric detection, and the photoelectric detector directly converts the photoelectric signal into current or voltage.

8. The device according to claim 7, wherein It also includes: The fourth acquisition module is used to acquire multiple sets of data, and each set of data includes an image to be processed, an image area of the optoelectronic detection head, a valid area, diagnostic information, a diagnostic visualization image, and a fused image; The second display module is used to display multiple fused images; The storage module is used to store multiple sets of data.

Images