Training method of laser point detection model based on deep neural network, laser point detection method and system, and laser lithotripsy system
Patent Information
- Application Number
- CN202611461182.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-20
- Publication Date
- 2026-11-03
Smart Images
Figure CN123033239A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target tracking technology, and more specifically, to a training method for a laser point detection model based on a deep neural network, a laser point detection method and system, and a laser lithotripsy system. Background Technology
[0002] Laser lithotripsy is one of the mainstream surgical methods for treating urinary tract stones such as kidney stones, ureteral stones, and bladder stones. In this procedure, the doctor first inserts a thin optical fiber into the location of the stone using an endoscope, and then, under endoscopic monitoring, uses the thermal effect of a laser to directly break up the stone.
[0003] In existing technologies, when a laser is activated, the surface of the stone undergoes thermal ablation and mechanical disintegration, producing a large amount of dust, tiny debris, and cavitation bubbles, which are suspended in the liquid and can produce flashing reflections. These appear as circular bright areas in the image, which are easily confused with the laser point in terms of visual characteristics. In addition, bleeding points on the surface of the urinary system mucosa can produce local bright reflections under laser irradiation, and their shape and size are similar to those of the laser point.
[0004] Therefore, when identifying laser points in images acquired by an endoscope, the aforementioned interference factors often lead to false detections, which may result in the instantaneous firing of the laser due to misjudgment, causing medical accidents and reducing the safety of the surgery. Summary of the Invention
[0005] The purpose of this invention is to provide a training method for a laser point detection model based on a deep neural network, a laser point detection method and system, and a laser lithotripsy system. This invention solves the technical problem in the prior art where interference terms lead to false detections of laser points in endoscopic images, which may result in instantaneous laser firing due to misjudgment, causing medical accidents and reducing surgical safety.
[0006] As a first aspect of the present invention, the present invention provides a training method for a laser point detection model based on a deep neural network, the laser point detection model being used to detect target laser points in endoscopic images detected by an endoscope. The training method includes: A dataset is constructed based on multiple historical images acquired by the endoscope and the corresponding label data for each historical image, and the dataset is divided into a training set and a validation set. The training set is input into the laser point detection model to be trained. The laser point detection model to be trained includes a classification branch network and a regression branch network, and the classification branch network and the regression branch network process the historical image simultaneously. Using the laser point detection model to be trained, the prediction result of the historical image is obtained based on the historical images in the training set; Based on the prediction results of the historical images and the corresponding label data of the historical images, the model parameters of the laser point detection model to be trained are adjusted according to the composite loss function. The composite loss function is composed of a weighted sum of the classification loss function and the regression loss function.
[0007] In one embodiment of the present invention, the regression loss function is: ; in, L DIOU For the regression loss value, IOU The intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box. The center point of the prediction box. The center point of the real bounding box. It is the Euclidean distance between the predicted bounding box and the ground truth bounding box. C The length of the diagonal of the smallest closure region containing the predicted bounding box and the ground truth bounding box.
[0008] In one embodiment of the present invention, the classification loss function is: ; in, FL ( p t ) represents the classification loss value. p t To predict probabilities, α t These are the weighting coefficients corresponding to the classification loss. γ For focusing parameters.
[0009] In one embodiment of the present invention, the prediction result of the historical image is obtained based on the historical images in the training set using the laser point detection model to be trained, including: Extract feature maps from the historical images; Calculate the centrality between each location in the feature map and the target center point of the target bounding box; Calculate the offset between each position in the feature map and each edge of the target bounding box; Based on the centrality between each position in the feature map and the target center point of the target bounding box, and the offset between each position in the feature map and each side of the target bounding box, predict whether there is a target laser point in the feature map and the location information of the target laser point.
[0010] In one embodiment of the present invention, the method further includes: The initial historical endoscopic images were divided into multiple regions; Detect the pixel value of each region block and determine the valid range of pixel values for multiple region blocks; The region block whose pixel values exceed the effective range is identified as the target region block; The pixel cumulative distribution function is determined based on the pixel values of the region block within the effective range; The mapping function corresponding to the target region block is determined based on the pixel cumulative distribution function and the pixel values of the target region block; The pixel values of the target region block are remapped according to the mapping function corresponding to the target region block.
[0011] In one embodiment of the present invention, the method further includes: cropping the initial historical endoscopic image using a preset mask according to a preset rule to obtain an effective historical image; The valid historical images are dynamically cropped according to preset cropping parameters.
[0012] As a second aspect of the present invention, the present invention also provides a method for laser point detection in endoscopic images, comprising: Acquire endoscopic images of the target detection area captured by the endoscope; The endoscope image is input into the laser point detection model for detection to detect target laser points in the endoscope image, and the detection result is output. The detection result includes the target laser point detection box and the unidentified target laser points. The laser point detection model was trained using the training method described above.
[0013] In one embodiment of the present invention, the detection method further includes: Obtain the stone detection frame of the endoscopic image; When the detection result is a target laser point detection frame, the intersection-union ratio (IUGR) of the stone detection frame and the target laser point detection frame is calculated. When the IUGR is greater than or equal to a preset threshold, an excitation control command is generated. When the IUGR is less than the preset threshold, an early warning message is generated. The excitation control command is used to indicate that the laser generator is activated to emit laser light, and the early warning message is used to prompt the laser generator to be turned off and issue an alarm message. When the detection result is that no target laser point is identified, invalid information is generated, which is used to indicate that the laser generator is kept in standby mode.
[0014] As a third aspect of the present invention, the present invention also provides a laser point detection system for endoscopic images, comprising: A laser point detection model, wherein the laser point detection model is trained using the training method described above; An image acquisition unit is communicatively connected to an endoscope and is used to acquire endoscopic images of the target detection area acquired by the endoscope.
[0015] As a fourth aspect of the present invention, the present invention also provides a laser lithotripsy system, comprising: A laser generator is used to emit a laser beam to the target detection area to form a laser spot; An endoscope is used to acquire images of the target detection area; and The laser spot detection system for endoscopic images described above.
[0016] This invention provides a training method for a laser point detection model based on a deep neural network. The laser point detection model to be trained adopts a decoupled structure (i.e., a classification branch network and a regression branch network). The classification branch network relies on the texture of the central region, while the regression branch network relies on edge position information. That is, the decoupled structure handles class probability prediction and bounding box coordinate regression separately, while simultaneously allowing information interaction between the two branches. This allows the classification branch to improve localization accuracy with the boundary information of the regression branch, and the regression branch to improve stability with the center information of the classification branch. Furthermore, the regression branch utilizes a regression loss function that incorporates a center point distance penalty term, employing a distance intersection-union loss. It uses the relative distance between each position in the feature map and the target center point of the target bounding box to guide convergence, improving tracking stability when the laser point moves rapidly. The classification branch uses a modulation factor (1- p t This allows for the effective shielding of gradient noise generated by massive amounts of simple negative samples during millions of iterations, ensuring that the gradient descent direction always points towards optimizing the features of the most difficult-to-classify laser points. This enables the laser point detection model to be trained to effectively learn images with greater difficulty, such as difficult-to-classify partially occluded laser points. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0018] Figure 1 The diagram shown is a flowchart illustrating a training method for a laser point detection model based on a deep neural network, according to an embodiment of the present invention.
[0019] Figure 2 The diagram shown is a flowchart illustrating a training method for a laser point detection model based on a deep neural network, according to the second embodiment of the present invention.
[0020] Figure 3 The diagram shown is a flowchart illustrating a training method for a laser point detection model based on a deep neural network, provided in the third embodiment of the present invention.
[0021] Figure 4 The diagram shown is a flowchart illustrating a laser point detection method for endoscopic images according to an embodiment of the present invention.
[0022] Figure 5 The diagram shown is a flowchart illustrating a laser point detection method for endoscopic images according to another embodiment of the present invention.
[0023] Figure 6 The diagram shown is a working block diagram of a laser point detection system for endoscopic images provided in an embodiment of the present invention.
[0024] Figure 7 The diagram shown is a working block diagram of a laser lithotripsy system provided in an embodiment of the present invention. Detailed Implementation
[0025] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] In the description of this invention, it should be noted that the terms "upper", "lower", "front", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0027] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "installation" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a mechanical connection or an electrical connection; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0028] Exemplary methods As a first aspect of the present invention, the present invention provides a training method for a laser point detection model based on a deep neural network, the laser point detection model being used to detect target laser points in endoscopic images detected by an endoscope. Figure 1The diagram shown is a flowchart illustrating a training method for a laser point detection model based on a deep neural network according to an embodiment of the present invention. Figure 1 As shown, the training method for a laser point detection model based on a deep neural network provided by this invention includes the following steps: S100: Construct a dataset based on multiple historical images acquired by the endoscope and the label data corresponding to each historical image, and divide the dataset into a training set and a validation set; Specifically, multiple historical images acquired by the endoscope are labeled, including both positive and negative labels. Positive labels include laser points, while negative labels include bubbles, debris, bleeding points, dust, etc. Historical images and their corresponding positive labels serve as positive samples. Similarly, historical images and their corresponding negative labels serve as negative samples. These negative samples help teach the laser point detection model to identify "what is not a laser," thereby improving the accuracy of laser point detection.
[0029] After each historical image is labeled, multiple historical images and their corresponding labels constitute a dataset. The dataset is then divided into a training set and a validation set.
[0030] S200: Input the training set into the laser point detection model to be trained; wherein the laser point detection model to be trained includes a classification branch network and a regression branch network, and the classification branch network and the regression branch network process the historical image simultaneously; Classification branch networks are used to learn category-related features such as local texture, shape contour, and region contrast of targets (e.g., laser points, bleeding points, debris, bubbles, etc.) to optimize classification tasks and avoid feature expression shift caused by simultaneously fitting location information.
[0031] Regression branch networks are used to learn the boundary location, spatial distribution, and geometric shape of targets (such as laser points, bleeding points, debris, bubbles, etc.) and other features related to localization, thus avoiding inaccurate boundary localization caused by interference from category semantic information.
[0032] By having the classification branch network and the regression branch network process historical images simultaneously, the same set of feature maps can be processed by the classification branch network and the regression branch network respectively with different feature transformation methods, thereby improving classification accuracy and localization precision without increasing the amount of inference computation.
[0033] That is, the laser point detection model to be trained in this invention adopts a decoupled structure (i.e., a classification branch network and a regression branch network). The classification branch network relies on the texture of the central region, while the regression branch network relies on the edge position information. In other words, the decoupled structure is used to process the category probability prediction and the bounding box coordinate regression separately, and at the same time, the two branches interact with each other. This allows the classification branch to improve the positioning accuracy with the boundary information of the regression branch, and the regression branch to improve the stability with the center information of the classification branch.
[0034] Optionally, a multi-scale spatial pyramid pooling module is set at the end of the backbone network. This module adopts a parallel multi-branch pooling structure, with each branch having a different receptive field. Global context information is extracted through parallel pooling operations of different receptive fields, which enhances the detection model's ability to distinguish subtle differences between scattered halos and real laser points.
[0035] In addition, the laser point detection model to be trained in this invention adopts a cross-stage local residual structure. The cross-stage local residual structure divides the input feature map into two branches, one of which undergoes dense residual processing and the other is directly passed, and finally merges the output, thereby optimizing the gradient propagation path while ensuring the network depth.
[0036] S300: Using the laser point detection model to be trained, the prediction results of historical images are obtained based on historical images in the training set; Historical images and their corresponding label data are input into the laser point detection model to be trained for calculation to obtain the prediction result corresponding to the historical image.
[0037] Specifically, the prediction results can be: laser point or non-laser point.
[0038] Optionally, when predicting historical images in the training set, a bidirectional feature transfer structure is adopted. On the basis of the top-down semantic enhancement path, a bottom-up localization feature transfer path is added, which can fuse shallow edge and corner information with deep semantic information, greatly improving the detection model's ability to capture tiny laser points (pixel-level targets).
[0039] S400: Based on the prediction results of historical images and the corresponding label data of historical images, adjust the model parameters of the laser point detection model to be trained according to the composite loss function; wherein, the composite loss function is composed of a weighted sum of the classification loss function and the regression loss function.
[0040] Optionally, the regression loss function is: ; in, L DIOU For the regression loss value, IOU The intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box. The center point of the prediction box, The center point of the true bounding box. It is the Euclidean distance between the predicted bounding box and the ground truth bounding box. C It is the diagonal length of the smallest closure region containing both the predicted and ground truth boxes.
[0041] The regression branch network calculates the relative distance (i.e., offset) between each location in the feature map and each edge of the target bounding box. A regression loss function incorporating a center point distance penalty is used, employing a distance intersection-union loss. The relative distance between each location in the feature map and the target center point of the target bounding box guides convergence, improving tracking stability when the laser point moves rapidly.
[0042] Optionally, the classification loss function is: ; in, FL ( p t ) represents the classification loss value. p t To predict probabilities, α t These are the weighting coefficients corresponding to the classification loss. γ For focusing parameters.
[0043] Through classification branch network, through modulation factor (1- p t This allows for the effective shielding of gradient noise generated by massive amounts of simple negative samples during millions of iterations, ensuring that the gradient descent direction always points towards optimizing the features of the most difficult-to-classify laser points. This enables the laser point detection model to be trained to effectively learn images with greater difficulty, such as difficult-to-classify partially occluded laser points.
[0044] This invention provides a training method for a laser point detection model based on a deep neural network. The laser point detection model to be trained adopts a decoupled structure (i.e., a classification branch network and a regression branch network). The classification branch network relies on the texture of the central region, while the regression branch network relies on edge position information. That is, the decoupled structure handles class probability prediction and bounding box coordinate regression separately, while simultaneously allowing information interaction between the two branches. This allows the classification branch to improve localization accuracy with the boundary information of the regression branch, and the regression branch to improve stability with the center information of the classification branch. Furthermore, the regression branch utilizes a regression loss function that incorporates a center point distance penalty term, employing a distance intersection-union loss. It uses the relative distance between each position in the feature map and the target center point of the target bounding box to guide convergence, improving tracking stability when the laser point moves rapidly. The classification branch uses a modulation factor (1- p tThis allows for the effective shielding of gradient noise generated by massive amounts of simple negative samples during millions of iterations, ensuring that the gradient descent direction always points towards optimizing the features of the most difficult-to-classify laser points. This enables the laser point detection model to be trained to effectively learn images with greater difficulty, such as difficult-to-classify partially occluded laser points.
[0045] In one embodiment of the present invention, Figure 2 The diagram shown is a flowchart illustrating a training method for a laser point detection model based on a deep neural network, according to another embodiment of the present invention. Figure 2 As shown, S300 (obtaining the prediction result of the historical image based on the historical images in the training set using the laser point detection model to be trained) specifically includes the following steps S310-S340: S310: Extract feature maps from historical images; S320: Calculate the centrality between each location in the feature map and the target center point of the target bounding box; Specifically, the target center point is the geometric center region of the target bounding box. The target bounding box can adaptively detect various scales and aspect ratios without requiring predefined outlines.
[0046] Centrality measures whether each location in the feature map is near the center point of the target bounding box, and can represent the confidence that the location is near the center point of the target.
[0047] S330: Calculate the offset between each location in the feature map and each edge of the target bounding box; Specifically, the offset refers to the relative distance between each location in the feature map and each edge of the target bounding box.
[0048] S340: Based on the centrality between each position in the feature map and the target center point of the target bounding box, and the offset between each position in the feature map and each side of the target bounding box, predict whether there is a target laser point in the historical image and the location information of the target laser point.
[0049] The feature map is predicted based on centrality to determine whether the target center point is included. If the center point is included, the position and size of the target bounding box are predicted based on the offset.
[0050] This invention uses centrality to predict whether each location contains the target center point, and uses offset to predict the position and size of the target bounding box. That is, it uses centrality and offset for joint prediction, which improves the model's generalization ability to different shapes of target laser points, bleed points, debris, reflective points, bubbles, etc., so as to improve the accuracy of the prediction results of target laser points.
[0051] In another embodiment of the invention, such as Figure 3As shown, before S100 (constructing a dataset based on multiple historical images acquired by the endoscope and the label data corresponding to each historical image), the training method for a laser point detection model based on a deep neural network provided by this invention also requires preprocessing of the initial historical endoscopic images acquired by the endoscope. Specifically, the preprocessing method for the initial historical endoscopic images, that is, before S100, the training method provided by this invention further includes the following steps: S101: Divide the initial historical endoscopic images into multiple region blocks; S102: Detect the pixel value of each region block and determine the valid range of pixel values for multiple region blocks; Specifically, based on the pixel values of each region block, the pixel value distribution of the initial historical endoscopic image is determined, and the effective range of the pixel values is determined based on the pixel value distribution.
[0052] S103: Determine the region block whose pixel values are outside the valid range as the target region block; Once the effective range is determined, the target area block that exceeds the effective range can be determined based on the pixel value of each area block and the effective range. That is, the target area block is the area that needs to be re-mapped for brightness and contrast.
[0053] S104: Determine the pixel cumulative distribution function based on the pixel values of the region block within the effective range; Once the effective range is determined, the cumulative pixel distribution function can be determined based on the pixel values of the regions within the effective range.
[0054] The cumulative pixel distribution function refers to the proportion of pixels in an image whose grayscale value is less than or equal to a threshold.
[0055] S105: Determine the mapping function corresponding to the target region block based on the pixel cumulative distribution function and the pixel values of the target region block; S106: Remap the pixel values of the target region block according to the mapping function corresponding to the target region block.
[0056] This involves remapping the contrast and brightness of the target area block.
[0057] S101-S106 can be used to preprocess the initial historical endoscopic images acquired by the endoscope (brightness and contrast remapping of local areas) to obtain the preprocessed historical images.
[0058] This invention employs an adaptive histogram enhancement algorithm (such as CLAHE) to remap brightness and contrast in local blocks (i.e., target area blocks, areas where pixel values exceed the effective range). This enhances the texture of dark stone areas while avoiding overexposure of the laser center highlights caused by global enhancement, thereby improving the performance of subsequent classification and regression tasks. It may also significantly enhance the edge response of real targets, thereby improving the accuracy of target laser point recognition and reducing the probability of misidentification.
[0059] In another embodiment of the present invention, the preprocessing method for preprocessing the initial historical endoscopic images can also be as follows: that is, before S100, the training method provided by the present invention further includes the following steps: S107: According to preset rules, the initial historical endoscopic image is cropped using a preset mask to obtain a valid historical image; That is, according to preset rules (such as cropping out areas with gray values close to 0), the initial historical endoscopic image is cropped using a preset mask. This static cropping method is used to eliminate the interference of edge areas on feature extraction, so that during model training, attention is paid to the core area of the image, so that the relative position of the target laser point in the image is stable.
[0060] S108: Dynamically crop the valid historical image according to the preset cropping parameters; Dynamic cropping can adaptively preserve valid content from existing historical images.
[0061] This invention employs a combination of static and dynamic clipping, which can concentrate computational resources on the core area of the surgical field of view, thereby reducing computational redundancy.
[0062] As a second aspect of the invention, the invention also provides a method for laser point detection in endoscopic images. Figure 4 The diagram shown is a flowchart illustrating a laser point detection method for endoscopic images according to an embodiment of the present invention. Figure 4 As shown, the laser point detection method for endoscopic images provided by this invention includes the following steps: S1: Acquire endoscopic images of the target detection area captured by the endoscope; During the surgery, an endoscope is used to acquire real-time endoscopic images of the target detection area. Specifically, the target detection area can be any area observed using the endoscope.
[0063] S2: Input the endoscope image into the laser point detection model for detection to detect the target laser point in the endoscope image and output the detection result, which includes the target laser point detection box and the unidentified target laser point; The laser point detection model was trained using the deep neural network-based laser point detection model training method described above.
[0064] When a target laser point is detected in an endoscopic image, the detection results include the position and size of the target laser point detection box.
[0065] The present invention provides a laser point detection method for endoscopic images, which adopts a decoupled structure to process category probability prediction and bounding box coordinate regression separately, and simultaneously interacts with the information between the two branches. The classification branch improves the positioning accuracy with the boundary information of the regression branch, and the regression branch improves the stability with the center information of the classification branch. In this way, the target laser point can be accurately identified, thereby improving the detection accuracy of the target laser point.
[0066] Optionally, after acquiring the endoscopic images, the images can be preprocessed. Specific preprocessing methods can include the following two: (1) Remapping the brightness and contrast of local areas of the endoscopic image, specifically including the following steps: Divide the endoscopic image into multiple regions; Detect the pixel value of each region block and determine the valid range of pixel values for multiple region blocks; Identify the region block whose pixel values are outside the valid range as the target region block; The pixel cumulative distribution function is determined based on the pixel values of the region blocks within the effective range; The mapping function corresponding to the target region block is determined based on the pixel cumulative distribution function and the pixel values of the target region block; The pixel values of the target region block are remapped according to the mapping function corresponding to the target region block.
[0067] (2) Static and dynamic cropping of endoscopic images, specifically including the following steps: According to preset rules, the endoscopic image is cropped using a preset mask to obtain a valid image; Dynamically crop the valid historical images according to preset cropping parameters; In one embodiment of the present invention, such as Figure 5 As shown, after S2, the detection method provided by the present invention further includes the following steps: S3: Obtain the stone detection box of the endoscopic image; S4: When the detection result is the target laser point detection box, calculate the intersection-union ratio (IU / R) between the stone detection box and the target laser point detection box. When the IU / R is greater than or equal to a preset threshold, generate an excitation control command; when the IU / R is less than the preset threshold, generate an early warning message. The excitation control command is used to indicate that the laser generator is activated to emit laser light, and the early warning message is used to prompt the laser generator to be turned off and issue an alarm message. Once the target laser point is detected, the crossover ratio (CRO) between the target laser point detection frame and the stone detection frame can be calculated. When the CRO is greater than or equal to a preset threshold, it indicates that the target laser point overlaps with the stone, and the laser generator can be activated to emit a laser to break the stone.
[0068] When the cross-union ratio between the target laser point detection frame and the stone detection frame is less than the preset threshold, it indicates that the laser point may not have landed on the stone, but may have landed on soft tissue, metal instruments, or guide wires. At this time, an early warning message is generated and the laser generator is instantly cut off to forcibly shut down the laser generator in order to avoid medical accidents.
[0069] S5: When the detection result is that no target laser point is identified, invalid information is generated. Invalid information is used to indicate that the laser generator is kept in standby mode.
[0070] When no target laser point is detected, invalid information is generated, indicating that the laser point was not captured or was severely obstructed. In this case, the laser generator remains in standby mode.
[0071] As a third aspect of the invention, the invention also provides a laser point detection system for endoscopic images, such as... Figure 6 As shown, the laser spot detection system 10 includes a laser spot detection model 110, which is trained using the training method described above. Image acquisition unit 120 is communicatively connected to the endoscope and is used to acquire endoscopic images of the target detection area acquired by the endoscope.
[0072] As a third aspect of the invention, the invention also provides a laser lithotripsy system 1, such as... Figure 7 As shown, the laser lithotripsy system 1 includes: Laser generator 20 is used to emit a laser beam to the target detection area to form a laser spot; Endoscope 30 is used to acquire images of the target detection area; and 10. Laser spot detection system for endoscopic images.
[0073] The endoscope 30 transmits the image of the target detection area to the CMOS / CCD sensor of the camera via optical fiber and outputs a standard digital video signal (such as HDMI 2.0). The laser point detection system 10 decodes the digital video signal to obtain multiple sequence frame images.
[0074] Optionally, this invention also employs a professional-grade image acquisition card based on a PCIe interface. This image acquisition card supports Direct Memory Access (DMA) and Zero-Copy technologies. Video frame data can be directly written to the GPU memory via the PCIe bus.
[0075] The laser generator communicates with the laser point detection system 10 in the following two ways: (1) Digital I / O control interface: used for millisecond-level real-time excitation control. The IPC outputs TTL level signals through a dedicated IO control card.
[0076] (2) Serial Communication Interface: Used for parameter adjustment and status monitoring. The laser spot detection system 10 can be connected to the control port of the laser generator via RS-232 or RS-485 interface.
[0077] Exemplary electronic devices As a fourth aspect of the present invention, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it executes the laser point detection method for an endoscope image described above.
[0078] Specifically, the electronic device includes a processor, memory, network interface, and input device connected via a device bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices and computer programs. The internal memory provides an environment for the operation of the operating devices and computer programs in the non-volatile storage medium. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it follows the steps of a laser point detection method for endoscopic images according to various embodiments of this specification, as described in the above embodiments.
[0079] The processor may include the main processor, as well as baseband chips, modems, etc.
[0080] The memory stores a program that executes the technical solution of this invention, and may also store operating devices and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.
[0081] The processor can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0082] The control handle may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.
[0083] Output devices may include devices that allow information to be output to the user, such as displays, printers, speakers, etc.
[0084] The communication interface may include any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.
[0085] The processor executes the program stored in the memory and calls other devices, which can be used to implement the various steps of the laser point detection method for endoscopic images provided in the above embodiments of this specification.
[0086] The electronic device may also include a display component and a voice component. The display component may be a liquid crystal display or an e-ink display. The input device of the controller may be a touch layer covering the display component, or a button, trackball, or touchpad set on the controller housing, or an external keyboard, touchpad, or mouse, etc.
[0087] Exemplary computer program products and storage media In addition to the methods and devices described above, the laser spot detection method for an endoscope image provided in the embodiments of this specification can also be a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to perform the steps in the laser spot detection method for an endoscope image according to various embodiments of this specification as described in the "Exemplary Methods" section above.
[0088] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0089] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this specification. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages.
[0090] Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps in a laser spot detection method for an endoscope image according to various embodiments of this specification as described in the "Exemplary Methods" section above.
[0091] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0093] The embodiments described above are merely illustrative of several implementation methods outlined in this specification. While the descriptions are specific and detailed, they should not be construed as limiting the scope of the solutions provided in this specification. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection for this patent should be determined by the appended claims.
Claims
1. A training method for a laser point detection model based on a deep neural network, characterized in that, The laser spot detection model is used to detect target laser spots in endoscopic images detected by the endoscope. The training method includes: A dataset is constructed based on multiple historical images acquired by the endoscope and the corresponding label data for each historical image, and the dataset is divided into a training set and a validation set. The training set is input into the laser point detection model to be trained. The laser point detection model to be trained includes a classification branch network and a regression branch network, and the classification branch network and the regression branch network process the historical image simultaneously. Using the laser point detection model to be trained, the prediction result of the historical image is obtained based on the historical images in the training set; Based on the prediction results of the historical images and the corresponding label data of the historical images, the model parameters of the laser point detection model to be trained are adjusted according to the composite loss function. The composite loss function is composed of a weighted sum of the classification loss function and the regression loss function.
2. The training method according to claim 1, characterized in that, The regression loss function is: ； in, L DIOU For the regression loss value, IOU The intersection-union ratio (IUU) between the predicted bounding box and the ground truth bounding box. The center point of the prediction box. The center point of the real bounding box. It is the Euclidean distance between the predicted bounding box and the ground truth bounding box. C The length of the diagonal of the smallest closure region containing the predicted bounding box and the ground truth bounding box.
3. The training method according to claim 1, characterized in that, The classification loss function is: ； in, FL ( p t ) represents the classification loss value. p t To predict probabilities, α t These are the weighting coefficients corresponding to the classification loss. γ For focusing parameters.
4. The training method according to claim 1, characterized in that, Using the laser point detection model to be trained, prediction results for historical images are obtained based on historical images in the training set, including: Extract feature maps from the historical images; Calculate the centrality between each location in the feature map and the target center point of the target bounding box; Calculate the offset between each position in the feature map and each edge of the target bounding box; Based on the centrality between each position in the feature map and the target center point of the target bounding box, and the offset between each position in the feature map and each side of the target bounding box, predict whether there is a target laser point in the feature map and the location information of the target laser point.
5. The training method according to claim 4, characterized in that, The training method also includes: The initial historical endoscopic images were divided into multiple regions; Detect the pixel value of each region block and determine the valid range of pixel values for multiple region blocks; The region block whose pixel values exceed the effective range is identified as the target region block; The pixel cumulative distribution function is determined based on the pixel values of the region block within the effective range; The mapping function corresponding to the target region block is determined based on the pixel cumulative distribution function and the pixel values of the target region block; The pixel values of the target region block are remapped according to the mapping function corresponding to the target region block.
6. The training method according to claim 4, characterized in that, The training method also includes: According to preset rules, the initial historical endoscopic images are cropped using a preset mask to obtain valid historical images; The valid historical images are dynamically cropped according to preset cropping parameters.
7. A method for laser point detection in endoscopic images, characterized in that, include: Acquire endoscopic images of the target detection area captured by the endoscope; The endoscope image is input into the laser point detection model for detection to detect target laser points in the endoscope image, and the detection result is output. The detection result includes the target laser point detection box and the unidentified target laser points. The laser point detection model is trained using the training method described in any one of claims 1-6.
8. The detection method according to claim 7, characterized in that, Also includes: Obtain the stone detection frame of the endoscopic image; When the detection result is a target laser point detection frame, the intersection-union ratio (IUU) of the stone detection frame and the target laser point detection frame is calculated. When the IUU is greater than or equal to a preset threshold, an excitation control command is generated. When the crossover ratio is less than the preset threshold, an early warning message is generated; wherein, the excitation control command is used to indicate that the laser generator is activated to emit laser light, and the early warning message is used to prompt the laser generator to be turned off and to issue an alarm message; When the detection result is that no target laser point is identified, invalid information is generated, which is used to indicate that the laser generator is kept in standby mode.
9. A laser point detection system for endoscopic images, characterized in that, include: A laser point detection model, wherein the laser point detection model is trained by the training method described in any one of claims 1-6; An image acquisition unit is communicatively connected to an endoscope and is used to acquire endoscopic images of the target detection area acquired by the endoscope.
10. A laser lithotripsy system, characterized in that, include: A laser generator is used to emit a laser beam to the target detection area to form a laser spot; An endoscope is used to acquire images of the target detection area; as well as The laser dot detection system for endoscopic images as described in claim 9.