Target identification method, device, surgical system, computer equipment and storage medium

Abstract

The present application relates to a target recognition method, device, computer equipment, storage medium and computer program product. The method comprises: inputting an image to be recognized into a preset target decision model, obtaining a model recognition result of a target in the image to be recognized and a confidence level corresponding to the model recognition result; determining a model recognition result whose confidence level is less than or equal to a preset confidence threshold as a model recognition result to be verified, and receiving a verification instruction for the model recognition result to be verified input from the outside; verifying the model recognition result to be verified based on the verification instruction to obtain a verification recognition result; and determining a target recognition result of the target in the image to be recognized based on the verification recognition result. The use of this method can improve the accuracy of target recognition.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a target recognition method, device, surgical system, computer equipment, storage medium and computer program product. Background Art

[0002] In multi-target resection surgery, due to the different relative sizes, positions, and shapes of the targets, the resection target decision needs to be re-made according to the new tissue exposure as the surgery progresses. Before the target decision is made, the accuracy of target identification must be ensured.

[0003] In traditional technology, the target recognition method first obtains the image to be recognized, then extracts features of the image to be recognized to obtain image features, and then recognizes the target based on the image features to obtain the recognition result of the target, which includes the location information and size information of the target.

[0004] However, the use of traditional technology may result in missed or false positives, resulting in low accuracy in target identification. Summary of the invention

[0005] Based on this, it is necessary to provide a target identification method, device, surgical system, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of target identification in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a target identification method. The method comprises:

[0007] Inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result;

[0008] Determine the model recognition result whose confidence is less than or equal to the preset confidence threshold as the model recognition result to be verified, and receive a verification instruction for the model recognition result to be verified input from the outside;

[0009] Verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result;

[0010] According to the verification recognition result, a target recognition result of the target point in the image to be recognized is determined.

[0011] In one embodiment, determining the target recognition result of the target in the image to be recognized according to the verification recognition result includes:

[0012] The model recognition result whose confidence is less than or equal to the preset confidence threshold and the verification recognition result are integrated to obtain the target recognition result of the target in the image to be recognized.

[0013] In one embodiment, the fusion of the model recognition result whose confidence is less than or equal to a preset confidence threshold and the verification recognition result to obtain the target recognition result of the target in the image to be recognized includes:

[0014] Performing weighted summation on the model recognition results whose confidence is less than or equal to a preset confidence threshold and the verification recognition results to obtain a weighted summation result;

[0015] According to the weighted summation result, a target recognition result of the target point in the image to be recognized is determined.

[0016] In one embodiment, the target decision model includes a target recognition model and a confidence calculation model; the step of inputting the image to be recognized into the preset target decision model to obtain the model recognition result of the target in the image to be recognized and the confidence corresponding to the model recognition result includes:

[0017] Inputting the image to be identified into the target recognition model to obtain a model recognition result of the target in the image to be identified and a decision vector corresponding to the model recognition result;

[0018] The decision vector is input into the confidence calculation model to obtain the confidence corresponding to the model recognition result.

[0019] In one embodiment, the step of inputting the image to be identified into the target recognition model to obtain a model recognition result of the target in the image to be identified and a decision vector corresponding to the model recognition result includes:

[0020] By using the target recognition model, multi-scale feature extraction is performed on the image to be recognized to obtain feature maps of different scales;

[0021] Obtaining image edge features in the feature maps of different scales through the target recognition model;

[0022] By using the target recognition model, the image edge features are respectively merged with the feature maps of corresponding scales to obtain image merge features of different scales;

[0023] The target recognition model is used to determine the model recognition result of the target in the image to be recognized and the decision vector corresponding to the model recognition result according to the fusion result of the image merging features of different scales and the image edge features.

[0024] In one embodiment, the confidence calculation model is a probabilistic graph neural network model; the step of inputting the decision vector into the confidence calculation model to obtain the confidence corresponding to the model recognition result includes:

[0025] The decision vector is input into the probabilistic graph neural network model to obtain the confidence level corresponding to the model recognition result.

[0026] In one embodiment, the receiving of the externally input verification instruction for the recognition result of the model to be verified includes:

[0027] Displaying the recognition result of the model to be verified through a display;

[0028] A verification instruction for the recognition result of the model to be verified is received from an external input through a human-computer interaction interface.

[0029] In one embodiment, the step of verifying the recognition result of the model to be verified based on the verification instruction to obtain the verification recognition result includes:

[0030] Parsing the verification instruction to obtain an instruction parsing result;

[0031] The recognition result of the model to be verified is verified according to the instruction analysis result to obtain a verification recognition result.

[0032] In one embodiment, the verification instructions include verification instructions of multiple different modes;

[0033] The step of parsing the verification instruction to obtain an instruction parsing result includes:

[0034] Analyze the calibration instructions of various modes separately to obtain different instruction analysis results;

[0035] The step of verifying the recognition result of the model to be verified according to the instruction parsing result to obtain the verification recognition result includes:

[0036] Fusion of different instruction parsing results to obtain instruction fusion results;

[0037] The recognition result of the model to be verified is verified according to the instruction fusion result to obtain a verification recognition result. In a second aspect, the present application also provides a target recognition device. The device includes:

[0038] An image processing module is used to input the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result;

[0039] An instruction receiving module, used for determining the model recognition result whose confidence is less than or equal to a preset confidence threshold as the model recognition result to be verified, and receiving an externally input verification instruction for the model recognition result to be verified;

[0040] A result verification module, used for verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result;

[0041] The target point determination module is used to determine the target point recognition result in the image to be recognized according to the verification recognition result.

[0042] In a third aspect, the present application also provides a surgical system. The system comprises:

[0043] The image host system is used to input the image to be identified into a preset target decision model, obtain the model recognition result of the target in the image to be identified and the confidence corresponding to the model recognition result, and determine the model recognition result whose confidence is less than or equal to the preset confidence threshold as the model recognition result to be verified;

[0044] A human-computer interaction interface, used to receive an externally input verification instruction for the recognition result of the model to be verified;

[0045] The image host system is also used to verify the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result, and determine the target recognition result of the target in the image to be recognized based on the verification recognition result.

[0046] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0047] Inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result;

[0048] Determine the model recognition result whose confidence is less than or equal to the preset confidence threshold as the model recognition result to be verified, and receive a verification instruction for the model recognition result to be verified input from the outside;

[0049] Verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result;

[0050] According to the verification recognition result, a target recognition result of the target point in the image to be recognized is determined.

[0051] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result;

[0053] Determine the model recognition result whose confidence is less than or equal to the preset confidence threshold as the model recognition result to be verified, and receive an externally input verification instruction for the model recognition result to be verified;

[0054] Verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result;

[0055] According to the verification recognition result, a target recognition result of the target point in the image to be recognized is determined.

[0056] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0057] Inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result;

[0058] Determine the model recognition result whose confidence is less than or equal to the preset confidence threshold as the model recognition result to be verified, and receive an externally input verification instruction for the model recognition result to be verified;

[0059] Verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result;

[0060] According to the verification recognition result, a target recognition result of the target point in the image to be recognized is determined.

[0061] The above-mentioned target recognition method, device, surgical system, computer equipment, storage medium and computer program product first select the model recognition result with a lower confidence level, and consider the corresponding detection object to be a suspected target. For the suspected target, it will be further verified whether it is a target according to the verification instruction transmitted from the outside, and finally the target recognition result that the detection object in the image to be identified is the target is determined based on the verification recognition result. It can be seen that in the processing of suspected targets, the present application will further adopt external verification instructions, such as manual verification instructions, to verify whether it is a target. In this way, for suspected targets with a higher recognition error rate, even if the model recognition error occurs, it can be corrected in time through verification, which is conducive to improving the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a schematic flow chart of a target identification method in one embodiment;

[0063] Figure 2 A diagram of an application environment of a target recognition method in an embodiment;

[0064] Figure 3 A data flow diagram of a target identification method in an embodiment;

[0065] Figure 4 A structural diagram of a multi-scale edge self-attention target recognition network model in one embodiment;

[0066] Figure 5 A data flow diagram of decision fusion in an embodiment;

[0067] Figure 6 A structural diagram of a probabilistic graph neural network model in one embodiment;

[0068] Figure 7 A schematic diagram of a process for modeling manual decision making in one embodiment;

[0069] Figure 8 is a schematic flow chart of a target identification method in one embodiment;

[0070] Fig. 9 An interactive effect diagram in an embodiment;

[0071] Fig.10 is a structural block diagram of a target recognition device in one embodiment;

[0072] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0073] In one embodiment, Figure 1 As shown, a target recognition method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] Step S102, inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result.

[0075] The confidence level corresponding to the model recognition result is used to evaluate whether the model recognition result of the target needs to be subsequently verified. The lower the confidence level, the higher the error rate of the model in identifying the detection object as the target; conversely, the higher the confidence level, the lower the error rate of the model in identifying the detection object as the target. The confidence level can be obtained by calculation from the decision vector extracted from the image to be identified.

[0076] In a specific example, if Figure 2 and Figure 3 As shown, the terminal includes a binocular endoscopic vision system, an image host system and a human-computer interaction system (including a human-computer interaction interface). The binocular endoscopic vision system is used to collect endoscopic images in the surgical scene in real time, obtain the image to be identified, and then transmit the image to be identified to the image host system. The image host system is used to input the image to be identified into a preset target decision model to obtain the model recognition result of the target and the confidence level corresponding to the model recognition result.

[0077] Step S104: determine the model recognition result whose confidence level is less than or equal to the preset confidence level threshold as the model recognition result to be verified, and receive an external verification instruction for the model recognition result to be verified.

[0078] Specifically, in one embodiment, the image host system compares the confidence level with a preset confidence threshold, and sends the model recognition result with a confidence level less than or equal to the confidence threshold as the model recognition result to be verified to the human-computer interaction system, so as to obtain an external input verification instruction for the model recognition result to be verified. The human-computer interaction system can realize visualization on the doctor's side and realize human-computer interaction through its own interface. It can be understood that sending the model recognition results of targets with high model recognition error rates to manual personnel for further manual verification processing can timely correct the model recognition results with model recognition errors, thereby effectively reducing the recognition error rate of the targets.

[0079] In another embodiment, the image host system may also send the recognition result of the model to be verified to an external result verification device. The result verification device analyzes the recognition result of the model to be verified, generates a verification instruction, and returns it to the image host system.

[0080] Step S106, verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result.

[0081] Specifically, the image host system parses the verification instruction to obtain the instruction parsing result, and then parses the model recognition result to be verified according to the instruction parsing result to obtain the verification recognition result. Optionally, in one embodiment, the instruction parsing result may be a verification recognition result identifier. The image host system parses the verification instruction to obtain the verification recognition result identifier, and then determines the verification recognition result of the model recognition result to be verified according to the verification recognition result identifier. In another embodiment, the image host system parses the verification instruction to obtain the instruction parsing result, and then parses the model recognition result to be verified according to the verification algorithm corresponding to the instruction parsing result to obtain the verification recognition result.

[0082] Step S108, determining the target recognition result of the target point in the image to be recognized according to the verification recognition result.

[0083] Specifically, in one embodiment, in order to consider the impact of the model and manual on target recognition, the image host system fuses the recognition result of the model to be verified and the verification recognition result to obtain the target recognition result of the target in the image to be recognized. Optionally, the image host system performs weighted fusion according to the weights corresponding to the recognition result of the model to be verified and the recognition result of the model to be verified, the verification recognition result and the weights corresponding to the verification recognition result to obtain the target recognition result of the target in the image to be recognized. In this embodiment, it is considered that there may be decision overlap between manual decision and machine decision, and the credibility of manual decision is usually higher than that of machine decision. Therefore, in decision fusion, machine decision and manual decision will be implemented in a weighted fusion manner. After fusion, the final decision is achieved through post-processing optimization.

[0084] In the above-mentioned target recognition method, the model recognition result with a lower confidence level is first selected, and the corresponding detection object is considered to be a suspected target. For the suspected target, it will be further verified whether it is a target based on the verification instruction transmitted from the outside, and finally the target recognition result that the detection object in the image to be recognized is the target is determined based on the verification recognition result. It can be seen that in the processing of suspected targets, the present application will further adopt external verification instructions, such as manual verification instructions, to verify whether it is a target. In this way, for suspected targets with a higher recognition error rate, even if the model recognition error occurs, it can be corrected in time through verification, which is conducive to improving the accuracy of target recognition.

[0085] In one embodiment, the target decision model includes a target recognition model and a confidence calculation model. Based on this, in one embodiment, a possible implementation method of the above step S102 "inputting the image to be recognized into the preset target decision model to obtain the model recognition result of the target in the image to be recognized and the confidence corresponding to the model recognition result" is involved. Based on the above embodiment, step S102 can be specifically implemented by the following steps:

[0086] Step S1022, inputting the image to be identified into the target recognition model to obtain the model recognition result of the target in the image to be identified and the decision vector corresponding to the model recognition result;

[0087] Step S1024, input the decision vector into the confidence calculation model to obtain the confidence corresponding to the model recognition result.

[0088] Specifically, the image host system inputs the image to be identified into a preset target recognition model to obtain the model recognition result of the target in the image to be identified and the decision vector corresponding to the model recognition result. The decision vector contains both low-level information such as position contour and high-level decision information extracted by the network. Then, the image host system inputs the decision vector into a preset confidence calculation model to obtain the confidence corresponding to the model recognition result. Optionally, Figure 4 and Figure 5 As shown, the image host system inputs the decision vector into a preset probabilistic graph neural network model to obtain the confidence level corresponding to the model recognition result, and calculates it through the cascade graph neural network structure of the model, ultimately achieving the evaluation of machine decisions and the diversion of manual decisions.

[0089] In this embodiment, a deep graph neural network is used to solve the probability estimation of surgical targets in suspected areas, make better use of decision vectors, and more accurately determine the needs of manual decision-making. At the same time, human-machine collaboration is used to correct machine-located targets instead of model accuracy evaluation, thereby improving the utilization rate of manual decision-making.

[0090] Furthermore, in one embodiment, a possible implementation method of the above step S1022 "inputting the image to be identified into the target recognition model to obtain the model recognition result of the target in the image to be identified and the decision vector corresponding to the model recognition result" is involved. Based on the above embodiment, step S1022 can be specifically implemented by the following steps:

[0091] Step S112, performing multi-scale feature extraction on the image to be identified through the target recognition model to obtain feature maps of different scales;

[0092] Step S114, obtaining image edge features in feature maps of different scales through a target recognition model;

[0093] Step S116, using the target recognition model, merging the image edge features with the feature maps of the corresponding scales to obtain image merging features of different scales;

[0094] Step S118, through the target recognition model, according to the fusion results of image merging features at different scales and image edge features, determine the model recognition result of the target in the image to be recognized and the decision vector corresponding to the model recognition result.

[0095] Alternatively, if Figure 6As shown, the target recognition model can be a multi-scale edge self-attention target recognition network model. The image host system extracts the size, position, color, contour and high-level features extracted by the network of the surgical target at different scale levels through the multi-scale feature extraction structure in the multi-scale edge self-attention target recognition network model, and these features constitute feature maps of different scales. The multi-scale feature extraction structure transmits the feature maps of different scales to the multiple scale feature calculation structures in the multi-scale edge self-attention target recognition network model. On the other hand, the multi-scale feature extraction structure transmits the feature maps of different scales to the edge feature calculation structure. The edge feature calculation structure decodes the edge contour information in the feature maps of different scales to obtain the image edge features. The edge feature calculation structure transmits the image edge features to the multiple scale feature calculation structures, provides edge information for the multiple scale feature calculation structures, and assists in the recognition of targets at different scales.

[0096] Each scale feature calculation structure merges the image edge features with the feature map of the corresponding scale to obtain image merge features of different scales. The fusion calculation structure in the multi-scale edge self-attention target recognition network model fuses the image merge features of different scales with the image edge features to obtain a fusion result. Finally, the surgical target positioning structure in the multi-scale edge self-attention target recognition network model determines the model recognition result of the target in the image to be recognized based on the fusion result. The decision vector calculation structure in the multi-scale edge self-attention target recognition network model determines the decision vector corresponding to the model recognition result based on the fusion result.

[0097] In this embodiment, a multi-scale self-supervised network is used to realize target recognition in complex environments, which can better meet the application of laparoscopic surgery scenarios and is conducive to improving the accurate recognition of difficult targets. Then, the target decision of the machine multi-scale edge self-attention target recognition network is integrated with the manual target decision, and the machine decision replaces part of the doctor's decision in the task of surgical target correction while retaining the surgeon's supervision and control ability of safety, which is conducive to improving the efficiency of the doctor's real-time target decision during surgery and improving the safety and reliability of surgery.

[0098] The part diverted to manual decision-making will be visualized through the image host system and transmitted to the human-computer interaction system for interactive manual decision-making. In one embodiment, a possible implementation method of the above step "receiving external input verification instructions for the recognition results of the model to be verified" is involved. Based on the above embodiment, this step can be specifically implemented by the following steps:

[0099] Step S1042, displaying the recognition result of the model to be verified on a display;

[0100] Step S1044, receiving a verification instruction for the recognition result of the model to be verified from an external input through the human-computer interaction interface.

[0101] Specifically, the image host system displays the recognition results of the model to be verified with a confidence level less than or equal to the confidence threshold to the doctor through the display in the human-computer interaction system. The human-computer interaction system supports doctors to make decisions through multiple interactive methods such as voice, gestures and interfaces, and receives manual verification instructions from doctors for the recognition results of the model to be verified from external input through the human-computer interaction interface in the human-computer interaction system.

[0102] In one embodiment, step S106 includes the following steps:

[0103] Step S1062, parsing the verification instruction to obtain the instruction parsing result;

[0104] Step S1064, based on the instruction parsing result of the manual decision instruction, the recognition result of the model to be verified is verified to obtain a verified recognition result.

[0105] Furthermore, the verification instructions include verification instructions of multiple different modes, such as voice verification instructions, interface verification instructions, gesture verification instructions, etc. In one embodiment, the image host system parses the verification instructions of multiple different modes respectively to obtain different instruction parsing results, then fuses the different instruction parsing results to obtain an instruction fusion result, and finally calibrates the recognition result of the model to be verified according to the instruction fusion result to obtain a verification recognition result.

[0106] Specifically, Figure 7 As shown, in the process of multimodal manual decision analysis and decision modeling, the process of transforming the high-level cognition of manual decision into machine-readable decision instructions is realized through five steps: multimodal instruction analysis, analysis instruction modeling, multimodal instruction fusion, decision preprocessing and manual decision generation, thereby reducing the operating difficulty of the surgeon and realizing the utilization of the surgeon's high-level cognition.

[0107] In this embodiment, by setting up a variety of human-computer interaction modes for decision-making, simultaneous analysis and processing of multiple input signals can be achieved without setting up separate logic for each input. This helps to reduce the difficulty of doctors' decision-making and the complexity of interaction, and better utilize advanced cognitive information.

[0108] In one embodiment, the image to be identified may be a nodule image. Based on this, the target identification method includes the following steps:

[0109] Step S201, the binocular endoscope vision system transmits the real-time sampling nodule surgery scene image to the image host system;

[0110] Step S202, the image host system adjusts image parameters;

[0111] Step S203, determining whether the target point update time has been reached;

[0112] If yes, then execute step S204, extract feature maps at different downsampling levels to perform feature calculation to adapt to complex nodules in the environment, and execute step S205, the edge-sensitive decoding layer performs feature extraction and decoding, extracts contour information of all suspected nodule areas to assist in determining the position and contour of the suspected nodules at different scales; if no, execute step S211;

[0113] Step S206, the detection sensitivity in the network is increased to screen out suspected nodule areas as much as possible, and then a target decision vector is generated according to the shape, position and other information of each detected suspected nodule area and the deep features extracted by the network through the fusion calculation and decision vector calculation module;

[0114] Step S207, the decision vector will be used in the probability estimation graph neural network to determine whether manual decision is required;

[0115] Step S208, determining whether manual decision is required;

[0116] If yes, then execute step S209. For the target that is determined to require manual decision, the system transmits the prompt information to the human-computer interaction system. After receiving the prompt information, the operator gives instructions through the voice, gesture, interface and other interfaces provided by the interactive system, and finally generates a manual decision through manual decision modeling. If no, then execute step S210.

[0117] Step S210, decision fusion;

[0118] Step S211, displaying the fusion decision through the naked eye 3D vision system.

[0119] In one embodiment, the binocular endoscope vision system includes a physical system such as a left eye lens, a right eye lens, a cold light source, a cable, an interface (including an image host system interface), and a micro-vision control system for adjusting image quality such as brightness and white balance. Using the binocular endoscope vision system as the system carrier, the overall structure is more compact and efficient. Naked-eye 3D cylindrical lens technology is used in the binocular endoscope vision system. Naked-eye 3D cylindrical lens technology is also called micro cylindrical lens 3D technology. Its principle is to make the image plane of the liquid crystal screen located on the focal plane of the lens, so that the pixels of the image under each cylindrical lens are divided into several sub-pixels, so that the lens can project each sub-pixel in different directions. So when the eyes watch the display screen from different angles, they see different sub-pixels, forming 3D vision. This technology is an important part of the naked-eye 3D interactive system.

[0120] The naked-eye 3D interactive system includes a two-way interface for interacting with the image host system, a display and interactive control system that provides an interface for the human-computer interaction system, and left and right eye lenses for realizing naked-eye 3D vision. In the human-to-machine decision-making process of human-computer interaction, the panel in the human-computer interaction system simultaneously receives three signal inputs: voice, gesture, and interface. The display and interactive control system will simply process the signal and transmit it to the image host system through the image host system interface for decision analysis and modeling.

[0121] In the process of sending visualization from machine to human, the visualization information sent by the image host system will be further processed in the display and interactive control system, and will be generated to the left eye vision and the right eye vision, and finally provide stereoscopic vision on the naked eye 3D display. In the process of human-computer interaction, the interactive data from the doctor's end to the image host system goes through two intermediate nodes in the input and output process. Among them, in the input process, the human-computer interaction system reads the doctor's abstract instructions through the input interface and simply encapsulates them. The image host system preliminarily parses them into readable information of the algorithm model, and then the information is transmitted to the algorithm layer as the input of the multimodal human-computer collaborative surgical target correction algorithm. In the output process, the final output result of the algorithm will pass through the same intermediate node and provide visualization for the surgeon through the naked eye 3D vision system of the human-computer interaction system.

[0122] The image host system includes an endoscope system image input interface, an image preprocessing and optimization module, a human-machine collaboration subsystem hardware platform, and a visualization post-processing module. The endoscope system image input interface is used to receive input from a binocular endoscope vision system. The image preprocessing and optimization module is used to optimize image quality and details through visual algorithms. The human-machine collaboration subsystem hardware platform is used to provide the computing power and computing environment required by the algorithm and provide input and output interfaces. The visualization post-processing module is used to visualize the results of the algorithm and manual decision-making.

[0123] In a specific application scenario, take the surgery of multiple micro-nodule target resection as an example. During the surgery, due to the multiple targets and their different relative sizes, positions, and shapes, the decision on the resection target needs to be re-made according to the new tissue exposure as the surgery progresses, which puts a lot of pressure on manual decision-making. In fact, most nodules have obvious characteristics, and machine decision-making can achieve a high accuracy rate. This system will greatly reduce the workload of manual decision-making. The interactive effects in this application are as follows: Fig. 9 Show.

[0124] In the embodiment of the present application, the credibility of the suspected target judgment obtained by the multi-scale edge self-attention target recognition network is evaluated, and a multi-modal human-computer interaction interface is provided to autonomously correct the difficult targets identified by the machine in a collaborative decision-making manner. This can change the current technical status of the separation of machine decision-making and manual decision-making in laparoscopic surgery, and realize human-computer collaboration to reduce the decision-making workload of doctors while ensuring safety.

[0125] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] Based on the same inventive concept, the embodiment of the present application also provides a target recognition device for implementing the target recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more target recognition device embodiments provided below can refer to the limitations of the target recognition method above, and will not be repeated here.

[0127] In one embodiment, Fig.10 As shown, a target recognition device is provided, comprising: an image processing module 202, an instruction receiving module 204, a result checking module 206 and a target determination module 208, wherein:

[0128] The image processing module 202 is used to input the image to be identified into a preset target decision model to obtain the model recognition result of the target in the image to be identified and the confidence level corresponding to the model recognition result;

[0129] The instruction receiving module 204 is used to determine the model recognition result whose confidence level is less than or equal to the preset confidence level threshold as the model recognition result to be verified, and receive a verification instruction for the model recognition result to be verified input from the outside;

[0130] A result verification module 206, used to verify the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result;

[0131] The target point determination module 208 is used to determine the target point recognition result in the image to be recognized according to the verification recognition result.

[0132] In one embodiment, the target determination module 208 is specifically used to fuse the model recognition results with confidence less than or equal to a preset confidence threshold and the verification recognition results to obtain the target recognition result of the target in the image to be recognized.

[0133] In one embodiment, the target determination module 208 is specifically used to perform weighted summation on the model recognition results and the verification recognition results whose confidence is less than or equal to a preset confidence threshold, and calculate the target recognition result of the target in the image to be recognized.

[0134] In one embodiment, the target decision model includes a target recognition model and a confidence calculation model; the image processing module 202 is specifically used to input the image to be recognized into the target recognition model to obtain the model recognition result of the target in the image to be recognized and the decision vector corresponding to the model recognition result; the decision vector is input into the confidence calculation model to obtain the confidence corresponding to the model recognition result.

[0135] In one embodiment, the image processing module 202 is specifically used to perform multi-scale feature extraction on the image to be identified through a target recognition model to obtain feature maps of different scales; obtain image edge features in feature maps of different scales through the target recognition model; merge the image edge features with the feature maps of corresponding scales through the target recognition model to obtain image merged features of different scales; determine the model recognition result of the target in the image to be identified and the decision vector corresponding to the model recognition result based on the fusion results of the image merged features of different scales and the image edge features through the target recognition model.

[0136] In one embodiment, the confidence calculation model includes a probabilistic graph neural network model; the image processing module 202 is specifically used to input the decision vector into a preset probabilistic graph neural network model to obtain the confidence corresponding to the machine recognition result.

[0137] In one embodiment, the manual decision module 204 is specifically used to display the recognition result of the model to be verified through a display; and receive a verification instruction for the recognition result of the model to be verified input from the outside through a human-computer interaction interface.

[0138] In one embodiment, the manual decision module 206 is specifically used to parse the verification instruction to obtain the instruction parsing result; and parse the recognition result of the model to be verified according to the instruction parsing result to obtain the verification recognition result.

[0139] In one embodiment, the verification instruction includes verification instructions of multiple different modes; the manual decision module 206 is specifically used to parse the verification instructions of multiple different modes respectively to obtain different instruction parsing results; fuse the different instruction parsing results to obtain the instruction fusion result; verify the recognition result of the model to be verified according to the instruction fusion result to obtain the verification recognition result. Each module in the above-mentioned target recognition device can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0140] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a target recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0141] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a surgical system is provided, comprising:

[0143] The image host system is used to input the image to be identified into a preset target decision model, obtain the model recognition result of the target in the image to be identified and the confidence level corresponding to the model recognition result, and determine the model recognition result with a confidence level less than or equal to a preset confidence threshold as the model recognition result to be verified;

[0144] A human-computer interaction interface, used to receive external input of a verification instruction for the recognition result of the model to be verified;

[0145] The image host system is also used to calibrate the recognition result of the model to be calibrated based on the calibration instruction to obtain the calibration recognition result, and determine the target recognition result of the target point in the image to be recognized based on the calibration recognition result.

[0146] In one embodiment, the image host system is specifically used to fuse the model recognition results with confidence less than or equal to a preset confidence threshold and the verification recognition results to obtain the target recognition result of the target in the image to be recognized.

[0147] In one embodiment, the image host system is specifically used to input the image to be identified into a target recognition model to obtain a model recognition result of the target in the image to be identified and a decision vector corresponding to the model recognition result; input the decision vector into a confidence calculation model to obtain a confidence corresponding to the model recognition result.

[0148] In one embodiment, the image host system is specifically used to perform multi-scale feature extraction on the image to be identified through a target recognition model to obtain feature maps of different scales; obtain image edge features in feature maps of different scales through the target recognition model; merge the image edge features with the feature maps of corresponding scales through the target recognition model to obtain image merged features of different scales; determine the model recognition result of the target in the image to be identified and the decision vector corresponding to the model recognition result based on the image merged features of different scales and the fusion results of the image edge features through the target recognition model.

[0149] In one embodiment, the system further comprises:

[0150] The display is used to display the recognition result of the model to be verified.

[0151] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0153] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0156] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target identification method, characterized in that: The method comprises: Inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result, wherein the image to be identified is an endoscopic image collected in real time during surgery; Determine the model recognition result whose confidence is less than or equal to the preset confidence threshold as the model recognition result to be verified, and receive an externally input verification instruction for the model recognition result to be verified; Based on the verification instruction, the recognition result of the model to be verified is verified to obtain a verification recognition result, wherein the verification recognition result is obtained by verifying the recognition result of the model to be verified according to a verification algorithm corresponding to the instruction parsing result, and the instruction parsing result is obtained by parsing the verification instruction; According to the verification recognition result, a target recognition result of the target point in the image to be recognized is determined.

2. The method according to claim 1, characterized in that: Determining the target recognition result of the target in the image to be recognized according to the verification recognition result includes: The model recognition result whose confidence is less than or equal to the preset confidence threshold and the verification recognition result are integrated to obtain the target recognition result of the target in the image to be recognized.

3. The method according to claim 2, characterized in that The step of fusing the model recognition result whose confidence is less than or equal to a preset confidence threshold and the verification recognition result to obtain a target recognition result of the target in the image to be recognized includes: The model recognition results whose confidence is less than or equal to the preset confidence threshold and the verification recognition results are weightedly summed to obtain the target recognition result of the target in the image to be recognized.

4. The method according to claim 1, characterized in that: The target decision model includes a target recognition model and a confidence calculation model; the step of inputting the image to be recognized into the preset target decision model to obtain the model recognition result of the target in the image to be recognized and the confidence corresponding to the model recognition result includes: Inputting the image to be identified into the target recognition model to obtain a model recognition result of the target in the image to be identified and a decision vector corresponding to the model recognition result; The decision vector is input into the confidence calculation model to obtain the confidence corresponding to the model recognition result.

5. The method according to claim 4, characterized in that The step of inputting the image to be identified into the target recognition model to obtain a model recognition result of the target in the image to be identified and a decision vector corresponding to the model recognition result includes: By using the target recognition model, multi-scale feature extraction is performed on the image to be recognized to obtain feature maps of different scales; Obtaining image edge features in the feature maps of different scales through the target recognition model; By using the target recognition model, the image edge features are respectively merged with the feature maps of corresponding scales to obtain image merge features of different scales; The target recognition model is used to determine the model recognition result of the target in the image to be recognized and the decision vector corresponding to the model recognition result according to the fusion result of the image merging features of different scales and the image edge features.

6. The method according to claim 4 or 5, characterized in that: The confidence calculation model is a probabilistic graph neural network model; the step of inputting the decision vector into the confidence calculation model to obtain the confidence corresponding to the model recognition result includes: The decision vector is input into the probabilistic graph neural network model to obtain the confidence level corresponding to the model recognition result.

7. The method according to any one of claims 1 to 4, characterized in that: The receiving of the externally input verification instruction for the recognition result of the model to be verified includes: Displaying the recognition result of the model to be verified through a display; A verification instruction for the recognition result of the model to be verified is received from an external input through a human-computer interaction interface.

8. The method according to claim 1, characterized in that The step of verifying the recognition result of the model to be verified based on the verification instruction to obtain the verification recognition result includes: Parsing the verification instruction to obtain an instruction parsing result; The recognition result of the model to be verified is verified according to the instruction analysis result to obtain a verification recognition result.

9. The method according to claim 8, characterized in that The verification instructions include verification instructions of multiple different modes; The step of parsing the verification instruction to obtain an instruction parsing result includes: Analyze the calibration instructions of various modes separately to obtain different instruction analysis results; The step of verifying the recognition result of the model to be verified according to the instruction parsing result to obtain the verification recognition result includes: Fusion of different instruction parsing results to obtain instruction fusion results; The recognition result of the model to be verified is verified according to the instruction fusion result to obtain a verification recognition result.

10. A target recognition device, characterized in that: The device comprises: An image processing module, used for inputting the image to be identified into a preset target decision model to obtain a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result, wherein the image to be identified is an endoscopic image collected in real time during surgery; An instruction receiving module, used for determining the model recognition result whose confidence is less than or equal to a preset confidence threshold as the model recognition result to be verified, and receiving an externally input verification instruction for the model recognition result to be verified; A result verification module, used for verifying the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result, wherein the verification recognition result is obtained by verifying the recognition result of the model to be verified according to the verification algorithm corresponding to the instruction parsing result, and the instruction parsing result is obtained by parsing the verification instruction; The target point determination module is used to determine the target point recognition result in the image to be recognized according to the verification recognition result.

11. A surgical system, characterized in that: The system comprises: An image host system, used for inputting the image to be identified into a preset target decision model, obtaining a model recognition result of the target in the image to be identified and a confidence level corresponding to the model recognition result, and determining a model recognition result whose confidence level is less than or equal to a preset confidence threshold as a model recognition result to be verified, wherein the image to be identified is an endoscopic image collected in real time during surgery; A human-computer interaction interface, used to receive an externally input verification instruction for the recognition result of the model to be verified; The image host system is also used to verify the recognition result of the model to be verified based on the verification instruction to obtain a verification recognition result, and to determine the target recognition result of the target in the image to be recognized based on the verification recognition result; wherein the verification recognition result is obtained by verifying the recognition result of the model to be verified according to the verification algorithm corresponding to the instruction parsing result, and the instruction parsing result is obtained by parsing the verification instruction.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images