Deep Learning-Based Image Recognition Systems and Methods

Abstract

This invention discloses an image recognition system and method based on deep learning, relating to the field of image processing technology. The system includes: acquiring a first image frame and a second image frame in a medical image that are temporally adjacent and have a time interval greater than a preset threshold; acquiring first texture data of the corresponding surgical instrument jaw region; locating a first target in the first image frame; locating at least one candidate target in the second image frame; extracting a first set of feature vectors based on the first image frame and at least one preceding image frame; extracting a second feature vector of at least one candidate target; generating a third feature vector based on a morphological prediction model using the first set of feature vectors and matching it with the second feature vector; and determining whether a continuing target is identified or whether a target is lost. Its beneficial effects include: enabling stable tracking and accurate identification of surgical instruments in low-frame-rate medical images, improving the continuity and robustness of target recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image recognition system and method based on deep learning. Background Technology

[0002] Image recognition technology started in the 1990s and has evolved from traditional computer vision algorithms based on edges and textures to statistical learning methods and then to deep learning models. In particular, after the emergence of deep neural networks, architectures such as convolutional neural networks, region proposal networks, YOLO and Transformer have driven the rapid development of object detection and image understanding, and have been widely used in scenarios such as autonomous driving, video surveillance, and smart manufacturing. Today, as the accuracy of algorithms continues to improve, research trends are gradually shifting to low-latency recognition, edge device deployment, and enhanced robustness of recognition in complex dynamic environments.

[0003] Existing deep learning image recognition methods can achieve high recognition accuracy by learning spatial features in static frames. However, for medical images, such as laparoscopic minimally invasive surgical images, which are usually acquired and recorded by endoscopic cameras for purposes such as intraoperative auxiliary recognition, postoperative behavior analysis, and clinical teaching, the frame rate is often set to low during acquisition or processing due to factors such as long operation time, large image storage pressure, high cost of manual annotation, and limited bandwidth of remote transmission. In some cases, these images are even provided in 1fps format in public research datasets. Compared with high frame rate videos, the time interval between adjacent frames in low frame rate videos is large, which may cause significant positional shifts, angular rotations, or deformations of surgical instruments between frames. This makes it difficult for traditional image recognition methods based on spatial feature consistency to stably extract target features or accurately complete the target association between frames. Problems such as target identity loss, jumps, or misidentification are prone to occur. This recognition discontinuity will directly affect the stability and accuracy of key applications such as instrument trajectory reconstruction, surgical stage recognition, and intraoperative navigation.

[0004] Therefore, a deep learning-based image recognition system and method are proposed. Summary of the Invention

[0005] In view of the above-mentioned prior art, this application is hereby filed. Embodiments of this application provide a deep learning-based image recognition system and method, which can improve the accuracy of continuous identification of surgical instruments in low frame rate medical images.

[0006] According to one aspect of this application, a deep learning-based image recognition method is provided, comprising: acquiring a first image frame and a second image frame in a medical image that are temporally adjacent and have a time interval greater than a preset interval threshold; acquiring first texture data of the jaw region of a corresponding surgical instrument in the medical image; locating a first target in the first image frame based on the first texture data; locating at least one candidate target in the second image frame, the candidate target being a possible continuation target of the first target in the second image frame; extracting a first feature vector of the first target in the first image frame and at least one preceding image frame to form a first feature vector set; extracting a second feature vector of each candidate target; generating a third feature vector representing the evolutionary state of the first target between the first image frame and the second image frame based on the first feature vector set using a deep learning-based morphological prediction model; matching the third feature vector with each of the second feature vectors to determine whether there exists a second feature vector with the highest matching degree exceeding a preset matching threshold; if so, determining the candidate target corresponding to the second feature vector as a continuation target of the first target in the second image frame; otherwise, determining that the first target is lost in the second image frame.

[0007] According to another aspect of this application, a deep learning-based image recognition system is provided, comprising: an image frame acquisition module for acquiring a first image frame and a second image frame in a medical image that are temporally adjacent and have a time interval greater than a preset interval threshold; a texture data acquisition module for acquiring first texture data of the jaw region of a corresponding surgical instrument in the medical image; a first target localization module for localizing a first target in the first image frame based on the first texture data; a candidate target localization module for localizing at least one candidate target in the second image frame, wherein the candidate target is a possible continuation target of the first target in the second image frame; and a historical feature extraction module for extracting at least one image of the first target in the first image frame and prior to it. The first feature vector in the frame constitutes a first feature vector set; the candidate feature extraction module is used to extract the second feature vector of each candidate target; the morphology prediction module is used to generate a third feature vector representing the evolution state of the first target between the first image frame and the second image frame based on the first feature vector set using a deep learning-based morphology prediction model; the matching decision module is used to match the third feature vector with each of the second feature vectors, and determine whether there is a second feature vector with the highest matching degree that exceeds a preset matching threshold. If there is, the candidate target corresponding to the second feature vector is determined as the continuation target of the first target in the second image frame; otherwise, it is determined that the first target is lost in the second image frame.

[0008] According to another aspect of this application, an electronic device is provided, including a memory and a processor, the memory being used to store computer-executable instructions, and the processor being used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method described above.

[0009] According to another aspect of this application, a computer storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, implement the steps of the method described above.

[0010] Compared with the prior art, the deep learning-based image recognition system and method according to the embodiments of this application can achieve stable tracking and accurate identification of surgical instruments in low frame rate medical images, improve the continuity and robustness of target recognition, effectively reduce recognition interruption or misjudgment caused by image blurring, insufficient frame rate or target occlusion, and thus improve the reliability of image-assisted analysis and navigation. Attached Figure Description

[0011] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is a flowchart of the deep learning-based image recognition method of the present invention.

[0013] Figure 2 This is a flowchart illustrating the secondary search for the continuation target in the deep learning-based image recognition method of this invention.

[0014] Figure 3 This is a flowchart illustrating the dynamic adjustment of the preset matching threshold in the deep learning-based image recognition method of the present invention.

[0015] Figure 4 This is a block diagram of the deep learning-based image recognition system of the present invention.

[0016] Figure 5 This is a block diagram of an electronic device according to the present invention. Detailed Implementation

[0017] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0018] Application Overview

[0019] In existing technologies, image recognition technology has evolved from traditional computer vision algorithms to deep learning models, but it still faces significant challenges in low frame rate medical imaging scenarios. For low frame rate medical images, the large interval between adjacent frames can cause surgical instruments to shift significantly in position, rotate at an angle, or deform. Traditional recognition methods based on spatial feature consistency are difficult to effectively associate targets in previous and subsequent frames, resulting in loss or misidentification of target identity, which affects the accuracy of instrument trajectory reconstruction and intraoperative navigation.

[0020] In existing technologies, the core difficulty in target association in low frame rate images lies in the lack of inter-frame state and insufficient prediction of morphological changes. Through analysis, it has been found that the texture of key parts of the instrument has relative stability and can be used as an initial positioning basis. At the same time, the target's motion trajectory in time is continuous, and the evolution trend can be predicted by modeling historical features. Based on this, the basic concept of this application is to propose a technical route that combines texture feature localization, multi-frame feature accumulation and deep learning prediction model. By predicting the target's evolution state and dynamically matching it with candidate features, the problem of inter-frame association can be solved.

[0021] Exemplary methods

[0022] Figures 1-3 The illustration shows an image recognition method based on deep learning according to an embodiment of this application, including: acquiring a first image frame and a second image frame in a medical image that are temporally adjacent and have a time interval greater than a preset interval threshold; acquiring first texture data of the jaw region of a corresponding surgical instrument in the medical image; locating a first target in the first image frame based on the first texture data; locating at least one candidate target in the second image frame, the candidate target being a possible continuation target of the first target in the second image frame; extracting a first feature vector of the first target in the first image frame and at least one preceding image frame to form a first feature vector set; extracting a second feature vector of each candidate target; generating a third feature vector representing the evolutionary state of the first target between the first image frame and the second image frame based on the first feature vector set using a morphological prediction model based on deep learning; matching the third feature vector with each second feature vector to determine whether there exists a second feature vector with the highest matching degree exceeding a preset matching threshold; if so, determining the candidate target corresponding to the second feature vector as a continuation target of the first target in the second image frame; otherwise, determining that the first target is lost in the second image frame.

[0023] The first texture data refers to the set of microscopic texture features on the surface of the surgical instrument jaws. In this application, the first texture data can be obtained by matching it with a pre-built database of surgical instrument texture features. The morphological prediction model refers to a neural network that can learn the temporal evolution of the target's morphology. Specifically, it can be implemented using a long short-term memory network with an attention mechanism, which can predict the morphological change trend of the target during frame intervals by analyzing historical feature sequences.

[0024] Specifically, this method filters keyframe pairs with a sufficiently large time span by using a preset interval threshold, avoiding the processing of redundant frame data. Initial target localization is performed using the unique texture features of the instrument jaw region, ensuring basic detection accuracy. After establishing a candidate target set in subsequent frames, a feature set containing temporal evolution information is constructed by extracting the target's depth features in the current frame and multiple historical frames. The morphological prediction model performs temporal modeling on this feature set, generating predicted features reflecting the target's possible evolutionary states. By performing similarity matching between the predicted features and the real-time features of the candidate targets, accurate cross-frame target association is ultimately achieved.

[0025] Through the above technical solution, this application effectively solves the problem of difficult cross-frame association of surgical instruments in low frame rate medical images. By combining texture feature localization with multi-frame feature prediction, the accuracy of cross-frame target matching is improved while ensuring the initial detection accuracy, ensuring the continuity of the instrument recognition process, and providing a reliable data foundation for subsequent surgical navigation and behavior analysis.

[0026] This application further proposes that before determining that the first target is lost in the second image frame, the method includes: determining a predicted candidate region of the first target in the second image frame based on a third feature vector, wherein the predicted candidate region is an image region of a preset size centered at the position corresponding to the third feature vector; extracting multiple new candidate targets within the predicted candidate region, and extracting a fourth feature vector for each new candidate target; matching the third feature vector with each fourth feature vector, and if there is a fourth feature vector with the highest matching degree that exceeds a preset secondary threshold, then determining the new candidate target corresponding to the fourth feature vector as the continuation target of the first target in the second image frame.

[0027] The predicted candidate region refers to the candidate search range generated based on the spatial location corresponding to the third feature vector output by the morphological prediction model. Specifically, it can be implemented as a rectangular region centered at the predicted coordinates with a side length of a preset pixel value, used to cover the possible displacement range of the instrument in low frame rate images. The new candidate target refers to the potential instrument region extracted from the predicted candidate region using a sliding window or region proposal network. Specifically, it can be implemented by using a multi-scale sliding window combined with an edge detection algorithm to generate candidate boxes, used to supplement omissions in the original candidate target set. The preset secondary threshold refers to the feature similarity judgment standard below the preset matching threshold. Specifically, it can be implemented as a fixed value of 0.8 times the primary matching threshold or a value dynamically adjusted based on historical matching data, used to relax the matching conditions within the expanded search range.

[0028] Specifically, when the initial candidate target matching fails, an image region covering the possible displacement range of the instrument is generated based on the spatial coordinates corresponding to the third feature vector output by the morphological prediction model. Multiple new candidate targets are extracted and their depth features are extracted by performing dense sampling or region proposal operations within this region. The similarity between the third feature vector and the depth features of the new candidate targets is calculated. When the highest matching degree exceeds a secondary threshold, the candidate target is associated as a continuation instance of the same instrument. For example, when the instrument undergoes rapid rotation in low frame rate images, causing a mismatch in the features of the original candidate targets, a predicted candidate region is used to cover the new position of the instrument after rotation, and a secondary matching mechanism is used to capture the deformed instrument instance.

[0029] Through the above technical solution, this application improves the continuous recognition capability of instrument targets in low frame rate medical images by expanding the candidate region and matching again when the instrument moves rapidly or undergoes a sudden change in shape, and further reduces the failure of target association caused by excessive changes between frames.

[0030] This application further proposes that before determining whether the matching degree exceeds a preset matching threshold, the method includes: calculating the average sharpness of the first image frame and the second image frame; determining the average sharpness: if it is lower than a first sharpness threshold, then setting the preset matching threshold to the first matching threshold; if it is not lower than the first sharpness threshold and is lower than a second sharpness threshold, then setting the preset matching threshold to the second matching threshold; if it is not lower than the second sharpness threshold, then setting the preset matching threshold to the third matching threshold; wherein the values ​​of the first matching threshold, the second matching threshold, and the third matching threshold increase sequentially.

[0031] Average sharpness refers to an indicator that quantifies the degree of detail preservation in an image by calculating the statistical average of pixel gradient magnitudes across image frames. Specifically, it can be achieved by using the Sobel operator to extract image edges and then calculating the average gradient magnitude. This indicator reflects the degree of image blur. The first and second sharpness thresholds are pre-calibrated image quality grading thresholds, determined experimentally. These can be determined by statistically analyzing the inflection points of feature matching accuracy at different sharpness levels on a standard test set, used to classify image quality into low, medium, and high levels. The preset matching threshold is the critical value for feature vector similarity judgment, specifically implemented using cosine similarity or Euclidean distance transformation confidence values. Its numerical setting directly affects the strictness of the matching results.

[0032] Specifically, in low frame rate medical image processing, the Sobel operator is used to calculate the gradient magnitude of each pixel in two frames of images, and the average of all pixel gradient magnitudes is taken as the average sharpness. When this value is lower than a pre-defined first sharpness threshold, it indicates that the image is significantly blurred, and the reliability of feature extraction decreases. The matching threshold is then lowered to the first matching threshold to avoid missed detections due to feature bias. When the average sharpness is in the middle range, a second matching threshold with moderate stringency is used to balance recognition accuracy and recall. When the image quality reaches a high-resolution standard, a higher third matching threshold is used to suppress false matches. This hierarchical control mechanism establishes a positive correlation between image quality and the matching threshold, effectively mitigating the problem of missed detections in low-quality images while ensuring recognition accuracy in high-resolution scenarios.

[0033] Through the above technical solution, this application can maintain the continuous tracking of the instrument target by adaptively reducing the matching requirements when there is intermittent quality degradation in low frame rate medical images, and automatically improve the matching standard when the image quality is restored to ensure recognition accuracy, effectively solving the problem of misjudgment of instrument recognition caused by fluctuations in image clarity.

[0034] This application further proposes that after determining the average sharpness, the method includes: extracting the region in the first image frame other than the first target as the first background region; extracting the region in the second image frame other than at least one candidate target region as the second background region; calculating the average texture complexity of the first background region and the second background region; determining whether the average texture complexity is higher than a preset complexity threshold; if so, increasing the preset matching threshold according to the difference between the average texture complexity and the preset complexity threshold through a preset first adjustment function.

[0035] The first and second background regions refer to the non-target regions remaining after excluding the identified instrument target regions and suspected instrument target regions in the first image frame through image segmentation technology. Specifically, edge detection-based region segmentation methods can be used to isolate the interference of instrument targets on background texture analysis. Average texture complexity refers to the statistical quantity of background region texture features calculated using the gray-level co-occurrence matrix. Specifically, a weighted combination of energy, contrast, and entropy values ​​can be used to quantify the distribution density of textures similar to the instrument target in the background. The preset complexity threshold is a critical value for background interference intensity obtained through training on historical data. Specifically, a probability threshold output by a machine learning classifier can be used to determine whether to initiate matching threshold adjustment. The first adjustment function is a mathematical formula that dynamically adjusts the matching threshold based on the difference between background complexity and the threshold. Specifically, a linear proportional function or an exponential function can be used to establish a positive correlation between background interference intensity and matching strictness.

[0036] Specifically, firstly, non-target regions in the preceding and following frames are extracted as background analysis objects using a region segmentation method. Then, a texture feature extraction algorithm is used to quantitatively evaluate the two background regions, calculating a comprehensive index reflecting texture complexity. When this index exceeds a preset threshold, it indicates the presence of numerous interfering features in the background similar to the texture of instrument jaws. At this point, a predefined adjustment function is used to increase the matching threshold requirement based on the deviation between the actual complexity and the threshold, ensuring that candidate targets must have a higher similarity to the predicted features to be identified as continuing targets. This dynamic adjustment avoids mismatches caused by similar textures in complex backgrounds while preserving the matching possibility of the true target within a reasonable deformation range.

[0037] Through the above technical solution, this application can dynamically optimize the matching threshold setting according to the background texture features of the actual surgical scene, effectively distinguish the movement of real instruments from background interference features in low frame rate medical images with complex tissue textures, reduce the probability of mismatch caused by similar textures, and avoid missing the match of real targets due to overly strict threshold settings, thereby further ensuring the stability of continuous tracking of instrument targets between consecutive frames.

[0038] This application further proposes:

[0039] Locating a first target in a first image frame based on first texture data includes: calculating a first similarity between each image region in the first image frame and the first texture data; and identifying the image region with the highest first similarity that exceeds a preset similarity threshold as the first target.

[0040] Locating at least one candidate target in the second image frame includes: calculating the second similarity between each image region in the second image frame and the first texture data; and selecting image regions whose second similarity exceeds a preset similarity threshold as candidate targets.

[0041] The first and second similarities refer to the degree of matching between image regions and standard texture features. Specifically, this can be achieved using structural similarity index or cosine similarity calculation of convolutional neural network features, used to quantify the correlation between regions and target textures. The preset similarity threshold refers to the minimum similarity requirement for filtering valid matching regions.

[0042] This application further proposes to extract a first feature vector of a first target in a first image frame and at least one preceding image frame to obtain a first feature vector set, including: obtaining at least one preceding image frame of the first image frame; extracting the position information of the first target in the first image frame; predicting the possible position region of the first target in at least one image frame based on the position information; determining whether there is an associated image region in at least one image frame that is located in the possible position region and has a similarity to the first texture data exceeding a preset judgment threshold; and extracting each associated image region and the first feature vector of the first target to jointly constitute the first feature vector set.

[0043] In this context, "at least one preceding image frame" refers to a series of historical frames arranged chronologically before the current frame. This can be achieved by using a sliding window mechanism to select the most recent N frames, thus expanding the temporal feature sources. Location information refers to the spatial coordinates and region of the target within the current frame. The potential location region refers to the range of the target in historical frames inferred from the current frame's location. This can be achieved by using a motion trajectory extrapolation algorithm to generate a rectangular region, thus limiting the search range of historical frames. Associated image regions refer to image blocks in historical frames that meet spatial constraints and have matching textures. These can be filtered using a combination of region segmentation and similarity calculation to ensure consistency between historical features and the current target.

[0044] Specifically, in low frame rate medical image processing, a sliding window is used to acquire several historical frames preceding the current frame. The motion trajectory of the target is constructed using the detection box coordinates in the current frame, and the possible rectangular regions where the target may exist in the historical frames are calculated. Image segmentation is performed within the predicted region for each historical frame, and the similarity between each sub-region and a preset jaw texture is calculated. Candidate regions exceeding a threshold are selected. The feature vectors of these candidate regions are then aggregated with the target features of the current frame to form a dataset containing time-series features. This process effectively integrates multi-frame feature information through a dual spatiotemporal constraint mechanism: it utilizes motion continuity to narrow the search range and texture similarity to eliminate interfering regions.

[0045] Exemplary System

[0046] Figure 4 The figure illustrates a deep learning-based image recognition system according to an embodiment of this application, comprising: an image frame acquisition module for acquiring a first image frame and a second image frame in a medical image that are temporally adjacent and have a time interval greater than a preset interval threshold; a texture data acquisition module for acquiring first texture data of the jaw region corresponding to a surgical instrument in the medical image; a first target localization module for localizing a first target in the first image frame based on the first texture data; a candidate target localization module for localizing at least one candidate target in the second image frame, wherein the candidate target is a possible continuation target of the first target in the second image frame; and a historical feature extraction module for extracting at least one image of the first target in the first image frame and preceding images. The first feature vector in the frame constitutes the first feature vector set; the candidate feature extraction module is used to extract the second feature vector of each candidate target; the morphology prediction module is used to generate a third feature vector representing the evolution state of the first target between the first image frame and the second image frame by using a deep learning-based morphology prediction model; the matching decision module is used to match the third feature vector with each second feature vector, and determine whether there is a second feature vector with the highest matching degree that exceeds a preset matching threshold. If it exists, the candidate target corresponding to the second feature vector is determined as the continuation target of the first target in the second image frame; otherwise, it is determined that the first target is lost in the second image frame.

[0047] In one example, before the matching decision module determines that the first target is lost in the second image frame, it further includes: determining a predicted candidate region of the first target in the second image frame based on the third feature vector, wherein the predicted candidate region is an image region of a preset size centered at the position corresponding to the third feature vector; extracting multiple new candidate targets within the predicted candidate region, and extracting the fourth feature vector of each new candidate target respectively; matching the third feature vector with each fourth feature vector, and if there is a fourth feature vector with the highest matching degree that exceeds a preset secondary threshold, then the new candidate target corresponding to the fourth feature vector is determined as the continuation target of the first target in the second image frame.

[0048] In one example, before determining whether the matching degree exceeds a preset matching threshold, the matching decision module further includes: calculating the average sharpness of the first image frame and the second image frame; determining the average sharpness: if it is lower than a first sharpness threshold, then the preset matching threshold is set to the first matching threshold; if it is not lower than the first sharpness threshold and is lower than a second sharpness threshold, then the preset matching threshold is set to the second matching threshold; if it is not lower than the second sharpness threshold, then the preset matching threshold is set to the third matching threshold; wherein, the values ​​of the first matching threshold, the second matching threshold, and the third matching threshold increase sequentially.

[0049] In one example, after determining the average sharpness, the matching decision module further includes: extracting the region in the first image frame other than the first target as the first background region; extracting the region in the second image frame other than at least one candidate target region as the second background region; calculating the average texture complexity of the first background region and the second background region; determining whether the average texture complexity is higher than a preset complexity threshold; if so, increasing the preset matching threshold according to the difference between the average texture complexity and the preset complexity threshold through a preset first adjustment function.

[0050] In one example, the first target localization module locates the first target in the first image frame based on the first texture data by: calculating the first similarity between each image region in the first image frame and the first texture data; and determining the image region with the highest first similarity and exceeding a preset similarity threshold as the first target.

[0051] In one example, the candidate target localization module locates at least one candidate target in the second image frame by: calculating a second similarity between each image region in the second image frame and the first texture data; and selecting image regions whose second similarity exceeds a preset similarity threshold as candidate targets.

[0052] In one example, the historical feature extraction module extracts the first feature vector of the first target in the first image frame and at least one preceding image frame to form the first feature vector set, including: obtaining at least one preceding image frame of the first image frame; extracting the position information of the first target in the first image frame; predicting the possible position region of the first target in at least one image frame based on the position information; determining whether there is an associated image region in at least one image frame that is located in the possible position region and has a similarity to the first texture data exceeding a preset judgment threshold; and extracting each associated image region and the first feature vector of the first target to form the first feature vector set.

[0053] Exemplary electronic devices

[0054] Figure 5 An electronic device according to an embodiment of this application is illustrated. The electronic device may be the mobile device itself, or a standalone device independent of it, which may communicate with the mobile device to receive collected input signals from it and send selected target driving behaviors to it.

[0055] Figure 5 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0056] like Figure 5 As shown, the electronic device includes one or more processors and memory.

[0057] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0058] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the driving behavior decision-making methods of the various embodiments of this application described above, and / or other desired functions.

[0059] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0060] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device may include any other suitable components depending on the specific application.

[0061] Exemplary computer-readable media

[0062] Embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the driving behavior decision-making methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0063] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0064] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0065] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0066] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0067] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0068] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A deep learning-based image recognition method applied to device recognition in low frame rate medical images, characterized in that, include: Acquire the first and second image frames in medical images that are temporally adjacent and have a time interval greater than a preset interval threshold; Obtain the first texture data of the jaw region of the corresponding surgical instrument in the medical image; The first target is located in the first image frame based on the first texture data; Locate at least one candidate target in the second image frame, wherein the candidate target is a possible continuation target of the first target in the second image frame; Extract the first feature vector of the first target in the first image frame and at least one preceding image frame to form a first feature vector set; Extract the second feature vector of each of the candidate targets; A third feature vector representing the evolutionary state of the first target between the first image frame and the second image frame is generated based on the first feature vector set using a deep learning-based morphological prediction model. The third feature vector is matched with each of the second feature vectors to determine whether there is a second feature vector with the highest matching degree that exceeds a preset matching threshold. If there is, the candidate target corresponding to the second feature vector is determined as the continuation target of the first target in the second image frame; otherwise, it is determined that the first target is lost in the second image frame.

2. The image recognition method based on deep learning according to claim 1, characterized in that, The step of determining that the first target is lost in the second image frame further includes: The prediction candidate region of the first target in the second image frame is determined based on the third feature vector, wherein the prediction candidate region is an image region of a preset size centered at the position corresponding to the third feature vector; Multiple new candidate targets are extracted within the predicted candidate region, and the fourth feature vector of each new candidate target is extracted respectively; The third feature vector is matched with each of the fourth feature vectors. If there is a fourth feature vector with the highest matching degree that exceeds a preset secondary threshold, then the new candidate target corresponding to the fourth feature vector is determined as the continuation target of the first target in the second image frame.

3. The image recognition method based on deep learning according to claim 1, characterized in that, Before determining whether the matching degree exceeds the preset matching threshold, the following steps are also included: Calculate the average sharpness of the first image frame and the second image frame; Determine the average sharpness: If the resolution is lower than the first clarity threshold, then the preset matching threshold is set to the first matching threshold; If the resolution is not lower than the first resolution threshold but lower than the second resolution threshold, then the preset matching threshold is set to the second matching threshold. If the resolution is not lower than the second clarity threshold, then the preset matching threshold is set to the third matching threshold; Among them, the first matching threshold, the second matching threshold, and the third matching threshold are incremented.

4. The image recognition method based on deep learning according to claim 3, characterized in that, The process of determining the average sharpness further includes: Extract the region in the first image frame excluding the first target as the first background region; Extract the region in the second image frame other than the at least one candidate target region as the second background region; Calculate the average texture complexity of the first background region and the second background region; Determine whether the average texture complexity is higher than a preset complexity threshold. If so, increase the preset matching threshold based on the difference between the average texture complexity and the preset complexity threshold using a preset first adjustment function.

5. The image recognition method based on deep learning according to claim 1, characterized in that, The step of locating the first target in the first image frame based on the first texture data includes: Calculate the first similarity between each image region in the first image frame and the first texture data; The image region with the highest similarity that exceeds a preset similarity threshold is identified as the first target.

6. The image recognition method based on deep learning according to claim 5, characterized in that, Locating at least one candidate target in the second image frame includes: Calculate the second similarity between each image region in the second image frame and the first texture data; Image regions whose second similarity exceeds the preset similarity threshold are selected as candidate targets.

7. The image recognition method based on deep learning according to claim 1, characterized in that, The step of extracting the first feature vector of the first target in the first image frame and at least one preceding image frame to form a first feature vector set includes: Obtain at least one image frame preceding the first image frame; Extract the position information of the first target in the first image frame; Based on the location information, predict the possible location region of the first target in the at least one image frame; Determine whether there exists an associated image region in the at least one image frame that is located within the possible location region and whose similarity to the first texture data exceeds a preset determination threshold; The first feature vector set is formed by extracting each of the associated image regions and the first feature vector of the first target.

8. A deep learning-based image recognition system, characterized in that, include: The image frame acquisition module is used to acquire the first and second image frames in medical images that are temporally adjacent and have a time interval greater than a preset interval threshold. The texture data acquisition module is used to acquire the first texture data of the jaw region of the corresponding surgical instrument in the medical image; The first target localization module is used to locate the first target in the first image frame based on the first texture data; A candidate target localization module is used to locate at least one candidate target in the second image frame, wherein the candidate target is a possible continuation target of the first target in the second image frame; The historical feature extraction module is used to extract the first feature vector of the first target in the first image frame and at least one preceding image frame to form a first feature vector set. A candidate feature extraction module is used to extract the second feature vector of each of the candidate targets; The morphology prediction module is used to generate a third feature vector representing the evolutionary state of the first target between the first image frame and the second image frame based on the first feature vector set using a deep learning-based morphology prediction model. The matching decision module is used to match the third feature vector with each of the second feature vectors, and determine whether there is a second feature vector with the highest matching degree that exceeds a preset matching threshold. If there is, the candidate target corresponding to the second feature vector is determined as the continuation target of the first target in the second image frame; otherwise, it is determined that the first target is lost in the second image frame.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 7.

10. A computer storage medium storing computer-executable instructions thereon, characterized in that: When the computer-executable instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images