Endoscopic image processing method, device, electronic device, and computer program

Abstract

An endoscopic image processing method, device, electronic device, and storage medium, the method including: acquiring an endoscopic video stream carrying an original endoscopic image; detecting the original endoscopic image in a corresponding video frame by a first thread; transmitting the detection result of the original endoscopic image to an integration module; forming a control command by the integration module according to the detection result of the original endoscopic image; and adjusting an output result in the second thread by a second thread in response to the control command.

Description

[Technical field]

[0001] This application is filed based on and claims priority to a Chinese patent application having application number 202010076394.4 and filing date on January 23, 2020, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to medical image processing technology, and in particular to an endoscopic image processing method, device, electronic device, and storage medium using artificial intelligence technology. [Background technology]

[0003] Up to now, deep learning for recognizing various categories has been an important means for solving the classification of large amounts of data in various application scenarios. For example, in application scenarios such as image and natural language processing, large-scale classification and recognition can be realized for large amounts of data, so that the relevant classification prediction results can be obtained quickly and accurately, and the realization of the functions of the corresponding application scenarios can be accelerated.

[0004] Various images of medical images are continuously generated, for example, by continuously taking pictures of the digestive tract using an endoscope, forming a large amount of data, and it is necessary to perform classification prediction to realize large-scale classification and recognition.

[0005] In the related art, target detection schemes in real-time video streams (e.g., colonoscopy video streams) generally integrate relatively high-precision target detection models (e.g., various target detection algorithms such as Retina Net, Faster RCNN, YOLO, etc.), each of which sequentially inputs image frames in the video stream into the model to detect targets. However, the processing speed of the model is limited and does not match the speed of the real-time video stream, so the detection model in the related art cannot match the frame rate of the real-time video, which affects the output effect of the model. Summary of the Invention [Means for solving the problem]

[0006] In view of this, the embodiments of the present application provide an endoscopic image processing method, device, electronic device, and storage medium that can process an original endoscopic image using parallel first and second threads, thereby enabling the output result to be adapted to the usage environment of the endoscopic video stream.

[0007] The technical solutions of the embodiments of the present application are realized as follows. An embodiment of the present application provides an endoscopic image processing method, obtaining an endoscopic video stream comprising original endoscopic images; Detecting an original endoscopic image in a corresponding video frame by a first thread, and transmitting the detection result of the original endoscopic image to an integration module; forming a control command according to the detection result of the original endoscopic image by an integration module; and adjusting an output result in the second thread by a second thread in response to the control command, so that the output result is adapted to a usage environment of the endoscope video stream; The first thread and the second thread are parallel threads.

[0008] An embodiment of the present application further provides an endoscopic image processing method, An endoscopic image processing device obtains an endoscopic video stream transmitted by an endoscope, the endoscopic video stream including an original endoscopic image, and is used to inspect a corresponding lesion of a target object; The endoscope image processing device detects an original endoscope image in a corresponding video frame through a first thread, and transmits the detection result of the original endoscope image to an integration module; forming a control command in accordance with the detection result of the original endoscope image by an integration module in the endoscope image processing device; The endoscopic image processing device adjusts an output result of the second thread by a second thread in response to the control command, so that the output result is adapted to a usage environment of the endoscopic video stream, and the first thread and the second thread are parallel threads; The method includes a step in which the endoscopic image processing device sends a corresponding target endoscopic image frame to a medical device in contact with the target object according to the adjusted output result of the second thread, and the medical device outputs the corresponding target endoscopic image frame.

[0009] An embodiment of the present application further provides an endoscopic image processing device, an information transmission module configured to obtain an endoscopic video stream including an original endoscopic image; an information processing module configured to detect an original endoscopic image in a corresponding video frame by a first thread, and transmit the detection result of the original endoscopic image to the integration module; The information processing module is configured to form a control command according to a detection result of the original endoscopic image by an integration module; The information processing module is configured to adjust an output result of the second thread by a second thread in response to the control command, so that the output result is adapted to a use environment of the endoscopic video stream; The apparatus, wherein the first thread and the second thread are parallel threads.

[0010] In the above scheme, The information processing module is configured to extract endoscopic video frames in pathology information of a target object; The information processing module is configured to perform a resolution enhancement process on the endoscopic video frames; The information processing module is configured to obtain a plurality of endoscopic video frames by converting a format of the endoscopic video frames from a current encoding format to a grayscale value encoding format; The information processing module is configured to encode and compress the endoscopic video frames to form an endoscopic video stream in a video stream state.

[0011] In the above scheme, The information processing module is configured to encode and compress each of the endoscopic video frames in an encoding manner conforming to a target format; the information processing module is configured to generate data packets conforming to a format of the endoscopic video frames by writing a serial number corresponding to the video frames and generating a time stamp corresponding to the image data packets; The information processing module is configured to perform a splicing process on the image data packets to form a plurality of endoscopic video frames.

[0012] In the above scheme, The information processing module is configured to scale the original endoscopic image to a target size to generate a standard endoscopic image frame; The information processing module is configured to filter the standard endoscopic image frames to eliminate interfering standard endoscopic image frames to form target endoscopic image frames including different target endoscopic image frames.

[0013] In the above scheme, The information processing module is configured to detect an existing foreign object by a detector in the first thread, and obtain a foreign object box distributed in the target endoscopic image frame, the foreign object box being for indicating an area in the target endoscopic image frame where a foreign object exists; The information processing module is configured to locate a lesion region and a lesion category in the target endoscopic image frame by filtering the target endoscopic image frame based on the foreign object box; The information processing module is configured to transmit a lesion region in the target endoscopic image frame located by the foreign object box to an integration module.

[0014] In the above scheme, The information processing module is configured to stop detecting a current target endoscopic image frame when a detection time of the detector in the first thread exceeds a detection time threshold; The information processing module is configured to acquire a subsequent target endoscopic image frame for detection.

[0015] In the above scheme, The information processing module is configured to: when the tracker in the second thread is in a tracking continuation state, a tracking box corresponding to the tracker is aligned with a foreign object box corresponding to the detector, and the tracker in the second thread outputs a current target endoscopic image frame; The information processing module is configured such that when the tracker of the second thread is in a reset state, a tracking box corresponding to the tracker is not aligned with a foreign object box corresponding to the detector, and the tracker in the second thread outputs a new target endoscopic image frame.

[0016] In the above scheme, The information processing module is configured to determine, when a foreign object box is determined to exist in the target endoscopic image frame by the detector in the first thread, corresponding box regression parameters; The information processing module is configured to activate a tracker in the second thread according to the box regression parameters.

[0017] In the above scheme, The information processing module is configured to assign a target identifier to a foreign object in the current target endoscopic image frame when the tracker in the second thread is activated; The information processing module is configured to realize continuous tracking of the foreign object in the current target endoscopic image frame by initializing the target identifier when the foreign object in the target endoscopic image frame is the same when the tracker in the second thread is in an active state, a tracking continuation state, and a reset state.

[0018] In the above scheme, The information processing module is configured to send an adjustment command in response to an output result in the second thread to obtain new pathology information by adjusting a detection state of a medical device in contact with the target object.

[0019] An embodiment of the present application further provides an endoscopic image processing system, an endoscope configured to transmit an endoscopic video stream to an endoscopic image processor; an endoscopic image processing device configured to acquire an endoscopic video stream transmitted by the endoscope; the endoscopy video stream includes an original endoscopy image and is used to inspect a corresponding lesion of a target object; The endoscopic image processing device is configured to detect an original endoscopic image in a corresponding video frame through a first thread, and transmit the detection result of the original endoscopic image to an integration module; The endoscopic image processing device is configured to form a control command according to a detection result of the original endoscopic image by an integration module; the endoscopic image processing device is configured to adjust an output result in the second thread by a second thread in response to the control command, so that the output result is adapted to a usage environment of the endoscopic video stream, the first thread and the second thread being parallel threads; The endoscopic image processing device is configured to send a corresponding target endoscopic image frame to a medical device in contact with the target object according to the adjusted output result of the second thread; The medical device is configured to output a corresponding target endoscopic image frame.

[0020] An embodiment of the present application further provides an electronic device, a memory configured to store executable instructions; and a processor configured, when executing executable instructions stored in the memory, to implement the endoscopic image processing method described above.

[0021] An embodiment of the present application further provides a computer-readable storage medium having executable instructions stored thereon, the executable instructions, when executed by a processor, causing the computer-readable storage medium to implement the endoscopic image processing method described above. Effect of the Invention

[0022] The embodiments of the present application have the following beneficial effects: By obtaining an endoscopic video stream including an original endoscopic image, detecting the original endoscopic image in a corresponding video frame by a first thread, transmitting the detection result of the original endoscopic image to an integration module, forming a control command by the integration module according to the detection result of the original endoscopic image, and adjusting the output result in the second thread in response to the control command, the present application controls the output result by the parallel first thread and second thread to adapt to the usage environment of the endoscopic video stream, thereby improving the real-time accuracy of endoscopic image processing and enhancing the real-time reproducibility.

[0023] In order to more clearly explain the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces drawings necessary for describing the embodiments or related technologies. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without making any inventive efforts. [Brief description of the drawings]

[0024] [Figure 1] 1 is a schematic diagram of an environment in which an endoscopic image processing method provided in an embodiment of the present application is used. [Diagram 2] 1 is a schematic diagram of an electronic device provided in an embodiment of the present application. [Diagram 3] 1 is a schematic flow diagram of one preferred embodiment of an endoscopic image processing method provided in an embodiment of the present application. [Figure 4A] 1 is a schematic flow diagram of one preferred embodiment of an endoscopic image processing method provided in an embodiment of the present application. [Figure 4B] 1 is a schematic flow diagram of one preferred embodiment of an endoscopic image processing method provided in an embodiment of the present application. [Diagram 5] 1 is a schematic flow diagram of one preferred embodiment of an endoscopic image processing method provided in an embodiment of the present application. [Figure 6] FIG. 2 is a schematic diagram of different threads in an endoscopic image processing method according to an embodiment of the present application. [Figure 7]FIG. 13 is a schematic diagram of an initialization state of a tracker in an endoscopic image processing method according to an embodiment of the present application. [Figure 8] 1 is a schematic diagram of a tracking continuation state in an endoscopic image processing method according to an embodiment of the present application; [Figure 9] 1 is a schematic diagram of a tracking reset / tracking continuation state in an endoscopic image processing method according to an embodiment of the present application. FIG. [Figure 10] 1 is a schematic diagram of a tracking reset / tracking continuation state in an endoscopic image processing method according to an embodiment of the present application. FIG. [Figure 11] 1 is a schematic diagram of a process for determining a same target in an endoscopic image processing method according to an embodiment of the present application. [Figure 12] 1 is a schematic diagram of the display effect of an endoscopic image processing method according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the present application is described in more detail below with reference to the drawings, and the described embodiments are not limitations on the present application, and all other embodiments obtained by those skilled in the art without exerting any inventive effort fall within the scope of protection of the present application.

[0026] In the following description, when "some embodiments" are used, a subset of all possible embodiments is described; however, it should be understood that "some embodiments" may be the same or different subsets of all possible embodiments, and may be combined with each other where not inconsistent.

[0027] Before describing the embodiments of the present application in more detail, the nouns and terms related to the embodiments of the present application will be explained, and the following interpretations will be applied to the nouns and terms related to the embodiments of the present application.

[0028] 1) "Response" is used to express a condition or state upon which an operation to be performed depends, and when the dependent condition or state is satisfied, the operation or operations to be performed may be in real time or may have a delay, and unless otherwise stated, the operations to be performed do not have any execution order restrictions.

[0029] 2) "AD Computer Aided Diagnosis (Chinese name)" is used to assist in the detection of lesions and improve the accuracy of diagnosis by using imaging, medical image processing technology and other possible physiological and biochemical means, in combination with computer analysis and calculation.

[0030] 3) "Endoscopic video stream" refers to video-like pathological information formed by image acquisition of a living body part (different target organs of the human body or lesions of the human body) by an image acquisition device (e.g., an endoscope).

[0031] 4) "Lesion" generally refers to the area in an organism where a disease has occurred. In other words, diseased tissue containing limited pathogenic microorganisms is called a lesion.

[0032] 5) "YUV (Yellow-UV)" is a color coding method in which "Y" stands for luminance (Luma), i.e., grayscale value, and "U" and "V" stand for chrominance (Chroma), whose function is to describe the color and saturation of an image and is used to specify the color of a pixel.

[0033] 6) "RGB" is a three-primary color coding method, also known as RGB color mode, which is an industry color standard that obtains various colors by varying and superimposing the three channels of red (R), green (G), and blue (B). RGB represents the colors of the three channels of red, green, and blue. This standard contains almost all colors that the human eye can recognize, and is one of the most widely used color systems.

[0034] FIG. 1 is a schematic diagram of a usage scenario of the endoscopic image processing method provided in the embodiment of the present application. Referring to FIG. 1, a terminal (including terminal 10-1 and terminal 10-2) is provided with a corresponding client capable of performing different functions, and the client obtains and views different pathological information from a corresponding server 200 via a network 300, the terminal is connected to the server 200 via the network 300, the network 300 may be a wide area network or a local area network, or a combination thereof, and a wireless link is used to realize the transmission of data, and the types of pathological information obtained by the terminal (including terminal 10-1 and terminal 10-2) from the corresponding server 200 via the network 300 may be the same or different, for example, the terminal (including terminal 10-1 and terminal 10-2) may obtain a pathological image or pathological video aligned with a target object from the corresponding server 200 via the network 300, or may obtain and view a pathological video (e.g., an endoscopic video stream) aligned only with a current target from the corresponding server 200 via the network 300. The server 200 may store pathological information corresponding to each of the different target objects, or may store supporting analysis information that matches the pathological information of the target objects. In some embodiments of the present application, the different types of pathological information stored in the server 200 may be an endoscopic video stream captured by an endoscope. The at least two original endoscopic images in the endoscopic video stream in this embodiment are a set of multi-view pathological pictures obtained by repeatedly observing a confusing lesion area by operations such as moving the camera and switching the magnification when a doctor uses an endoscope, and information of a specific view of the endoscope is fused.Since the endoscopic video stream records all the information in the endoscopic field of view when the doctor observes the patient's lesion, the information of a patient's lesion observed by the doctor within the endoscopic field of view is utilized as a continuous video stream, which avoids the doctor ignoring minute lesion areas when the doctor moves the endoscope quickly, thereby providing more information for the doctor to diagnose and find minute lesion areas than a single frame picture.

[0035] Here, the observation of the patient's lesion by the endoscope (the medical device that contacts the target object) may include multiple different application scenarios, such as different video stream screening, such as diabetic retinopathy lesion screening, cervical cancer early screening, etc. The endoscopic image processing method according to this embodiment can be applied to multiple application scenarios, which facilitates remote checking and use by doctors.

[0036] The server 200 transmits pathological information of the same target object to the terminal (terminal 10-1 and / or terminal 10-2) via the network 300, thereby enabling the user of the terminal (terminal 10-1 and / or terminal 10-2) to analyze the pathological information of the target object. As an example, the server 200 installs a neural network model configured to acquire a video stream of an endoscope device 400, the endoscope video stream including an original endoscope image, a first thread detects the original endoscope image in a corresponding video frame, and transmits the detection result of the original endoscope image to an integration module, the integration module forms a control command according to the detection result of the original endoscope image, and in response to the control command, a second thread adjusts an output result in the second thread, thereby realizing that the output result is adapted to a use environment of the endoscope video stream, and the first thread and the second thread are parallel threads.

[0037] The configuration of the electronic device in the embodiment of the present application will be described in detail below, and the electronic device can be implemented in various forms, for example, a dedicated terminal having an endoscopic image processing function, an electronic device having an endoscopic image processing function, or a cloud server, for example, the server 200 in Fig. 1. Fig. 2 is a schematic diagram of the configuration of the electronic device provided in the embodiment of the present application, where Fig. 2 shows only an exemplary configuration of the electronic device, not all the configurations, and some or all of the configurations in Fig. 2 can be implemented as necessary.

[0038] The electronic device provided in the embodiment of the present application includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. Each component in the electronic device is coupled by a bus system 205. It can be understood that the bus system 205 is configured to realize connection communication between these components. The bus system 205 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for clarity of explanation, various buses are all described as the bus system 205 in FIG. 2.

[0039] The user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or touch screen, or the like.

[0040] Here, the memory 202 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The memory 202 in the embodiment of the present application may store data for supporting the operation of the terminal (e.g., 10-1). Examples of these data include any computer programs for operating on the terminal (e.g., 10-1), such as an operating system and application programs. The operating system includes various system programs, such as a framework layer, a kernel library layer, a driver layer, etc., and is configured to realize various basic services and process tasks by hardware. The application programs may include various application programs.

[0041] In some embodiments, the endoscopic image processing device provided in the embodiments of the present application may be realized by a combination of software and hardware, for example, the endoscopic image processing device provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the endoscopic image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0042] As an example in which the endoscopic image processing device provided in the embodiments of the present application is implemented by a combination of software and hardware, the endoscopic image processing device provided in the embodiments of the present application can be directly embodied as a combination of software modules executed by the processor 201. The software modules may be located in a storage medium, and the storage medium is disposed in the memory 202. The processor 201 reads executable instructions included in the software modules in the memory 202, and combines them with necessary hardware (e.g., including the processor 201 and other components connected to the bus 205) to complete the endoscopic image processing method provided in the embodiments of the present application.

[0043] By way of example, the processor 201 may be an integrated circuit chip with signal processing capabilities, such as a general purpose processor, a digital signal processor (DSP), other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., where the general purpose processor may be a microprocessor or any conventional processor, etc.

[0044] As an example of the endoscopic image processing device provided in the embodiments of the present application being implemented in the form of hardware, the endoscopic image processing device provided in the embodiments of the present application can be implemented by directly employing a processor 201 in the form of a hardware decoding processor, for example, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements to execute and realize the endoscopic image processing method provided in the embodiments of the present application.

[0045] The memory 202 in the embodiment of the present application is configured to store various types of data to support the operation of the electronic device, examples of which may include any executable instructions for operating the electronic device, such as a program implementing the endoscopic image processing method of the embodiment of the present application.

[0046] In some other embodiments, the endoscopic image processing device provided in the embodiments of the present application can be realized in the form of software. FIG. 2 shows an endoscopic image processing device 2020 stored in the memory 202, which may be software in the form of a program or plug-in, and may include a series of modules. As an example of a program stored in the memory 202, the endoscopic image processing device 2020 may include: an information transmission module 2081 configured to obtain an endoscopic video stream including an original endoscopic image; and an information processing module 2082 configured to detect, by a first thread, an original endoscopic image in a corresponding video frame, and transmit the detection result of the original endoscopic image to the integration module.

[0047] The information processing module 2082 is configured to form a control command according to the detection result of the original endoscopic image by an integration module.

[0048] The information processing module 2082 is configured to respond to the control command by adjusting the output result in the second thread using a second thread so that the output result is adapted to the usage environment of the endoscopic video stream, and the first thread and the second thread are parallel threads.

[0049] The endoscopic image processing method provided in the embodiment of the present application will be described with reference to the electronic device shown in Fig. 2, and Fig. 3 is a preferred flow diagram of the endoscopic image processing method provided in the embodiment of the present application, and it can be understood that the steps in Fig. 3 can be executed by various servers that operate an endoscopic image processing device, such as a dedicated terminal, a server, a cloud server, or a server cluster having an endoscopic image processing function. The steps shown in Fig. 3 will be described below.

[0050] In step 301, an endoscopic imaging device acquires an endoscopic video stream.

[0051] The endoscopic video stream includes original endoscopic images, which are video streams captured by an endoscope in a medical environment, such as an actual use environment in a hospital. Under the movement and shooting of the endoscope, the endoscopic video stream displays the endoscopic video captured by the lens of the endoscope. Therefore, a number of consecutive frames of original endoscopic images are obtained from the endoscopic video stream, and the obtained original endoscopic images of each frame all represent the endoscopic video captured by the endoscope at a certain time point, so that recognition of the endoscopic image is realized based on the original endoscopic images of each frame.

[0052] In the realized endoscopic image recognition, an endoscope captures an endoscopic video stream inside an organism such as a human body. Exemplarily, the endoscope captures an endoscopic video stream in a lumen communicating with the outside or a sealed body cavity. For example, a lumen communicating with the outside may be a digestive tract, an airway, a urinary tract, etc., and a sealed body cavity is a cavity into which an endoscope needs to be inserted through a notch, such as a thoracic cavity, an abdominal cavity, a joint cavity, etc. By capturing and recognizing the endoscopic video stream by the endoscope, the organ situation in the corresponding lumen is known. In the process of inspecting a lumen with the endoscope, the obtained endoscopic video stream is accessed for the endoscopic image recognition being performed. In addition, the realized endoscopic image recognition may be performed on a previously obtained endoscopic video stream, such as a historical endoscopic video stream, and the realized endoscopic image recognition is not limited to real-time recognition, but may be performed on a large number of stored endoscopic video streams.

[0053] In some embodiments of the present application, acquisition of an endoscopic video stream may be accomplished as follows. Extract an endoscopic video frame in the pathology information of the target object, perform resolution enhancement processing on the endoscopic video frame, convert the format of the endoscopic video frame from a current encoding format to a grayscale value encoding format, obtain a plurality of endoscopic video frames, and encode and compress the endoscopic video frame to form an endoscopic video stream in a video stream state. Convert the format of the endoscopic video frame from a current encoding format to a grayscale value encoding format, obtain a plurality of endoscopic video frames, and encode and compress all the obtained endoscopic video frames to form the pathology information in a video state, thereby reducing the amount of data transmission and reducing freezing, while improving motion blur and improving image quality, further improving the user experience, and allowing the user (doctor) of the remote terminal to more accurately judge the pathology information of the target object.

[0054] In some embodiments of the present application, converting the format of the endoscopic video frames from a current encoding format to a grayscale value encoding format may be achieved as follows. In the encoding method for matching the target format, the endoscopic video frames are encoded and compressed, serial numbers corresponding to the video frames are written, and a time stamp corresponding to the image data packets is generated to generate data packets that match the format of the endoscopic video frames, and a splicing process is performed on the image data packets to form a plurality of endoscopic video frames. When encoding and compressing each image frame to form a video stream, pre-installed video encoding software such as ffmpeg encoding software may be adopted, which is free software that can perform recording, conversion, and streaming functions of multiple audio and video formats, and includes audio and video decoding / encoding libraries. Specifically, the video stream may be formed by encoding and compressing the image frames using a video compression algorithm in the video encoding software, such as a video compression algorithm such as H264 or X264, where H264 and X264 are digital video compression formats and video codec standards. In this process, YUV format refers to a pixel format that indicates luminance parameters and chrominance parameters respectively. The advantage of separating luminance parameters and chrominance parameters is that they avoid mutual interference, and the sampling rate of chrominance is reduced, which does not greatly affect the quality of the image. YUV does not require simultaneous transmission of three independent video signals like RGB, so converting and transmitting an image frame into YUV format can occupy very little bandwidth and save a lot of resources. YUV format is divided into planar and packed. In the case of planar YUV format, the luminance of every pixel point is stored continuously, then the color of every pixel point is stored, and then the saturation of every pixel point is stored. In the case of packed YUV format, the luminance and saturation of each pixel point are stored continuously and alternately.Image frames in YUV format can be stored in storage formats such as YUV422P, YUV420, and YUV420sp, thereby saving storage resources of the terminal device and improving the transmission efficiency of pathology information.

[0055] In addition, in the process of transmitting the endoscopic video stream to the terminal device according to the preset transmission protocol, the transmission protocol may be a real-time data transmission protocol such as UDP protocol, where UDP protocol (User Data Packet Protocol) is a connectionless protocol for processing data packets, and the type of data transmission protocol is not specifically limited in this application. In some preferred embodiments of this application, the video stream may be transmitted to the terminal device by UDP protocol.

[0056] In some embodiments of the present application, the method comprises: The method further includes generating a standard endoscopic image frame by adjusting the original endoscopic image to a target size, and filtering the standard endoscopic image frame to remove interfering standard endoscopic image frames to form a target endoscopic image frame including a different target endoscopic image frame.

[0057] In step 302, the endoscope image processing device detects an original endoscope image in a corresponding video frame through a first thread, and transmits the detection result of the original endoscope image to the integration module.

[0058] In step 303, the endoscopic image processing device uses an integration module to form a control command according to the detection result of the original endoscopic image.

[0059] In step 304, the endoscopic image processing device adjusts the output result in the second thread by the second thread in response to the control command.

[0060] The first thread and the second thread are parallel threads.

[0061] This makes it possible to adapt the output result to the usage environment of the endoscopic video stream.

[0062] Continuing to refer to the electronic device shown in Fig. 2, the endoscopic image processing method provided in the embodiment of the present application will be described, and Fig. 4A is a preferred flow diagram of the endoscopic image processing method provided in the embodiment of the present application, where it can be understood that the steps shown in Fig. 4A can be executed by various servers that operate the endoscopic image processing device, and can be, for example, a dedicated terminal, a server, a cloud server, or a server cluster having an endoscopic image processing function. The steps shown in Fig. 4A will be described below.

[0063] In step 401, the endoscopic image processing device detects an existing foreign object through a detector in the first thread, and obtains a foreign object box distributed in the target endoscopic image frame.

[0064] The foreign object box is used to indicate an area in the target endoscopic image frame where a foreign object is present.

[0065] In step 402, the endoscopic image processing device locates a lesion region in the target endoscopic image frame and the lesion category to which it belongs by filtering the target endoscopic image frame based on the foreign object box.

[0066] In step 403, the endoscopic image processing device transmits the lesion region in the target endoscopic image frame located by the foreign object box to a merging module.

[0067] In some embodiments of the present application, an endoscopic image processing method includes: The method further includes a step of stopping detection of a current target endoscopic image frame and acquiring a subsequent target endoscopic image frame for detection when the detection time of the detector in the first thread exceeds a detection time threshold.

[0068] In some embodiments of the present application, the second thread responding to the control command to adjust the output result in the second thread may be realized as follows. When the tracker in the second thread is in a tracking continuation state, the tracking box corresponding to the tracker is aligned with the foreign object box corresponding to the detector, and the tracker in the second thread outputs a current target endoscopic image frame.

[0069] In some embodiments of the present application, the second thread responding to the control command to adjust the output result in the second thread may be realized as follows. When the tracker of the second thread is in a reset state and the tracking box corresponding to the tracker is not aligned with the foreign object box corresponding to the detector, the tracker in the second thread outputs a new target endoscopic image frame.

[0070] In some embodiments of the present application, an endoscopic image processing method includes: The method further includes, when a detector in the first thread determines that a foreign object box exists in the target endoscopic image frame, determining corresponding box regression parameters and activating a tracker in the second thread with the box regression parameters.

[0071] In some embodiments of the present application, an endoscopic image processing method includes: The method further includes a step of assigning a target identifier to a foreign object in the current target endoscopic image frame when the tracker in the second thread is activated, and initializing the target identifier when the foreign object in the target endoscopic image frame is the same when the tracker in the second thread is in an active state, a tracking continuation state and a reset state, thereby realizing continuous tracking of the foreign object in the current target endoscopic image frame.

[0072] In some embodiments of the present application, an endoscopic image processing method includes: The method further includes a step of transmitting an adjustment command to obtain new pathological information by adjusting a detection state of a medical device in contact with the target object in response to an output result in the second thread.

[0073] 2, the endoscopic image processing method provided in the embodiment of the present application will be described. In the endoscopic image processing system, reference will be made to Fig. 4B, which is a preferred flow diagram of the endoscopic image processing method provided in the embodiment of the present application, where it can be understood that the steps in Fig. 4B can be executed by various servers that operate the endoscopic image processing device, and can be, for example, a dedicated terminal, a server, a cloud server, or a server cluster having an endoscopic image processing function. The steps shown in Fig. 4B will be described below.

[0074] In step 4001, the endoscope transmits an endoscopic video stream to an endoscopic image processor. In step 4002, an endoscopic image processing device acquires an endoscopic video stream transmitted by an endoscope.

[0075] The endoscopic video stream includes original endoscopic images that are used to inspect the corresponding lesions of the target object.

[0076] In step 4003, the endoscopic image processing device detects an original endoscopic image in a corresponding video frame through a first thread, and transmits the detection result of the original endoscopic image to the integration module.

[0077] In step 4004, the endoscopic image processing device uses an integration module to form a control command according to the detection result of the original endoscopic image.

[0078] In step 4005, the endoscopic image processing device adjusts the output result in the second thread by the second thread in response to the control command.

[0079] This makes it possible to realize that the output result is adapted to the usage environment of the endoscopic video stream, and the first thread and the second thread are parallel threads.

[0080] In step 4006, the endoscopic image processing device sends a corresponding target endoscopic image frame to a medical device in contact with the target object according to the adjusted output result in the second thread.

[0081] In step 4007, the medical device outputs the corresponding target endoscopic image frame.

[0082] The technical solution of this embodiment realizes that the output result is controlled to suit the usage environment of the endoscopic video stream through the parallel first thread and the second thread, thereby improving the real-time accuracy of processing the endoscopic image, improving the real-time repeatability, and outputting the corresponding target endoscopic image frame through the medical device, so that the inspection status of the target object can be known in real time.

[0083] Hereinafter, the endoscopic image processing method provided in the present application will be described using as an example determining polyps (foreign bodies) in colon images using an endoscopic video stream. Various images of medical videos formed by an endoscopic device are continuously generated, for example, by an endoscope continuously taking pictures inside the digestive tract, which results in a large amount of data. Therefore, it is necessary to realize large-scale classification and recognition by performing classification prediction.

[0084] Therefore, artificial intelligence technology (AI, Artificial Intelligence) provides a scheme to support the above applications by training an appropriate support analysis information formation network. Artificial intelligence is a theory, method, technology, and application system that uses digital computers or machines based on digital computers to simulate, extend, and expand human intelligence, recognize the environment, obtain knowledge, and use knowledge to obtain optimal results. Artificial intelligence studies the design principles and realization methods of various intelligent machines, endowing the machines with the functions of sensing, reasoning, and decision-making, and in the AI ​​medical field, realizes low-latency video and message recognition by digital computers or machines based on digital computers, and realizes convenient acquisition of support analysis information according to pathology analysis results.

[0085] However, in the related art, the scheme of the endoscopic video stream (e.g., colonoscopy video stream) target detection solution generally integrates various target detection algorithms such as one target detection model with high accuracy, such as the retina detection algorithm (Retina Net), the region convolution neural network algorithm (Faster RCNN), and the one-glance algorithm (YOLO You Only Look Once), all of which sequentially input the image frames in the video stream into the model to detect the target. However, the processing speed of the model is limited and does not match the speed of the real-time video stream (most detection models cannot keep up with the frame rate of real-time video), which affects the output effect of the model.

[0086] In the traditional technology, the process of processing the original endoscopic images contained in the endoscopic video stream to determine polyps (foreign bodies) using a neural network model includes the following: 1) The neural network model directly processes image frames in the real-time video stream. However, the frame rate of the real-time video stream is generally 25 fps or higher, and the frame interval is smaller than 40 ms, so the processing speed of the model cannot keep up with the real-time video stream speed. 2) Extract and detect some frames from only the video stream (for example, extract every N frames), and do not process the video frames that are not processed immediately (frame skip processing). 3) In addition to traditional detection, a tracking algorithm (such as various target tracking algorithms, such as KCF, CSRDCF, SiamFC, SiamRPN, etc.) is added, for example, the step processing method finds the initial frame by the detection model, and in the subsequent steps, the target is located instead of detection by tracking, this scheme guarantees real-time to a certain extent, but the accuracy of polyp detection is reduced; or the model ensemble method regards the tracking algorithm as another detection model, and executes the detection model and tracking algorithm for every frame, this scheme guarantees the accuracy of polyp detection, but the real-time performance is reduced, so doctors cannot use it continuously.

[0087] Referring to the above summary of the related art, in the process of processing colonoscopy video streams by traditional neural network models, the deficiencies mainly include: 1) The detection model takes a long time to process and cannot keep up with the video frame rate, resulting in detection stagnation and distorted results. When the speed of the detection model cannot keep up with the real-time video stream frame rate, the usual phenomenon is that the screen detection box stagnates and remains in place, but the target in the video is no longer in place. Under the specific evaluation value indicators, this phenomenon will significantly reduce the real-time precision, real-time recall, and real-time F-value, and from the user's visual experience, this phenomenon will cause a deterioration of the user experience, which will not only seriously affect the diagnosis experience of doctors, but also cause detection to be missed. 2) The detection model is not very stable and robust. In variable and unpredictable scenarios such as video streams, the detection model's display is prone to jitter and instability, and typical trigger scenarios include lens blur, rapidly changing lens, deforming target, partial occlusion of target, target moving in and out of the field of view, etc. 3) Lack of identity recognition of identical targets The detection model does not have the ability to recognize the ID of the same target, and in a target detection scenario, it is desirable to trigger a warning or prompt when a new target is found, but since the detection model has low robustness and does not have the ability to recognize the ID of the same target, the warning / prompt is frequently and repeatedly triggered, which is inconvenient for the operator / user.

[0088] In order to solve the above-mentioned deficiencies, please refer to FIG. 5, which is a preferred flow diagram of an endoscopic image processing method provided in an embodiment of the present application, in which a user is a user who operates a colonoscope, and specifically includes the following steps: In step 501, an endoscopic video stream containing original endoscopic images is obtained. In step 502, a detector in a first thread detects an original endoscopic image in a corresponding video frame, and transmits the detection result of the original endoscopic image to a merging module. In step 503, the integration module forms control commands according to the detection results of the original endoscopic images. In step 504, the tracker in the second thread adjusts the output result of the tracker in the second thread in response to the control command, so that the output result is aligned with the delay parameter of the endoscopic video stream, and the first thread and the second thread are parallel threads.

[0089] Continue to refer to FIG. 6, which is a schematic diagram of different threads in an endoscopic image processing method in an embodiment of the present application. 1) In the timeline of the endoscopic video stream, each point represents one frame. If the input frame rate of the real-time video is 25 fps, the point / frame interval represents 40 ms. 2) The Tracker receives all output detection boxes from the Tracker in real time and directly controls the video output. 3) The Detector is an integrated module that maintains and instructs the Tracker's next behavior and indirectly controls the video output. 4) The intersection lines represent the information transmission between components and can note the time points where the information transmission begins and ends.

[0090] The tracker is used to ensure real-time performance and improve robustness. The tracker receives control commands from the integration module and can directly return the detection box to the video stream in real time. In addition, the tracker used in the second thread must meet the low latency requirements. Any low-latency real-time tracker such as manual feature tracking algorithm (CSR-DCF), SiamRPN, etc. can be adopted to realize direct information interaction with the video stream.

[0091] A detector is used to ensure accuracy, specifically, the integrated module can send corresponding control commands to realize the initialization, continuation, reinitialization and termination of the tracker. Here, the detector does not need to meet real-time requirements, and if the previous frame is not completed, it may give up detecting the frame and adopt any detector algorithm such as YOLOv3, RetinaNet, Faster RCNN, etc., and the detector does not directly interact with the video stream.

[0092] An Integration Module is used to collect information from two parallel lines, jointly calling the Detector and Integration Module.

[0093] Continuing to refer to Figures 7 to 10, in the process in which tracking in the second thread operates, the states of the tracker include a wait state (wait), an initialization state (init), a tracking continuation state (cont), a tracking reset state (reinit) and a stop state (stop).

[0094] The wait state indicates that the detector has not found a target and the tracker is in a wait state.

[0095] Refer to FIG. 7, which is a schematic diagram of the initialization state of the tracker in the endoscopic image processing method of an embodiment of the present application. The initialization state (init) indicates that the detector has found the first frame target and its box regression parameters (bbox Bounding-Box regression), and the box regression parameters are used to activate and initialize the tracker.

[0096] Please refer to FIG. 8. FIG. 8 is a schematic diagram of a tracking continuation state in an endoscopic image processing method according to an embodiment of the present application. In the tracking continuation state (cont), if the degree of agreement between the box provided by the tracker and the instruction box provided by the detector during tracking is high (IoU>=μ), the tracker continues tracking since it is still tracking the target well (the target has not been missed).

[0097] Refer to FIG. 9, which is a schematic diagram of the tracking reset-continue state in the endoscopic image processing method of the embodiment of the present application. The tracking reset state (reinit) indicates that the tracker has lost the target or the tracking is not accurate when the box provided by the tracker and the indication box provided by the detector have a low matching degree (IoU < μ, μ is dynamically adjusted according to the use environment of the endoscopic video stream) during tracking, so the tracker is reactivated and initialized using the bbox of the detector.

[0098] Referring to FIG. 10, FIG. 10 is a schematic diagram of the tracking reset-continue state in the endoscopic image processing method of the embodiment of the present application, and the end state (stop) ends the tracker if the detector does not find a valid target for N consecutive frames during tracking, since it indicates that the target is likely not within the field of view.

[0099] Continuing to refer to FIG. 11, FIG. 11 is a schematic diagram of a process of judging the same target in the endoscopic image processing method of the embodiment of the present application. When the endoscopic video stream of a patient is detected by the endoscopic image processing method provided in the present application, 1) Appearance of a new target: A target in the init state is considered a new target that has been discovered for the first time, and is assigned a new ID. 2) Retention of old targets: If targets in the init, cont, and reinit states are the same target, they are assigned an initialized ID. This recognition policy can reduce warnings and prompts for duplicate targets, reducing unnecessary interruptions to the user.

[0100] The endoscopic image processing method provided in this application (Asynchronous in Parallel Detection and Tracking, AIPDT) allows monitoring the effectiveness of the use of the endoscopic image processing method in the process of determining polyps in colon images.

[0101] The video evaluation of the endoscopic video stream needs to take into account the video frame rate. Generally, the real-time frame rate is 25 fps, that is, each frame is 40 ms. If the time taken by the model is more than 40 ms, the result of the previous frame is directly taken as the current result (represented as the screen output stagnation), and the indicators shown in Table 1 are calculated based on the pre-evaluation. [Table 1]

[0102] If the detection algorithm is YOLOv3 and the tracking algorithm is CSR-DCF, (1) See Table 2 for speed index improvement [Table 2]

[0103] When the detector employs RetinaNet and more complex networks, the latency of the detector scheme is generally between 100-200ms. (2) See Table 3 for real-time accuracy improvements [Table 3]

[0104] Refer to FIG. 12, which is a schematic diagram of the display effect of the endoscopic image processing method of the embodiment of the present application. In the conventional technology (Detector scheme), due to high delay, the tracking box corresponding to the tracker and the foreign object box corresponding to the detector are not aligned with each other, and many tracking boxes continue to use the results of the previous frame, leading to a decrease in precision and recall, and the visual effect of the endoscopic image is greatly reduced. In contrast, the tracking box provided in the endoscopic image processing method of the embodiment of the present application (AIPDT real-time asynchronous parallel frame) has a more accurate position and can be aligned with the foreign object box, and its robustness is also better, which contributes to the continuous operation of the endoscope.

[0105] The beneficial technical effects are as follows: An endoscopic video stream including an original endoscopic image is obtained, and a first thread detects the original endoscopic image in a corresponding video frame, and transmits the detection result of the original endoscopic image to an integration module, and the integration module forms a control command according to the detection result of the original endoscopic image, and in response to the control command, a second thread adjusts the output result in the second thread, where the first thread and the second thread are parallel threads, thereby realizing that the output result is adapted to the usage environment of the endoscopic video stream, thereby improving the real-time accuracy of endoscopic image processing, and enhancing the real-time reproducibility.

[0106] The above are examples of the present application, and do not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should fall within the scope of protection of the present application. [Industrial Applicability]

[0107] In an embodiment of the present application, an endoscopic video stream containing an original endoscopic image is obtained, a first thread detects the original endoscopic image in a corresponding video frame, and the detection result of the original endoscopic image is transmitted to an integration module, and the integration module forms a control command according to the detection result of the original endoscopic image, and in response to the control command, a second thread adjusts the output result in the second thread so as to realize that the output result is adapted to the usage environment of the endoscopic video stream, and the first thread and the second thread are parallel threads, and the parallel first thread and second thread control the output result to be adapted to the usage environment of the endoscopic video stream, thereby improving the real-time accuracy of endoscopic image processing and enhancing the real-time reproducibility. [Explanation of symbols]

[0108] 10-1 Terminal 10-2 Terminal 200 Servers 201 Processor 202 Memory 203 User Interface 204 Network Interface 205 Bus System 300 Network 400 Endoscopic device 2020 Endoscopic image processing device 2081 Information Transmission Module 2082 Information Processing Module

Claims

1. An endoscopic image processing method executed by an electronic device, comprising: acquiring an endoscopic video stream including endoscopic images; A first thread is executed for each of a plurality of frames to detect a foreign object in a target endoscopic image frame in a corresponding video frame, and transmits a lesion area based on the foreign object to an integration module, detecting a foreign object present in the target endoscopic image frame by a detector in the first thread, and obtaining a foreign object box for indicating an area where the foreign object exists; locating a lesion region in the target endoscopic image frame by filtering the target endoscopic image frame based on the foreign object box; transmitting a lesion area in the target endoscopic image frame located by the foreign object box to an integration module; a transmitting step including: activating a tracker in a second thread when the lesion area is transmitted to the integration module; determining, by the integration module, whether to continue tracking by the tracker in the second thread, If a degree of agreement between a tracking box corresponding to the tracker in the second thread and the lesion area detected by the detector is equal to or greater than a predetermined threshold, the integration module sets the tracker to a tracking continuation state; or If a degree of agreement between a tracking box corresponding to the tracker in the second thread and the lesion area detected by the detector is lower than the predetermined threshold, the integration module resets the tracker. A step of determining deactivating the tracker in the second thread if the time during which the lesion region is not detected by the detector exceeds a predetermined threshold; Including, The method, wherein the first thread and the second thread are parallel threads.

2. The step of acquiring an endoscopic video stream includes: Extracting endoscopic video frames in pathology information of a target object; performing a resolution enhancement process on the endoscopic video frames; obtaining a plurality of endoscopic video frames by converting a format of the endoscopic video frames from a current encoding format to a grayscale value encoding format; 2. The method of claim 1, further comprising: encoding and compressing the endoscopic video frames to form an endoscopic video stream in a video stream state.

3. The step of converting the format of the endoscopic video frames from a current encoding format to a grayscale value encoding format includes: encoding and compressing each of the endoscopic video frames with an encoding method that matches a target format; generating data packets conforming to the format of the endoscopic video frames by writing serial numbers corresponding to the video frames and generating time stamps corresponding to the image data packets; 3. The method of claim 2, further comprising performing a splicing process on the image data packets to form a plurality of endoscopic video frames.

4. An endoscopic image processing method executed by an endoscopic image processing system, comprising: acquiring an endoscopic video stream including endoscopic images; A first thread is executed for each of a plurality of frames to detect a foreign object in a target endoscopic image frame in a corresponding video frame, and transmits a lesion area based on the foreign object to an integration module, detecting a foreign object present in the target endoscopic image frame by a detector in the first thread, and obtaining a foreign object box for indicating an area where the foreign object exists; locating a lesion region in the target endoscopic image frame by filtering the target endoscopic image frame based on the foreign object box; transmitting a lesion area in the target endoscopic image frame located by the foreign object box to an integration module; a transmitting step including: activating a tracker in a second thread when the lesion area is transmitted to the integration module; determining, by the integration module, whether to continue tracking by the tracker in the second thread, If a degree of agreement between a tracking box corresponding to the tracker in the second thread and the lesion area detected by the detector is equal to or greater than a predetermined threshold, the integration module sets the tracker to a tracking continuation state; or If a degree of agreement between a tracking box corresponding to the tracker in the second thread and the lesion area detected by the detector is lower than the predetermined threshold, the integration module resets the tracker. A step of determining deactivating the tracker in the second thread if the time during which the lesion region is not detected by the detector exceeds a predetermined threshold; Including, The method, wherein the first thread and the second thread are parallel threads.

5. The endoscopic image processing method is performed by an endoscopic image processing system, the system comprising: an endoscope configured to transmit an endoscopic video stream to an endoscopic image processor; an endoscopic image processing device configured to acquire an endoscopic video stream transmitted by the endoscope; the endoscopy video stream includes an endoscopy image and is used to inspect a corresponding lesion of a target object; The endoscope image processing device is configured to detect a foreign object in a target endoscope image frame in a corresponding video frame by a first thread executed for each plurality of frames, and transmit a lesion area based on the foreign object to an integration module, the transmitting step including: Detecting a foreign object present in the target endoscopic image frame by a detector in the first thread and obtaining a foreign object box for indicating an area where the foreign object exists; locating a lesion region in the target endoscopic image frame by filtering the target endoscopic image frame based on the foreign object box; transmitting a lesion area in the target endoscopic image frame located by the foreign object box to an integration module; Including, The endoscopic image processing device is configured to activate a tracker in a second thread when the lesion region is transmitted to the integration module; The integration module is configured to determine whether to continue tracking by the tracker in the second thread, the determining including: When a degree of agreement between a tracking box corresponding to the tracker in the second thread and the lesion area detected by the detector is equal to or greater than a predetermined threshold, the integration module sets the tracker to a tracking continuation state; or If a degree of agreement between a tracking box corresponding to the tracker in the second thread and the lesion area detected by the detector is lower than the predetermined threshold, the integration module resets the tracker; The endoscopic image processing device is configured to deactivate the tracker in the second thread when a time during which the lesion region is not detected by the detector exceeds a predetermined threshold; the first thread and the second thread are parallel threads. The endoscopic image processing method according to any one of claims 1 to 4.

6. a memory configured to store executable instructions; An electronic device comprising: a processor configured to, when executing executable instructions stored in the memory, realize an endoscopic image processing method described in any one of claims 1 to 3, or an endoscopic image processing method described in claim 4 or 5.

7. A computer program for implementing the endoscopic image processing method according to any one of claims 1 to 3, or the endoscopic image processing method according to claim 4 or 5.

Images