An image enhancement method, device, electronic equipment and endoscope system

Abstract

The application provides an image enhancement method and device, electronic equipment and an endoscope system. The method comprises the following steps: using a pre-trained network model to extract semantic information of an image to be enhanced; determining a processing strategy for performing enhancement processing on the image to be enhanced according to the semantic information; and performing enhancement processing on the image to be enhanced according to the processing strategy. The method can optimize the image enhancement effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image enhancement method, apparatus, electronic device and endoscope system. Background Technology

[0002] In endoscopic video imaging systems, image enhancement techniques are typically used to post-process images to improve image quality. For example, image sharpness and contrast enhancement can improve the clarity of intra-abdominal tissues (blood vessels, organ edges, etc.); saturation correction can be applied to achieve better subjective quality; and enhancement processing can be performed on images with non-uniform lighting.

[0003] In the post-processing image enhancement of endoscopic video images, overprocessing issues can easily occur. For example, during normal surgery, enhancing the saturation of the image can provide a better subjective visual experience. However, if a major hemorrhage occurs in the field of vision, the saturation of the red blood will overflow due to the enhanced saturation. Another example is that water vapor may form during surgery due to cutting operations, affecting the doctor's visual experience. Usually, the image is dehazed in post-processing. Since fascia and fog are very similar, both being large areas of white, global dehazing can easily "remove" the fascia as well.

[0004] Traditional image enhancement techniques mainly involve manually designing features to adjust the enhancement intensity for different regions. For example, the chromaticity value of the current pixel is used to adjust the saturation of the red region to avoid red overflow; the variance of the local region is used to adaptively adjust the contrast enhancement intensity; and blood vessels are detected through color information to enhance the display of blood vessels.

[0005] Practice has shown that in traditional image enhancement techniques, adaptive image enhancement methods based on manually designed features rely on researchers' deep understanding of medical images and rich image processing experience, which has a high threshold. Moreover, the extracted features are usually general features, resulting in poor robustness and unsatisfactory performance in some extreme scenarios. Summary of the Invention

[0006] In view of this, this application provides an image enhancement method, apparatus, electronic device, and endoscope system.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] According to a first aspect of the embodiments of this application, an image enhancement method is provided, comprising:

[0009] Using a pre-trained network model, semantic information of the image to be enhanced is extracted; the semantic information includes one or more of target information, scene classification information, and event information.

[0010] A processing strategy for enhancing the image to be enhanced is determined based on the semantic information; wherein, when the semantic information is switched from the first semantic information to the second semantic information, the processing strategy is adjusted from the first enhancement processing strategy to the second enhancement processing strategy.

[0011] The image to be enhanced is processed according to the aforementioned processing strategy.

[0012] According to a second aspect of the embodiments of this application, an image enhancement apparatus is provided, comprising:

[0013] The extraction unit is used to extract semantic information of the image to be enhanced using a pre-trained network model; the semantic information includes one or more of target information, scene classification information, and event information.

[0014] The determining unit is configured to determine a processing strategy for enhancing the image to be enhanced based on the semantic information; wherein, when the semantic information is switched from first semantic information to second semantic information, the processing strategy is adjusted from the first enhancement processing strategy to the second enhancement processing strategy.

[0015] The processing unit is used to perform enhancement processing on the image to be enhanced according to the processing strategy.

[0016] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0017] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the method of the first aspect.

[0018] According to a fifth aspect of the embodiments of this application, an endoscope system is provided, comprising: an endoscope device, a light source device, a camera system host, a display device, and a storage device; wherein:

[0019] The endoscope device is used to photograph the observation area of ​​the subject and generate image data;

[0020] The light source device is used to provide illumination light emitted from the front end of the endoscope device;

[0021] The camera system host is used to process the image data generated by the endoscope device according to the method provided in the first aspect, and to control the overall operation of the endoscope system.

[0022] The display device is used to display an image corresponding to the image data processed by the camera system host;

[0023] The storage device is used to store images corresponding to the image data processed by the camera system host.

[0024] The image enhancement method of this application embodiment utilizes a pre-trained network model to extract semantic information of the image to be enhanced, determines a processing strategy for enhancing the image based on the semantic information, and performs enhancement processing on the image based on the processing strategy. By determining the processing strategy for enhancing the image based on the semantic information of the image, the image enhancement effect is optimized. Attached Figure Description

[0025] Figure 1 This is a schematic flowchart illustrating an image enhancement method according to an exemplary embodiment of this application;

[0026] Figure 2 This is a schematic diagram illustrating an endoscopic video image enhancement process according to an exemplary embodiment of this application;

[0027] Figure 3 This is a schematic diagram of an FCN algorithm flow illustrated in an exemplary embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the algorithm flow of Faster R-CNN shown in an exemplary embodiment of this application;

[0029] Figure 5 This is a schematic diagram illustrating a scene classification algorithm based on deep learning, as shown in an exemplary embodiment of this application.

[0030] Figure 6 This is a schematic diagram illustrating an event detection algorithm based on deep learning, as shown in an exemplary embodiment of this application.

[0031] Figure 7 This is a schematic diagram illustrating a neural network parameter set pre-training process according to an exemplary embodiment of this application;

[0032] Figure 8A and Figure 8B This is a schematic diagram of a mask obtained by extracting the fascia position in the laparoscopic surgical field of view, as shown in an exemplary embodiment of this application.

[0033] Figure 9 This is a schematic diagram illustrating an HSV color space according to an exemplary embodiment of this application;

[0034] Figure 10 This is a schematic diagram of the structure of an image enhancement device shown in an exemplary embodiment of this application;

[0035] Figure 11 This is a schematic diagram of the hardware structure of an electronic device illustrated in an exemplary embodiment of this application;

[0036] Figure 12 This is a schematic diagram of the structure of an endoscope system shown in an exemplary embodiment of this application;

[0037] Figure 13A This is a schematic diagram of the general structure of an endoscope system shown in an exemplary embodiment of this application;

[0038] Figure 13B This is a functional structure diagram of an endoscope system shown in an exemplary embodiment of this application. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0041] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0042] Please see Figure 1 This is a flowchart illustrating an image enhancement method provided in an embodiment of this application, as shown below. Figure 1 As shown, the image enhancement method may include the following steps:

[0043] Step S100: Extract semantic information of the image to be enhanced using a pre-trained network model.

[0044] Step S110: Determine the processing strategy for enhancing the image to be enhanced based on semantic information.

[0045] Step S120: Perform enhancement processing on the image to be enhanced according to the processing strategy.

[0046] In this embodiment of the application, for images that need to be enhanced (referred to as images to be enhanced herein), such as endoscopic video images, semantic information of images to be enhanced can be extracted using deep learning methods.

[0047] In one example, a pre-trained network model can be used to extract semantic information from the image to be enhanced. The specific implementation will be explained with examples below.

[0048] For example, semantic information may include, but is not limited to, one or more of target information, scene classification information, and event information.

[0049] Take endoscopic video images as an example.

[0050] Target information can be described by organs and tissues (organs, organ margins, blood vessels, fascia, bile ducts, lesions, etc.) and non-tissue organs (surgical instruments, gauze, clips, catheters, suture needles, sutures, etc.), and the information described includes attributes, location, and area.

[0051] Scene classification information can describe the global features of images and videos, such as the shooting location (e.g., nasal cavity, throat) and imaging effects (whether there is bleeding, whether there is fogging, etc.).

[0052] Event information can be described by human actions (including but not limited to separation, resection, suturing, flushing, placement of drainage tubes, etc.) and passively triggered events (including but not limited to bleeding, fogging, etc.). The description information includes event category, event location, etc.

[0053] For example, different types of semantic information are extracted using different network models, while semantic information of the same type is extracted using one or more network models.

[0054] To optimize image enhancement, a processing strategy for enhancing the image can be determined based on the extracted semantic information. This allows for different strategies to be applied to different regions when the semantic information of the image meets specific conditions.

[0055] For example, different semantic information may correspond to different processing strategies for enhancing the image to be enhanced.

[0056] For example, when the semantic information to be enhanced is switched from one type of speech information (referred to as the first speech information in this document) to another type of semantic information (referred to as the second semantic information in this document), the processing strategy is also switched from the current processing strategy (referred to as the first enhancement processing strategy in this document) to another processing strategy (referred to as the second enhancement processing strategy in this document).

[0057] For example, when saturation enhancement of endoscopic video images is detected, different methods can be used to enhance the saturation of the bleeding area and the non-bleeding area. For example, the saturation enhancement operation can be weakened or not performed on the bleeding area, while the saturation enhancement operation can be performed on the non-bleeding area.

[0058] exist Figure 1 In the method flow shown, the semantic information of the image to be enhanced is extracted by using deep learning methods, and the processing strategy for enhancing the image to be enhanced is determined based on the semantic information. Then, the image to be enhanced is enhanced according to the determined processing strategy, which can optimize the image enhancement effect.

[0059] As one possible embodiment, the above processing strategy may include a global processing strategy, that is, uniform enhancement processing of the entire frame of the image to be enhanced, or / and a partitioned processing strategy, that is, different enhancement processing is performed on different regions of the image to be enhanced.

[0060] For example, if the image to be enhanced contains moiré patterns and bleed, then when enhancing the image, we can first perform global demoiré processing, and then perform zonal processing based on the bleed areas.

[0061] In one example, when the processing strategy includes a partitioning processing strategy, step S110, determining the processing strategy for enhancing the image to be enhanced based on semantic information, may include:

[0062] The image to be enhanced is divided into regions based on semantic information to obtain at least two partitions;

[0063] Determine the processing strategy for enhancing at least two partitions, with different processing strategies for different partitions.

[0064] For example, when the semantic information of the image to be enhanced is extracted, the image to be enhanced can be divided into regions based on the extracted semantic information to obtain at least two partitions.

[0065] For example, when semantic information includes event location, locations where a specified event (such as bleeding) exists and locations where a specified event does not exist can be divided into different partitions.

[0066] For example, when semantic information includes target location, the locations where a specified target (such as fascia) exists and the locations where a specified target does not exist can be divided into different partitions.

[0067] For the above at least two partitions, a processing strategy for enhancing the processing of the at least two partitions can be determined respectively.

[0068] For example, different partitions may have different processing strategies.

[0069] In one example, the above at least two partitions may include a first partition and a second partition, wherein the first partition is the region where the specified event or the specified target exists, and the second partition is the region where the specified event or the specified target does not exist.

[0070] For example, when the semantic information includes event information, the area where a specified event, such as bleeding, exists can be divided into one partition (referred to as the first partition in this document) based on the event location information in the event information, and the area where no specified event exists can be divided into another partition (referred to as the second partition in this document).

[0071] Alternatively, when the semantic information includes target information, the region where the specified target, such as fascia, exists can be divided into the first partition (i.e., the first zone) based on the target location in the target information, and the region where the specified target does not exist can be divided into another partition (i.e., the second zone).

[0072] In another example, the above at least two partitions may include a first partition, a third partition, and a fourth partition, wherein the first partition is the area where the specified event or the specified target exists, the third partition is the area within a specified range outside the edge of the area where the specified event or the specified target exists, and the fourth partition is the remaining area.

[0073] For example, in order to optimize the image display effect, for the junction of the region where a specified event or target exists and the region where no specified event or target exists, different processing strategies can be used for the regions where the specified event or target exists and the regions where no specified event or target exists.

[0074] When dividing the image to be enhanced into regions based on semantic information, the region containing a specified event or target can be divided into a partition (referred to as the first partition), the region outside the specified range of the region containing the specified event or target can be divided into a partition (referred to as the third partition in this paper), and the remaining regions can be divided into a partition (referred to as the fourth partition in this paper).

[0075] For example, different pixel values ​​can be set for pixel positions corresponding to different partitions in the mask by generating a mask of the image to be enhanced, and different enhancement processing strategies can be set for different pixel values ​​in the mask.

[0076] As one possible embodiment, the above-mentioned region division of the image to be enhanced based on semantic information to obtain at least two partitions may include:

[0077] A mask corresponding to the image to be enhanced is generated based on semantic information. The mask includes at least two partitions with different pixel values ​​in different partitions.

[0078] In one example, the mask includes a first partition and a second partition. The first partition is the area where a specified event or a specified target exists, and the second partition is the area where no specified event or specified target exists. The pixel values ​​of the first partition are first values, and the pixel values ​​of the second partition are second values.

[0079] For example, the pixel value of the mask corresponding to the pixel position where a specified event exists (which can be called the pixel value of the first partition) can be set to 0, and the pixel value of the mask corresponding to the pixel position where no specified event exists (which can be called the pixel value of the second partition) can be set to 1.

[0080] In another example, the mask includes a first partition, a third partition, and a fourth partition. The first partition is the area where a specified event or a specified target exists. The third partition is the area within a specified range outside the edge of the area where the specified event or the specified target exists. The fourth partition is the remaining area. The pixel values ​​of the first partition are first values, the pixel values ​​of the third partition are third values, and the pixel values ​​of the fourth partition are fourth values.

[0081] For example, the pixel value of the mask corresponding to the pixel position where a specified event exists (which can be called the pixel value of the first partition) can be set to 0, the pixel value of the mask corresponding to the region outside the edge of the region where the specified event or target exists (which can be called the pixel value of the third partition) can be set to 0.5, and the pixel value corresponding to the other positions (which can be called the pixel value of the fourth partition) can be set to 1.

[0082] In one example, when the above-mentioned at least two partitions include a first partition and a second partition, the above-mentioned processing strategies for determining the enhancement processing of at least two partitions may include:

[0083] For any pixel location in the image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the enhancement processing strategy is determined to be no enhancement processing or weak enhancement processing; if the pixel value of the mask corresponding to that pixel location is the second value, the enhancement processing strategy is determined to be enhancement processing.

[0084] For example, to avoid overprocessing, for locations where a specified target or event exists (i.e., the first partition), no enhancement processing or weak enhancement processing may be performed; for locations where no specified target or event exists (i.e., the second partition), normal enhancement processing may be performed.

[0085] In another example, when the aforementioned at least two partitions include a first partition, a third partition, and a fourth partition, the aforementioned processing strategies for determining the enhancement processing of at least two partitions may include:

[0086] For any pixel location in the image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the enhancement processing strategy is determined to be no enhancement processing; if the pixel value of the mask corresponding to that pixel location is the third value, the enhancement processing strategy is determined to be weak enhancement processing; if the pixel value of the mask corresponding to that pixel location is the fourth value, the enhancement processing strategy is determined to be enhancement processing.

[0087] For example, to avoid overprocessing and to optimize the connection between regions with specified targets or events and regions without specified targets or events, no enhancement processing may be performed on the location with specified targets or events (i.e., the first partition); weak enhancement processing may be performed on the region within a specified range outside the edge of the region with specified events or targets (i.e., the third partition); and normal enhancement processing may be performed on the remaining regions.

[0088] The following explanations will use saturation enhancement and contrast enhancement as examples.

[0089] As one possible embodiment, when the enhancement process includes saturation enhancement processing, step S120, performing enhancement processing on the image to be enhanced according to the processing strategy, may include:

[0090] The image to be enhanced is converted from an RGB image to an HSV image to obtain the HSV image to be enhanced;

[0091] Based on the pixel values ​​at each pixel position in the mask, the S-components at each pixel position in the HSV image to be enhanced are enhanced to obtain the enhanced HSV image.

[0092] The enhanced HSV image is converted to an RGB image to obtain the enhanced RGB image.

[0093] For example, the implementation of converting an RGB image to an HSV image can be described below with specific examples, and will not be repeated here in the embodiments of this application.

[0094] For example, the image to be enhanced can be converted from an RGB (Red, Green, Blue) image to an HSV (Hue, Saturation, Value) image, and the S component of each pixel position in the image to be enhanced can be enhanced according to the pixel value of each pixel position in the mask, so as to optimize the image display effect while avoiding overprocessing.

[0095] In one example, when the above at least two partitions include the first partition and the second partition,

[0096] The above-mentioned enhancement processing of the S-component at each pixel position in the HSV image to be enhanced, based on the pixel values ​​at each pixel position in the mask, may include:

[0097] For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to that pixel location is the first value, the S component of that pixel location is preserved.

[0098] When the pixel value of the mask corresponding to the pixel position is the second value, the S component of the pixel position is enhanced using the first enhancement coefficient, which is greater than 1.

[0099] For example, for regions with a specified target or event, the S-component at the corresponding pixel position may not be enhanced. For regions without a specified target or event, the S-component at the corresponding pixel position may be enhanced normally. This is to optimize the image display effect while avoiding overprocessing. The specific implementation can be explained in the following text with specific examples.

[0100] In another instance, when at least two of the aforementioned partitions include the first partition, the third partition, and the fourth partition,

[0101] The above-mentioned enhancement processing of the S-component at each pixel position in the HSV image to be enhanced, based on the pixel values ​​at each pixel position in the mask, may include:

[0102] For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to that pixel location is the first value, the S component of that pixel location is preserved.

[0103] When the pixel value of the mask corresponding to the pixel position is the third value, the second enhancement coefficient is used to enhance the S component of the pixel position. The second enhancement coefficient is greater than 1.

[0104] When the pixel value of the mask corresponding to the pixel position is the fourth value, the third enhancement coefficient is used to enhance the S component of the pixel position. The third enhancement coefficient is greater than the second enhancement coefficient.

[0105] For example, for regions where a specified target or event exists, the S-component at the corresponding pixel location may not be enhanced. For regions outside the specified range outside the edge of the region where a specified event or target exists, the S-component at the corresponding pixel location may be weakly enhanced. For other regions, the S-component at the corresponding pixel location may be enhanced normally.

[0106] For example, the enhancement factor for weak enhancement (referred to as the second enhancement factor in this document) is smaller than the enhancement factor for normal enhancement (referred to as the third enhancement factor in this document).

[0107] For example, considering that even if a pixel location in an image to be enhanced does not have a specified target or event, but the hue of that pixel location meets certain conditions, enhancing that pixel location may still result in overprocessing.

[0108] For example, taking the anti-red overflow treatment for bleeding situations as an example, for areas that are red but do not have bleeding, saturation enhancement treatment may not be performed to avoid red distortion.

[0109] As one possible embodiment, the above-described enhancement processing of the S-component at each pixel location in the HSV image to be enhanced based on the pixel values ​​at each pixel location in the mask may include:

[0110] Based on the pixel values ​​at each pixel location in the mask and the H component values ​​at each pixel location in the HSV image to be enhanced, the S component at each pixel location in the HSV image to be enhanced is enhanced.

[0111] For example, the value of the H component that meets specific conditions can be preset. When performing enhancement processing, in addition to the pixel value of the mask, the processing strategy of enhancement processing can also be determined based on the value of the H component.

[0112] In one example, when at least two partitions include the first partition and the second partition,

[0113] The above-mentioned enhancement processing of the S-component at each pixel location in the HSV image to be enhanced, based on the pixel values ​​at each pixel location in the mask and the H-component values ​​at each pixel location in the HSV image to be enhanced, may include:

[0114] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to the pixel location is the first value, or if the value of the H component of the pixel location is within a preset range, the S component of the pixel location is preserved; otherwise, the S component of the pixel location is enhanced using the fourth enhancement coefficient, which is greater than 1.

[0115] In another example, when at least two of the aforementioned partitions include the first partition, the third partition, and the fourth partition,

[0116] The above-mentioned enhancement processing of the S-component at each pixel location in the HSV image to be enhanced, based on the pixel values ​​at each pixel location in the mask and the H-component values ​​at each pixel location in the HSV image to be enhanced, may include:

[0117] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to the pixel location is a first value, or if the value of the H component of the pixel location is within a preset range, the S component of the pixel location is maintained.

[0118] When the pixel value of the mask corresponding to the pixel position is the third value, and the H component of the pixel position is not within the preset range, the fifth enhancement coefficient is used to enhance the S component of the pixel position, and the fifth enhancement coefficient is greater than 1.

[0119] When the pixel value of the mask corresponding to the pixel position is the fourth value, and the H component of the pixel position is not within the preset range, the sixth enhancement coefficient is used to enhance the S component of the pixel position. The sixth enhancement coefficient is greater than the fifth enhancement coefficient.

[0120] For example, the range of H components that do not require enhancement processing can be preset (referred to as the preset range in this document). For pixel positions where the H component is within the preset range, no enhancement processing is required. The specific implementation can be explained in the following text with specific examples.

[0121] As one possible embodiment, when the enhancement process includes contrast enhancement processing, step S120, performing enhancement processing on the image to be enhanced according to the processing strategy, may include:

[0122] Identify the high-frequency and low-frequency components of the image to be enhanced;

[0123] Based on the pixel values ​​at each pixel position in the mask, the high-frequency components of the image to be enhanced are processed.

[0124] For example, the contrast of the image to be enhanced can be improved by enhancing the high-frequency components of the image.

[0125] To avoid overprocessing, the areas that need contrast enhancement can be determined based on the pixel values ​​at each pixel position in the mask, and the high-frequency parts of those areas can be enhanced for contrast.

[0126] In one example, determining the high-frequency and low-frequency components of the image to be enhanced may include:

[0127] Low-pass filtering is applied to the image to be enhanced to obtain the low-frequency components of the image.

[0128] Based on the image to be enhanced and its low-frequency components, the high-frequency components of the image to be enhanced are determined.

[0129] For example, the low-frequency part of the image to be enhanced can be obtained by performing a low-pass filter on the image to be enhanced, and the high-frequency part of the image to be enhanced can be obtained by subtracting the low-frequency part of the image to be enhanced from the original image.

[0130] In one example, when the above at least two partitions include the first partition and the second partition,

[0131] The above-mentioned enhancement processing of the high-frequency components of the image to be enhanced based on the pixel values ​​at each pixel position in the mask can include:

[0132] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the pixel value of that pixel location is maintained.

[0133] When the pixel value of the mask corresponding to the pixel position is the second value, the high-frequency part of the pixel position is enhanced according to the first gain, where the first gain is greater than 1.

[0134] In another example, when at least two of the aforementioned partitions include the first partition, the third partition, and the fourth partition,

[0135] The above-mentioned enhancement processing of the high-frequency components of the image to be enhanced based on the pixel values ​​at each pixel position in the mask can include:

[0136] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the pixel value of that pixel location is maintained.

[0137] When the pixel value of the mask corresponding to the pixel position is the third value, the high-frequency part of the pixel position is enhanced according to the second gain, and the second gain is greater than 1;

[0138] When the pixel value of the mask corresponding to the pixel position is the fourth value, the high-frequency part of the pixel position is enhanced according to the third gain, and the third gain is greater than the second gain.

[0139] For example, the gain (first gain, second gain, or third gain) used to enhance the high-frequency portion of the pixel location is a constant, or determined based on the local mean square error.

[0140] For example, for any pixel location, the local mean square error corresponding to that pixel location is determined based on the pixel values ​​within a window with a preset width and height centered on that pixel location. The specific implementation can be explained below with specific examples.

[0141] In one example, when the processing strategy includes a global processing strategy, step S120, performing enhancement processing on the image to be enhanced according to the processing strategy, may include:

[0142] When the semantic information is scene classification information, and the scene classification information includes the image to be enhanced as an ear, nose, or throat scene, the entire frame image is enhanced to remove red distortion.

[0143] When the semantic information is scene classification information, and the scene classification information includes the presence of moiré patterns in the image to be enhanced, the entire frame image is enhanced by removing the moiré patterns.

[0144] For example, when the processing strategy is a global processing strategy, the entire frame of the image (the image to be enhanced) is uniformly enhanced.

[0145] For example, if the extracted semantic information determines that the image to be enhanced is an ear, nose, or throat scene, then the frame-level red overflow prevention function is enabled to remove red distortion from the entire frame image and prevent red distortion.

[0146] This application proposes using deep learning technology to extract semantic information from endoscopic videos, such as the location of blood vessels, fascia, whether a major hemorrhage has occurred, and the location of the hemorrhage. During post-processing of the acquired endoscopic images, this semantic information is used to perform frame-by-frame or region-by-region processing. For example: ① On a frame-by-frame basis, based on the hemorrhage information, targeted enhancement processing is performed only on frames showing hemorrhage; ② Enhancement processing is performed on key tissues (e.g., blood vessels) to avoid affecting other areas; ③ During global processing, inappropriate post-processing of key organs and tissues (e.g., fascia) is reduced or avoided to maintain the authenticity of key parts of the image.

[0147] When it is determined from the extracted semantic information that moiré patterns exist in the image to be enhanced, the entire frame of the image can be enhanced to remove the moiré patterns and optimize the image display effect.

[0148] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below in conjunction with specific application scenarios.

[0149] Take endoscopic video image enhancement processing as an example.

[0150] For the acquired endoscopic video images, the information extraction module can extract semantic information from the images. The target detection model, image semantic segmentation model, scene classification model, and event detection model used in the information extraction process are generated by the pre-training module. The post-processing control module transforms the semantic information into a control mask of the same size as the image. Based on the control mask, the post-processing module performs image processing on each pixel to different degrees and in different ways. A schematic diagram can be seen as follows: Figure 2 As shown.

[0151] For example, semantic information may include:

[0152] (1) Target information: Target information describes organs and tissues (organs, organ edges, blood vessels, fascia, bile ducts, lesions, etc.) and non-tissue organs (surgical instruments, gauze, clips, catheters, suture needles, sutures, etc.). The information described may include, but is not limited to, attributes, location and area.

[0153] (2) Scene classification information: Scene classification information mainly describes the global features of images and videos (shooting location, imaging effect), including presence / absence and category information.

[0154] (3) Event information: Event information describes human behavior (including but not limited to separation, resection, suturing, flushing, placement of drainage tubes, etc.) and passively triggered events (including but not limited to bleeding, fogging, etc.). The description information may include but is not limited to event category, event location, etc.

[0155] The functions of each module are explained in detail below.

[0156] I. Information Extraction Module

[0157] The information extraction module may include, but is not limited to, one or more modules from semantic segmentation, object detection, scene classification, and event detection.

[0158] 1.1 Semantic Segmentation

[0159] Deep learning-based semantic segmentation techniques can include, but are not limited to, FCN, SegNet, UNet, and the DeepLab series.

[0160] Taking FCN as an example, the algorithm flow can be found in [link to FCN algorithm]. Figure 3 .

[0161] like Figure 3 As shown, firstly, a series of convolution and pooling operations are performed on a single frame image to obtain the feature representation of the image; secondly, upsampling / deconvolution and other operations are used to obtain a feature vector of the same size as the input image; finally, logistic regression is performed on the feature vector to obtain the classification result corresponding to each pixel in the original image.

[0162] 1.2 Target Detection

[0163] Deep learning-based object detection techniques may include, but are not limited to, YOLO, SSD, Fast R-CNN, and Faster R-CNN.

[0164] For example, the algorithm flowchart for Faster R-CNN can be found here. Figure 4 .

[0165] like Figure 4As shown, firstly, a series of convolution and pooling operations are performed on a single frame image to obtain the image's feature information; secondly, the image's feature information is input into RPN (Region Proposal Networks) to extract candidate regions containing the target to be detected; finally, the extracted candidate regions are pooled using Roi-pooling, and the features after pooling are used for further target classification and target location regression to obtain the category and location information of the target to be detected.

[0166] For example, this scheme can be used to locate and classify organs, surgical instruments, etc. A separate network can be trained for each target, or a single network can be trained to locate and classify all targets simultaneously.

[0167] 1.3 Scene Classification

[0168] The process of the deep learning-based scene classification algorithm can be found in [link / reference]. Figure 5 .

[0169] like Figure 5 As shown, firstly, a series of convolution and pooling operations are performed on a single frame image to obtain the feature information of the image; then, the feature information of the image is input into a fully connected layer to obtain the probability of each category, and the category with the highest probability corresponds to the category of the current image.

[0170] 1.4 Event Detection

[0171] First, multiple frames of video images are input into a neural network. The neural network extracts features from the video frame images to generate multi-dimensional behavioral feature vectors. Then, the feature vectors are input into a fully connected layer for logical classification to obtain the probability of each category. The event category with the highest probability corresponds to the event information of the current video segment.

[0172] For example, neural networks may include, but are not limited to, deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory (LSTM) neural networks, etc.

[0173] For example, the process of a deep learning-based event detection algorithm can be found in [reference needed]. Figure 6 .

[0174] like Figure 6 As shown, firstly, five frames of a video are taken as one input, and 3D convolution is used for feature encoding. Then, LSTM is used to process the frame features after feature encoding. Here, h(x) is the hidden state of LSTM, S(t) is the output result vector, S(t-5) is the process vector, and the output is S(t).

[0175] Then, the feature vector S(t) is classified logically, and the categories that are greater than a preset threshold are output. The sigmoid function is then used to output the final category of the feature vector S(t).

[0176] For example, the sigmoid function can be used to output the category of the feature vector S(t). The sigmoid function is defined by the following formula:

[0177]

[0178] For example, x is one dimension of the multidimensional feature vector S(t) generated after feature extraction from each event through a neural network, and each output category f(x) also corresponds to one dimension of the feature vector. f(x) is compared with a preset threshold (the criterion for distinguishing between bleeding / no bleeding and fogging / no fogging). If it is greater than the preset threshold, it is determined to belong to either bleeding or fogging, thus obtaining the final event detection result.

[0179] II. Pre-training Module

[0180] For example, for any of the above-mentioned specific tasks (such as object detection, scene classification, and event detection), the parameter set θ of the corresponding neural network NN needs to be obtained through pre-training. The flowchart can be found in [link to flowchart]. Figure 7 .

[0181] For example, taking the detection network used for object detection as an example, its main learnable part is the convolutional neural network (CN). The CN network structure parameters obtained through pre-training can include not only the weight parameters and offset parameters of the convolutional filters, but also, but not limited to, the number of convolutional filters and the size of the convolutional kernel.

[0182] For example, the training steps may include:

[0183] Step 2.1: Collect a large amount of endoscopic image data, label the position and category of the target objects in the data, and form a training sample with the image and its label (corresponding position and category). All training samples form the training set Ω.

[0184] Step 2.2: Initialize the network parameters of the convolutional neural network CN to θ0, and set appropriate high-level parameters related to training, such as learning rate and gradient descent algorithm.

[0185] Step 2.3: Perform parameter θ on the training set Ω. i The forward computation of the convolutional neural network (CN) is performed to obtain the output F(Y) of the convolutional neural network. A loss function is then designed to obtain the loss value L(θ). i The loss value is calculated from F(Y) and the label information in step 2.1.

[0186] Step 2.4: Adjust θ using the backpropagation algorithm to obtain θ i .

[0187] Step 2.5: Repeat steps 2.3 to 2.4 until the network converges, and output the parameter θ. final .

[0188] III. Post-processing control module

[0189] Post-processing control can be divided into global control and zone control.

[0190] 3.1 Global Control

[0191] Global control means that the entire frame of the image uses a unified control signal for a unified post-processing procedure.

[0192] Example 1

[0193] If the scene classification model in the information extraction module determines that the current image is an ear, nose, or throat scene, then the frame-level red overflow prevention function is enabled to prevent red distortion.

[0194] Example 2

[0195] If the scene classification model in the information extraction module determines that there are moiré patterns in the current image, it will output a global demoiré signal.

[0196] 3.2 Zone Control

[0197] Partition control means using different post-processing strategies for different regions of the entire image frame.

[0198] Example 3

[0199] During laparoscopic surgery, if the semantic segmentation model in the information extraction module detects that there is massive bleeding and the location of the bleeding in the current image, the partition anti-red overflow function will be activated to prevent red distortion when the bleeding location is saturated.

[0200] The post-processing control module sets the pixels corresponding to the bleed position on the mask to 0 (i.e., the first value mentioned above, no saturation enhancement operation or reduced saturation enhancement operation), and sets the rest to 1 (i.e., the second value mentioned above, saturation enhancement operation is performed), thus generating a saturation control mask.

[0201] Alternatively, the post-processing control module sets the pixels at the fascia location to 0 (i.e., the first value mentioned above, without performing saturation enhancement operation), sets the pixels at the junction of the bleeding area edge and the background (non-bleeding area) within a certain radius to 0.5 (i.e., the third value mentioned above, reducing saturation enhancement operation), and sets the remaining parts to 1 (i.e., the fourth value mentioned above, performing saturation enhancement), thus generating a saturation control mask.

[0202] Example 4

[0203] The semantic segmentation model in the information extraction module extracted the fascia location in the laparoscopic surgical field. A schematic diagram of the extracted mask can be found in [reference needed]. Figure 8A and Figure 8B .

[0204] The post-processing control module sets the pixels corresponding to the fascia position on the mask to 0 (i.e., the first value mentioned above, no dehazing operation is performed), and sets the rest to 1 (i.e., the second value mentioned above, dehazing is performed), thus generating a dehazing control mask.

[0205] Alternatively, the post-processing control module sets the pixels at the fascia position to 0 (i.e., the first value mentioned above, no dehazing operation), sets the pixels at the junction of the fascia edge and the background to 0.5 within a certain radius (i.e., the third value mentioned above, a weak dehazing operation), and sets the rest to 1 (i.e., the fourth value mentioned above, dehazing operation), thus generating a dehazing control mask.

[0206] IV. Post-processing module

[0207] The post-processing module performs corresponding image post-processing based on the global signal or control mask output by the post-processing control module.

[0208] Example 5

[0209] Regarding the red overflow prevention handling in Section 3.2, one possible implementation is as follows:

[0210] 4.1 Convert the RGB image to the HSV color space. The HSV representation of a color image consists of three parts: Hue, Saturation, and Value.

[0211] For example, it can be used Figure 9 The cylinder shown represents the HSV color space. The cross-section of the cylinder can be regarded as a polar coordinate system. H is represented by the polar angle, S by the polar axis length, and V by the height of the cylinder's central axis.

[0212] The specific conversion formula from RGB to HSV is shown below:

[0213]

[0214]

[0215] C max =max(R, G, B)

[0216] C min =min(R, G, B)

[0217] Δ=C max -Cmin

[0218]

[0219]

[0220] V = C max

[0221] 4.2. Use the saturation control mask CMask corresponding to each pixel to filter out the bleed area, and then reduce the weighted S component of the corresponding pixel position to achieve the effect of red suppression.

[0222] For example, depending on whether CMask adjusts the saturation value, you can choose to adjust it or not:

[0223] For any pixel position, if the value of CMask is 1, then S = S*(1+C), that is, the S component is enhanced using (1+C) (i.e., the first enhancement coefficient mentioned above); if the value of CMask is 0, then S = S, that is, the S component remains unchanged.

[0224] CMask represents the saturation control mask (0 means no saturation enhancement, 1 means saturation enhancement), S represents the saturation of the current pixel value (0 means pure white, the value range is 0 to 100%, the larger the value, the more saturated the color), C can be a constant to adjust the degree of saturation enhancement, for example, take 1 / 2.

[0225] For example, the saturation control mask CMask and the H component of the pixel position can be used together for post-processing:

[0226] The H component can be measured in angles, ranging from 0 to 360°, and represents color information, i.e., the position of the color in the spectrum. All colors on the color ring are colors in the spectrum. Starting from red and rotating counterclockwise, an H component value of 0 represents red, an H component value of 120 represents green, and an H component value of 240 represents blue.

[0227] Assume that the range of values ​​for the red H component (i.e. the preset range mentioned above) is 0 to 20 degrees and 340 to 360 degrees.

[0228] For any pixel position, if the value of CMask is 0, or the value of the H component is within the above preset range, then the S component of the current pixel position remains unchanged.

[0229] Otherwise, if the value of CMask is not 0 and the value of the H component is not within the above-mentioned preset range, then S = S*(1+C), that is, the S component is enhanced using (1+C) (i.e., the above-mentioned fourth enhancement coefficient).

[0230] For example, the fourth enhancement coefficient and the first enhancement coefficient can be the same or different.

[0231] 4.3 Convert the image from HSV space back to RGB space.

[0232] If 0 ≤ H ≤ 360, 0 ≤ S ≤ 1 and 0 ≤ V ≤ 1, then:

[0233] C = V × S

[0234]

[0235] m=VC

[0236]

[0237] (R, G, B) = ((R′+m)×255, (G′+m)×255, (B′+m)×255)

[0238] Example 6

[0239] For the dehazing process described in Section 3.2, it can be achieved by using Adaptive Contrast Enhancement (ACE) combined with the dehazing control mask CMask output by the post-processing control module.

[0240] The principle of ACE is as follows: an image is divided into two parts: a low-frequency part, which can be obtained through low-pass filtering (smoothing blur); and a high-frequency part, which can be obtained by subtracting the low-frequency part from the original image. The goal of the algorithm is to enhance the high-frequency part, which represents details, by multiplying the high-frequency part by a certain gain value and then reconstructing it to obtain the enhanced image. Therefore, the core of the ACE algorithm is the calculation of the gain coefficient of the high-frequency part.

[0241] In one example, the gain can be set to a fixed value.

[0242] In another example, the gain value can be represented as a quantity related to the variance.

[0243] For example, assuming a pixel in an image is represented as x(i, j), then the local mean and variance of a region centered at (i, j) with a window size of (2n+1)*(2n+1) (i.e., taking a preset width and height of 2n+1 as an example) can be expressed as:

[0244]

[0245]

[0246] For pixel x(i,j), the mean m x(i,j) can be used as the background, x(i,j)-m x (i, j) represents the high-frequency component at that pixel location. Taking the gain product of the high frequencies, we get:

[0247] f(i,j)=m x (i,j)+G(i,j)[x(i,j)-m x (i, j)

[0248] G(i,j) is the gain at that pixel position (i.e., the first gain mentioned above).

[0249] In one example, G(i, j) can take the value of a constant greater than 1.

[0250] In another example, G(i,j) can be a change in value that is inversely proportional to the local mean square error:

[0251]

[0252] By combining control masks, contrast enhancement filtering can be applied to pixel locations where dehazing needs to be enabled:

[0253] For any pixel position, if the value of CMask is 1, then:

[0254] f(i,j)=m x (i,j)+G(i,j)[x(i,j)-m x (i, j)]

[0255] That is, G(i,j) is used to enhance the high-frequency part of the pixel position.

[0256] If the value of CMask is 0, then the pixel value at that pixel position is preserved.

[0257] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0258] Please see Figure 10 This is a schematic diagram of the structure of an image enhancement device provided in an embodiment of this application, as shown below. Figure 10 As shown, the image enhancement device may include:

[0259] Extraction unit 1010 is used to extract semantic information of the image to be enhanced using a pre-trained network model; the semantic information includes one or more of target information, scene classification information, and event information;

[0260] The determining unit 1020 is used to determine a processing strategy for enhancing the image to be enhanced based on the semantic information; wherein, when the semantic information is switched from first semantic information to second semantic information, the processing strategy is adjusted from the first enhancement processing strategy to the second enhancement processing strategy.

[0261] The processing unit 1030 is used to perform enhancement processing on the image to be enhanced according to the processing strategy.

[0262] In some embodiments, different types of semantic information are extracted using different network models, while semantic information of the same type is extracted using one or more network models.

[0263] In some embodiments, the processing strategy includes a global processing strategy and / or a partitioned processing strategy.

[0264] In some embodiments, when the processing strategy includes a partitioning processing strategy, the determining unit 1020 determines a processing strategy for enhancing the image to be enhanced based on the semantic information, including:

[0265] The image to be enhanced is divided into regions based on the semantic information to obtain at least two partitions;

[0266] A processing strategy is determined for enhancing the at least two partitions, with different processing strategies for different partitions.

[0267] In this application, deep learning technology can be used to extract high-level semantic information, which can then be used to generate global control signals or regional control masks. This allows for adaptive control of the post-processing type and intensity for different frames or different regions within the same frame. Adaptively performing post-processing on different frames or different regions within the same frame can enhance the clarity of key parts of the image while reducing or avoiding inappropriate post-processing of critical organs and tissues (such as fascia).

[0268] In some embodiments, the determining unit 1020 divides the image to be enhanced into regions based on the semantic information to obtain at least two partitions, including:

[0269] A mask corresponding to the image to be enhanced is generated based on the semantic information. The mask includes at least two partitions, and the pixel values ​​of the different partitions are different.

[0270] In some embodiments, the at least two partitions include a first partition and a second partition, wherein the first partition is an area where a specified event or a specified target exists, and the second partition is an area where no specified event or specified target exists, wherein the pixel value of the first partition is a first value, and the pixel value of the second partition is a second value;

[0271] or,

[0272] The at least two partitions include a first partition, a third partition, and a fourth partition. The first partition is the area where a specified event or a specified target exists. The third partition is the area within a specified range outside the edge of the area where the specified event or the specified target exists. The fourth partition is the remaining area. The pixel value of the first partition is a first value. The pixel value of the third partition is a third value. The pixel value of the fourth partition is a fourth value.

[0273] In some embodiments, when the at least two partitions include a first partition and a second partition, the determining unit 1020 determines a processing strategy for enhancing the at least two partitions, including:

[0274] For any pixel location in the image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the enhancement processing strategy is determined to be no enhancement processing or weak enhancement processing; if the pixel value of the mask corresponding to that pixel location is the second value, the enhancement processing strategy is determined to be enhancement processing.

[0275] In some embodiments, when the at least two partitions include a first partition, a third partition, and a fourth partition, the determining unit 1020 determines a processing strategy for enhancing the at least two partitions, including:

[0276] For any pixel location in the image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the enhancement processing strategy is determined to be no enhancement processing; if the pixel value of the mask corresponding to that pixel location is the third value, the enhancement processing strategy is determined to be weak enhancement processing; if the pixel value of the mask corresponding to that pixel location is the fourth value, the enhancement processing strategy is determined to be enhancement processing.

[0277] In some embodiments, the enhancement process includes saturation enhancement processing;

[0278] The processing unit 1030 performs enhancement processing on the image to be enhanced according to the processing strategy, including:

[0279] The image to be enhanced is converted from an RGB image to an HSV image to obtain the HSV image to be enhanced;

[0280] Based on the pixel values ​​of each pixel position in the mask, the S-component of each pixel position in the HSV image to be enhanced is enhanced to obtain the enhanced HSV image.

[0281] The enhanced HSV image is converted into an RGB image to obtain the enhanced RGB image.

[0282] In some embodiments, when the at least two partitions include a first partition and a second partition...

[0283] The processing unit 1030 enhances the S-component of each pixel position in the HSV image to be enhanced based on the pixel value of each pixel position in the mask, including:

[0284] For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to that pixel location is the first value, the S component of that pixel location is preserved;

[0285] When the pixel value of the mask corresponding to the pixel position is the second value, the S component of the pixel position is enhanced using a first enhancement coefficient, where the first enhancement coefficient is greater than 1.

[0286] In some embodiments, when the at least two partitions include a first partition, a third partition, and a fourth partition...

[0287] The processing unit 1030 enhances the S-component of each pixel position in the HSV image to be enhanced based on the pixel value of each pixel position in the mask, including:

[0288] For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to that pixel location is the first value, the S component of that pixel location is preserved;

[0289] When the pixel value of the mask corresponding to the pixel position is the third value, the S component of the pixel position is enhanced using a second enhancement coefficient, where the second enhancement coefficient is greater than 1.

[0290] When the pixel value of the mask corresponding to the pixel position is the fourth value, the S component of the pixel position is enhanced using a third enhancement coefficient, which is greater than the second enhancement coefficient.

[0291] In some embodiments, the processing unit 1030 enhances the S-component of each pixel position in the HSV image to be enhanced based on the pixel value of each pixel position in the mask, including:

[0292] Based on the pixel values ​​at each pixel location in the mask and the H component values ​​at each pixel location in the HSV image to be enhanced, the S component at each pixel location in the HSV image to be enhanced is enhanced.

[0293] In some embodiments, when the at least two partitions include a first partition and a second partition...

[0294] The processing unit 1030 enhances the S component of each pixel position in the HSV image to be enhanced based on the pixel value of each pixel position in the mask and the H component value of each pixel position in the HSV image to be enhanced, including:

[0295] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to the pixel location is the first value, or if the value of the H component of the pixel location is within a preset range, the S component of the pixel location is maintained; otherwise, a fourth enhancement coefficient is used to enhance the S component of the pixel location, wherein the fourth enhancement coefficient is greater than 1.

[0296] In some embodiments, when the at least two partitions include a first partition, a third partition, and a fourth partition...

[0297] The processing unit 1030 enhances the S component of each pixel position in the HSV image to be enhanced based on the pixel value of each pixel position in the mask and the H component value of each pixel position in the HSV image to be enhanced, including:

[0298] For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to the pixel location is the first value, or when the value of the H component of the pixel location is within a preset range, the S component of the pixel location is maintained.

[0299] When the pixel value of the mask corresponding to the pixel position is the third value, and the H component of the pixel position is not within the preset range, the S component of the pixel position is enhanced using the fifth enhancement coefficient, where the fifth enhancement coefficient is greater than 1.

[0300] When the pixel value of the mask corresponding to the pixel position is the fourth value, and the H component of the pixel position is not within the preset range, the S component of the pixel position is enhanced by the sixth enhancement coefficient, and the sixth enhancement coefficient is greater than the fifth enhancement coefficient.

[0301] In some embodiments, the enhancement process includes contrast enhancement processing;

[0302] The processing unit 1030 performs enhancement processing on the image to be enhanced according to the processing strategy, including:

[0303] Determine the high-frequency and low-frequency components of the image to be enhanced;

[0304] Based on the pixel values ​​at each pixel position in the mask, the high-frequency components of the image to be enhanced are subjected to enhancement processing.

[0305] In some embodiments, the processing unit 1030 determines the high-frequency and low-frequency components of the image to be enhanced, including:

[0306] The image to be enhanced is subjected to low-pass filtering to obtain the low-frequency component of the image to be enhanced;

[0307] Based on the image to be enhanced and its low-frequency components, the high-frequency components of the image to be enhanced are determined.

[0308] In some embodiments, when the at least two partitions include a first partition and a second partition...

[0309] The processing unit 1030 performs enhancement processing on the high-frequency components of the image to be enhanced based on the pixel values ​​at each pixel position in the mask, including:

[0310] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the pixel value of that pixel location is maintained.

[0311] When the pixel value of the mask corresponding to the pixel position is the second value, the high-frequency part of the pixel position is enhanced according to the first gain, where the first gain is greater than 1.

[0312] In some embodiments, when the at least two partitions include a first partition, a third partition, and a fourth partition...

[0313] The processing unit 1030 performs enhancement processing on the high-frequency components of the image to be enhanced based on the pixel values ​​at each pixel position in the mask, including:

[0314] For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the pixel value of that pixel location is maintained.

[0315] When the pixel value of the mask corresponding to the pixel position is the third value, the high-frequency part of the pixel position is enhanced according to the second gain, where the second gain is greater than 1.

[0316] When the pixel value of the mask corresponding to the pixel position is the fourth value, the high-frequency part of the pixel position is enhanced according to the third gain, where the third gain is greater than the second gain.

[0317] In some embodiments, the gain used to enhance the high-frequency portion of a pixel location is a constant, or determined based on the local mean square error. For any pixel location, the local mean square error corresponding to that pixel location is determined based on the pixel values ​​within a window with a preset width and height centered on that pixel location.

[0318] In some embodiments, when the processing strategy includes a global processing strategy, the processing unit 1030 performs enhancement processing on the image to be enhanced according to the processing strategy, including:

[0319] When the semantic information is scene classification information, and the scene classification information includes the image to be enhanced as an ear, nose, or throat scene, the entire frame image is enhanced to remove red distortion.

[0320] When the semantic information is scene classification information, and the scene classification information includes the presence of moiré patterns in the image to be enhanced, the entire frame image is subjected to enhancement processing to remove moiré patterns.

[0321] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the image enhancement method described above.

[0322] Please see Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 1101 and a memory 1102 storing machine-executable instructions. The processor 1101 and the memory 1102 can communicate via a system bus 1103. Furthermore, by reading and executing the machine-executable instructions corresponding to the image enhancement logic in the memory 1102, the processor 1101 can execute the image enhancement method described above.

[0323] The memory 1102 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0324] In some embodiments, a machine-readable storage medium, such as Figure 11The memory 1102 in the memory medium stores machine-executable instructions that, when executed by a processor, implement the image enhancement method described above. For example, the machine-executable storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0325] This application also provides a computer program stored in a machine-readable storage medium, such as... Figure 11 The memory 1102 in the processor, and when the processor executes the computer program, causes the processor 1101 to execute the image enhancement method described above.

[0326] Please see Figure 12 This is a schematic diagram of the structure of an endoscope system provided in an embodiment of this application, as shown below. Figure 12 As shown, the endoscope system may include: an endoscope device 1210, a light source device 1220, a camera system host 1230, a display device 1240, and a storage device 1250; wherein:

[0327] The endoscope device 1210 is used to photograph the observation area of ​​the subject and generate image data;

[0328] The light source device 1220 is used to provide illumination light emitted from the front end of the endoscope device;

[0329] The camera system host 1230 is used to process the image data generated by the endoscope device;

[0330] For example, the camera system host 1230 can process the image data generated by the endoscope device in the manner described in the above embodiments.

[0331] The display device 1240 is used to display an image corresponding to the image data processed by the camera system host;

[0332] The storage device 1250 is used to store images corresponding to the image data processed by the camera system host.

[0333] For example, the above image enhancement method is applied to an endoscope system.

[0334] For example, the endoscope device 1210 can acquire observation sites (including but not limited to the ear, nose, throat, or abdominal cavity) of an object being examined (such as a person undergoing endoscopic examination) and generate image data. The endoscope device can be illuminated by the light source device 1220 to acquire clearly visible image details.

[0335] For the image data acquired by the endoscope device 1210, the camera system host 1230 can perform enhancement processing on the image data in the manner described in the above method embodiment to optimize the image display effect.

[0336] In one example, the camera system host 1230 can also be used to control the overall operation of the endoscope system, such as controlling the illumination mode of the light source, the image processing mode, the intelligent processing mode, and the video encoding mode.

[0337] The image data processed by the camera system host 1230 can be displayed on the display device 1240 for relevant personnel (such as medical staff) to view; on the other hand, it can also be stored in the storage device 1250, such as after being encoded in a specified video format.

[0338] The structure and function of the endoscope system will be described in detail below.

[0339] Please see Figure 13A and Figure 13B , Figure 13A This is a schematic diagram of the general structure of an endoscope system. Figure 13B This is a schematic diagram of the functional structure of an endoscope system.

[0340] like Figure 13A As shown, the endoscope system may include an endoscope (i.e., the endoscope device 1210 mentioned above), a light source (i.e., the light source device 1220 mentioned above), a camera system host, a display device, and a storage device.

[0341] Operators (such as medical professionals) can insert an endoscope into the patient's body and output the captured images to an external display and storage device. By observing the images displayed on the device, operators can check for bleeding sites, tumors, and abnormalities at the target site, and the images provide real-time images for surgical treatment. Other personnel using the endoscopic system (such as medical professionals) can access the videos stored on the storage device for postoperative review and surgical training.

[0342] A light source provides illumination light emitted from the tip of the endoscope device. The camera system host processes the image data generated by the endoscope in the manner described in the above method embodiments and uniformly controls the overall operation of the endoscope system. The display device displays the image corresponding to the image data from the camera system host. The storage device stores the image corresponding to the image data from the camera system host.

[0343] like Figure 13B As shown, an endoscope may include a camera optical system, an imaging unit, a processing unit, and an operating unit.

[0344] For example, a camera optical system is used to focus light from the area being observed. The camera optical system is constructed using one or more lenses.

[0345] The imaging unit is used to perform photoelectric conversion on the light received by each pixel to generate image data.

[0346] For example, the imaging unit may consist of an image sensor such as CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge-coupled Device).

[0347] The processing unit converts the image data generated by the imaging unit into digital signals and sends the converted digital signals to the camera system host.

[0348] The operation unit receives inputs of instruction signals for switching the endoscope and for switching the illumination light of the light source device, and outputs the instruction signals to the camera system host.

[0349] For example, the operating unit may include, but is not limited to, switches, buttons, and touch panels.

[0350] like Figure 13B As shown, the light source device may include a lighting control unit and a lighting unit.

[0351] For example, the lighting control unit is used to receive an instruction signal from the camera system host and control the lighting unit to provide illumination light to the endoscope.

[0352] For example, the camera system host is used to process image data received from the endoscope and transmit it to a display device and a storage device.

[0353] For example, the display device and storage device can be external devices.

[0354] like Figure 13B As shown, the camera system host may include an image input unit, an image processing unit, an intelligent processing unit, a video encoding unit, a control unit, and an operation unit.

[0355] For example, the image input unit is used to receive digital signals sent by the endoscope and transmit the received digital signals to the image processing unit.

[0356] For example, the image processing unit is used to perform ISP (Image Signal Processing) operations on the signals of the image input unit, including but not limited to brightness transformation, sharpening, demoiring, and scaling.

[0357] The image processed by the image processing unit is transmitted to the intelligent processing unit, the video encoding unit, and / or the display device.

[0358] For example, the intelligent processing unit is used to perform intelligent analysis on the image processed by the image processing unit, including but not limited to scene classification based on deep learning, instrument head detection, gauze detection, moiré pattern classification, and dense fog classification.

[0359] The image processed by the intelligent processing unit is transmitted to the image processing unit and / or the video encoding unit.

[0360] The image processing unit processes the image after it has been processed by the intelligent processing unit in ways including but not limited to brightness transformation, demoiring, frame overlay, and scaling.

[0361] For example, the video encoding unit is used to encode and compress the image processed by the image processing unit or the intelligent processing unit, and transmit it to the storage device, whereby the storage device stores the encoded and compressed image.

[0362] For example, the control unit is used to control various modules of the endoscope system, including but not limited to the illumination mode of the light source, the image processing mode, the intelligent processing mode, and the video encoding mode.

[0363] For example, the operating unit may include, but is not limited to, switches, buttons, and touch panels, for receiving external indication signals and outputting the received indication signals to the control unit.

[0364] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0365] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image enhancement method, characterized in that, include: Using a pre-trained network model, semantic information of the image to be enhanced is extracted; the semantic information includes one or more of target information, scene classification information, and event information. The image to be enhanced includes endoscopic video images; the target information is used to describe organ tissues and non-organ tissues, and the description information includes the attributes, location, and area of ​​organ tissues and non-organ tissues; the scene classification information is used to describe the global features of the endoscopic video images, including the shooting location and imaging effect; the event information is used to describe human behavior and passively triggered events, and the description information includes the event category and event location. Based on the semantic information, the processing strategy for enhancing the image to be enhanced is determined to be a global processing strategy and / or a partitioned processing strategy; wherein, when the semantic information is switched from the first semantic information to the second semantic information, the processing strategy is adjusted from the first enhancement processing strategy to the second enhancement processing strategy. The image to be enhanced is processed according to the aforementioned processing strategy; Wherein, when the processing strategy for enhancing the image to be enhanced is determined to include a global processing strategy based on the semantic information, the step of enhancing the image to be enhanced according to the processing strategy includes: When the semantic information is scene classification information, and the scene classification information includes the image to be enhanced as an ear, nose, or throat scene, the entire frame image is enhanced to remove red distortion. When the semantic information is scene classification information, and the scene classification information includes the presence of moiré patterns in the image to be enhanced, the entire frame image is subjected to enhancement processing to remove moiré patterns. Wherein, when the processing strategy for enhancing the image to be enhanced is determined to include a global processing strategy and a partitioned processing strategy based on the semantic information, the image to be enhanced is subjected to global processing and partitioned processing respectively; The steps of performing global processing and partition processing on the image to be enhanced include: When the semantic information includes scene classification information and event information, and the scene classification information includes the presence of moiré patterns in the image to be enhanced, and the event information includes the location of bleed, the entire frame image is enhanced by removing moiré patterns, and the image to be enhanced is partitioned according to the location of bleed.

2. The method according to claim 1, characterized in that, Different types of semantic information are extracted using different network models, while semantic information of the same type is extracted using one or more network models.

3. The method according to claim 1, characterized in that, When the processing strategy includes a partitioning processing strategy, determining the processing strategy for enhancing the image to be enhanced based on the semantic information includes: The image to be enhanced is divided into regions based on the semantic information to obtain at least two partitions; A processing strategy is determined for enhancing the at least two partitions, with different processing strategies for different partitions.

4. The method according to claim 3, characterized in that, The step of dividing the image to be enhanced into regions based on the semantic information to obtain at least two partitions includes: A mask corresponding to the image to be enhanced is generated based on the semantic information. The mask includes two partitions with different pixel values ​​in different partitions. The two partitions include a first partition and a second partition. The first partition is an area where a specified event or a specified target exists, and the second partition is an area where no specified event or specified target exists. The pixel value of the first partition is a first value, and the pixel value of the second partition is a second value. The process of determining the enhancement strategies for the at least two partitions includes: For any pixel location in the image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the enhancement processing strategy is determined to be no enhancement processing or weak enhancement processing; if the pixel value of the mask corresponding to that pixel location is the second value, the enhancement processing strategy is determined to be enhancement processing.

5. The method according to claim 3, characterized in that, The step of dividing the image to be enhanced into regions based on the semantic information to obtain at least two partitions includes: A mask corresponding to the image to be enhanced is generated based on the semantic information. The mask includes three partitions, and the pixel values ​​of the different partitions are different. The three partitions include a first partition, a third partition, and a fourth partition. The first partition is the area where a specified event or a specified target exists. The third partition is the area within a specified range outside the edge of the area where the specified event or a specified target exists. The fourth partition is the remaining area. The pixel value of the first partition is a first value, the pixel value of the third partition is a third value, and the pixel value of the fourth partition is a fourth value. The process of determining the enhancement strategies for the at least two partitions includes: For any pixel location in the image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the enhancement processing strategy is determined to be no enhancement processing; if the pixel value of the mask corresponding to that pixel location is the third value, the enhancement processing strategy is determined to be weak enhancement processing; if the pixel value of the mask corresponding to that pixel location is the fourth value, the enhancement processing strategy is determined to be enhancement processing.

6. The method according to claim 3, characterized in that, The enhancement process includes saturation enhancement processing; The enhancement process for the image to be enhanced according to the processing strategy includes: The image to be enhanced is converted from an RGB image to an HSV image to obtain the HSV image to be enhanced; Based on the pixel values ​​at each pixel position in the mask, the S-components at each pixel position in the HSV image to be enhanced are enhanced to obtain the enhanced HSV image; wherein, the mask is a mask corresponding to the image to be enhanced generated based on the semantic information; The enhanced HSV image is converted into an RGB image to obtain the enhanced RGB image.

7. The method according to claim 6, characterized in that, When the at least two partitions include a first partition and a second partition. The enhancement process for the S-component of each pixel position in the HSV image to be enhanced, based on the pixel value of each pixel position in the mask, includes: For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to that pixel location is the first value, the S component of that pixel location is preserved; When the pixel value of the mask corresponding to the pixel position is the second value, the S component of the pixel position is enhanced using the first enhancement coefficient, where the first enhancement coefficient is greater than 1. And / or, when the at least two partitions include a first partition, a third partition, and a fourth partition, The step of enhancing the S-component of each pixel position in the HSV image to be enhanced based on the pixel value of each pixel position in the mask includes: For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to that pixel location is the first value, the S component of that pixel location is preserved; When the pixel value of the mask corresponding to the pixel position is the third value, the S component of the pixel position is enhanced by the second enhancement coefficient, and the second enhancement coefficient is greater than 1. When the pixel value of the mask corresponding to the pixel position is the fourth value, the S component of the pixel position is enhanced by the third enhancement coefficient, which is greater than the second enhancement coefficient.

8. The method according to claim 6, characterized in that, The enhancement process for the S-component of each pixel position in the HSV image to be enhanced, based on the pixel value of each pixel position in the mask, includes: Based on the pixel values ​​at each pixel position in the mask and the H component values ​​at each pixel position in the HSV image to be enhanced, the S component at each pixel position in the HSV image to be enhanced is enhanced. Wherein, when the at least two partitions include a first partition and a second partition, The enhancement process for the S component of each pixel position in the HSV image to be enhanced, based on the pixel value of each pixel position in the mask and the H component value of each pixel position in the HSV image to be enhanced, includes: For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to the pixel location is a first value, or if the value of the H component of the pixel location is within a preset range, the S component of the pixel location is maintained; otherwise, a fourth enhancement coefficient is used to enhance the S component of the pixel location, wherein the fourth enhancement coefficient is greater than 1. And / or, when the at least two partitions include a first partition, a third partition, and a fourth partition, The enhancement process for the S component of each pixel position in the HSV image to be enhanced, based on the pixel value of each pixel position in the mask and the H component value of each pixel position in the HSV image to be enhanced, includes: For any pixel location in the HSV image to be enhanced, when the pixel value of the mask corresponding to the pixel location is a first value, or when the value of the H component of the pixel location is within a preset range, the S component of the pixel location is maintained. When the pixel value of the mask corresponding to the pixel position is the third value, and the H component of the pixel position is not within the preset range, the S component of the pixel position is enhanced by the fifth enhancement coefficient, and the fifth enhancement coefficient is greater than 1. When the pixel value of the mask corresponding to the pixel position is the fourth value, and the H component of the pixel position is not within the preset range, the S component of the pixel position is enhanced by the sixth enhancement coefficient, and the sixth enhancement coefficient is greater than the fifth enhancement coefficient.

9. The method according to claim 4 or 5, characterized in that, The enhancement process includes contrast enhancement processing; The enhancement process for the image to be enhanced according to the processing strategy includes: The image to be enhanced is subjected to low-pass filtering to obtain the low-frequency component of the image to be enhanced; Based on the image to be enhanced and its low-frequency components, the high-frequency components of the image to be enhanced are determined. Based on the pixel values ​​at each pixel position in the mask, the high-frequency components of the image to be enhanced are enhanced. Wherein, when the at least two partitions include a first partition and a second partition, The enhancement process for the high-frequency components of the image to be enhanced, based on the pixel values ​​at each pixel position in the mask, includes: For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the pixel value of that pixel location is maintained. When the pixel value of the mask corresponding to the pixel position is the second value, the high-frequency part of the pixel position is enhanced according to the first gain, where the first gain is greater than 1. And / or, when the at least two partitions include a first partition, a third partition, and a fourth partition, The enhancement process for the high-frequency components of the image to be enhanced, based on the pixel values ​​at each pixel position in the mask, includes: For any pixel location in the HSV image to be enhanced, if the pixel value of the mask corresponding to that pixel location is the first value, the pixel value of that pixel location is maintained. When the pixel value of the mask corresponding to the pixel position is the third value, the high-frequency part of the pixel position is enhanced according to the second gain, where the second gain is greater than 1. When the pixel value of the mask corresponding to the pixel position is the fourth value, the high-frequency part of the pixel position is enhanced according to the third gain, where the third gain is greater than the second gain. The HSV image to be enhanced is obtained by converting the image to be enhanced from an RGB image to an HSV image.

10. The method according to claim 9, characterized in that, The gain used to enhance the high-frequency portion of a pixel location is a constant, or determined based on the local mean square error. For any pixel location, the local mean square error corresponding to that pixel location is determined based on the pixel values ​​within a window with a preset width and height centered on that pixel location.

11. An image enhancement device, characterized in that, include: The extraction unit is used to extract semantic information from the image to be enhanced using a pre-trained network model. The semantic information includes one or more of target information, scene classification information, and event information; the image to be enhanced includes endoscopic video images; the target information is used to describe organ tissues and non-organ tissues, and the description information includes the attributes, location, and area of ​​organ tissues and non-organ tissues; the scene classification information is used to describe the global features of the endoscopic video images, including the shooting location and imaging effect; the event information is used to describe human behavior and passively triggered events, and the description information includes the event category and event location. The determining unit is configured to determine, based on the semantic information, the processing strategy for enhancing the image to be enhanced as a global processing strategy and / or a partitioned processing strategy; wherein, when the semantic information is switched from first semantic information to second semantic information, the processing strategy is adjusted from the first enhancement processing strategy to the second enhancement processing strategy. A processing unit is configured to perform enhancement processing on the image to be enhanced according to the processing strategy. Wherein, when the processing strategy for enhancing the image to be enhanced is determined to include a global processing strategy based on the semantic information, the processing unit enhances the image to be enhanced according to the processing strategy, including: When the semantic information is scene classification information, and the scene classification information includes the presence of moiré patterns in the image to be enhanced, the entire frame image is subjected to enhancement processing to remove moiré patterns. The processing unit performs enhancement processing on the image to be enhanced according to the processing strategy, and further includes: When the processing strategy for enhancing the image to be enhanced is determined based on the semantic information to include a global processing strategy and a partitioned processing strategy, the image to be enhanced is subjected to global processing and partitioned processing respectively. The processing unit performs global processing and partition processing on the image to be enhanced, including: When the semantic information includes scene classification information and event information, and the scene classification information includes the presence of moiré patterns in the image to be enhanced, and the event information includes the location of bleed, the entire frame image is enhanced by removing moiré patterns, and the image to be enhanced is partitioned according to the location of bleed.

12. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-10.

13. An endoscope system, characterized in that, include: Endoscopic device, light source device, camera system host, display device, and storage device; wherein: The endoscope device is used to photograph the observation area of ​​the subject and generate image data; The light source device is used to provide illumination light emitted from the front end of the endoscope device; The camera system host is used to process the image data generated by the endoscope device according to the method described in any one of claims 1-10; The display device is used to display an image corresponding to the image data processed by the camera system host; The storage device is used to store images corresponding to the image data processed by the camera system host.

Images