Image processing method, device, and computer device

Abstract

The embodiments of the present application disclose an image processing method, device, and computer storage medium, which belong to the field of computer technology. The method includes: obtaining objects appearing in the region of interest of the target image, determining an image processing mode based on the objects appearing in the region of interest, at least two objects in different objects correspond to different image processing modes, processing the region of interest based on the image processing mode, and displaying the processed region of interest. The embodiments of the present application process the regions of interest where different objects are located through different image processing modes, and display the processed regions of interest. The displayed region of interest can achieve the best display effect desired by the doctor, which is conducive to the doctor's detailed observation of the region of interest, and then accurately judge the patient's condition, or during the operation, the doctor can operate more accurately.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image processing method, device, and computer device. Background Art

[0002] Images are acquired through camera shooting or scanning. In the process of acquiring images, that is, in the process of image formation, different processing can be performed according to user needs, and finally different quality images are displayed. Image processing is widely used in monitoring, video, medical and other fields. Combined with artificial intelligence technology, it can provide users with more intelligent and higher quality camera products.

[0003] For example, in the medical field, image processing is applied in endoscopy scenarios. The development of endoscopy systems has greatly improved the efficiency of doctors in disease diagnosis, surgical treatment and other aspects. In recent years, with the rapid development of image sensors, image signal processing and other related technologies, endoscopy systems have also been greatly improved in image resolution, frame rate, signal-to-noise ratio and other aspects. In order to further improve the work efficiency of doctors, it is necessary to improve the intelligence level of endoscopy systems and innovate more diagnostically meaningful image processing technologies, which has become a new hot spot for endoscopy systems. Summary of the invention

[0004] The embodiment of the present application provides an image processing method, device, and computer device, which can enable the displayed region of interest to achieve the best display effect desired by the doctor. The technical solution is as follows:

[0005] In one aspect, an image processing method is provided, the method comprising:

[0006] Obtaining objects appearing in a region of interest of a target image;

[0007] Determine an image processing mode based on the objects appearing in the region of interest, wherein at least two of the different objects correspond to different image processing modes;

[0008] Processing the region of interest based on the image processing mode;

[0009] The processed region of interest is displayed.

[0010] Optionally, determining an image processing mode based on an object appearing in the region of interest includes:

[0011] Determine the category of the object appearing in the region of interest based on a first mapping relationship, and determine the image processing mode based on a second mapping relationship, wherein the first mapping relationship indicates a correspondence between objects and categories, and the second mapping relationship indicates a correspondence between object categories and image processing modes; or

[0012] The image processing mode is determined based on a third mapping relationship, where the third mapping relationship indicates a correspondence between an object and an image processing mode.

[0013] Optionally, when the object appearing in the region of interest is a lesion region type object, the image processing mode includes an electronic magnification mode, and the electronic magnification processing mode is used to perform magnification processing on the region of interest; or,

[0014] When the object appearing in the region of interest is a microvascular object, the image processing mode includes a multispectral imaging mode, and the multispectral imaging mode is used to perform enhancement processing on different bands in the region of interest; or

[0015] When the object appearing in the region of interest is an intestinal / intestinal object, the image processing mode includes a contrast enhancement mode and a highlight suppression mode, wherein the contrast enhancement mode is used to perform contrast enhancement processing on the region of interest, and the highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest; or,

[0016] When the object appearing in the region of interest is a device-type object, the image processing mode includes a highlight suppression mode; or,

[0017] When the object appearing in the region of interest is a gauze-like object, the image processing mode includes a false color highlight mode, and the false color highlight mode is used to perform false color display processing on the object appearing in the region of interest; or

[0018] When the object appearing in the region of interest is a smoke-like object, the image processing mode includes a defogging processing mode, and the defogging processing mode is used to perform defogging processing on the smoke-like object appearing in the region of interest.

[0019] Optionally, when the object appearing in the region of interest is a smoke-type object, processing the region of interest based on the image processing mode includes:

[0020] Determine the concentration of smoke-like objects present in the region of interest;

[0021] If the concentration of the smoke object exceeds a concentration threshold, increasing the degree of defogging of the region of interest;

[0022] If the concentration of the smoke object is lower than the concentration threshold, the degree of defogging performed on the region of interest is reduced.

[0023] Optionally, acquiring the object appearing in the region of interest of the target image includes:

[0024] Identify objects appearing in the target image and location information of the objects based on a deep learning model;

[0025] The region of interest is determined based on the position information of the object, where the region of interest includes at least one object identified by the deep learning model.

[0026] Optionally, the number of objects appearing in the target image identified based on the deep learning model is multiple;

[0027] The determining the region of interest based on the position information of the object comprises:

[0028] determining a priority for each of the plurality of objects;

[0029] selecting a target object from the plurality of objects based on a priority of each of the plurality of objects;

[0030] The region of interest is determined based on the position information of the target object.

[0031] Optionally, acquiring the object appearing in the region of interest in the target image includes:

[0032] Determine the region of interest based on the specified region location information;

[0033] Objects appearing in the region of interest are identified based on a deep learning model.

[0034] Optionally, the region of interest after display processing includes:

[0035] Displaying the target image in the main screen area of ​​the display interface;

[0036] The processed region of interest is displayed in a secondary screen area of ​​the display interface, and the secondary screen area is superimposed on the main screen area.

[0037] Optionally, the display interface includes a plurality of auxiliary screen areas;

[0038] The multiple auxiliary screen areas are used to display images processed by different image processing modes for the same region of interest, or to display images processed by different regions of interest.

[0039] In another aspect, an image processing device is provided, the device comprising:

[0040] An acquisition module, used for acquiring objects appearing in a region of interest of a target image;

[0041] A determination module, used for determining an image processing mode based on the objects appearing in the region of interest, wherein at least two of the different objects correspond to different image processing modes;

[0042] A processing module, configured to process the region of interest based on the image processing mode;

[0043] The display module is used to display the processed region of interest.

[0044] Optionally, the determining module includes:

[0045] A first determination unit is used to determine the category of the object appearing in the region of interest based on a first mapping relationship, and to determine the image processing mode based on a second mapping relationship, wherein the first mapping relationship indicates the correspondence between the object and the category, and the second mapping relationship indicates the correspondence between the object category and the image processing mode; or, to determine the image processing mode based on a third mapping relationship, wherein the third mapping relationship indicates the correspondence between the object and the image processing mode.

[0046] Optionally, when the object appearing in the region of interest is a lesion region type object, the image processing mode includes an electronic magnification mode, and the electronic magnification processing mode is used to perform magnification processing on the region of interest; or,

[0047] When the object appearing in the region of interest is a microvascular object, the image processing mode includes a multispectral imaging mode, and the multispectral imaging mode is used to perform enhancement processing on different bands in the region of interest; or

[0048] When the object appearing in the region of interest is an intestinal / intestinal object, the image processing mode includes a contrast enhancement mode and a highlight suppression mode, wherein the contrast enhancement mode is used to perform contrast enhancement processing on the region of interest, and the highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest; or,

[0049] When the object appearing in the region of interest is a device-type object, the image processing mode includes a highlight suppression mode; or,

[0050] When the object appearing in the region of interest is a gauze-like object, the image processing mode includes a false color highlight mode, and the false color highlight mode is used to perform false color display processing on the object appearing in the region of interest; or

[0051] When the object appearing in the region of interest is a smoke-like object, the image processing mode includes a defogging processing mode, and the defogging processing mode is used to perform defogging processing on the smoke-like object appearing in the region of interest.

[0052] Optionally, when the object appearing in the region of interest is a smoke-type object, the processing module includes:

[0053] A second determining unit is used to determine the concentration of smoke-like objects appearing in the region of interest;

[0054] A first judgment unit, configured to increase the degree of defogging of the region of interest if the concentration of the smoke object exceeds a concentration threshold;

[0055] If the concentration of the smoke object is lower than the concentration threshold, the degree of defogging performed on the region of interest is reduced.

[0056] Optionally, the determining module further includes:

[0057] A first recognition unit, configured to recognize objects appearing in the target image and location information of the objects based on a deep learning model;

[0058] A third determining unit is used to determine the region of interest based on the position information of the object, where the region of interest includes at least one object recognized by the deep learning model.

[0059] Optionally, the number of objects appearing in the target image identified based on the deep learning model is multiple;

[0060] The third determining unit is used for:

[0061] determining a priority for each of the plurality of objects;

[0062] selecting a target object from the plurality of objects based on a priority of each of the plurality of objects;

[0063] The region of interest is determined based on the position information of the target object.

[0064] Optionally, the determining module further includes:

[0065] A fourth determining unit, configured to determine the region of interest based on the specified region location information;

[0066] The second recognition unit is used to recognize objects appearing in the region of interest based on a deep learning model.

[0067] Optionally, the display module includes:

[0068] A display unit, configured to display the target image in a main screen area of ​​a display interface;

[0069] The display unit is used to display the processed region of interest in the auxiliary screen area of ​​the display interface, and the auxiliary screen area and the main screen area are superimposed together.

[0070] Optionally, the display interface includes a plurality of auxiliary screen areas;

[0071] The multiple auxiliary screen areas are used to display images processed by different image processing modes for the same region of interest, or to display images processed by different regions of interest.

[0072] In a third aspect, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, an image processing method described in the above aspect is implemented.

[0073] According to a fourth aspect, a computer device is provided, the device comprising:

[0074] processor;

[0075] a memory for storing processor-executable instructions;

[0076] Wherein, the processor is configured to execute an image processing method described in the above aspect.

[0077] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the image processing method described in the above aspect.

[0078] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0079] The regions of interest corresponding to different objects are processed through different image processing modes, and the processed regions of interest are displayed. Since there are at least two objects among different objects corresponding to different image processing modes, the regions of interest corresponding to each object are processed in a targeted manner. Therefore, when the target image is an image captured by an endoscope, different image processing can be performed in the above-mentioned manner for different regions of interest acquired in different scenes. In the endoscope scene, the displayed region of interest can achieve the best display effect desired by the doctor, which is conducive to the doctor's detailed observation of the region of interest, and then accurately judge the patient's condition, or during the operation, the doctor can operate more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 It is a schematic diagram of the general structure of an endoscope system provided in an embodiment of the present application.

[0082] Figure 2 It is a detailed structural diagram of an endoscope system provided in an embodiment of the present application.

[0083] Figure 3 It is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application.

[0084] Figure 4 It is a flowchart of an image processing method provided in an embodiment of the present application.

[0085] Figure 5 It is a flowchart of another image processing method provided in an embodiment of the present application.

[0086] Figure 6 It is a flow chart of an image processing method provided in an embodiment of the present application.

[0087] Figure 7 This is a flow chart for determining a region of interest provided in an embodiment of the present application.

[0088] Figure 8 This is another flow chart for determining a region of interest provided in an embodiment of the present application.

[0089] Fig. 9 This is a highlight suppression curve mapping diagram provided by the real-time example of this application.

[0090] Fig.10 This is a method of realizing a false color highlight display provided by an embodiment of the present application.

[0091] Fig.11 It is an anti-red overflow curve mapping diagram provided in an embodiment of the present application.

[0092] Fig.12 A display interface diagram provided in an embodiment of the present application.

[0093] Fig.13 This is a picture-in-picture principle flow chart provided in an embodiment of the present application.

[0094] Fig.14 It is a schematic diagram of a display screen provided in an embodiment of the present application.

[0095] Fig.15 This is another display screen schematic diagram provided in an embodiment of the present application.

[0096] Fig.16 It is a structural schematic diagram of an image processing device provided in an embodiment of the present application.

[0097] Fig.17 It is a structural block diagram of a terminal provided in an embodiment of the present application.

[0098] Fig.18 It is a schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0100] For the convenience of subsequent explanation, the application scenarios of the embodiments of the present application are first introduced and explained here.

[0101] In the medical field, doctors usually need to observe and locate objects appearing in the area of ​​interest. For example, it is necessary to observe human tissues or organs such as the throat, sinuses, and recurrent laryngeal nerves in the diagnosis scene, and to observe and locate various surgical instruments in the surgical scene, such as surgical forceps, electric scalpels, gauze, etc. At this time, it is necessary to collect a target image, which includes the area of ​​interest. Since the target image also includes some other areas, when directly observing the objects appearing in the area of ​​interest through the target image, the features and details of the objects appearing in the area of ​​interest cannot be clearly and accurately observed. If it is necessary to clearly and accurately observe the objects appearing in the area of ​​interest, it is necessary to process the target image in a targeted manner and display the processed target image so that the doctor can clearly and accurately observe the objects appearing in the area of ​​interest, and perform the next step based on the features and details of the objects appearing in the area of ​​interest. This is conducive to the doctor's accurate judgment of the condition, or the doctor reduces the surgical risk when performing surgery.

[0102] The image processing method provided in the embodiment of the present application is applied to the above-mentioned scene of observing the target image collected by the endoscope system. For the convenience of subsequent description, the endoscope system involved in the embodiment of the present application is first introduced as a whole. Figure 1 As shown, Figure 1 Schematic diagram of the structure of an endoscope system provided in an embodiment of the present application. Figure 1 In the present invention, the endoscope system includes an endoscope, a light source, a camera system host, a display device and a storage device.

[0103] Among them, the endoscope is used to insert a long tube into the patient's body, photograph the part of the patient's body that needs to be observed, collect images of the part, and send the collected images to the camera system host. The light source device is used to emit illumination light from the front end of the long tube of the endoscope so that the endoscope can capture clear images. The camera system host is used to receive the image sent by the endoscope, process the image, and then send the processed image to the display device and storage device. The camera system host is also used to uniformly control the entire endoscope system, such as controlling the endoscope to send the collected image to the camera system host. The display device is used to receive the processed image sent by the camera system host, and then display the processed image on the display device. The storage device is used to receive the processed image sent by the camera system host and store the processed image.

[0104] pass Figure 1 In the endoscope system shown, the doctor observes the processed image displayed by the display device to observe whether the object in the area of ​​interest in the image has bleeding sites, tumor sites and abnormal sites. Figure 1 The endoscope system shown can provide real-time images during surgery. In addition, doctors can also obtain images in a storage device and conduct postoperative review and surgical training based on a video composed of multiple images.

[0105] In order to understand the principle of the endoscope system more clearly, the components of the endoscope system are explained here. Figure 2 As shown, Figure 2 It is a detailed structural diagram of an endoscope system provided in an embodiment of the present application.

[0106] exist Figure 2 In the embodiment, the camera system host in the endoscope system includes an image input unit, an image processing unit, an intelligent processing unit, a video encoding unit, a control unit and an operation unit.

[0107] The image input unit receives the image sent by the endoscope and transmits the received image to the image processing unit.

[0108] The image processing unit receives the image sent by the image input unit and processes the received image, that is, performs ISP (image signal processor) operation on the image. The ISP operation includes brightness transformation, sharpening, moiré removal, scaling and other operations on the image. After the image processing unit processes the image, it sends the processed image to the intelligent processing unit, the video encoding unit or the display device. In addition, the image processing unit is also used to receive the image after intelligent analysis by the intelligent processing unit, and perform ISP operation on the image after intelligent analysis again.

[0109] The intelligent processing unit receives the processed image sent by the image processing unit and performs intelligent analysis on the processed image, including scene classification, instrument or instrument head detection, gauze detection, moiré classification, and dense fog classification based on deep learning. After the intelligent processing unit intelligently analyzes the processed image, it sends the intelligently analyzed image to the image processing unit or the video encoding unit.

[0110] The video encoding unit is used to receive the image processed by the image processing unit or the image intelligently analyzed by the intelligent processing unit, encode and compress the processed image or the image intelligently analyzed, and send the compressed image to the storage device.

[0111] The control unit is used to send different functional instructions to each unit of the endoscope system, and is used to control each module of the endoscope system to perform certain functions, such as controlling the illumination of the light source, the image processing method of the image processing unit, the intelligent analysis method of the intelligent processing unit, and the encoding compression method of the video encoding unit. In addition, the control unit is also used to receive a trigger instruction sent by the operating unit, and respond to the trigger instruction to turn on the camera system host. When the user triggers the switch, button or touch panel on the camera system host, the operating unit is used to receive the user's trigger instruction and send the trigger instruction to the control unit.

[0112] exist Figure 2 In the embodiment, the light source in the endoscope system includes an illumination control unit and an illumination unit. After receiving the function instruction sent by the control unit in the camera system host, the illumination control unit sends an illumination instruction to the illumination unit to control the illumination unit to provide illumination light to the endoscope. The illumination unit receives the illumination instruction sent by the illumination control unit and provides illumination light to the endoscope.

[0113] exist Figure 2In the endoscope system, the endoscope has a camera optical system, an imaging unit, a processing unit and an operating unit. The camera optical system is composed of one or more lenses to focus the light from the part of the patient's body that needs to be observed so that the observed part can be clearly photographed. The imaging unit is composed of image sensors such as CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device), which is used to perform photoelectric conversion on the light received by each pixel to generate an image. The imaging unit sends the generated image to the processing unit. The processing unit receives the image sent by the imaging unit, converts the image into a digital signal image, and sends the converted image to the image input unit of the camera system host. When the user triggers the switch, button or touch panel on the endoscope, the operating unit is used to receive the user's trigger instruction and send the trigger instruction to the control unit of the camera system host.

[0114] The method provided in the embodiment of the present application is applied to the scenario of processing images using an endoscope system. Optionally, the image processing method provided in the embodiment of the present application can also be applied to other scenarios of processing images.

[0115] In order to implement the image processing method provided in the embodiment of the present application, the embodiment of the present application provides an image processing system. For the convenience of subsequent description, the image processing system is first explained in detail.

[0116] Figure 3 Schematic diagram of the architecture of an image processing system provided by an embodiment of the present application. Figure 3 As shown, the image processing system 300 includes a target recognition unit 301 , a control unit 302 , a ROI (region of interest) selection unit 303 , an image processing unit 304 , and a picture-in-picture overlay unit 305 .

[0117] Among them, the target recognition unit is used to recognize the object in the image. Since the object appearing in the region of interest in the image needs to be observed clearly and accurately, it is necessary to perform image processing on the region of interest in the image. The ROI selection unit is used to determine the region of interest. The image processing unit is used to process the region of interest in the image. The control unit is used to send function instructions to the ROI selection unit and the image processing unit to control the ROI selection unit to determine the region of interest, and control the image processing unit to process the region of interest in the image according to the specified image processing mode.

[0118] The image processing mode includes one or more of an image noise reduction mode, a sharpening mode, a contrast enhancement mode, a defogging mode, a multi-spectral imaging mode, a highlight suppression mode, an anti-red overflow mode, a false color highlight mode, an electronic amplification mode, a super-resolution mode, etc. The image processing mode may also be referred to as an image enhancement processing parameter.

[0119] The picture-in-picture overlay unit is used to overlay the processed image with the original image, so that both the processed image and the target image are displayed on the display device.

[0120] Based on the above-mentioned image processing system, the image processing method provided in the embodiment of the present application can be implemented in the following two ways.

[0121] In one possible implementation, Figure 4 As shown, Figure 4 1 is a flow chart of an image processing method provided in an embodiment of the present application. Figure 4 In the process, the target recognition unit receives the image captured by the endoscope, determines the object appearing in the image and the location information of the object, and then sends the determined object appearing in the image, the location information of the object and the image itself to the control unit. Figure 4 The location information shown in is the location information of the object, and the target type is the object.

[0122] The control unit receives the object appearing in the image and the position information of the object sent by the target recognition unit, and sends the position information of the object to the ROI selection unit to control the ROI selection unit to determine the region of interest of the image. The control unit is also used to determine the specified image processing mode based on the type of the object in the region of interest or the object appearing in the region of interest, that is, to determine the enhancement parameters of the image, and then send the image processing mode to the image processing unit, so as to control the image processing unit to process the region of interest of the image according to the specified image processing mode.

[0123] The ROI selection unit is used to receive the position information of the object sent by the control unit and the image collected by the endoscope, and determine the region of interest of the image according to the position information of the object. Figure 4 The ROI selection unit sends the determined ROI of the image to the image processing unit, and the ROI is a partial image in the image collected by the endoscope.

[0124] The image processing unit receives the image processing mode sent by the control unit and the region of interest of the image sent by the ROI selection unit. The image processing unit processes the region of interest of the image based on the image processing mode, and the processing is an enhancement processing, so as to observe the objects appearing in the region of interest more clearly and accurately. The image processing unit sends the processed image to the picture-in-picture overlay unit.

[0125] The picture-in-picture superposition unit receives the processed image and the image collected by the endoscope, and superimposes the processed image and the image collected by the endoscope, so that the processed image and the image collected by the endoscope are displayed on the display device.

[0126] In another possible implementation, Figure 5 As shown, Figure 5 is a flow chart of another image processing method provided by an embodiment of the present application. Figure 5 In the process, the control unit controls the ROI selection unit to receive the image collected by the endoscope, and determines the region of interest of the image according to the region position information specified by the user. The specified region position information is also called Figure 5 The ROI selection unit sends the determined region of interest of the image to the image processing unit and the target recognition unit.

[0127] The target recognition unit receives the region of interest of the image sent by the ROI selection unit, and then determines the object appearing in the region of interest based on the region of interest of the image. The target recognition unit sends the position information of the object in the region of interest and the object appearing in the region of interest to the control unit.

[0128] After receiving the position information of the object sent by the target recognition unit and the object appearing in the region of interest, the control unit determines the image processing mode based on the object appearing in the region of interest or the category to which the object belongs, that is, determines the enhancement parameters of the image. The image processing unit processes the region of interest of the image based on the determined image processing mode, and the processing may be enhancement processing, so as to observe the object appearing in the region of interest more clearly and accurately. Then the image processing mode is sent to the image processing unit, so as to control the image processing unit to process the region of interest of the image according to the specified image processing mode.

[0129] The image processing unit receives the image processing mode sent by the control unit and the region of interest of the image sent by the ROI selection unit. The image processing unit processes the region of interest of the image based on the image processing mode. The image processing unit sends the processed image to the picture-in-picture overlay unit.

[0130] The picture-in-picture superposition unit receives the processed image and the image collected by the endoscope, and superimposes the processed image and the image collected by the endoscope, so that the processed image and the image collected by the endoscope are displayed on the display device.

[0131] Above Figure 4 and Figure 5 Just Figure 3 The two possible image processing flows of the image processing system shown in the figure are applied Figure 3 When the image processing system is shown, the image processing flow of the image processing system can also be adjusted based on actual needs, and the embodiment of the present application is not limited thereto.

[0132] It should be noted that Figure 3 The various units in the image processing system shown can be centrally deployed in a terminal or in a server. Optionally, the various units in the image processing system can also be distributed and deployed on different devices, which is not limited in the embodiments of the present application.

[0133] also, Figure 1 Each unit of the image processing system is a software unit, and the naming of each unit is based on the function of the software unit. When applying the embodiment of the present application, different naming can be performed based on needs, and the embodiment of the present application does not limit the naming of the above-mentioned units.

[0134] based on Figure 3 The image processing system shown in the figure further describes the method provided in the embodiment of the present application. Figure 3 It can be seen from the image processing system shown that the execution subject of the image processing method is not limited. For the convenience of subsequent description, the following embodiments are described by taking the image processing system centrally deployed on the terminal as an example.

[0135] Figure 6 This is a flow chart of an image processing method provided by an embodiment of the present application. The image processing method may include the following steps:

[0136] Step 601: The terminal obtains an object appearing in a region of interest of a target image.

[0137] When the target image is an image captured by an endoscope, since different objects have different characteristics, the image can be processed in a targeted manner based on different objects. Figure 3 The target recognition unit in the target image first obtains the objects appearing in the region of interest of the target image, so as to determine different image processing modes based on the objects appearing in the region of interest.

[0138] Based on the above Figure 4 and Figure 5 It can be seen from the two flow charts that the terminal can obtain the objects appearing in the region of interest of the target image in the following two implementation methods.

[0139] (1) Figure 4 The image processing flow shown.

[0140] based on Figure 4 It can be seen from the image processing flow shown that the above-mentioned terminal obtains the object appearing in the region of interest of the target image in the following manner: the terminal identifies the object appearing in the target image and the location information of the object based on the deep learning model. The region of interest is determined based on the location information of the object, and the region of interest includes at least one object identified by the deep learning model.

[0141] The above-mentioned deep learning model is a pre-trained model for identifying objects in an image, and after identifying an object, the deep learning model may also determine the location information of the object. The location information of the object is specifically the position coordinates of the object in the target image. Based on the deep learning model, the intelligence level of the image processing system in the endoscopy scenario can be improved, and doctors do not need to perform debugging operations, which can reduce the doctor's additional debugging steps and improve the doctor's work efficiency.

[0142] The training process of the above-mentioned deep learning model can be: the terminal obtains multiple training samples, each of the multiple training samples includes an image and a label corresponding to the training sample, and the label corresponding to the training sample includes an object appearing in the image and the location information of the object. The terminal trains the initialized learning model based on the multiple training samples and the labels corresponding to each training sample to obtain the deep learning model.

[0143] Specifically, when the target image is an image collected by an endoscope, firstly, images collected by an endoscope in various medical scenarios are collected, and the collected images are divided according to one or more factors such as medical departments, application scenarios, and object types, and then the objects appearing in the collected images are annotated. The objects appearing in the images include human tissues or organs such as the throat, tympanic membrane, sinuses, and esophagus in the diagnosis scenario, and various instruments in the surgical scene, such as surgical forceps, electric scalpels, gauze, etc. Then, the position information of the object in the image is determined based on the object appearing in the image. In this way, the object appearing in the image and the position information of the object are used as the label of a sample. Typical samples and sample labels are selected from all samples to construct a training data set. The training data set is input into the initialized deep learning model for training to obtain the trained relationship function, which is the above-mentioned deep learning model. Among them, the deep learning model can be FRCNN (faster regions with convolutional neural network features, a target recognition technology that uses convolutional neural networks to extract regional features), SSD (single shot multi-box detector, a single-stage multi-box prediction classification detector), YOLO (you only look once, a target recognition network that only needs one time), etc.

[0144] In addition, the above-mentioned deep learning model can also use multi-frame fusion technology to verify the recognition results based on the deep learning model, which can further improve the stability and accuracy of recognition. In actual scenes, when there is motion, occlusion, etc. of the object in the image, the image quality will be reduced, affecting the accuracy of identifying the object in the image. Combined with multi-frame fusion technology, multiple frames of video images are input into the target recognition unit. The target recognition unit will combine the motion information, category information, etc. of the object in the previous and next frame images to more accurately determine the position and category of the object.

[0145] The above-mentioned terminal recognizes the objects appearing in the target image and the location information of the objects based on the deep learning model in the following manner: when the target image is an image acquired by an endoscope, the terminal inputs the target image acquired by the endoscope into the deep learning model, and the deep learning model outputs the objects appearing in the target image corresponding to the target image and the location information of the objects according to the characteristics of the target image. The number of objects appearing in the target image recognized based on the deep learning model can be one or more.

[0146] In addition, the above-mentioned implementation method of determining the region of interest based on the position information of the object may be: based on the position information of the identified object, an area within a reference range around the position information is used as the region of interest.

[0147] When the target image is an image collected by an endoscope, the identified object can be used as the center to extend outward to a certain range, and the area within the range can be used as the region of interest. For example, 10 pixels above the upper edge of the object is used as the upper edge of the region of interest, 10 pixels below the lower edge of the object is used as the lower edge of the region of interest, 10 pixels to the right of the right edge of the object is used as the right edge of the region of interest, and 10 pixels to the left of the left edge of the object is used as the left edge of the region of interest. For another example, if the object is a square with a side length of 10 pixels, 10 pixels from the left edge of the square can be used as the left edge of the region of interest, 10 pixels from the right edge of the square can be used as the right edge of the region of interest, 10 pixels from the upper edge of the square can be used as the upper edge of the region of interest, and 10 pixels from the lower edge of the square can be used as the lower edge of the region of interest. The width of the region of interest determined in this way is the width of the identified object plus 20 pixels, and the height is the height of the identified object plus 20 pixels, and the region of interest is a rectangle.

[0148] In addition, when there are multiple objects identified by the deep learning model, one possible implementation method for determining the region of interest based on the location information of the object is: the terminal extends a certain range outward from each of all objects as the center, and the area within the range is the region of interest. In this case, multiple regions of interest can be determined, and the operation of determining the image processing mode can be performed for each of the multiple regions of interest.

[0149] In addition, considering that different objects have different importances, different priorities can be set in advance for different objects. At this time, another possible implementation method of determining the region of interest based on the location information of the object is: the terminal determines the priority of each object among the multiple objects. Based on the priority of each object among the multiple objects, a target object is selected from the multiple objects. The region of interest is determined based on the location information of the target object.

[0150] When the target image is an image captured by an endoscope, the priority of each of the above multiple objects is set in advance through the doctor's preset operation. The doctor can set different priorities according to the importance levels of different objects. The more important the object, the higher the priority. The importance level is determined based on the current scene. For example, when the doctor examines the patient's throat, the image captured by the endoscope may also include the patient's mouth, tongue, etc., then the throat has the highest priority. By setting the priorities of multiple objects, the efficiency of acquiring the region of interest can be improved by prioritizing the object with a higher priority to determine the region of interest.

[0151] like Figure 7 As shown, Figure 7This is a flow chart for determining an area of ​​interest provided by an embodiment of the present application. Figure 7 In, first Figure 3 The target recognition unit in receives the target image, that is, Figure 7 The original image in the target image is then determined based on the deep learning model, and the object's location information is determined. The object in the target image is also called Figure 7 The key target in the target image is then determined. If there is no object, the target image is recaptured. If there is an object, it is determined whether there are multiple objects. If there are no multiple objects, the region of interest is determined based on the position information of an object. If there are multiple objects, the region of interest is determined based on the priority of multiple objects. Finally, the determined region of interest is output.

[0152] (2) Figure 5 The image processing flow shown.

[0153] based on Figure 5 It can be seen from the image processing flow shown that the terminal obtains the category of the object appearing in the region of interest of the target image by: determining the region of interest based on the specified region position information, and identifying the object appearing in the region of interest based on the deep learning model.

[0154] When the target image is an image captured by an endoscope, the terminal determines the region of interest based on the specified region position information by manually capturing a fixed region of interest in the target image through a preset region value or a user through a specified method. The preset region value is the location information of the region of interest set in advance by the doctor and the width and height of the region of interest.

[0155] The above-mentioned implementation method of identifying the objects appearing in the region of interest based on the deep learning model is as follows: the terminal inputs the region of interest in the target image to the deep learning model, and the deep learning model outputs the objects appearing in the region of interest in the target image based on the region of interest in the target image. Figure 3 The process shown in the figure uses the region of interest to determine the object, which significantly speeds up the speed of determining the object and better meets the higher real-time requirements in special medical scenarios.

[0156] like Figure 8 As shown, Figure 8 This is another flow chart for determining an area of ​​interest provided by an embodiment of the present application. Figure 8 In, first Figure 4 The target recognition unit in the image receives a preset region of interest and determines the object appearing in the region of interest based on the deep learning model. Then, it is determined whether there is an object. If there is no object, the region of interest is reacquired. If there is an object, the region of interest is output.

[0157] Step 602: The terminal determines an image processing mode based on the objects appearing in the region of interest, and at least two of the different objects correspond to different image processing modes.

[0158] When the target image is an image collected by an endoscope, in order to make the area of ​​interest achieve the best display effect desired by the doctor, the terminal needs to use Figure 3 The control unit in the image processing module determines different image processing modes for different objects.

[0159] In a possible implementation, the above-mentioned method of determining the image processing mode based on the object appearing in the region of interest is as follows: based on the first mapping relationship, the category of the object appearing in the region of interest is determined, and then based on the second mapping relationship, the image processing mode is determined. The first mapping relationship indicates the correspondence between the object and the category, and the second mapping relationship indicates the correspondence between the object category and the image processing mode.

[0160] The above-mentioned method of determining the category of the object appearing in the region of interest based on the first mapping relationship is as follows: the terminal stores the first mapping relationship, and the terminal searches the first mapping relationship for the object category corresponding to the object appearing in the region of interest. The first mapping relationship is manually determined and input.

[0161] When the target image is an image acquired by an endoscope, as shown in Table 1, Table 1 is the first mapping relationship, that is, the correspondence between objects and categories. In Table 1, objects such as inflammation and tumors belong to the lesion area class, objects such as throat, sinus, mucosal cortex, etc. belong to the microvascular class, objects such as esophagus, duodenum, urethra, etc. belong to the intestine / tract class, objects such as surgical forceps, electric scalpels or instrument heads belong to the instrument class, objects such as gauze belong to the gauze class, and objects such as smoke belong to the smoke class.

[0162] Table 1:

[0163]

[0164] The above-mentioned method of determining the image processing mode based on the second mapping relationship is as follows: the second mapping relationship is stored in the terminal, and after the terminal determines the category of the object based on the object, the terminal searches the second mapping relationship for the image processing mode corresponding to the object category according to the object category. The second mapping relationship is manually determined and input.

[0165] When the target image is an image acquired by an endoscope, the above object categories include lesion area objects, microvascular objects, intestinal / tract objects, instrument objects, gauze objects, smoke objects, etc. Among them, lesion area objects include inflammation, tumor and other objects, microvascular objects include throat, sinus, mucosal cortex and other objects, intestinal / tract objects include esophagus, duodenum, urethra and other objects, instrument objects include surgical forceps, electric scalpel or instrument head and other objects, gauze objects include gauze and other objects, smoke objects include dense fog and other objects.

[0166] The above-mentioned image processing modes include image noise reduction mode, sharpening mode, contrast enhancement mode, dehazing mode, multispectral imaging mode, highlight suppression mode, anti-red overflow mode, false color highlight mode, electronic magnification mode, super-resolution mode, etc. Various image processing modes can refer to relevant methods known in the industry.

[0167] As shown in Table 2, Table 2 is the second mapping relationship, that is, the correspondence between the object category and the image processing mode. In Table 2, the region of interest includes the lesion region class object can be processed in the electronic magnification mode, the region of interest includes the microvascular class object can be processed in the multispectral imaging mode, the region of interest includes the intestine / tract class object can be processed in the contrast enhancement mode and the highlight suppression mode, the region of interest includes the instrument class object can be processed in the highlight suppression mode, the region of interest includes the gauze class object can be processed in the pseudo color highlight mode, and the region of interest includes the smoke class object can be processed in the defogging processing mode.

[0168] Table 2:

[0169]

[0170] The image processing modes corresponding to different object categories in the second mapping relationship are determined based on the common characteristics of the objects in the object category. Specifically, when the target image is an image captured by an endoscope, the corresponding relationships between different object categories and different image processing modes are specifically divided into the following categories.

[0171] In the first category, since the lesion area object is small and difficult to be found, when the object appearing in the region of interest is a lesion area object, the image processing mode includes an electronic magnification mode, and the electronic magnification processing mode is used to magnify the region of interest.

[0172] The electronic magnification mode can magnify the lesion area objects in the area of ​​interest. After magnification, the lesion area objects can be clearer, which is convenient for doctors to observe the details. In the disease diagnosis stage, it can avoid the missed diagnosis of tiny early cancers, abnormal hyperplasia, etc. In the surgical treatment stage, it can also ensure that tumors and other diseases are completely removed without omissions.

[0173] In addition, when processing the region of interest where the lesion region object is located, it is not limited to the electronic magnification mode, but may also include one or more of the image noise reduction mode, sharpening mode, contrast enhancement mode, defogging mode, multi-spectral imaging mode, highlight suppression mode, anti-red overflow mode, super-resolution mode, etc. The specific modes and the specific parameters in each mode can be specified by the doctor through preset operations.

[0174] The second type is that since blood vessels near microvascular objects are difficult to observe, when the object appearing in the region of interest is a microvascular object, the image processing mode includes a multispectral imaging mode. The multispectral imaging mode is used to enhance different bands in the region of interest.

[0175] The multispectral imaging mode can present blood vessels of different depths and thicknesses near microvascular objects in the area of ​​interest with more significant contrast, making it easier for doctors to observe the morphology and structure of blood vessels, helping doctors to make earlier and more accurate diagnosis of diseases.

[0176] Among them, the multispectral imaging mode uses a special correction matrix to process the region of interest to enhance the contrast between targets such as blood vessels and surrounding tissues. The correction matrix is ​​shown below. The bands of the three channels of red (R), green (G), and blue (B) in the region of interest are enhanced. R_ehc is the output after the band enhancement of the red channel, G_ehc is the output after the band enhancement of the green channel, B_ehc is the output after the band enhancement of the blue channel, and R_ori, G_ori, and B_ori are the inputs of the bands of the red, green, and blue channels. oft_R, oft_G, and oft_B are offsets. The matrix represented by coef_X is a coefficient matrix, for example, coef_1. It should be noted that the correction matrix can be based on the global information of the image to be processed by multispectral imaging, or it can be based on the local information of the image to be processed by multispectral imaging.

[0177]

[0178] In addition, when processing the region of interest where microvascular objects are located, it is not limited to the multispectral imaging mode, but may also include one or more of the image noise reduction mode, sharpening mode, contrast enhancement mode, defogging mode, highlight suppression mode, anti-red overflow mode, electronic magnification mode, super-resolution mode, etc. The specific modes and the specific parameters in each mode can be specified by the doctor through preset operations.

[0179] The third category is that due to the long and narrow shape of intestinal objects, it is difficult to fully fill in the light at the location of the intestinal objects. For example, when using an endoscope to diagnose intestinal tissues or organs, the near view is obviously brighter because it is close to the light source, while the far view is obviously darker than the foreground area due to the dual effects of being far away from the light source and automatic exposure, which makes it difficult for doctors to accurately analyze the condition. Therefore, when the object appearing in the region of interest is an intestinal object, the image processing mode includes contrast enhancement mode and highlight suppression mode.

[0180] Among them, the contrast enhancement mode is used to enhance the contrast of the region of interest, and the highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest. The contrast enhancement mode and highlight suppression mode can automatically enhance the brightness and texture details of the region of interest without the doctor's additional interactive control, making the doctor's use of the endoscope more intelligent and convenient.

[0181] In addition, when processing the region of interest where the intestinal / tract objects are located, it is not limited to the contrast enhancement mode and the highlight suppression mode, but may also include one or more of the image noise reduction mode, the sharpening mode, the defogging mode, the multi-spectral imaging mode, the anti-red overflow mode, the electronic magnification mode, the super-resolution mode, etc. The specific modes and the specific parameters in each mode may be specified by the doctor through the preset operation.

[0182] The fourth category is that since instrument objects are made of metal, metal instruments often reflect light or are particularly bright. Therefore, when the object appearing in the region of interest is an instrument object, the image processing mode includes a highlight suppression mode.

[0183] The highlight suppression mode is used to suppress the brightness of the highlight part in the area of ​​interest. The highlight suppression mode can prevent the instrument from producing excessive visual stimulation to the doctor, and also enable the doctor to better locate the position of the surgical instrument during the operation, ensuring the accuracy of surgical operations such as cutting, suturing, and drainage, thereby improving the efficiency and safety of the surgical process.

[0184] In addition, when processing the region of interest where the instrument object is located, it is not limited to the highlight suppression mode, but may also include one or more of the image noise reduction mode, sharpening mode, contrast enhancement mode, defogging mode, multi-spectral imaging mode, anti-red overflow mode, electronic magnification mode, super-resolution mode, etc. The specific modes and the specific parameters in each mode can be specified by the doctor through preset operations.

[0185] The fifth category is that gauze-like objects may be left in the patient's body due to mistakes. Therefore, when the object appearing in the region of interest is a gauze-like object, the image processing mode includes a pseudo-color highlight mode, which is used to perform pseudo-color display processing on the object appearing in the region of interest. The pseudo-color highlight mode can determine the precise position information of the gauze. Combined with this information, the image processing unit can highlight the gauze area in the form of pseudo-color, significantly improving its recognition and avoiding missing it in the patient's body.

[0186] In addition, when processing the region of interest where the instrument object is located, it is not limited to the highlight suppression mode, but may also include one or more of the image noise reduction mode, sharpening mode, contrast enhancement mode, defogging mode, multi-spectral imaging mode, highlight suppression mode, anti-red overflow mode, electronic magnification mode, super-resolution mode, etc. The specific modes may be specified by the doctor through preset operations.

[0187] The sixth category, since the smoke density of smoke-like objects may affect surgical operations, when the objects appearing in the region of interest are smoke-like objects, the image processing mode includes a defogging mode, which is used to defog smoke-like objects appearing in the region of interest. The defogging mode can remove the smoke density, allowing doctors to perform surgical operations on patients more accurately.

[0188] In addition, when processing the region of interest where the smoke-like object is located, it is not limited to the highlight suppression mode, but may also include one or more of the image noise reduction mode, sharpening mode, contrast enhancement mode, multi-spectral imaging mode, highlight suppression mode, anti-red overflow mode, electronic magnification mode, super-resolution mode, etc. The specific modes and the specific parameters in each mode may be specified by the doctor through preset operations.

[0189] In another possible implementation, the above-mentioned method of determining the image processing mode based on the object appearing in the region of interest is as follows: determining the image processing mode based on a third mapping relationship, where the third mapping relationship indicates a correspondence between the object and the image processing mode.

[0190] The above-mentioned method of determining the image processing mode based on the third mapping relationship is as follows: the terminal stores the third mapping relationship, and the terminal searches the third mapping relationship for the image processing mode corresponding to the object according to the object appearing in the region of interest. The third mapping relationship is manually determined and input.

[0191] As shown in Table 3, Table 3 is the third mapping relationship, that is, the correspondence between the object and the image processing mode. In Table 3, the objects including inflammation, tumor and other regions of interest can be processed by the electronic magnification mode, the objects including the throat, sinus, mucosal cortex and other regions of interest can be processed by the multispectral imaging mode, the objects including the esophagus, duodenum, urethra and other regions of interest can be processed by the contrast enhancement mode and the highlight suppression mode, the objects including surgical forceps, electric scalpel or instrument head and other regions of interest can be processed by the highlight suppression mode, the objects including gauze and other regions of interest can be processed by the pseudo color highlight mode, and the objects including smoke and other regions of interest can be processed by the defogging processing mode.

[0192] Table 3:

[0193]

[0194] Step 603: The terminal processes the region of interest based on the image processing mode.

[0195] Based on step 602, different objects may correspond to different image processing modes, and therefore, the terminal processes the region of interest in different ways based on the image processing mode.

[0196] Based on the first category in step 602, when the object appearing in the region of interest is a lesion region class object, the image processing mode includes an electronic magnification mode, and the electronic magnification is used to magnify the region of interest. The magnification ratio can be preset or configured according to the needs of the doctor, and different magnification ratios can be preset according to the object in the specific region of interest.

[0197] Specifically, multiple pixels are interpolated in the region of interest, and the interpolated pixels are distributed around the original pixels. The multiple interpolated pixels are combined with the original pixels to obtain the enlarged region of interest. Interpolating multiple pixels in the region of interest can be achieved by nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, spline interpolation, etc., which is not limited in the embodiments of the present application.

[0198] Based on the second category in step 602, when the object appearing in the region of interest is a microvascular object, the image processing mode includes a multispectral imaging mode.

[0199] The multispectral imaging mode is used to enhance the bands of the red, green and blue channels of the region of interest. Specifically, for objects appearing in the region of interest, a combination of multiple multispectral wavelengths is preset, and the bands of the red, green and blue channels of the region of interest are enhanced according to the aforementioned correction matrix.

[0200] Based on the third category in step 602, when the object appearing in the region of interest is an intestinal / tract object, the image processing mode includes a contrast enhancement mode and a highlight suppression mode.

[0201] The contrast enhancement mode is used to perform contrast enhancement processing on the region of interest. In a possible implementation, histogram equalization can be used to perform contrast enhancement processing. Specifically, a histogram corresponding to the number of pixels in the region of interest and the grayscale of the pixels is constructed, in which the horizontal axis is the grayscale value of the pixel, and the vertical axis is the number of pixels corresponding to the grayscale value. The range where the grayscale value is concentrated is called grayscale. The range of grayscale levels in the region of interest is enlarged, that is, the area where the grayscale value is concentrated in the horizontal axis is widened, thereby increasing the contrast of the image.

[0202] Optionally, the contrast enhancement mode may also be implemented using Retinex (an image enhancement method) enhancement, gradient domain contrast enhancement, and other methods, which are not limited herein.

[0203] The highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest. Specifically, the highlight suppression mode can use global dynamic range compression or local dynamic range compression technology to suppress the highlight part of the region of interest while ensuring the overall contrast in the region of interest. The highlight suppression mode can also suppress the highlight area in the region of interest through a curve mapping method, and then weight the mapped region of interest with the target image to obtain the processed region of interest. The curve mapping method of highlight suppression can be based on the global information of the image to be subjected to highlight suppression processing, or it can be based on the local information of the image to be subjected to highlight suppression processing.

[0204] like Fig. 9 As shown, Fig. 9 This is a highlight suppression curve mapping diagram provided by the real-time example of this application. Fig. 9 In the case of no highlight suppression, the input brightness is proportional to the output brightness, that is, Fig. 9 The 45-degree dotted line in the figure shows that the output brightness is equal to the input brightness. When performing highlight suppression, the relationship between the input brightness and the output brightness is: Fig. 9 The solid line in , when the input brightness of the image is greater than m, it is obvious that the brightness of the output image will be less than the input brightness, achieving the effect of suppressing the brightness of the higher brightness part in the region of interest, that is, achieving the highlight suppression effect.

[0205] Based on the fourth category in step 602, when the object appearing in the region of interest is a device-type object, the image processing mode includes a highlight suppression mode. The highlight suppression mode is the same as the highlight suppression mode of the third category, and will not be described in detail here.

[0206] Based on the fifth category in step 602, when the object appearing in the region of interest is a gauze-like object, the image processing mode includes a pseudo-color highlight mode. The pseudo-color highlight mode is used to perform pseudo-color display processing on the object appearing in the region of interest. Specifically, based on the position information of the object, the object appearing in the region of interest is recolored to a preset color to achieve the purpose of highlighting. Fig.10 As shown, Fig.10 This is a method for realizing false color highlighting provided by an embodiment of the present application. Fig.10 In the present invention, the objects and the position information of the objects appearing in the region of interest are used to recolor the pixels of the region of interest, thereby obtaining the region of interest highlighted with false colors.

[0207] Based on the sixth category in step 602, when the object appearing in the region of interest is a smoke-like object, the image processing mode includes a defogging processing mode, and the defogging processing mode is used to perform defogging processing on the smoke-like object appearing in the region of interest. In a possible implementation, the concentration of the smoke-like object appearing in the region of interest may be determined; if the concentration of the smoke-like object exceeds a concentration threshold, the degree of defogging processing on the region of interest is increased; if the concentration of the smoke-like object is lower than the concentration threshold, the degree of defogging processing on the region of interest is reduced.

[0208] Specifically, the values ​​of the red, green and blue channels of each pixel in the region of interest are first divided by the A value (global datum value), that is, the values ​​of the red, green and blue channels are normalized, and then the minimum value of the three channels is taken to obtain a dark channel image. The dark channel image is filtered using a specified filter window to obtain a transmittance image, which is the image after defogging. Among them, the size of the filter window is selected according to the specific situation. If the concentration of smoke objects exceeds the concentration threshold, the larger the selected filter window, that is, the greater the degree of defogging of the region of interest. If the concentration of smoke objects is lower than the concentration threshold, the smaller the selected filter window, that is, the less the degree of defogging of the region of interest.

[0209] That is, in Figure 3 In the image processing system shown, the image processing unit detects the residual fog concentration in the current area of ​​interest in real time, and feeds back the residual fog concentration information to the control unit, which determines whether the defogging intensity needs to be further increased or decreased according to the preset target fog concentration value. The defogging processing mode can be implemented using an image defogging method based on a dark channel.

[0210] In addition, in step 602, the image noise reduction mode is to remove noise points in the region of interest, wherein the image noise reduction mode can be implemented by methods such as mean filtering, Gaussian filtering, bilateral filtering, guided filtering, and wavelet noise reduction.

[0211] The image sharpening mode is to reduce the edge of the pixel value mutation in the region of interest. The image sharpening mode can be achieved by differential sharpening, inverse mask sharpening and other methods.

[0212] The super-resolution mode can be implemented based on deep learning algorithms, such as SRCNN (superresolution convolutional neural network), ESPCN (efficient sub-pixel convolutional neural network), VDSR (Very Deep network for Super-Resolution), SRGAN (photo-realistic single image super-resolution using a generative adversarial network), EDSR (enhanced deep residual networks for single image super-resolution) and other super-resolution network structures.

[0213] The anti-red overflow function suppresses the intensity response of the red channel in the input image to prevent the red from overflowing easily in the bleeding scene. Fig.11 As shown, Fig.11 is a red overflow prevention curve mapping diagram provided in an embodiment of the present application. Fig.11 In , the horizontal axis represents the red channel response of the input image, and the vertical axis represents the red channel response of the output image. When the red overflow prevention process is not performed on the region of interest, the red channel response of the input image is proportional to the red channel response of the output image, that is, Fig.11 The 45-degree dotted line in the middle, at this time, the red channel response of the output image is equal to the red channel response of the input image. When anti-red overflow processing is performed, the red channel response of the input image and the red channel response of the output image are Fig. 9 The solid line in , when the red channel response of the input image is p, the corresponding red channel response of the output image is p. At this time, the red channel response of the output image is smaller than the red channel response of the input image, achieving the anti-red overflow effect. The curve mapping method for anti-red overflow can be based on the global information of the image to be processed for anti-red overflow, or it can be based on the local information of the image to be processed for anti-red overflow.

[0214] Step 604: The terminal displays the processed region of interest.

[0215] When the target image is an image acquired by an endoscope, in order to enable the doctor to observe the region of interest, the processed region of interest needs to be displayed on the terminal.

[0216] In a possible implementation, the terminal displays the processed region of interest in the following manner: the target image is displayed in the main screen area of ​​the display interface, and the processed region of interest is displayed in the auxiliary screen area of ​​the display interface, and the auxiliary screen area and the main screen area are superimposed together. There are one or more auxiliary screens. In this way, the target image can be retained while providing doctors with richer and more comprehensive diagnostic information.

[0217] like Fig.12 As shown, Fig.12 A display interface diagram provided in an embodiment of the present application. Fig.13 There is a main screen and three auxiliary screens.

[0218] The above-mentioned auxiliary screen area and main screen area are superimposed in the following way: after the area of ​​interest is processed by the image processing mode, the main screen detects that the area of ​​interest has been processed, and then obtains the target image, ensuring that the target image input to the screen superposition module belongs to the same frame of video screen as the area of ​​interest, and the screen superposition module embeds the auxiliary screen where the area of ​​interest is located into the main screen to achieve superposition. This can further meet the higher requirements of medical scenarios for system accuracy.

[0219] like Fig.13 As shown, Fig.13 This is a picture-in-picture principle flow chart provided by an embodiment of the present application. First, the image data is input into the DDR (double data rate) memory for caching, and then the region of interest is obtained from the DDR memory according to the position information of the region of interest output by the recognition model. After being processed by image processing modes such as noise reduction mode, sharpening mode, electronic magnification mode, contrast enhancement mode, and multi-spectral imaging mode, the processed region of interest is obtained. The processed region of interest is also called the data stream of the auxiliary screen. The main screen obtains the target image from the DDR memory based on the processed auxiliary screen data stream, and the target image is also called the main screen data stream. By controlling the timing to ensure that the target image input to the screen overlay module and the region of interest belong to the same frame of video screen, the screen overlay module embeds the auxiliary screen where the region of interest is located into the main screen to achieve overlay, and finally transmits the data to the display interface of the display device for display.

[0220] In addition, when the display interface includes multiple auxiliary screen areas, the multiple auxiliary screen areas are used to display images processed by different image processing modes for the same region of interest, or to display images processed by different regions of interest.

[0221] Among them, when multiple auxiliary screens display images processed by different image processing modes for the same region of interest, the priorities of the images processed by different image processing modes are pre-set, and the images processed by different image processing modes are displayed in the auxiliary screen in descending order of priority. When multiple auxiliary screens display different regions of interest, the priorities of different regions of interest are pre-set, and the different regions of interest are displayed in the auxiliary screen in descending order of priority.

[0222] like Fig.14 As shown, Fig.14 This is a schematic diagram of a display screen provided by an embodiment of the present application. In the figure, the region of interest is the region where the star is located, and at three auxiliary screen positions, images processed using different image processing modes for the region where the star is located are displayed respectively.

[0223] like Fig.15 As shown, Fig.15 This is another display screen schematic diagram provided by an embodiment of the present application. In the figure, there are three regions of interest, namely, the region where the star is located, the region where the right triangle is located, and the region where the isosceles triangle is located. In the three auxiliary screen positions, the processed image of the region where the star is located, the processed image of the region where the right triangle is located, and the processed image of the region where the isosceles triangle is located are displayed respectively.

[0224] In another possible implementation, the terminal displays the processed region of interest in the following manner: displaying the processed region of interest in the main screen area of ​​the display interface. At this time, the target image is not displayed.

[0225] In summary, in the embodiments of the present application, different image processing modes are used to process the regions of interest corresponding to different objects, and the processed regions of interest are displayed. Since there are at least two objects among different objects corresponding to different image processing modes, the regions of interest corresponding to each object are processed in a targeted manner. Therefore, when the target image is an image captured by an endoscope, different image processing can be performed in the above manner for different regions of interest acquired in different scenes. In the endoscope scene, the displayed region of interest can achieve the best display effect desired by the doctor, which is conducive to the doctor's detailed observation of the region of interest, and then accurately judge the patient's condition, or during the operation, the doctor can operate more accurately.

[0226] All the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and the embodiments of the present application will not be described in detail one by one.

[0227] Fig.16 16 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device can be implemented by software, hardware or a combination of both. The image processing device 1600 may include: an acquisition module 1601, a determination module 1602, a processing module 1603, and a display module 1604.

[0228] An acquisition module, used for acquiring objects appearing in a region of interest of a target image;

[0229] A determination module, used for determining an image processing mode based on objects appearing in the region of interest, wherein at least two of the different objects correspond to different image processing modes;

[0230] A processing module, used for processing the region of interest based on an image processing mode;

[0231] The display module is used to display the processed region of interest.

[0232] Optionally, a module is determined, including:

[0233] A first determination unit is used to determine the category of an object appearing in the region of interest based on a first mapping relationship, and to determine an image processing mode based on a second mapping relationship, wherein the first mapping relationship indicates a correspondence between an object and a category, and the second mapping relationship indicates a correspondence between an object category and an image processing mode; or, to determine the image processing mode based on a third mapping relationship, wherein the third mapping relationship indicates a correspondence between an object and an image processing mode.

[0234] Optionally, when the object appearing in the region of interest is a lesion region type object, the image processing mode includes an electronic magnification mode, and the electronic magnification processing mode is used for magnification processing of the region of interest; or,

[0235] When the object appearing in the region of interest is a microvessel, the image processing mode includes a multispectral imaging mode, which is used to enhance different bands in the region of interest; or,

[0236] When the object appearing in the region of interest is an intestinal / intestinal object, the image processing mode includes a contrast enhancement mode and a highlight suppression mode. The contrast enhancement mode is used to perform contrast enhancement processing on the region of interest, and the highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest; or,

[0237] When the object appearing in the region of interest is a device-type object, the image processing mode includes a highlight suppression mode; or,

[0238] When the object appearing in the region of interest is a gauze-like object, the image processing mode includes a false color highlight mode, and the false color highlight mode is used to perform false color display processing on the object appearing in the region of interest; or,

[0239] When the object appearing in the region of interest is a smoke-like object, the image processing mode includes a defogging processing mode, and the defogging processing mode is used to perform defogging processing on the smoke-like object appearing in the region of interest.

[0240] Optionally, when the object appearing in the region of interest is a smoke-like object, the processing module includes:

[0241] A second determining unit is used to determine the concentration of smoke-like objects appearing in the region of interest;

[0242] A first judgment unit, configured to increase the degree of defogging of the region of interest if the concentration of the smoke object exceeds a concentration threshold;

[0243] If the concentration of smoke-like objects is below the concentration threshold, the degree of dehazing of the region of interest is reduced.

[0244] Optionally, the determination module further includes:

[0245] A first recognition unit, configured to recognize objects appearing in a target image and location information of the objects based on a deep learning model;

[0246] The third determining unit is used to determine a region of interest based on the position information of the object, where the region of interest includes at least one object identified by the deep learning model.

[0247] Optionally, the number of objects appearing in the target image identified based on the deep learning model is multiple;

[0248] The third determining unit is used for:

[0249] determining a priority for each of the plurality of objects;

[0250] selecting a target object from the plurality of objects based on a priority of each of the plurality of objects;

[0251] The region of interest is determined based on the location information of the target object.

[0252] Optionally, the determination module further includes:

[0253] A fourth determining unit, configured to determine an area of ​​interest based on the specified area location information;

[0254] The second recognition unit is used to recognize objects appearing in the region of interest based on the deep learning model.

[0255] Optionally, the display module includes:

[0256] A display unit, used to display the target image in the main screen area of ​​the display interface;

[0257] The display unit is used to display the processed region of interest in the auxiliary screen area of ​​the display interface, and the auxiliary screen area and the main screen area are superimposed together.

[0258] Optionally, the display interface includes multiple auxiliary screen areas;

[0259] The multiple auxiliary screen areas are used to display images processed by different image processing modes for the same region of interest, or to display images processed for different regions of interest.

[0260] In summary, in the embodiments of the present application, different image processing modes are used to process the regions of interest corresponding to different objects, and the processed regions of interest are displayed. Since there are at least two objects among different objects corresponding to different image processing modes, the regions of interest corresponding to each object are processed in a targeted manner. Therefore, when the target image is an image captured by an endoscope, different image processing can be performed in the above manner for different regions of interest acquired in different scenes. In the endoscope scene, the displayed region of interest can achieve the best display effect desired by the doctor, which is conducive to the doctor's detailed observation of the region of interest, and then accurately judge the patient's condition, or during the operation, the doctor can operate more accurately.

[0261] It should be noted that: when the image processing device provided in the above embodiment performs image processing, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0262] Fig.171 is a block diagram of a terminal 1700 provided in an embodiment of the present application. The terminal 1700 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1700 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0263] Typically, the terminal 1700 includes a processor 1701 and a memory 1702 .

[0264] The processor 1701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0265] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1702 is used to store at least one instruction, which is used to be executed by the processor 1701 to implement the image processing method provided in the method embodiment of the present application.

[0266] In some embodiments, the terminal 1700 may also optionally include: a peripheral device interface 1703 and at least one peripheral device. The processor 1701, the memory 1702 and the peripheral device interface 1703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707, a positioning assembly 1708 and a power supply 1709.

[0267] The peripheral device interface 1703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1701 and the memory 1702. In some embodiments, the processor 1701, the memory 1702, and the peripheral device interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1701, the memory 1702, and the peripheral device interface 1703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0268] The radio frequency circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1704 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1704 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1704 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0269] The display screen 1705 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1705 is a touch display screen, the display screen 1705 also has the ability to collect touch signals on the surface or above the surface of the display screen 1705. The touch signal can be input to the processor 1701 as a control signal for processing. At this time, the display screen 1705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1705 can be one, and the front panel of the terminal 1700 is set; in other embodiments, the display screen 1705 can be at least two, which are respectively set on different surfaces of the terminal 1700 or are folded; in other embodiments, the display screen 1705 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1700. Even, the display screen 1705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0270] The camera assembly 1706 is used to capture images or videos. Optionally, the camera assembly 1706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0271] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1701 for processing, or input them into the radio frequency circuit 1704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0272] Positioning component 1708 is used to locate the current geographical location of terminal 1700 to implement navigation or LBS (Location Based Service). Positioning component 1708 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0273] The power supply 1709 is used to power various components in the terminal 1700. The power supply 1709 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1709 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0274] In some embodiments, the terminal 1700 further includes one or more sensors 1710 , including but not limited to: an acceleration sensor 1711 , a gyroscope sensor 1712 , a pressure sensor 1713 , a fingerprint sensor 1714 , an optical sensor 1715 , and a proximity sensor 1716 .

[0275] The acceleration sensor 1711 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1700. For example, the acceleration sensor 1711 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1701 can control the display screen 1705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1711. The acceleration sensor 1711 can also be used for collecting game or user motion data.

[0276] The gyro sensor 1712 can detect the body direction and rotation angle of the terminal 1700, and the gyro sensor 1712 can cooperate with the acceleration sensor 1711 to collect the user's 3D actions on the terminal 1700. The processor 1701 can implement the following functions based on the data collected by the gyro sensor 1712: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0277] The pressure sensor 1713 can be set on the side frame of the terminal 1700 and / or the lower layer of the display screen 1705. When the pressure sensor 1713 is set on the side frame of the terminal 1700, the user's holding signal of the terminal 1700 can be detected, and the processor 1701 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1713. When the pressure sensor 1713 is set on the lower layer of the display screen 1705, the processor 1701 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0278] The fingerprint sensor 1714 is used to collect the user's fingerprint, and the processor 1701 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1714, or the fingerprint sensor 1714 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1701 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1714 can be set on the front, back, or side of the terminal 1700. When a physical button or a manufacturer logo is set on the terminal 1700, the fingerprint sensor 1714 can be integrated with the physical button or the manufacturer logo.

[0279] The optical sensor 1715 is used to collect the ambient light intensity. In one embodiment, the processor 1701 can control the display brightness of the display screen 1705 according to the ambient light intensity collected by the optical sensor 1715. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1705 is increased; when the ambient light intensity is low, the display brightness of the display screen 1705 is reduced. In another embodiment, the processor 1701 can also dynamically adjust the shooting parameters of the camera assembly 1706 according to the ambient light intensity collected by the optical sensor 1715.

[0280] The proximity sensor 1716, also called a distance sensor, is usually arranged on the front panel of the terminal 1700. The proximity sensor 1716 is used to collect the distance between the user and the front of the terminal 1700. In one embodiment, when the proximity sensor 1716 detects that the distance between the user and the front of the terminal 1700 is gradually decreasing, the processor 1701 controls the display screen 1705 to switch from the screen-on state to the screen-off state; when the proximity sensor 1716 detects that the distance between the user and the front of the terminal 1700 is gradually increasing, the processor 1701 controls the display screen 1705 to switch from the screen-off state to the screen-on state.

[0281] Those skilled in the art will understand that Fig.17 The structure shown in the figure does not constitute a limitation on the terminal 1700, and the terminal 1700 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0282] The embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the image processing method provided in the above embodiment.

[0283] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a terminal, enables the terminal to execute the image processing method provided in the above embodiment.

[0284] Fig.18 Schematic diagram of a server structure provided in an embodiment of the present application. The server may be a server in a backend server cluster. Specifically:

[0285] The server 1800 includes a central processing unit (CPU) 1801, a system memory 1804 including a random access memory (RAM) 1802 and a read-only memory (ROM) 1803, and a system bus 1805 connecting the system memory 1804 and the central processing unit 1801. The server 1800 also includes a basic input / output system (I / O system) 1806 that helps transfer information between various components within the computer, and a mass storage device 1807 for storing an operating system 1813, application programs 1814, and other program modules 1815.

[0286] The basic input / output system 1806 includes a display 1808 for displaying information and an input device 1809 such as a mouse and a keyboard for user inputting information. The display 1808 and the input device 1809 are connected to the central processing unit 1801 through an input / output controller 1810 connected to the system bus 1805. The basic input / output system 1806 may also include an input / output controller 1810 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1810 also provides output to a display screen, a printer, or other types of output devices.

[0287] The mass storage device 1807 is connected to the central processing unit 1801 through a mass storage controller (not shown) connected to the system bus 1805. The mass storage device 1807 and its associated computer-readable media provide non-volatile storage for the server 1800. That is, the mass storage device 1807 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.

[0288] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1804 and mass storage device 1807 can be collectively referred to as memory.

[0289] According to various embodiments of the present application, the server 1800 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1800 can be connected to the network 1812 through the network interface unit 1811 connected to the system bus 1805, or the network interface unit 1811 can be used to connect to other types of networks or remote computer systems (not shown).

[0290] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU. The one or more programs include instructions for performing the image processing method provided in the embodiment of the present application.

[0291] The embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the server, the server can execute the image processing method provided by the above embodiment.

[0292] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a server, enables the server to execute the image processing method provided in the above embodiment.

[0293] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0294] The above description is only a preferred embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining objects appearing in a region of interest of a target image; Based on a first mapping relationship, determining a category of an object appearing in the region of interest, and based on a second mapping relationship, determining an image processing mode, wherein the first mapping relationship indicates a correspondence between an object and a category, and the second mapping relationship indicates a correspondence between an object category and an image processing mode; or, based on a third mapping relationship, determining the image processing mode, wherein the third mapping relationship indicates a correspondence between an object and an image processing mode; at least two of the different objects correspond to different image processing modes; Processing the region of interest based on the image processing mode; displaying the processed region of interest; Wherein, when the object appearing in the region of interest is a lesion region type object, the image processing mode includes an electronic magnification mode, and the electronic magnification mode is used to magnify the region of interest; or, When the object appearing in the region of interest is a microvascular object, the image processing mode includes a multispectral imaging mode, and the multispectral imaging mode is used to perform enhancement processing on different bands in the region of interest; or When the object appearing in the region of interest is an intestinal / intestinal object, the image processing mode includes a contrast enhancement mode and a highlight suppression mode, wherein the contrast enhancement mode is used to perform contrast enhancement processing on the region of interest, and the highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest; or, When the object appearing in the region of interest is a device-type object, the image processing mode includes a highlight suppression mode; or, When the object appearing in the region of interest is a gauze-like object, the image processing mode includes a false color highlight mode, and the false color highlight mode is used to perform false color display processing on the object appearing in the region of interest; or When the object appearing in the region of interest is a smoke-like object, the image processing mode includes a defogging processing mode, and the defogging processing mode is used to perform defogging processing on the smoke-like object appearing in the region of interest.

2. The method according to claim 1, characterized in that When the object appearing in the region of interest is a smoke-type object, processing the region of interest based on the image processing mode includes: Determine the concentration of smoke-like objects present in the region of interest; If the concentration of the smoke object exceeds a concentration threshold, increasing the degree of defogging of the region of interest; If the concentration of the smoke object is lower than the concentration threshold, the degree of defogging performed on the region of interest is reduced.

3. The method according to claim 1, characterized in that The acquiring of the objects appearing in the region of interest of the target image comprises: Identify objects appearing in the target image and location information of the objects based on a deep learning model; The region of interest is determined based on the position information of the object, where the region of interest includes at least one object identified by the deep learning model.

4. The method according to claim 3, characterized in that Identify, based on the deep learning model, that the number of objects appearing in the target image is multiple; The determining the region of interest based on the position information of the object comprises: determining a priority for each of the plurality of objects; selecting a target object from the plurality of objects based on a priority of each of the plurality of objects; The region of interest is determined based on the position information of the target object.

5. The method according to claim 1, characterized in that The acquiring of the object appearing in the region of interest in the target image comprises: Determine the region of interest based on the specified region location information; Objects appearing in the region of interest are identified based on a deep learning model.

6. The method according to claim 1, characterized in that The region of interest after display processing includes: Displaying the target image in the main screen area of ​​the display interface; The processed region of interest is displayed in a secondary screen area of ​​the display interface, and the secondary screen area is superimposed on the main screen area.

7. The method according to claim 6, characterized in that The display interface includes a plurality of auxiliary screen areas; The multiple auxiliary screen areas are used to display images processed by different image processing modes for the same region of interest, or to display images processed by different regions of interest.

8. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring objects appearing in a region of interest of a target image; a determination module, configured to determine the objects appearing in the region of interest based on a first mapping relationship, and to determine the image processing mode based on a second mapping relationship, wherein the first mapping relationship indicates a correspondence between objects and categories, and the second mapping relationship indicates a correspondence between object categories and image processing modes; or to determine the image processing mode based on a third mapping relationship, wherein the third mapping relationship indicates a correspondence between objects and image processing modes; at least two of the different objects correspond to different image processing modes; A processing module, configured to process the region of interest based on the image processing mode; A display module, used for displaying the processed region of interest; Wherein, when the object appearing in the region of interest is a lesion region type object, the image processing mode includes an electronic magnification mode, and the electronic magnification mode is used to magnify the region of interest; or, When the object appearing in the region of interest is a microvascular object, the image processing mode includes a multispectral imaging mode, and the multispectral imaging mode is used to perform enhancement processing on different bands in the region of interest; or When the object appearing in the region of interest is an intestinal / intestinal object, the image processing mode includes a contrast enhancement mode and a highlight suppression mode, wherein the contrast enhancement mode is used to perform contrast enhancement processing on the region of interest, and the highlight suppression mode is used to suppress the brightness of the highlight part in the region of interest; or, When the object appearing in the region of interest is a device-type object, the image processing mode includes a highlight suppression mode; or, When the object appearing in the region of interest is a gauze-like object, the image processing mode includes a false color highlight mode, and the false color highlight mode is used to perform false color display processing on the object appearing in the region of interest; or When the object appearing in the region of interest is a smoke-like object, the image processing mode includes a defogging processing mode, and the defogging processing mode is used to perform defogging processing on the smoke-like object appearing in the region of interest.

9. The device according to claim 8, characterized in that The determining module comprises: a first determining unit, configured to determine the category of the object appearing in the region of interest based on a first mapping relationship, and determine the image processing mode based on a second mapping relationship, wherein the first mapping relationship indicates a correspondence between the object and the category, and the second mapping relationship indicates a correspondence between the object category and the image processing mode; or determine the image processing mode based on a third mapping relationship, wherein the third mapping relationship indicates a correspondence between the object and the image processing mode; Wherein, when the object appearing in the region of interest is a smoke-like object, the processing module includes: A second determining unit is used to determine the concentration of smoke-like objects appearing in the region of interest; A first judgment unit, configured to increase the degree of defogging of the region of interest if the concentration of the smoke object exceeds a concentration threshold; If the concentration of the smoke object is lower than the concentration threshold, reducing the degree of defogging of the region of interest; Wherein, the determination module further includes: A first recognition unit, configured to recognize objects appearing in the target image and location information of the objects based on a deep learning model; A third determining unit, configured to determine the region of interest based on the position information of the object, wherein the region of interest includes at least one object identified by the deep learning model; wherein the number of objects appearing in the target image is identified as multiple based on the deep learning model; The third determining unit is used for: determining a priority for each of the plurality of objects; selecting a target object from the plurality of objects based on a priority of each of the plurality of objects; Determine the region of interest based on the location information of the target object; Wherein, the determination module includes: A fourth determining unit, configured to determine the region of interest based on the specified region location information; A second recognition unit, configured to recognize objects appearing in the region of interest based on a deep learning model; Wherein, the display module comprises: A display unit, configured to display the target image in a main screen area of ​​a display interface; The display unit is used to display the processed region of interest in the auxiliary screen area of ​​the display interface, and the auxiliary screen area is superimposed on the main screen area; Wherein, the display interface includes a plurality of auxiliary screen areas; The multiple auxiliary screen areas are used to display images processed by different image processing modes for the same region of interest, or to display images processed by different regions of interest.

10. A computer device, characterized in that: The device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the steps of the method described in any one of claims 1 to 7.

Images