Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment. The image processing method comprises the following steps: acquiring a plurality of endoscope images corresponding to an image processing instruction; according to the image processing instruction, the display position, the interest area feature and the display size of each endoscope image are obtained; cutting each endoscope image according to the features of the region of interest and the display size to obtain a plurality of first cut images; and displaying the plurality of first cut images on the display interface according to the display position.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as computer vision, and more particularly to an image processing method and electronic device. Background Art

[0002] Endoscopic systems are widely used in modern medicine to examine and diagnose internal tissues of the human body. Existing endoscopy technology mainly relies on a variety of imaging modes, such as visible light images, fluorescence images, infrared images, narrow-band imaging images, etc. These imaging modes provide doctors with richer diagnostic evidence through different spectral information, which helps to identify lesions and their details.

[0003] Current endoscope systems usually use a single image display mode, that is, one imaging mode image (such as visible light or fluorescence image) is displayed on the main window at a time. In this mode, images of other imaging modes, such as infrared images, narrow-band imaging images, etc., are usually displayed in the form of small windows.

[0004] However, since only one mode of image can be displayed in the main window at a time, images of other modes can only be displayed in small windows, which makes it impossible to comprehensively compare the details of images of multiple modes at the same time, resulting in weak diagnostic support capabilities of the endoscope system for multi-modal images. Summary of the invention

[0005] In order to solve at least one of the above technical problems, the present disclosure provides an image processing method and an electronic device.

[0006] According to one aspect of the present disclosure, there is provided an image processing method, comprising: acquiring a plurality of endoscopic images corresponding to the image processing instructions; acquiring the display position, the feature of the region of interest and the display size of each endoscopic image respectively according to the image processing instruction; Cropping each endoscopic image according to the feature of the region of interest and the display size to obtain a plurality of first cropped images; The plurality of first cropped images are displayed on a display interface according to the display positions.

[0007] According to the image processing method of at least one embodiment of the present disclosure, the step of acquiring the display position, the feature of the region of interest and the display size of each endoscopic image according to the image processing instruction comprises: Acquire a split-screen mode corresponding to the image processing instruction and a feature of a region of interest of each endoscopic image; The display position and display size of each endoscopic image are respectively acquired according to the split-screen mode.

[0008] According to the image processing method of at least one embodiment of the present disclosure, when the split screen mode is primary and secondary split screen, respectively acquiring the display position and display size of each endoscopic image according to the split screen mode includes: determining a primary display image and at least one secondary display image among the plurality of endoscopic images according to the image processing instruction; Acquire a main display area and at least one secondary display area corresponding to the split-screen mode, where the number of the secondary display areas is the same as the number of the secondary display images; The main display image is associated with the main display area according to the image processing instruction, and the at least one secondary display image is associated with the at least one secondary display area, respectively, to obtain the display position and display size of each endoscopic image.

[0009] According to the image processing method of at least one embodiment of the present disclosure, when the split screen mode is contrast split screen, respectively acquiring the display position and display size of each endoscopic image according to the split screen mode includes: obtaining the number of images of the plurality of endoscopic images; Dividing the display area corresponding to the display interface according to the number of images to obtain a plurality of sub-areas; The multiple endoscopic images are respectively associated with the multiple sub-areas to obtain a display position and a display size of each endoscopic image.

[0010] According to the image processing method of at least one embodiment of the present disclosure, the acquiring of a plurality of endoscopic images corresponding to the image processing indication comprises: Acquire multi-source image data through multi-mode imaging equipment; storing the multi-source image data in a clock synchronization buffer; A plurality of endoscopic images synchronized with the clock are read from the clock synchronization buffer according to the image processing instruction.

[0011] According to at least one embodiment of the present disclosure, the image processing method, before storing the multi-source image data in the clock synchronization buffer, further includes: storing the multi-source image data in an input buffer; The storing the multi-source image data in the clock synchronization cache includes: reading the multi-source image data from the input cache and then writing the data into the clock synchronization cache.

[0012] According to the image processing method of at least one embodiment of the present disclosure, the reading of the multi-source image data from the input buffer includes: Performing integrity judgment on the multi-source image data stored in the input buffer to obtain a judgment result; When the judgment result is complete, the multi-source image data is read from the input buffer.

[0013] According to at least one embodiment of the present disclosure, the image processing method further includes, after displaying the plurality of first cropped images: In response to movement indications of the plurality of endoscopic images, acquiring a plurality of moved images corresponding to the movement indications; Each moved image is cropped according to the feature of the region of interest and the display size of each endoscopic image to obtain a plurality of second cropped images; The plurality of second cropped images are displayed on a display interface according to the display positions.

[0014] According to at least one embodiment of the present disclosure, the image processing method further includes, after displaying the plurality of first cropped images: In response to the switching instructions of the plurality of endoscopic images, respectively acquiring a post-switching position, a post-switching feature, and a post-switching size of each endoscopic image according to the switching instructions; cropping each endoscopic image according to the switched features and the switched sizes to obtain a plurality of third cropped images; The plurality of third cropped images are displayed on a display interface according to the switched positions.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the image processing method of any one embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 The process of an image processing method according to an embodiment of the present invention is Figure 1 .

[0018] Figure 2 yes Figure 1 A flowchart of a parameter acquisition method in the image processing method shown.

[0019] Figure 3 yes Figure 2 The flowchart of the primary and secondary screen splitting method in the parameter acquisition method shown.

[0020] Figure 4 yes Figure 2The flowchart of the comparative screen splitting method in the parameter acquisition method shown.

[0021] Figure 5 yes Figure 1 The process of the image acquisition method in the image processing method shown Figure 1 .

[0022] Figure 6 yes Figure 1 The process of the image acquisition method in the image processing method shown Figure 2 .

[0023] Figure 7 yes Figure 5 A flowchart of a data reading method in an image acquisition method is shown.

[0024] Figure 8 The process of an image processing method according to an embodiment of the present invention is Figure 2 .

[0025] Fig. 9 The process of an image processing method according to an embodiment of the present invention is Figure 3 .

[0026] Fig.10 yes Figure 1 A flowchart of an image cropping method in the image processing method shown.

[0027] Fig.11 is a schematic flowchart of an image processing method according to an embodiment of the present disclosure.

[0028] Fig.12 yes Fig.11 A schematic diagram of primary and secondary split-screen display in the image processing method shown.

[0029] Fig.13 yes Fig.11 Schematic diagram of the image processing method for comparing split screen display Figure 1 .

[0030] Fig.14 yes Fig.11 Schematic diagram of the image processing method for comparing split screen display Figure 2 .

[0031] Fig.15 yes Fig.11 Schematic diagram of the image processing method for comparing split screen display Figure 3 .

[0032] Fig.16 yes Fig.11 Schematic diagram of the image processing method for comparing split screen display Figure 4 .

[0033] Fig.17 It is a schematic block diagram of the structure of an image processing device according to one embodiment of the present disclosure.

[0034] Fig.18 The present invention is a block diagram showing a structure of an electronic device equipped with an image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0036] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] The endoscope collects image information from the user's body or other parts through the multi-mode imaging device at its front end, converts the collected optical signals into electrical signals, transmits them to the processing unit for processing and adjustment, and finally displays them through a monitor for doctors or operators to view. In complex diagnostic processes, the endoscope system may need to display images in multiple modes at the same time (such as visible light and fluorescent images at the same time), usually displaying other images in a small window, which are generally zoomed secondary views, and displaying the main image of current focus in the main window. Since only one image can be displayed in the main window at a time, images in other modes are displayed in small windows, and the small windows may compress the images or reduce their size. The small windows display fewer details, which makes it impossible to comprehensively compare the details of images in multiple modes at the same time, resulting in weak diagnostic support capabilities of the endoscope system for multi-modal images.

[0038] To this end, the present disclosure proposes an image processing method and electronic device, which crops each endoscopic image according to the characteristics of the region of interest and the display size, obtains multiple first cropped images and then displays them, thereby realizing the processing of the endoscopic image. When a user needs to process an endoscopic image, an image processing instruction can be sent to an electronic device such as a server, and the server and other electronic devices execute the corresponding endoscopic image processing process after receiving the image processing instruction. In particular, the server and other electronic devices can be specifically a video processing unit in the endoscope system, or a video processing structure outside the endoscope system, such as a video processing server outside the endoscope system.

[0039] For the convenience of description and to make the technical solutions of the specific embodiments of the present disclosure easier to understand, before describing the image processing method implemented in the present disclosure, the technical terms involved in the specific embodiments of the present disclosure are defined as follows: Primary and Secondary Split Screen is a technology that displays two or more different contents on the same screen at the same time, with one screen area being the primary screen and one or more being the secondary screens. The primary screen is used to display the most important and core information, usually occupying a larger display area; the secondary screen is used to display auxiliary information or secondary data, usually occupying a smaller area.

[0040] Split Screen Comparison is a technique for displaying two or more images, videos or data with different perspectives, contents or versions on the same screen at the same time. Each screen area of ​​the split screen comparison usually has the same weight and importance.

[0041] Package alignment refers to ensuring the consistency of the spatial position and orientation of multiple sensors when they are packaged or installed in a multi-sensor system.

[0042] The master-slave mode refers to organizing and communicating multiple sensors in a master-slave relationship. It is mainly used to coordinate the work between multiple sensors to achieve unified data collection and processing.

[0043] Figure 1 FIG. 2 is a schematic diagram showing the overall flow of an image processing method M100 according to an embodiment of the present disclosure. Figure 1 The method shown includes steps S110 to S140. The method can be executed by an electronic device such as a server.

[0044] Specifically, Figure 1 The methods shown include: Step S110, acquiring a plurality of endoscopic images corresponding to the image processing instructions.

[0045] In some embodiments of the present disclosure, step S110 can acquire multiple endoscopic images through a multi-mode imaging device of an endoscope system, and the multiple endoscopic images can be images of different modes. The image processing instruction in step S110 can carry split-screen control parameters, including split-screen mode, identification and coordinates of multiple endoscopic images to be split, etc.; the above split-screen control parameters can be manually input by the user. In order to improve the efficiency of parameter input, coordinate suggestions corresponding to different numbers of images in different modes can also be pre-set, so that the user can determine the final split-screen control parameters according to the coordinate suggestions. The split-screen control parameters can include, for example, primary and secondary split screens, main image ID, main image coordinates, first image ID, first image coordinates, second image ID, second image coordinates, etc.

[0046] In some embodiments of the present disclosure, the multiple endoscopic images acquired through step S110 may include at least two of visible light images, fluorescent images, infrared images, narrow-band imaging images, combined light images, and pseudo-color images.

[0047] Step S120, obtaining the display position, region of interest features and display size of each endoscopic image respectively according to the image processing instruction.

[0048] In some embodiments of the present disclosure, step S120 can determine the specific display position of a certain endoscopic image, which part of the endoscopic image the user is interested in, and the size of the endoscopic image. The region of interest features of multiple endoscopic images can be the same or different. The region of interest feature in step S120 can be color, shape, etc.

[0049] Step S130 , cropping each endoscopic image according to the features of the region of interest and the display size to obtain a plurality of first cropped images.

[0050] In some embodiments of the present disclosure, each endoscopic image is cropped separately in step S130, so that unnecessary parts of the endoscopic image can be removed and only the parts of interest are retained.

[0051] Step S140: displaying a plurality of first cropped images on a display interface according to display positions.

[0052] In some embodiments of the present disclosure, multiple first cropped images can be displayed on the display interface of the endoscope system through step S140. The process of displaying multiple first cropped images through step S140 can be a direct image replacement process, that is, replacing the current display image on the display interface with multiple first cropped images according to the display position; or it can be a process of displaying after image splicing, that is, splicing multiple first cropped images according to the display position to obtain a spliced ​​image; and displaying the spliced ​​image on the display interface.

[0053] The image processing method provided by the present disclosure crops and displays multiple endoscopic images, so that the display interface can simultaneously display the details of multiple mode images, thereby realizing the simultaneous display of multiple modes and different types of images, and improving the ease of comparison of different types of images; this method solves the problem in the prior art that only one mode of image can be displayed in the main window at a time, and the images of other modes can only be displayed in the form of a small window, which makes it impossible to comprehensively compare the details of multiple modes of images at the same time, resulting in the weak diagnostic support ability of the endoscope system for multi-mode images. In addition, this method simultaneously displays multiple modes and different types of images without the need for image switching, and can also solve the problem in the prior art that switching different image types requires operations such as handle buttons, and the operation method is cumbersome.

[0054] Regarding step S120, in some embodiments of the present disclosure, it may include the following steps: Figure 2 Steps S121 to S122 are shown.

[0055] Step S121, obtaining the split screen mode corresponding to the image processing instruction and the feature of the region of interest of each endoscopic image.

[0056] In some embodiments of the present disclosure, the split screen mode in step S121 may include primary and secondary split screen, contrast split screen, parallel split screen, etc. The interest region feature obtained in step S121 is information used to describe the characteristics of the user's interest region; the interest region feature may be a color feature, a texture feature, a shape feature, an edge feature, etc.

[0057] Step S122, respectively obtaining the display position and display size of each endoscopic image according to the split-screen mode.

[0058] In some embodiments of the present disclosure, for different split-screen modes, the distribution of multiple endoscopic images is different. For example, when the screen is split into primary and secondary, one endoscopic image among the multiple endoscopic images is used as the primary image, and the other endoscopic images are used as secondary images. When there are multiple secondary images, the display space of different secondary spaces can be the same, or different secondary images can be divided according to importance to ensure that important secondary images have more display space, while less important secondary images occupy less space. The layout of the primary and secondary split screens can be a multi-column layout (the primary image is distributed on one side, and the secondary images are distributed in multiple columns on the other side), a grid layout (the primary image is placed in a group of grids, and the secondary images are distributed in at least another group of grids), etc., and the image processing instruction can control which layout method is used.

[0059] When contrasting split screen, the display area sizes corresponding to multiple endoscopic images are usually the same. Users can associate important images with focus areas (areas that users often pay attention to based on their usage habits) and other images with other areas according to their usage habits. They can also associate multiple endoscopic images with multiple display areas at will. The display method of contrasting split screen can be two-dimensional grid split screen, that is, the screen is divided into multiple two-dimensional grids, each grid displays a different endoscopic image; it can also be side-to-side split screen, that is, the screen is divided into two or more vertical or horizontal areas, each area displays a different endoscopic image, etc.

[0060] The split screen control parameters carried by the image processing instruction can be used to control which split screen mode is used and which layout mode of the split screen mode is used for display. Different layout modes can be flexibly selected according to the number of endoscopic images to be displayed, the display purpose, etc.

[0061] Step S121 and step S122 perform display control through the split-screen mode, so that the user only needs to distribute the image to different areas without accurately setting the coordinates of the image, thereby lowering the technical threshold for use; and the split-screen mode allows the user to quickly adjust the layout and size of different images, thereby achieving flexible layout.

[0062] Regarding step S122, when the split screen mode is primary and secondary split screen, in some embodiments of the present disclosure, it may include the following: Figure 3 Steps S1221 to S1223 are shown.

[0063] Step S1221, determining a main display image and at least one secondary display image among a plurality of endoscopic images according to an image processing instruction.

[0064] In some embodiments of the present disclosure, step S1221 can determine which endoscopic image is to be used as the main display image, which endoscopic image or images are to be used as secondary display images, and the relative positions of multiple secondary display images when multiple secondary display images are included based on the split-screen control parameters carried by the image processing indication, the identifier of the endoscopic image, and the coordinates of the endoscopic image.

[0065] Step S1222, obtaining a main display area and at least one secondary display area corresponding to the split-screen mode.

[0066] In some embodiments of the present disclosure, the number of secondary display areas in step S1222 is the same as the number of secondary display images in step S1221. When the number of secondary display areas in step S1222 is greater than 1, the sizes of different secondary display areas may be the same or different.

[0067] Step S1223, according to the image processing instruction, the main display image is associated with the main display area, and at least one secondary display image is associated with at least one secondary display area, so as to obtain the display position and display size of each endoscopic image.

[0068] In some embodiments of the present disclosure, when associating an endoscopic image, if the endoscopic image is a primary display image, the endoscopic image can be directly associated with the primary display area to obtain the display position and display size of the endoscopic image; if the endoscopic image is a secondary display image, the secondary display image can be randomly associated with one of the unassociated secondary display areas, or the secondary display image can be associated with a secondary display area based on its importance, etc. Different secondary display images can be associated with different secondary display areas.

[0069] The primary and secondary split-screen control is achieved through steps S1221 to S1223, so that the user can view the primary and secondary images at the same time without switching windows, and the main image can be highlighted, thereby improving information accessibility.

[0070] Regarding step S122, when the split-screen mode is a comparison split-screen, in some embodiments of the present disclosure, it may include steps S1224 to S1226 as Figure 4 shown.

[0071] Step S1224, obtaining the number of images of multiple endoscopic images.

[0072] In some embodiments of the present disclosure, the number of images obtained through step S1224 is a positive integer greater than or equal to 2, and this number of images is related to the mode of the multimodal imaging device and the image processing instruction; that is, this number of images is less than or equal to the number of modes of the multimodal imaging device. The modes of the multimodal imaging device may include: multiple imaging modes such as white light imaging mode, fluorescence imaging mode, narrowband imaging mode, infrared imaging mode, ultrasonic imaging mode, spectral imaging mode, and polarization imaging mode. Assuming that the multimodal imaging device has 4 modes and the image processing instruction requires obtaining all images, then the number of images obtained through step S1224 is also 4; if the image processing instruction requires obtaining white light images and fluorescence images, then the number of images obtained through step S1224 is 2.

[0073] Step S1225, dividing the display area corresponding to the display interface according to the number of images to obtain multiple sub-areas.

[0074] In some embodiments of the present disclosure, through step S1225, the display area can be evenly divided horizontally or vertically, or the display area can be evenly divided according to a two-dimensional grid, etc.; for example, if the display area includes 120 grids and the number of images is 4, then 30 grids can be used as a group to divide the display area into a "field" grid.

[0075] Step S1226, respectively associating multiple endoscopic images with multiple sub-areas to obtain the display position and display size of each endoscopic image.

[0076] In some embodiments of the present disclosure, for a certain endoscopic image, through step S1226, the endoscopic image can be randomly associated with one of the unassociated sub-areas, or the endoscopic image can be associated with the sub-area according to importance, and different endoscopic images are associated with different sub-areas.

[0077] By implementing the comparison split-screen control through steps S1224 to S1226, the differences between multiple images can be intuitively compared, and the image comparison efficiency can be improved.

[0078] Regarding step S110, in some embodiments of the present disclosure, it may include steps S111 to S113 as Figure 5 shown.

[0079] Step S111, acquiring multi-source image data through a multi-mode imaging device.

[0080] In some embodiments of the present disclosure, the multi-source image data acquired by step S111 may include at least two of visible light image data, fluorescent image data, infrared image data, narrowband imaging image data, combined light image data, and pseudo-color image data. In particular, in order to enable the multi-mode imaging device to simultaneously display images of the same part, the multi-channel sensors of the multi-mode imaging device in step S111 can be packaged and aligned to make the captured image field of view consistent; in order to ensure data consistency, the multi-channel sensors of the multi-mode imaging device in step S111 can also be configured in master-slave mode so that the image data output by the multi-mode imaging device can be synchronized in time. Through time and space synchronization alignment, it is possible to ensure that the display frame rate is consistent with the data source frame rate, ensure smooth images, avoid adverse effects such as image tearing or mosaics caused by frame loss, and improve image processing efficiency.

[0081] Step S112: storing the multi-source image data in a clock synchronization buffer.

[0082] Step S113, reading a plurality of clock-synchronized endoscopic images from the clock synchronization buffer according to the image processing instruction.

[0083] Steps S111 to S113 synchronize and align multiple endoscopic images through clock synchronization cache to ensure the consistency of each endoscopic image, thereby improving the reliability of image processing.

[0084] Furthermore, if Figure 6 As shown, before step S112, step S110 may further include: Step S114: storing the multi-source image data in an input buffer.

[0085] At this time, step S112 specifically includes reading the multi-source image data from the input buffer and writing the data into the clock synchronization buffer.

[0086] In some embodiments of the present disclosure, the input buffer in step S114 may be a large-capacity storage device, such as a double data rate synchronous dynamic random access memory (DDR), a phase change memory (PCM), etc. When reading multi-source image data in step S112, in order to increase data processing speed and reduce latency, the multi-source image data in the input buffer may be read out in units of lines using a clock higher than the write rate.

[0087] In some embodiments of the present disclosure, the input cache in step S114 can optimize the endoscopic image by reducing delay and response speed, and the consistency of the endoscopic image can be further improved by the input cache.

[0088] Regarding step S112, in some embodiments of the present disclosure, it may include the following steps: Figure 7 Steps S1121 to S1122 are shown.

[0089] Step S1121, performing integrity judgment on the multi-source image data stored in the input buffer to obtain a judgment result.

[0090] In some embodiments of the present disclosure, when the judgment result obtained by step S1121 is that the multi-source image data is complete (including complete frames), step S1122 is executed; when the judgment result obtained by step S1121 is that the multi-source image data is incomplete (excluding complete frames), data can continue to be stored through the input cache, and step S1121 can be continuously judged until the judgment result is complete or a preset termination condition is reached, and the preset termination condition can be a condition such as the number of judgments or the judgment time.

[0091] Step S1122, reading multi-source image data from the input buffer.

[0092] Step S1121 and step S1122 read the multi-source image data from the input cache when the multi-source image data is complete, thereby avoiding inter-frame crosstalk and screen tearing caused by inconsistent read and write rates, improving data consistency, reliability and security, reducing errors and anomalies, and improving overall performance and user experience.

[0093] Furthermore, if Figure 8 As shown, the image processing method provided by the present disclosure may further include, after step S140: Step S150 , in response to the movement instructions of the plurality of endoscopic images, acquiring a plurality of moved images corresponding to the movement instructions.

[0094] In some embodiments of the present disclosure, after viewing the displayed image, the user can move the multi-mode imaging device of the endoscope system to control the image movement. When the multi-mode imaging device is moved, the multiple moved images are the images after the multiple endoscopic images are moved.

[0095] Step S160 , cropping each moved image according to the feature of the region of interest and the display size of each endoscopic image to obtain a plurality of second cropped images.

[0096] In some embodiments of the present disclosure, the method of cropping the moved image in step S160 is similar to the method of cropping the multiple endoscopic images in step S130.

[0097] Step S170: display a plurality of second cropped images on the display interface according to the display positions.

[0098] In some embodiments of the present disclosure, the manner of displaying the plurality of second cropped images in step S170 is similar to the manner of displaying the plurality of first cropped images in step S140.

[0099] Through step S150 to step S170, the position of the displayed image can be fine-tuned by simply moving the multi-mode imaging device, thereby achieving flexible adjustment and display of the image, making the operation simpler, more intuitive and efficient.

[0100] Furthermore, if Fig. 9 As shown, the image processing method provided by the present disclosure may further include, after step S140: Step S180, in response to the switching instructions of the plurality of endoscopic images, respectively acquiring the switched position, switched feature and switched size of each endoscopic image according to the switching instructions.

[0101] In some embodiments of the present disclosure, after viewing the displayed image, the user can adjust the control parameters according to usage needs, and adjust the displayed features, display position, display size, etc. through the control parameters.

[0102] Step S190, cropping each endoscopic image according to the switched features and the switched sizes to obtain a plurality of third cropped images.

[0103] In some embodiments of the present disclosure, the manner of cropping the image in step S190 is similar to the manner of cropping the plurality of endoscopic images in step S130.

[0104] Step S191, displaying a plurality of third cropped images on the display interface according to the switched positions.

[0105] In some embodiments of the present disclosure, the manner of displaying the plurality of third cropped images through step S191 is similar to the manner of displaying the plurality of first cropped images through step S140.

[0106] Through steps S180 to S191, the type, position and size of the displayed image can be switched by simply adjusting the parameters, thereby achieving flexible switching of the image display, making the operation simpler, more intuitive and more efficient.

[0107] For any endoscopic image, regarding step S130, in some embodiments of the present disclosure, the following may be included: Fig.10 Steps S131 to S134 are shown.

[0108] Step S131, identifying a feature region corresponding to the feature of the region of interest from the endoscopic image.

[0109] In some embodiments of the present disclosure, the region of interest feature of the endoscopic image can be matched with the endoscopic image through step S131, and the position of the region of interest feature in the endoscopic image and the corresponding feature region can be determined according to the matching result.

[0110] Step S132, determining whether the size of the feature area is larger than the display size.

[0111] In some embodiments of the present disclosure, when it is determined by step S132 that the size of the feature area is not larger than the display size, step S133 is executed; when it is determined by step S132 that the size of the feature area is larger than the display size, step S134 is executed.

[0112] Step S133 , cutting out an image of a display size including the characteristic area from the endoscopic image.

[0113] In some embodiments of the present disclosure, taking the size of the feature area as 300 unit areas and the size of the display size as 500 unit areas as an example, an image of 500 unit areas including the feature area of ​​300 unit areas can be cropped from the endoscopic image. The 500 unit areas can be obtained by uniformly expanding the periphery of the 300 unit areas, or by expanding the 300 unit areas in one or several directions.

[0114] Step S134, cropping the characteristic region from the endoscopic image and reducing the characteristic region according to the display size.

[0115] In some embodiments of the present disclosure, taking the size of the feature area as 300 unit areas and the size of the display size as 150 unit areas as an example, after the feature area of ​​300 unit areas is cropped from the endoscopic image, the feature area is reduced to 150 unit areas. The process of obtaining the first cropped image through step S134 may include: calculating the reduction scale factor 300 / 150=2, and then reducing the image of the 300 feature area based on the scale factor. The reduction process can be implemented by algorithms such as bilinear interpolation, bicubic interpolation, and Lancelot interpolation; in particular, in order to ensure the quality of the image as much as possible, image processing processes such as sharpening and anti-aliasing can also be combined in the reduction process.

[0116] The image processing method provided by the present invention can meet the doctor's flexible observation of different types of images (such as white light images, fluorescent images, combined light images, pseudo-color images, etc.) of the same affected area during surgery, which is convenient for doctors to compare details more clearly and accurately locate the affected area, making the treatment of lesions during surgery more accurate and convenient, thereby improving the accuracy and safety of the surgery.

[0117] Fig.11 A schematic flowchart of the image processing method according to the present invention is shown.

[0118] Fig.11 In the schematic flow chart shown, the image processing method includes: 1. Receive control parameters sent by the user through the host computer according to actual needs.

[0119] 2. Acquire multi-source image data through multi-mode imaging equipment.

[0120] 3. Store multi-source image data in the input buffer.

[0121] 4. Read the multi-source image data from the input buffer and write it into the clock synchronization buffer.

[0122] 5. Read multiple clock-synchronized endoscopic images from a clock synchronization cache according to image processing instructions.

[0123] 6. Analyze the control parameters to obtain the split-screen mode, display position, interest area characteristics and display size.

[0124] 7. Crop each endoscopic image according to the features of the region of interest and the display size to obtain a plurality of first cropped images.

[0125] 8. Replace the current data of the display interface with the multiple first cropped images according to the display position.

[0126] In some embodiments of the present disclosure, this step can be used to replace the current data at corresponding positions of the display interface with corresponding first cropped images according to the coordinates, thereby realizing the display of multiple types of images.

[0127] Through the above steps, it is convenient for users to watch different types of images of the same scene. The display position can be changed by simply moving the multi-mode imaging device. No complicated operation is required and the operation is simple. In addition, the display position and display size of multiple types of images can be adjusted by controlling parameters to meet the needs of different users.

[0128] In some embodiments of the present disclosure, the display position can be controlled by coordinates. Taking multi-source image data including white light image data, fluorescent image data, combined light image data, pseudo-color image data and narrow-band image data as an example, the multiple endoscopic images obtained may include white light images, fluorescent images, combined light images and pseudo-color images. When the split screen mode used is primary and secondary split screen, the four vertex coordinates (1,1), (480,1), (1,360) and (480,360) of the first cropped image corresponding to the secondary display image 1, i.e. the white light image, can be obtained, and the size is 480*360; the secondary display image 2 That is, the four vertex coordinates of the first cropped image corresponding to the fluorescent image are (1,361), (480, 361), (1,720) and (480, 720), and the size is 480*360; the four vertex coordinates of the first cropped image corresponding to the secondary display image 3, i.e. the pseudo-color image, are (1,721), (480, 721), (1,1080) and (480, 1080), and the size is 480*360; the remaining area of ​​the display interface 1920*1080 is the main display image, i.e. the combined light image. The image displayed by the above process can be as follows Fig.12 shown.

[0129] Taking the acquisition of multiple endoscopic images including white light images and fluorescence images as an example, when the split-screen mode adopted is contrast split-screen, the four vertex coordinates of the first cropped image corresponding to the first display image, i.e., the white light image, are (1,1), (960, 1), (1, 1080), (960, 1080), and the size is 960*1080; the four vertex coordinates of the first cropped image corresponding to the second display image, i.e., the fluorescence image, are (961, 1), (961, 1080), (1920, 1), (1920, 1080), and the size is 960*1080. The features of the first display image and the second display image displayed by the above process can be the same or different. Taking the original endoscopic image including two parts A and B as an example, the white light image is image type 1, the fluorescence image is image type 2, the two regions of interest have the same features, and the region of interest features correspond to part A in the endoscopic image, the schematic diagram of displaying multiple first cropped images can be as follows Fig.13 As shown; when the features of the two regions of interest are the same and the features of the regions of interest correspond to part B in the endoscopic image, a schematic diagram showing multiple first cropped images can be as follows Fig.14 As shown; when the features of the two regions of interest are different, the features of the region of interest of image type 1 correspond to part A in the endoscopic image, and the features of the region of interest of image type 2 correspond to part B in the endoscopic image, the schematic diagram showing multiple first cropped images can be as follows Fig.15As shown; when the features of the two regions of interest are different, the features of the region of interest of image type 1 correspond to part B in the endoscopic image, and the features of the region of interest of image type 2 correspond to part A in the endoscopic image, the schematic diagram showing multiple first cropped images can be as follows Fig.16 shown.

[0130] In particular, in order to enable different types of images to be stitched into the original complete image based on the cropped images after being cropped, the stitching position of the cropped images should be the same as that of the original images. For example, when the original image includes four parts, ABCD, and the multiple endoscopic images include four types, part A can be cropped from the first type, part B can be cropped from the second type, part C can be cropped from the third type, and part D can be cropped from the fourth type, and the four parts, ABCD, can be stitched in the order of the original image.

[0131] Different types of images with different features in the area of ​​interest can be displayed on the same screen, and images of different types and fields of view can be spliced ​​into a complete image. This can meet the user's flexible needs for different types of images during use and improve the flexibility of image processing.

[0132] The display mode of the contrast split screen can be 1:1 left and right as shown in the figure above, or 1:1 top and bottom. When the number of endoscopic images is greater than 2, the area size corresponding to each split screen mode is also the same; for example, when the number of endoscopic images is 3, the screen is split in a ratio of 1:1:1.

[0133] Based on any of the above implementations, the present disclosure also provides an image processing device (corresponding to an image processing method).

[0134] Fig.17 It is a schematic block diagram of the structure of an image processing device according to one embodiment of the present disclosure.

[0135] like Fig.17 As shown, the image processing device comprises: The image acquisition module 110 is used to acquire a plurality of endoscopic images corresponding to the image processing instructions.

[0136] The parameter acquisition module 120 is used to respectively acquire the display position, the feature of the region of interest and the display size of each endoscopic image according to the image processing instruction.

[0137] The image cropping module 130 is used to crop each endoscopic image according to the characteristics of the region of interest and the display size to obtain a plurality of first cropped images.

[0138] The image display module 140 is used to display a plurality of first cropped images on a display interface according to display positions.

[0139] The above modules can be implemented by computer program modules.

[0140] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0141] The execution subject of the image processing method in the specific implementation manner of the present disclosure may be an electronic device such as a server.

[0142] Therefore, based on any of the above embodiments, the present disclosure further provides an electronic device, which can execute the image processing method of any of the above embodiments of the present disclosure, and the image processing device of any of the above embodiments can be configured on the electronic device.

[0143] Fig.18 FIG. 1 is a schematic block diagram of the structure of an electronic device 1000 equipped with an image processing device according to an embodiment of the present disclosure.

[0144] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0145] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0146] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0147] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0148] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0149] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0154] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0155] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. An image processing method, characterized in that: include: acquiring a plurality of endoscopic images corresponding to the image processing instructions; acquiring the display position, the feature of the region of interest and the display size of each endoscopic image respectively according to the image processing instruction; Cropping each endoscopic image according to the feature of the region of interest and the display size to obtain a plurality of first cropped images; as well as The plurality of first cropped images are displayed on a display interface according to the display positions.

2. The image processing method according to claim 1, characterized in that: The obtaining, according to the image processing instruction, the display position, the feature of the region of interest and the display size of each endoscopic image respectively comprises: Acquire the split screen mode corresponding to the image processing instruction and the feature of the region of interest of each endoscopic image; and The display position and display size of each endoscopic image are respectively acquired according to the split-screen mode.

3. The image processing method according to claim 2, characterized in that: When the split screen mode is primary and secondary split screen, respectively obtaining the display position and display size of each endoscopic image according to the split screen mode includes: determining a primary display image and at least one secondary display image among the plurality of endoscopic images according to the image processing instruction; Acquire a main display area and at least one secondary display area corresponding to the split-screen mode, wherein the number of the secondary display areas is the same as the number of the secondary display images; and The main display image is associated with the main display area according to the image processing instruction, and the at least one secondary display image is associated with the at least one secondary display area, respectively, to obtain the display position and display size of each endoscopic image.

4. The image processing method according to claim 2, characterized in that: When the split screen mode is contrast split screen, respectively acquiring the display position and display size of each endoscopic image according to the split screen mode includes: obtaining the number of images of the plurality of endoscopic images; Dividing the display area corresponding to the display interface according to the number of images to obtain a plurality of sub-areas; and The multiple endoscopic images are respectively associated with the multiple sub-areas to obtain a display position and a display size of each endoscopic image.

5. The image processing method according to any one of claims 1 to 4, characterized in that: The acquiring of a plurality of endoscopic images corresponding to the image processing instructions comprises: Acquire multi-source image data through multi-mode imaging equipment; storing the multi-source image data in a clock synchronization buffer; and A plurality of endoscopic images synchronized with the clock are read from the clock synchronization buffer according to the image processing instruction.

6. The image processing method according to claim 5, characterized in that: Before storing the multi-source image data in the clock synchronization buffer, the method further includes: storing the multi-source image data in an input buffer; and The storing the multi-source image data in the clock synchronization cache includes: reading the multi-source image data from the input cache and then writing the data into the clock synchronization cache.

7. The image processing method according to claim 6, characterized in that: The reading the multi-source image data from the input buffer comprises: Performing integrity judgment on the multi-source image data stored in the input buffer to obtain a judgment result; and When the judgment result is complete, the multi-source image data is read from the input buffer.

8. The image processing method according to any one of claims 1 to 4, characterized in that: After displaying the plurality of first cropped images, the method further includes: In response to movement indications of the plurality of endoscopic images, acquiring a plurality of moved images corresponding to the movement indications; cropping each moved image according to the feature of the region of interest and the display size of each endoscopic image to obtain a plurality of second cropped images; and The plurality of second cropped images are displayed on a display interface according to the display positions.

9. The image processing method according to any one of claims 1 to 4, characterized in that: After displaying the plurality of first cropped images, the method further includes: In response to the switching instructions of the plurality of endoscopic images, respectively acquiring a post-switching position, a post-switching feature, and a post-switching size of each endoscopic image according to the switching instructions; cropping each endoscopic image according to the switched features and the switched sizes to obtain a plurality of third cropped images; and The plurality of third cropped images are displayed on a display interface according to the switched positions.

10. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the image processing method according to any one of claims 1 to 9.

Images