Image processing method, device, electronic device, and computer-readable storage medium

Abstract

The present application provides an image processing method, device, electronic device, computer-readable storage medium, and computer program product, the image processing method including: performing collection processing on an object using various narrowband filters to obtain narrowband channel images including the object; and performing fusion processing on multiple narrowband channel images that correspond one-to-one to the various types of narrowband filters to obtain a color image including the contours of the object.

Description

[Technical Field]

[0001] This application is based on and claims priority from a Chinese patent application bearing application number 202210114314.9 and filed on January 30, 2022. The entire contents of the Chinese patent application are incorporated herein by reference.

[0002] This application relates to graphic image processing technology, and in particular to image processing methods, devices, electronic equipment, and computer-readable storage medium On the body Regarding. [Background technology]

[0003] With the advancement of image processing technology, electronic devices can produce richer and more vivid images. Compared with text, images convey richer and more expressive information, and are therefore more popular with users.

[0004] By collecting images of an object using an electronic device, the object is included in the collected images. The position of the object is identified using the collected images, and subsequent operations are performed based on the position of the object. For example, in the case of a pathology image, the pathological site is quickly identified using the pathology image, and subsequent accurate pathology sampling is performed based on the pathological site.

[0005] However, in the related art, the image collected by the electronic device cannot clearly distinguish between the target object and other non-target objects, which reduces the imaging effect of the image. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide an image processing method, an apparatus, an electronic device, and the like, which can effectively display an object and improve the imaging effect of an image. and computer-readable storage medium Body provide. [Means for solving the problem]

[0007] The technical solution of the embodiment of the present application is realized as follows.

[0008] An embodiment of the present application is an image processing method executed by an electronic device including multiple narrow-band filters with different wavelength bands through which optical signals pass, the image processing method comprising: collecting and processing a photosensitive signal, which is an optical signal of an object, using each of the multiple narrow-band filters to obtain a narrow-band channel image including the object; performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; Including fruit, When the number of types of narrow-band filters is at least four, performing a fusion process on a plurality of narrow-band channel images corresponding one-to-one to the various types of narrow-band filters to obtain a color image including the contour of the object, Identifying any three types of narrow band filters from the multiple types of narrow band filters; performing a synthesis process on the narrowband channel images corresponding to the three types of narrowband filters to obtain a candidate color image; performing an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images; determining the candidate color image with the most edge features as a color image including the contour of the object; Including An image processing method is provided.

[0009] An embodiment of the present application includes an acquisition module configured to acquire a narrowband channel image including the object by acquiring a photosensitive signal, which is an optical signal of the object, using each of a variety of narrowband filters having different wavelength bands through which the optical signal passes, and a fusion module configured to acquire a color image including the contour of the object by performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters. and when there are at least four types of narrowband filters, the fusion module is configured to perform a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object. The fusion module identifies any three types of narrowband filters among the various types of narrowband filters, performs a synthesis process on the narrowband channel images corresponding to the any three types of narrowband filters to obtain candidate color images, performs an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images, and determines the candidate color image with the most edge features as the color image including the contour of the object. An image processing device is provided.

[0010] An embodiment of the present application provides an electronic device including a memory that stores computer-executable commands, and a processor that, when executing the computer-executable commands stored in the memory, realizes the image processing method provided by the embodiment of the present application.

[0011] An embodiment of the present application includes a housing forming a partially sealed space, multiple narrowband filters located on a photosensitive chip inside the housing for outputting photosensitive signals and collecting optical signals, and a processor located inside the housing and receiving the photosensitive signals output by the photosensitive chip, the processor generating narrowband channel images including the object based on the photosensitive signals, which are optical signals of the object collected by the multiple narrowband filters, and performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the multiple narrowband filters, to obtain a color image including the contour of the object. and when there are at least four types of narrowband filters, the processor performs a fusion process on a plurality of narrowband channel images that correspond one-to-one to the various types of narrowband filters, and when acquiring a color image including the contour of the object, identifies any three types of narrowband filters from the various types of narrowband filters, performs a synthesis process on the narrowband channel images that correspond to the arbitrary three types of narrowband filters, acquires candidate color images, performs an edge detection process on each of the candidate color images, acquires edge features of each of the candidate color images, and determines the candidate color image with the most edge features as the color image including the contour of the object. Provide electronic devices.

[0012] An embodiment of the present application is a computer-readable storage medium storing a program executed by a computer including multiple types of narrowband filters with different wavelength bands through which optical signals pass, the program causing the computer to execute the steps of: collecting and processing photosensitive signals, which are optical signals of an object, using each of the multiple types of narrowband filters to obtain narrowband channel images including the object; and performing a fusion process on multiple narrowband channel images that correspond one-to-one to the multiple types of narrowband filters to obtain a color image including the contour of the object. death , When there are at least four types of narrowband filters, the step of performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object includes the steps of: identifying any three types of narrowband filters among the various types of narrowband filters; performing a synthesis process on the narrowband channel images corresponding to the arbitrary three types of narrowband filters to obtain candidate color images; performing an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images; and selecting the candidate color image with the most edge features as the color image including the contour of the object. A computer-readable storage medium is provided. [Effects of the Invention]

[0014] The embodiments of the present application have the following beneficial effects:

[0015] By collecting and processing the object using a narrowband filter, the object can be collected accurately, and narrowband channel images corresponding to various types of narrowband filters are fused to obtain a color image including the contours of the object, so that the color image collected by the electronic device can clearly distinguish between the object and non-object in the color image, thereby effectively displaying the object and improving the imaging effect of the image. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic diagram of an X-ray radiographic image provided by the related art. [Figure 2] FIG. 2 is an architecture diagram of an image processing system provided by an embodiment of the present application. [Figure 3] FIG. 3 is a structural schematic diagram of an image processing device provided by an embodiment of the present application. [Figure 4] FIG. 4 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. [Figure 5A] FIG. 5A is a flowchart of an image processing method provided by an embodiment of the present application. [Figure 5B] FIG. 5B is a flowchart of an image processing method provided by an embodiment of the present application. [Figure 6] FIG. 6 is a structural schematic diagram of a short-wave infrared camera provided in an embodiment of the present application. [Figure 7] FIG. 7 is a schematic diagram of a composite color image provided by an embodiment of the present application. [Figure 8] FIG. 8 is a schematic diagram of a composite color image provided by an embodiment of the present application. [Figure 9A] FIG. 9A is a schematic diagram of a bracket-type imaging system provided by an embodiment of the present application. [Figure 9B] FIG. 9B is a schematic diagram of a handheld imaging system provided by an embodiment of the present application. [Figure 10A] FIG. 10A is a schematic diagram of a normal lens provided in an embodiment of the present application. [Figure 10B] FIG. 10B is a schematic diagram of an infrared apochromatic lens provided in an embodiment of the present application. [Figure 11A] FIG. 11A is a schematic diagram of the constant lighting mode provided by an embodiment of the present application. [Figure 11B] FIG. 11B is a schematic diagram of the flash mode provided by the embodiment of the present application. [Figure 12A] FIG. 12A is a schematic diagram of a focus ring provided in an embodiment of the present application. [Figure 12B]FIG. 12B is a schematic diagram of a liquid lens provided in an embodiment of the present application. [Figure 13] FIG. 13 is a comparison diagram between a synthetic infrared color image taken by a narrowband infrared color camera provided in an embodiment of the present application and a color image taken by a normal color camera. DETAILED DESCRIPTION OF THE INVENTION

[0017] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the drawings. The described embodiments should not be considered as limitations on the present application, and any other embodiments obtained by those skilled in the art without any creative efforts belong to the protection scope of the present application.

[0018] In the following description, the terms "first / second" used herein are used only to distinguish between similar objects and do not indicate a particular order for the objects. Note that "first / second" can be used interchangeably with a particular order or a sequence of events where permitted, and the embodiments of the present application described herein can be performed in an order other than that shown or described.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art of the present application. The terminology used herein is for the purpose of describing the embodiments of the present application, and is not intended to limit the present application.

[0020] Before describing the examples of the present application in more detail, the nouns and technical terms used in the examples of the present application will be explained. The following interpretations apply to the nouns and technical terms used in the examples of the present application.

[0021] (1) "In response to..." means that the operation to be performed is in response to a condition or state on which the operation depends. When the dependent condition or state is satisfied, one or more operations may be performed in real time or with a predetermined delay. Unless otherwise specified, the order in which the operations are performed is not limited.

[0022] (2) "Pathological images" are images that show the pathological morphology of organs, tissues, or cells in the body. Based on pathological images, it is possible to investigate the cause of lesions, the mechanism of disease onset, and the progression of lesions. Pathological images include slice images of lesions (images created by cutting out a certain size of lesion tissue to observe changes in the lesion), endoscopic images, etc.

[0023] (3) "Narrow-band filter," also known as a narrow-band optical filter, is an optical filter that is a subdivision of a bandpass filter. A narrow-band optical filter allows optical signals in a specific wavelength band to pass, while blocking optical signals on either side of this wavelength band. The passband of a narrow-band optical filter is relatively narrow, less than 5% of the center wavelength value, and the passband half-width of a narrow-band optical filter is less than 10 nanometers.

[0024] In related technology, accurate determination of tumor margins during cancer resection surgery (e.g., conservative surgery) allows for complete removal of the tumor area, preventing recurrence of the patient's condition and avoiding reoperation. Postoperative pathological analysis of tissue is the standard for tumor diagnosis. Tissues removed during a patient's surgery are cut into tissue blocks of appropriate volume, which are then selected by a physician and fixed using methods such as formalin immersion. Subsequently, the tissue blocks undergo a series of processes, including dehydration, embedding, slicing, and staining, ultimately producing pathological sections that can be observed under a microscope. The process by which a physician selects tissue blocks is particularly important in obtaining accurate information about the patient's lesions. If tissue blocks containing lesions are omitted from selection, the physician's ability to make more accurate judgments is limited. Selecting a large number of tissue blocks significantly increases the workload for sectioning, reducing medical efficiency.

[0025] The main criteria for physicians to detect tumor margins during surgery and select pathological samples after surgery can be divided into two main aspects. In hospitals with limited medical resources, physicians primarily rely on naked-eye observation and palpation to identify tumor areas and select tissue blocks. This method is extremely difficult for inexperienced physicians, especially when the tumor bed is hidden. A pathologist's visual inspection alone cannot distinguish between normal tissue areas and lesions, and palpation is highly subjective. Alternatively, radiographic methods, such as optical imaging platforms, can be used to identify lesions and assist in pathological sampling. These platforms provide X-ray images of the excised tissue (as shown in Figure 1, the strip 101 is a marker pin for marking the location), helping physicians locate lesions more accurately during surgery and aiding in pathological sampling during biopsy. However, interpretation of X-ray images primarily depends on the physician's experience, resulting in a certain degree of subjectivity and disagreement among physicians. Also, X-ray equipment is not cheap.

[0026] To solve the above problems, embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can effectively display objects and improve the imaging effect of images.

[0027] The image processing method provided in the embodiments of the present application may be realized by a terminal alone, or may be realized by a terminal and a server working together. For example, the terminal alone executes the image processing method described below, or the terminal sends an image collection request (including narrowband channel images of the object) to the server, and the server executes the image processing method according to the received image collection request, and performs fusion processing based on narrowband channel images corresponding to multiple narrowband filters to obtain a color image including the contour of the object.

[0028] The electronic device provided by the embodiments of the present application may be various types of terminals or servers with imaging capabilities. Here, the server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Here, the cloud service may be an image processing service for terminals to make calls. The terminal may be, but is not limited to, a smart camera, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart voice interaction device, an in-vehicle terminal, etc. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application is not limited thereto.

[0029] Please refer to Fig. 2. Fig. 2 shows an image processing system provided by an embodiment of the present application. 10 1 is an architecture diagram of a terminal 200 connected to a server 100 via a network 300. Here, the network 300 may be a wide area network or a local area network, or a combination of both.

[0030] In some embodiments, the electronic device ImagingThe image processing method provided in the embodiments of the present application is implemented by the terminal 200. For example, a user performs an acquisition process on an object using a narrowband filter in the terminal 200 to obtain a narrowband channel image (also referred to as a single channel image) containing the object, and then performs a fusion process based on the narrowband channel images corresponding to the various narrowband filters to obtain a color image containing the contours of the object. For example, for a pathological image, the user performs an acquisition process on pathological tissue using a narrowband filter in the terminal 200 to obtain a narrowband channel image containing the pathological tissue, and then performs a fusion process based on the narrowband channel images corresponding to the various narrowband filters to obtain a color image (i.e., a pathological image) containing the contours of the pathological tissue. By quickly identifying the pathological site using the pathological image, subsequent accurate pathological sampling can be performed based on the pathological site.

[0031] In some embodiments, the image processing method provided by the embodiments of the present application may be implemented by a terminal and a server working together. For example, a user performs an acquisition process on an object using a narrowband filter in the terminal 200 to obtain a narrowband channel image containing the object, and the terminal 200 transmits the narrowband channel image containing the object to the server 100. The server 100 performs a fusion process based on the narrowband channel images corresponding to various narrowband filters to obtain a color image containing the contours of the object, which is then transmitted to the terminal 200, and the terminal 200 displays the color image. For example, for a pathological image, the terminal 200 performs an acquisition process on pathological tissue using a narrowband filter to obtain a narrowband channel image containing the pathological tissue, which is then transmitted to the server 100. The server 100 performs a fusion process based on the narrowband channel images corresponding to various narrowband filters to obtain a color image containing the contours of the pathological tissue (i.e., a pathological image), which is then transmitted to the terminal 200 for display. This allows the pathological site to be quickly identified using the pathological image, and subsequent accurate pathological sampling can be performed based on the pathological site.

[0032] In some embodiments, a terminal or a server can execute a computer program to implement the image processing method provided by the embodiments of the present application. For example, the computer program may be a native program or software module in an operating system, a native application (APP) that is, a program that can be installed and executed in an operating system, a mini program that is, a program that can be downloaded to a browser environment and executed, or an applet that can be incorporated into any APP. In short, the computer program may be any type of application program, module, or plug-in.

[0033] The embodiments of this application are realized using cloud technology. Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology based on the application of the cloud computing business model. It forms a resource pool that can be used as needed, providing flexibility and convenience. Cloud computing technology is an important pillar. The back-end services of technical network systems require a large amount of computing and storage resources.

[0034] In some embodiments, multiple servers may constitute a blockchain, and the server 100 may be a node of the blockchain. Information connections may exist between nodes in the blockchain, and information can be transmitted between nodes through the information connections. Here, data related to the image processing method provided by the embodiments of the present application (e.g., image processing logic, color images) is stored in the blockchain, and the accuracy of the data in the blockchain is maintained because the blockchain has the property of being tamper-proof.

[0035] Please refer to FIG. 3. FIG. 3 is a structural schematic diagram of an image processing device provided by an embodiment of the present application, and will be described by taking as an example that the device is a terminal with imaging functions. The image processing device shown in FIG. 3 includes at least one processor 510, a memory 550, and at least one network interface 520. The components of the image processing device 500 are coupled by a bus system 540. The bus system 540 is used to realize communication connections between these components. The bus system 540 includes a data bus, as well as a power bus, a control bus, and a status signal bus. However, for clarity, various buses are shown as the bus system 540 in FIG. 3.

[0036] The processor 510 may be, for example, a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or other integrated circuit chip having signal processing capabilities, where the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0037] Memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 550 may include, for example, one or more storage devices that are physically remote from processor 510.

[0038] The memory 550 may include volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 550 described in the embodiments of the present application may include any suitable type of memory.

[0039] In some embodiments, memory 550 may store data to support various operations, including, for example, programs, modules, and data structures, or a subset or superset thereof, as generally described below.

[0040] The operation system 551 includes system programs for processing various basic system services, such as a frame layer, a core library layer, and a driver layer, and for executing hardware-related tasks, and is used to realize various basic services and process hardware-based tasks.

[0041] The network communication module 552 connects to other electronic devices via one or more (wired or wireless) network interfaces 520, example network interfaces 520 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0042] The presentation module 553 can present information via one or more output devices (e.g., a display, a speaker, etc.) associated with the user interface 530 (e.g., peripheral devices, displayed content, user interface for manipulating information).

[0043] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and to interpret the detected inputs or interactions.

[0044] In some embodiments, the image processing device provided by the embodiments of the present application is implemented in a software manner. Figure 3 shows an image processing device 555 stored in memory 550. The image processing device 555 may be software in the form of a program, plug-in, or the like, and includes an acquisition module 5551 and a fusion module 5552. These modules are logical, so they can be arbitrarily combined or further divided according to the functions to be implemented. The functions of each module are described below.

[0045] The hardware structure of the electronic device provided by the embodiment of the present application will be described below.

[0046] Please refer to Fig. 4. Fig. 4 is a schematic diagram of the hardware structure of electronic device 40 provided by an embodiment of the present application, which includes a housing 41 forming a partially sealed space, a variety of narrowband filters 42 located on a photosensitive chip inside housing 41 for outputting photosensitive signals and collecting the optical signals, and a processor 43 located inside housing 41 for receiving the photosensitive signals, which are optical signals of an object output by the photosensitive chip and collected by the variety of narrowband filters, and generating narrowband channel images including the object based on the photosensitive signals, and performing a fusion process on the multiple narrowband channel images that correspond one-to-one to the variety of narrowband filters 42, thereby obtaining a color image including the contours of the object.

[0047] By collecting and processing the object using a narrowband filter, the object can be collected accurately, and narrowband channel images corresponding to various types of narrowband filters are fused to obtain a color image including the contours of the object, so that the color image collected by the electronic device can clearly distinguish between the object and non-object in the color image, thereby effectively displaying the object and improving the imaging effect of the image.

[0048] Here, the electronic device 30 may be any electronic device with an imaging function, such as a mobile phone or a camera. Here, the housing 41 of the electronic device 40 may be a bracket-type housing (including a bracket 904 as shown in FIG. 9A), which can fix the electronic device and prevent it from shaking. The housing 41 of the electronic device 40 may be a handheld housing (including a handle 905 as shown in FIG. 9B), which can be used to hold the electronic device, making it convenient for quick photography. Here, the photosensitive chip may be an infrared photosensitive chip or a normal light photosensitive chip. The infrared photosensitive chip has a higher photosensitivity effect than the normal light photosensitive chip.

[0049] The narrowband channel images exhibit two dimensional characteristics: the first dimensional characteristic is the difference in absorption peaks for different narrowband light wave spectra of the target (e.g., pathological tissue), and the second dimensional characteristic is the difference in penetration depth of light waves within the range of these absorption peaks into the target. Therefore, the contrast effect of the narrowband channel images is remarkable.

[0050] For example, with a 1450 nanometer (nm) filter, the contours of the object stand out and a narrowband channel image with a light transmission wavelength of 1450 nm can be obtained; with a 1300 nm filter, the contours of the object stand out and a narrowband channel image with a light transmission wavelength of 1300 nm can be obtained; and with a 1050 nm filter, the contours of the object stand out and a narrowband channel image with a light transmission wavelength of 1050 nm can be obtained.

[0051] The size of each narrowband filter matches the size of a pixel on the photosensitive chip (i.e., the size of a pixel on the photosensitive chip is equal to or greater than the size of each narrowband filter). For example, if the side length of a pixel on the photosensitive chip is 5 micrometers, the side length of the small section of each narrowband filter may be 5 micrometers, and if the size of a pixel on the photosensitive chip is 8 micrometers x 8 micrometers, the size of each narrowband filter may be 8 micrometers x 8 micrometers. This ensures that there is one narrowband filter for each pixel on the photosensitive chip.

[0052] In some embodiments, the various narrow-band filters are regularly arranged on the photosensitive chip in a matrix format, and the processor further generates an image including the object based on the photosensitive signal, performs channel division processing on the image including the object based on the various narrow-band filters on the photosensitive chip to obtain intermediate channel images corresponding to the various narrow-band filters, and performs interpolation processing on the intermediate channel images corresponding to the various narrow-band filters to obtain narrow-band channel images corresponding to the various narrow-band filters.

[0053] For example, the following technical solution can be implemented to perform channel division processing on an image including an object based on various narrow-band filters in the above-mentioned photosensitive chip, and obtain intermediate channel images corresponding to the various narrow-band filters: Each For a narrowband filter, the position of the narrowband filter on the photosensitive chip is identified, and based on the position of the narrowband filter on the photosensitive chip, pixel extraction processing of the corresponding channel is performed on an image including the object to obtain the pixels of the narrowband filter, and based on the arrangement of the narrowband filter on the photosensitive chip, array processing is performed on the pixels of the narrowband filter to obtain an intermediate channel image corresponding to the narrowband filter.

[0054] As shown in FIG. 6, the filter matrix 603 includes various narrowband filters (1450 nm narrowband filters, 1300 nm narrowband filters, and 1050 nm narrowband filters shown in FIG. 6), and the filter matrix 603 is repeatedly arranged on the photosensitive chip in a 2×2 cycle to collect regularly arranged narrowband channel images.

[0055] 8, a customized filter matrix 801 includes a variety of narrowband filters (a 1450nm narrowband filter 802, a 1300nm narrowband filter 803, a 1050nm narrowband filter 804, and a wideband filter 806). The 1450nm narrowband filter 802 is used as an example. Pixels corresponding to the 1450nm channel covered by the 1450nm narrowband filter 802 are extracted from an image containing an object as pixels corresponding to the 1450nm filter. Based on the arrangement of the 1450nm narrowband filter on the photosensitive chip, an array process is performed on the pixels of the 1450nm narrowband filter to obtain an intermediate channel image corresponding to the 1450nm filter. For example, if the 1450 nm narrowband filter is positioned at (0,2), (0,4), (0,6), and (0,8) on the sensitive chip, the extracted pixels are arranged at (0,2), (0,4), (0,6), and (0,8) to obtain the intermediate channel image 805 corresponding to the 1450 nm filter.

[0056] As an example, we will continue to use the 1450 nm narrow band filter as shown in Figure 8. The empty pixels in the intermediate channel image 805 are interpolated, i.e., the empty pixels are assigned pixel values ​​that are the pixel values ​​of the non-empty pixels in the intermediate channel image 805.

[0057] For example, taking a 1450nm narrowband filter, a 1300nm narrowband filter, and a 1050 narrowband filter as examples, for an image containing an object collected, the pixels covered by the 1450nm filter are extracted from the image containing the object to become the corresponding pixels of the 1450nm filter, and based on the arrangement of the narrowband filters on the photosensitive chip, the pixels of the narrowband filters are subjected to array processing to obtain an intermediate channel image corresponding to the corresponding 1450nm filter, and the 1450nm intermediate channel image is subjected to interpolation processing to obtain a 1450nm narrowband channel image (i.e., a narrowband channel image with an optical transmission wavelength of 1450nm). Pixels covered by the 1300 nm filter are extracted from the image including the object to be pixels corresponding to the 1300 nm filter, and an alignment process is performed on the pixels of the narrow band filter based on the arrangement of the narrow band filters on the photosensitive chip to obtain an intermediate channel image corresponding to the 1300 nm filter, and the 1300 nm intermediate channel image is interpolated to obtain a 1300 nm narrow band channel image (i.e., a narrow band channel image with an optical transmission wavelength of 1300 nm). Pixels covered by the 1050 nm filter are extracted from the image including the object to be pixels corresponding to the 1050 nm filter, and an alignment process is performed on the pixels of the narrow band filter based on the arrangement of the narrow band filters on the photosensitive chip to obtain an intermediate channel image corresponding to the 1050 nm filter, and the 1050 nm intermediate channel image is interpolated to obtain a 1050 nm narrow band channel image (i.e., a narrow band channel image with an optical transmission wavelength of 1050 nm).

[0058] In some embodiments, the plurality of narrowband channel images have the same size. The plurality of narrowband channel images are obtained by performing a channel decomposition process on an image including an object. The processor further comprises: eachFor each pixel, determine the pixel values ​​of the corresponding pixels in each narrowband channel image, perform a synthesis process on the pixel values ​​of the corresponding pixels in the multiple narrowband channel images to obtain a multi-channel pixel value of the pixel, and perform a stitching process on the multi-channel pixel values ​​of the multiple pixels to obtain a color image including the contour of the object. According to the embodiment of the present application, the multiple pixel values ​​of the same pixel can be effectively synthesized to improve the imaging effect of the color image.

[0059] As an example, the following description will be given in conjunction with FIG. 8. The multiple narrowband channel images have the same size. The multiple narrowband channel images are obtained by performing channel decomposition processing on an image (original image) containing an object. In other words, the multiple narrowband channel images have the same size as the original image. each Let us take an example where the pixel in (1) is located at the position (0,0) in the original image. Pixel values ​​corresponding to the pixel at the position (0,0) in the multiple narrowband channel images are identified, i.e., multiple pixel values ​​are identified. The synthesis process is an average calculation process of the multiple pixel values. The average result is the multi-channel pixel value corresponding to the pixel at the position (0,0). Based on the above method, multi-channel pixel values ​​corresponding to pixels at positions such as (0,1) are obtained. The multi-channel pixel values ​​corresponding to pixels at all positions in the original image are stitched together to obtain the final color image.

[0060] For example, taking a 1450 nm narrowband filter, a 1300 nm narrowband filter, and a 1050 nm narrowband filter as examples, the narrowband channel images include a narrowband channel image with a light transmission wavelength of 1450 nm, a narrowband channel image with a light transmission wavelength of 1300 nm, and a narrowband channel image with a light transmission wavelength of 1050 nm, and the narrowband channel image with a light transmission wavelength of 1450 nm, the narrowband channel image with a light transmission wavelength of 1300 nm, and the narrowband channel image with a light transmission wavelength of 1050 nm are the same size. A pixel value corresponding to each identical pixel in each narrowband channel image is identified, a synthesis process is performed on the pixel values ​​corresponding to each identical pixel in the multiple narrowband channel images to obtain a multi-channel pixel value of the identical pixel, and the multi-channel pixel values ​​of the pixels are stitched according to the pixel position to obtain a color image including the contour of the object, thereby synthesizing the narrowband channel image with a light transmission wavelength of 1450 nm, the narrowband channel image with a light transmission wavelength of 1300 nm, and the narrowband channel image with a light transmission wavelength of 1050 nm into a color image.

[0061] In some embodiments, the electronics further includes an achromatic lens disposed within the housing for focusing the narrowband waves collected by the various narrowband filters onto the photosensitive chip, such that the narrowband waves can be collected by the electronics at a later stage to collect narrowband channel images.

[0062] 4, the electronic device 40 includes an achromatic lens 44 (e.g., an infrared apochromatic lens), where the achromatic lens 44 ensures that several narrow-band wavelengths are clearly focused on the surface of the photosensitive chip at the same time, resulting in higher image clarity.

[0063] The achromatic lens 44 can be a normal lens, which does not have the achromatic function, i.e., cannot ensure that several narrow-band wavelengths are clearly focused on the surface of the photosensitive chip at the same time.

[0064] In some embodiments, the achromatic lens further includes a focus ring mounted on the surface of the achromatic lens for adjusting the focal length of the achromatic lens, or a liquid lens mounted on the achromatic lens for adjusting the focal length of the achromatic lens.

[0065] As shown in Fig. 4, the electronic device 40 includes an achromatic lens 44 (e.g., an infrared apochromatic lens). As shown in Fig. 12A, a focus ring 1201 on the infrared apochromatic lens is manually rotated to focus the lens. As shown in Fig. 12B, an additional liquid lens 1202 is attached to the infrared apochromatic lens, and in combination with electronics, the liquid lens 1202 can be used to search for the focal plane, thereby achieving autofocus of the lens.

[0066] In some embodiments, the electronic device further includes a window disposed inside the housing for protecting the photosensitive chip and the processor, wherein the type of window includes at least one of a long-pass filter, a polarizer, and an attenuator.

[0067] As shown in Figure 4, the electronic device 40 includes a window 45. 45 may be a long-pass filter for filtering visible light (e.g., an infrared long-pass filter), a polarizer for collecting polarized light, or an attenuator for attenuating incident light, etc.

[0068] In some embodiments, the electronic device further includes a light source device disposed inside the housing for providing at least one light source, the light source type including at least one of a broadband light source and a narrowband light source, and an illumination mode of the light source device including at least one of a steady mode and a flash mode.

[0069] 4, the light source provided by the light source device 46 may be a broadband light source or multiple narrowband light sources that are turned on simultaneously. Since the electronic device of the embodiment of the present application employs narrowband filters, it is not necessary to separately turn on light sources of different wavelengths and capture images to obtain multispectral information. Instead, all light source devices can be turned on simultaneously, or a broadband light source can be directly used for illumination, and then filtered by a matrix of narrowband filters to obtain a multispectral image.

[0070] For example, when the illumination mode of the light source device is set to the constant-on mode, the brightness of the constant-on mode can be maintained for a long period of time, which is suitable for observing samples in real time and recording videos. When the illumination mode of the light source device is set to the flash mode, the flash mode can burst extremely large light source power in a very short period of time, which is suitable for collecting dynamic samples or scenes with strong ambient light. In the flash mode, the camera only needs to select a very small shutter speed (approximately 1 / 100,000 s (seconds)) to ensure the collection of blur-free images.

[0071] As described above, the image processing method provided by the embodiment of the present application is implemented by electronic equipment, and the electronic equipment includes various narrow-band filters. Please refer to Figure 5A. Figure 5A is a flowchart of the image processing method provided by the embodiment of the present application. This will be described in conjunction with the steps shown in Figure 5A.

[0072] The electronic device may be any electronic device with an imaging function, such as a mobile phone or a camera. Narrow-band filters are optical filters that are a subdivision of bandpass filters, allowing optical signals in a specific wavelength band to pass and blocking optical signals on either side of this wavelength band. Examples include 980 nm narrow-band filters, 1000 nm narrow-band filters, 1050 nm narrow-band filters, 1064 nm narrow-band filters, and 1080 nm narrow-band filters.

[0073] In step 101, an acquisition process is performed on the object using various narrowband filters to obtain narrowband channel images containing the object.

[0074] For example, a narrow-band channel image obtained by performing acquisition processing on an object using a narrow-band filter can clearly distinguish the object from other non-objects, and can display the object effectively enough.

[0075] In some embodiments, the electronic device includes a photosensitive chip, and the various narrowband filters are regularly arranged on the photosensitive chip in a matrix format. In step 101, performing an acquisition process on the object using the various narrowband filters to obtain narrowband channel images including the object can be achieved by: performing an image acquisition process on the object using the photosensitive chip to obtain an image including the object; performing a channel splitting process on the image including the object based on the various narrowband filters on the photosensitive chip to obtain intermediate channel images corresponding to the various narrowband filters; and performing an interpolation process on the intermediate channel images corresponding to the various narrowband filters to obtain narrowband channel images corresponding to the various narrowband filters. Embodiments of the present application can accurately obtain narrowband channel images corresponding to the various narrowband filters.

[0076] As shown in Figure 6, the filter matrix contains various narrowband filters (1450 nm narrowband filter, 1300 nm narrowband filter, and 1050 nm narrowband filter shown in Figure 6), and the filter matrix is ​​repeatedly arranged on the photosensitive chip in a 2 × 2 cycle to collect regularly arranged narrowband channel images.

[0077] 8, the customized filter matrix 801 includes a variety of narrowband filters (a 1450nm narrowband filter 802, a 1300nm narrowband filter 803, a 1050nm narrowband filter 804, and a wideband filter 806). The 1450nm narrowband filter 802 is taken as an example. The pixels of the 1450nm channel covered by the 1450nm narrowband filter 802 are extracted from the image containing the object as the pixels corresponding to the 1450nm filter. Based on the arrangement of the 1450nm narrowband filter on the photosensitive chip, the pixels of the 1450nm narrowband filter are subjected to array processing to obtain an intermediate channel image corresponding to the 1450nm filter. For example, if the 1450 nm narrowband filter is positioned at (0,2), (0,4), (0,6), and (0,8) on the sensitive chip, the extracted pixels are arranged at (0,2), (0,4), (0,6), and (0,8) to obtain the intermediate channel image 805 corresponding to the 1450 nm filter.

[0078] As an example, we will continue to use the 1450 nm narrow band filter as shown in Figure 8. The empty pixels in the intermediate channel image 805 are interpolated, i.e., the empty pixels are assigned pixel values ​​that are the pixel values ​​of the non-empty pixels in the intermediate channel image 805.

[0079] For example, taking a 1450nm narrowband filter, a 1300nm narrowband filter, and a 1050 narrowband filter as examples, for a collected image containing an object, the pixels covered by the 1450nm filter are extracted from the image containing the object and are determined as the pixels corresponding to the 1450nm filter. Based on the arrangement of the narrowband filters on the photosensitive chip, an array process is performed on the pixels of the narrowband filters to obtain an intermediate channel image corresponding to the 1450nm filter. The 1450nm intermediate channel image is then interpolated to obtain a 1450nm narrowband channel image (i.e., a narrowband channel image with an optical transmission wavelength of 1450nm). Pixels covered by the 1300 nm filter are extracted from the image including the object to be pixels corresponding to the 1300 nm filter, and an alignment process is performed on the pixels of the narrow band filter based on the arrangement of the narrow band filters on the photosensitive chip to obtain an intermediate channel image corresponding to the 1300 nm filter, and the 1300 nm intermediate channel image is interpolated to obtain a 1300 nm narrow band channel image (i.e., a narrow band channel image with an optical transmission wavelength of 1300 nm). Pixels covered by the 1050 nm filter are extracted from the image including the object to be pixels corresponding to the 1050 nm filter, and an alignment process is performed on the pixels of the narrow band filter based on the arrangement of the narrow band filters on the photosensitive chip to obtain an intermediate channel image corresponding to the 1050 nm filter, and the 1050 nm intermediate channel image is interpolated to obtain a 1050 nm narrow band channel image (i.e., a narrow band channel image with an optical transmission wavelength of 1050 nm).

[0080] In some embodiments, the channel splitting process for the image including the object based on the various narrow-band filters in the photosensitive chip to obtain the intermediate channel images corresponding to the various narrow-band filters can be realized by the following technical solution: eachFor the narrow-band filter, the position of the narrow-band filter on the photosensitive chip is identified, and a pixel extraction process is performed on an image including the object based on the position of the narrow-band filter on the photosensitive chip to obtain the pixels of the narrow-band filter, and an alignment process is performed on the pixels of the narrow-band filter based on the arrangement of the narrow-band filter on the photosensitive chip to obtain an intermediate channel image corresponding to the narrow-band filter. According to an embodiment of the present application, an accurate intermediate channel image can be obtained by aligning the pixels of the narrow-band filter using the arrangement of the narrow-band filter on the photosensitive chip.

[0081] As an example, performing pixel extraction processing on an image including an object based on the position of the narrow-band filter on a photosensitive chip and acquiring pixels of the narrow-band filter can be realized by performing pixel extraction processing on an image including an object based on the position of the narrow-band filter on a photosensitive chip and acquiring pixels of the narrow-band filter corresponding to a channel, where the channel corresponds to the narrow-band filter, thereby realizing channel division.

[0082] 8, a customized filter matrix 801 includes a variety of narrowband filters (a 1450nm narrowband filter 802, a 1300nm narrowband filter 803, a 1050nm narrowband filter 804, and a wideband filter 806). The 1450nm narrowband filter 802 is used as an example. Pixels corresponding to the 1450nm channel, which are covered by the 1450nm narrowband filter 802, are extracted from an image containing an object to become pixels corresponding to the 1450nm filter. Based on the arrangement of the 1450nm narrowband filter on the photosensitive chip, an array process is performed on the pixels of the 1450nm narrowband filter to obtain an intermediate channel image corresponding to the 1450nm filter. For example, if the 1450 nm narrowband filter is positioned at (0,2), (0,4), (0,6), and (0,8) on the sensitive chip, the extracted pixels are arranged at (0,2), (0,4), (0,6), and (0,8) to obtain the intermediate channel image 805 corresponding to the 1450 nm filter.

[0083] For example, taking a 1450 nm narrow-band filter, a 1300 nm narrow-band filter, and a 1050 nm narrow-band filter as examples, pixels of an image containing an object are extracted at the position of the narrow-band filter on the photosensitive chip, the pixels of the narrow-band filter are obtained, and the pixels of the narrow-band filter are arranged according to the arrangement of the narrow-band filter on the photosensitive chip to obtain intermediate channel images corresponding to the narrow-band filters, i.e., the narrow-band channel image with a light transmission wavelength of 1450 nm, the narrow-band channel image with a light transmission wavelength of 1300 nm, and the narrow-band channel image with a light transmission wavelength of 1050 nm.

[0084] In step 102, a fusion process is performed on a plurality of narrowband channel images that correspond one-to-one to various types of narrowband filters to obtain a color image including the contour of the object.

[0085] For example, after narrowband channel images corresponding to various types of narrowband filters are obtained, the narrowband channel images corresponding to various types of narrowband filters are synthesized to obtain a color image, so that the color image collected by the electronic device can clearly distinguish between the target object and the non-target object in the color image, thereby effectively displaying the target object and improving the imaging effect of the image.

[0086] In some embodiments, the sizes of the narrowband channel images corresponding to the various narrowband filters one-to-one are the same. Performing a fusion process on the narrowband channel images corresponding to the various narrowband filters one-to-one to obtain a color image including the contour of the object is a method of obtaining a color image including the contour of the object in an image including the object. each For each pixel, determine the pixel value of each corresponding pixel in each narrowband channel image, perform a synthesis process on the pixel values ​​of the corresponding pixels in the multiple narrowband channel images to obtain a multi-channel pixel value of the pixel, and perform a stitching process on the multi-channel pixel values ​​of the multiple pixels to obtain a color image including the contour of the object. According to the embodiment of the present application, a color image with accurate imaging effect can be obtained by synthesizing from the pixel dimension.

[0087] As an example, the following description will be given in conjunction with FIG. 8. The multiple narrowband channel images have the same size. The multiple narrowband channel images are obtained by performing channel decomposition processing on an image (original image) containing an object. In other words, the multiple narrowband channel images have the same size as the original image. each Let us take an example where the pixel in (1) is located at the position (0,0) in the original image. Pixel values ​​corresponding to the pixel at the position (0,0) in the multiple narrowband channel images are identified, i.e., multiple pixel values ​​are identified. The synthesis process is an average calculation process of the multiple pixel values. The average result is the multi-channel pixel value corresponding to the pixel at the position (0,0). Based on the above method, multi-channel pixel values ​​corresponding to pixels at positions such as (0,1) are obtained. The multi-channel pixel values ​​corresponding to pixels at all positions in the original image are stitched together to obtain the final color image.

[0088] For example, taking a 1450 nm narrowband filter, a 1300 nm narrowband filter, and a 1050 nm narrowband filter as examples, the narrowband channel images include a narrowband channel image with a light transmission wavelength of 1450 nm, a narrowband channel image with a light transmission wavelength of 1300 nm, and a narrowband channel image with a light transmission wavelength of 1050 nm, and the narrowband channel image with a light transmission wavelength of 1450 nm, the narrowband channel image with a light transmission wavelength of 1300 nm, and the narrowband channel image with a light transmission wavelength of 1050 nm are the same size. The pixel value of any one pixel corresponding to the narrowband channel images corresponding to the various types of narrowband filters is identified, a synthesis process is performed on the pixel values ​​of the multiple corresponding pixels to obtain the multi-channel pixel value of any one pixel, and the multi-channel pixel values ​​of the pixels are stitched according to the pixel position to obtain a color image including the contour of the object, thereby synthesizing the narrowband channel image with a light transmission wavelength of 1450 nm, the narrowband channel image with a light transmission wavelength of 1300 nm, and the narrowband channel image with a light transmission wavelength of 1050 nm into a color image.

[0089] In some embodiments, fusing a plurality of narrowband channel images corresponding one-to-one to a variety of narrowband filters to obtain a color image including the contour of an object can be achieved by identifying some narrowband filters among the variety of narrowband filters, and performing a fusion process on the narrowband channel images corresponding to the some narrowband filters to obtain a color image including the contour of the object. The embodiments of the present application can improve the imaging effect of color images.

[0090] For example, rather than a color image having good imaging effect obtained by combining narrowband channel images corresponding to all narrowband filters, the embodiments of the present application can obtain a color image with the best imaging effect by combining only narrowband channel images corresponding to some narrowband filters.

[0091] In some embodiments, identifying a portion of the narrowband filters among the multiple types of narrowband filters can be achieved by displaying filtering indicators of the multiple types of narrowband filters, and in response to a selection operation on the filtering indicator, specifying the narrowband filter corresponding to the selected filtering indicator as a portion of the narrowband filters.

[0092] For example, when filtering indices of a variety of narrowband filters, such as 980 nm, 1000 nm, 1050 nm, 1064 nm, 1050 nm, 1080 nm, 1300 nm, and 1450 nm, are displayed, and any three filtering indices, such as 1050 nm, 1300 nm, and 1450 nm, are manually selected, the narrowband filters corresponding to 1050 nm, 1300 nm, and 1450 nm, respectively, are selected as some of the narrowband filters, and narrowband channel images corresponding to the 1050 nm narrowband filter, 1300 nm narrowband filter, and 1450 nm narrowband filter are synthesized to obtain a color image, thereby synthesizing a color image that meets the user's requirements based on the user's selection.

[0093] In some embodiments, identifying some narrowband filters in the above-mentioned multiple types of narrowband filters can be achieved by obtaining the frequencies at which the multiple types of narrowband filters are used, sorting the multiple types of narrowband filters in descending order based on the frequencies at which the multiple types of narrowband filters are used, and determining the multiple types of narrowband filters that are arranged first in the descending order as some of the narrowband filters.

[0094] For example, the frequencies at which various narrowband filters are used are obtained, and the various narrowband filters are sorted in descending order based on the frequencies at which the narrowband filters are used, and the first three narrowband filters in the descending order are considered to be some of the narrowband filters. For example, if a 1050nm narrowband filter, a 1300nm narrowband filter, and a 1450nm narrowband filter are the first three narrowband filters in the descending order, the narrowband filters corresponding to 1050nm, 1300nm, and 1450nm, respectively, are considered to be some of the narrowband filters, and narrowband channel images corresponding to the 1050nm narrowband filter, 1300nm narrowband filter, and 1450nm narrowband filter are synthesized to obtain a color image, thereby automatically synthesizing a color image that meets the user's requirements based on the user's preferences.

[0095] In some embodiments, when there are four types of narrowband filters, performing a fusion process on multiple narrowband channel images that correspond one-to-one to the multiple types of narrowband filters to obtain a color image including the contour of the target can be achieved by identifying any three types of narrowband filters from the multiple types of narrowband filters, performing a synthesis process on the narrowband images that correspond to the any three types of narrowband filters to obtain candidate color images, performing an edge detection process on each candidate color image to obtain edge features of each candidate color image, and selecting the candidate color image with the most edge features as the color image including the contour of the target.

[0096] For example, there are many methods for synthesizing narrowband filter images, and manually trying them one by one is too tedious. Therefore, in order to improve synthesis efficiency, the embodiment of the present application performs synthesis processing on narrowband channel images corresponding to any three types of narrowband filters to obtain candidate color images, and then performs edge detection processing on all candidate color images to obtain the edge features of each candidate color image. The more edge features a candidate color image has, the more prominent the edges will be and the better the imaging effect will be. Therefore, the candidate color image with the most edge features will be the color image containing the contour of the object.

[0097] Edge detection is used to detect pixel points in an image where there is a clear change in brightness (i.e., pixel points where the change in brightness exceeds a change threshold), and a group of pixel points is represented as a contour (i.e., an edge). Image edge detection can significantly reduce the amount of data, remove irrelevant information, and retain important structural attributes of the image (i.e., edges). Here, in the edge detection process of the embodiment of this application, operators such as a first-order differential edge operator, a Roberts edge detection operator, a Prewitt edge detection operator, and a second-order differential operator are used.

[0098] Please refer to FIG. 5B. FIG. 5B is a flowchart of an image processing method provided by an embodiment of the present application. FIG. 5B shows that FIG. 5A further includes steps 103 to 104, and the electronic device further includes a wideband filter. In step 103: Wide Step 10: Perform acquisition processing on the target using a bandpass filter to obtain a wideband channel image containing the target. 4 In the step (c), an image enhancement process is performed on a color image including the contour of an object based on the broadband channel image, and an enhanced color image is obtained.

[0099] As shown in Figure 6, the filter matrix includes various narrowband filters (1450nm narrowband filter, 1300nm narrowband filter, and 1050nm narrowband filter shown in Figure 6) and wideband filter 604, and the filter matrix is ​​repeatedly arranged in a 2x2 cycle on photosensitive chip 602. The narrowband channel images include a narrowband channel image with a light transmission wavelength of 1450nm, a narrowband channel image with a light transmission wavelength of 1300nm, and a narrowband channel image with a light transmission wavelength of 1050nm, and the narrowband channel image with a light transmission wavelength of 1450nm, the narrowband channel image with a light transmission wavelength of 1300nm, the narrowband channel image with a light transmission wavelength of 1050nm, and the wideband channel image are all the same size. When a narrowband channel image with a light transmission wavelength of 1450 nm, a narrowband channel image with a light transmission wavelength of 1300 nm, and a narrowband channel image with a light transmission wavelength of 1050 nm are synthesized into a color image, the synthesized color image and the wideband channel image have the same size. The pixel value of any one pixel corresponding to the color image and the pixel value of any one pixel corresponding to the wideband channel image are identified, and the ratio between the pixel value of any one pixel corresponding to the color image and the pixel value of any one pixel corresponding to the wideband channel image is taken as the enhancement value of any one pixel, and the pixel enhancement values ​​are stitched according to the pixel position to obtain an enhanced color image.

[0100] In some embodiments, for the imaging scene of pathological sampling, collection processing is performed on pathological tissue using various narrowband filters to obtain narrowband channel pathological images containing pathological tissue, and fusion processing is performed on multiple narrowband channel pathological images that correspond one-to-one to various types of narrowband filters to obtain color pathological images containing the contours of the pathological tissue. The color pathological images can assist in the detection of lesions remaining during surgery and in post-operative pathological sampling, thereby improving the success rate of surgery and the accuracy and consistency of post-operative pathological sampling, and reducing the difficulty of pathological sampling.

[0101] An exemplary application of the embodiment of the present application in a practical application scenario will be described below.

[0102] The embodiments of the present application can be applied to various imaging scenes, for example, in the imaging scene of pathological sampling, electronic devices are used to image pathological tissue, and the imaged images are used to detect residual lesions during surgery and to assist in pathological sampling after surgery.

[0103] An example of an imaging scene for pathological sampling will be described below.

[0104] The main criteria for physicians to detect tumor margins during surgery and select pathological samples after surgery can be divided into two main aspects. In hospitals with limited medical resources, physicians primarily rely on naked-eye observation and palpation to identify tumor areas and select tissue blocks. This method is extremely difficult for inexperienced physicians, especially when the tumor bed is hidden. A pathologist's visual inspection alone cannot distinguish between normal tissue and lesions, and palpation is highly subjective. Alternatively, radiographic methods, such as optical imaging platforms, can be used to identify lesions and assist in pathological sampling. These platforms provide X-ray images of the excised tissue, helping physicians locate lesions more accurately during surgery and aiding in pathological sampling during biopsy. However, interpretation of X-ray images primarily depends on the physician's experience, resulting in a certain degree of subjectivity and discrepancies in interpretation results. Furthermore, X-ray equipment is expensive.

[0105] To address the above-mentioned challenges, the present application provides a short-wave infrared camera (also known as a narrowband infrared color camera, implemented by electronics) using an on-chip mosaic photosensitive chip customized with narrowband filters to assist in the detection of residual lesions during surgery and post-operative pathological sampling. This snapshot-based auxiliary sampling system eliminates the need for image registration across different wavelengths, enables high-speed, real-time imaging, and eliminates the need to block ambient light, allowing for open operation and great convenience. A flash photography mode can eliminate blurring and further reduce the effects of ambient light. The present application also improves the success rate of surgery and the accuracy and consistency of post-operative pathological sampling, reducing the difficulty of pathological sampling, while requiring lower hardware costs than radiological imaging systems.

[0106] As shown in Figure 6, an embodiment of the present application provides a customized patch 602 (a filter matrix attached to a photosensitive chip on the shortwave infrared camera) on a shortwave infrared camera 601. The attached filters are a 2x2 repeated filter matrix 603. Each 2x2 filter matrix includes a 1450 nm narrowband filter, a 1300 nm narrowband filter, a 1050 nm narrowband filter, and one wideband filter. Here, the wideband filter is made of ordinary glass or other similar material that fully transmits the shortwave infrared wavelength band, or a material with brightness attenuation function. The first three narrowband wavelength bands (1450 nm narrowband filter, 1300 nm narrowband filter, and 1050 nm narrowband filter) are used to combine three infrared color channels, and the wideband can output a gray narrowband channel image without the narrowband filtering effect.

[0107] As shown in Figure 6, a mosaic matrix is ​​attached to the shortwave infrared photosensitive chip. These mosaic matrices are 2x2 units, and each 2x2 filter matrix has four small rectangular units. Three units are 1450nm, 1300nm, and 1050nm narrowband filters, respectively. The fourth unit is either regular glass or a window with brightness attenuation function to provide a regular grayscale image. The size of each unit is the same as the pixel size of the attached photosensitive chip, ensuring a one-to-one correspondence when attached. The three narrowband filters in the 2x2 filter matrix in Figure 6 are 1050nm, 1300nm, and 1450nm. However, the narrowband wavelength bands can be changed in actual use. The three wavelength bands (1050nm, 1300nm, and 1450nm) here are merely three empirical bands effective for auxiliary sampling.

[0108] The size of each filter corresponds to the size of the pixel on the photosensitive chip. For example, if the pixel on the photosensitive chip has a side length of 5 micrometers, the side length of each filter subdivision in FIG. 6 may be 5 micrometers. If the pixel on the photosensitive chip has a side length of 8 micrometers, the side length of each filter subdivision in FIG. 6 may be 8 micrometers. This ensures that there is one filter for each pixel. For each frame image collected, the pixels covered by the 1450 nm filter are extracted, and the resulting image is a narrowband channel image with a light transmission wavelength of 1450 nm. The pixels covered by the 1300 nm filter are extracted, and the resulting image is a narrowband channel image with a light transmission wavelength of 1300 nm. The pixels covered by the 1050 nm filter are extracted, and the resulting image is a narrowband channel image with a light transmission wavelength of 1050 nm.

[0109] As shown in Figure 7, the narrowband channel images of these three channels can be directly combined into a single color image. Alternatively, all pixels corresponding to the broadband filter can be extracted, and the resulting grayscale image can be used to collect a wavelength-insensitive image. Of course, external illumination can also be used to collect images of specific wavelength bands by selecting the appropriate light source.

[0110] As shown in Figure 8, each collected frame image is split into channels using 1050nm, 1300nm, and 1450nm narrowband filters. After channel splitting, interpolation is performed to obtain a narrowband channel image with a light transmission wavelength of 1450nm, a narrowband channel image with a light transmission wavelength of 1300nm, and a narrowband channel image with a light transmission wavelength of 1050nm. The narrowband channel image with a light transmission wavelength of 1450nm, the narrowband channel image with a light transmission wavelength of 1300nm, and the narrowband channel image with a light transmission wavelength of 1050nm are then synthesized to obtain a narrowband infrared color image.

[0111] The snapshot shortwave infrared camera shown in the examples of this application can capture narrowband infrared color images in real time, something that current multispectral and hyperspectral systems struggle to do in real time.

[0112] FIG. 9A shows a bracket-type imaging system, which includes a customized short-wave infrared camera 901 (customized photosensitive chip shown in FIG. 6), an infrared apochromatic lens 902, a window 903, and a bracket 904.

[0113] Here, the infrared apochromatic lens 902 can keep several narrow-band wavelengths clearly focused on the surface of the photosensitive chip simultaneously, and the window 903 can be an infrared long-pass filter for filtering visible light, a polarizer for collecting polarized light, or an attenuator for attenuating the incident light.

[0114] 9B shows a handheld imaging system, which includes a customized short-wave infrared camera 901 (the customized photosensitive chip shown in FIG. 6), an infrared apochromatic lens 902, a window 903, and a handle 905. Here, the infrared apochromatic lens 902 can keep several narrow-band wavelengths clearly focused on the surface of the photosensitive chip simultaneously. The window 903 can be an infrared long-pass filter for filtering visible light, a polarizer for obtaining polarized light, or an attenuator for attenuating incident light.

[0115] 9A and 9B may be either a regular lens or an infrared apochromatic lens. When using a regular lens, different wavelengths have different refractive indices, which can affect the clarity of the image. However, using an infrared apochromatic lens can solve this problem. The infrared apochromatic lens uses coatings and material processes to converge light rays of different wavelengths onto the same plane, further enhancing the clarity of the image.

[0116] In the ordinary lens shown in Figure 10A, light rays of different wavelengths have different refractive indices, so they pass through the ordinary lens and converge on different planes, resulting in images of some wavelengths being clear while images of other wavelengths becoming unclear. The infrared apochromatic lens shown in Figure 10B is coated on the lens, and by carefully selecting materials and manufacturing processes, it can converge light rays of wavelengths within a certain range (e.g., 400 to 1700 nm) on one plane, avoiding the chromatic aberration that occurs with ordinary lenses.

[0117] 9A and 9B may be a broadband light source or multiple narrowband light sources lit simultaneously. With a snapshot shortwave infrared camera like the one in FIG. 6, it is not necessary to turn on light sources of different wavelengths and capture images to obtain multispectral information. Alternatively, all light sources can be turned on simultaneously, or a broadband light source can be directly used for illumination, followed by filtering with a filter matrix to obtain a multispectral image.

[0118] The constant-on mode (illumination mode of the light source) shown in Figure 11A can maintain a constant brightness for a long period of time, making it suitable for real-time sample observation and video recording. The flash mode (illumination mode of the light source) shown in Figure 11B can burst extremely large light source power for a very short period of time, making it suitable for collecting dynamic samples or scenes with strong ambient light. In flash mode, the camera only needs to select a very small shutter speed (approximately 1 / 100,000 s (seconds)) to ensure the collection of blur-free sample images.

[0119] As shown in Figure 12A, the focus of the infrared apochromatic lens is adjusted by manually rotating the focus ring 1201. As shown in Figure 12B, an additional liquid lens 1202 is attached to the infrared apochromatic lens, and the lens can be autofocused by searching for the focal plane in combination with a camera.

[0120] As shown in Figure 13, an embodiment of the present application provides a comparison between a composite infrared color image captured by a narrowband infrared color camera and a color image captured by a conventional color camera. Hollow organ tissues mainly include colon cancer tissue, rectal cancer tissue, gastric cancer tissue, and esophageal cancer tissue. For these different tumor tissues, narrowband infrared color images have relatively good tissue boundary differentiation, with similar imaged colors, demonstrating significant advantages for identifying the muscular layers of hollow organs. Narrowband infrared color images are clearer than conventional color images when determining tumor boundaries. Here, a composite color image captured by 1050 nm, 1300 nm, and 1450 nm wavelengths clearly displays the extent of tumor tissue, with different tissues displayed in different intensities of yellow to orange. In breast pathology cases, narrowband infrared color images clearly depict tumor contours and tumor tissue compared to conventional color images. The displayed contours have the highest matching with the whole-field digital pathology slide (WSI).

[0121] As described above, the technical solution provided by the embodiments of the present application uses infrared multispectral narrowbands to synthesize color images, providing physicians with information not observable with the naked eye to predict the lesion area of ​​resected tissue, and offering a new solution for identifying tumor margins during surgery and assisting in postoperative pathological sampling. This is more reliable than the physician's naked eye observation and palpation methods, and its consistency is more guaranteed. The system of the embodiments of the present application is a snapshot imaging system, capable of real-time imaging (e.g., 120 fps (frames per second)), directly solving the problem of image registration with different spectra and saving time. Another advantage of the narrowband filter matrix is ​​that it can better filter out the effects of indoor light sources and ensure spectral purity, allowing physicians to work in open spaces without the need for a light-blocking box or environmental restrictions. The flash design further eliminates the effects of ambient light (short exposure eliminates the effects of ambient light), which is a great advantage when photographing dynamic samples (e.g., movements caused by breathing or shaking). On the other hand, imaging systems are damage-free, contact-free, and ionizing radiation-free, are virtually unaffected by ambient light, are easy to use in open spaces, and have hardware system costs lower than those of radiation systems.The imaging system of the present application is more useful for sampling tumor tissues that have been excised after surgery and fixed in formalin than conventional color images, providing more information about the tumor bed to the sampling physician and improving sampling efficiency.

[0122] The image processing method provided by the embodiments of the present application has been described above in combination with exemplary applications and implementations of the electronic device provided by the embodiments of the present application. Hereinafter, we will continue to describe how each module in the image processing device provided by the embodiments of the present application cooperates to achieve image processing.

[0123] The acquisition module 5551 is configured to perform acquisition processing on the object using various narrowband filters to obtain a narrowband image including the object, and the fusion module 5552 is configured to perform fusion processing on multiple narrowband channel images that correspond one-to-one to the various narrowband filters to obtain a color image including the contour of the object.

[0124] In some embodiments, the image processing device includes a photosensitive chip, and the various narrow-band filters are regularly arranged on the photosensitive chip in a matrix format. The acquisition module 5551 is further configured to perform an image acquisition process on the object through the photosensitive chip to obtain an image including the object, perform a channel splitting process on the image including the object according to the various narrow-band filters on the photosensitive chip to obtain intermediate channel images corresponding to the various narrow-band filters, and perform an interpolation process on the intermediate channel images corresponding to the various narrow-band filters to obtain the narrow-band channel images corresponding to the various narrow-band filters.

[0125] In some embodiments, the collection module 5551 further comprises: each For the narrow-band filter, the position of the narrow-band filter on the photosensitive chip is identified, pixel extraction processing is performed on an image including the object based on the position of the narrow-band filter on the photosensitive chip to obtain pixels of the narrow-band filter, and array processing is performed on the pixels of the narrow-band filter based on the arrangement of the narrow-band filter on the photosensitive chip to obtain an intermediate channel image corresponding to the narrow-band filter.

[0126] In some embodiments, the sizes of the narrowband channel images corresponding to the various narrowband filters are the same. eachThe method is configured to identify, for each pixel, a pixel value of a corresponding pixel in each narrowband channel image, perform a synthesis process on the multiple pixel values ​​of the corresponding pixels to obtain a multi-channel pixel value of the pixel, and perform a stitching process on the multi-channel pixel values ​​of the multiple pixels to obtain a color image including the contour of the object.

[0127] In some embodiments, the fusion module 5552 is further configured to identify some narrowband filters among the multiple types of narrowband filters, and perform a fusion process on the narrowband channel images corresponding to the some narrowband filters to obtain a color image including the contours of the object.

[0128] In some embodiments, the fusion module 5552 is further configured to display filtering indices of various types of narrowband filters, and in response to a selection operation on the filtering indices, make the narrowband filter corresponding to the selected filtering indices into a part of the narrowband filters.

[0129] In some embodiments, the fusion module 5552 is further configured to obtain frequencies at which the various types of narrowband filters are used, sort the various types of narrowband filters in descending order based on the frequencies at which the various types of narrowband filters are used, and include the various types of narrowband filters that are arranged first in the result of the descending order as some of the narrowband filters.

[0130] In some embodiments, when there are four types of narrowband filters, the fusion module 5552 is further configured to identify any three types of narrowband filters among the multiple types of narrowband filters, perform a synthesis process on the narrowband images corresponding to the any three types of narrowband filters, obtain candidate color images, perform an edge detection process on each candidate color image, obtain edge features of each candidate color image, and select the candidate color image with the most edge features as the color image including the contour of the target.

[0131] In some embodiments, the image processing device further includes a wideband filter, and is configured to perform a fusion process on a plurality of narrowband channel images that correspond one-to-one to various types of narrowband filters to obtain a color image including the contours of the object, and then the acquisition module 5551 is further configured to perform an acquisition process on the object using the wideband filter to obtain a wideband channel image including the object, and perform an image enhancement process on the color image including the contours of the object based on the wideband channel image to obtain an enhanced color image.

[0132] In some embodiments, the acquisition module 5551 is further configured to perform an acquisition process on the pathological tissue using various narrowband filters to obtain narrowband channel pathological images containing the pathological tissue, and the fusion module 5552 is further configured to perform a fusion process on the multiple narrowband channel pathological images corresponding to the various narrowband filters in a one-to-one correspondence to obtain a color pathological image containing the contours of the pathological tissue.

[0133] An embodiment of the present application provides a computer program product, the computer program product including a computer program or computer commands, the computer-executable commands being stored in a computer-readable storage medium, a processor of an electronic device reading the computer-executable commands from the computer-readable storage medium, and the processor executing the computer-executable commands to cause the electronic device to perform the image processing method of the embodiment of the present application.

[0134] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored and, when executed by a processor, cause the processor to perform an image processing method provided by the embodiment of the present application, such as the image processing method shown in Figures 5A-5B.

[0135] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM, or may be any device including one or any combination of the above memories.

[0136] In some embodiments, the computer-executable commands take the form of a program, software, software module, script, or code, compiled according to any type of programming language (including compiled or interpreted, or declarative or procedural languages), deployed according to any type, including as a stand-alone program or deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0137] As an example, the executable commands may or may not correspond to a file in a file system and may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program being discussed, or in multiple associated files (e.g., files that store one or more modules, subprograms, or code portions).

[0138] As an example, the computer-executable commands may be deployed to be executed on a single electronic device, or on multiple electronic devices located at a single location, or on multiple electronic devices distributed across multiple locations and interconnected via a communications network.

[0139] The above is merely an example of the present application and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are all included in the scope of protection of the present application.

Claims

1. An image processing method executed by an electronic device including many types of narrow-band filters that pass optical signals in different wavelength bands, comprising: The image processing method includes: Acquiring a narrowband channel image including the object by collecting and processing a photosensitive signal, which is an optical signal of the object, using each of the multiple narrowband filters; performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; Including, When the number of types of narrowband filters is at least four, performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object, Identifying any three types of narrow band filters from the multiple types of narrow band filters; performing a synthesis process on the narrowband channel images corresponding to the three types of narrowband filters to obtain a candidate color image; performing an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images; determining the candidate color image with the most edge features as a color image including the contour of the object; An image processing method comprising:

2. The electronic device includes a photosensitive chip, and the various types of narrow-band filters are regularly arranged on the photosensitive chip in a matrix format; Acquiring a narrowband channel image including the object by collecting and processing a photosensitive signal, which is an optical signal of the object, using each of the multiple types of narrowband filters, includes: acquiring an image including the object based on the photosensitive signal; performing a channel division process on an image including the object based on the multiple narrow-band filters on the photosensitive chip to obtain intermediate channel images corresponding to the multiple narrow-band filters; performing an interpolation process on intermediate channel images corresponding to the multiple types of narrowband filters to obtain narrowband channel images corresponding to the multiple types of narrowband filters; 2. The image processing method according to claim 1, further comprising:

3. performing a channel division process on an image including the object based on the various narrow-band filters in the photosensitive chip, and obtaining intermediate channel images corresponding to the various narrow-band filters; For each narrow band filter in the photosensitive chip: Identifying the location of the narrow band filter on the photosensitive chip; performing a pixel extraction process on an image including the object based on the position of the narrow-band filter on the photosensitive chip, and acquiring pixels of the narrow-band filter; performing an array process on pixels of the narrow-band filters based on the arrangement of the narrow-band filters on the photosensitive chip to obtain intermediate channel images corresponding to the narrow-band filters; 3. The image processing method according to claim 2, further comprising:

4. the plurality of narrowband channel images have the same size, and the plurality of narrowband channel images are obtained by performing a channel decomposition process on an image including the object; performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; For each pixel in the image containing the object, identifying pixel values ​​for the corresponding pixels of each of the narrowband channel images; performing a synthesis process on pixel values ​​of the corresponding pixels of the plurality of stored narrowband channel images to obtain multi-channel pixel values ​​of the pixels; performing a stitching process on the multi-channel pixel values ​​of the plurality of pixels to obtain a color image including the contour of the object; 2. The image processing method according to claim 1, further comprising:

5. performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; Identifying a subset of narrowband filters from the plurality of types of narrowband filters; performing a fusion process on the narrowband channel images corresponding to the part of the narrowband filters to obtain a color image including the contour of the object; 2. The image processing method according to claim 1, further comprising:

6. Identifying the part of narrow band filters among the multiple types of narrow band filters includes: Displaying filtering indicators of various types of said narrow band filters; In response to a selection operation for the filtering index, the narrowband filter corresponding to the selected filtering index is set as one of the narrowband filters; 6. The image processing method according to claim 5, further comprising:

7. Identifying the part of narrow band filters among the multiple types of narrow band filters includes: Obtaining frequencies at which various types of the narrowband filters are used; sorting the various types of narrowband filters in descending order based on the frequencies at which the various types of narrowband filters are used, and determining the various types of narrowband filters that are arranged first in the result of sorting in descending order as the some of the narrowband filters; 6. The image processing method according to claim 5, further comprising:

8. the electronic device further includes a wideband filter; The image processing method further comprises: Acquiring a wideband channel image including the object by collecting and processing a photosensitive signal, which is an optical signal of the object, using the wideband filter; performing an image enhancement process on a color image including the contour of the object based on the broadband channel image to obtain the enhanced color image; 2. The image processing method according to claim 1, further comprising:

9. Acquiring a narrowband channel image including the object by collecting and processing a photosensitive signal, which is an optical signal of the object, using each of the multiple types of narrowband filters, includes: collecting and processing photosensitive signals, which are optical signals of the pathological tissue, which is the object, using each of the multiple types of narrowband filters to obtain narrowband channel pathological images including the pathological tissue; performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; performing fusion processing on a plurality of narrowband channel pathological images corresponding one-to-one to the various types of narrowband filters to obtain a color pathological image including the contour of the pathological tissue; 2. The image processing method according to claim 1, further comprising:

10. a housing forming a partially sealed space; a variety of narrow-band filters located on a photosensitive chip inside the housing for outputting photosensitive signals, and collecting optical signals; a processor located inside the housing, receiving a photosensitive signal output by the photosensitive chip, the processor generating a narrowband channel image including the object based on the photosensitive signal, which is an optical signal of the object collected by the various types of narrowband filters, and performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; The processor: When the types of narrow-band filters are at least four, a fusion process is performed on a plurality of narrow-band channel images corresponding one-to-one to the various types of narrow-band filters to obtain a color image including the contour of the object, Identifying any three types of narrow band filters from the multiple types of narrow band filters; performing a synthesis process on the narrowband channel images corresponding to the three types of narrowband filters to obtain a candidate color image; performing an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images; The candidate color image having the most edge features is determined to be the color image including the contour of the object. An electronic device characterized by:

11. The various narrow-band filters are regularly arranged on the photosensitive chip in a matrix format; The processor further comprises: generating an image including the object based on the photosensitive signal; performing a channel division process on the image including the object based on the multiple narrow-band filters in the photosensitive chip to obtain intermediate channel images corresponding to the multiple narrow-band filters; performing an interpolation process on the intermediate channel images corresponding to the various types of narrowband filters to obtain narrowband channel images corresponding to the various types of narrowband filters; 11. The electronic device according to claim 10.

12. The optical system further includes an achromatic lens disposed inside the housing for collecting narrow-band waves collected by the various narrow-band filters onto the photosensitive chip.

11. The electronic device according to claim 10.

13. The achromatic lens further comprises: a focus ring disposed on the surface of the achromatic lens for adjusting the focal length of the achromatic lens; or a liquid ring disposed on the achromatic lens to adjust the focal length of the achromatic lens; 13. The electronic device according to claim 12, comprising:

14. a window disposed inside the housing to protect the photosensitive chip and the processor; The window type includes at least one of a long-pass filter, a polarizer, and an attenuator.

11. The electronic device according to claim 10.

15. a light source device disposed inside the housing for providing at least one light source, the light source type including at least one of a broadband light source and a narrowband light source; The illumination mode of the light source device includes at least one of a constant lighting mode and a flash mode.

11. The electronic device according to claim 10.

16. an acquisition module configured to acquire narrowband channel images including the object by collecting and processing photosensitive signals, which are optical signals of the object, using each of a variety of narrowband filters that pass optical signals in different wavelength bands; a fusion module configured to perform a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; Including, When the number of types of narrowband filters is at least four, the fusion module is configured to perform a fusion process on a plurality of narrowband channel images that correspond one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object, Identifying any three types of narrow band filters from the multiple types of narrow band filters; performing a synthesis process on the narrowband channel images corresponding to the three types of narrowband filters to obtain a candidate color image; performing an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images; The candidate color image having the most edge features is determined to be the color image including the contour of the object.

1. An image processing device comprising:

17. a memory for storing executable commands; a processor that, when executing executable commands stored in said memory, implements the image processing method of any one of claims 1 to 9; 17. The image processing device according to claim 16, further comprising:

18. On the computer, A computer-readable storage medium storing a program executed by a computer, the computer-readable storage medium including various types of narrow-band filters that pass optical signals in different wavelength bands, Acquiring narrowband channel images including the object by collecting and processing photosensitive signals, which are optical signals of the object, using each of the multiple narrowband filters; performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object; storing the program for causing the computer to execute the above; When the number of types of narrowband filters is at least four, the step of performing a fusion process on a plurality of narrowband channel images corresponding one-to-one to the various types of narrowband filters to obtain a color image including the contour of the object includes: Identifying any three types of narrow band filters from the multiple types of narrow band filters; performing a synthesis process on the narrowband channel images corresponding to the three types of narrowband filters to obtain a candidate color image; performing an edge detection process on each of the candidate color images to obtain edge features of each of the candidate color images; determining the candidate color image with the most edge features as a color image including the contour of the object; Including, A computer-readable storage medium comprising:

Images