A visual augmentation method and system based on AR glasses

Abstract

The application provides a visual enhancement method and system based on AR glasses, the method comprising: respectively pre-processing a target visible light image and a target infrared light image; respectively performing pyramid image registration on the processed visible light image and the processed infrared light image; respectively performing image transformation decomposition on the visible light registration image and the infrared light registration image; fusing visible light low-frequency coefficients and infrared light low-frequency coefficients, and fusing visible light high-frequency coefficients and infrared light high-frequency coefficients; and performing inverse image transformation decomposition on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain an enhanced visual image. The application combines heat source information of an infrared image and detail texture information of a faint light image, and significantly improves the visual effect and the definition of the image.

Description

Technical Field

[0001] This invention belongs to the technical field of visual enhancement, and specifically relates to a visual enhancement method and system based on AR glasses. Background Technology

[0002] AR glasses, by integrating camera technology, achieve a seamless fusion of reality and virtuality, incorporating virtual information into people's vision. The technical principles of AR glasses mainly consist of three parts: computer vision, sensors, and projection technology. AR glasses adopt a head-mounted design, allowing users to easily access various applications and services through simple voice commands or gestures. The display module is the core module of AR glasses, primarily responsible for delivering virtual information to the user's eyes, producing a visually fusion effect. Currently, the mainstream display technology for AR glasses is based on diffractive waveguides with embossed surface gratings.

[0003] To improve the recognition accuracy of traditional AR glasses, visible light images and infrared images are often fused together, thereby improving the visual effect and the accuracy of target recognition. However, traditional fusion algorithms have drawbacks such as easy loss of image details, low contrast, and insufficient information, which affect the visual effect of AR glasses. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a visual enhancement method and system based on AR glasses, which solves the technical problems in the prior art.

[0005] In a first aspect, the present invention provides the following technical solution: a visual enhancement method based on AR glasses, comprising: The visible light image and infrared light image of the target are acquired through AR glasses. The visible light image and the infrared light image of the target are preprocessed respectively to obtain the processed visible light image and the processed infrared light image. The processed visible light image and the processed infrared light image are respectively subjected to pyramid image registration to obtain a visible light registered image and an infrared light registered image; The visible light registered image and the infrared light registered image are respectively subjected to image transformation decomposition to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient; The visible light low-frequency coefficient and the infrared light low-frequency coefficient are fused to obtain a fused low-frequency coefficient, and the visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. The fused low-frequency coefficients and the fused high-frequency coefficients are decomposed by inverse image transformation to obtain an enhanced visual image.

[0006] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention acquires a target visible light image and a target infrared light image through AR glasses, and preprocesses the target visible light image and the target infrared light image respectively to obtain a processed visible light image and a processed infrared light image; pyramid image registration is performed on the processed visible light image and the processed infrared light image respectively to obtain a visible light registered image and an infrared light registered image; image transform decomposition is performed on the visible light registered image and the infrared light registered image respectively to obtain visible light low-frequency coefficients, visible light high-frequency coefficients, infrared light low-frequency coefficients, and infrared light high-frequency coefficients; the visible light low-frequency coefficients and infrared light low-frequency coefficients are fused to obtain fused low-frequency coefficients, and the visible light high-frequency coefficients and infrared light high-frequency coefficients are fused to obtain fused high-frequency coefficients; the fused low-frequency coefficients and fused high-frequency coefficients are subjected to inverse image transform decomposition to obtain an enhanced visual image. This invention registers the two images, which can meet the spatial matching requirements of subsequent image fusion, and can also reduce the fusion complexity. Then, the two images are fused, combining the heat source information of the infrared image and the detailed texture information of the low-light image, which significantly improves the visual effect and clarity of the image.

[0007] Preferably, the step of performing pyramid image registration on the processed visible light image and the processed infrared light image respectively to obtain a visible light registered image and an infrared light registered image includes: The processed visible light image and the processed infrared light image are respectively subjected to four-level multi-resolution decomposition to obtain visible light pyramid images with decreasing resolution. Infrared pyramid images : ; , ; In the formula, For the first Layered pyramid images in Pixel value at that location, For standard Gaussian kernels, For local coordinate indices within the Gaussian kernel, This is a visible light image of the pyramids. Infrared pyramid image; The initial transformation parameters between the two images are calculated starting with the lowest resolution visible light pyramid image and infrared light pyramid image. : ; In the formula, For mutual information calculation, For the first Transformation parameters for layer estimation; The first Initial transformation parameters of the layer Passed to the The layers are used as initial values ​​to optimize the transformation parameters, in order to obtain updated transformation parameters. : ; In the formula, , The first Visible light pyramid images and infrared light pyramid images of the layers. For the first Transformation parameters for layer estimation; The optimization process of the transformation parameters is repeated until the highest resolution pyramid layer is reached to obtain the final transformation parameters. Based on the final transformation parameters, image registration is performed to obtain a visible light registered image and an infrared light registered image.

[0008] Preferably, the step of performing image transformation decomposition on the visible light registered image and the infrared light registered image respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared low-frequency coefficient, and infrared high-frequency coefficient includes: Obtain the first and second filter banks in NSPFB, and determine the first filter based on the first filter bank. : ; In the formula, These are the low-pass and high-pass filters in the first filter bank, respectively. The second filter is determined based on the second filter bank. : , ; In the formula, These are the sector, checkerboard, and parallelogram filters in the second filter bank, respectively. The visible light registration image and the infrared light registration image are processed by the first filter and the second filter respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient.

[0009] Preferably, the step of fusing the visible light low-frequency coefficient and the infrared light low-frequency coefficient to obtain a fused low-frequency coefficient includes: Calculate the first low-frequency fusion weight With the second low-frequency fusion weight : , ; In the formula, To integrate low-frequency coefficients in Pixel value at that location, These are the pixel mean and pixel variance, respectively, after fusing low-frequency coefficients. This is the adjustment coefficient; Based on the first low-frequency fusion weight With the second low-frequency fusion weight The visible light low-frequency coefficient will be... The infrared light low-frequency coefficient Fusion is performed to obtain the fused low-frequency coefficients. : .

[0010] Preferably, the step of fusing the visible light high-frequency coefficient and the infrared light high-frequency coefficient to obtain a fused high-frequency coefficient includes: Calculate the spatial frequency of the visible light high-frequency coefficient. and the spatial frequency of the infrared light high-frequency coefficient : ; ; In the formula, These represent the horizontal and vertical spatial frequencies of the visible light high-frequency coefficient, respectively. These are the horizontal and vertical spatial frequencies of the high-frequency coefficient of infrared light, respectively. Calculate the energy characteristics of the visible light high-frequency coefficient. and the energy characteristics of the high-frequency coefficient of the infrared light : ; ; In the formula, To preset window size, The high frequency coefficients of visible light are respectively Pixel value at that location, The high frequency coefficients of infrared light are respectively in Pixel value at; Spatial frequency based on visible light high-frequency coefficient Spatial frequency of infrared high-frequency coefficient Energy characteristics of high-frequency coefficients in visible light Energy characteristics of high frequency coefficients in infrared light Calculate the high-frequency characteristics of the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients : ; ; High-frequency characteristics based on the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients Calculate high-frequency fusion weights : ; Based on the high-frequency fusion weight The visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. : .

[0011] Secondly, the present invention provides the following technical solution: a visual enhancement system based on AR glasses, the system comprising: The preprocessing module is used to acquire a target visible light image and a target infrared light image through AR glasses, and to preprocess the target visible light image and the target infrared light image respectively to obtain a processed visible light image and a processed infrared light image. The registration module is used to perform pyramid image registration on the processed visible light image and the processed infrared light image respectively, so as to obtain a visible light registered image and an infrared light registered image; The transformation module is used to perform image transformation decomposition on the visible light registration image and the infrared light registration image respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient. The fusion module is used to fuse the visible light low-frequency coefficient and the infrared light low-frequency coefficient to obtain a fused low-frequency coefficient, and to fuse the visible light high-frequency coefficient and the infrared light high-frequency coefficient to obtain a fused high-frequency coefficient. The inverse transform module is used to perform inverse image transform decomposition on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain an enhanced visual image.

[0012] Preferably, the registration module is specifically used for: The processed visible light image and the processed infrared light image are respectively subjected to four-level multi-resolution decomposition to obtain visible light pyramid images with decreasing resolution. Infrared pyramid images : ; , ; In the formula, For the first Layered pyramid images in Pixel value at that location, For standard Gaussian kernels, For local coordinate indices within the Gaussian kernel, This is a visible light image of the pyramids. Infrared pyramid image; The initial transformation parameters between the two images are calculated starting with the lowest resolution visible light pyramid image and infrared light pyramid image. : ; In the formula, For mutual information calculation, For the first Transformation parameters for layer estimation; The first Initial transformation parameters of the layer Passed to the The layers are used as initial values ​​to optimize the transformation parameters, in order to obtain updated transformation parameters. : ; In the formula, , The first Visible light pyramid images and infrared light pyramid images of the layers. For the first Transformation parameters for layer estimation; The optimization process of the transformation parameters is repeated until the highest resolution pyramid layer is reached to obtain the final transformation parameters. Based on the final transformation parameters, image registration is performed to obtain a visible light registered image and an infrared light registered image.

[0013] Preferably, the transformation module is specifically used for: Obtain the first and second filter banks in NSPFB, and determine the first filter based on the first filter bank. : ; In the formula, These are the low-pass and high-pass filters in the first filter bank, respectively. The second filter is determined based on the second filter bank. : , ; In the formula, These are the sector, checkerboard, and parallelogram filters in the second filter bank, respectively. The visible light registration image and the infrared light registration image are processed by the first filter and the second filter respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient.

[0014] Thirdly, the present invention provides the following technical solution: a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual enhancement method based on AR glasses as described above.

[0015] Fourthly, the present invention provides the following technical solution: a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-described visual enhancement method based on AR glasses. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a vision enhancement method based on AR glasses provided in Embodiment 1 of the present invention; Figure 2 This is a structural block diagram of the visual enhancement system based on AR glasses provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the hardware structure of a computer provided for another embodiment of the present invention.

[0018] The embodiments of the present invention will be further described below with reference to the accompanying drawings. Detailed Implementation

[0019] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain embodiments of the present invention, and should not be construed as limiting the present invention.

[0020] Example 1 In Embodiment 1 of the present invention, as Figure 1 As shown, a visual enhancement method based on AR glasses includes: S1. Obtain a visible light image and an infrared light image of the target through AR glasses, and preprocess the visible light image and the infrared light image of the target respectively to obtain a processed visible light image and a processed infrared light image. Specifically, the preprocessing process here is a common method used in the prior art for processing visible light and infrared light images, so it will not be described in detail here.

[0021] S2. Perform pyramid image registration on the processed visible light image and the processed infrared light image respectively to obtain a visible light registered image and an infrared light registered image; Step S2 includes: S21. Perform four-level multi-resolution decomposition on the processed visible light image and the processed infrared light image respectively to obtain visible light pyramid images with decreasing resolution. Infrared pyramid images : ; , ; In the formula, For the first Layered pyramid images in Pixel value at that location, For standard Gaussian kernels, For local coordinate indices within the Gaussian kernel, This is a visible light image of the pyramids. Infrared pyramid image; Among them, layer 0 is the pyramid layer with the highest resolution, and layer 3 is the pyramid layer with the lowest resolution.

[0022] S22. Calculate the initial transformation parameters between the two images, starting with the lowest resolution visible light pyramid image and infrared light pyramid image. : ; In the formula, For mutual information calculation, For the first Transformation parameters for layer estimation.

[0023] S23, the first Initial transformation parameters of the layer Passed to the The layers are used as initial values ​​to optimize the transformation parameters, in order to obtain updated transformation parameters. : ; In the formula, , The first Visible light pyramid images and infrared light pyramid images of the layers. For the first Transformation parameters for layer estimation.

[0024] S24. Repeat the optimization process of transformation parameters until the highest resolution pyramid layer is reached to obtain the final transformation parameters. Based on the final transformation parameters, perform image registration to obtain a visible light registered image and an infrared light registered image.

[0025] S3. Perform image transformation decomposition on the visible light registration image and the infrared light registration image respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient. Step S3 includes: S31. Obtain the first filter bank and the second filter bank in NSPFB, and determine the first filter bank based on the first filter bank. : ; In the formula, These are the low-pass and high-pass filters in the first filter bank, respectively. S32. Determine the second filter based on the second filter bank. : , ; In the formula, These are the sector, checkerboard, and parallelogram filters in the second filter bank, respectively. S33. The visible light registration image and the infrared light registration image are processed by the first filter and the second filter respectively to obtain the visible light low-frequency coefficient, the visible light high-frequency coefficient, the infrared light low-frequency coefficient, and the infrared light high-frequency coefficient. Specifically, the above process is the NSCT transform process, but this application convolves the tower filter bank in NSPFB and the directional filter bank in NSDFB respectively, decomposing the image through l layers of NSPFB to generate l+1 images, including one low-frequency sub-band image and l high-frequency sub-band images. Each high-frequency sub-band image needs to be processed... The first-stage directional filter generates directional sub-bands.

[0026] S4. The visible light low-frequency coefficient and the infrared light low-frequency coefficient are fused to obtain a fused low-frequency coefficient, and the visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. The step of fusing the visible light low-frequency coefficient and the infrared light low-frequency coefficient to obtain the fused low-frequency coefficient includes: S411, Calculate the first low-frequency fusion weight. With the second low-frequency fusion weight : , ; In the formula, To integrate low-frequency coefficients in Pixel value at that location, These are the pixel mean and pixel variance, respectively, after fusing low-frequency coefficients. This is the adjustment coefficient; S412, Based on the first low-frequency fusion weight With the second low-frequency fusion weight The visible light low-frequency coefficient will be... The infrared light low-frequency coefficient Fusion is performed to obtain the fused low-frequency coefficients. : .

[0027] Specifically, the adjustment coefficient is 0.8. In the low-frequency fusion process of this application, it can effectively solve the problem that the simple weighted fusion algorithm has poor performance in preserving global and contour information of the image. This method uses a fuzzy Gaussian membership function to adaptively fuse the low-frequency coefficients, which has a better effect in preserving the global and contour information of the original image.

[0028] The step of fusing the visible light high-frequency coefficient and the infrared light high-frequency coefficient to obtain a fused high-frequency coefficient includes: S421. Calculate the spatial frequency of the visible light high-frequency coefficient. and the spatial frequency of the infrared light high-frequency coefficient : ; ; In the formula, These represent the horizontal and vertical spatial frequencies of the visible light high-frequency coefficient, respectively. These are the horizontal and vertical spatial frequencies of the high-frequency coefficient of infrared light, respectively. S422. Calculate the energy characteristics of the visible light high-frequency coefficient. and the energy characteristics of the high-frequency coefficient of the infrared light : ; ; In the formula, To preset window size, The high frequency coefficients of visible light are respectively Pixel value at that location, The high frequency coefficients of infrared light are respectively in Pixel value at; S423, Spatial frequency based on visible light high-frequency coefficient Spatial frequency of infrared high-frequency coefficient Energy characteristics of high-frequency coefficients in visible light Energy characteristics of high frequency coefficients in infrared light Calculate the high-frequency characteristics of the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients : ; ; S424. High-frequency characteristics based on the visible light high-frequency coefficient. High-frequency characteristics of infrared light high-frequency coefficients Calculate high-frequency fusion weights : ; S425, Based on the aforementioned high-frequency fusion weights The visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. : ; Specifically, in the high-frequency fusion process, the high-frequency coefficients obtained from scale decomposition represent the image's detail information and sharpness. To improve the richness and sharpness of image details, this application adopts a typical maxima strategy, while also enhancing the image's detail richness and sharpness by designing reasonable neighborhood feature indicators.

[0029] S5. Perform inverse image transform decomposition on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain an enhanced visual image; Specifically, the enhanced visual image can be obtained by performing the inverse NSCT transform.

[0030] The visual enhancement method based on AR glasses provided in Embodiment 1 of this invention involves acquiring a visible light image and a target infrared image using AR glasses. The visible light and infrared images are preprocessed to obtain processed visible light and processed infrared images, respectively. Pyramid image registration is then performed on the processed visible light and processed infrared images to obtain a registered visible light image and a registered infrared image. Finally, image transform decomposition is performed on the registered visible light and registered infrared images to obtain low-frequency coefficients, high-frequency coefficients, low-frequency coefficients, and infrared coefficients. The invention employs a method of fusing two images: high-frequency coefficients of light and low-frequency coefficients of infrared light to obtain a fused low-frequency coefficient, and high-frequency coefficients of light and infrared light to obtain a fused high-frequency coefficient. The fused low-frequency coefficient and fused high-frequency coefficient are then subjected to inverse image transform decomposition to obtain an enhanced visual image. This invention registers the two images, which can meet the spatial matching requirements of subsequent image fusion and reduce fusion complexity. The two images are then fused, combining the heat source information of the infrared image with the detailed texture information of the low-light image, significantly improving the visual effect and clarity of the image.

[0031] Example 2 like Figure 2 As shown, in Embodiment 2 of the present invention, a visual enhancement system based on AR glasses is provided, the system comprising: The preprocessing module 1 is used to acquire a target visible light image and a target infrared light image through AR glasses, and to preprocess the target visible light image and the target infrared light image respectively to obtain a processed visible light image and a processed infrared light image. Registration module 2 is used to perform pyramid image registration on the processed visible light image and the processed infrared light image respectively, so as to obtain a visible light registered image and an infrared light registered image; Transformation module 3 is used to perform image transformation decomposition on the visible light registration image and the infrared light registration image respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient. The fusion module 4 is used to fuse the visible light low-frequency coefficient and the infrared light low-frequency coefficient to obtain a fused low-frequency coefficient, and to fuse the visible light high-frequency coefficient and the infrared light high-frequency coefficient to obtain a fused high-frequency coefficient. The inverse transform module 5 is used to perform inverse image transform decomposition on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain an enhanced visual image.

[0032] Specifically, the registration module 2 is used for: The processed visible light image and the processed infrared light image are respectively subjected to four-level multi-resolution decomposition to obtain visible light pyramid images with decreasing resolution. Infrared pyramid images : ; , ; In the formula, For the first Layered pyramid images in Pixel value at that location, For standard Gaussian kernels, For local coordinate indices within the Gaussian kernel, This is a visible light image of the pyramids. Infrared pyramid image; The initial transformation parameters between the two images are calculated starting with the lowest resolution visible light pyramid image and infrared light pyramid image. : ; In the formula, For mutual information calculation, For the first Transformation parameters for layer estimation; The first Initial transformation parameters of the layer Passed to the The layers are used as initial values ​​to optimize the transformation parameters, in order to obtain updated transformation parameters. : ; In the formula, , The first Visible light pyramid images and infrared light pyramid images of the layers. For the first Transformation parameters for layer estimation; The optimization process of the transformation parameters is repeated until the highest resolution pyramid layer is reached to obtain the final transformation parameters. Based on the final transformation parameters, image registration is performed to obtain a visible light registered image and an infrared light registered image.

[0033] Specifically, the transformation module 3 is used for: Obtain the first and second filter banks in NSPFB, and determine the first filter based on the first filter bank. : ; In the formula, These are the low-pass and high-pass filters in the first filter bank, respectively. The second filter is determined based on the second filter bank. : , ; In the formula, These are the sector, checkerboard, and parallelogram filters in the second filter bank, respectively. The visible light registration image and the infrared light registration image are processed by the first filter and the second filter respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient.

[0034] Specifically, the fusion module 4 is used for: Calculate the first low-frequency fusion weight With the second low-frequency fusion weight : , ; In the formula, To integrate low-frequency coefficients in Pixel value at that location, These are the pixel mean and pixel variance, respectively, after fusing low-frequency coefficients. This is the adjustment coefficient; Based on the first low-frequency fusion weight With the second low-frequency fusion weight The visible light low-frequency coefficient will be... The infrared light low-frequency coefficient Fusion is performed to obtain the fused low-frequency coefficients. : .

[0035] The fusion module 4 is further used for: Calculate the spatial frequency of the visible light high-frequency coefficient. and the spatial frequency of the infrared light high-frequency coefficient : ; ; In the formula, These represent the horizontal and vertical spatial frequencies of the visible light high-frequency coefficient, respectively. These are the horizontal and vertical spatial frequencies of the high-frequency coefficient of infrared light, respectively. Calculate the energy characteristics of the visible light high-frequency coefficient. and the energy characteristics of the high-frequency coefficient of the infrared light : ; ; In the formula, To preset window size, The high frequency coefficients of visible light are respectively Pixel value at that location, The high frequency coefficients of infrared light are respectively in Pixel value at; Spatial frequency based on visible light high-frequency coefficient Spatial frequency of infrared high-frequency coefficient Energy characteristics of high-frequency coefficients in visible light Energy characteristics of high frequency coefficients in infrared light Calculate the high-frequency characteristics of the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients : ; ; High-frequency characteristics based on the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients Calculate high-frequency fusion weights : ; Based on the high-frequency fusion weight The visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. : .

[0036] In other embodiments of the present invention, the present invention provides the following technical solution: a computer, including a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, wherein the processor 101 executes the computer program to implement the visual enhancement method based on AR glasses as described above.

[0037] Specifically, the processor 101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0038] The memory 102 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 102 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 102 may include removable or non-removable (or fixed) media. Where appropriate, the memory 102 may be internal or external to a data processing device. In a particular embodiment, the memory 102 is non-volatile memory. In a particular embodiment, the memory 102 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random Access Memory (FPMDRAM), Extended Data Out Dynamic Random Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0039] The memory 102 can be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 101.

[0040] The processor 101 implements the above-described visual enhancement method based on AR glasses by reading and executing computer program instructions stored in the memory 102.

[0041] In some embodiments, the computer may further include a communication interface 103 and a bus 100. For example, Figure 3 As shown, the processor 101, memory 102, and communication interface 103 are connected through bus 100 and complete communication with each other.

[0042] The communication interface 103 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of the present invention. The communication interface 103 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0043] Bus 100 includes hardware, software, or both, that couples components of a computer device together. Bus 100 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 100 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 100 may include one or more buses. Although specific buses are described and illustrated in the embodiments of the present invention, the present invention is contemplated by any suitable bus or interconnect.

[0044] The computer can execute the AR glasses-based visual enhancement method of the present invention based on the acquired AR glasses-based visual enhancement system, thereby realizing AR glasses-based visual enhancement.

[0045] In some further embodiments of the present invention, in conjunction with the above-described visual enhancement method based on AR glasses, the present invention provides the following technical solution: a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-described visual enhancement method based on AR glasses.

[0046] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0047] More specific examples of readable media (a non-exhaustive list) include: electrical connections (electronic devices) with one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0048] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0050] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A visual enhancement method based on AR glasses, characterized in that, include: The visible light image and infrared light image of the target are acquired through AR glasses. The visible light image and the infrared light image of the target are preprocessed respectively to obtain the processed visible light image and the processed infrared light image. The processed visible light image and the processed infrared light image are respectively subjected to pyramid image registration to obtain a visible light registered image and an infrared light registered image; The visible light registered image and the infrared light registered image are respectively subjected to image transformation decomposition to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient; The visible light low-frequency coefficient and the infrared light low-frequency coefficient are fused to obtain a fused low-frequency coefficient, and the visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. The fused low-frequency coefficients and the fused high-frequency coefficients are decomposed by inverse image transformation to obtain an enhanced visual image.

2. The visual enhancement method based on AR glasses according to claim 1, characterized in that, The step of performing pyramid image registration on the processed visible light image and the processed infrared light image respectively to obtain a visible light registered image and an infrared light registered image includes: The processed visible light image and the processed infrared light image are respectively subjected to four-level multi-resolution decomposition to obtain visible light pyramid images with decreasing resolution. Infrared pyramid images : ； ， ； In the formula, For the first Layered pyramid images in Pixel value at that location, For standard Gaussian kernels, For local coordinate indices within the Gaussian kernel, This is a visible light image of the pyramids. Infrared pyramid image; The initial transformation parameters between the two images are calculated starting with the lowest resolution visible light pyramid image and infrared light pyramid image. : ； In the formula, For mutual information calculation, For the first Transformation parameters for layer estimation; The first Initial transformation parameters of the layer Passed to the The layers are used as initial values ​​to optimize the transformation parameters, in order to obtain updated transformation parameters. : ； In the formula, , The first Visible light pyramid images and infrared light pyramid images of the layers. For the first Transformation parameters for layer estimation; The optimization process of the transformation parameters is repeated until the highest resolution pyramid layer is reached to obtain the final transformation parameters. Based on the final transformation parameters, image registration is performed to obtain a visible light registered image and an infrared light registered image.

3. The visual enhancement method based on AR glasses according to claim 1, characterized in that, The step of performing image transformation decomposition on the visible light registered image and the infrared light registered image respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared low-frequency coefficient, and infrared high-frequency coefficient includes: Obtain the first and second filter banks in NSPFB, and determine the first filter based on the first filter bank. : ； In the formula, These are the low-pass and high-pass filters in the first filter bank, respectively. The second filter is determined based on the second filter bank. : ， ； In the formula, These are the sector, checkerboard, and parallelogram filters in the second filter bank, respectively. The visible light registration image and the infrared light registration image are processed by the first filter and the second filter respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient.

4. The visual enhancement method based on AR glasses according to claim 1, characterized in that, The step of fusing the visible light low-frequency coefficient and the infrared light low-frequency coefficient to obtain the fused low-frequency coefficient includes: Calculate the first low-frequency fusion weight With the second low-frequency fusion weight : ， ； In the formula, To integrate low-frequency coefficients in Pixel value at that location, These are the pixel mean and pixel variance, respectively, after fusing low-frequency coefficients. This is the adjustment coefficient; Based on the first low-frequency fusion weight With the second low-frequency fusion weight The visible light low-frequency coefficient will be... The infrared light low-frequency coefficient Fusion is performed to obtain the fused low-frequency coefficients. : 。 5. The visual enhancement method based on AR glasses according to claim 1, characterized in that, The step of fusing the visible light high-frequency coefficient and the infrared light high-frequency coefficient to obtain a fused high-frequency coefficient includes: Calculate the spatial frequency of the visible light high-frequency coefficient. and the spatial frequency of the infrared light high-frequency coefficient : ； ； In the formula, These represent the horizontal and vertical spatial frequencies of the visible light high-frequency coefficient, respectively. These are the horizontal and vertical spatial frequencies of the high-frequency coefficient of infrared light, respectively. Calculate the energy characteristics of the visible light high-frequency coefficient. and the energy characteristics of the high-frequency coefficient of the infrared light : ； ； In the formula, To preset window size, The high frequency coefficients of visible light are respectively in Pixel value at that location, The high frequency coefficients of infrared light are respectively in Pixel value at; Spatial frequency based on visible light high-frequency coefficient Spatial frequency of infrared high-frequency coefficient Energy characteristics of high-frequency coefficients in visible light Energy characteristics of high frequency coefficients in infrared light Calculate the high-frequency characteristics of the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients : ； ； High-frequency characteristics based on the visible light high-frequency coefficient High-frequency characteristics of infrared light high-frequency coefficients Calculate high-frequency fusion weights : ； Based on the high-frequency fusion weight The visible light high-frequency coefficient and the infrared light high-frequency coefficient are fused to obtain a fused high-frequency coefficient. : 。 6. A visual enhancement system based on AR glasses, characterized in that, The system includes: The preprocessing module is used to acquire a target visible light image and a target infrared light image through AR glasses, and to preprocess the target visible light image and the target infrared light image respectively to obtain a processed visible light image and a processed infrared light image. The registration module is used to perform pyramid image registration on the processed visible light image and the processed infrared light image respectively, so as to obtain a visible light registered image and an infrared light registered image; The transformation module is used to perform image transformation decomposition on the visible light registration image and the infrared light registration image respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient. The fusion module is used to fuse the visible light low-frequency coefficient and the infrared light low-frequency coefficient to obtain a fused low-frequency coefficient, and to fuse the visible light high-frequency coefficient and the infrared light high-frequency coefficient to obtain a fused high-frequency coefficient. The inverse transform module is used to perform inverse image transform decomposition on the fused low-frequency coefficients and the fused high-frequency coefficients to obtain an enhanced visual image.

7. The visual enhancement system based on AR glasses according to claim 1, characterized in that, The registration module is specifically used for: The processed visible light image and the processed infrared light image are respectively subjected to four-level multi-resolution decomposition to obtain visible light pyramid images with decreasing resolution. Infrared pyramid images : ； ， ； In the formula, For the first Layered pyramid images in Pixel value at that location, For standard Gaussian kernels, For local coordinate indices within the Gaussian kernel, This is a visible light image of the pyramids. Infrared pyramid image; The initial transformation parameters between the two images are calculated starting with the lowest resolution visible light pyramid image and infrared light pyramid image. : ； In the formula, For mutual information calculation, For the first Transformation parameters for layer estimation; The first Initial transformation parameters of the layer Passed to the The layers are used as initial values ​​to optimize the transformation parameters, in order to obtain updated transformation parameters. : ； In the formula, , The first Visible light pyramid images and infrared light pyramid images of the layers. For the first Transformation parameters for layer estimation; The optimization process of the transformation parameters is repeated until the highest resolution pyramid layer is reached to obtain the final transformation parameters. Based on the final transformation parameters, image registration is performed to obtain a visible light registered image and an infrared light registered image.

8. The visual enhancement system based on AR glasses according to claim 6, characterized in that, The transformation module is specifically used for: Obtain the first and second filter banks in NSPFB, and determine the first filter based on the first filter bank. : ； In the formula, These are the low-pass and high-pass filters in the first filter bank, respectively. The second filter is determined based on the second filter bank. : ， ； In the formula, These are the sector, checkerboard, and parallelogram filters in the second filter bank, respectively. The visible light registration image and the infrared light registration image are processed by the first filter and the second filter respectively to obtain the visible light low-frequency coefficient, visible light high-frequency coefficient, infrared light low-frequency coefficient, and infrared light high-frequency coefficient.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the visual enhancement method based on AR glasses as described in any one of claims 1 to 5.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the visual enhancement method based on AR glasses as described in any one of claims 1 to 5.

Images