A multispectral image fusion binocular optical waveguide display intelligent digital twin glasses

Abstract

The embodiment of the application provides a kind of multispectral image fusion binocular light waveguide display intelligent digital twin glasses, through target contour fusion, scene can be fused and displayed with the imaging contour of different waveband spectral camera while retaining the perspective function of light waveguide glasses, distinguish using color and brightness;And using target recognition algorithm, specific type target is fused into the display image in a specific color way;It has visual perception, depth perception and temperature perception ability, and has the ability to access various image and video acquisition terminals, visible light cameras, infrared cameras and other spectral cameras of different wavebands can be selected to access;It has the ability to access external information, such as weapon equipment fire control, GIS, map, audio, and can be driven and displayed in character and graphic mode.

Description

Technical Field

[0001] This invention relates to the field of smart glasses imaging technology, and more particularly to a multispectral image fusion binocular waveguide display smart digital twin glasses. Background Technology

[0002] With the development of spectral imaging technologies such as ultraviolet, visible light, low light, near-infrared, infrared, and terahertz, human inventions of optoelectronic products have become increasingly abundant. For example, the night vision glasses disclosed in Chinese design patent CN307019834S can achieve infrared thermal imaging night vision. Although it does not require the use of hands, naked-eye vision is lost during use, and the glasses must be removed to restore visual perception. The infrared thermal imager disclosed in Chinese design patent CN306111713S achieves four-light fusion multi-functional handheld ranging, but requires both hands to use normally. The binocular night vision device disclosed in Chinese design patent CN306979706S has a head-mounted mode, but glasses need to be connected to the eyepiece to observe the image during use, which also results in the loss of naked-eye vision and external perception.

[0003] In existing technologies, when using a telescope, the user first needs to bring their eye close to the eyepiece, then move the telescope until the target object is observed within its observation area. The telescope's focal length is then adjusted to magnify the image of the target object to a reasonable size before observing it. However, these technologies lack voice control, cannot free the user's hands, require manual operation, and completely consume visual energy. They also lack pixel-level fusion and depth perception capabilities; they lack the ability to automatically detect, identify, and track targets using artificial intelligence; they cannot access and analyze external information; and they lack the ability to construct digital twin models using multiple sensors and information equipment. Summary of the Invention

[0004] This invention provides a multispectral image fusion binocular waveguide display smart digital twin glasses. Target contour fusion can retain the see-through function of the waveguide glasses while displaying the scene by fusing the imaging contours of cameras with different spectral bands, and distinguishing them by color and brightness; and using a target recognition algorithm to fuse specific types of targets into the displayed image with specific colors and other methods.

[0005] In a first aspect, embodiments of the present invention provide a multispectral image fusion binocular waveguide display smart digital twin glasses, including a multispectral image data acquisition unit, a waveguide display and voice transmission unit, and an information data intelligent processing integrated control unit;

[0006] The multispectral image data acquisition unit is used to acquire spectral images of multiple bands and transmit them to the information data intelligent processing integrated control unit.

[0007] The information data intelligent processing integrated control unit is used to perform target detection and contour extraction on spectral images of multiple bands respectively, and to perform multispectral fusion on the image contours of the extracted spectral images of multiple bands.

[0008] The optical waveguide display and voice transmission unit is used to display contour information and image information after multispectral fusion based on optical waveguide display technology; and to output voice prompt signals for the glasses.

[0009] Preferably, it also includes an external information input unit, which is used to connect external non-spectral imaging information to use the external non-spectral imaging information as an important information source for the digital twin. Depending on the type of external non-spectral imaging information connected by the external information input unit, it switches to a pre-matched digital twin model.

[0010] Preferably, it also includes a voice input unit, which is used to collect sound signals and transmit them to the information data intelligent processing integrated control unit;

[0011] The information data intelligent processing integrated control unit includes a voice recognition module, which receives the sound signal transmitted by the voice input unit, performs voiceprint decryption and recognition analysis, and converts it into control commands that can be executed by the glasses.

[0012] Preferably, the information data intelligent processing integrated control unit includes an information analysis module, an image receiving module, and a driving display module;

[0013] The information analysis module is connected to the external information input unit and is used to receive external information, including fire control information, command information, GIS information, and map information.

[0014] The image receiving module is connected to the multispectral image data acquisition unit and is used to receive spectral images of multiple bands transmitted by the multispectral image data acquisition unit. The spectral images include infrared spectral images and visible spectral images.

[0015] The driving display module is connected to the optical waveguide display and voice transmission unit, and is used to drive the optical waveguide display and voice transmission unit to display the contour information and image information after multispectral fusion.

[0016] Preferably, the information data intelligent processing integrated control unit further includes an image fusion module and an AI computing module, wherein the image fusion module and the AI ​​computing module are connected to the image receiving module;

[0017] The image fusion module is used to fuse received spectral images of multiple bands.

[0018] The AI ​​computing module is used to perform visual perception computing, depth perception computing, and temperature perception computing to determine people and objects in the visual scene, as well as the distances between them; and to perform digital twin reconstruction of spectral images of multiple bands and external non-spectral imaging information obtained by the information analysis module, so as to integrate and process multi-source information to form digital information that is acquired and updated synchronously.

[0019] Preferably, the image fusion module is specifically used to identify the registration coordinate feature points using the Accelerated Robust Feature SURF feature point selection method; pair the feature points of the visible light image and the infrared image according to the information features in the scene to form coordinate pairs; and calculate the registration parameters of the visible light spectral image and the infrared spectral image using affine transformation.

[0020] Based on the gradient operator, the visible light image contour and infrared light image contour are detected using visible light spectrum and infrared light spectrum, and the visible light image contour and infrared light image contour are registered using the registration parameters of the visible light spectrum image and the infrared light spectrum image.

[0021] The contours of visible light images and infrared light images are fused based on the nonmaximum suppression method.

[0022] Preferably, the AI ​​computing module is specifically used to detect, classify, track, identify, and analyze people and objects in the scene based on deep convolutional neural network algorithms; calculate the distance between people and objects in the scene based on dual-camera stereo imaging technology; and determine the temperature distribution state in the scene based on infrared thermal imaging technology, and determine the highest temperature target and the lowest temperature target in the scene.

[0023] Preferably, the information data intelligent processing integrated control unit further includes a menu interaction module and a rendering processing module;

[0024] The menu interaction module is used to control the glasses, generate menu display changes, provide feedback to the eyes, and confirm the voice control results;

[0025] The rendering module is used to perform depth calculations based on stereo vision technology, and to achieve a three-dimensional display effect.

[0026] This invention provides a multispectral image fusion binocular waveguide display smart digital twin glasses. Through target contour fusion, it retains the see-through function of the waveguide glasses while displaying the scene using the fused imaging contours of different spectral cameras, distinguishing them by color and brightness. It also utilizes a target recognition algorithm to fuse specific types of targets into the displayed image using specific colors. The glasses possess visual perception, depth perception, and temperature perception capabilities, and can connect to various image and video acquisition terminals, including visible light cameras, infrared cameras, and other spectral cameras of different bands. Furthermore, they can access external information such as weapon fire control systems, GIS, maps, and audio, driving the display in character and graphic formats. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A block diagram of a multispectral image fusion binocular waveguide display intelligent digital twin glasses system according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of a specific contour fusion algorithm according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] In the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0032] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a system, product, or device that includes a series of components or units is not limited to the listed components or units, but may optionally include unlisted components or units, or may optionally include other components or units inherent to such products or devices. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] In existing technologies, when using a telescope, the user first needs to bring their eye close to the eyepiece, then move the telescope until the target object is observed within its observation area. The telescope's focal length is then adjusted to magnify the image of the target object to a reasonable size before observing it. However, these technologies lack voice control, cannot free the user's hands, require manual operation, and completely consume visual energy. They also lack pixel-level fusion and depth perception capabilities; they lack the ability to automatically detect, identify, and track targets using artificial intelligence; they cannot access and analyze external information; and they lack the ability to construct digital twin models using multiple sensors and information equipment.

[0035] Therefore, embodiments of the present invention provide a multispectral image fusion binocular waveguide display smart digital twin glasses. Through target contour fusion, it can retain the see-through function of the waveguide glasses while displaying the scene using the fused imaging contours of different spectral cameras, distinguishing them by color and brightness. Furthermore, it utilizes a target recognition algorithm to fuse specific types of targets into the displayed image using specific colors, etc. It possesses visual perception, depth perception, and temperature perception capabilities, and can connect to various image and video acquisition terminals, including visible light cameras, infrared cameras, and other spectral cameras of different bands. It also has the ability to access external information, such as weapon fire control systems, GIS, and maps, driving the display in character and graphic formats. The following will elaborate and describe this through several embodiments.

[0036] Figure 1This invention provides a multispectral image fusion binocular waveguide display smart digital twin glasses, including a multispectral image data acquisition unit, an external information input unit, a voice input unit, a waveguide display and voice transmission unit, and an information data intelligent processing integrated control unit;

[0037] The multispectral image data acquisition unit is used to acquire spectral images of multiple bands and transmit them to the information data intelligent processing integrated control unit. In this embodiment, the multispectral image data acquisition unit is equipped with a variety of image and video acquisition terminals, and spectral cameras of different bands such as visible light cameras, low-light cameras, short, medium and long-wave infrared cameras, ultraviolet cameras, radar, and sonar can be selected for access.

[0038] An external information input unit is used to connect to external non-spectral imaging information, using this information as a crucial information source for the digital twin. Depending on the type of external non-spectral imaging information connected to the external information input unit, it switches to a pre-matched digital twin model. The external information input unit connects to external non-spectral imaging information, serving as a vital information source for the digital twin. Different digital twin models can be switched depending on the type of external information input unit. The external information input unit has the capability to connect to various image and video acquisition terminals, including visible light cameras, infrared cameras, and other spectral cameras of different wavelengths. It also has the ability to access external information, such as weapon fire control systems, GIS, and maps, driving the display in character and graphic formats.

[0039] The voice input unit is used to collect sound signals and transmit them to the information data intelligent processing integrated control unit. The voice input unit uses voice input to replace buttons, which can free up the hands to operate other supporting devices, and has flexibility and encryption.

[0040] The waveguide display and voice transmission unit is used to display multispectral fusion contour information and image information based on waveguide display technology; and to output voice prompts for the glasses. The lenses of the display glasses utilize waveguide display technology, employing display devices such as OLED, LCD, QLED, mini LED, and micro LED to illuminate the waveguide lens grating, displaying multispectral fusion contour information, image information, external input information, etc.; the display can also be turned off by voice control as needed, restoring the lenses to a transparent, naked-eye viewing state. The voice output is via an external headset, which converts the output information into a voice signal and outputs it to the wearer.

[0041] The information data intelligent processing integrated control unit is used to perform target detection and contour extraction on spectral images of multiple bands respectively, and to perform multispectral fusion on the image contours of the extracted spectral images of multiple bands.

[0042] In this embodiment, the intelligent information data processing integrated control unit includes a voice recognition module, an information analysis unit, an image receiving module, a display driving module, an image fusion module, an AI computing module, a menu interaction module, and a rendering processing module. The voice recognition module receives the sound signal transmitted by the voice input unit, performs voiceprint decryption and recognition analysis, and converts it into control commands executable by the glasses, eliminating the need for manual or button operation. The built-in voice recognition module allows for voice control of display / sleep mode switching, different display modes, different display content switching, menu control, etc., without manual operation.

[0043] The intelligent data processing and integrated control unit further includes an image fusion module and an AI computing module, which are connected to the image receiving module. The image receiving module is connected to the multispectral image data acquisition unit and is used to receive spectral images of multiple bands transmitted by the multispectral image data acquisition unit. These spectral images include infrared and visible light spectral images, and can be expanded to include spectral imaging devices or sensing devices of other bands. The interface can employ wireless or wired connection transmission. The image fusion module is used to fuse the received spectral images of multiple bands, retaining useful information suitable for display.

[0044] Specifically, such as Figure 2 As shown, the image fusion module is specifically used to confirm the registration coordinate feature points using the Accelerated Robust Feature Selection (SURF) method; pair feature points of the visible light image and infrared image according to the information features in the scene to form coordinate pairs; calculate the registration parameters of the visible light spectral image and infrared spectral image using affine transformation; detect the visible light image contour and infrared image contour of the visible light spectrum and infrared spectrum based on the gradient operator, and detect targets in the visible light image and infrared image respectively using a deep convolutional neural network to obtain the bounding rectangle coordinates of the target of interest. Using the registration parameters of the visible light spectral image and infrared spectral image, the visible light image contour and infrared image contour are registered, and the size is normalized to a suitable display resolution; the visible light image contour and infrared image contour are fused based on the non-maximum suppression method; the bounding rectangle region of the detected target is used to define the key area of ​​the contour and highlight it to highlight the target; non-target areas are displayed with the background color.

[0045] Target contour fusion can display the scene by fusing the imaging contours of cameras with different spectral bands while preserving the see-through function of the optical waveguide glasses, and distinguish them by color and brightness; and use target recognition algorithms to fuse specific types of targets into the displayed image with specific colors and other methods.

[0046] The first function of the AI ​​computing module is to realize visual perception computing, depth perception computing, and temperature perception computing. Visual perception utilizes deep convolutional neural network algorithms to detect, classify, track, identify, and analyze objects such as people and vehicles in a scene. This visual perception algorithm enables the glasses in this embodiment of the invention to have visual perception capabilities, that is, the ability to visually "see" and "interpret" the surrounding environment. For example, the camera system connected to the processor can detect animals, plants, people, cars, and environmental objects in the scene being viewed by the camera, identifying people, vehicles, animals, and other objects in the scene. Depth perception uses dual-camera stereo imaging technology to calculate the distance to people, vehicles, and other objects in the scene. This depth perception algorithm enables the glasses in this embodiment of the invention to have depth perception capabilities, which is the ability of artificial intelligence to understand the distance of objects. In computer vision terminology, "depth" simply means "how far". Temperature perception uses infrared thermal imaging technology to calculate the temperature distribution in a scene, especially the detection and perception of the highest and lowest temperature targets in the scene. This temperature perception algorithm enables the glasses in this embodiment of the invention to have temperature perception capabilities, enabling infrared thermal imaging technology to "confirm" the difference in temperature distribution between the environment and the target.

[0047] The second function of the AI ​​computing module is to perform digital twin reconstruction of multispectral image information and various data obtained by the information analysis module. It integrates and processes multi-source information to form digital information that is acquired and updated synchronously, giving it a greater advantage over remote data acquisition terminals.

[0048] The information analysis module is connected to the external information input unit and is used to receive external information, including fire control information, command information, GIS information, and map information.

[0049] The driving display module connects to the optical waveguide display and voice transmission unit, and is used to drive the optical waveguide display and voice transmission unit to display the contour information and image information after multispectral fusion. It drives external screen displays and is compatible with OLED, LCD, QLED, mini LED, micro LED, etc.

[0050] The menu interaction module is used to control the glasses, generate menu display changes, provide feedback to the eyes, and confirm the voice control results; the wearer of the glasses controls the system through menu interaction, generates menu display changes, provides feedback to the eyes, and confirms the voice control results.

[0051] The glasses in this invention, unlike traditional observation and aiming telescopes which require handheld observation, free up the hands. When using traditional observation and aiming telescopes, the human eye loses visual perception of the environment because it is not facing the optical eyepiece. When the ambient light is good and does not affect naked-eye observation, the image and video display function can be turned off by voice control. When the ambient light is poor or used at night, different light source cameras can be selected by voice for multi-light source image fusion. Or, when there is heavy fog, the infrared camera can be selected, and the infrared fog-penetrating ability can be used to observe targets at a greater distance.

[0052] The rendering module is used to perform depth calculations based on stereo vision technology, and to achieve a three-dimensional display effect.

[0053] The various embodiments of the present invention can be combined arbitrarily to achieve different technical effects.

[0054] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0055] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A smart digital twin glasses with multispectral image fusion and binocular waveguide display, characterized in that, The system includes a multispectral image data acquisition unit, an optical waveguide display and voice transmission unit, and an intelligent information data processing integrated control unit. The multispectral image data acquisition unit acquires spectral images of multiple bands and transmits them to the intelligent information data processing integrated control unit. The intelligent information data processing integrated control unit performs target detection and contour extraction on the spectral images of multiple bands, and performs multispectral fusion on the image contours of the extracted spectral images of multiple bands. The optical waveguide display and voice transmission unit displays the contour information and image information after multispectral fusion based on optical waveguide display technology, and outputs voice prompts for the glasses. It also includes an external information input unit, which connects to external non-spectral imaging information to use this information as an important information source for the digital twin. Depending on the type of external non-spectral imaging information connected to the meta-connector, the system switches to a pre-matched digital twin model. The information data intelligent processing integrated control unit also includes an image fusion module and an AI computing module, which are connected to the image receiving module. The image fusion module is used to fuse received spectral images of multiple bands. The AI ​​computing module is used to perform visual perception calculations, depth perception calculations, and temperature perception calculations to determine people and objects in the visual scene, as well as the distances to them. It also performs digital twin reconstruction on spectral images of multiple bands and external non-spectral imaging information obtained by the information analysis module, integrating multi-source information to form synchronous acquisition and updating of digital information. The AI ​​computing module uses a deep convolutional neural network algorithm to detect, classify, track, identify, and analyze people and objects in the scene. Based on dual-camera stereo imaging technology, the distances to people and objects in the scene are calculated; Infrared thermal imaging technology is used to determine the temperature distribution in a scene, and to identify the highest and lowest temperature targets in the scene.

2. The multispectral image fusion binocular waveguide display smart digital twin glasses according to claim 1, characterized in that, It also includes a voice input unit, which is used to collect sound signals and transmit them to the information data intelligent processing integrated control unit; the information data intelligent processing integrated control unit includes a voice recognition module, which is used to receive the sound signals transmitted by the voice input unit, and to perform voiceprint decryption and recognition analysis, converting them into control commands that can be executed by the glasses.

3. The multispectral image fusion binocular waveguide display smart digital twin glasses according to claim 2, characterized in that, The intelligent information data processing and integrated control unit includes an information analysis module, an image receiving module, and a driving display module. The information analysis module is connected to the external information input unit and is used to receive external information, including fire control information, command information, GIS information, and map information. The image receiving module is connected to the multispectral image data acquisition unit and is used to receive spectral images of multiple bands transmitted by the multispectral image data acquisition unit, including infrared and visible light spectral images. The driving display module is connected to the optical waveguide display and voice transmission unit and is used to drive the optical waveguide display and voice transmission unit to display the contour information and image information after multispectral fusion.

4. The multispectral image fusion binocular waveguide display smart digital twin glasses according to claim 2, characterized in that, The image fusion module is specifically used to: identify registration coordinate feature points using the Accelerated Robust Feature Selection (SURF) method; pair feature points of the visible light image and infrared image according to information features in the scene to form coordinate pairs; calculate and obtain registration parameters of the visible light spectral image and infrared spectral image using affine transformation; detect the visible light image contour and infrared image contour of the visible light spectrum and infrared light spectrum based on the gradient operator; register the visible light image contour and infrared image contour using the registration parameters of the visible light spectral image and infrared light spectral image; and fuse the visible light image contour and infrared image contour based on the nonmaximum suppression method.

5. The multispectral image fusion binocular waveguide display smart digital twin glasses according to claim 2, characterized in that, The information data intelligent processing integrated control unit also includes a menu interaction module and a rendering processing module; the menu interaction module is used to control the glasses, generate menu display changes, provide feedback to the eyes, and confirm the voice control results; the rendering processing module is used to perform depth calculation based on stereo vision technology, and realize three-dimensional display effects through the rendering processing module.

Images