Live-action target and information enhancement system and method based on optics

Abstract

The invention discloses a live-action target and information enhancement system and method based on optics. Four through holes are formed in one side wall of the shell, the four through holes are divided into an upper row and a lower row, the first visible light camera, the infrared emitter and the second visible light camera are arranged right opposite to the three through holes in the upper row respectively, and the first visible light camera and the second visible light camera are used for obtaining visible light images under different visual angles; the laser projection module is arranged opposite to one downward through hole, the invisible light imaging device is fixedly installed on the inner side wall of the shell, the infrared emitter is used for emitting infrared light to a preset target area, the infrared light enters the invisible light imaging device through the optical focusing module after being reflected by the target area, and then an invisible light image is generated. According to the invention, through collaborative innovation of infrared scanning modeling, a phase compensation algorithm, a double-path projection light path and an automatic correction mechanism, a high-precision and high-robustness live-action enhancement effect is realized, and the projection precision is remarkably improved on a non-planar carrier.

Description

Technical Field

[0001] This invention belongs to the field of reality enhancement technology, and particularly relates to an optical-based reality target and information enhancement system and method. Background Technology

[0002] Augmented reality (AR) technology uses virtual objects to "enhance" the display of real-world scenes. It boasts advantages such as high realism and low modeling workload, making it widely applicable in fields like engineering design, modern displays, healthcare, military, education, entertainment, and tourism. However, several issues remain: most AR devices rely on head-mounted displays, resulting in poor user comfort and difficulty in sharing the field of view with others; in terms of interactivity, most AR devices passively receive information, failing to actively integrate user ideas with the real-world environment, leading to poor interactivity with the real world.

[0003] From a methodological perspective, augmented reality generally follows a process of "perception / localization → geometric registration → rendering / projection → error compensation and feedback." The main implementation paths include two types of head-mounted display methods: Video See-Through (VST) and Optical See-Through (OST), and Spatial Augmentation / Projection (SAR) methods. Projection methods naturally support multi-user sharing and wear-free operation, but they rely more heavily on the geometric calibration and photometric (color and brightness) compensation of the "projection-camera" system. They require surface mapping and color and brightness modeling to offset distortions caused by non-uniform materials and ambient light, and to achieve projection compensation on complex surfaces. Simultaneously, occlusion must be handled to ensure the correct relationship between the virtual and real elements. Recent research trends have shifted from stepwise geometric / photometric correction to end-to-end full compensation and real-time occlusion processing for dynamic scenes to improve robustness and practicality. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, this invention proposes an optical-based real-scene target and information enhancement system and method.

[0005] The technical solution adopted in this invention is: This invention includes a housing and a first visible light camera, a second visible light camera, an infrared emitter, an optical focusing module, an invisible light imaging device, and a laser projection module installed within the housing. Four through holes are provided on one side wall of the housing, arranged in two rows. The first visible light camera, infrared emitter, second visible light camera, and invisible light imaging device are all fixedly installed on the same inner side wall of the housing. The first visible light camera, infrared emitter, and second visible light camera are respectively positioned opposite the three upper through holes. The first and second visible light cameras are used to acquire visible light images from different perspectives. The laser projection module is positioned opposite one of the lower through holes. The invisible light imaging device is fixedly installed on the inner side wall of the housing. The optical focusing module is fixedly connected to the invisible light imaging device. The infrared emitter emits infrared light towards a preset target area. After being reflected by the target area, the infrared light passes through the optical focusing module and enters the invisible light imaging device, thereby generating an invisible light image.

[0006] It also includes a control processing module, which is fixedly installed on the inner side wall of the housing. The control processing module is electrically connected to the first visible light camera, the second visible light camera, the infrared emitter, the invisible light imaging device, and the laser projection module, respectively. It is used to control the infrared emitter to emit infrared light towards the target area, process the visible light image and the invisible light image to generate enhanced information, and then control the laser projection module to project the enhanced information onto the target area.

[0007] The invisible light imaging device is an infrared camera.

[0008] The control processing module includes: An image preprocessing model is used to receive invisible light images acquired by an invisible light imaging device, preprocess them, and obtain preprocessing results. A contour extraction model is used to extract contour images from invisible light images, preprocessed results, and two visible light images. A laser projection generation model is used to generate laser projection images based on contour images, preprocessing results, and two visible light images. A fitting and self-calibration model is used to fit and self-calibrate two visible light images, an invisible light image, and a projection image to obtain enhanced information for calibrating the laser projection image.

[0009] The laser projection module includes a visible light laser module, two total internal reflection prisms, two beam-splitting mirror groups, and two optical galvanometer modules. The visible light laser module includes three light sources. The first beam-splitting mirror group includes three semi-transparent and semi-reflective mirrors, and the second beam-splitting mirror group includes three highly reflective mirrors. The visible light laser module has three sets of light sources emitting red, green, and blue lasers respectively. These three colors of laser light are incident on the three semi-transparent and semi-reflective mirrors in the first beam-splitting and turning mirror group, where they are reflected and transmitted respectively. The reflected light from the three semi-transparent and semi-reflective mirrors in the first beam-splitting and turning mirror group is reflected by the first total internal reflection prism and then incident on the first optical galvanometer module. After being reflected by the first optical galvanometer module, it is projected onto the target area. The transmitted light from the three semi-transparent and semi-reflective mirrors in the first beam-splitting and turning mirror group is incident on the three highly reflective turning mirrors in the second beam-splitting and turning mirror group, where it is reflected and transmitted respectively. The reflected light from the three highly reflective turning mirrors in the second beam-splitting and turning mirror group is reflected by the second total internal reflection prism and then incident on the second optical galvanometer module. After being reflected by the second optical galvanometer module, it is projected onto the target area. The visible light laser module is electrically connected to the control and processing module and is used to control the wavelength of the laser emitted by the visible light laser module. The optical galvanometer module is electrically connected to the control and processing module and is used to drive the galvanometer to rotate and change the optical path.

[0010] Optical-based methods for augmenting real-world targets and information include the following steps: S1. The first visible light camera and the second visible light camera respectively acquire the first visible light image and the second visible light image, and the invisible light imaging device acquires the invisible light image. S2. Input the invisible light image into the image preprocessing model for preprocessing to obtain the preprocessing result. Then, input the invisible light image, the preprocessing result, the first visible light image, and the second visible light image into the contour extraction model for contour extraction to obtain the contour image. S3. Input the contour image, preprocessing result, first visible light image and second visible light image into the laser projection generation model to obtain the laser projection image; S4. The laser projection image is sent to the laser projection module for projection to obtain the projection image. The projection image, the invisible light image, the first visible light image and the second visible light image are input together into the fitting and self-calibration module to obtain the enhancement information. S5. After feeding back the enhanced information to the laser projection module, the image is projected to obtain the calibrated projection image.

[0011] The preprocessing in step S2 includes denoising and contrast enhancement of the input image, and obtaining the names and location parameters of objects in the image based on object detection and classification algorithms.

[0012] The contour extraction in step S2 specifically involves: 1) Extract features from the invisible light image to obtain a multi-scale feature map of the invisible light image; 2) Extract features from the first visible light image and the second visible light image respectively to obtain the first visible light feature map and the second visible light feature map, and then stitch the first visible light feature map and the second visible light feature map together to obtain the fused visible light feature map; 3) Based on the preprocessing results, target region cropping and size alignment operations are performed on the multi-scale feature map and the fused visible light feature map to obtain invisible light local features and visible light local features. Then, the invisible light local features and visible light local features are fused to obtain the target fused feature map. 4) Extract contours from the target fusion feature map to obtain several ordered contour points, and all the ordered contour point columns together constitute the contour image.

[0013] The laser projection generation model is specifically processed according to the following steps: a) Interpolate and smooth the contour image to obtain the basic trajectory segment used for galvanometer scanning; b) Extract scene features from the first visible light image and the second visible light image to obtain scene information; c) Align the basic trajectory segments, scene information, preprocessing results, and preset projection control parameters with the target region, and stitch them together in the channel dimension to obtain target-level projection features, and then process them to obtain a laser projection image.

[0014] In step S4, the fitting and self-calibration are processed according to the following formula: in, Indicates coordinates as Laser projection intensity at the location, This represents the average of the maximum and minimum laser projection intensities. Indicates the intensity modulation factor. It is the phase change of the captured invisible light image. It is a phase compensation term.

[0015] The beneficial effects of this invention are as follows: (1) The present invention proposes a system and method based on optical and real-world target and information enhancement. The system integrates an optical image acquisition module, an information processing module and a laser projection module, and can realize functions such as target contour enhancement, danger warning and interactive information guidance.

[0016] (2) The method proposed in this invention adopts a two-string-one-parallel neural network architecture. By decoupling functions, target recognition, contour extraction and projection generation are processed in stages, reducing the complexity of a single model and achieving millisecond-level response. The command parser converts user input into mode parameters (such as priority, refresh rate, pattern type, color constraints, etc.). The projection generation module uses this to link the strategy library to complete pattern arrangement and scanning timing planning, and outputs the galvanometer trajectory and RGB gating signal. During mode switching, low latency and high robustness are maintained through delay compensation and incremental updates.

[0017] (3) In the method proposed in this invention, the contour extraction part uses both visible light images and invisible light images as input. The visible light images are used for information enhancement feedback and correction, while the invisible light images are used to collect the structural information of the target, which effectively solves the problem that traditional methods such as the Canny operator are easily interfered with in complex backgrounds.

[0018] (4) In the system proposed in this invention, a high-performance AI computing board is used as the information processing module. The microcontroller and its peripheral circuit drive the laser projection module, which simultaneously processes the upper-level image information and drives the lower-level actuator, thus taking into account both efficient image processing and precise control and rapid response of the laser projection peripheral.

[0019] (5) In the system proposed in this invention, through the collaborative innovation of infrared scanning modeling, phase compensation algorithm, dual-path projection optical path and automatic correction mechanism, a high-precision and high-robust real scene enhancement effect is achieved, which significantly improves the projection accuracy on non-planar carriers. Attached Figure Description

[0020] Figure 1 This is a system block diagram of an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the system flow according to an embodiment of the present invention.

[0022] Figure 3 This is a left-side view of an embodiment of the present invention.

[0023] Figure 4 This is a front view of an embodiment of the present invention.

[0024] Figure 5 This is a front view of an embodiment of the present invention.

[0025] Figure 6 This is a rear view of an embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram of the optical path of the laser projection module in this invention.

[0027] The components include: 1. Display screen; 2. Lithium battery; 3. K230 information processing host computer; 4. Sub-computer; 5. Power expansion board; 6. Optical galvanometer module; 7. Beam-splitting steering mirror group; 8. Visible light laser module; 9. Charging port; 10. Visible light camera; 11. Front panel; 12. Left side panel; 13. Frame inclined groove; 14. Right side panel; 15. Back panel. Detailed Implementation

[0028] The invention will be further described below with reference to the accompanying drawings. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this disclosure. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly used in the art to which this invention pertains.

[0029] Specific embodiments of the present invention are as follows: This embodiment includes a housing and a first visible light camera, a second visible light camera, an infrared emitter, an optical focusing module, an invisible light imaging device, and a laser projection module installed inside the housing. Four through holes are provided on one side wall of the housing, arranged in two rows. The first visible light camera, infrared emitter, second visible light camera, and invisible light imaging device are all fixedly installed on the same inner side wall of the housing. The first visible light camera, infrared emitter, and second visible light camera are respectively positioned opposite the three upper through holes. The first and second visible light cameras are used to acquire visible light images from different perspectives. The laser projection module is positioned opposite one of the lower through holes. The invisible light imaging device is fixedly installed on the inner side wall of the housing. The optical focusing module is fixedly connected to the invisible light imaging device. The infrared emitter emits infrared light towards a preset target area. After being reflected by the target area, the infrared light passes through the optical focusing module and enters the invisible light imaging device, thereby generating an invisible light image. The infrared emitter generates infrared light using an infrared light generator, which then passes through an optical focusing module to narrow the optical path angle and increase the intensity of the infrared light. The infrared light is then scanned and detected, and finally, an invisible light imaging device acquires an invisible light image.

[0030] The visible light camera 10 includes a first visible light camera and a second visible light camera.

[0031] Figure 1 This is a schematic diagram of an optical-based method and system for augmenting real-world targets and information, as described in an embodiment of this application. Figure 1 As shown in the figure, the optical-based real-scene target and information enhancement system of this application includes an infrared information processing module, a visible light information processing module, and a laser projection module. The infrared and visible light information is acquired from the target, processed in K230, and fed back to the laser projection module for projection onto the object.

[0032] Figure 2This is a flowchart illustrating an optical-based method and system for augmenting real-world targets and information, as described in an embodiment of this application. Figure 1 As shown, the optical-based real-scene target and information enhancement method and system of this application includes an optical image acquisition module, an information processing module and a laser projection module, which form a closed-loop workflow through data interaction and control feedback.

[0033] It also includes a control processing module, which is fixedly installed on the inner side wall of the housing. The control processing module is electrically connected to the first visible light camera, the second visible light camera, the infrared emitter, the invisible light imaging device, and the laser projection module. It is used to control the infrared emitter to emit infrared light towards the target area, and to process the visible light images acquired by the first and second visible light cameras and the invisible light images acquired by the invisible light imaging device to generate enhanced information, and then control the laser projection module to project the enhanced information onto the target area.

[0034] Invisible light images are used to acquire target structure information, while visible light images are used for information enhancement feedback and correction. The control processing module first performs target recognition on the image information, extracts the contour of the acquired target, and then generates output geometric shapes such as points, lines, surfaces, and colors to be projected based on the target structure and characteristics. The laser projection module projects the output image onto the target surface through a laser projector. The left and right channels of the binocular camera acquire visible light images img1 and img2, respectively, while the infrared camera acquires the invisible light image img. Visible light images are used for texture feature extraction and projection effect feedback, while invisible light images are used for target structure recognition and depth calculation.

[0035] The invisible light imaging device is an infrared camera.

[0036] The system design in this embodiment is as follows: Figure 5 The image shows a front view of the system. Figure 6 The rear view is shown. The overall system casing consists of four panels and a frame with a sloping groove 13. The front panel 11 has a 5cm diameter laser emission hole in the center, with two 2cm diameter camera acquisition holes symmetrically distributed on both sides of the emission hole. Below the left panel 12, there is a 3.5cm diameter opening for fixing the system's main electrical switch; to the right of the switch is a rectangular groove for exposing the SD card slot on the circuit board; above the switch are two vertically arranged rectangular grooves for installing a Type-C interface for connecting to the host computer 3 for debugging and for installing a toggle switch to switch debugging modes. A 45° sloping surface at the upper rear of the frame has corresponding mounting holes for fixing the display screen 1. The back panel 15 also has corresponding mounting holes for connecting and fixing to the internal circuit board. The right panel 14 has a 1cm diameter opening at its lower left to expose the system's charging port 9.

[0037] The control processing module includes: An image preprocessing model is used to receive invisible light images acquired by an invisible light imaging device, preprocess them, and obtain preprocessing results. A contour extraction model is used to extract contour images from invisible light images, preprocessed results, and two visible light images. A laser projection model is used to generate a laser projection image based on a contour image, preprocessing results, and two visible light images. A fitting and self-calibration model is used to fit and self-calibrate two visible light images, an invisible light image, and a projection image to obtain enhanced information for calibrating the laser projection image.

[0038] The information processing module integrates a three-layer intelligent processing architecture: the image preprocessing model performs target detection and classification on non-visible light images, outputting the target name and spatial location; the contour extraction model fuses visible light binocular images and infrared images, and generates the target contour and its three-dimensional coordinate parameters based on the preprocessing results; the laser projection model generates laser projection data containing point / line / surface geometry and RGB color encoding according to target attributes, scene requirements and user instructions, and dynamically adjusts the projection intensity and position using the infrared structured light phase change compensation formula through the fitting and self-calibration calculation unit.

[0039] The laser projection module includes a visible light laser module 8, two total reflection prisms, two beam-splitting and steering mirror groups 7, and two optical galvanometer modules 6. The visible light laser module 8 includes three light sources, and one beam-splitting and steering mirror group 7 includes three identical semi-transparent and semi-reflective mirrors or three identical high-reflectivity steering mirrors. The three light sources in the visible light laser module 8 emit red, green, and blue lasers respectively. These three colors of laser light are incident on the first beam-splitting mirror group (a non-polarizing 50 / 50 beam-splitting prism), where they undergo reflection and transmission: the reflected light is reflected internally by the first right-angle prism and then incident on the first optical galvanometer, from which it is projected onto the target area; the transmitted light continues into the second stage. The second beam-splitting mirror group is a high-reflectivity galvanometer that reflects the three transmitted beams from the first stage. These beams are then reflected internally by the second right-angle prism and incident on the second optical galvanometer, from which they are projected onto the target area.

[0040] The three colors of laser light are incident on the first beam-splitting mirror group in a one-to-one manner, with one color of light incident on one semi-transparent and semi-reflective mirror. Similarly, the transmitted light enters the second beam-splitting mirror group, with one transmitted light corresponding to one highly reflective steering mirror.

[0041] The visible light laser module 8 is electrically connected to the control and processing module and is used to control the wavelength of the laser emitted by the visible light laser module 8. The optical galvanometer module 6 is electrically connected to the control and processing module and is used to drive the galvanometer to rotate and change the optical path.

[0042] The visible light laser module 8 includes three semiconductor lasers (red, green, and blue), each with its own collimating / beam expander and beam combining optics, as well as a laser constant current drive and temperature control submodule. This submodule is electrically connected to the control processing module and is used to control the wavelength and stable output power of the laser emitted by the visible light laser module 8. The optical galvanometer module 6 includes an X-axis galvanometer and a Y-axis galvanometer, a photoelectric position detector, and a servo drive amplifier circuit. This servo drive amplifier circuit is electrically connected to the control processing module and is used to drive the galvanometer to rotate and change the optical path according to control signals, thereby achieving deflection and scanning of the laser beam in the X / Y directions.

[0043] Figure 4 The second part shown illustrates the front view of the system. The laser beam, emitted by the semiconductor emitter, is reflected by a semi-transparent prism and transmitted upwards. A laser galvanometer alters the X and Y components of the light path, resulting in a forward-facing beam after two reflections. The laser projection module employs a dual-channel galvanometer scanning architecture. The laser emitted by the RGB laser module is split by the semi-transparent prism and then deflected along the X and Y axes by two sets of two-dimensional galvanometer modules, simultaneously projecting two different patterns (as illustrated by the window outline and the red warning box in the figure). The ESP32 lower-level machine 4 is responsible for actuator control, and its integrated data storage module saves the galvanometer trajectory file.

[0044] The visible light laser module 8 emits light in a direction parallel to the horizontal direction of the beam-splitting mirror. The optical galvanometer module 6, with two reflective surfaces in two dimensions, forms one incident light direction parallel to the vertical direction of the beam-splitting mirror. The optical galvanometer module 6 also has another incident light direction formed by two reflective surfaces in two dimensions, parallel to the horizontal direction of the beam-splitting mirror. The visible light laser module 8 emits a laser of a specified frequency. After passing through a semi-transparent mirror, the laser is split into two paths with angle deflections of 0 degrees and 90 degrees. These paths are then converted to a 90-degree path by a high-reflectivity galvanometer. The laser is then input into the optical galvanometer module 6, where it undergoes two corresponding deflections. Both laser paths are then emitted synchronously to the imaging area. The two laser paths cover different or the same area of ​​the target, simultaneously drawing different patterns. When only one laser path is needed, the other laser path can be deflected by the galvanometer to an angle that would otherwise be blocked from emission.

[0045] The system also includes a display screen 1, which is mounted on the upper surface of the housing and electrically connected to the control processing module for visualizing the data processed by the control processing module. In this embodiment, the system is powered by a 12.6V lithium battery 2. The overall housing of the system consists of four panels and a frame, which are formed by photopolymerization of LEDO6060 material. The frame is fixed to the four panels with M25mm self-tapping screws, and the internal circuit board is connected and fixed to the panels with M3 copper pillars.

[0046] In step S4, the laser projection module receives the projection data output from the laser projection model in step S3, performs calculation and timing encoding on the data, and generates driving signals for two X and Y galvanometers and R, G, and B three-color laser switching signals. Under the dual-optical-path structure of the visible light laser module 8, which is irradiated by the first optical galvanometer through the first beam-splitting mirror group and the first total internal reflection prism, and irradiated by the second optical galvanometer through the second beam-splitting mirror group and the second total internal reflection prism, the first and second optical galvanometers are driven to scan synchronously to form two projection patterns in the target area or to superimpose projections on the same area. During the galvanometer transition segment, blanking control is implemented on the laser to avoid motion blur, and the constant current drive and temperature control submodule ensures stable output of laser wavelength and optical power. The galvanometer is controlled in a closed loop by a servo drive amplifier circuit with position detection to achieve precise deflection and scanning in the X and Y directions. At the same time, the enhancement information output by the fitting and self-calibration module is superimposed in real time as position and intensity compensation to the galvanometer drive and laser intensity modulation, thereby outputting the calibrated projection image.

[0047] An optical-based method for augmenting real-world targets and information includes the following steps: S1. The system acquires several image combinations. Each image combination includes two visible light images and one invisible light image. That is, the first visible light camera and the second visible light camera acquire the first visible light image and the second visible light image, respectively, and the invisible light imaging device acquires the invisible light image. S2. Input the invisible light image into the image preprocessing model to preprocess and obtain the preprocessing result. Then, input the invisible light image, the preprocessing result, the first visible light image and the second visible light image into the contour extraction model to extract the contour and obtain the contour image. S3. Input the contour image, preprocessing result, first visible light image and second visible light image into the laser projection generation model to obtain the laser projection image; S4. The laser projection image is sent to the laser projection module for projection to obtain the projection image. The projection image, the invisible light image, the first visible light image and the second visible light image are input together into the fitting and self-calibration module to obtain the enhancement information. The enhancement information is compensation information used to calibrate the laser projection image, including: projection position deviation information, intensity adjustment information, and geometric mapping parameters; wherein the position deviation is obtained by the difference between the projected image and the real scene image, the intensity adjustment is based on the phase change of the infrared structured light, and the geometric mapping is determined by spherical fitting and affine transformation.

[0048] S5. After feeding the enhanced information back to the laser projection model, the image is projected to obtain the calibrated projection image.

[0049] The preprocessing in step S2 includes optional denoising and contrast enhancement of the input image, and obtaining the name and position parameters of the main objects in the image based on a real-time detection algorithm based on a convolutional neural network. The position parameters include at least the bounding rectangle, center point coordinates and / or pixel-level segmentation mask, i.e., determining the target region.

[0050] The target detection and classification algorithm is a real-time detection algorithm based on convolutional neural networks.

[0051] The contour extraction in step S2 specifically involves: 1) Extract features from the invisible light image to obtain a multi-scale feature map of the invisible light image; 2) Extract features from the first visible light image and the second visible light image respectively to obtain the first visible light feature map and the second visible light feature map, and then stitch the first visible light feature map and the second visible light feature map together to obtain the fused visible light feature map; 3) Based on the preprocessing results, target region cropping and size alignment operations are performed on the multi-scale feature map and the fused visible light feature map to obtain invisible light local features and visible light local features. Then, the invisible light local features and visible light local features are fused to obtain the target fused feature map. 4) Extract contours from the target fusion feature map to obtain several ordered contour points, and all the ordered contour point columns together constitute the contour image.

[0052] Step S2, contour extraction, includes: based on the preprocessed target name and location, performing edge detection and segmentation on the image within the corresponding region of interest to obtain the target's outer contour and optional inner contour; performing connected component filtering and contour tracking on the contour to obtain a closed contour; vectorizing, smoothing, and simplifying the closed contour; resampling according to the scanning order to generate a control point sequence; and converting it to the projection coordinate system using system geometric calibration parameters, outputting the contour data as the laser projection module. "Contour" refers to the aforementioned outer contour and optional inner contour, and "subject" refers to the main object in the image.

[0053] In step S3, the laser projection module includes a visible light laser module 8, two total internal reflection prisms, two beam-splitting mirror groups 7, and two optical galvanometer modules 6; the visible light laser module 8 includes three light sources; one beam-splitting mirror group 7 includes three identical semi-transparent and semi-reflective mirrors or three identical high-reflectivity steering mirrors.

[0054] In step S3, corresponding external commands can be added as needed to assist in various daily life tasks, such as driving navigation, visual impairment assistance, meeting assistance, and crowd control. The information processing mode includes additional information beyond the target outline information. Based on the target outline and its location, as well as the background scene and its location, necessary image, animation, and text information content is added.

[0055] Specifically, the contour extraction model uses raw invisible light images acquired by an infrared camera as its basis. The input is processed sequentially through multiple convolutional layers, batch normalization, activation, and downsampling operations, outputting multi-scale feature maps. It mainly carries information about the target's brightness distribution and rough shape; Image preprocessing link As input, the preprocessed model yields the category label, bounding box, and other results for each target. This serves as a priori for subsequent detailed reasoning only within the target region; Binocular visible light link with left and right visible light images As input, the data is fed into a convolutional backbone network with shared parameters to obtain feature maps rich in edge and texture details. And through feature stitching, a fused visible light feature containing texture and parallax information is formed. .

[0056] The model is based on the preprocessing results Will and The corresponding target region is cropped and aligned to a fixed size using regions of interest, resulting in local feature blocks for each target. Then splice them together along the channel dimension and pass through The convolution and attention modules complete feature fusion and output the fused features. .

[0057] Based on this, an encoder-decoder convolutional network is used to... The network performs progressive upsampling and convolution, and at the end, it directly outputs the ordered contour point sequence of the target in the laser projection coordinate system through a coordinate regression head. That is, the laser scanning coordinates and connection order of each point.

[0058] During the training phase, a registration dataset is constructed for the three input paths mentioned above: each sample simultaneously contains one frame of invisible light image. Registered binocular visible light images and in The target categories, bounding boxes, and high-precision ordered contour lines are manually labeled; combined with the camera projection geometry obtained from offline calibration, the contour points in the pixel coordinate system are fitted to the true contour point sequence in the laser projection coordinate system using least squares. , as a supervisory label. First, with It is labeled as a supervised, independently trained image preprocessing model, enabling it to output stable object detection results. Subsequently, during the training of the contour extraction model, and online generation The data is fed into the complete network, and after invisible light convolutional coding, binocular convolutional coding, region of interest alignment, and feature fusion, the encoder-decoder convolutional network directly regresses and predicts the ordered contour point sequence of each target in the laser projection coordinate system. The loss function is used to predict the contour point sequence. With true contour points Between The loss function is the primary factor, with additional small-weighted regularization terms based on contour smoothness, contour length, and topological consistency to suppress jagged edges and isolated points.

[0059] In step S3, the laser projection model receives contour data, object name, location information, and user commands to generate projection data. The laser projection model includes an application strategy library, which selects and combines at least one of the following strategies according to user commands to construct laser projection images for different applications: S3-1, Vehicle Navigation and Road Condition Alert Strategy: Perform edge tracking and prioritize the annotation of protrusions / potholes within the road area to generate the projected outlines of lane lines, curb lines, speed bump / pothole markers, and directional arrows; S3-2, Visual Impairment Assistance Strategy: Project the outer contour of surrounding obstacles first, highlight the closed contour or wireframe of nearby obstacles according to the safe distance threshold, and provide guide lines in passable areas; S3-3, Navigation Tips Strategy: Project labels, outlines, and access paths to designated targets or areas; S3-4 Warning Signage Strategy: Generate flashing or intermittent scanning warning outlines and boundary lines for hazardous sources or restricted areas.

[0060] The laser projection model generation process is as follows: a) Interpolate and smooth the contour image to obtain the basic trajectory segment used for galvanometer scanning; b) Extract scene features from the first visible light image and the second visible light image to obtain scene information; c) Align the basic trajectory segments, scene information, preprocessing results, and preset projection control parameters with the target region, and stitch them together in the channel dimension to obtain target-level projection features, and then process them to obtain a laser projection image.

[0061] Specifically, the laser projection model uses the ordered sequence of target contour points in the laser projection coordinate system output by the contour extraction model. Image preprocessing results The binocular visible light image of the current frame and user commands (i.e., the preset projection control parameters) are the input: Contour link for each Interpolation and smoothing are performed to generate a basic trajectory segment suitable for galvanometer scanning. ; Scene Link Pair Scene features are extracted using lightweight convolution and downsampling to distinguish road areas, obstacles, and background. Command chain will link user commands The target category is encoded as a control vector, representing the projection mode currently required, such as navigation guidance, visual impairment assistance, and hazard warning.

[0062] The three features are aligned at the target level and concatenated along the channel dimension before being fed into the policy sub-network. Several fully connected layers and attention structures generate the projection instruction vector for each target. This specifies whether the target's corresponding contour is highlighted, filled, or has an overlaid arrow, symbol, color, and blinking mode. Based on this, the end-stage timing encoding and power constraint module integrates all... With trajectory fragments Under the premise of satisfying the constraints of galvanometer speed, acceleration and visual persistence, the point series is resampled and sorted, and the scanning trajectory of the two galvanometers changing over time, as well as the synchronous RGB laser switch and intensity sequence, are output as "laser projection data" to directly drive the laser projection module.

[0063] Training process: Offline phase construction includes Corresponding preprocessing results Contour Truth Value and user commands The dataset was obtained by manually designing the desired projection effect in the laser projection coordinate system to obtain true projection data. ; Use a pre-trained contour extraction model to fix the output ,Will Input laser projection model, regression prediction projection data Aligned with trajectory coordinates and time sampling The loss is mainly due to the cross-entropy loss of the laser switching sequence, and regularization terms for velocity and acceleration, distance deviation between the projected contour and the original contour, and color and brightness smoothness are superimposed to ensure that the output satisfies hardware constraints while maintaining a clear and stable projection effect. After system deployment, the position and intensity deviation between the "actual projected image" and the theoretical projection result given by the fitting and self-calibration module are used to update some parameters or strategy coefficients of the laser projection model in small steps.

[0064] In step S4, the fitting and self-calibration are processed according to the following formula in the module: in, Represents projected coordinates Laser projection intensity at the location; The maximum value of the projection intensity within this local window. and minimum value The average value; Intensity modulation factor; The phase change is obtained by phase solving from an infrared structured light captured image (i.e., an invisible light image); This is a phase compensation term used to correct phase deviations introduced by system geometric errors, galvanometer timing / hysteresis, and uneven target surface topography and reflectivity. In step S4, the binocular visible light image is used to determine the location of the target area and establish camera coordinates and projection coordinates. The correspondence is established; based on this alignment, the actual phase distribution obtained from infrared structured light is compared with the theoretical projection model to estimate the difference. The aforementioned In step S5, the projection control is updated as an enhanced information feedback to achieve self-calibration.

[0065] In this embodiment, the information processing module, from a hardware perspective, includes a high-performance computing development board, a slave microcontroller, and its peripheral circuitry. The high-performance computing development board is responsible for processing upper-level image information, including stereo matching of binocular visible light images, fusion of visible and non-visible light images, and intelligent inference of target features. The slave microcontroller and its peripheral circuitry are responsible for processing lower-level actuator drives, including outputting analog-to-digital conversion signals for driving the laser galvanometer and logic level signals for driving the laser diode to emit light.

[0066] The microcontroller and its peripheral circuits include the microcontroller, a data storage module, a visible light emission driver module, a galvanometer driver module, and a power supply driver module. The microcontroller transmits and receives control signals; the data storage module stores the laser animation galvanometer trajectory control file; the visible light emission driver module controls the laser module to output lasers of various visible wavelengths through a switching circuit. The galvanometer driver module consists of a motion structure and a feedback structure. It controls the rotor movement through current magnitude and uses the feedback structure to achieve precise control of the galvanometer angle, changing the X-axis and Y-axis deflection of the laser projection. Due to the persistence of vision effect in the human eye, the information reflected from the laser projection on the target is retained on the retina for a period of time, allowing the laser's trajectory to form a laser image. As the image changes over time, it creates a laser animation. The power supply driver module performs level conversion, providing appropriate voltage levels to each module.

[0067] In the optical-based real-scene target and information augmentation method, some steps are implemented using artificial intelligence models. The pre-training and recognition processes of the artificial intelligence models are as follows: D1. Constructing the Training Dataset: Sampled images from both visible light and non-visible light acquisition modules are used as source material. Each source material unit contains two images acquired by the binocular visible light imaging module and one image acquired by the non-visible light imaging module. The pixel count of the three images is reduced to the same level, and the target's name and distance from the camera are labeled. Simultaneously, the target's outline information in the non-visible light image is hand-drawn. This dataset is then divided into a training set and a test set in an 8:2 ratio.

[0068] D2. Training Process: Construct a neural network model. The input to the neural network model is the training image data, and the output of the neural network is the target name and contour pattern. The training results are used to generate a training model for recognition in step D3.

[0069] D3. Recognition Process: The pre-training results are used for target recognition in practical system applications. The trained model is used to recognize the acquired binocular visible and non-visible light images, and the target name, target location, and target contour image are output.

[0070] In step D2, the neural network model is a combination of three basic functional models. For the input non-visible light image, the target location and name are obtained through the image preprocessing model. The target contour and coordinate position are obtained through the contour extraction model, which simultaneously inputs both non-visible and visible light binocular images and the output of the image preprocessing model. The outputs of the image preprocessing model and the contour extraction model, along with the original visible light image and the display information required by the system, are simultaneously input into the laser projection model, finally generating the laser projection pattern.

[0071] The depth information and structural features of the projection position are further determined by scanning with an infrared emitter. This information is then fused with spherical fitting and affine transformations performed on the acquired images. Based on the target structure, output geometric figures such as points, lines, surfaces, and colors are generated for projection. These output figures are then projected onto the target surface using a laser projector. The intensity of the laser projection is related to the phase change of the image captured by the infrared structured light.

[0072] The output laser projection result, mixed with the real-world image obtained through laser scanning gaps, yields projection information through image interpolation. This projection information is then compared with the original calculated projection output information to determine the offset positions of each pixel, thus obtaining the deviation information for each pixel. This deviation information is input into the laser projection galvanometer, enabling the system to obtain a precise projection position without calibration.

[0073] This invention provides an optical-based system for real-world target and information enhancement. Visible and invisible light information is emitted by a laser semiconductor emitter and an infrared emitter, respectively. This information is received by a binocular visible light imaging device (i.e., a first and second visible light camera) and an invisible light imaging device (i.e., an infrared camera). After processing, the received information is used to perform target recognition and contour extraction using deep learning methods. Specifically, the deep learning method involves inputting the original infrared and visible light information into a pre-trained neural network model, which then processes the data to obtain a reconstructed projection pattern. Simultaneously, the system acquires and compares the original and projected images, calculates the fit, and performs self-calibration. An optical image acquisition module acquires image information from the environment. After preprocessing and analysis by the information processing module, this information drives the laser projection module to perform projection fitting, accurately projecting the generated image onto the real-world scene. In this way, functions such as target contour enhancement, target hazard warnings, and interactive information guidance are achieved. The system allows users to select different information processing and projection modes to meet diverse application needs. This feature is designed to provide users with targeted information display and projection services, thereby assisting in performing a variety of daily life tasks, such as navigation, target recognition, and object marking.

[0074] The system achieves closed-loop control through the feedback loop indicated by the dashed arrow: after comparing and analyzing the difference between the actual projected image and the theoretical model, the deviation information is sent to the galvanometer control system for fine-tuning, thereby achieving sub-pixel-level projection accuracy without manual calibration. An application scenario example shows that when a pedestrian target is detected, the system can project a red warning line around its outline (as shown in the red box in the image), while simultaneously overlaying navigation arrows on the building facade. This multimodal information enhancement feature makes it suitable for complex scenarios such as visual impairment assistance, construction safety warnings, and pedestrian flow guidance in public places.

[0075] like Figure 3The diagram shows the side view of the system and the assembly of its internal components. The overall dimensions of the system are 13cm*13cm*15cm, with a 45° slope at the top rear. The K230 information processing host computer 3 receives input information including binocular visible light and infrared invisible light images output from the image optical acquisition module, user-selected function commands, and the projected scene image. The optical acquisition module includes a binocular visible light imaging device and a central infrared emitter and invisible light imaging device. The binocular visible light imaging device can acquire visible light images. The infrared emitter generates infrared light, which is then focused by the optical focusing module to narrow the optical path angle and increase the infrared light intensity, and scans the detection area. Finally, the invisible light imaging device acquires the invisible light image. The projected scene image is used for fitting and self-calibration calculations with the unprojected scene image. The formula for fusion of spherical fitting and affine transformation is: in and These are the coordinates of the transformed point. and These are the coordinates of the original point. , These are linear transformation coefficients. and It is a translation vector. The laser projection intensity is related to the phase change of the image captured by infrared structured light, and the relevant formula is: The lower-level microcontroller 4 is used to perform the projection action. Information output from the upper-level microcontroller 3 is transmitted to the lower-level microcontroller 4 via the SPI protocol. It is then amplified by a differential amplifier circuit at twice the gain and phase-flipped by an inverting circuit. The relevant formula is: in These are the control signals for the lower-level microcontroller 4, ranging from 1.024V to 3.072V. It is a reference level of 2.048V; and This is the processed signal input to the galvanometer drive module; the two signals are exactly out of phase, with a range of -2.048V to 2.048V. The processed signal is input to the galvanometer drive module to control the movement state of the optical galvanometer module 6, used to change the X-axis and Y-axis deflection of the laser projection, thereby forming a pattern. The power expansion board 5 processes all electrical signals. The 12.6V lithium battery 2 serves as an external power source; its voltage is boosted to 18V by a boost module, and then converted to ±15V by a charge pump buck module. In addition, the 12.6V voltage is further bucked to 3.3V by a buck module. The power expansion board 5 can generate ±15V, 12.6V, and 3.3V levels, meeting the power supply needs of various peripherals.

[0076] The above embodiments and accompanying drawings are for illustrative purposes only and do not constitute any limitation. The actual scope of protection of this invention is set forth in the claims. It should be understood that any modifications and changes can be made without departing from the spirit of this invention.

Claims

1. An optical-based real-scene target and information enhancement system, characterized in that: The device includes a housing and a first visible light camera, a second visible light camera, an infrared emitter, an optical focusing module, an invisible light imaging device, and a laser projection module installed within the housing. Four through holes are located on one side wall of the housing, arranged in two rows. The first visible light camera, infrared emitter, second visible light camera, and invisible light imaging device are all fixedly mounted on the same inner side wall of the housing. The first visible light camera, infrared emitter, and second visible light camera are respectively positioned opposite the three upper through holes. The first and second visible light cameras are used to acquire visible light images from different perspectives. The laser projection module is positioned opposite one of the lower through holes. The invisible light imaging device is fixedly mounted on the inner side wall of the housing. The optical focusing module is fixedly connected to the invisible light imaging device. The infrared emitter emits infrared light towards a preset target area. After being reflected by the target area, the infrared light passes through the optical focusing module and enters the invisible light imaging device, thereby generating an invisible light image.

2. The optical-based real-scene target and information enhancement system according to claim 1, characterized in that: It also includes a control processing module, which is fixedly installed on the inner side wall of the housing. The control processing module is electrically connected to the first visible light camera, the second visible light camera, the infrared emitter, the invisible light imaging device, and the laser projection module, respectively. It is used to control the infrared emitter to emit infrared light towards the target area, process the visible light image and the invisible light image to generate enhanced information, and then control the laser projection module to project the enhanced information onto the target area.

3. The optical-based real-scene target and information enhancement system according to claim 1, characterized in that: The invisible light imaging device is an infrared camera.

4. The optical-based real-scene target and information enhancement system according to claim 2, characterized in that, The control processing module includes: An image preprocessing model is used to receive invisible light images acquired by an invisible light imaging device, preprocess them, and obtain preprocessing results. A contour extraction model is used to extract contour images from invisible light images, preprocessed results, and two visible light images. A laser projection generation model is used to generate laser projection images based on contour images, preprocessing results, and two visible light images. A fitting and self-calibration model is used to fit and self-calibrate two visible light images, an invisible light image, and a projection image to obtain enhanced information for calibrating the laser projection image.

5. The optical-based real-scene target and information enhancement system according to claim 1, characterized in that: The laser projection module includes a visible light laser module (8), two total reflection prisms, two beam-splitting mirror groups (7) and two optical galvanometer modules (6). The visible light laser module (8) includes three light sources. The first beam-splitting mirror group (7) includes three semi-transparent and semi-reflective mirrors, and the second beam-splitting mirror group (7) includes three highly reflective mirrors. The three light sources in the visible light laser module (8) emit red, green and blue lasers respectively. The three colors of lasers are incident on the three semi-transparent and semi-reflective mirrors in the first beam-directing mirror group respectively, and are reflected and transmitted respectively. The reflected light from the three semi-transparent and semi-reflective mirrors in the first beam-directing mirror group is reflected by the first total internal reflection prism and then incident on the first optical galvanometer module. After being reflected by the first optical galvanometer module, it is projected onto the target area. The transmitted light from the three semi-transparent and semi-reflective mirrors in the first group of optical turning mirrors is reflected by the three highly reflective turning mirrors in the second group of optical turning mirrors. The reflected light from the three highly reflective turning mirrors in the second group of optical turning mirrors is reflected by the second total internal reflection prism and then enters the second optical galvanometer module. After being reflected by the second optical galvanometer module, it is projected onto the target area. The visible light laser module (8) is electrically connected to the control and processing module and is used to control the wavelength of the laser emitted by the visible light laser module (8). The optical galvanometer module (6) is electrically connected to the control and processing module and is used to drive the galvanometer to rotate and change the optical path.

6. An optical-based method for augmenting real-world targets and information in a system as described in any one of claims 1-5, characterized in that, The method includes the following steps: S1. The first visible light camera and the second visible light camera respectively acquire the first visible light image and the second visible light image, and the invisible light imaging device acquires the invisible light image. S2. Input the invisible light image into the image preprocessing model for preprocessing to obtain the preprocessing result. Then, input the invisible light image, the preprocessing result, the first visible light image and the second visible light image into the contour extraction model for contour extraction to obtain the contour image. S3. Input the contour image, preprocessing result, first visible light image and second visible light image into the laser projection generation model to obtain the laser projection image; S4. The laser projection image is sent to the laser projection module for projection to obtain the projection image. The projection image, the invisible light image, the first visible light image and the second visible light image are input together into the fitting and self-calibration module to obtain the enhancement information. S5. After feeding back the enhanced information to the laser projection module, the image is projected to obtain the calibrated projection image.

7. The optical-based real-scene target and information enhancement method according to claim 6, characterized in that: The preprocessing in step S2 includes denoising and contrast enhancement of the input image, and obtaining the names and location parameters of objects in the image based on object detection and classification algorithms.

8. The optical-based real-scene target and information enhancement method according to claim 6, characterized in that: The contour extraction in step S2 specifically involves: 1) Extract features from the invisible light image to obtain a multi-scale feature map of the invisible light image; 2) Extract features from the first visible light image and the second visible light image respectively to obtain the first visible light feature map and the second visible light feature map, and then stitch the first visible light feature map and the second visible light feature map together to obtain the fused visible light feature map; 3) Based on the preprocessing results, target region cropping and size alignment operations are performed on the multi-scale feature map and the fused visible light feature map to obtain invisible light local features and visible light local features. Then, the invisible light local features and visible light local features are fused to obtain the target fused feature map. 4) Extract contours from the target fusion feature map to obtain several ordered contour points, and all the ordered contour point columns together constitute the contour image.

9. The optical-based real-scene target and information enhancement method according to claim 6, characterized in that: The laser projection generation model is specifically processed according to the following steps: a) Interpolate and smooth the contour image to obtain the basic trajectory segment used for galvanometer scanning; b) Extract scene features from the first visible light image and the second visible light image to obtain scene information; c) Align the basic trajectory segments, scene information, preprocessing results, and preset projection control parameters with the target region, and stitch them together in the channel dimension to obtain target-level projection features, and then process them to obtain a laser projection image.

10. The optical-based real-scene target and information enhancement method according to claim 6, characterized in that: In step S4, the fitting and self-calibration are processed according to the following formula: in, Indicates coordinates as Laser projection intensity at the location, This represents the average of the maximum and minimum laser projection intensities. Indicates the intensity modulation factor. It is the phase change of the captured invisible light image. It is a phase compensation term.

Images