Intelligent head-mounted device with camera capable of rotating along with movement

Abstract

The invention discloses an intelligent head-mounted device with a camera capable of rotating along with the camera, and the device comprises an eye movement tracking camera which is used for collecting and measuring the image data of the iris of the two eyes of a wearer; the IMU sensor is used for detecting head movement data of a wearer; the image acquisition and processing module is used for generating eyeball movement data according to the image data of the iris of the two eyes, and converting the eyeball movement data and the head movement data into a rotation instruction; the binocular camera motion control device executes the rotation instruction, drives the binocular camera to move, moves along with the iris and the eyeball, and collects image data in real time to realize line-of-sight tracking; and the image acquisition and processing module is also used for carrying out image and / or video fusion processing according to the received image acquired by the binocular camera in combination with the head movement data and the eyeball movement data. According to the invention, the problem of limitation in the aspects of acquiring depth information and expanding the visual field can be solved, and a more flexible camera tracking and adjusting mechanism is designed, so that wider and clearer image acquisition is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of binocular vision and image fusion, and more specifically to an intelligent head-mounted device with a camera that can rotate with the user, which uses an auxiliary camera to track the movement of human eyes and control the follow-up movement of the binocular camera to achieve the purpose of expanding the binocular field of view. Background Art

[0002] According to the imaging principle of convex lenses, when the object distance is greater than twice the focal length, the image distance is between one and two focal lengths, forming an inverted, reduced real image. At this time, the image distance is smaller than the object distance, the image is smaller than the object, and the object and the image are on opposite sides. This principle is used in photography and videography. The structure of the human eye is equivalent to a convex lens with variable focal length. The visual imaging of the human eye is that the reflected light of the object is refracted through the lens and formed on the retina, and then perceived by the visual nerve and transmitted to the brain, so that people can see the object. The image of the external object on the retina is a real image, and the image is adjusted by the cerebral cortex to form an upright image. When the two eyes observe the same object at the same time, two images of the object are formed on their respective retinas, and then transmitted to the same area of ​​the cortical visual center through the optic nerves on both sides, and merged into a complete, single stereoscopic perception experience of the object.

[0003] Similarly, binocular machine vision is one of the stereo vision technologies that people are studying. The premise of binocular vision is to accurately know the internal and external parameters of the visual system, that is, the internal and external parameters of the camera. The process of solving the internal and external parameters of the camera is the so-called camera calibration. The accuracy of camera calibration directly affects the quality of vision. Usually, manufacturers strictly fix the relative position of the two cameras of the binocular camera, thereby fixing the relative coordinates of the imaging of the two cameras, thereby simplifying image fusion. However, with such a design, the relative position of the camera is fixed, and the relative position of the camera relative to the host device is also fixed.

[0004] Obtaining image depth information has become a hot topic because of the various methods that have been developed, but most of the methods have certain defects. Currently, a more mature method is to use multiple cameras of the same model with a fixed focal length to obtain depth information. This method is very low-cost, and within a certain range, the depth accuracy it obtains is also high, but once the measured target exceeds this range, the accuracy of the depth information obtained will be greatly reduced. The reason is that the focal lengths of the two cameras used to obtain information are fixed, so the field of view obtained is also fixed. When the measured target is far away from the binocular camera, its image is also blurry, so the amount of information obtained is small, and the accuracy of the depth information obtained is also low.

[0005] Therefore, people thought of using a variable-focus camera for binocular imaging, changing the focal length of the camera according to the selected target, to achieve the purpose of clear target imaging and make the measured depth information more accurate. However, for targets at different distances, the camera's focus adjustment amount is also different, and each focus adjustment will change the camera's focal length, so it is difficult to keep the focal lengths of the two cameras consistent.

[0006] For some wearable devices, such as XR glasses and smart helmets, people also hope to obtain a wider field of view and images with spatial information through binocular vision. The field of view of wearable devices using the aforementioned fixed binocular cameras is limited to the field of view of a single camera. If you want to further expand the field of view, or if the user intends to see a wider field of view without moving the wearable device, it is not possible.

[0007] Therefore, current technologies have some limitations in acquiring depth information and expanding the field of view, especially in wearable devices where the field of view and image clarity are limited. Summary of the invention

[0008] In view of this, the present invention provides an intelligent head-mounted device with a camera that can rotate with the headset, which can solve the limitations in obtaining depth information and expanding the field of view. The present invention designs a more flexible camera tracking and adjustment mechanism to achieve wider and clearer image acquisition.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] The present invention provides an intelligent head-mounted device with a camera capable of rotating with the head, comprising: a head-mounted body, and an eye-tracking camera, a binocular camera, an image acquisition and processing module, an IMU sensor, and a binocular camera motion control device installed on the head-mounted body;

[0011] Wherein, the image acquisition and processing module is respectively connected to the eye tracking camera, the binocular camera, the IMU sensor and the binocular camera motion control device;

[0012] The eye tracking camera is used to collect and measure image data of the wearer's irises and send it to the image acquisition and processing module;

[0013] The IMU sensor is used to detect the wearer's head movement data and send it to the image acquisition and processing module;

[0014] The image acquisition and processing module generates eye movement data according to the image data of the binocular irises, and converts the eye movement data and the head movement data into a rotation instruction to be sent to the binocular camera motion control device;

[0015] The binocular camera motion control device executes the rotation instruction to drive the binocular camera to move, follow the movement of the iris and eyeball, and collect image data in real time to achieve tracking and synchronization of sight lines;

[0016] The image acquisition and processing module performs image and / or video fusion processing based on the images received from the binocular camera and in combination with the head movement data and the eye movement data.

[0017] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following technical advantages:

[0018] The present invention uses a zoomable and movable dual camera as the image acquisition device of the device. Zooming is for the need of a deeper field of view, while the axial movement of the camera expands the field of view in the horizontal and vertical directions. The present invention uses a multi-coordinate imaging correlation algorithm to calculate and fuse the synthesized image and video in real time according to the change of imaging coordinates when the camera body moves and the focal length changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0020] Figure 1 This is a structural block diagram of the intelligent head-mounted device with a camera that can rotate with the device provided by the present invention.

[0021] Figure 2 This is a block diagram of the generation of eye movement data in the image acquisition and processing module provided by the present invention.

[0022] Figure 3 This is a block diagram of the image acquisition and processing module provided by the present invention when generating a rotation instruction.

[0023] Figure 4 This is a block diagram of image fusion in the image acquisition and processing module provided by the present invention.

[0024] Figure 5 The present invention provides a flowchart of image fusion in the image acquisition and processing module.

[0025] Figure 6 A flow chart of the operating principle of the intelligent head-mounted device with a rotating camera provided by the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The embodiment of the present invention discloses a smart head-mounted device with a camera capable of rotating with the user. Figure 1 As shown, including:

[0028] A head-mounted body (not shown in the figure), and an eye tracking camera, a binocular camera, an image acquisition and processing module, an IMU sensor, a binocular camera motion control device, a near-eye display, a data output unit, and a power supply unit installed on the head-mounted body;

[0029] Among them, the image acquisition and processing module is respectively connected to the eye tracking camera, the binocular camera, the IMU sensor and the binocular camera motion control device.

[0030] The details of each part are as follows:

[0031] 1. Eye tracking camera:

[0032] It is used to collect and measure the image data of the wearer's irises and send it to the image acquisition and processing module through the serial port bus.

[0033] The fixed eye tracking camera can adopt a dual camera design or a fixed camera group design. According to the specific form of the headwear: glasses, helmets, half helmets or bandages, headbands, etc., two or two groups of eye tracking cameras can be installed on the bracket or fixed on the headwear body. The specific design is based on the principle that the two cameras are aimed at the wearer's eyes without affecting their line of sight. The eye tracking camera can also use existing mature technology to collect and measure the direction and position information of the irises of both eyes and the angle information formed between the directions of the irises of both eyes. The image acquisition and processing module uses this information to control the focus direction and angle of the follow-up binocular camera, follow the movement of the iris and eyeballs, and the binocular follow-up camera and the eyeball focus on the same external world, so as to achieve the tracking and tracking synchronization of the line of sight.

[0034] In specific implementation, for example, an eye tracking camera uses two fixed micro-focus cameras to focus on the middle part of the wearer's left and right eyes respectively, and the camera sensor can be a common product that is sensitive to visible light or an infrared sensitive product with a resolution of more than one million pixels. The eye tracking camera quickly and continuously collects images of the wearer's eyeballs, with an imaging speed of more than hundreds of images per second.

[0035] 2. Binocular camera:

[0036] The binocular camera is the "eye" of the device, which is used to collect environmental images of interest to the wearer of the device. The binocular camera is connected to the image acquisition and processing module through a high-speed parallel interface. The binocular vision camera module includes two homogeneous cameras with adjustable focal lengths. The resolution of the cameras is generally above millions of pixels and the field of view is large. The two homogeneous cameras are respectively fixed on a binocular camera motion control device, which includes, for example: an electric rotating base and a micro motor; the micro motor drives the rotating base, and the base can move up and down and left and right in a plane; the camera is installed on the rotating base, and the movement of the base drives the movement of the camera, so as to simulate the rotation of the human eyeball and the up and down and left and right scanning of the human eye.

[0037] The binocular camera that can perform rotational motion can be two independent camera modules or a physically integrated binocular module. The camera can be a fixed focal length camera, or an optical zoom camera or a combination thereof. However, in order to overcome the difficulty of fusion calculation due to the large difference in vision between the "binocular" eyes, the key parameters of the binocular camera such as the image sensor, imaging pixel, and field of view should be consistent, and the image signal processor and its algorithm of the binocular image should be consistent. Whether it is two independent cameras or an integrated binocular module, the camera body is fixed on a micro motor assembly that can perform rotational motion.

[0038] 3.IMU sensor:

[0039] The IMU sensor includes a MEMS accelerometer and a micro gyroscope. The IMU sensor is used to detect, track and calculate the wearer's head movement data. The data is transmitted to the image acquisition and processing module through a serial interface. It is also used for coordinate adjustment of binocular imaging fusion and for detecting the device's usage status, including device vibration, jitter detection and placement detection.

[0040] 4. Near-eye display:

[0041] Reference Figure 1 As shown, the smart head-mounted device also includes a near-eye display, which includes a low-power micro display driver, an optical mechanical driver, and an optical waveguide. The near-eye display is connected to the image acquisition and processing module through a high-speed parallel display interface, and the complete video stream collected and synthesized by the image acquisition and processing module is displayed in front of the wearer.

[0042] 5. Data output unit;

[0043] Reference Figure 1 As shown, the data output unit is connected to the image acquisition processing module through a high-speed serial port, and the complete video data stream collected and synthesized by the image acquisition processing module is selectively and synchronously transmitted to the outside.

[0044] 6. Image acquisition and processing module:

[0045] The image acquisition and processing module is the data input, output, processing and control center of the entire head-mounted device. The hardware structure of the image acquisition and processing module includes a central processing unit (CPU), an image signal processor (ISP), a storage unit (MCP) and an interface unit (SIU). The storage unit (MCP) includes a dynamic data storage space and a static program storage space. The former is used for temporary storage of running codes and intermediate data, and the latter is used for storage of firmware programs and running parameters. The interface unit (SIU) includes a motion interface and a data interface, wherein the data interface includes a high-speed serial interface connected to the aforementioned data output module, a serial interface for receiving eye movement data output by the eye tracking camera, and a high-speed parallel display data interface.

[0046] The functions implemented by the image acquisition and processing module are as follows 6.1~6.3:

[0047] 6.1. Generate eye movement data based on the image data collected by the eye tracking camera;

[0048] Reference Figure 2 As shown, the image acquisition processing module generates eye movement data based on the image data of the irises of both eyes, including:

[0049] An image preprocessing unit performs grayscale conversion, noise reduction and contrast enhancement on the image data collected by the eye tracking camera in sequence;

[0050] The feature extraction unit realizes the positioning of pupil and corneal reflection point based on Canny edge detection and Hough transform;

[0051] A coordinate system conversion and calibration unit, which constructs a two-dimensional coordinate system with the upper left corner of the image as the origin, represents the pupil location and corneal reflection point location data in the two-dimensional coordinate system, and performs a calibration operation;

[0052] The eye movement data generating unit obtains the coordinates of the eye gaze point according to the relative position relationship between the pupil center and the corneal reflection point combined with the calibrated parameters; the coordinates of the eye gaze point obtained by continuously collecting images are connected in chronological order to form eye movement data.

[0053] The received eye tracking camera image data is analyzed and calculated to form eye movement data. The specific process is as follows:

[0054] (1) Image acquisition

[0055] The eye tracking camera first captures the image information of the human eye in real time, usually at a certain frame rate (such as dozens of frames per second or even higher) to ensure that the dynamic changes of the eyeball can be fully recorded. The position and angle of the camera need to be properly calibrated so that it can focus clearly on the eyeball and obtain high-quality raw image data for subsequent analysis.

[0056] (2) Image preprocessing

[0057] Grayscale processing: The images captured by the camera are often color images, but for the subsequent eye movement analysis, the color images are often converted to grayscale images first, which can reduce the amount of data and highlight key features such as the image outline, which is convenient for subsequent processing. The grayscale operation is relatively simple, and the color pixel values ​​are converted to grayscale values ​​according to the color channel weights through a specific algorithm. For example, the commonly used weighted average method has a calculation formula of: Gray = 0.299 × R + 0.587 × G + 0.114 × B (where R, G, and B represent the pixel values ​​of the red, green, and blue color channels, respectively).

[0058] Noise reduction: During the image acquisition process, noise may be generated due to factors such as ambient light and the noise of the electronic device itself. Common noise reduction methods include mean filtering and median filtering. Mean filtering replaces the value of the central pixel with the average value of the pixels in the neighborhood, thereby making the image smoother and reducing noise interference; median filtering sorts the pixel values ​​in the neighborhood and takes the middle value as the central pixel value, which is better for removing salt and pepper noise.

[0059] Enhance contrast: In order to more clearly distinguish the contours of key parts such as the eyeball and pupil, contrast enhancement technology is used, such as the histogram equalization method. It redistributes the grayscale histogram of the image to make the details of the originally darker or brighter areas in the image more obvious, stretching the overall contrast of the image, making it easier to identify related features later.

[0060] (3) Feature extraction

[0061] Pupil location: This is a key step. Various algorithms are used to find the location of the pupil in the preprocessed image. Common methods include edge detection-based algorithms. For example, the Canny edge detection operator is used to extract edge information in the image. Then, based on the condition that the pupil is generally circular and has specific grayscale features, the pupil is located by detecting the circle through methods such as Hough Transform. Hough Transform can convert the circle detection problem in the image space into the parameter space for voting statistics, and find the area that meets the circular features, which is most likely the location of the pupil.

[0062] Corneal reflection point positioning: In addition to the pupil, the corneal reflection point is also an important feature. When an external light source (such as the device's own infrared light source, etc.) is irradiated, a reflective bright spot will form on the corneal surface. Similarly, its brightness, position and other characteristics are used in combination with the geometric relationship analysis of the image to accurately find the coordinate position of the corneal reflection point in the image.

[0063] Identification of other eye features: Based on the specific eye tracking accuracy requirements and other conditions, we will further identify features such as eyelid position and eye contour. For example, by detecting the grayscale changes at the eyelid edge, we can determine the eyelid open and closed state and its position, providing more reference for a more comprehensive analysis of eye movement and eye status.

[0064] (4) Coordinate transformation and calibration

[0065] Establish a coordinate system: Establish a two-dimensional coordinate system on the image plane, determine the upper left corner of the image as the origin, the right as the positive direction of the x-axis, and the downward as the positive direction of the y-axis, and express the pixel coordinates of the extracted pupil, corneal reflection point and other features in this coordinate system.

[0066] Calibration operation: Due to differences in eye size, position, and camera installation angle between individuals, calibration is required to improve data accuracy. The calibration process generally requires the subject to look at multiple calibration points at known positions on the screen (such as multiple fixed points in the common nine-square grid format), record the coordinate data of the corresponding eye feature points at this time, and then calculate the conversion relationship through a mathematical model (such as a linear regression model, etc.), so that the subsequent actual measured coordinate data can accurately correspond to the actual gaze angle, position and other information.

[0067] (5) Calculation of eye movement parameters

[0068] Gaze point calculation: According to the relative position relationship between the pupil center and the corneal reflection point, combined with the calibrated parameters, the coordinate position of the gaze point of the human eye on the screen or in the actual observation scene is inferred through geometric calculations, optical models and other methods. For example, the commonly used pupil-corneal reflection vector method (P-CR vector method) uses the geometric relationship between the vector formed by the two and the gaze direction to determine the location of the gaze point.

[0069] Eye movement trajectory generation: By continuously collecting images over time and repeating the above analysis steps, the continuous gaze point coordinates are connected in chronological order to form the trajectory of eye movement, which intuitively shows the movement path of the eyeball.

[0070] 6.2, image acquisition processing module, used to convert eye movement data and head movement data into rotation instructions, refer to Figure 3 As shown, including:

[0071] A head and eye movement parameter acquisition unit is used to convert eye movement data and head movement data into a unified global coordinate system;

[0072] The azimuth adjustment unit is used to convert the movement direction and angle of the head and eyeballs in the global coordinate system into adjustment data of the binocular camera motion control device according to the principle of motion reverse compensation.

[0073] In this embodiment, the image acquisition and processing module converts the eye movement data and the head movement data into rotation instructions and sends them to the binocular camera motion control device; the binocular camera motion control device executes the rotation instructions to drive the binocular camera to move and achieve following the movement of the iris and eyeballs.

[0074] The eye movement data of the eye tracking camera is converted into the motion data of the base motor of the binocular vision camera module after calculation by the central processing unit.

[0075] 1). Obtain head and eye movement parameters

[0076] (1) Sensor data acquisition: IMU sensors are used to detect head movement information, such as the angular velocity of the head (usually represented by a vector, including angular velocity components rotating around different coordinate axes, such as ω1, ω2, and ω3 around the x, y, and z axes, respectively) and linear acceleration data. For eye movement, the eye rotation angle (including horizontal rotation angle α and vertical rotation angle β, etc.) can be determined based on the output of the above-mentioned eye movement data generation unit, such as corneal reflection, pupil center positioning, etc.

[0077] (2) Coordinate transformation and unification: These data from different sources are often in their own local coordinate systems and need to be transformed into a unified global coordinate system to facilitate subsequent calculations. This process involves the use of rotation transformation matrices and translation vectors. For example, for a posture change of the head, it is transformed from its own coordinate system to the global coordinate system. Assuming that the head coordinate system has rotation angles θ1, θ2, and θ3 around its own coordinate axis, there is a 3×3 rotation matrix R1 that can be constructed through the rotation transformation formula (such as the derivation of the rotation matrix based on Euler angles, etc.), which transforms the coordinate point (x1, y1, z1) in the head coordinate system to the global coordinate system (x2, y2, z2), satisfying the relationship: [x2, y2, z2] T =R1[x1,y1,z1] T ( T In other words, there is a corresponding transformation relationship from the eyeball coordinate system to the global coordinate system.

[0078] 2). Adjust the binocular camera azimuth calculation

[0079] (1) Based on the principle of motion reverse compensation: According to the direction and angle of movement of the head and eyeballs, calculate the angle that needs to be rotated to offset these movements and keep the camera relatively stable. For example, if the head rotates to the left by an angle of θ (a certain axial angle changes in the global coordinate system), then in theory the binocular camera needs to be rotated to the right by an angle of θ (this involves a motor or other drive device controlling the rotation of the camera, and the corresponding control signal often has a certain conversion relationship with the rotation angle, such as the corresponding relationship between the motor rotation angle and the number of control pulses. If the motor rotates 1 degree every time it receives N pulses, it needs to send N×θ pulses to drive the motor to rotate by an angle of θ).

[0080] (2) Coordinated adjustment between binocular cameras: The binocular cameras need to maintain a certain relative position relationship and coordinate when adjusting the azimuth angle. For example, for horizontal rotation, the two cameras must rotate synchronously at the same angle to ensure that their fields of view always maintain a relatively fixed geometric relationship, which is convenient for subsequent stereo vision calculations. Assume that the azimuth angle that camera A needs to rotate is The azimuth angle that camera B needs to rotate is Under normal circumstances (When rotating in the horizontal direction), the same is true for rotating in the vertical direction, and such rotation angle adjustment is accurately achieved through respective drive control mechanisms.

[0081] 6.3. The image acquisition and processing module performs image and / or video fusion processing based on the images captured by the binocular camera and combined with the head movement data and the eye movement data.

[0082] The raw data collected by the aforementioned binocular vision camera is transmitted to the image acquisition and processing module through a high-speed parallel communication interface, first enters the ISP image signal processor, and then is transmitted to the central processing unit (CPU) through a set of parallel interfaces within the module. The central processing unit is responsible for image fusion processing.

[0083] Reference Figure 4 As shown, the image acquisition processing module, when used for image and / or video fusion processing, includes:

[0084] The image signal processing unit is used to perform bad pixel correction, filtering and denoising, color correction and gamma correction on the received left and right video data in sequence;

[0085] The image fusion unit is used to spatially align the processed left and right video data, and fuse them using pixel value-based fusion, Laplace pyramid fusion, multi-resolution fusion or Poisson fusion algorithm, and then perform contrast enhancement and sharpening on the fused image.

[0086] The central processing unit uses a high-performance digital image processor to complete the real-time image fusion of the binocular video stream. Image fusion generally requires three steps. First, extract the feature points of the left and right binocular images, and divide the image processed by the ISP output by the integrated image acquisition board into N groups of S layers of Gaussian difference pyramids through, for example, the BRISK algorithm. In each layer of the scale space pyramid, a suitable feature point detection algorithm should be used, and the same threshold should be applied to detect and extract key points for each image group and interpolation group; then purify the matching point pairs, and use, for example, a random sampling consensus algorithm to screen out mismatches in the matching pairs. The RANSAC algorithm randomly selects some samples, iterates repeatedly, and determines the error values ​​between the transformations for comparison and judgment. It is determined whether the feature point is an inner point or an outer point. A stable maximum value appears, which is the optimal parameter; finally, the overlapping areas between the left and right binocular vision images to be spliced ​​are fused, and seamlessly spliced ​​into a high-resolution image. The fusion coefficient is selected using a genetic algorithm and the characteristics of the human visual system, and the fusion coefficient is used to perform an inverse wavelet transform to reconstruct the fused image, and the left and right binocular vision images are spliced ​​into a complete image with a wider field of view.

[0087] The image acquisition and processing module receives the continuous video data stream collected by the binocular vision camera through the parallel communication interface, and then undergoes ISP image signal processing and image fusion in turn to synthesize the dual video data streams of the left and right binocular cameras into a complete fused video data stream. Figure 5 As shown, the specific process is as follows:

[0088] 1) Data Reception

[0089] The binocular vision camera transmits the collected continuous video data stream to the image acquisition and processing module through a parallel communication interface. During the transmission process, it will contain image data and related synchronization signals, such as line synchronization signal (HREF), field synchronization signal (VSYNC) and pixel clock signal (PCLK), which are used to identify the effective area and transmission rhythm of image data.

[0090] 2)ISP image signal processing

[0091] (1) Bad pixel correction: During the imaging process, the camera may produce bad pixels due to individual pixel failures of the sensor, which appear as bright or dark spots in the image. The ISP will detect and correct these bad pixels through algorithms. For example, for dead pixels, the values ​​of the surrounding normal pixels can be used to estimate and replace their original erroneous values, thereby improving the quality of the image.

[0092] (2) Denoising filter: Various noises are inevitably introduced during the image acquisition process, such as thermal noise and shot noise. ISP uses filtering algorithms, such as Gaussian filtering and median filtering, to denoise the image. Gaussian filtering smoothes the image and reduces the impact of noise by taking a weighted average of each pixel in the image and its neighborhood. Median filtering sorts the grayscale values ​​in the pixel neighborhood and takes the middle value as the new grayscale value of the pixel. It has a good effect on removing salt and pepper noise, etc.12.

[0093] (3) Color correction: Different camera sensors may have different responses to colors, resulting in inconsistency between the captured image color and the color of the real scene. ISP uses a color correction algorithm to adjust the red, green, and blue channels of the image according to a preset color correction matrix to make the image color more accurate and natural, close to the real color seen by the human eye12.

[0094] (4) Gamma correction: Gamma correction is used to adjust the brightness and contrast of an image so that the image can present a more appropriate visual effect on different display devices. It performs nonlinear transformation on the grayscale value of the image to enhance the dark or bright details of the image and improve the layering and visual comfort of the image127.

[0095] 3) Image Fusion

[0096] (1) Image registration: Before fusion, the images captured by the left and right binocular cameras need to be registered so that they are aligned in space and the corresponding pixels are as consistent as possible in geometric position. This usually involves performing transformation operations such as translation, rotation, and scaling on the image to find the best matching position. Common registration methods include feature point-based registration algorithms, such as SIFT, SURF and other feature point extraction algorithms, which extract significant feature points from the image and find corresponding feature point pairs in the left and right images. Then, the transformation relationship between the images is calculated based on these feature point pairs, thereby achieving image registration.

[0097] (2) Fusion algorithm selection: Select an appropriate image fusion algorithm based on specific application requirements and image characteristics. Common fusion algorithms include pixel value-based fusion, Laplace pyramid fusion, multi-resolution fusion, Poisson fusion, etc.

[0098] Fusion based on pixel values: Simply perform weighted average operations on the grayscale values ​​or color values ​​of the corresponding pixels of the left and right images to obtain the fused pixel value. For example, for a grayscale image, the average grayscale value of the corresponding pixels of the left and right images can be directly calculated as the grayscale value of the fused pixel; for a color image, the red, green, and blue channels can be weighted averaged separately.

[0099] Laplacian pyramid fusion: First, the left and right images are constructed into Laplacian pyramids respectively, and then the corresponding image details are fused at different pyramid levels according to certain fusion rules, and finally the fused image is obtained through reconstruction algorithm. This method can better preserve the detailed information of the image and make the fused image more natural.

[0100] Multi-resolution fusion: decompose the left and right images into image sequences of different resolutions, and then fuse them at each resolution level, gradually synthesizing the final fused image from low resolution to high resolution. This can make full use of the information of images at different resolutions and improve the quality and stability of the fusion effect.

[0101] Poisson fusion: Based on the Poisson equation, by solving a minimization problem, the brightness and color changes of the fused image in the fusion area are made as smooth as possible, thus achieving a natural and seamless fusion effect.

[0102] (3) Post-fusion processing: Perform necessary post-processing on the fused image to further optimize the image quality. For example, the fused image can be subjected to contrast enhancement and sharpening to highlight the target object and detail information in the image, making the fused video data stream clearer and more vivid, which is more conducive to subsequent analysis and processing.

[0103] Figure 6 This is a flow chart of the operating principle of the intelligent head-mounted device with a camera that can rotate with the mobile provided by the present invention. After the head-mounted device is powered on, the firmware program is copied from the aforementioned static program storage space to the dynamic memory space and starts running. The program first completes the initialization of the camera and the display, including the initialization of the eye tracking camera and the initialization of the binocular vision camera. The program will detect and complete the calibration of the coordinates of both eyes according to the actual wearing situation of the user. Then, the eye tracking acquisition is started. The movement of the user's eyes includes left and right rotation and up and down rotation. The up and down, left and right movement information is used as the basis for calculating the movement angle of the binocular vision camera. By controlling the movement of the motor and adjusting the position of the camera, the movement information is also used for the adjustment of the dual imaging coordinate system of the image fusion processing; if it is detected that the user is not currently rotating the eyeball, the user is currently in the state of "looking", and this information will also participate in the image fusion calculation. After the binocular vision image completes image processing and image fusion, it is sent to the near-eye display for display, and can also be synchronously transmitted to other devices for storage and display.

[0104] The present invention adopts a zoomable and movable dual camera as an image acquisition device for a head-mounted device. Zooming can achieve a greater depth of vision, while the follow-up movement of the camera expands the user's plane vision.

[0105] In addition, the device of the present invention can adjust the movement of the binocular camera and the synthesis of binocular images according to the combination of eye movement and head movement, including the situation that the head is moving while the eyeballs are not moving relative to the outside world (focus), the head is moving while the eyeballs follow the head (scanning), and the head is not moving while the eyeballs are moving, the head and the eyeballs are both moving, etc. When the head is moving but the eyeballs are not moving, it is interpreted as focusing, the camera focuses on the front and does not move, and the superimposed synthetic image processing can be performed; when the head moves and the eyeballs follow the movement, it is interpreted as scanning, the camera focuses on the front and does not move, and a panoramic image can be synthesized; when the head is not moving but the eyeballs move, the horizontal and vertical angles of the binocular camera are adjusted according to the corresponding movement angles of the eyeballs in the horizontal and vertical directions, and a synthetic wide-view image can be calculated at the same time; and it is a normal state that both the head and the eyeballs are moving, and it is necessary to adjust the azimuth angle of the binocular camera at the same time, and adjust the coordinates of the image synthesis to synthesize a stable image. The entire process involves multi-faceted coordinate transformation, motion parameter acquisition and conversion, and calculation based on visual models. By continuously adjusting the camera azimuth angle and image synthesis coordinates in real time, the influence of head and eye movement can be overcome to generate stable and usable images.

[0106] At the same time, the device has the ability to judge vibration and shaking during the wearing of the device. The eye movement data caused by these unconscious movements are filtered through filters to accurately remove interference signals according to the frequency characteristics of the signal (for example, device shaking may cause high-frequency (such as greater than 10Hz) interference components in the eye movement signal. By setting the cutoff frequency of the low-pass filter to 5Hz, these high-frequency interferences can be filtered out, and only the effective eye movement signals with lower frequencies (such as the signal frequencies corresponding to eye movement behaviors such as saccades and stable tracking) are retained). The sampling data of the binocular camera is corrected. First, the binocular camera is calibrated to determine the camera's internal parameters (such as focal length, principal point position, etc.) and external parameters (such as rotation and translation relationships). When the device shakes, etc., the parameters obtained by calibration are used to correct the sampling data through methods such as coordinate transformation. For example, if the relative position and posture relationship of the binocular camera is known (obtained through calibration), when the device shakes and causes a slight change in the camera position, the coordinates of the sampling points can be converted to the coordinate system when the device is not shaking based on the geometric relationship and calibration parameters, thereby correcting the sampling data to form a smooth image data stream.

[0107] The device of the present invention can achieve wider vision and even ultra-viewing angle images. The human eye has a viewing angle of about 120°, while a general camera has a viewing angle of about 70°. A wide-angle camera and an ultra-wide-angle camera can reach 120°. With the rotation of the lens, including up and down, left and right movement, a 180° or even 360° panoramic image can be obtained, and even a 720° panoramic image (360° horizontally and 360° vertically) can be generated.

[0108] The device of the present invention can achieve deeper vision and realize remote observation. The device can focus on remote targets that human eyes cannot reach but cameras can focus on, thereby expanding the user's field of vision.

[0109] The device of the present invention can realize tracking analysis. The device determines the center of the picture that the observer is interested in through sampling analysis, so that the system performs focused analysis on the area and gives prompts.

[0110] The device of the present invention also has power saving characteristics. When it is detected that no human eye is looking at the tracking camera, it can be judged that the device is not in use, and after a certain period of time, the external camera image acquisition and calculation related circuits are turned off, and the local display and remote transmission are turned off. The basic principle is that the device also includes: an infrared light transmitting device, which uses the infrared light transmitting device to emit infrared light to the area where human eyes may exist, and the structures of the human eye such as the cornea and lens will produce specific reflection characteristics for infrared light. When the camera cooperates with components such as infrared filters to receive the reflected infrared light, under normal circumstances, if the human eye looks at the direction of the camera, the characteristics of its reflected light will present specific patterns such as light spots and contours in the image (for example, a clearer pupil contour can be seen). When there is no human eye looking, the received reflected light image will lack these characteristic expressions. The infrared reflection image collected by the camera is analyzed by an image recognition algorithm to determine whether there are characteristics that meet the human eye gaze, so as to determine whether there is human eye gaze. On the contrary, if it is detected that the human eye is approaching (that is, the user wears the device well, and the eye is in a reasonable spatial position set by the device to accurately capture the eye state and movement), the external camera camera and image processing function circuit are turned on. The principle is: use the infrared distance sensor to emit infrared signals. When the human eye approaches, the infrared signal will be reflected back. The sensor calculates the distance between the human eye and the sensor based on the intensity and time of the reflected signal. When the distance is less than the set threshold (for example, the distance threshold is set to 8 cm, and when the distance is detected within 8 cm, it is determined that the human eye is approaching), it can be determined that the human eye is approaching.

[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0112] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A smart head-mounted device with a camera that can rotate with the user, characterized in that: include: A head-mounted body, and an eye-tracking camera, a binocular camera, an image acquisition and processing module, an IMU sensor, and a binocular camera motion control device installed on the head-mounted body; Wherein, the image acquisition and processing module is respectively connected to the eye tracking camera, the binocular camera, the IMU sensor and the binocular camera motion control device; The eye tracking camera is used to collect and measure image data of the wearer's irises and send it to the image acquisition and processing module; The IMU sensor is used to detect the wearer's head movement data and send it to the image acquisition and processing module; The image acquisition and processing module generates eye movement data according to the image data of the binocular irises, and converts the eye movement data and the head movement data into a rotation instruction to be sent to the binocular camera motion control device; The binocular camera motion control device executes the rotation instruction to drive the binocular camera to move, follow the movement of the iris and eyeball, and collect image data in real time to achieve tracking and synchronization of sight lines; The image acquisition and processing module performs image and / or video fusion processing based on the images received from the binocular camera and in combination with the head movement data and the eye movement data.

2. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: The eye tracking camera is connected to the image acquisition and processing module via a serial interface; the eye tracking camera uses two fixed micro-focus cameras to focus on the middle part of the wearer's left and right eyes respectively, and continuously acquires the wearer's eyeball images according to the imaging speed.

3. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: The binocular camera is connected to the image acquisition processing module via a parallel interface; The binocular cameras are two independent identical camera modules or physically integrated binocular modules; the camera module or the camera of the binocular module is a fixed-focus camera, an optical zoom camera or a combination thereof.

4. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: The IMU sensor is connected to the image acquisition and processing module through a serial interface, and is also used for coordinate adjustment of binocular imaging fusion and detection of equipment usage status, including equipment vibration, jitter detection and placement detection.

5. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: Also includes: Near-eye displays; The near-eye display is connected to the image acquisition and processing module via a parallel display interface, and is used to display the complete image after fusion processing.

6. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: Also includes: Data output unit; The data output unit is connected to the image acquisition and processing module via a serial port, and is used to provide synchronous external transmission services.

7. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: The image acquisition and processing module generates eye movement data according to the image data of the irises of both eyes, including: An image preprocessing unit performs grayscale conversion, noise reduction and contrast enhancement on the image data collected by the eye tracking camera in sequence; The feature extraction unit realizes the positioning of pupil and corneal reflection point based on Canny edge detection and Hough transform; A coordinate system conversion and calibration unit, which constructs a two-dimensional coordinate system with the upper left corner of the image as the origin, represents the pupil location and corneal reflection point location data in the two-dimensional coordinate system, and performs a calibration operation; The eye movement data generating unit obtains the coordinates of the eye gaze point according to the relative position relationship between the pupil center and the corneal reflection point combined with the calibrated parameters; the coordinates of the eye gaze point obtained by continuously collecting images are connected in chronological order to form eye movement data.

8. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: The image acquisition and processing module is used to convert the eye movement data and the head movement data into a rotation instruction, including: A head and eye movement parameter acquisition unit is used to convert eye movement data and head movement data into a unified global coordinate system; The azimuth adjustment unit is used to convert the movement direction and angle of the head and eyeballs in the global coordinate system into adjustment data of the binocular camera motion control device according to the principle of motion reverse compensation.

9. The intelligent head-mounted device with a camera that can rotate with the user according to claim 1, characterized in that: The image acquisition and processing module, when used for image and / or video fusion processing, includes: The image signal processing unit is used to perform bad pixel correction, filtering and denoising, color correction and gamma correction on the received left and right video data in sequence; The image fusion unit is used to spatially align the processed left and right video data, and fuse them using pixel value-based fusion, Laplace pyramid fusion, multi-resolution fusion or Poisson fusion algorithm, and then perform contrast enhancement and sharpening on the fused image.

Images