Precise mapping method and device based on image gain optimization and application

Abstract

The invention relates to a precise mapping method and device based on image gain optimization and application. A sensing hardware group is established, and an exposure strategy is initialized; establishing a visual residual optimization scheme based on image gain; acquiring feedback data of the sensing hardware group based on an initialized exposure strategy, inputting all data at the current moment into a control end, updating the exposure strategy, optimizing image features based on a visual residual optimization scheme, and outputting an updated image; the method is applied to VR equipment mapping in an environment in which light does not reach a preset standard. The device adopting the method is configured at the main control end of the VR equipment. According to the method, more accurate map points can be obtained through optimization in dark light, strong light and weak texture environments, the number of outer points of the map is small, and the map points are more stable; under the condition that the optimized map points are accurate, the point precision of each projection is higher, the mapping efficiency is improved, and the mapping effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a precise mapping method, device and application based on image gain optimization. Background Art

[0002] With the rapid development of the leisure and entertainment industry, many terminal simulators for virtual reality environments have emerged to meet users' needs for a stronger sense of control and immersion.

[0003] With the development of computer vision technology, the inside-out-in positioning method has gradually replaced the outside-out-in positioning method and has made certain progress in the fields of virtual reality and augmented reality.

[0004] The outside-in positioning method mainly performs positioning and mapping through the fusion of visual sensors and inertial measurement units (IMUs). However, in conventional applications, this method is easily affected by image noise in the process of extracting 2D features from images in dark light, strong light, and weak texture environments and obtaining 3D points through triangulation and optimizer processing. The calculated depth is inaccurate, which leads to many outliers on the map.

[0005] The inside-out positioning method can track the position through a camera combined with computer vision technology, which greatly reduces the difficulty of environmental configuration and has lower space requirements. However, it also has technical defects, including relatively low tracking accuracy, tracking blind spots, and being greatly affected by ambient lighting.

[0006] Therefore, it is urgent to combine the inside-out and outside-in positioning methods and optimize the mapping solution through environmental factors. Summary of the invention

[0007] The present invention solves the problems existing in the prior art and provides a precise mapping method, device and application based on image gain optimization.

[0008] The technical solution adopted by the present invention is a precise mapping method based on image gain optimization, which comprises the following steps:

[0009] S1: Establish the sensing hardware group and initialize the exposure strategy; establish the visual residual optimization scheme based on image gain;

[0010] S2 obtains feedback data of the sensor hardware group based on the initialization exposure strategy;

[0011] S3 inputs all data at the current moment into the control terminal, updates the exposure strategy, optimizes the image features based on the visual residual optimization scheme, and outputs the updated image;

[0012] S4 repeats S3 or ends based on the instruction.

[0013] Preferably, in S1, the sensing hardware group includes a visual sensor and an inertial measurement unit.

[0014] Preferably, the image gain-based visual residual optimization solution satisfies:

[0015]

[0016] in, is the variance, satisfying , is the visual residual represented by the i-th feature, is a robust kernel function.

[0017] In the present invention, the optimization scheme minimizes all visual residuals of multiple frame images, indicating high accuracy.

[0018] Preferably,

[0019]

[0020] Where c is the adjustment coefficient, , is the gain of the image where the i-th feature is located, () is the median function.

[0021] Preferably, a feature is extracted from the image, the feature being the central position of a certain pixel block in the image, and each feature is optimized using the visual residual optimization scheme.

[0022] In the present invention, the optimization here essentially refers to optimizing the spatial position of the feature and optimizing the reprojection error and minimizing it.

[0023] Preferably, if the sensing hardware group feedbacks that the current state is static, c is fixed until the state becomes non-static.

[0024] Preferably, if the residual optimization value is less than a preset value, the visual residual is fixed.

[0025] Preferably, in S3, all data at the current moment include an image generated by the visual sensor, a current image gain, and measurement data of an inertial measurement unit.

[0026] An application of the precise mapping method based on image gain optimization is applied to mapping of VR devices in an environment where light does not meet preset standards.

[0027] A precise mapping device based on image gain optimization is configured at the main control end of a VR device, and includes a light collection mechanism and a controller. The controller collects feedback data of a sensing hardware group and exposure strategy data of the main control end based on a light intensity signal of the light collection mechanism, and uses the precise mapping method based on image gain optimization to obtain optimized image features, and feeds back the optimized image features to the main control end.

[0028] The present invention relates to a precise mapping method, device and application based on image gain optimization, which includes establishing a sensor hardware group and initializing an exposure strategy; establishing a visual residual optimization scheme based on image gain; obtaining feedback data of the sensor hardware group based on the initialization exposure strategy, inputting all data at the current moment into a control end, updating the exposure strategy, optimizing image features based on a visual residual optimization scheme, and outputting an updated image; applying the method to VR device mapping in an environment where light does not meet a preset standard; and configuring a device using the method at a main control end of the VR device.

[0029] The beneficial effects of the present invention are:

[0030] (1) It can optimize and obtain more accurate map points in dark light, strong light and weak texture environments. There are few outliers on the map, and the map points are more stable.

[0031] (2) When the optimized map points are accurate, the point accuracy of each projection is higher, which improves the mapping efficiency and the mapping effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 Schematic diagram for effect comparison, wherein (a) is the mapping result without using the method of the present invention, and (b) is the mapping result with using the method of the present invention;

[0034] Figure 3 It is a schematic diagram of the device structure of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0036] The present invention relates to an accurate mapping method based on image gain optimization. The method updates the image gain according to data features of implementation feedback during the mapping process, thereby optimizing the mapping and obtaining a better mapping effect.

[0037] The method comprises the following steps:

[0038] (1) Establish a sensor hardware group and initialize the exposure strategy; establish a visual residual optimization scheme based on image gain;

[0039] (2) obtaining feedback data from the sensor hardware group based on the initialization exposure strategy;

[0040] (3) inputting all data at the current moment into the control end, updating the exposure strategy, optimizing the image features based on the visual residual optimization scheme, and outputting the updated image;

[0041] (4) Repeat (3) or end based on the instruction.

[0042] The following is a detailed description of the steps.

[0043] (1) Establish a sensor hardware group and initialize the exposure strategy; establish a visual residual optimization scheme based on image gain;

[0044] The sensing hardware group includes vision sensors and inertial measurement units.

[0045] In the present invention, a microcontroller unit (MCU) is set at the VR head display end to perform hardware synchronization between the visual sensor and the inertial measurement unit. The hardware synchronization here includes hardware grouping and timing synchronization, so that the data and signals output by the visual sensor and the inertial measurement unit are synchronized during feedback, which is convenient for better optimization of mapping.

[0046] Furthermore, considering that the visual sensor will adjust the exposure time and gain according to the exposure strategy in different environments, specifically, in the case of dark light and weak texture, the exposure time and gain are large, and at the same time, large image noise is generated, while in the case of strong light, the exposure time and gain are small, and the image noise is also large due to the influence of strong light. Therefore, a visual residual optimization scheme based on image gain is established to meet,

[0047]

[0048] in, is the variance, satisfying , is the visual residual represented by the i-th feature, is a robust kernel function.

[0049]

[0050] Where c is the adjustment coefficient, , is the gain of the image where the i-th feature is located, () is the median function, and in practical applications 1 / c is taken as 1.4826.

[0051] In actual application, features are extracted from images acquired by visual sensors. The features are the center positions of certain pixel blocks in the images. Each feature is optimized using the visual residual optimization scheme. The position points here can generally be extracted through FAST corner point detection. After optimization, the reprojection error values ​​of all position points are minimized.

[0052] If the sensing hardware group reports that the current state is static, that is, when the VR device is static, the image gain will have a certain jump, then c is fixed unchanged until the state is non-static, and the optimization function jump is avoided by fixing the image gain factor.

[0053] If the residual optimization value is less than the preset value, such as 50, the visual residual is fixed to reduce jitter in a weak texture environment, that is, the optimization solution is not executed.

[0054] (2) obtaining feedback data from the sensor hardware group based on the initialization exposure strategy;

[0055] In the present invention, the feedback data here refers to the image generated by the visual sensor and the data measured by the inertial measurement unit.

[0056] (3) inputting all data at the current moment into the control end, updating the exposure strategy, optimizing the image features based on the visual residual optimization scheme, and outputting the updated image;

[0057] All data at the current moment include feedback data and control data, including the image generated by the visual sensor, the exposure time and the corresponding image gain obtained based on the initialized exposure strategy, and the data measured at the inertial measurement unit.

[0058] Updating the exposure strategy involves the following steps:

[0059] S3.1 calculates the average brightness based on the pixel values ​​of the image;

[0060] S3.2 Perform two-dimensional mapping according to the average brightness and the number of features; the two-dimensional mapping here is through SVM training data, and it is expected that the average brightness of the image is generally set to 50 and the number of features is 100.

[0061] Specifically, a large number of images are collected for training, different exposure times and gains are obtained under the same environment, and different images are obtained. The average brightness is 50 and the feature is 100, which is the best. The corresponding optimal exposure time and gain data set is obtained for standby use; after obtaining a large number of corresponding optimal exposure time and gain data sets, a one-dimensional curve is fitted;

[0062] In the application, the default exposure time and gain are first obtained to obtain the initial image. When updating the exposure strategy, the corresponding exposure time and gain of the next setting are searched and set according to the average brightness and number of features of the image according to the corresponding two-dimensional curve;

[0063] (4) Repeat (3) or end based on the instruction.

[0064] The present invention also relates to an application of the precise mapping method based on image gain optimization, which is applied to mapping of VR devices in an environment where light does not meet preset standards.

[0065] In the present invention, light failing to meet the preset standard includes situations where the light intensity is too high, too low, or unevenly distributed. Generally speaking, the average grayscale value of the image is 50 as the preset standard, and the application is triggered when the average grayscale value of the collected image is less than 50.

[0066] The present invention also relates to a precise mapping device based on image gain optimization, which is configured at the main control end of a VR device and includes a light collection mechanism and a controller. The controller collects feedback data of a sensing hardware group and exposure strategy data of the main control end based on a light intensity signal of the light collection mechanism, and uses the precise mapping method based on image gain optimization to obtain optimized image features, and feeds back the optimized image features to the main control end.

[0067] In the present invention, the device herein includes but is not limited to a medium, an interface, etc., which is configured at the main control end of the VR device, and uses a light collection mechanism to collect current lighting condition signals. The controller collects the collected light intensity signals. If the conditions are not met, the feedback data of the sensor hardware group and the exposure strategy data of the main control end are collected. The image gain involved in the exposure strategy data is optimized based on a visual residual optimization scheme of the image gain, and then the optimized image features are obtained, which are output to the head display of the VR device through the main control end.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A precise mapping method based on image gain optimization, characterized in that: The method comprises the following steps: S1: Establish the sensing hardware group and initialize the exposure strategy; establish the visual residual optimization scheme based on image gain; S2 obtains feedback data of the sensor hardware group based on the initialization exposure strategy; S3 inputs all data at the current moment into the control terminal, updates the exposure strategy, optimizes the image features based on the visual residual optimization scheme, and outputs the updated image; S4 repeats S3 or ends based on the instruction.

2. The accurate mapping method based on image gain optimization according to claim 1, characterized in that: In S1, the sensing hardware group includes vision sensors and inertial measurement units.

3. The accurate mapping method based on image gain optimization according to claim 1, characterized in that: The image gain-based visual residual optimization scheme satisfies, ， in, is the variance, satisfying , is the visual residual represented by the i-th feature, is a robust kernel function.

4. The accurate mapping method based on image gain optimization according to claim 3, characterized in that: ， Where c is the adjustment coefficient, , is the gain of the image where the i-th feature is located, () is the median function.

5. The accurate mapping method based on image gain optimization according to claim 3, characterized in that: Features are extracted from the image, where the feature is the center position of a pixel block in the image, and each feature is optimized using the visual residual optimization scheme.

6. The accurate mapping method based on image gain optimization according to claim 3, characterized in that: If the sensing hardware group feedbacks that the current state is static, c is fixed and remains unchanged until the state becomes non-static.

7. The accurate mapping method based on image gain optimization according to claim 3, characterized in that: If the residual optimization value is less than the preset value, the visual residual will be fixed.

8. The accurate mapping method based on image gain optimization according to claim 2, characterized in that: In S3, all data at the current moment include the image generated by the visual sensor, the current image gain, and the measurement data of the inertial measurement unit.

9. An application of the precise mapping method based on image gain optimization according to any one of claims 1 to 8, characterized in that: Applicable to VR device mapping in environments where the light does not meet the preset standards.

10. A precise mapping device based on image gain optimization, characterized in that: A main control end configured at a VR device includes a light collection mechanism and a controller. The controller collects feedback data of a sensing hardware group and exposure strategy data of the main control end based on a light intensity signal of the light collection mechanism, and uses the precise mapping method based on image gain optimization as described in any one of claims 1 to 8 to obtain optimized image features, and feeds back the optimized image features to the main control end.

Images