Positioning method of virtual reality equipment and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides a positioning method of virtual reality equipment and the virtual reality equipment, and is used for improving the accuracy and robustness of SLAM positioning. According to an interaction scene with illumination variation, environment images synchronously collected by a multi-view camera are spliced into a spliced image, so that the edge features of the environment images collected by a monocular camera are fully utilized, the subsequent calculation accuracy of a 6Dof pose based on feature point matching is improved, meanwhile, the calling frequency of an image exposure adjustment algorithm is reduced, and the image exposure adjustment efficiency is improved. And therefore, the SLAM positioning algorithm can stably and efficiently run in a robust manner. For the overexposure or underexposure spliced image with the light intensity ratio not within the preset interval, different algorithms are adopted to adjust the image brightness so as to improve the contrast ratio of the image in the illumination change scene, so that the feature difference between different spliced images is more obvious, the feature discrimination degree is effectively improved, and the image quality is improved. And thus, the accuracy and robustness of the subsequent 6Dof pose based on feature point matching are improved.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality (VR), and provides a positioning method for a virtual reality device and a virtual reality device. Background Art

[0002] VR technology has been widely used in all walks of life by providing users with an immersive experience. In recent years, with the introduction of the concept of the metaverse and the improvement of mobile computing capabilities, the technical requirements for users to fully interact in VR scenes have become increasingly higher. The translation and rotation of the user's head in three-dimensional space is an important part of the VR interactive scene. Therefore, the key performance parameters such as the 6-degree-of-freedom (DoF) posture positioning technology, positioning accuracy and positioning stability of the user's head have become key indicators that directly affect the user's actual VR experience.

[0003] At present, the Simultaneous Localization and Mapping (SLAM) technology is usually used to calculate the 6Dof pose of the head. For SLAM technology, continuously and stably outputting high-accuracy feature point matching pairs and establishing accurate and robust maps play a decisive role in the calculation results of the 6Dof pose of the head. However, in the current VR interaction scenes, most of them are indoor scenes with variable lighting conditions, sometimes bright and sometimes dark, such as game rooms equipped with breathing lights, living rooms that change from cloudy to sunny, and rooms with thunder and lightning. During the interaction process, the illumination of the collected images changes continuously, which leads to a small number of feature point matching pairs extracted by the SLAM positioning algorithm or errors, and it is impossible to provide representative and stable feature point pairs, which leads to inaccurate mapping and reduces positioning accuracy. Summary of the invention

[0004] The embodiments of the present application provide a positioning method for a virtual reality device and a virtual reality device, which are used to reduce the influence of ambient light and improve positioning accuracy.

[0005] On the one hand, an embodiment of the present application provides a positioning method for a virtual reality device, including:

[0006] Acquire environmental images continuously captured by multiple cameras, and perform stitching processing on the environmental images synchronously captured by the multiple cameras to obtain multiple-frame stitching images;

[0007] For each stitched image, perform the following operations:

[0008] Determining a light intensity ratio of the stitched image according to a pixel value of the stitched image;

[0009] If the light intensity ratio is greater than a preset upper limit of the intensity ratio, a histogram equalization process with limited contrast is performed on the stitched image to reduce the image brightness; if the light intensity ratio is less than or equal to a preset lower limit of the intensity ratio, an image enhancement process based on deep learning is performed on the stitched image to increase the image brightness;

[0010] According to the acquisition order of the multiple-frame stitched images, by changing the number of image channels, every n-frame stitched images in the multiple-frame stitched images are stacked to obtain an input image set, where n is an integer greater than 1;

[0011] According to the input image set and combined with its own IMU data, the 6DOf pose of the user's head is determined.

[0012] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, a display, and a multi-eye camera, wherein the multi-eye camera, the display, the memory, and the processor are connected via a bus:

[0013] The memory stores computer program instructions, and the processor performs the following operations according to the computer program instructions:

[0014] Acquire the environment images continuously captured by the multi-eye camera, and perform stitching processing on the environment images synchronously captured by the multi-eye camera to obtain a multi-frame stitching image;

[0015] For each stitched image, perform the following operations:

[0016] Determining a light intensity ratio of the stitched image according to a pixel value of the stitched image;

[0017] If the light intensity ratio is greater than a preset intensity ratio upper limit, performing a contrast-limited histogram equalization process on the stitched image to reduce image brightness;

[0018] If the light intensity ratio is less than or equal to a preset intensity ratio lower limit, performing deep learning-based image enhancement processing on the stitched image to increase image brightness;

[0019] According to the acquisition order of the multiple-frame stitched images, by changing the number of image channels, every n-frame stitched images in the multiple-frame stitched images are stacked to obtain an input image set, where n is an integer greater than 1;

[0020] The 6DOf posture of the user's head is determined according to the input image set and combined with its own IMU data, and the 6DOf posture is used to control the application interface displayed on the display.

[0021] Optionally, the processor determines the light intensity ratio of the stitched image according to the pixel value of the stitched image, and the specific operation is:

[0022] Performing grayscale processing on the stitched image to obtain a pixel mean value and a maximum pixel value after the grayscale processing;

[0023] The ratio of the pixel mean value to the maximum pixel value is used as the light intensity ratio of the stitched image.

[0024] Optionally, the processor performs a contrast-limited histogram equalization process on the stitched image to reduce image brightness, and the specific operation is:

[0025] Divide the grayscaled stitched image into M*M image blocks, and obtain the cumulative probability distribution of each gray level according to the probability distribution corresponding to each gray level in each image block, where M is an integer greater than or equal to 2;

[0026] According to the cumulative probability distribution of each gray level in each image block, the number of pixels in the corresponding image block whose gray levels exceed the pixel number threshold is obtained, and the exceeding number of pixels is evenly distributed to each gray level to obtain a new cumulative probability distribution;

[0027] For pixel values ​​in different position areas in the spliced ​​image, different mappings are performed using the new cumulative probability distribution of each image block to obtain new pixel values ​​of grayscale histogram equalization;

[0028] According to the new pixel value of each image block, a spliced ​​image with reduced image brightness is obtained.

[0029] Optionally, the processor calculates a new pixel value of a corresponding image block using a new cumulative probability distribution according to the position of each image block in the spliced ​​image, and the specific operation is:

[0030] For the pixel values ​​in the four corner regions of the stitched image, the product of the new cumulative probability distribution of the target image block where the four corner regions are located and the maximum pixel value is taken as the new pixel value in the four corner regions;

[0031] For the pixel values ​​in the edge area of ​​the stitched image, the new cumulative probability distributions of the target image block where the original pixel value is located and an adjacent image block of the target image block are respectively multiplied by the product of the maximum pixel value to obtain two initial pixel values, and the center distance ratios of the original pixel value to the target image block and the adjacent image block are calculated, and the new pixel value in the edge area is obtained by combining the two initial pixel values;

[0032] For the pixel values ​​in the middle area of ​​the stitched image, the new cumulative probability distributions of the target image block where the original pixel value is located and the three adjacent image blocks of the target image block are multiplied by the maximum pixel value respectively to obtain four initial pixel values, and the center distance ratios of the original pixel value to the target image block and each adjacent image block are calculated, and the new pixel value in the middle area is obtained by combining the four initial pixel values.

[0033] Optionally, the processor performs deep learning-based image enhancement processing on the stitched image to increase image brightness, and the specific operation is:

[0034] Decomposing the stitched image into an initial reflection image and an initial illumination image by convolution processing;

[0035] Obtaining an initial illumination image after noise removal according to the initial reflection image and the initial illumination image;

[0036] According to the initial illumination image after noise removal and the initial reflection image, a multi-scale brightness attention mechanism is used to restore the light reflectance information to obtain a target reflection image;

[0037] According to the illumination adjustment parameter corresponding to the illumination intensity ratio of the stitched image, the initial illumination image after noise removal is adjusted to increase the brightness to obtain a target illumination image;

[0038] The target illumination image and the target reflection image are multiplied pixel by pixel to obtain a spliced ​​image with enhanced image brightness.

[0039] Optionally, the processor uses a multi-scale brightness attention mechanism to restore light reflectance information according to the initial illumination image after noise removal and the initial reflection image to obtain the target reflection image, and the specific operation is:

[0040] After performing a multi-channel convolution operation and a multi-channel activation function processing on the initial reflection image, the initial reflection image is multiplied pixel by pixel with the initial illumination image after noise removal to obtain a fused image;

[0041] After performing pooling processing on the fused images respectively using pooling layers with different numbers of layers, a multi-scale brightness attention mechanism is used to perform convolution and deconvolution operations on the pooled images respectively and a convolution operation is performed on the fused images to restore light reflectance information;

[0042] The initial reflection image after convolution activation is adjusted according to the light reflectivity information to obtain a target reflection image.

[0043] Optionally, the processor increases the brightness of the initial illumination image after noise removal according to the illumination adjustment parameter corresponding to the illumination intensity ratio of the stitched image to obtain the target illumination image, and the specific operation is:

[0044] The illumination adjustment parameter is used as a constant channel to be spliced ​​with the initial illumination image after noise removal to obtain an illumination enhanced image;

[0045] The illumination enhancement image is subjected to convolution and activation function processing to obtain a target illumination image.

[0046] Optionally, the formula of the illumination adjustment parameter is expressed as:

[0047]

[0048] Wherein, α represents the illumination adjustment parameter, β represents the illumination intensity ratio of the spliced ​​image, and P mean represents the pixel mean of the stitched image, P max Indicates the maximum pixel value of the stitched image.

[0049] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a positioning method for a virtual reality device provided in an embodiment of the present application.

[0050] The beneficial effects of the positioning method of the virtual reality device provided in the embodiment of the present application are as follows:

[0051] By stitching together the environmental images collected synchronously by multiple cameras into one stitched image, the edge features on the environmental images collected individually by each camera can be fully utilized, thereby improving the accuracy of the subsequent 6Dof pose calculation based on feature points. At the same time, stitching together the environmental images collected synchronously by multiple cameras into one stitched image for processing can reduce the number of algorithm calls and improve the positioning speed. On the other hand, in order to reduce the impact of illumination changes on feature point extraction, it is determined whether the light intensity ratio determined by the pixel value of each frame of the stitched image is within the preset range, and different algorithms are used to adjust the image brightness for the overexposure and underexposure of the image that is not within the preset range, thereby improving the contrast of the stitched image in the illumination change scene, making the feature differences between different stitched images more obvious, effectively improving the feature distinction, and thus improving the accuracy and robustness of the 6Dof pose calculation based on feature points.

[0052] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A schematic diagram of a lighting change scene provided in an embodiment of the present application;

[0055] Figure 2 A positioning framework diagram of a virtual display device provided in an embodiment of the present application;

[0056] Figure 3 A flow chart of a positioning method for a virtual reality device provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the multi-camera stitching process provided in an embodiment of the present application;

[0058] Figure 5 A flowchart of the adaptive histogram equalization process with limited contrast provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of a process for determining pixel values ​​for histogram equalization according to an embodiment of the present application;

[0060] Figure 7 A schematic diagram of contrast-limited adaptive histogram equalization provided in an embodiment of the present application;

[0061] Figure 8 A flowchart of adjusting a lightweight deep learning image enhancement network provided in an embodiment of the present application;

[0062] Fig. 9 A structural diagram of a lightweight deep learning enhancement network provided in an embodiment of the present application;

[0063] Fig.10 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0065] Currently, VR devices are mostly used in indoor scenes, where the brightness of light may change.

[0066] For example, Figure 1 As shown, when the user is experiencing VR in an indoor scene equipped with a breathing light, the head will move continuously. Since the light of the breathing light will gradually change from bright to dark under the control of the microcomputer, the brightness of the environmental images captured in the two frames before and after the movement will be different.

[0067] If the positioning and tracking technology of the head's 6Dof posture is not accurate and robust enough, it may affect the feature points extracted from the image, and then affect the positioning results of the VR device in scenes with changing lighting, making it difficult for the VR device to cope with special scenes such as lighting changes. This will inevitably cause certain troubles in the user experience, such as: the virtual scene cannot be positioned, drifts, etc., and thus cannot meet the user's normal interactive experience needs.

[0068] In view of this, the embodiment of the present application provides a positioning method for a virtual reality device, which adopts deep learning technology and histogram equalization technology, designs an adaptive image exposure adjustment algorithm, and can adjust the brightness of the environment image collected by the multi-eye camera on the virtual reality device in a special interactive scene with changing lighting, thereby improving the image contrast, making the feature difference between the images more obvious, effectively improving the feature distinction, and then improving the accuracy and robustness of the subsequent 6Dof pose calculation based on feature points. At the same time, in order to further improve the accuracy of feature point matching, reduce the time consumption of the positioning algorithm, and enable the algorithm to run stably and robustly, before adjusting the brightness of the environment image collected by the multi-eye camera, multiple environment images collected synchronously by the multi-eye camera are spliced ​​into a spliced ​​image, which can make full use of the edge features on the environment image collected by each camera separately and reduce the number of algorithm calls, thereby improving the positioning accuracy and efficiency, ensuring the robustness of the virtual reality device positioning in the virtual interactive scene, and enabling the user of the virtual reality device to have a better immersive experience.

[0069] See also Figure 2, which is a positioning framework diagram of a virtual display device provided in an embodiment of the present application, mainly includes two parts: adaptive illumination change adjustment and SLAM positioning. Among them, in the adaptive illumination change adjustment part, the environment image collected by the multi-eye camera on the virtual reality device is firstly spliced ​​to improve the subsequent positioning accuracy and positioning efficiency based on feature point matching; then, for each frame of the spliced ​​image, the light intensity ratio of the spliced ​​image is calculated, and the light intensity ratio is compared with the preset threshold. When the image is normally exposed, no processing is performed, and when the image is overexposed, an adaptive histogram equalization method with limited contrast is used to reduce the brightness of the spliced ​​image, and when the image is underexposed, a deep learning network is used to increase the brightness of the spliced ​​image, thereby reducing the influence of the illumination change ring on the SLAM positioning result and improving the accuracy and robustness of SLAM positioning; finally, the spliced ​​image after brightness adjustment is stacked into a multi-channel input image, thereby reducing the number of calls to the SLAM positioning algorithm and improving the positioning efficiency. In the SLAM positioning part, it mainly includes three steps: front-end feature extraction, back-end pose estimation and loop detection. The front-end feature extraction is used to calculate the camera pose change between two frames based on the camera's captured images, and pass the change to the back-end for optimization; the back-end pose estimation is used to estimate the optimal trajectory and map from the noisy pose data transmitted from the front end; loop detection is used to reduce the pose drift generated by the back-end estimation, constrain the same pose to the same point, and output an accurate 6Dof pose.

[0070] The positioning method of the virtual reality device provided in the embodiment of the present application is as follows: Figure 3 As shown, it mainly includes the following steps:

[0071] S301: Acquire environmental images continuously captured by multiple cameras, and perform stitching processing on the environmental images synchronously captured by the multiple cameras to obtain a multi-frame stitching image.

[0072] Most virtual display devices have multiple cameras (such as binocular or quadruple cameras) installed, so the images collected by each camera need to be processed. At present, when using a single-eye adaptive image exposure adjustment algorithm to process multi-eye data, most of them are implemented by calling the monocular image algorithm multiple times. Although this implementation method is simple, the algorithm is called in series and cannot fully utilize the parallel processing capabilities of the CPU, resulting in multiple wastes of computing resources and increased time consumption.

[0073] In an optional implementation, the embodiment of the present application performs stitching processing on environmental images synchronously captured by multiple cameras, wherein the image stitching method is consistent with the camera position to ensure that the edge pixel changes of the camera at each position have a certain regularity.

[0074] Take the four-eye camera as an example. Figure 4As shown in the figure, the Float32 type stitching images are stitched in the following manners: 0: image of the lower left camera of the VR device, 1: image of the lower right camera of the VR device, 2: image of the upper left camera of the VR device, and 3: image of the upper right camera of the VR device. At this time, assuming that the resolution of the monocular image is 640×480×1, the resolution of the image stitched by the four-camera stitching is 2560×1920×1.

[0075] By stitching together the environmental images synchronously captured by multiple cameras, the brightness of the multiple images can be uniformly adjusted by calling the adaptive image exposure adjustment only once, thereby improving the efficiency of the adaptive image exposure adjustment. At the same time, the image edges of the multiple cameras in the stitched image can form additional edge features relative to the edges of the independent monocular images. This ensures that more abundant and stable feature points can be extracted from the stitched image, thereby improving the stable and robust operation of the SLAM positioning algorithm.

[0076] For each frame of the stitched image, when the adaptive image exposure adjustment algorithm is called, the following operations S302 to S306 are performed.

[0077] S302: Determine the light intensity ratio of the stitched image according to the pixel value of the stitched image.

[0078] In indoor lighting change scenes, lighting parameters are constantly changing, and an indicator that can evaluate light levels is needed to evaluate the relative light changes between images. For grayscale images, the pixel value is 0 to 255. When the pixel value is 0, the image appears black, and when the pixel value is 255, the image appears white. Therefore, the evaluation indicator of light levels can be determined based on the pixel value of the spliced ​​image after grayscale processing.

[0079] In one example, the stitched image is grayed, the pixel mean and maximum pixel value after the graying process are obtained, and the ratio of the pixel mean and the maximum pixel value is used as the light intensity ratio of the stitched image, which is used as an evaluation index of the light level. The formula is as follows:

[0080]

[0081] Among them, β represents the illumination intensity ratio of the stitched image, P mean represents the pixel mean of the spliced ​​image, P max Indicates the maximum pixel value of the stitched image.

[0082] S303: Determine whether the light intensity ratio is greater than a preset intensity ratio upper limit, if so, execute S304, if not, execute S305.

[0083] Light changes may cause the image to be too bright or too dark. Therefore, the intensity ratio threshold can be set to determine whether the image is overexposed or underexposed.

[0084] Optionally, the upper limit of the preset intensity ratio is 0.8.

[0085] S304: performing a contrast-limited histogram equalization process on the stitched image to reduce image brightness.

[0086] When the illumination intensity ratio is greater than the preset upper limit of the intensity ratio, at this time, most of the pixel values ​​in the stitched image are close to the maximum pixel value of 255 of the grayscale image, that is, the pixel values ​​are too concentrated on one end, resulting in unclear image details. The image appears overexposed and brighter. The image brightness can be reduced through adaptive histogram equalization processing with limited contrast, making the distribution of pixel values ​​more uniform, thereby enhancing the contrast of the stitched image.

[0087] In specific implementation, the adaptive histogram equalization process with limited contrast is as follows: Figure 5 As shown, it mainly includes the following steps:

[0088] S304 1: Divide the grayscaled spliced ​​image into M*M image blocks, and obtain the cumulative probability distribution of each gray level according to the probability distribution corresponding to each gray level in each image block.

[0089] The traditional histogram equalization (Hisogram Equalization) image enhances the global image when enhancing the image contrast, and does not consider the local information. Therefore, the embodiment of the present application divides the spliced ​​image after grayscale processing into M*M image blocks, thereby realizing adaptive histogram equalization (Adaptive Hisogram Equalization) of local image blocks. Wherein, M is an integer greater than or equal to 2, and optionally, M=8, such as Figure 6 shown.

[0090] At the same time, in order to avoid image discontinuity and over-enhancement caused by AHE, the embodiment of the present application adopts a method of limiting histogram distribution to enhance image contrast.

[0091] In a specific implementation, for each image block, the grayscale of the image block is determined according to the maximum pixel value and the minimum pixel value of the image block, the number of pixel values ​​in each grayscale in the entire image block is counted, and the probability distribution corresponding to each grayscale is determined in combination with the total number of pixels in the image block, and the cumulative probability distribution of each grayscale is obtained according to the probability distribution corresponding to each grayscale, such as the cumulative probability distribution of the first grayscale is the probability distribution of the grayscale (that is, the ratio of the number of pixels of the grayscale to the total number of pixels), the cumulative probability distribution of the second grayscale is the sum of the probability distribution of the grayscale and the cumulative probability distribution of the previous grayscale, the cumulative probability distribution of the second grayscale is the sum of the probability distribution of the grayscale and the cumulative probability distribution of the previous two grayscales, and so on.

[0092] S3042: According to the cumulative probability distribution of each gray level in each image block, the number of pixels in the corresponding image block whose gray levels exceed the pixel number threshold is obtained, and the excess number of pixels is evenly distributed to each gray level to obtain a new cumulative probability distribution.

[0093] In one example, the pixel number threshold for limiting the histogram can be determined according to the width and height of each image block. Specifically, the pixel number threshold for each image block is Among them, W is the width of the stitched image, H is the height of the stitched image, and M is the number of divided image blocks, that is, each image block is divided into the number of pixels of each gray level, and then 4 times is taken as the pixel number threshold of the grayscale histogram.

[0094] After determining the pixel number threshold of the grayscale histogram of each image block, for each image block, according to the cumulative probability distribution of each gray level in the image block, the number of pixels in each gray level in the image block that exceeds the pixel number threshold is obtained, and the exceeding number of pixels is evenly distributed to each gray level to obtain a new cumulative probability distribution.

[0095] The grayscale histogram equalization process with limited contrast is as follows Figure 7 As shown, the number of pixels exceeding the pixel number threshold is filled with a slash. After this part of the pixels is evenly distributed to each gray level, the situation where the number of pixels at a certain gray level is too high is reduced, making the gray distribution histogram of the image block more balanced.

[0096] S3043: For pixel values ​​in different position areas in the spliced ​​image, different mappings are performed using the new cumulative probability distribution of each image block to obtain new pixel values ​​of grayscale histogram equalization.

[0097] Generally, the entire stitched image consists of three parts: the four corner areas, the edge area, and the middle area, such as Figure 6 After obtaining the new cumulative probability distribution of each image block, different mapping rules are used to map the pixel values ​​of different position areas in each frame of the spliced ​​image to obtain a new balanced pixel value.

[0098] For the pixel values ​​in the four corner regions of the stitched image, the product of the new cumulative probability distribution of the target image block where the four corner regions are located and the maximum pixel value is taken as the new pixel value in the four corner regions.

[0099] like Figure 6 As shown in the local enlarged image of the 4*4 image block in the figure, for the upper left corner area of ​​the spliced ​​image, the new cumulative probability distribution of the target image block is recorded as P b , then the new pixel value after equalization in the upper left corner area is P = P b *255.

[0100] For the pixel values ​​in the edge area of ​​the stitched image, the new cumulative probability distribution of the target image block where the original pixel value is located and an adjacent image block of the target image block are multiplied by the maximum pixel value to obtain two initial pixel values, and the center distance ratio of the original pixel value to the target image block and the adjacent image block is calculated. The two initial pixel values ​​are combined to obtain the new pixel value in the edge area.

[0101] like Figure 6 As shown in the partial enlarged image of the 4*4 image block in the figure, the original pixel values ​​in the upper edge area of ​​the spliced ​​image are represented by a five-pointed star, and the new cumulative probability distribution of the target image block where it is located is denoted by P b1 , the new cumulative probability distribution of an adjacent image block on the right side of the target image block is recorded as P b2 , then the initial pixel value of the original pixel value is P1=P b1 *255, another initial pixel value is P2=P b2 *255, the first distance from the original pixel value to the center point of the target image block (represented by a small circle) is recorded as d b1 The second distance from the original pixel value to the center point of the adjacent image block (represented by a small circle) is recorded as d b2 , then the equalization weight of the target image block to the original pixel value is The equalization weight of the original pixel value of the adjacent image blocks is C2=1-C1. At this time, the new pixel value after the original pixel value in the upper edge area is equalized is P=P1*C1+P2*C2.

[0102] For the pixel values ​​in the middle area of ​​the stitched image, the new cumulative probability distribution of the target image block where the original pixel value is located and the three adjacent image blocks of the target image block are multiplied by the maximum pixel value to obtain four initial pixel values, and the center distance ratios of the original pixel value to the target image block and each adjacent image block are calculated. The new pixel value in the middle area is obtained by combining the four initial pixel values.

[0103] like Figure 6As shown in the partial enlarged image of the 4*4 image block in the figure, the original pixel values ​​in the middle edge area of ​​the spliced ​​image are represented by a small square box, and the new cumulative probability distribution of the target image block where it is located is denoted by P b1 , the new cumulative probability distributions of the three adjacent image blocks above, to the upper left, and to the left of the target image block are recorded as P b2 , P b3 , P b4 , then the four initial pixel values ​​of the original pixel value are P1 = P b1 *255, P2=P b2 *255, P3=P b3 *255, P4=P b4 *255, the first horizontal distance from the original pixel value to the center point of the upper left adjacent image block (represented by a small circle) is recorded as d b1 , the first longitudinal distance is denoted as d b2 The second horizontal distance from the original pixel value to the center point of the target image block (represented by a small circle) is recorded as d b3 , the second longitudinal distance is recorded as d b4 , then the horizontal equalization weight of the upper left adjacent image block to the original pixel value is The vertical balance weight is The horizontal equalization weight of the upper adjacent image block for the original pixel value is 1-C1, and the vertical equalization weight is C2. The horizontal equalization weight of the left adjacent image block for the original pixel value is C1, and the vertical equalization weight is 1-C2. The horizontal equalization weight of the target image block for the original pixel value is 1-C1, and the vertical equalization weight is 1-C2. At this time, the new pixel value after the original pixel value in the middle area is equalized is P=P1*C1*C2+P2*(1-C1)*C2+P3*C1*(1-C2)+P4*(1-C1)*(1-C2).

[0104] S3044: Obtain a spliced ​​image with reduced image brightness according to the new pixel value of each image block.

[0105] In an embodiment of the present application, for overexposed stitched images, adaptive histogram equalization processing with limited contrast is performed to make the new pixel values ​​in each image block more evenly distributed, thereby reducing the overall brightness of the stitched image and improving the image contrast. In this way, when the SLAM technology based on feature point matching is subsequently adopted, the difference in feature points between different image frames can be increased, the accuracy of feature matching can be improved, and the accuracy of SLAM positioning can be improved.

[0106] In the embodiment of the present application, for an overexposed stitched image, when the brightness is adjusted by using an adaptive histogram with limited contrast, the stitched image is divided into M*M small image blocks, and then the cumulative probability distribution of each small image block is calculated, and a pixel number threshold is set, and the number of pixels close to the maximum pixel value that exceeds the preset number threshold is evenly distributed in the overall histogram of the pixel values ​​of each gray level, so that the distribution of the pixel values ​​is more uniform, and then according to the mapping rules of the image area where each pixel value is located, different methods are used to calculate the new pixel value after equalization, thereby reducing the overall brightness of the image. Compared with hardware automatic exposure technology, such as: changing the sensitivity, exposure time and using flash, the embodiment of the present application can suppress image noise while changing the image brightness, and will not introduce additional highlights and unbalanced light that do not belong to the scene on the image, and provide a stable brightness and scene stable image as the input of the SLAM positioning technology, thereby improving the stability and accuracy of positioning.

[0107] S305: Determine whether the light intensity ratio is less than or equal to a preset intensity ratio lower limit, if so, execute S306, otherwise no processing is performed.

[0108] When the stitched image does not have an overexposure problem, it is also necessary to compare the light intensity ratio with the preset intensity ratio lower limit to determine whether the stitched image has an underexposure problem. If there is neither an overexposure problem nor an underexposure problem, it is considered to be normally exposed and there is no need to adjust the brightness of the stitched image.

[0109] Optionally, the preset lower limit of the intensity ratio is 0.3.

[0110] S306: Perform deep learning-based image enhancement processing on the stitched image to increase image brightness.

[0111] When the illumination intensity ratio is less than or equal to the preset intensity ratio lower limit, at this time, most of the pixel values ​​in the stitched image are concentrated in the dark gray value area, resulting in unclear image details, underexposure of the image, and overall darker appearance. Image enhancement processing can be performed through a lightweight deep learning image enhancement network to increase the brightness of the image details while ensuring the operating efficiency of the overall algorithm.

[0112] In specific implementation, the brightness adjustment process based on the lightweight deep learning image enhancement network is as follows: Figure 8 As shown, it mainly includes the following steps:

[0113] S3061: Decomposing the stitched image into an initial reflection image and an initial illumination image through convolution processing.

[0114] For images captured in an environment with weak global light, if the visibility is enhanced directly by increasing the brightness, it will easily cause partial overexposure of the image and introduce noise because the actual lighting factors are not taken into account. Based on the Retinex theory, it can be known that an image is composed of the pixel-by-pixel multiplication of the illumination image and the reflected image of the object, and the reflected image of the object is the constant image determined by the object, which will not change due to changes in illumination. Therefore, the purpose of image enhancement can be achieved by correcting the illumination component. Therefore, the embodiment of the present application decomposes the underexposed stitched image into the initial reflected image and the initial illumination image through a layer of convolution operation, and decouples the stitched image processed with brightness enhancement in the original space into two smaller image spaces for processing, thereby solving the problem of partial image overexposure.

[0115] S3062: Obtain an initial illumination image after noise removal according to the initial reflection image and the initial illumination image.

[0116] After decomposing the underexposed stitched image into an initial reflection image and an initial illumination image, the initial reflection image is convolved and then cascaded with the initial illumination image to obtain a cascaded image, which is then processed by single-channel convolution and single-channel activation functions. This eliminates the unextracted reflection texture features (i.e., considered as noise) from the initial illumination image, and obtains the initial illumination image after noise removal, thereby improving the accuracy of image brightness enhancement processing.

[0117] Since the initial reflection image and the initial illumination image are decomposed from the same frame of the stitched image, the resolution of the initial reflection image and the initial illumination image is the same and equal to the resolution of the stitched image. Different from the length and width stitching between images, the cascade between images can be understood as the superposition of the number of channels, and the formula is (x1, y1, z1) + (x1, y1, z2) = (x1, y1, z1 + z2), where (x1, y1) represents the resolution of the initial reflection image and the initial illumination image, z1 and z2 represent the number of channels of the initial reflection image and the initial illumination image respectively, and the values ​​of z1 and z2 of the grayscale image are 1.

[0118] S3063: According to the initial illumination image and the initial reflection image, a multi-scale brightness attention mechanism is used to restore the light reflectance information to obtain the target reflection image.

[0119] Different from the decomposed initial illumination image, the initial reflection image will be processed by multi-channel convolution operation and multi-channel activation function, and then it will be multiplied pixel by pixel with the initial illumination image after noise removal to obtain a fused image. After the fused image is pooled by different numbers of pooling layers, the multi-scale brightness attention mechanism is used to perform convolution and deconvolution operations on the pooled image and convolution operation on the fused image to restore the light reflectance information. The initial reflection image after convolution activation is adjusted according to the light reflectance information to obtain the target reflection image.

[0120] Since noise mainly appears in areas with low light levels and the noise attenuation distribution depends on the light distribution, and the single-scale fully convolutional neural network ignores the impact of image brightness on image noise at different scales, the embodiment of the present application uses a multi-scale brightness attention mechanism to replace the traditional fully convolutional neural network in the reflectivity restoration branch, thereby more accurately repairing the reflection information based on the image brightness features at image levels of different scales and improving the accuracy of brightness enhancement.

[0121] S3064: According to the illumination adjustment parameter corresponding to the illumination intensity ratio of the stitched image, the initial illumination image after noise removal is adjusted to increase the brightness, so as to obtain a target illumination image.

[0122] In one example, when a lightweight deep learning-based image enhancement network is used to perform brightness enhancement processing on underexposed stitched images, the light intensity ratio index used to evaluate the light level is used as part of the network's light intensity adjustment parameter, so that the network can adaptively adjust the brightness of the output image according to the characteristics of the image itself. Among them, when the degree of underexposure of the input image is higher (that is, the value of the light intensity ratio β is smaller), the brightness of the output image needs to be higher (that is, the value of the light adjustment parameter α is higher), and the formula of the light adjustment parameter is expressed as:

[0123] α=(0.5-β)+1 Formula 2

[0124] Among them, α represents the illumination adjustment parameter, and β represents the illumination intensity ratio of the stitched image. From Formula 2, it can be seen that when β≤0.3, α>1, which means adjusting the lower level of light to a higher level.

[0125] Specifically, the brightness increase adjustment process of the initial illumination image after noise removal includes: the illumination adjustment parameter is a parameter of the network itself, which can be expanded to a vector matrix with the same dimension as the initial illumination image after noise removal. Therefore, the illumination adjustment parameter can be used as a constant channel to be spliced ​​with the initial illumination image after noise removal to obtain an illumination enhanced image, and then the illumination enhanced image is subjected to 3-layer convolution operation, single-channel convolution operation and single-channel activation function processing to obtain a target illumination image.

[0126] S3065: Multiply the target illumination image and the target reflection image pixel by pixel to obtain a spliced ​​image with enhanced image brightness.

[0127] The target illumination image and target reflection image obtained by spatial decoupling of two smaller images are merged into a mosaic image with enhanced image brightness by pixel-by-pixel multiplication. The formula is expressed as:

[0128] I=R3°L3 Formula 3

[0129] Among them, I represents the stitched image, R3 represents the target reflection image, L3 represents the target illumination image, and ° represents pixel-by-pixel multiplication.

[0130] In one example, the structure of a lightweight deep learning enhancement network for brightness enhancement adjustment of underexposed stitched images is as follows: Fig. 9 As shown in the figure, the typical 5-layer U-Net algorithm is cropped and a 3-layer U-Net network is used as the processing network of the reflection image branch, which mainly includes the image decomposition part (D1 and D2 modules), the reflectivity recovery part (D3 module), the illumination adjustment part (D4 module) and the image fusion part (D5 module). Among them, the image decomposition part is used to decompose the input image into a reflection image and an illumination image; the reflectivity recovery part is used to generate a new reflection image according to the reflection image and the image illumination image; the illumination adjustment part is used to generate a new illumination image according to the illumination image and the illumination adjustment parameters; the image fusion part is used to fuse the new reflection image and the new illumination image into an image enhanced.

[0131] Specifically, the input image I is decomposed into an initial reflection image R and an initial illumination image L after the first convolution layer in the three-layer U-Net network in the D1 module; the D2 module cascades the initial illumination image L and the initial reflection image R1 processed by the three-layer U-Net network, excludes the reflection texture features not extracted by the network (i.e., regarded as noise) from the initial illumination image L, and obtains the initial illumination image L2 after noise removal; since the distribution of noise is related to the illumination distribution, the initial illumination image L2 processed by the D2 module and the initial illumination image L1 processed by the D1 module are restored after the reflectivity recovery module D3. The reflected image R2 is input into the reflectivity recovery module D3 to obtain the target reflected image R3, wherein D3 includes two maximum pooling layers, three convolution layers and two deconvolution layers; the initial illumination image L2 processed by the D2 module and the illumination adjustment parameter α determined by the illumination intensity ratio are input into the illumination adjustment module D4, and after splicing, convolution and activation processing, the target illumination image L3 after brightness adjustment of the initial illumination image L2 is obtained; the D5 module fuses the target illumination image L3 output by the D4 module and the target reflected image R3 output by D3, and outputs the stitched image after image enhancement.

[0132] In the embodiment of the present application, for underexposed stitched images, by decomposing them into illumination images and reflection images, decoupling the illumination images and reflection images for image enhancement is performed, thereby preventing overexposure of image enhancement due to uneven illumination distribution and improving the accuracy of brightness enhancement. At the same time, in order to increase the speed of feature detection, the embodiment of the present application simplifies the deep learning algorithm. Compared with general image enhancement methods, by using a multi-scale brightness attention mechanism, the input image is used at multiple levels in terms of brightness on the basis of increasing the running speed, thereby improving the accuracy of feature detection, and being able to better process the brightness change information in the image while keeping the image resolution unchanged, improving the reflectivity recovery and noise reduction effects, and being more efficient than the encoding and decoding deep learning algorithm.

[0133] S307: According to the acquisition order of the multiple-frame stitching images, by changing the number of image channels, every n-frame stitching images in the multiple-frame stitching images are stacked to obtain an input image set.

[0134] Currently, in SLAM positioning scenarios, in order to realize the parallel calling of monocular image algorithms, related technologies usually fuse multi-eye data into one image as input and embed the position information between multi-eye cameras.

[0135] Taking a four-eye camera as an example, the resolution of a monocular image is 640×480 (pixels). n (n is an integer greater than 1, such as previous and next frames) monocular grayscale images of the four-eye camera, 4×640×480×1, are packaged into n data of 2560×1920×1, which are then called in parallel by the image processing algorithm deployed by the platform.

[0136] However, not all platforms where image processing algorithms are deployed support packaged data as input, so this method has certain limitations and cannot be widely promoted.

[0137] In order to complete the adaptive image exposure adjustment task of multi-view images in a single processing time and to be applicable to the support of parallel processing on most platforms, a batch data stacking method is designed in the embodiment of the present application. For the multi-frame stitched images after brightness processing by the multi-view camera, every n frames of the stitched images in the multi-frame stitched images are stacked according to the image acquisition order by changing the number of image channels to obtain the input image set of the SLAM algorithm.

[0138] Still taking the images captured by the four-eye camera as an example, unlike the batch processing method of packing n grayscale images of 4×640×480×1 into n×2560×1920×1, the embodiment of the present application packs n grayscale stitched images of 2560×1920×1 into 2560×1920×n n-channel images. In this way, n grayscale stitched images of 2560×1920×1 will be regarded as 1 n-channel image of 2560×1920×n and input into the SLAM algorithm, that is, the SLAM model only needs a single execution to complete the key point estimation of the multi-eye image, thereby improving the positioning speed. At the same time, the input of the model is a single n-channel data, and the algorithm framework does not need to support the Batch input function, and has a wider range of applications.

[0139] S308: Determine the 6DOf pose of the user's head based on the input image set and its own IMU data.

[0140] In the SLAM positioning process, feature points are extracted from the input image and feature matching is performed. The relative pose of the two frames before and after the virtual reality device is calculated based on the matching results, and the relative pose is optimized in combination with its own IMU data to obtain an accurate and robust 6DOf pose of the head.

[0141] In order to enable the virtual reality device to still extract accurate and robust feature points in the exchange scene with illumination change, thereby improving the accuracy and robustness of positioning and mapping, the embodiment of the present application designs an adaptive image exposure adjustment algorithm based on the deep learning algorithm and the histogram equalization algorithm, and stitches the environmental images synchronously acquired by multiple cameras into a stitched image, thereby performing adaptive image exposure adjustment on the stitched image. Since the stitched image can effectively utilize the edge information on the image generated by each camera separately, it helps to improve the accuracy of feature point matching in the subsequent SLAM algorithm. At the same time, after multiple environmental images are stitched into one image, the adaptive image exposure adjustment algorithm only needs to be called once to achieve the brightness adjustment of the multi-eye images, thereby reducing the time consumption of the algorithm, enabling the algorithm to run stably and robustly, and improving the positioning efficiency. Moreover, in the process of image exposure adjustment, the exposure of the current stitched image is judged by calculating the light intensity ratio used to measure the light level in each frame of the stitched image after grayscale conversion. When the light intensity ratio is greater than the preset upper limit of the intensity ratio, it is determined that the current stitched image has an overexposure problem, and the exposure is suppressed and adjusted by using an adaptive histogram equalization method with limited contrast. When the light intensity ratio is less than or equal to the preset lower limit of the intensity ratio, it is determined that the current stitched image has an underexposure problem. At this time, a lightweight deep learning image enhancement network is used to enhance the brightness, and the light intensity ratio is used as part of the illumination adjustment parameters in the network to improve the image brightness of the output image. When the light intensity ratio is greater than the preset lower limit of the intensity ratio and less than or equal to the preset upper limit of the intensity ratio, it is regarded as normal exposure and no processing is performed. Through adaptive image exposure adjustment, the impact of illumination changes on the image can be reduced, the contrast of the image can be improved, the difference between image features can be more obvious, and the extracted features can be more distinguishable, so as to achieve stable and robust indoor scene positioning and mapping, and ensure a good immersive experience for users. Compared with single low-light image enhancement algorithms and traditional image processing methods, the adaptive image exposure adjustment algorithm can adaptively adjust image exposure according to scene lighting changes, and no longer uses fixed exposure parameters, thereby more intelligently and efficiently improving the accuracy of positioning and mapping of virtual reality devices in lighting-changing scenes.

[0142] Based on the same technical concept, an embodiment of the present application provides a virtual reality device, which can implement the steps of the positioning method of the above-mentioned virtual reality device and achieve the same technical effect.

[0143] See also Fig.10 The virtual reality device includes a processor 1001, a memory 1002, a display 1003 and a multi-eye camera 1004. The multi-eye camera 1004, the display 1003, the memory 1002 and the processor 1001 are connected via a bus 1005:

[0144] The memory 1002 stores computer program instructions, and the processor 1001 performs the following operations according to the computer program instructions:

[0145] Acquire the environment images continuously captured by the multi-eye camera 1004, and perform stitching processing on the environment images synchronously captured by the multi-eye camera 1004 to obtain a multi-frame stitching image;

[0146] For each stitched image, perform the following operations:

[0147] Determine the light intensity ratio of the stitched image according to the pixel value of the stitched image;

[0148] If the light intensity ratio is greater than a preset upper limit of the intensity ratio, a histogram equalization process with limited contrast is performed on the stitched image to reduce the image brightness;

[0149] If the light intensity ratio is less than or equal to the preset intensity ratio lower limit, the stitched image is subjected to image enhancement processing based on deep learning to increase the image brightness;

[0150] According to the acquisition order of the multi-frame stitching images, by changing the number of image channels, every n-frame stitching images in the multi-frame stitching images are stacked to obtain an input image set;

[0151] According to the input image set and combined with its own IMU data, the 6DOf posture of the user's head is determined, and the 6DOf posture is used to control the application interface displayed on the display 1003.

[0152] Optionally, the processor 1001 determines the light intensity ratio of the stitched image according to the pixel value of the stitched image, and the specific operation is:

[0153] Grayscale the stitched image and obtain the pixel mean and maximum pixel value after grayscale processing;

[0154] The ratio of the pixel mean and the maximum pixel value is used as the light intensity ratio of the stitched image.

[0155] Optionally, the processor 1001 performs a contrast-limited histogram equalization process on the stitched image to reduce image brightness, and the specific operation is:

[0156] The grayscaled stitched image is divided into M*M image blocks, and the cumulative probability distribution of each gray level is obtained according to the probability distribution corresponding to each gray level in each image block, where M is an integer greater than or equal to 2;

[0157] According to the cumulative probability distribution of each gray level in each image block, the number of pixels in the corresponding image block whose gray levels exceed the pixel number threshold is obtained, and the excess number of pixels is evenly distributed to each gray level to obtain a new cumulative probability distribution;

[0158] For the pixel values ​​of different position areas in the spliced ​​image, the new cumulative probability distribution of each image block is used for different mapping to obtain the new pixel value of grayscale histogram equalization;

[0159] According to the new pixel value of each image block, a spliced ​​image with reduced image brightness is obtained.

[0160] Optionally, the processor 1001 calculates a new pixel value of a corresponding image block using a new cumulative probability distribution according to the position of each image block in the stitched image. The specific operation is:

[0161] For the pixel values ​​in the four corners of the stitched image, the product of the new cumulative probability distribution of the target image block where the four corners are located and the maximum pixel value is used as the new pixel value in the four corners;

[0162] For the pixel values ​​in the edge area of ​​the stitched image, the new cumulative probability distribution of the target image block where the original pixel value is located and an adjacent image block of the target image block are multiplied by the product of the maximum pixel value to obtain two initial pixel values, and the center distance ratios of the original pixel value to the target image block and the adjacent image block are calculated, and the two initial pixel values ​​are combined to obtain the new pixel value in the edge area;

[0163] For the pixel values ​​in the middle area of ​​the stitched image, the new cumulative probability distribution of the target image block where the original pixel value is located and the three adjacent image blocks of the target image block are multiplied by the maximum pixel value to obtain four initial pixel values, and the center distance ratios of the original pixel value to the target image block and each adjacent image block are calculated. The new pixel value in the middle area is obtained by combining the four initial pixel values.

[0164] Optionally, the processor 1001 performs deep learning-based image enhancement processing on the stitched image to increase image brightness, and the specific operations are:

[0165] Through convolution processing, the spliced ​​image is decomposed into an initial reflection image and an initial illumination image;

[0166] According to the initial reflection image and the initial illumination image, an initial illumination image after noise removal is obtained;

[0167] According to the initial illumination image and initial reflection image after noise removal, a multi-scale brightness attention mechanism is used to restore the light reflectance information to obtain the target reflection image.

[0168] According to the illumination adjustment parameter corresponding to the illumination intensity ratio of the stitched image, the brightness of the initial illumination image after noise removal is increased to obtain the target illumination image;

[0169] The target illumination image and the target reflection image are multiplied pixel by pixel to obtain a stitched image with enhanced image brightness.

[0170] Optionally, the processor 1001 uses a multi-scale brightness attention mechanism to restore light reflectance information according to the initial illumination image and the initial reflection image after noise removal to obtain a target reflection image. The specific operation is:

[0171] After the initial reflection image is processed by multi-channel convolution and multi-channel activation function, it is multiplied pixel by pixel with the initial illumination image after noise removal to obtain a fused image.

[0172] After pooling the fused images with different numbers of pooling layers, a multi-scale brightness attention mechanism is used to perform convolution and deconvolution operations on the pooled images and convolution operations on the fused images to restore the light reflectance information.

[0173] The initial reflection image after convolution activation is adjusted according to the light reflectance information to obtain the target reflection image.

[0174] Optionally, the processor 1001 increases the brightness of the initial illumination image after noise removal according to the illumination adjustment parameter corresponding to the illumination intensity ratio of the stitched image to obtain the target illumination image. The specific operation is:

[0175] The illumination adjustment parameter is used as a constant channel to be spliced ​​with the initial illumination image after noise removal to obtain an illumination enhanced image;

[0176] The illumination enhanced image is processed by convolution and activation function to obtain the target illumination image.

[0177] Optionally, the formula for the lighting adjustment parameter is expressed as:

[0178]

[0179] Among them, α represents the illumination adjustment parameter, β represents the illumination intensity ratio of the stitched image, and P mean represents the pixel mean of the spliced ​​image, P max Indicates the maximum pixel value of the stitched image.

[0180] The memory 1002 may be a volatile memory, such as a random-access memory (RAM); the memory 1002 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1002 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1002 may be a combination of the above memories;

[0181] The processor 1001 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0182] It should be noted that Fig.10 This is just an example, and provides the necessary hardware for the virtual reality device to execute the steps of the positioning method of the virtual reality device provided in the embodiment of the present application. If not shown, the virtual reality device may also include conventional hardware such as a speaker, a pickup, a power supply, a communication interface, and an IMU.

[0183] The virtual reality device provided by the embodiment of the present application can make full use of the edge features on the environmental images collected by each camera separately by stitching the environmental images collected synchronously by multiple cameras into a stitched image, thereby improving the accuracy of the subsequent 6Dof pose calculation based on feature points. At the same time, stitching the environmental images collected synchronously by multiple cameras into a stitched image for processing can reduce the number of algorithm calls and improve the positioning speed. On the other hand, in order to reduce the impact of illumination changes on feature point extraction, it is determined whether the light intensity ratio determined by the pixel value of each frame of the stitched image is within a preset range, and different algorithms are used to adjust the image brightness for the overexposure and underexposure of the image that is not within the preset range, thereby improving the contrast of the stitched image in the illumination change scene, making the feature differences between different stitched images more obvious, effectively improving the feature distinction, and thus improving the accuracy and robustness of the 6Dof pose calculation based on feature points.

[0184] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the positioning method of the virtual reality device in the aforementioned embodiment.

[0185] The embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute the positioning method of the virtual reality device in the aforementioned embodiment.

[0186] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0187] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0188] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0190] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A positioning method for a virtual reality device, characterized in that: include: Acquire environmental images continuously captured by multiple cameras, and perform stitching processing on the environmental images synchronously captured by the multiple cameras to obtain multiple-frame stitching images; For each stitched image, perform the following operations: Determining a light intensity ratio of the stitched image according to a pixel value of the stitched image; If the light intensity ratio is greater than a preset intensity ratio upper limit, performing a contrast-limited histogram equalization process on the stitched image to reduce image brightness; If the light intensity ratio is less than or equal to a preset intensity ratio lower limit, performing deep learning-based image enhancement processing on the stitched image to increase image brightness; According to the acquisition order of the multiple-frame stitched images, by changing the number of image channels, every n-frame stitched images in the multiple-frame stitched images are stacked to obtain an input image set, where n is an integer greater than 1; According to the input image set and combined with its own IMU data, the 6DOf pose of the user's head is determined.

2. The method according to claim 1, characterized in that Determining the light intensity ratio of the stitched image according to the pixel value of the stitched image includes: Performing grayscale processing on the stitched image to obtain a pixel mean value and a maximum pixel value after the grayscale processing; The ratio of the pixel mean value to the maximum pixel value is used as the light intensity ratio of the stitched image.

3. The method according to claim 1, characterized in that The step of performing a contrast-limited histogram equalization process on the stitched image to reduce image brightness includes: Divide the grayscaled stitched image into M*M image blocks, and obtain the cumulative probability distribution of each gray level according to the probability distribution corresponding to each gray level in each image block, where M is an integer greater than or equal to 2; According to the cumulative probability distribution of each gray level in each image block, the number of pixels in the corresponding image block whose gray levels exceed the pixel number threshold is obtained, and the exceeding number of pixels is evenly distributed to each gray level to obtain a new cumulative probability distribution; For pixel values ​​in different position areas in the spliced ​​image, different mappings are performed using the new cumulative probability distribution of each image block to obtain new pixel values ​​of grayscale histogram equalization; According to the new pixel value of each image block, a spliced ​​image with reduced image brightness is obtained.

4. The method according to claim 3, characterized in that The step of calculating a new pixel value of a corresponding image block by using a new cumulative probability distribution according to a position of each image block in the spliced ​​image comprises: For the pixel values ​​in the four corner regions of the stitched image, the product of the new cumulative probability distribution of the target image block where the four corner regions are located and the maximum pixel value is taken as the new pixel value in the four corner regions; For the pixel values ​​in the edge area of ​​the stitched image, the new cumulative probability distributions of the target image block where the original pixel value is located and an adjacent image block of the target image block are respectively multiplied by the product of the maximum pixel value to obtain two initial pixel values, and the center distance ratios of the original pixel value to the target image block and the adjacent image block are calculated, and the new pixel value in the edge area is obtained by combining the two initial pixel values; For the pixel values ​​in the middle area of ​​the stitched image, the new cumulative probability distributions of the target image block where the original pixel value is located and the three adjacent image blocks of the target image block are multiplied by the maximum pixel value respectively to obtain four initial pixel values, and the center distance ratios of the original pixel value to the target image block and each adjacent image block are calculated, and the new pixel value in the middle area is obtained by combining the four initial pixel values.

5. The method according to claim 1, characterized in that The performing deep learning-based image enhancement processing on the stitched image to increase image brightness includes: Decomposing the stitched image into an initial reflection image and an initial illumination image by convolution processing; Obtaining an initial illumination image after noise removal according to the initial reflection image and the initial illumination image; According to the initial illumination image after noise removal and the initial reflection image, a multi-scale brightness attention mechanism is used to restore the light reflectance information to obtain a target reflection image; According to the illumination adjustment parameter corresponding to the illumination intensity ratio of the stitched image, the initial illumination image after noise removal is adjusted to increase the brightness to obtain a target illumination image; The target illumination image and the target reflection image are multiplied pixel by pixel to obtain a spliced ​​image with enhanced image brightness.

6. The method according to claim 5, characterized in that The method of restoring light reflectance information by using a multi-scale brightness attention mechanism based on the initial illumination image after noise removal and the initial reflection image to obtain a target reflection image includes: After performing a multi-channel convolution operation and a multi-channel activation function processing on the initial reflection image, the initial reflection image is multiplied pixel by pixel with the initial illumination image after noise removal to obtain a fused image; After performing pooling processing on the fused images respectively using pooling layers with different numbers of layers, a multi-scale brightness attention mechanism is used to perform convolution and deconvolution operations on the pooled images respectively and a convolution operation is performed on the fused images to restore light reflectance information; The initial reflection image after convolution activation is adjusted according to the light reflectivity information to obtain a target reflection image.

7. The method according to claim 6, characterized in that The step of increasing the brightness of the initial illumination image after noise removal according to the illumination adjustment parameter corresponding to the illumination intensity ratio of the spliced ​​image to obtain the target illumination image includes: The illumination adjustment parameter is used as a constant channel to be spliced ​​with the initial illumination image after noise removal to obtain an illumination enhanced image; The illumination enhancement image is subjected to convolution and activation function processing to obtain a target illumination image.

8. The method according to any one of claims 5 to 7, characterized in that The formula of the illumination adjustment parameter is expressed as: α＝(0.5-β)+1， Wherein, α represents the illumination adjustment parameter, β represents the illumination intensity ratio of the spliced ​​image, and P mean represents the pixel mean of the stitched image, P max Indicates the maximum pixel value of the stitched image.

9. A virtual reality device, characterized in that: It includes a processor, a memory, a display and a multi-eye camera, wherein the multi-eye camera, the display, the memory and the processor are connected via a bus: The memory stores computer program instructions, and the processor performs the following operations according to the computer program instructions: Acquire the environment images continuously captured by the multi-eye camera, and perform stitching processing on the environment images synchronously captured by the multi-eye camera to obtain a multi-frame stitching image; For each stitched image, perform the following operations: Determining a light intensity ratio of the stitched image according to a pixel value of the stitched image; If the light intensity ratio is greater than a preset intensity ratio upper limit, performing a contrast-limited histogram equalization process on the stitched image to reduce image brightness; If the light intensity ratio is less than or equal to a preset intensity ratio lower limit, performing deep learning-based image enhancement processing on the stitched image to increase image brightness; According to the acquisition order of the multiple-frame stitched images, by changing the number of image channels, every n-frame stitched images in the multiple-frame stitched images are stacked to obtain an input image set, where n is an integer greater than 1; The 6DOf posture of the user's head is determined according to the input image set and combined with its own IMU data, and the 6DOf posture is used to control the application interface displayed on the display.

10. The virtual reality device according to claim 9, characterized in that: The processor determines the light intensity ratio of the stitched image according to the pixel value of the stitched image, and the specific operation is: Performing grayscale processing on the stitched image to obtain a pixel mean value and a maximum pixel value after the grayscale processing; The ratio of the pixel mean value to the maximum pixel value is used as the light intensity ratio of the stitched image.

Images