Positioning mapping method and device, near-to-eye display equipment and storage medium

Abstract

The invention relates to the field of augmented reality, and provides a positioning mapping method, device and equipment for intelligent glasses and a computer storage medium, and the method comprises the steps: obtaining original visual information shot by a shooting device and sensor information of a position sensor; expanding the original visual information based on a time dimension to obtain first visual information; expanding the original visual information according to the sensor information based on a spatial dimension to obtain second visual information; and fusing the original visual information, the first visual information and the second visual information to obtain global image information of an environment where the intelligent glasses are located. By generating the first visual information and the second visual information, effective utilization of the original visual information is realized, so that the visual map of the surrounding environment is constructed more accurately.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a positioning and mapping method, apparatus, near-eye display device, and storage medium. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) stands for real-time positioning and mapping. For example, in the field of augmented reality (AR), it can complete positioning (Localization) and mapping (Mapping) in an unknown environment, so that the near-eye display device can perceive the surrounding environment in order to arrange the content to be displayed reasonably. However, the positioning and mapping technology in the related art is easily blocked and interfered by factors such as weather, lighting, and terrain when acquiring environmental images, resulting in poor data collection quality, inability to fully utilize the collected data, and affecting the accuracy of subsequent positioning and mapping. Summary of the invention

[0003] The main purpose of this application is to provide a positioning and mapping method, device, near-eye display device and storage medium, aiming to reduce the impact of external interference on original visual information.

[0004] In a first aspect, the present application provides a positioning mapping method, the positioning mapping method is applied to a near-eye display device, the near-eye display device is provided with a shooting device and a position sensor, and the positioning mapping method comprises the following steps:

[0005] Acquiring original visual information captured by the camera and sensor information of the position sensor;

[0006] Expanding the original visual information based on a time dimension to obtain first visual information;

[0007] Expanding the original visual information according to the sensor information based on the spatial dimension to obtain second visual information;

[0008] The original visual information, the first visual information, and the second visual information are fused to obtain global image information of the environment in which the near-eye display device is located.

[0009] In a second aspect, the present application further provides a positioning and mapping device, which is applied to a near-eye display device, and the positioning and mapping device includes:

[0010] An information acquisition module, used to acquire original visual information captured by the camera and sensor information of the position sensor;

[0011] A first expansion module, used to expand the original visual information based on a time dimension to obtain first visual information;

[0012] A second expansion module, configured to expand the original visual information according to the sensor information based on a spatial dimension to obtain second visual information;

[0013] An information fusion module is used to fuse the original visual information, the first visual information, and the second visual information to obtain global image information of the environment in which the near-eye display device is located.

[0014] In a third aspect, the present application also provides a near-eye display device, comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, the positioning and mapping method as described above is implemented.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the positioning and mapping method as described above is implemented.

[0016] The present application provides a positioning and mapping method, device, equipment and computer storage medium. The present application obtains the original visual information captured by the shooting device and the sensor information of the position sensor; expands the original visual information based on the time dimension to obtain the first visual information; expands the original visual information based on the sensor information based on the space dimension to obtain the second visual information; fuses the original visual information, the first visual information and the second visual information to obtain the global image information of the environment where the near-eye display device is located. By generating the first visual information and the second visual information, the impact of external interference on the original visual information is reduced, the utilization rate of the original visual information is improved, and the global image information of the surrounding environment is constructed more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a positioning and mapping method provided in one embodiment of the present application;

[0019] Figure 2 A schematic block diagram of a positioning and mapping method provided in an embodiment of the present application;

[0020] Figure 3a An image corresponding to the original visual information provided by an embodiment of the present application;

[0021] Figure 3b An image corresponding to the second visual information provided by an embodiment of the present application;

[0022] Figure 3c A schematic diagram of a matching result of matching original visual information with second visual information provided in an embodiment of the present application;

[0023] Figure 4 A schematic block diagram of a positioning and mapping device provided in one embodiment of the present application;

[0024] Figure 5 This is a schematic block diagram of the structure of a near-eye display device involved in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] Embodiments of the present application provide a positioning and mapping method, apparatus, near-eye display device, and computer-readable storage medium.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A flowchart of a positioning and mapping method provided in an embodiment of the present application. The positioning and mapping method can be used in a near-eye display device, and positioning and mapping are performed based on the original visual information captured by the camera of the near-eye display device and the position information collected by the position sensor.

[0030] Please refer to Figure 2 , Figure 2 This is a usage scenario diagram provided by an embodiment of the present application. Figure 2As shown, original visual information and sensor information are obtained through a shooting device and a position sensor respectively, and first visual information and second visual information are generated based on the two, so as to fuse and obtain global image information.

[0031] like Figure 1 As shown, the positioning and mapping method includes steps S101 to S104.

[0032] Step S101: Acquire original visual information captured by the shooting device and sensor information of the position sensor.

[0033] Exemplarily, the near-eye display device provided in the embodiment of the present application may be, for example, smart glasses, on which a shooting device and a position sensor are installed, wherein the shooting device may be, for example, a visual camera for acquiring a two-dimensional image of the surrounding environment, and the position sensor may include, for example, a depth sensor for measuring the distance between an object and the sensor. Of course, this is not limited to this, and the visual camera and the depth sensor may also be mounted on the same device, such as a depth camera, which is not limited here.

[0034] For example, SLAM technology is a commonly used technology in AR glasses, which can locate its own position in an unknown environment and build an environmental map to ensure the geometric consistency of virtual objects superimposed on the real environment. Among them, visual positioning is crucial in SLAM. Visual positioning technology captures the environment in advance, extracts visual features and matches them, so that AR glasses can understand the environment they are in, and accurately superimpose virtual information in the real world, so that users can interact with virtual objects naturally and intuitively, enhancing user experience. Visual positioning is based on the visual information obtained by the shooting device. However, there may be various forms of interference in the actual environment, such as insufficient lighting, occlusion, etc., resulting in the loss of visual information.

[0035] Exemplarily, in order to effectively utilize the visual information acquired by the camera, the positioning and mapping method provided in the embodiment of the present application acquires the original visual information captured by the camera of the near-eye display device and the sensor information of the position sensor, so as to subsequently process these two types of information and more accurately construct a visual map of the surrounding environment. Among them, the original visual information, for example, includes a video stream captured for the surrounding environment, and the sensor information, for example, includes acceleration information, angular velocity information, etc. collected by a gyroscope.

[0036] Step S102: Expand the original visual information based on the time dimension to obtain first visual information.

[0037] Exemplarily, for the disturbed part of the original visual information captured by the shooting device, the content of this part can be supplemented based on the video generation algorithm, for example, the occluded part is predicted based on the original visual information to obtain the first visual information, thereby improving the integrity of the visual information. Among them, the method for determining whether the original visual information is disturbed is not limited here. Specifically, the edge detection and variance analysis algorithm can be used to determine whether there is interference in the original video information. When the video screen is blocked, the corresponding area is usually blurred, the edge information is reduced, and the pixel variance of the area is also reduced. By performing edge detection on two adjacent frames and calculating the variance of the two, it can be determined whether there is occlusion; or the original visual information can also be detected for brightness anomalies. If there is a large area of ​​occlusion on the monitoring area screen due to human or non-human factors in the video screen, these areas usually show brightness abnormalities. By detecting the brightness changes of the image, these abnormal areas can be identified.

[0038] Exemplarily, the video generation algorithm can

[0039] In some implementations, the expanding the original visual information based on the time dimension to obtain the first visual information includes:

[0040] Identifying an original video frame in the original visual information;

[0041] In the case where it is detected that there is an interfered video frame in the original visual information, first visual information for replacing the interfered video frame is generated according to a context video frame of the interfered video frame.

[0042] Exemplarily, the method for identifying the original video frame in the original visual information can adopt the frame difference method, for example, by performing a differential operation on two adjacent frames in the video image sequence, calculating the brightness difference between the two frames, and judging whether the brightness difference is greater than a threshold, judging whether the video frame is disturbed by strong light or shadow. Specifically, if the brightness of the video frame changes by more than a threshold, it means that strong light or shadow may appear in the video frame; an occlusion rate detection algorithm can also be adopted, by binarizing the color image, the dark part is the foreground, and the other part is the background. Connected area detection is performed on the foreground to obtain the maximum connected area area, and compared with the area of ​​the entire image to obtain the occlusion rate, and the video frame is judged to be occluded when the occlusion rate is greater than the preset occlusion rate. Of course, it is not limited to this, and it is also possible to detect whether there is an interfered video frame in the original visual information based on a deep learning algorithm, etc., which is not limited here.

[0043] Exemplarily, when it is determined that there is an interfered video frame in the original visual information, the first visual information for replacing the interfered video frame is generated based on the context video frame of the interfered video frame based on the time dimension, thereby improving the targetedness of video generation and avoiding occupying too many computing resources for generating the first visual information.

[0044] In some implementations, the generating, according to the context video frame of the disturbed video frame, the first visual information for replacing the disturbed video frame comprises:

[0045] Input the context video frame into a preset generator, and obtain a video frame to be determined output by the generator according to the context video frame;

[0046] Based on a preset discriminator, the video frame to be discriminated is discriminated, and the parameters of the generator are adjusted according to the discrimination result of the discriminator;

[0047] Based on the generator, the first visual information is generated according to a context video frame of the disturbed video frame.

[0048] Exemplarily, a Generative Adversarial Networks (GAN) is used to generate the first video information. Specifically, GAN generates the first visual information similar to the original visual information by training a generator and a discriminator. The goal of the generator is to generate a video frame to be identified that is as realistic as possible. It is usually a deep neural network that can generate data from random noise. The output of the generator is a video frame to be identified that is similar to the context video frame; while the goal of the discriminator is to distinguish the video frame to be identified generated by the generator from the real context video frame. It is also a deep neural network that usually uses a different structure from the generator. The output of the discriminator is a probability value that represents the probability that the input video frame to be identified is real data. Through the continuous confrontation between the generator and the discriminator during the training process, the generator attempts to generate more and more realistic video frames to be identified, while the discriminator attempts to more accurately distinguish the context video frame from the video frame to be identified. This confrontation process promotes the performance improvement of the generator and the discriminator, so that the generator can generate as realistic video frames to be identified as possible to replace the disturbed video frame.

[0049] For example, the value function of the adversarial network can be optimized: Among them, V(D,G) is the value function of the adversarial network,

[0050] x is a sample randomly drawn from the actual video frame distribution of the original visual information, i.e., the real context video frame, p data(x) represents the distribution of the actual video frames, that is, the distribution of the training data set, x~p data (x) represents x sampled from the actual video frame distribution; z is a random sample drawn from the noise distribution of the input generator, p z (z) represents the distribution of the noise vector z input to the generator, usually a standard normal distribution, z~p z (z) represents z sampled from the noise distribution; D(x) is the output of the discriminator for the real sample x, indicating the probability that the sample x is real; D(G(z)) is the output of the discriminator for the sample G(z) generated by the generator, indicating the probability that the sample G(z) is real; the first item is the expected loss of the discriminator on the actual video frame, which encourages the discriminator to correctly identify the actual video frame; the second term is the expected loss of the discriminator on the generated video frames to be judged, which encourages the discriminator to correctly identify the generated video frames to be judged as fake; It means that the goal of the generator is to minimize the value function, that is, the generator hopes that the generated video frames to be judged can deceive the discriminator as much as possible, so that the discriminator can judge these generated video frames to be judged as real data; This means that the discriminator's goal is to maximize the value function, that is, the discriminator hopes to distinguish the actual video frames from the video frames to be discriminated as much as possible. During the training process, an alternating optimization method is usually used. The generator G is fixed while the discriminator D is optimized, and the generator G is optimized while the discriminator D is fixed. The generator and the discriminator are constantly confronted during the training process, and finally reach a balance state. The generator can generate realistic video frames to be discriminated, and the discriminator can distinguish between the actual video frames and the video frames to be discriminated, so that the first visual information generated by the adversarial network achieves the best effect.

[0051] Through the above method, a generator with a certain accuracy is obtained, which is used to generate the first visual information that can be used to replace the interfered video frame according to the context video frame of the interfered video frame in the original visual information.

[0052] In some implementations, the generating, according to the context video frame of the disturbed video frame, the first visual information for replacing the disturbed video frame comprises:

[0053] The context video frame is input into a preset temporal convolutional network, and the interfered video frame is predicted to obtain the first visual information.

[0054] Exemplarily, the first visual information can also be generated based on a temporal convolutional network (TCN). The convolutional layer of the TCN can capture local temporal patterns in time series, and can learn more complex temporal dependencies by stacking multiple convolutional layers. It can process the temporal information in the video data for prediction tasks related to the time dimension and predict the interfered video frame based on the context video frame.

[0055] Specifically, the length of each sequence in which the TCN model processes the original visual information is preset as the time step t. For example, each time step t can be 100 frames of original visual information. For each time step t, the output y(t) of the temporal convolutional layer of the TCN model can be calculated as follows: Among them, x(t) is the input value at time step t, y(t) is the output value at time step t, w(k) is the weight of the convolution kernel at position k, and K is the size of the convolution kernel, that is, the number of time steps covered by the convolution kernel. By stacking multiple such convolutional layers, and possibly combining dilated convolutions and residual connections, TCN is able to learn complex time series data patterns and achieve the generation of first visual information.

[0056] Step S103: Expand the original visual information based on the sensor information based on the spatial dimension to obtain second visual information.

[0057] Exemplarily, the angle of the original visual information obtained in step S101 is relatively single. In order to enrich the amount of information in the visual information, second visual information at other angles may be generated according to the original visual information based on the spatial dimension.

[0058] In some implementations, the expanding the original visual information based on the sensor information based on the spatial dimension to obtain the second visual information includes:

[0059] Acquire original key frames in the original visual information;

[0060] According to the sensor information and the original key frame, the original key frame is transformed in perspective to obtain the second visual information under a preset perspective.

[0061] For example, the original visual information usually contains a large number of frames. Detailed processing of each frame will consume a lot of computing resources. By filtering key frames, the number of frames that need to be processed can be reduced, thereby reducing the computing cost. The changes between key frames are usually more significant than those between ordinary frames. By filtering out the original key frames in the original visual information, the main changes in the video can be captured more effectively, improving the efficiency of the generation process.

[0062] Exemplarily, the original visual information is expanded based on the spatial dimension, that is, the same object photographed from a different perspective from the original visual information is generated, so as to more comprehensively determine the shape of the object in the field of view, which is conducive to improving the accuracy of positioning and mapping. Specifically, the first visual information generated in the time dimension is essentially a prediction of visual information with a different shooting time from the original visual information, and the second visual information generated in the spatial dimension is essentially a prediction of visual information with the same shooting time and different shooting angle as the original visual information, that is, the shooting perspective of the object is changed in the spatial dimension, so that the multi-angle visual information of the object is obtained through the second visual information.

[0063] In some implementations, the sensor information includes original position information and depth information; and the perspective transformation is performed on the original key frame according to the sensor information and the original key frame, and the second visual information under a preset perspective includes:

[0064] Based on a preset perspective transformation matrix, determining target position information of the second visual information at the preset perspective according to the depth information and the original position information;

[0065] Based on a preset multi-view diffusion model, the corresponding second visual information is generated according to the original key frame and the target position information.

[0066] Exemplarily, the perspective transformation matrix is ​​pre-set Where R is a 3x3 rotation matrix, T is a 3x1 translation vector, and the last row [0 0 0 1] is a standardized row in the homogeneous coordinate system. R and T can be set according to the actual needs of the perspective transformation and are not limited here.

[0067] Specifically, the depth information D(x, y) of the original key frame is obtained, and the coordinates (x′, y′, z′) = T·(x, y, D(x, y)) under the new perspective are determined by using the perspective transformation algorithm, where (x, y, D(x, y)) are the coordinates under the original perspective, and the second video information under the new perspective is generated through a multi-view diffusion model, where the multi-view diffusion model can be established based on a deep learning model, such as a convolutional neural network (CNN) or a variational autoencoder (VAE), etc., but is of course not limited to this and is not limited here.

[0068] Step S104: fuse the original visual information, the first visual information, and the second visual information to obtain global image information of the environment in which the near-eye display device is located.

[0069] Exemplarily, the original visual information collected by the camera device and the first visual information and the second visual information generated according to the original visual information are fused to obtain global image information with more environmental information.

[0070] In some embodiments, the fusing the original visual information, the first visual information, and the second visual information to obtain global image information of the environment in which the near-eye display device is located includes:

[0071] Matching the first visual information and the second visual information with the original video information respectively to obtain a first matching degree between the first visual information and the original video information and a second matching degree between the second visual information and the original video information;

[0072] Determining the first visual information that meets the preset condition with the first matching degree and the second visual information that meets the preset condition with the second matching degree as target visual information;

[0073] The target visual information is fused with the original visual information to obtain global image information of the environment in which the near-eye display device is located.

[0074] Exemplarily, the generated first visual information and second visual information can be screened. Specifically, edges, corners or significant texture areas in the environment are used as feature points to help match the first visual information and the second visual information with the original visual information. Please refer to Figure 3a-3c , Figure 3a-3c A schematic diagram of matching results of matching visual information provided by an embodiment of the present application. Figure 3a is the original visual information, Figure 3b The second visual information from other perspectives generated based on the spatial dimension is matched by a matching algorithm, and the matching degree is determined according to the matching results of the feature points. The first visual information and the second visual information with insufficient matching degree are filtered out to obtain the target visual information.

[0075] For example, after obtaining the target visual information, the target visual information is fused with the original visual information according to the matching result. Assuming that the coordinate transformation matrix The visual information of cam1 can be converted to the visual information of cam0, then Connect visual information from different angles, Represents the coordinate transformation matrix from the cam1 coordinate system to the world coordinate system. Represents the coordinate transformation matrix from the body coordinate system (usually the carrier coordinate system of the smart camera) to the world coordinate system. Represents the coordinate transformation matrix from the cam0 coordinate system to the body coordinate system, It represents the transformation matrix from the cam1 coordinate system to the cam0 coordinate system. It can be understood that the above coordinate system can be the coordinate system of the original visual information actually captured, or the coordinate system of the generated target visual information, which is not limited here.

[0076] For example, by fusing the target visual information with the original visual information in the same coordinate system, a common view association relationship is established and the cumulative error is reduced, the amount of information in each sub-map is increased, thereby increasing the amount of information in the global image information and improving the robustness of positioning and mapping.

[0077] The positioning and mapping method provided in the above embodiment obtains the original visual information captured by the shooting device and the sensor information of the position sensor; expands the original visual information based on the time dimension to obtain the first visual information; expands the original visual information according to the sensor information based on the space dimension to obtain the second visual information; and fuses the original visual information, the first visual information, and the second visual information to obtain the global image information of the environment in which the near-eye display device is located. By generating the first visual information and the second visual information, the original visual information is effectively utilized, thereby more accurately constructing a visual map of the surrounding environment.

[0078] See also Figure 4 , Figure 4 is a schematic diagram of a positioning mapping device provided in an embodiment of the present application. The positioning mapping device can be configured in a near-eye display device to execute the aforementioned positioning mapping method.

[0079] like Figure 4 As shown, the positioning and mapping device includes: an information acquisition module 110 , a first expansion module 120 , a second expansion module 130 , and an information fusion module 140 .

[0080] An information acquisition module 110, used to acquire original visual information captured by the camera and sensor information of the position sensor;

[0081] A first expansion module 120, configured to expand the original visual information based on a time dimension to obtain first visual information;

[0082] A second expansion module 130, configured to expand the original visual information according to the sensor information based on a spatial dimension to obtain second visual information;

[0083] The information fusion module 140 is used to fuse the original visual information, the first visual information, and the second visual information to obtain global image information of the environment in which the near-eye display device is located.

[0084] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0085] The method and apparatus of the present application can be used in many general or special computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0086] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 5 The device is shown running on a near-eye display.

[0087] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a near-eye display device provided in an embodiment of the present application. The near-eye display device may be a server or a terminal.

[0088] like Figure 5 As shown, the near-eye display device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0089] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any positioning and mapping method.

[0090] The processor is used to provide computing and control capabilities to support the operation of the entire near-eye display device.

[0091] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any positioning and mapping method.

[0092] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the near-eye display device to which the scheme of the present application is applied. The specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0094] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0095] Acquiring original visual information captured by the camera and sensor information of the position sensor;

[0096] Expanding the original visual information based on a time dimension to obtain first visual information;

[0097] Expanding the original visual information according to the sensor information based on the spatial dimension to obtain second visual information;

[0098] The original visual information, the first visual information, and the second visual information are fused to obtain global image information of the environment in which the near-eye display device is located.

[0099] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of positioning and mapping described above can refer to the corresponding process in the aforementioned positioning and mapping control method embodiment, and will not be repeated here.

[0100] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the positioning and mapping method of the present application.

[0101] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0102] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0103] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0104] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A positioning and mapping method, characterized in that: The positioning and mapping method is applied to a near-eye display device, wherein the near-eye display device is provided with a shooting device and a position sensor, and the method comprises: Acquiring original visual information captured by the camera and sensor information of the position sensor; Expanding the original visual information based on a time dimension to obtain first visual information; Expanding the original visual information according to the sensor information based on the spatial dimension to obtain second visual information; The original visual information, the first visual information, and the second visual information are fused to obtain global image information of the environment in which the near-eye display device is located.

2. The positioning and mapping method according to claim 1, characterized in that: The step of expanding the original visual information based on the time dimension to obtain the first visual information includes: Identifying an original video frame in the original visual information; In the case where it is detected that there is an interfered video frame in the original visual information, first visual information for replacing the interfered video frame is generated according to a context video frame of the interfered video frame.

3. The positioning and mapping method according to claim 2, characterized in that: The step of generating first visual information for replacing the disturbed video frame according to the context video frame of the disturbed video frame comprises: Input the context video frame into a preset generator, and obtain a video frame to be determined output by the generator according to the context video frame; Based on a preset discriminator, the video frame to be discriminated is discriminated, and the parameters of the generator are adjusted according to the discrimination result of the discriminator; Based on the generator with adjusted parameters, the first visual information is generated according to the context video frame of the disturbed video frame.

4. The positioning and mapping method according to claim 2, characterized in that: The step of generating first visual information for replacing the disturbed video frame according to the context video frame of the disturbed video frame comprises: The context video frame is input into a preset temporal convolutional network, and the interfered video frame is predicted to obtain the first visual information.

5. The positioning and mapping method according to claim 1, characterized in that: The step of expanding the original visual information based on the sensor information based on the spatial dimension to obtain the second visual information includes: Acquire original key frames in the original visual information; According to the sensor information and the original key frame, the original key frame is transformed in perspective to obtain the second visual information under a preset perspective.

6. The positioning and mapping method according to claim 5, characterized in that: The sensor information includes original position information and depth information; The step of performing a perspective transformation on the original key frame according to the sensor information and the original key frame to obtain the second visual information at a preset perspective includes: Based on a preset perspective transformation matrix, determining target position information of the second visual information at the preset perspective according to the depth information and the original position information; Based on a preset multi-view diffusion model, the corresponding second visual information is generated according to the original key frame and the target position information.

7. The positioning and mapping method according to any one of claims 1 to 6, characterized in that: The fusing the original visual information, the first visual information, and the second visual information to obtain global image information of the environment in which the near-eye display device is located includes: Matching the first visual information and the second visual information with the original video information respectively to obtain a first matching degree between the first visual information and the original video information and a second matching degree between the second visual information and the original video information; Determining the first visual information that meets the preset condition with the first matching degree and the second visual information that meets the preset condition with the second matching degree as target visual information; The target visual information is fused with the original visual information to obtain global image information of the environment in which the near-eye display device is located.

8. A positioning and mapping device, characterized in that: The positioning and mapping device is applied to a near-eye display device, comprising: An information acquisition module, used to acquire original visual information captured by the camera and sensor information of the position sensor; A first expansion module, used to expand the original visual information based on a time dimension to obtain first visual information; A second expansion module, configured to expand the original visual information according to the sensor information based on a spatial dimension to obtain second visual information; An information fusion module is used to fuse the original visual information, the first visual information, and the second visual information to obtain global image information of the environment in which the near-eye display device is located.

9. A near-eye display device, characterized in that: The near-eye display device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the positioning and mapping method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the positioning and mapping method according to any one of claims 1 to 7 are implemented.

Images